class: title, self-paced Intensive Kubernetes<br/>Advanced Concepts<br/><br/>Application Configuration<br/>and Stateful Applications<br/> .nav[*Self-paced version*] .debug[ ``` ``` These slides have been built from commit: 474fe43e [shared/title.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/shared/title.md)] --- class: title, in-person Intensive Kubernetes<br/>Advanced Concepts<br/><br/>Application Configuration<br/>and Stateful Applications<br/><br/></br> .footnote[ **Slides[:](https://www.youtube.com/watch?v=h16zyxiwDLY) https://2020-05-ardan.container.training/** ] <!-- WiFi: **Something**<br/> Password: **Something** **Be kind to the WiFi!**<br/> *Use the 5G network.* *Don't use your hotspot.*<br/> *Don't stream videos or download big files during the workshop*<br/> *Thank you!* --> .debug[[shared/title.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/shared/title.md)] --- ## Intros - Hello! We are: - .emoji[👷🏻‍♀️] AJ ([@s0ulshake], [EphemeraSearch]) - .emoji[🐳] Jérôme ([@jpetazzo], Ardan Labs) - The training will run for 4 hours, with a 10 minutes break every hour (the middle break will be a bit longer) - Feel free to ask questions at any time - *Especially when you see full screen container pictures!* - Live feedback, questions, help: [Gitter](https://gitter.im/jpetazzo/training-20200504-online) [EphemeraSearch]: https://ephemerasearch.com/ [@s0ulshake]: https://twitter.com/s0ulshake [@jpetazzo]: https://twitter.com/jpetazzo .debug[[logistics.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/logistics.md)] --- ## A brief introduction - This was initially written by [Jérôme Petazzoni](https://twitter.com/jpetazzo) to support in-person, instructor-led workshops and tutorials - Credit is also due to [multiple contributors](https://github.com/jpetazzo/container.training/graphs/contributors) — thank you! - You can also follow along on your own, at your own pace - We included as much information as possible in these slides - We recommend having a mentor to help you ... - ... Or be comfortable spending some time reading the Kubernetes [documentation](https://kubernetes.io/docs/) ... - ... And looking for answers on [StackOverflow](http://stackoverflow.com/questions/tagged/kubernetes) and other outlets .debug[[k8s/intro.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/intro.md)] --- class: self-paced ## Hands on, you shall practice - Nobody ever became a Jedi by spending their lives reading Wookiepedia - Likewise, it will take more than merely *reading* these slides to make you an expert - These slides include *tons* of exercises and examples - They assume that you have access to a Kubernetes cluster - If you are attending a workshop or tutorial: <br/>you will be given specific instructions to access your cluster - If you are doing this on your own: <br/>the first chapter will give you various options to get your own cluster .debug[[k8s/intro.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/intro.md)] --- ## Accessing these slides now - We recommend that you open these slides in your browser: https://2020-05-ardan.container.training/ - Use arrows to move to next/previous slide (up, down, left, right, page up, page down) - Type a slide number + ENTER to go to that slide - The slide number is also visible in the URL bar (e.g. .../#123 for slide 123) .debug[[shared/about-slides.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/shared/about-slides.md)] --- ## Accessing these slides later - Slides will remain online so you can review them later if needed (let's say we'll keep them online at least 1 year, how about that?) - You can download the slides using that URL: https://2020-05-ardan.container.training/slides.zip (then open the file `k8s-adv-4.yml.html`) - You will to find new versions of these slides on: https://container.training/ .debug[[shared/about-slides.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/shared/about-slides.md)] --- ## These slides are open source - You are welcome to use, re-use, share these slides - These slides are written in markdown - The sources of these slides are available in a public GitHub repository: https://github.com/jpetazzo/container.training - Typos? Mistakes? Questions? Feel free to hover over the bottom of the slide ... .footnote[.emoji[👇] Try it! The source file will be shown and you can view it on GitHub and fork and edit it.] <!-- .exercise[ ```open https://github.com/jpetazzo/container.training/tree/master/slides/common/about-slides.md``` ] --> .debug[[shared/about-slides.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/shared/about-slides.md)] --- class: extra-details ## Extra details - This slide has a little magnifying glass in the top left corner - This magnifying glass indicates slides that provide extra details - Feel free to skip them if: - you are in a hurry - you are new to this and want to avoid cognitive overload - you want only the most essential information - You can review these slides another time if you want, they'll be waiting for you ☺ .debug[[shared/about-slides.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/shared/about-slides.md)] --- ## Use the chat! - We have set up a chat room on [Gitter](https://gitter.im/jpetazzo/training-20200504-online) (clicking the link above will take you to the chat room) - Don't hesitate to use it to ask questions, or get help, or share feedback - We will *not* use the Zoom chat room for Q&A (we've tried it in past training sessions, and participants preferred [Gitter](https://gitter.im/jpetazzo/training-20200504-online)) - Feel free to ask questions at any time - Sometimes we will wait a bit to answer ... ... but don't worry, we'll make sure to address all your questions! .debug[[shared/chat-room-zoom-webinar.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/shared/chat-room-zoom-webinar.md)] --- ## Use non-verbal communication cues - ... wait, what?!? -- - In the chat room, you are welcome (even encouraged!) to use emojis! - Some of our favorites: .emoji[🤔✔️👍🏻👍🏼👍🏽👍🏾👍🏿⚠️🛑] - During the session, we'll often ask audience participation questions - Feel free to answer in the chat room, any way you like! (short message, emoji reaction ...) .debug[[shared/chat-room-zoom-webinar.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/shared/chat-room-zoom-webinar.md)] --- name: toc-module-0 ## Table of contents - [Volumes](#toc-volumes) - [Managing configuration](#toc-managing-configuration) - [Stateful sets](#toc-stateful-sets) - [Running a Consul cluster](#toc-running-a-consul-cluster) - [Persistent Volumes Claims](#toc-persistent-volumes-claims) - [Local Persistent Volumes](#toc-local-persistent-volumes) - [Highly available Persistent Volumes](#toc-highly-available-persistent-volumes) .debug[(auto-generated TOC)] .debug[[shared/toc.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/shared/toc.md)] --- class: pic .interstitial[] --- name: toc-volumes class: title Volumes .nav[ [Previous section](#toc-) | [Back to table of contents](#toc-module-0) | [Next section](#toc-managing-configuration) ] .debug[(automatically generated title slide)] --- # Volumes - Volumes are special directories that are mounted in containers - Volumes can have many different purposes: - share files and directories between containers running on the same machine - share files and directories between containers and their host - centralize configuration information in Kubernetes and expose it to containers - manage credentials and secrets and expose them securely to containers - store persistent data for stateful services - access storage systems (like Ceph, EBS, NFS, Portworx, and many others) .debug[[k8s/volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/volumes.md)] --- class: extra-details ## Kubernetes volumes vs. Docker volumes - Kubernetes and Docker volumes are very similar (the [Kubernetes documentation](https://kubernetes.io/docs/concepts/storage/volumes/) says otherwise ... <br/> but it refers to Docker 1.7, which was released in 2015!) - Docker volumes allow us to share data between containers running on the same host - Kubernetes volumes allow us to share data between containers in the same pod - Both Docker and Kubernetes volumes enable access to storage systems - Kubernetes volumes are also used to expose configuration and secrets - Docker has specific concepts for configuration and secrets <br/> (but under the hood, the technical implementation is similar) - If you're not familiar with Docker volumes, you can safely ignore this slide! .debug[[k8s/volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/volumes.md)] --- ## Volumes ≠ Persistent Volumes - Volumes and Persistent Volumes are related, but very different! - *Volumes*: - appear in Pod specifications (we'll see that in a few slides) - do not exist as API resources (**cannot** do `kubectl get volumes`) - *Persistent Volumes*: - are API resources (**can** do `kubectl get persistentvolumes`) - correspond to concrete volumes (e.g. on a SAN, EBS, etc.) - cannot be associated with a Pod directly; but through a Persistent Volume Claim - won't be discussed further in this section .debug[[k8s/volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/volumes.md)] --- ## Adding a volume to a Pod - We will start with the simplest Pod manifest we can find - We will add a volume to that Pod manifest - We will mount that volume in a container in the Pod - By default, this volume will be an `emptyDir` (an empty directory) - It will "shadow" the directory where it's mounted .debug[[k8s/volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/volumes.md)] --- ## Our basic Pod ```yaml apiVersion: v1 kind: Pod metadata: name: nginx-without-volume spec: containers: - name: nginx image: nginx ``` This is a MVP! (Minimum Viable Pod😉) It runs a single NGINX container. .debug[[k8s/volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/volumes.md)] --- ## Trying the basic pod .exercise[ - Create the Pod: ```bash kubectl create -f ~/container.training/k8s/nginx-1-without-volume.yaml ``` <!-- ```bash kubectl wait pod/nginx-without-volume --for condition=ready ``` --> - Get its IP address: ```bash IPADDR=$(kubectl get pod nginx-without-volume -o jsonpath={.status.podIP}) ``` - Send a request with curl: ```bash curl $IPADDR ``` ] (We should see the "Welcome to NGINX" page.) .debug[[k8s/volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/volumes.md)] --- ## Adding a volume - We need to add the volume in two places: - at the Pod level (to declare the volume) - at the container level (to mount the volume) - We will declare a volume named `www` - No type is specified, so it will default to `emptyDir` (as the name implies, it will be initialized as an empty directory at pod creation) - In that pod, there is also a container named `nginx` - That container mounts the volume `www` to path `/usr/share/nginx/html/` .debug[[k8s/volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/volumes.md)] --- ## The Pod with a volume ```yaml apiVersion: v1 kind: Pod metadata: name: nginx-with-volume spec: volumes: - name: www containers: - name: nginx image: nginx volumeMounts: - name: www mountPath: /usr/share/nginx/html/ ``` .debug[[k8s/volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/volumes.md)] --- ## Trying the Pod with a volume .exercise[ - Create the Pod: ```bash kubectl create -f ~/container.training/k8s/nginx-2-with-volume.yaml ``` <!-- ```bash kubectl wait pod/nginx-with-volume --for condition=ready ``` --> - Get its IP address: ```bash IPADDR=$(kubectl get pod nginx-with-volume -o jsonpath={.status.podIP}) ``` - Send a request with curl: ```bash curl $IPADDR ``` ] (We should now see a "403 Forbidden" error page.) .debug[[k8s/volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/volumes.md)] --- ## Populating the volume with another container - Let's add another container to the Pod - Let's mount the volume in *both* containers - That container will populate the volume with static files - NGINX will then serve these static files - To populate the volume, we will clone the Spoon-Knife repository - this repository is https://github.com/octocat/Spoon-Knife - it's very popular (more than 100K stars!) .debug[[k8s/volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/volumes.md)] --- ## Sharing a volume between two containers .small[ ```yaml apiVersion: v1 kind: Pod metadata: name: nginx-with-git spec: volumes: - name: www containers: - name: nginx image: nginx volumeMounts: - name: www mountPath: /usr/share/nginx/html/ - name: git image: alpine command: [ "sh", "-c", "apk add git && git clone https://github.com/octocat/Spoon-Knife /www" ] volumeMounts: - name: www mountPath: /www/ restartPolicy: OnFailure ``` ] .debug[[k8s/volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/volumes.md)] --- ## Sharing a volume, explained - We added another container to the pod - That container mounts the `www` volume on a different path (`/www`) - It uses the `alpine` image - When started, it installs `git` and clones the `octocat/Spoon-Knife` repository (that repository contains a tiny HTML website) - As a result, NGINX now serves this website .debug[[k8s/volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/volumes.md)] --- ## Trying the shared volume - This one will be time-sensitive! - We need to catch the Pod IP address *as soon as it's created* - Then send a request to it *as fast as possible* .exercise[ - Watch the pods (so that we can catch the Pod IP address) ```bash kubectl get pods -o wide --watch ``` <!-- ```wait NAME``` ```tmux split-pane -v``` --> ] .debug[[k8s/volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/volumes.md)] --- ## Shared volume in action .exercise[ - Create the pod: ```bash kubectl create -f ~/container.training/k8s/nginx-3-with-git.yaml ``` <!-- ```bash kubectl wait pod/nginx-with-git --for condition=initialized``` ```bash IP=$(kubectl get pod nginx-with-git -o jsonpath={.status.podIP})``` --> - As soon as we see its IP address, access it: ```bash curl `$IP` ``` <!-- ```bash /bin/sleep 5``` --> - A few seconds later, the state of the pod will change; access it again: ```bash curl `$IP` ``` ] The first time, we should see "403 Forbidden". The second time, we should see the HTML file from the Spoon-Knife repository. .debug[[k8s/volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/volumes.md)] --- ## Explanations - Both containers are started at the same time - NGINX starts very quickly (it can serve requests immediately) - But at this point, the volume is empty (NGINX serves "403 Forbidden") - The other containers installs git and clones the repository (this takes a bit longer) - When the other container is done, the volume holds the repository (NGINX serves the HTML file) .debug[[k8s/volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/volumes.md)] --- ## The devil is in the details - The default `restartPolicy` is `Always` - This would cause our `git` container to run again ... and again ... and again (with an exponential back-off delay, as explained [in the documentation](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy)) - That's why we specified `restartPolicy: OnFailure` .debug[[k8s/volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/volumes.md)] --- ## Inconsistencies - There is a short period of time during which the website is not available (because the `git` container hasn't done its job yet) - With a bigger website, we could get inconsistent results (where only a part of the content is ready) - In real applications, this could cause incorrect results - How can we avoid that? .debug[[k8s/volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/volumes.md)] --- ## Init Containers - We can define containers that should execute *before* the main ones - They will be executed in order (instead of in parallel) - They must all succeed before the main containers are started - This is *exactly* what we need here! - Let's see one in action .footnote[See [Init Containers](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/) documentation for all the details.] .debug[[k8s/volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/volumes.md)] --- ## Defining Init Containers .small[ ```yaml apiVersion: v1 kind: Pod metadata: name: nginx-with-init spec: volumes: - name: www containers: - name: nginx image: nginx volumeMounts: - name: www mountPath: /usr/share/nginx/html/ initContainers: - name: git image: alpine command: [ "sh", "-c", "apk add git && git clone https://github.com/octocat/Spoon-Knife /www" ] volumeMounts: - name: www mountPath: /www/ ``` ] .debug[[k8s/volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/volumes.md)] --- ## Trying the init container .exercise[ - Create the pod: ```bash kubectl create -f ~/container.training/k8s/nginx-4-with-init.yaml ``` - Try to send HTTP requests as soon as the pod comes up <!-- ```key ^D``` ```key ^C``` --> ] - This time, instead of "403 Forbidden" we get a "connection refused" - NGINX doesn't start until the git container has done its job - We never get inconsistent results (a "half-ready" container) .debug[[k8s/volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/volumes.md)] --- ## Other uses of init containers - Load content - Generate configuration (or certificates) - Database migrations - Waiting for other services to be up (to avoid flurry of connection errors in main container) - etc. .debug[[k8s/volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/volumes.md)] --- ## Volume lifecycle - The lifecycle of a volume is linked to the pod's lifecycle - This means that a volume is created when the pod is created - This is mostly relevant for `emptyDir` volumes (other volumes, like remote storage, are not "created" but rather "attached" ) - A volume survives across container restarts - A volume is destroyed (or, for remote storage, detached) when the pod is destroyed ??? :EN:- Sharing data between containers with volumes :EN:- When and how to use Init Containers :FR:- Partager des données grâce aux volumes :FR:- Quand et comment utiliser un *Init Container* .debug[[k8s/volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/volumes.md)] --- class: pic .interstitial[] --- name: toc-managing-configuration class: title Managing configuration .nav[ [Previous section](#toc-volumes) | [Back to table of contents](#toc-module-0) | [Next section](#toc-stateful-sets) ] .debug[(automatically generated title slide)] --- # Managing configuration - Some applications need to be configured (obviously!) - There are many ways for our code to pick up configuration: - command-line arguments - environment variables - configuration files - configuration servers (getting configuration from a database, an API...) - ... and more (because programmers can be very creative!) - How can we do these things with containers and Kubernetes? .debug[[k8s/configuration.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/configuration.md)] --- ## Passing configuration to containers - There are many ways to pass configuration to code running in a container: - baking it into a custom image - command-line arguments - environment variables - injecting configuration files - exposing it over the Kubernetes API - configuration servers - Let's review these different strategies! .debug[[k8s/configuration.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/configuration.md)] --- ## Baking custom images - Put the configuration in the image (it can be in a configuration file, but also `ENV` or `CMD` actions) - It's easy! It's simple! - Unfortunately, it also has downsides: - multiplication of images - different images for dev, staging, prod ... - minor reconfigurations require a whole build/push/pull cycle - Avoid doing it unless you don't have the time to figure out other options .debug[[k8s/configuration.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/configuration.md)] --- ## Command-line arguments - Pass options to `args` array in the container specification - Example ([source](https://github.com/coreos/pods/blob/master/kubernetes.yaml#L29)): ```yaml args: - "--data-dir=/var/lib/etcd" - "--advertise-client-urls=http://127.0.0.1:2379" - "--listen-client-urls=http://127.0.0.1:2379" - "--listen-peer-urls=http://127.0.0.1:2380" - "--name=etcd" ``` - The options can be passed directly to the program that we run ... ... or to a wrapper script that will use them to e.g. generate a config file .debug[[k8s/configuration.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/configuration.md)] --- ## Command-line arguments, pros & cons - Works great when options are passed directly to the running program (otherwise, a wrapper script can work around the issue) - Works great when there aren't too many parameters (to avoid a 20-lines `args` array) - Requires documentation and/or understanding of the underlying program ("which parameters and flags do I need, again?") - Well-suited for mandatory parameters (without default values) - Not ideal when we need to pass a real configuration file anyway .debug[[k8s/configuration.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/configuration.md)] --- ## Environment variables - Pass options through the `env` map in the container specification - Example: ```yaml env: - name: ADMIN_PORT value: "8080" - name: ADMIN_AUTH value: Basic - name: ADMIN_CRED value: "admin:0pensesame!" ``` .warning[`value` must be a string! Make sure that numbers and fancy strings are quoted.] 🤔 Why this weird `{name: xxx, value: yyy}` scheme? It will be revealed soon! .debug[[k8s/configuration.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/configuration.md)] --- ## The downward API - In the previous example, environment variables have fixed values - We can also use a mechanism called the *downward API* - The downward API allows exposing pod or container information - either through special files (we won't show that for now) - or through environment variables - The value of these environment variables is computed when the container is started - Remember: environment variables won't (can't) change after container start - Let's see a few concrete examples! .debug[[k8s/configuration.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/configuration.md)] --- ## Exposing the pod's namespace ```yaml - name: MY_POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace ``` - Useful to generate FQDN of services (in some contexts, a short name is not enough) - For instance, the two commands should be equivalent: ``` curl api-backend curl api-backend.$MY_POD_NAMESPACE.svc.cluster.local ``` .debug[[k8s/configuration.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/configuration.md)] --- ## Exposing the pod's IP address ```yaml - name: MY_POD_IP valueFrom: fieldRef: fieldPath: status.podIP ``` - Useful if we need to know our IP address (we could also read it from `eth0`, but this is more solid) .debug[[k8s/configuration.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/configuration.md)] --- ## Exposing the container's resource limits ```yaml - name: MY_MEM_LIMIT valueFrom: resourceFieldRef: containerName: test-container resource: limits.memory ``` - Useful for runtimes where memory is garbage collected - Example: the JVM (the memory available to the JVM should be set with the `-Xmx ` flag) - Best practice: set a memory limit, and pass it to the runtime - Note: recent versions of the JVM can do this automatically (see [JDK-8146115](https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8146115)) and [this blog post](https://very-serio.us/2017/12/05/running-jvms-in-kubernetes/) for detailed examples) .debug[[k8s/configuration.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/configuration.md)] --- ## More about the downward API - [This documentation page](https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/) tells more about these environment variables - And [this one](https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/) explains the other way to use the downward API (through files that get created in the container filesystem) .debug[[k8s/configuration.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/configuration.md)] --- ## Environment variables, pros and cons - Works great when the running program expects these variables - Works great for optional parameters with reasonable defaults (since the container image can provide these defaults) - Sort of auto-documented (we can see which environment variables are defined in the image, and their values) - Can be (ab)used with longer values ... - ... You *can* put an entire Tomcat configuration file in an environment ... - ... But *should* you? (Do it if you really need to, we're not judging! But we'll see better ways.) .debug[[k8s/configuration.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/configuration.md)] --- ## Injecting configuration files - Sometimes, there is no way around it: we need to inject a full config file - Kubernetes provides a mechanism for that purpose: `configmaps` - A configmap is a Kubernetes resource that exists in a namespace - Conceptually, it's a key/value map (values are arbitrary strings) - We can think about them in (at least) two different ways: - as holding entire configuration file(s) - as holding individual configuration parameters *Note: to hold sensitive information, we can use "Secrets", which are another type of resource behaving very much like configmaps. We'll cover them just after!* .debug[[k8s/configuration.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/configuration.md)] --- ## Configmaps storing entire files - In this case, each key/value pair corresponds to a configuration file - Key = name of the file - Value = content of the file - There can be one key/value pair, or as many as necessary (for complex apps with multiple configuration files) - Examples: ``` # Create a configmap with a single key, "app.conf" kubectl create configmap my-app-config --from-file=app.conf # Create a configmap with a single key, "app.conf" but another file kubectl create configmap my-app-config --from-file=app.conf=app-prod.conf # Create a configmap with multiple keys (one per file in the config.d directory) kubectl create configmap my-app-config --from-file=config.d/ ``` .debug[[k8s/configuration.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/configuration.md)] --- ## Configmaps storing individual parameters - In this case, each key/value pair corresponds to a parameter - Key = name of the parameter - Value = value of the parameter - Examples: ``` # Create a configmap with two keys kubectl create cm my-app-config \ --from-literal=foreground=red \ --from-literal=background=blue # Create a configmap from a file containing key=val pairs kubectl create cm my-app-config \ --from-env-file=app.conf ``` .debug[[k8s/configuration.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/configuration.md)] --- ## Exposing configmaps to containers - Configmaps can be exposed as plain files in the filesystem of a container - this is achieved by declaring a volume and mounting it in the container - this is particularly effective for configmaps containing whole files - Configmaps can be exposed as environment variables in the container - this is achieved with the downward API - this is particularly effective for configmaps containing individual parameters - Let's see how to do both! .debug[[k8s/configuration.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/configuration.md)] --- ## Passing a configuration file with a configmap - We will start a load balancer powered by HAProxy - We will use the [official `haproxy` image](https://hub.docker.com/_/haproxy/) - It expects to find its configuration in `/usr/local/etc/haproxy/haproxy.cfg` - We will provide a simple HAproxy configuration, `k8s/haproxy.cfg` - It listens on port 80, and load balances connections between IBM and Google .debug[[k8s/configuration.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/configuration.md)] --- ## Creating the configmap .exercise[ - Go to the `k8s` directory in the repository: ```bash cd ~/container.training/k8s ``` - Create a configmap named `haproxy` and holding the configuration file: ```bash kubectl create configmap haproxy --from-file=haproxy.cfg ``` - Check what our configmap looks like: ```bash kubectl get configmap haproxy -o yaml ``` ] .debug[[k8s/configuration.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/configuration.md)] --- ## Using the configmap We are going to use the following pod definition: ```yaml apiVersion: v1 kind: Pod metadata: name: haproxy spec: volumes: - name: config configMap: name: haproxy containers: - name: haproxy image: haproxy volumeMounts: - name: config mountPath: /usr/local/etc/haproxy/ ``` .debug[[k8s/configuration.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/configuration.md)] --- ## Using the configmap - The resource definition from the previous slide is in `k8s/haproxy.yaml` .exercise[ - Create the HAProxy pod: ```bash kubectl apply -f ~/container.training/k8s/haproxy.yaml ``` <!-- ```hide kubectl wait pod haproxy --for condition=ready``` --> - Check the IP address allocated to the pod: ```bash kubectl get pod haproxy -o wide IP=$(kubectl get pod haproxy -o json | jq -r .status.podIP) ``` ] .debug[[k8s/configuration.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/configuration.md)] --- ## Testing our load balancer - The load balancer will send: - half of the connections to Google - the other half to IBM .exercise[ - Access the load balancer a few times: ```bash curl $IP curl $IP curl $IP ``` ] We should see connections served by Google, and others served by IBM. <br/> (Each server sends us a redirect page. Look at the URL that they send us to!) .debug[[k8s/configuration.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/configuration.md)] --- ## Exposing configmaps with the downward API - We are going to run a Docker registry on a custom port - By default, the registry listens on port 5000 - This can be changed by setting environment variable `REGISTRY_HTTP_ADDR` - We are going to store the port number in a configmap - Then we will expose that configmap as a container environment variable .debug[[k8s/configuration.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/configuration.md)] --- ## Creating the configmap .exercise[ - Our configmap will have a single key, `http.addr`: ```bash kubectl create configmap registry --from-literal=http.addr=0.0.0.0:80 ``` - Check our configmap: ```bash kubectl get configmap registry -o yaml ``` ] .debug[[k8s/configuration.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/configuration.md)] --- ## Using the configmap We are going to use the following pod definition: ```yaml apiVersion: v1 kind: Pod metadata: name: registry spec: containers: - name: registry image: registry env: - name: REGISTRY_HTTP_ADDR valueFrom: configMapKeyRef: name: registry key: http.addr ``` .debug[[k8s/configuration.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/configuration.md)] --- ## Using the configmap - The resource definition from the previous slide is in `k8s/registry.yaml` .exercise[ - Create the registry pod: ```bash kubectl apply -f ~/container.training/k8s/registry.yaml ``` <!-- ```hide kubectl wait pod registry --for condition=ready``` --> - Check the IP address allocated to the pod: ```bash kubectl get pod registry -o wide IP=$(kubectl get pod registry -o json | jq -r .status.podIP) ``` - Confirm that the registry is available on port 80: ```bash curl $IP/v2/_catalog ``` ] .debug[[k8s/configuration.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/configuration.md)] --- ## Passwords, tokens, sensitive information - For sensitive information, there is another special resource: *Secrets* - Secrets and Configmaps work almost the same way (we'll expose the differences on the next slide) - The *intent* is different, though: *"You should use secrets for things which are actually secret like API keys, credentials, etc., and use config map for not-secret configuration data."* *"In the future there will likely be some differentiators for secrets like rotation or support for backing the secret API w/ HSMs, etc."* (Source: [the author of both features](https://stackoverflow.com/a/36925553/580281 )) .debug[[k8s/configuration.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/configuration.md)] --- ## Differences between configmaps and secrets - Secrets are base64-encoded when shown with `kubectl get secrets -o yaml` - keep in mind that this is just *encoding*, not *encryption* - it is very easy to [automatically extract and decode secrets](https://medium.com/@mveritym/decoding-kubernetes-secrets-60deed7a96a3) - [Secrets can be encrypted at rest](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/) - With RBAC, we can authorize a user to access configmaps, but not secrets (since they are two different kinds of resources) ??? :EN:- Managing application configuration :EN:- Exposing configuration with the downward API :EN:- Exposing configuration with Config Maps and Secrets :FR:- Gérer la configuration des applications :FR:- Configuration au travers de la *downward API* :FR:- Configuration via les *Config Maps* et *Secrets* .debug[[k8s/configuration.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/configuration.md)] --- class: pic .interstitial[] --- name: toc-stateful-sets class: title Stateful sets .nav[ [Previous section](#toc-managing-configuration) | [Back to table of contents](#toc-module-0) | [Next section](#toc-running-a-consul-cluster) ] .debug[(automatically generated title slide)] --- # Stateful sets - Stateful sets are a type of resource in the Kubernetes API (like pods, deployments, services...) - They offer mechanisms to deploy scaled stateful applications - At a first glance, they look like *deployments*: - a stateful set defines a pod spec and a number of replicas *R* - it will make sure that *R* copies of the pod are running - that number can be changed while the stateful set is running - updating the pod spec will cause a rolling update to happen - But they also have some significant differences .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- ## Stateful sets unique features - Pods in a stateful set are numbered (from 0 to *R-1*) and ordered - They are started and updated in order (from 0 to *R-1*) - A pod is started (or updated) only when the previous one is ready - They are stopped in reverse order (from *R-1* to 0) - Each pod know its identity (i.e. which number it is in the set) - Each pod can discover the IP address of the others easily - The pods can persist data on attached volumes 🤔 Wait a minute ... Can't we already attach volumes to pods and deployments? .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- ## Revisiting volumes - [Volumes](https://kubernetes.io/docs/concepts/storage/volumes/) are used for many purposes: - sharing data between containers in a pod - exposing configuration information and secrets to containers - accessing storage systems - Let's see examples of the latter usage .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- ## Volumes types - There are many [types of volumes](https://kubernetes.io/docs/concepts/storage/volumes/#types-of-volumes) available: - public cloud storage (GCEPersistentDisk, AWSElasticBlockStore, AzureDisk...) - private cloud storage (Cinder, VsphereVolume...) - traditional storage systems (NFS, iSCSI, FC...) - distributed storage (Ceph, Glusterfs, Portworx...) - Using a persistent volume requires: - creating the volume out-of-band (outside of the Kubernetes API) - referencing the volume in the pod description, with all its parameters .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- ## Using a cloud volume Here is a pod definition using an AWS EBS volume (that has to be created first): ```yaml apiVersion: v1 kind: Pod metadata: name: pod-using-my-ebs-volume spec: containers: - image: ... name: container-using-my-ebs-volume volumeMounts: - mountPath: /my-ebs name: my-ebs-volume volumes: - name: my-ebs-volume awsElasticBlockStore: volumeID: vol-049df61146c4d7901 fsType: ext4 ``` .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- ## Using an NFS volume Here is another example using a volume on an NFS server: ```yaml apiVersion: v1 kind: Pod metadata: name: pod-using-my-nfs-volume spec: containers: - image: ... name: container-using-my-nfs-volume volumeMounts: - mountPath: /my-nfs name: my-nfs-volume volumes: - name: my-nfs-volume nfs: server: 192.168.0.55 path: "/exports/assets" ``` .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- ## Shortcomings of volumes - Their lifecycle (creation, deletion...) is managed outside of the Kubernetes API (we can't just use `kubectl apply/create/delete/...` to manage them) - If a Deployment uses a volume, all replicas end up using the same volume - That volume must then support concurrent access - some volumes do (e.g. NFS servers support multiple read/write access) - some volumes support concurrent reads - some volumes support concurrent access for colocated pods - What we really need is a way for each replica to have its own volume .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- ## Individual volumes - The Pods of a Stateful set can have individual volumes (i.e. in a Stateful set with 3 replicas, there will be 3 volumes) - These volumes can be either: - allocated from a pool of pre-existing volumes (disks, partitions ...) - created dynamically using a storage system - This introduces a bunch of new Kubernetes resource types: Persistent Volumes, Persistent Volume Claims, Storage Classes (and also `volumeClaimTemplates`, that appear within Stateful Set manifests!) .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- ## Stateful set recap - A Stateful sets manages a number of identical pods (like a Deployment) - These pods are numbered, and started/upgraded/stopped in a specific order - These pods are aware of their number (e.g., #0 can decide to be the primary, and #1 can be secondary) - These pods can find the IP addresses of the other pods in the set (through a *headless service*) - These pods can each have their own persistent storage (Deployments cannot do that) .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- class: pic .interstitial[] --- name: toc-running-a-consul-cluster class: title Running a Consul cluster .nav[ [Previous section](#toc-stateful-sets) | [Back to table of contents](#toc-module-0) | [Next section](#toc-persistent-volumes-claims) ] .debug[(automatically generated title slide)] --- # Running a Consul cluster - Here is a good use-case for Stateful sets! - We are going to deploy a Consul cluster with 3 nodes - Consul is a highly-available key/value store (like etcd or Zookeeper) - One easy way to bootstrap a cluster is to tell each node: - the addresses of other nodes - how many nodes are expected (to know when quorum is reached) .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- ## Bootstrapping a Consul cluster *After reading the Consul documentation carefully (and/or asking around), we figure out the minimal command-line to run our Consul cluster.* ``` consul agent -data-dir=/consul/data -client=0.0.0.0 -server -ui \ -bootstrap-expect=3 \ -retry-join=`X.X.X.X` \ -retry-join=`Y.Y.Y.Y` ``` - Replace X.X.X.X and Y.Y.Y.Y with the addresses of other nodes - The same command-line can be used on all nodes (convenient!) .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- ## Cloud Auto-join - Since version 1.4.0, Consul can use the Kubernetes API to find its peers - This is called [Cloud Auto-join] - Instead of passing an IP address, we need to pass a parameter like this: ``` consul agent -retry-join "provider=k8s label_selector=\"app=consul\"" ``` - Consul needs to be able to talk to the Kubernetes API - We can provide a `kubeconfig` file - If Consul runs in a pod, it will use the *service account* of the pod [Cloud Auto-join]: https://www.consul.io/docs/agent/cloud-auto-join.html#kubernetes-k8s- .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- ## Setting up Cloud auto-join - We need to create a service account for Consul - We need to create a role that can `list` and `get` pods - We need to bind that role to the service account - And of course, we need to make sure that Consul pods use that service account .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- ## Putting it all together - The file `k8s/consul.yaml` defines the required resources (service account, cluster role, cluster role binding, service, stateful set) - It has a few extra touches: - a `podAntiAffinity` prevents two pods from running on the same node - a `preStop` hook makes the pod leave the cluster when shutdown gracefully This was inspired by this [excellent tutorial](https://github.com/kelseyhightower/consul-on-kubernetes) by Kelsey Hightower. Some features from the original tutorial (TLS authentication between nodes and encryption of gossip traffic) were removed for simplicity. .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- ## Running our Consul cluster - We'll use the provided YAML file .exercise[ - Create the stateful set and associated service: ```bash kubectl apply -f ~/container.training/k8s/consul.yaml ``` - Check the logs as the pods come up one after another: ```bash stern consul ``` <!-- ```wait Synced node info``` ```key ^C``` --> - Check the health of the cluster: ```bash kubectl exec consul-0 consul members ``` ] .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- ## Caveats - We aren't using actual persistence yet (no `volumeClaimTemplate`, Persistent Volume, etc.) - What happens if we lose a pod? - a new pod gets rescheduled (with an empty state) - the new pod tries to connect to the two others - it will be accepted (after 1-2 minutes of instability) - and it will retrieve the data from the other pods .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- ## Failure modes - What happens if we lose two pods? - manual repair will be required - we will need to instruct the remaining one to act solo - then rejoin new pods - What happens if we lose three pods? (aka all of them) - we lose all the data (ouch) - If we run Consul without persistent storage, backups are a good idea! .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- class: pic .interstitial[] --- name: toc-persistent-volumes-claims class: title Persistent Volumes Claims .nav[ [Previous section](#toc-running-a-consul-cluster) | [Back to table of contents](#toc-module-0) | [Next section](#toc-local-persistent-volumes) ] .debug[(automatically generated title slide)] --- # Persistent Volumes Claims - Our Pods can use a special volume type: a *Persistent Volume Claim* - A Persistent Volume Claim (PVC) is also a Kubernetes resource (visible with `kubectl get persistentvolumeclaims` or `kubectl get pvc`) - A PVC is not a volume; it is a *request for a volume* - It should indicate at least: - the size of the volume (e.g. "5 GiB") - the access mode (e.g. "read-write by a single pod") .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- ## What's in a PVC? - A PVC contains at least: - a list of *access modes* (ReadWriteOnce, ReadOnlyMany, ReadWriteMany) - a size (interpreted as the minimal storage space needed) - It can also contain optional elements: - a selector (to restrict which actual volumes it can use) - a *storage class* (used by dynamic provisioning, more on that later) .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- ## What does a PVC look like? Here is a manifest for a basic PVC: ```yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: my-claim spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi ``` .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- ## Using a Persistent Volume Claim Here is a Pod definition like the ones shown earlier, but using a PVC: ```yaml apiVersion: v1 kind: Pod metadata: name: pod-using-a-claim spec: containers: - image: ... name: container-using-a-claim volumeMounts: - mountPath: /my-vol name: my-volume volumes: - name: my-volume persistentVolumeClaim: claimName: my-claim ``` .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- ## Creating and using Persistent Volume Claims - PVCs can be created manually and used explicitly (as shown on the previous slides) - They can also be created and used through Stateful Sets (this will be shown later) .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- ## Lifecycle of Persistent Volume Claims - When a PVC is created, it starts existing in "Unbound" state (without an associated volume) - A Pod referencing an unbound PVC will not start (the scheduler will wait until the PVC is bound to place it) - A special controller continuously monitors PVCs to associate them with PVs - If no PV is available, one must be created: - manually (by operator intervention) - using a *dynamic provisioner* (more on that later) .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- class: extra-details ## Which PV gets associated to a PVC? - The PV must satisfy the PVC constraints (access mode, size, optional selector, optional storage class) - The PVs with the closest access mode are picked - Then the PVs with the closest size - It is possible to specify a `claimRef` when creating a PV (this will associate it to the specified PVC, but only if the PV satisfies all the requirements of the PVC; otherwise another PV might end up being picked) - For all the details about the PersistentVolumeClaimBinder, check [this doc](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/persistent-storage.md#matching-and-binding) .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- ## Persistent Volume Claims and Stateful sets - A Stateful set can define one (or more) `volumeClaimTemplate` - Each `volumeClaimTemplate` will create one Persistent Volume Claim per pod - Each pod will therefore have its own individual volume - These volumes are numbered (like the pods) - Example: - a Stateful set is named `db` - it is scaled to replicas - it has a `volumeClaimTemplate` named `data` - then it will create pods `db-0`, `db-1`, `db-2` - these pods will have volumes named `data-db-0`, `data-db-1`, `data-db-2` .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- ## Persistent Volume Claims are sticky - When updating the stateful set (e.g. image upgrade), each pod keeps its volume - When pods get rescheduled (e.g. node failure), they keep their volume (this requires a storage system that is not node-local) - These volumes are not automatically deleted (when the stateful set is scaled down or deleted) - If a stateful set is scaled back up later, the pods get their data back .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- ## Dynamic provisioners - A *dynamic provisioner* monitors unbound PVCs - It can create volumes (and the corresponding PV) on the fly - This requires the PVCs to have a *storage class* (annotation `volume.beta.kubernetes.io/storage-provisioner`) - A dynamic provisioner only acts on PVCs with the right storage class (it ignores the other ones) - Just like `LoadBalancer` services, dynamic provisioners are optional (i.e. our cluster may or may not have one pre-installed) .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- ## What's a Storage Class? - A Storage Class is yet another Kubernetes API resource (visible with e.g. `kubectl get storageclass` or `kubectl get sc`) - It indicates which *provisioner* to use (which controller will create the actual volume) - And arbitrary parameters for that provisioner (replication levels, type of disk ... anything relevant!) - Storage Classes are required if we want to use [dynamic provisioning](https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/) (but we can also create volumes manually, and ignore Storage Classes) .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- ## The default storage class - At most one storage class can be marked as the default class (by annotating it with `storageclass.kubernetes.io/is-default-class=true`) - When a PVC is created, it will be annotated with the default storage class (unless it specifies an explicit storage class) - This only happens at PVC creation (existing PVCs are not updated when we mark a class as the default one) .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- ## Dynamic provisioning setup This is how we can achieve fully automated provisioning of persistent storage. 1. Configure a storage system. (It needs to have an API, or be capable of automated provisioning of volumes.) 2. Install a dynamic provisioner for this storage system. (This is some specific controller code.) 3. Create a Storage Class for this system. (It has to match what the dynamic provisioner is expecting.) 4. Annotate the Storage Class to be the default one. .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- ## Dynamic provisioning usage After setting up the system (previous slide), all we need to do is: *Create a Stateful Set that makes use of a `volumeClaimTemplate`.* This will trigger the following actions. 1. The Stateful Set creates PVCs according to the `volumeClaimTemplate`. 2. The Stateful Set creates Pods using these PVCs. 3. The PVCs are automatically annotated with our Storage Class. 4. The dynamic provisioner provisions volumes and creates the corresponding PVs. 5. The PersistentVolumeClaimBinder associates the PVs and the PVCs together. 6. PVCs are now bound, the Pods can start. ??? :EN:- Deploying apps with Stateful Sets :EN:- Example: deploying a Consul cluster :EN:- Understanding Persistent Volume Claims and Storage Classes :FR:- Déployer une application avec un *Stateful Set* :FR:- Example : lancer un cluster Consul :FR:- Comprendre les *Persistent Volume Claims* et *Storage Classes* .debug[[k8s/statefulsets.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/statefulsets.md)] --- class: pic .interstitial[] --- name: toc-local-persistent-volumes class: title Local Persistent Volumes .nav[ [Previous section](#toc-persistent-volumes-claims) | [Back to table of contents](#toc-module-0) | [Next section](#toc-highly-available-persistent-volumes) ] .debug[(automatically generated title slide)] --- # Local Persistent Volumes - We want to run that Consul cluster *and* actually persist data - But we don't have a distributed storage system - We are going to use local volumes instead (similar conceptually to `hostPath` volumes) - We can use local volumes without installing extra plugins - However, they are tied to a node - If that node goes down, the volume becomes unavailable .debug[[k8s/local-persistent-volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/local-persistent-volumes.md)] --- ## With or without dynamic provisioning - We will deploy a Consul cluster *with* persistence - That cluster's StatefulSet will create PVCs - These PVCs will remain unbound¹, until we will create local volumes manually (we will basically do the job of the dynamic provisioner) - Then, we will see how to automate that with a dynamic provisioner .footnote[¹Unbound = without an associated Persistent Volume.] .debug[[k8s/local-persistent-volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/local-persistent-volumes.md)] --- ## If we have a dynamic provisioner ... - The labs in this section assume that we *do not* have a dynamic provisioner - If we do have one, we need to disable it .exercise[ - Check if we have a dynamic provisioner: ```bash kubectl get storageclass ``` - If the output contains a line with `(default)`, run this command: ```bash kubectl annotate sc storageclass.kubernetes.io/is-default-class- --all ``` - Check again that it is no longer marked as `(default)` ] .debug[[k8s/local-persistent-volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/local-persistent-volumes.md)] --- ## Deploying Consul - We will use a slightly different YAML file - The only differences between that file and the previous one are: - `volumeClaimTemplate` defined in the Stateful Set spec - the corresponding `volumeMounts` in the Pod spec - the label `consul` has been changed to `persistentconsul` <br/> (to avoid conflicts with the other Stateful Set) .exercise[ - Apply the persistent Consul YAML file: ```bash kubectl apply -f ~/container.training/k8s/persistent-consul.yaml ``` ] .debug[[k8s/local-persistent-volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/local-persistent-volumes.md)] --- ## Observing the situation - Let's look at Persistent Volume Claims and Pods .exercise[ - Check that we now have an unbound Persistent Volume Claim: ```bash kubectl get pvc ``` - We don't have any Persistent Volume: ```bash kubectl get pv ``` - The Pod `persistentconsul-0` is not scheduled yet: ```bash kubectl get pods -o wide ``` ] *Hint: leave these commands running with `-w` in different windows.* .debug[[k8s/local-persistent-volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/local-persistent-volumes.md)] --- ## Explanations - In a Stateful Set, the Pods are started one by one - `persistentconsul-1` won't be created until `persistentconsul-0` is running - `persistentconsul-0` has a dependency on an unbound Persistent Volume Claim - The scheduler won't schedule the Pod until the PVC is bound (because the PVC might be bound to a volume that is only available on a subset of nodes; for instance EBS are tied to an availability zone) .debug[[k8s/local-persistent-volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/local-persistent-volumes.md)] --- ## Creating Persistent Volumes - Let's create 3 local directories (`/mnt/consul`) on node2, node3, node4 - Then create 3 Persistent Volumes corresponding to these directories .exercise[ - Create the local directories: ```bash for NODE in node2 node3 node4; do ssh $NODE sudo mkdir -p /mnt/consul done ``` - Create the PV objects: ```bash kubectl apply -f ~/container.training/k8s/volumes-for-consul.yaml ``` ] .debug[[k8s/local-persistent-volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/local-persistent-volumes.md)] --- ## Check our Consul cluster - The PVs that we created will be automatically matched with the PVCs - Once a PVC is bound, its pod can start normally - Once the pod `persistentconsul-0` has started, `persistentconsul-1` can be created, etc. - Eventually, our Consul cluster is up, and backend by "persistent" volumes .exercise[ - Check that our Consul clusters has 3 members indeed: ```bash kubectl exec persistentconsul-0 consul members ``` ] .debug[[k8s/local-persistent-volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/local-persistent-volumes.md)] --- ## Devil is in the details (1/2) - The size of the Persistent Volumes is bogus (it is used when matching PVs and PVCs together, but there is no actual quota or limit) .debug[[k8s/local-persistent-volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/local-persistent-volumes.md)] --- ## Devil is in the details (2/2) - This specific example worked because we had exactly 1 free PV per node: - if we had created multiple PVs per node ... - we could have ended with two PVCs bound to PVs on the same node ... - which would have required two pods to be on the same node ... - which is forbidden by the anti-affinity constraints in the StatefulSet - To avoid that, we need to associated the PVs with a Storage Class that has: ```yaml volumeBindingMode: WaitForFirstConsumer ``` (this means that a PVC will be bound to a PV only after being used by a Pod) - See [this blog post](https://kubernetes.io/blog/2018/04/13/local-persistent-volumes-beta/) for more details .debug[[k8s/local-persistent-volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/local-persistent-volumes.md)] --- ## Bulk provisioning - It's not practical to manually create directories and PVs for each app - We *could* pre-provision a number of PVs across our fleet - We could even automate that with a Daemon Set: - creating a number of directories on each node - creating the corresponding PV objects - We also need to recycle volumes - ... This can quickly get out of hand .debug[[k8s/local-persistent-volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/local-persistent-volumes.md)] --- ## Dynamic provisioning - We could also write our own provisioner, which would: - watch the PVCs across all namespaces - when a PVC is created, create a corresponding PV on a node - Or we could use one of the dynamic provisioners for local persistent volumes (for instance the [Rancher local path provisioner](https://github.com/rancher/local-path-provisioner)) .debug[[k8s/local-persistent-volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/local-persistent-volumes.md)] --- ## Strategies for local persistent volumes - Remember, when a node goes down, the volumes on that node become unavailable - High availability will require another layer of replication (like what we've just seen with Consul; or primary/secondary; etc) - Pre-provisioning PVs makes sense for machines with local storage (e.g. cloud instance storage; or storage directly attached to a physical machine) - Dynamic provisioning makes sense for large number of applications (when we can't or won't dedicate a whole disk to a volume) - It's possible to mix both (using distinct Storage Classes) ??? :EN:- Static vs dynamic volume provisioning :EN:- Example: local persistent volume provisioner :FR:- Création statique ou dynamique de volumes :FR:- Exemple : création de volumes locaux .debug[[k8s/local-persistent-volumes.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/local-persistent-volumes.md)] --- class: pic .interstitial[] --- name: toc-highly-available-persistent-volumes class: title Highly available Persistent Volumes .nav[ [Previous section](#toc-local-persistent-volumes) | [Back to table of contents](#toc-module-0) | [Next section](#toc-) ] .debug[(automatically generated title slide)] --- # Highly available Persistent Volumes - How can we achieve true durability? - How can we store data that would survive the loss of a node? -- - We need to use Persistent Volumes backed by highly available storage systems - There are many ways to achieve that: - leveraging our cloud's storage APIs - using NAS/SAN systems or file servers - distributed storage systems -- - We are going to see one distributed storage system in action .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- ## Our test scenario - We will set up a distributed storage system on our cluster - We will use it to deploy a SQL database (PostgreSQL) - We will insert some test data in the database - We will disrupt the node running the database - We will see how it recovers .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- ## Portworx - Portworx is a *commercial* persistent storage solution for containers - It works with Kubernetes, but also Mesos, Swarm ... - It provides [hyper-converged](https://en.wikipedia.org/wiki/Hyper-converged_infrastructure) storage (=storage is provided by regular compute nodes) - We're going to use it here because it can be deployed on any Kubernetes cluster (it doesn't require any particular infrastructure) - We don't endorse or support Portworx in any particular way (but we appreciate that it's super easy to install!) .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- ## A useful reminder - We're installing Portworx because we need a storage system - If you are using AKS, EKS, GKE ... you already have a storage system (but you might want another one, e.g. to leverage local storage) - If you have setup Kubernetes yourself, there are other solutions available too - on premises, you can use a good old SAN/NAS - on a private cloud like OpenStack, you can use e.g. Cinder - everywhere, you can use other systems, e.g. Gluster, StorageOS .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- ## Portworx requirements - Kubernetes cluster ✔️ - Optional key/value store (etcd or Consul) ❌ - At least one available block device ❌ .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- ## The key-value store - In the current version of Portworx (1.4) it is recommended to use etcd or Consul - But Portworx also has beta support for an embedded key/value store - For simplicity, we are going to use the latter option (but if we have deployed Consul or etcd, we can use that, too) .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- ## One available block device - Block device = disk or partition on a disk - We can see block devices with `lsblk` (or `cat /proc/partitions` if we're old school like that!) - If we don't have a spare disk or partition, we can use a *loop device* - A loop device is a block device actually backed by a file - These are frequently used to mount ISO (CD/DVD) images or VM disk images .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- ## Setting up a loop device - We are going to create a 10 GB (empty) file on each node - Then make a loop device from it, to be used by Portworx .exercise[ - Create a 10 GB file on each node: ```bash for N in $(seq 1 4); do ssh node$N sudo truncate --size 10G /portworx.blk; done ``` (If SSH asks to confirm host keys, enter `yes` each time.) - Associate the file to a loop device on each node: ```bash for N in $(seq 1 4); do ssh node$N sudo losetup /dev/loop4 /portworx.blk; done ``` ] .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- ## Installing Portworx - To install Portworx, we need to go to https://install.portworx.com/ - This website will ask us a bunch of questions about our cluster - Then, it will generate a YAML file that we should apply to our cluster -- - Or, we can just apply that YAML file directly (it's in `k8s/portworx.yaml`) .exercise[ - Install Portworx: ```bash kubectl apply -f ~/container.training/k8s/portworx.yaml ``` ] .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- class: extra-details ## Generating a custom YAML file If you want to generate a YAML file tailored to your own needs, the easiest way is to use https://install.portworx.com/. FYI, this is how we obtained the YAML file used earlier: ``` KBVER=$(kubectl version -o json | jq -r .serverVersion.gitVersion) BLKDEV=/dev/loop4 curl https://install.portworx.com/1.4/?kbver=$KBVER&b=true&s=$BLKDEV&c=px-workshop&stork=true&lh=true ``` If you want to use an external key/value store, add one of the following: ``` &k=etcd://`XXX`:2379 &k=consul://`XXX`:8500 ``` ... where `XXX` is the name or address of your etcd or Consul server. .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- ## Waiting for Portworx to be ready - The installation process will take a few minutes .exercise[ - Check out the logs: ```bash stern -n kube-system portworx ``` - Wait until it gets quiet (you should see `portworx service is healthy`, too) <!-- ```longwait PX node status reports portworx service is healthy``` ```key ^C``` --> ] .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- ## Dynamic provisioning of persistent volumes - We are going to run PostgreSQL in a Stateful set - The Stateful set will specify a `volumeClaimTemplate` - That `volumeClaimTemplate` will create Persistent Volume Claims - Kubernetes' [dynamic provisioning](https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/) will satisfy these Persistent Volume Claims (by creating Persistent Volumes and binding them to the claims) - The Persistent Volumes are then available for the PostgreSQL pods .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- ## Storage Classes - It's possible that multiple storage systems are available - Or, that a storage system offers multiple tiers of storage (SSD vs. magnetic; mirrored or not; etc.) - We need to tell Kubernetes *which* system and tier to use - This is achieved by creating a Storage Class - A `volumeClaimTemplate` can indicate which Storage Class to use - It is also possible to mark a Storage Class as "default" (it will be used if a `volumeClaimTemplate` doesn't specify one) .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- ## Check our default Storage Class - The YAML manifest applied earlier should define a default storage class .exercise[ - Check that we have a default storage class: ```bash kubectl get storageclass ``` ] There should be a storage class showing as `portworx-replicated (default)`. .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- class: extra-details ## Our default Storage Class This is our Storage Class (in `k8s/storage-class.yaml`): ```yaml kind: StorageClass apiVersion: storage.k8s.io/v1beta1 metadata: name: portworx-replicated annotations: storageclass.kubernetes.io/is-default-class: "true" provisioner: kubernetes.io/portworx-volume parameters: repl: "2" priority_io: "high" ``` - It says "use Portworx to create volumes" - It tells Portworx to "keep 2 replicas of these volumes" - It marks the Storage Class as being the default one .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- ## Our Postgres Stateful set - The next slide shows `k8s/postgres.yaml` - It defines a Stateful set - With a `volumeClaimTemplate` requesting a 1 GB volume - That volume will be mounted to `/var/lib/postgresql/data` - There is another little detail: we enable the `stork` scheduler - The `stork` scheduler is optional (it's specific to Portworx) - It helps the Kubernetes scheduler to colocate the pod with its volume (see [this blog post](https://portworx.com/stork-storage-orchestration-kubernetes/) for more details about that) .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- .small[ ```yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: postgres spec: selector: matchLabels: app: postgres serviceName: postgres template: metadata: labels: app: postgres spec: schedulerName: stork containers: - name: postgres image: postgres:11 volumeMounts: - mountPath: /var/lib/postgresql/data name: postgres volumeClaimTemplates: - metadata: name: postgres spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 1Gi ``` ] .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- ## Creating the Stateful set - Before applying the YAML, watch what's going on with `kubectl get events -w` .exercise[ - Apply that YAML: ```bash kubectl apply -f ~/container.training/k8s/postgres.yaml ``` <!-- ```hide kubectl wait pod postgres-0 --for condition=ready``` --> ] .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- ## Testing our PostgreSQL pod - We will use `kubectl exec` to get a shell in the pod - Good to know: we need to use the `postgres` user in the pod .exercise[ - Get a shell in the pod, as the `postgres` user: ```bash kubectl exec -ti postgres-0 su postgres ``` <!-- autopilot prompt detection expects $ or # at the beginning of the line. ```wait postgres@postgres``` ```keys PS1="\u@\h:\w\n\$ "``` ```key ^J``` --> - Check that default databases have been created correctly: ```bash psql -l ``` ] (This should show us 3 lines: postgres, template0, and template1.) .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- ## Inserting data in PostgreSQL - We will create a database and populate it with `pgbench` .exercise[ - Create a database named `demo`: ```bash createdb demo ``` - Populate it with `pgbench`: ```bash pgbench -i -s 10 demo ``` ] - The `-i` flag means "create tables" - The `-s 10` flag means "create 10 x 100,000 rows" .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- ## Checking how much data we have now - The `pgbench` tool inserts rows in table `pgbench_accounts` .exercise[ - Check that the `demo` base exists: ```bash psql -l ``` - Check how many rows we have in `pgbench_accounts`: ```bash psql demo -c "select count(*) from pgbench_accounts" ``` <!-- ```key ^D``` --> ] (We should see a count of 1,000,000 rows.) .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- ## Find out which node is hosting the database - We can find that information with `kubectl get pods -o wide` .exercise[ - Check the node running the database: ```bash kubectl get pod postgres-0 -o wide ``` ] We are going to disrupt that node. -- By "disrupt" we mean: "disconnect it from the network". .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- ## Disconnect the node - We will use `iptables` to block all traffic exiting the node (except SSH traffic, so we can repair the node later if needed) .exercise[ - SSH to the node to disrupt: ```bash ssh `nodeX` ``` - Allow SSH traffic leaving the node, but block all other traffic: ```bash sudo iptables -I OUTPUT -p tcp --sport 22 -j ACCEPT sudo iptables -I OUTPUT 2 -j DROP ``` ] .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- ## Check that the node is disconnected .exercise[ - Check that the node can't communicate with other nodes: ``` ping node1 ``` - Logout to go back on `node1` <!-- ```key ^D``` --> - Watch the events unfolding with `kubectl get events -w` and `kubectl get pods -w` ] - It will take some time for Kubernetes to mark the node as unhealthy - Then it will attempt to reschedule the pod to another node - In about a minute, our pod should be up and running again .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- ## Check that our data is still available - We are going to reconnect to the (new) pod and check .exercise[ - Get a shell on the pod: ```bash kubectl exec -ti postgres-0 su postgres ``` <!-- ```wait postgres@postgres``` ```keys PS1="\u@\h:\w\n\$ "``` ```key ^J``` --> - Check the number of rows in the `pgbench_accounts` table: ```bash psql demo -c "select count(*) from pgbench_accounts" ``` <!-- ```key ^D``` --> ] .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- ## Double-check that the pod has really moved - Just to make sure the system is not bluffing! .exercise[ - Look at which node the pod is now running on ```bash kubectl get pod postgres-0 -o wide ``` ] .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- ## Re-enable the node - Let's fix the node that we disconnected from the network .exercise[ - SSH to the node: ```bash ssh `nodeX` ``` - Remove the iptables rule blocking traffic: ```bash sudo iptables -D OUTPUT 2 ``` ] .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- class: extra-details ## A few words about this PostgreSQL setup - In a real deployment, you would want to set a password - This can be done by creating a `secret`: ``` kubectl create secret generic postgres \ --from-literal=password=$(base64 /dev/urandom | head -c16) ``` - And then passing that secret to the container: ```yaml env: - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: name: postgres key: password ``` .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- class: extra-details ## Troubleshooting Portworx - If we need to see what's going on with Portworx: ``` PXPOD=$(kubectl -n kube-system get pod -l name=portworx -o json | jq -r .items[0].metadata.name) kubectl -n kube-system exec $PXPOD -- /opt/pwx/bin/pxctl status ``` - We can also connect to Lighthouse (a web UI) - check the port with `kubectl -n kube-system get svc px-lighthouse` - connect to that port - the default login/password is `admin/Password1` - then specify `portworx-service` as the endpoint .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- class: extra-details ## Removing Portworx - Portworx provides a storage driver - It needs to place itself "above" the Kubelet (it installs itself straight on the nodes) - To remove it, we need to do more than just deleting its Kubernetes resources - It is done by applying a special label: ``` kubectl label nodes --all px/enabled=remove --overwrite ``` - Then removing a bunch of local files: ``` sudo chattr -i /etc/pwx/.private.json sudo rm -rf /etc/pwx /opt/pwx ``` (on each node where Portworx was running) .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- class: extra-details ## Dynamic provisioning without a provider - What if we want to use Stateful sets without a storage provider? - We will have to create volumes manually (by creating Persistent Volume objects) - These volumes will be automatically bound with matching Persistent Volume Claims - We can use local volumes (essentially bind mounts of host directories) - Of course, these volumes won't be available in case of node failure - Check [this blog post](https://kubernetes.io/blog/2018/04/13/local-persistent-volumes-beta/) for more information and gotchas .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- ## Acknowledgements The Portworx installation tutorial, and the PostgreSQL example, were inspired by [Portworx examples on Katacoda](https://katacoda.com/portworx/scenarios/), in particular: - [installing Portworx on Kubernetes](https://www.katacoda.com/portworx/scenarios/deploy-px-k8s) (with adapatations to use a loop device and an embedded key/value store) - [persistent volumes on Kubernetes using Portworx](https://www.katacoda.com/portworx/scenarios/px-k8s-vol-basic) (with adapatations to specify a default Storage Class) - [HA PostgreSQL on Kubernetes with Portworx](https://www.katacoda.com/portworx/scenarios/px-k8s-postgres-all-in-one) (with adaptations to use a Stateful Set and simplify PostgreSQL's setup) ??? :EN:- Using highly available persistent volumes :EN:- Example: deploying a database that can withstand node outages :FR:- Utilisation de volumes à haute disponibilité :FR:- Exemple : déployer une base de données survivant à la défaillance d'un nœud .debug[[k8s/portworx.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/k8s/portworx.md)] --- class: title, self-paced Thank you! .debug[[shared/thankyou.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/shared/thankyou.md)] --- class: title, in-person That's all, folks! <br/> Questions?  .debug[[shared/thankyou.md](https://github.com/jpetazzo/container.training/tree/2020-05-ardan/slides/shared/thankyou.md)]