If you like this post, you can learn more about operators from my book, The Kubernetes Operator Framework Book.

---

Recently my team and I were working on a feature for our operator when we came across a state-reconciliation bug that was potentially serious. What’s worse, we almost didn’t catch it. This was especially surprising as a diverse team of developers that were both new and experienced with Kubernetes. The tl;dr was that it is important for an operator to watch both its inputs and outputs.

The bug we missed

For some background, the operand managed by our operator accepts a fairly complex configuration to set up. So, we were working on an operator that watched Custom Resources (created by the user) and generated a ConfigMap based on those CRs. This ConfigMap is then used by the operand component as the source of its runtime settings, and the Custom Resources abstract the complexity of the underlying config from the user.

While making changes around the code that generated the ConfigMaps, we almost introduced a regression in which the operator would have stopped watching the generated ConfigMaps that it wrote. Thankfully, this was caught before the code merged. But it was only noticed by chance during code review, when it was something an automated pre-merge test should have caught.

Why is this important?

From a happy-path perspective, the operator worked fine. The user would create their CR, the operator would notice it, and the ConfigMap would be generated. If the user made any changes to their CR, those would be picked up too.

But what if something happened to generated ConfigMap? Say an accidental kubectl delete on the ConfigMap shuts down the component. Or worse, an attacker inserts their own malicious settings directly into the ConfigMap. If the operator isn’t keeping an eye on its output (just as it does with its input) then it can’t recover from situations like this. In this case, it is effectively useless as a state reconciliation controller.

State Reconciliation in Kubernetes

State reconciliation is not just an operator topic, it is one of the foundational concepts of Kubernetes. In my book, I briefly discuss this idea in comparing operators to built-in Kubernetes controllers:

These controllers work by monitoring the current state of the cluster and comparing it to the desired state. One example is a ReplicaSet with a specification to maintain three replicas of a Pod. Should one of the replicas fail, the ReplicaSet quickly identifies that there are now only two running replicas. It then creates a new Pod to bring stasis back to the cluster.

In this example, the ReplicaSet controller that’s watching the whole cluster has an input and an output:

• Input: The spec.replicas field on the ReplicaSet object, where the user says how many replicas they want. This is the desired state of the cluster.
• Output: The Pods created by the ReplicaSet controller that are now running in the cluster. This represents the actual state of the cluster.

The reconciliation of desired state vs actual state is a core function of Kubernetes. It’s almost taken for granted that Kubernetes will always maintain the number running Pods in the cluster to match the number you’ve set in your ReplicaSet (or Deployment).

In terms of our operator, the same idea still applies:

• Input / desired state: Custom Resources created by the user.
• Output / actual state: The generated ConfigMap written by the operator.

In our case, the operator failing to reconcile on any changes to its written ConfigMap is no different than a ReplicaSet failing to create new Pods. Obviously, that would not be an acceptable regression in Kubernetes. So why was this almost overlooked in our operator?

Operators can be confusing

Operators present a whole new field of possibilities in cloud architecture, with unfamiliar concepts and confusing terminology that may seem intimidating to new users and experienced Kubernetes developers alike. These new concepts can make it seem like you must forget everything you thought you knew to make room for something new. With a horizon this broad, it can be easy to disconnect from the familiar as the unknown begins to overwhelm.

But at their core, operators are just Kubernetes controllers with custom logic. This took me a long time to grasp as I fought my way through buzzword soup and exuberant documentation to understand this hot new tech. But ultimately, if you can understand how ReplicaSets, Deployments, or almost any other Kubernetes resource works, then you already have a good understanding of operator fundamentals.

Reducing the idea of operators down to a familiar concept helped me understand them much better. It helped me to have some background with upstream Kubernetes development, but I believe that it can also work the other way around. Lowering the barrier to working with operators helps make many of the development practices used by core Kubernetes authors more approachable to new contributors. Because at the end of the day, operators are just custom extensions of Kubernetes.

I talk about this idea with more context in my book, The Kubernetes Operator Framework Book, because I hope that it will help readers to understand not only operator development, but Kubernetes development as a whole.

As of today, I am officially a published author. My first book, The Kubernetes Operator Framework Book, is now available from Amazon and Packt (in both print and ebook).

In this book, I try to offer a casual introduction to Kubernetes Operators and the experience of building and maintaining one. If you or someone on your team is familiar with Kubernetes but curious about Operators, I think The Kubernetes Operator Framework Book will be a great resource for getting up to speed with the core concepts. It was my goal with this book to provide a friendly, welcoming tutorial with hands-on application of the most important topics, which I think new users will find easy to understand.

Writing this book was a personal challenge for myself, as well as an opportunity to contribute back to the Kubernetes community which has helped shape my own career. It’s an incredibly rewarding feeling to see it available for sale, but I was completely shocked to see that it was the #1 New Release in several categories!

It’s exciting to be a part of this technology. I’m grateful for the opportunity to have written this book, and I hope you will find it helpful, informative, and enjoyable to read. Looking forward to the next one!

If you are working on changes to a core Kubernetes component such as the scheduler, controller-manager, or apiserver, you may be confused about how to actually build and deploy your changes in a local environment to test them. This post will go over the method I use to build and test control plane component changes. There are probably more efficient ways to do it, but this works for me and gives me some flexibility over each step which I will explain.

I am usually developing custom schedulers, so that will be the focus for these steps (though as you will see, it pretty much applies to any kube control-plane pod). So this post may be particularly useful if you are also working on a custom scheduler, or plugins for the scheduler, or are simply trying to debug the default kube-scheduler.

Building control plane component images

The first step (after making any code changes) is to run make quick-release-images from the root of the k8s.io/kubernetes repository.

This will build release images for all the core components:

~/go/src/k8s.io/kubernetes$ make quick-release-images 
+++ [1103 14:32:11] Verifying Prerequisites....
+++ [1103 14:32:11] Using Docker for MacOS
+++ [1103 14:32:12] Building Docker image kube-build:build-841902342a-5-v1.15.2-1
+++ [1103 14:32:18] Syncing sources to container
+++ [1103 14:32:21] Running build command...
+++ [1103 14:33:25] Building go targets for linux/amd64:
    cmd/kube-apiserver
    cmd/kube-controller-manager
    cmd/kube-scheduler
    cmd/kube-proxy
    vendor/github.com/onsi/ginkgo/ginkgo
    test/e2e/e2e.test
    cluster/images/conformance/go-runner
    cmd/kubectl
+++ [1103 14:34:31] Syncing out of container
+++ [1103 14:34:40] Building images: linux-amd64
+++ [1103 14:34:40] Starting docker build for image: kube-apiserver-amd64
+++ [1103 14:34:40] Starting docker build for image: kube-controller-manager-amd64
+++ [1103 14:34:40] Starting docker build for image: kube-scheduler-amd64
+++ [1103 14:34:40] Starting docker build for image: kube-proxy-amd64
+++ [1103 14:34:40] Building conformance image for arch: amd64
+++ [1103 14:35:05] Deleting docker image k8s.gcr.io/kube-scheduler-amd64:v1.20.0-beta.1.5_c82d5ee0486191-dirty
+++ [1103 14:35:08] Deleting docker image k8s.gcr.io/kube-proxy-amd64:v1.20.0-beta.1.5_c82d5ee0486191-dirty
+++ [1103 14:35:15] Deleting docker image k8s.gcr.io/kube-controller-manager-amd64:v1.20.0-beta.1.5_c82d5ee0486191-dirty
+++ [1103 14:35:15] Deleting docker image k8s.gcr.io/kube-apiserver-amd64:v1.20.0-beta.1.5_c82d5ee0486191-dirty
+++ [1103 14:35:24] Deleting conformance image k8s.gcr.io/conformance-amd64:v1.20.0-beta.1.5_c82d5ee0486191-dirty
+++ [1103 14:35:24] Docker builds done

And it places them as individual .tar files in ./_output/release-images/amd64:

$ ls _output/release-images/amd64/
total 817M
drwxr-xr-x 7 mdame staff  224 Nov  3 14:35 .
drwxr-xr-x 3 mdame staff   96 Nov  3 14:34 ..
-rw-r--r-- 1 mdame staff 288M Nov  3 14:35 conformance-amd64.tar
-rw------- 2 mdame staff 159M Nov  3 14:35 kube-apiserver.tar
-rw------- 2 mdame staff 150M Nov  3 14:35 kube-controller-manager.tar
-rw------- 2 mdame staff 130M Nov  3 14:35 kube-proxy.tar
-rw------- 2 mdame staff  63M Nov  3 14:35 kube-scheduler.tar

These can be loaded into docker, tagged, and pushed to your registry of choice:

$ docker load -i _output/release-images/amd64/kube-scheduler.tar 
46d8985b3ff2: Loading layer [==================================================>]  60.37MB/60.37MB
Loaded image: k8s.gcr.io/kube-scheduler-amd64:v1.20.0-beta.1.5_c82d5ee0486191-dirty
$ docker tag k8s.gcr.io/kube-scheduler-amd64:v1.20.0-beta.1.5_c82d5ee0486191-dirty docker.io/mdame/kube-scheduler:test
$ docker push docker.io/mdame/kube-scheduler:test
The push refers to repository [docker.io/mdame/kube-scheduler]
46d8985b3ff2: Pushed 
c12e92a17b61: Layer already exists 
79d541cda6cb: Layer already exists 
printconfig: digest: sha256:c093dad1f4a60ce1830d91ced3a350982b921183c1b4cd9bfaae6f70c67af7ee size: 949

Deploying your custom image into a Kind cluster

(Note: Kind actually provides ways for you to build and load code changes into a cluster, see https://kind.sigs.k8s.io/docs/user/quick-start/#loading-an-image-into-your-cluster and https://kind.sigs.k8s.io/docs/user/quick-start/#building-images. So this may be overkill, but allows you to make changes to an already-running cluster or with images you’ve already built).

Start up a stock Kind cluster and get the current static pod manifest for the kube-scheduler (use docker ps to get the running cluster’s container, then the static pod manifests are located at /etc/kubernetes/manifests):

 

$ kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.19.1) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂

$ docker ps
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS                       NAMES
b4f712eff358        kindest/node:v1.19.1   "/usr/local/bin/entr…"   8 minutes ago       Up 8 minutes        127.0.0.1:58206->6443/tcp   kind-control-plane
$ docker exec -it b4f712eff358 cat /etc/kubernetes/manifests/kube-scheduler.yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-scheduler
    tier: control-plane
  name: kube-scheduler
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-scheduler
    - --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
    - --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
    - --bind-address=127.0.0.1
    - --kubeconfig=/etc/kubernetes/scheduler.conf
    - --leader-elect=true
    - --port=0
    image: k8s.gcr.io/kube-scheduler:v1.19.1
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10259
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: kube-scheduler
    resources:
      requests:
        cpu: 100m
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10259
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /etc/kubernetes/scheduler.conf
      name: kubeconfig
      readOnly: true
  hostNetwork: true
  priorityClassName: system-node-critical
  volumes:
  - hostPath:
      path: /etc/kubernetes/scheduler.conf
      type: FileOrCreate
    name: kubeconfig
status: {}

Copy this and edit it to refer to my custom scheduler image (truncated here):

$ cat kube-scheduler.yaml 
apiVersion: v1
kind: Pod
...
spec:
  containers:
  - command:
    - kube-scheduler
    - --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
    - --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
    - --bind-address=127.0.0.1
    - --kubeconfig=/etc/kubernetes/scheduler.conf
    - --leader-elect=true
    - --port=0
    image: docker.io/mdame/kube-scheduler:test
    imagePullPolicy: Always
...

Copy this manifest into the Kind cluster, overwriting the existing manifest:

$ docker cp kube-scheduler.yaml <cluster-container-ID>:/etc/kubernetes/manifests/kube-scheduler.yaml

And now the running pod should refer to your image:

$ kubectl get -o yaml pod/kube-scheduler-kind-control-plane -n kube-system
apiVersion: v1
kind: Pod
metadata:
...
  name: kube-scheduler-kind-control-plane
  namespace: kube-system
...
spec:
  containers:
  - command:
    - kube-scheduler
    - --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
    - --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
    - --bind-address=127.0.0.1
    - --kubeconfig=/etc/kubernetes/scheduler.conf
    - --leader-elect=true
    - --port=0
    image: docker.io/mdame/kube-scheduler:test
    imagePullPolicy: Always

 

Deploying changes to the image

If you need to deploy changes to the running pod, repeat the make quick-release-images and docker load/tag/push steps from above. Your new image won’t be pulled just by deleting the pod even with imagePullPolicy: Always set, because these are static pods mirroring a running container. So, to pull a new image you need to exec back into the cluster and stop the running container:

$ docker exec -it <container-ID> crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                      ATTEMPT             POD ID
cfcb6d257aa59       0c3dd0a038e11       47 minutes ago      Running             kube-scheduler            7                   83163425eb615
... ... ...
$ docker exec -it <container-ID> crictl stop cfcb6d257aa59
cfcb6d257aa59

And you should see a new pod start with your image pulled (as mentioned above, you could also load the changes directly into Kind’s registry and pull from there).

When I recently switched to a Mac for my personal/work laptop, I ran into this problem trying to build oc:

# github.com/apcera/gssapi
vendor/github.com/apcera/gssapi/name.go:213:9: could not determine kind of name for C.wrap_gss_canonicalize_name
cgo:
clang errors for preamble:
vendor/github.com/apcera/gssapi/name.go:90:2: error: unknown type name 'gss_const_name_t'
       gss_const_name_t input_name,
       ^
1 error generated.
make: *** [build] Error 2

This led me to this helpful comment that explains the gssapi headers on Mac are outdated. This was fixed by installing the heimdal HomeBrew package:

brew install heimdal

(Note: This post originally appeared on the Red Hat OpenShift Blog)

OpenShift Online recently made a new feature available to Pro accounts: Collaboration. Collaboration allows a Pro account user to provision cluster account access for other users, called collaborators. These collaborators have normal access to the same cluster as the Pro account (without any resource quotas or ability to create new projects) and thus can be granted permissions to work on projects owned by the Pro account.

Every Pro account has the ability to add up to 50 collaborators to their subscription. Collaborators are free cluster accounts provisioned by the Pro subscription owner. In addition, one collaborator account can be used for collaboration with multiple different Pro subscriptions (this is important to note for security reasons because, as we’ll discuss later in more detail, simply removing a collaborator from your subscription does not necessarily remove them from all cluster access).

Collaboration will greatly improve workflows for teams choosing to host their projects on OpenShift Online, as previously the only way to have multiple cluster accounts was to sign up for an additional Pro subscription for each account. Along with saving money, Collaboration also saves time by creating a simpler method for creating new accounts.

Adding Collaborators to Your Subscription

To get started, first you will need an OpenShift Online Pro account (if that wasn’t obvious already). From there, each user you wish to add as a collaborator will need to create a free account at developers.redhat.com.

Once your collaborator has confirmed their Red Hat Developers account, you can add them to your subscription. First, have the user sign into developers.redhat.com and click on their name in the upper-right corner to access their account details. Make note of the “Red Hat Login ID” on this page, as it is the username you will be required to enter in order to associate the Collaborator with your subscription.

Now sign in to manage.openshift.com and click on Manage Subscription under the cluster you wish to add them to.

Once you are in the subscription management console, click the new Manage link beside Collaborators, which will bring you to the collaborators page.

On the collaborator page, enter the Red Hat Login ID for the user in the username field and click Add collaborator

You should now see the user listed under your Collaborators, as well as the time that they were added and an option to remove them from your subscription.

Note that this does not automatically grant the user any access to your projects. Access will need to be granted manually by the project owner, most likely you, using OpenShift policy commands.

Granting Collaborators Project Access

Once a collaborator account has been provisioned on the cluster, they will have the ability to be given permissions to access any project on the cluster. They can also use the same account to collaborate under multiple different subscriptions.

Note that while this does mean that a collaborator provisioned by Subscription A can be given access to any other Subscription B ’s projects without counting toward the collaborator limit for Subscription B, if the collaborator is removed from Subscription A at any time that collaborator’s cluster account will be deprovisioned (and they will lose all access to Subscription B ’s projects).

Thus, the only way to guarantee persistence of collaborator accounts for as long as your subscription is active is to add them to your subscription through the “Manage Collaborators” page. Simply granting a user project access is not enough to make them permanent, but it is required for them to see your projects.

There are two ways to grant project access to a user on the cluster:

Granting Access with the CLI

One way is to log in to the cluster through the CLI using your access token and use oc policy add-role-to-user to give the user a role using the same username listed on the Collaboration page:

~$ oc login https://api.openshift.com --token=

Logged into "https://api.openshift.com:443" as "mdame" using the token provided.

You have one project on this server: "mdame-collab"

Using project "mdame-collab".

~$ oc policy add-role-to-user view collaborator-1234

role "view" added: "collaborator-1234"

(This example grants “view” access to the project for user “collaborator-1234”. Learn more about access roles here.)

Using the Web Console

Another method is to use the OpenShift Web Console (by clicking the Open Web Console link in the subscription manager) and navigating to Resources > Membership:

On the next page, click Edit Membership

From here you can add the collaborator by their username, select the appropriate role, and click Add. When you’re finished, click Done Editing

Now, when the user signs in to manage.openshift.com, they will see a card to log in to the web console for the same cluster as your subscription and, if they’ve been granted it, will have access to your projects on the cluster just like any other user.

Removing Collaborators

If at any time you wish to remove the user as a Collaborator from your subscription, you can do so on the same Collaboration page you used to add them (either by checking multiple collaborators and using the Remove Selected button to batch-remove multiple users, or by clicking the red Remove button next to each collaborator to remove them one-by-one).

It is important to note, however, that removing a collaborator from your account will not automatically remove any access roles you have assigned the user in your projects. These will need to be manually deleted (similar to how they were created) or the user may still have access to your projects. Because one user can be a collaborator on the same cluster for multiple subscriptions, simply removing them from your subscription may not remove their cluster access.

As part of the team that developed this feature, we are all very excited about the launch of Collaboration in OpenShift Online. This feature will help teams and organizations work together with the benefits of a hosted OpenShift cluster, and we hope you have as much fun using it as we did building it.

Part 1: Introduction to the OpenShift client
Part 2: Coding for Kubernetes

In the previous post, I went more in-depth into how Kubernetes works in the command line. For this post, let’s step back to the code from Part 1 of this series and modify it to run continuously, showing the list of pods currently in the cluster and updating every time a new pod is created.

The Watch Interface

Kubernetes’ Watch interface provides the ability to listen for several different types of events in the cluster. Using the channel functionality built into Go, this is perfect for us to set up an asynchronous controller that can run continuously. First we need to update our code to take advantage of channels. Go back to the code we had in Part 1 and change the main() function in your cmd/controller/cmd.go file to look like this:

func main() {
        config, err := clientcmd.DefaultClientConfig(pflag.NewFlagSet("empty", pflag.ContinueOnError)).ClientConfig()
        kubeClient, err := kclient.New(config)
        if err != nil {
                log.Printf("Error creating cluster config: %s", err)
                os.Exit(1)
        }
        openshiftClient, err := osclient.New(config)
        if err != nil {
                log.Printf("Error creating OpenShift client: %s", err)
                os.Exit(2)
        }

        c := controller.NewController(openshiftClient, kubeClient)
        stopChan := make(chan struct{})
        c.Run(stopChan)
        <-stopChan
}

What we’ve done is created a channel that will be used to safely send “stop” signals to our Go routines when the program ends, and passed that channel to our Run() function. Now, update your pkg/controller/controller.go file to look like this:

package controller

import (
        "fmt"
        "time" // New import

        osclient "github.com/openshift/origin/pkg/client"
        "github.com/openshift/origin/pkg/cmd/util/clientcmd"

        "github.com/spf13/pflag"
        kapi "k8s.io/kubernetes/pkg/api"
        "k8s.io/kubernetes/pkg/api/meta"
        kclient "k8s.io/kubernetes/pkg/client/unversioned"
        "k8s.io/kubernetes/pkg/runtime"
        "k8s.io/kubernetes/pkg/util/wait" // New import
        "k8s.io/kubernetes/pkg/watch"  // New import
)

type Controller struct {
        openshiftClient *osclient.Client
        kubeClient      *kclient.Client
        mapper          meta.RESTMapper
        typer           runtime.ObjectTyper
        f               *clientcmd.Factory
}

func NewController(os *osclient.Client, kc *kclient.Client) *Controller {

        f := clientcmd.New(pflag.NewFlagSet("empty", pflag.ContinueOnError))
        mapper, typer := f.Object()

        return &Controller{
                openshiftClient: os,
                kubeClient:      kc,
                mapper:          mapper,
                typer:           typer,
                f:               f,
        }
}

func (c *Controller) Run(stopChan <-chan struct{}) {
        // Run, aysnchronously, until receive a stop signal
        go wait.Until(func() {
                // Create a Watch Interface for Kubernetes Pods
                w, err := c.kubeClient.Pods(kapi.NamespaceAll).Watch(kapi.ListOptions{})
                if err != nil {
                        fmt.Println(err)
                }
                if w == nil {
                        return
                }

                // Listen for events on Watch Interface's channel
                for {
                        select {
                        case event, ok := <-w.ResultChan():
                                c.ProcessEvent(event, ok)
                        }
                }
        }, 1*time.Millisecond, stopChan)
}

// Function to handle incoming events
func (c *Controller) ProcessEvent(event watch.Event, ok bool) {
        if !ok {
                fmt.Println("Error received from watch channel")
        }
        if event.Type == watch.Error {
                fmt.Println("Watch channel error")
        }

        // Type switch, to handle different events
        switch t := event.Object.(type) {
        case *kapi.Pod:
                fmt.Printf("%s pod %s in namespace %s\n", event.Type, t.ObjectMeta.Name, t.ObjectMeta.Namespace)
        default:
                fmt.Printf("Unknown type\n")
        }
}

What we’ve done here is change our Run() function to:

• [41] Accept our stop channel as a parameter
• [43] Spawn a Go routine that will run continuously using Kubernetes’ wait.Until() function
• [45] Create a Watch Interface using the Kubernetes client to listen for pods in all namespaces
• [56] Listen for events over the Watch Interface’s result channel inside a select statement.
• [73-78] Process the event for the object type expected (in this case “pod”, but we could handle events for multiple types of objects) and print metadata about that event and the relevant resource object.

Deploy any simple app on your running and configured OpenShift cluster, then build and run your controller. You should see output similar to this:

ADDED pod ruby-hello-world-1-build in namespace test
ADDED pod docker-registry-1-deploy in namespace default

(Your output will obviously vary based on the app you use. I just used the sample Ruby hello world app.)

These events show up because, when starting up, a Watch Interface will receive ADDED events for all currently running pods. If I leave the controller running and, in another terminal, delete my ruby-hello-world pod, I see this output added:

MODIFIED pod ruby-hello-world-1-build in namespace test
MODIFIED pod ruby-hello-world-1-build in namespace test
DELETED pod ruby-hello-world-1-build in namespace test

So you can see how different interactions on your cluster can trigger different types of events.

Note that the OpenShift 3.3 client package includes easy access to Watch Interfaces for several additional types, such as Projects.

Fun: Cumulative Runtimes

As an exercise, let’s modify our controller to keep track of the cumulative runtime of all the pods for each namespace. Update your pkg/controller/controller.go file to change your ProcessEvent() function and add a new function, TimeSince(), like so:

func (c *Controller) ProcessEvent(event watch.Event, ok bool) {
        if !ok {
                fmt.Println("Error received from watch channel")
        }
        if event.Type == watch.Error {
                fmt.Println("Watch channel error")
        }

        var namespace string
        var runtime float64
        switch t := event.Object.(type) {
        case *kapi.Pod:
                podList, err := c.kubeClient.Pods(t.ObjectMeta.Namespace).List(kapi.ListOptions{})
                if err != nil {
                        fmt.Println(err)
                }
		for _, pod := range podList.Items {
                        runtime += c.TimeSince(pod.ObjectMeta.CreationTimestamp.String())
                }
                namespace = t.ObjectMeta.Namespace
        default:
                fmt.Printf("Unknown type\n")
        }
        fmt.Printf("Pods in namespace %v have been running for %v minutes.\n", namespace, runtime)
}

func (c *Controller) TimeSince(t string) float64 {
        startTime, err := time.Parse("2006-01-02 15:04:05 -0700 EDT", t)
        if err != nil {
                fmt.Println(err)
        }
        duration := time.Since(startTime)
        return duration.Minutes()
}

Now, whenever a new pod event is received (such as adding or deleting), we’ll trigger a client call to gather a list of all the running pods in the relevant namespace. From the CreationTimestamp of each pod, we use the time.Since() method to calculate how long it’s been running in minutes. From there it’s just a matter of summing up all the runtimes we’ve calculated. When you run it, the output should be similar to this:

Pods in namespace default have been running for 1112.2476349382832 minutes.
Pods in namespace test have been running for 3.097702110216667 minutes.

Try scaling the pods in a project up or down, and see how it triggers a new calculation each time. This is a very simple example, but hopefully it’s enough to get you on your way writing your own controllers for OpenShift!

Part 1: Introduction to the OpenShift client
Part 3: Writing a controller

In my last post, I went over how to set up an OpenShift environment for developing. That tutorial used the OpenShift Client API to make function calls which interacted with our cluster. However, there may be a situation where you need more direct interaction with your cluster’s resources (or perhaps you are interested in contributing to the open-source OpenShift Origin repository), and even the provided function calls aren’t enough to satiate your needs. In this case, it’s good to know how OpenShift interacts with Kubernetes directly to serve your cluster resources hot-and-ready.

Kubernetes Resource Objects

Going back to the code we used in the last post, you can recall that we used an OpenShift Project Interface to get a list of the projects in our cluster. If you examine the code for the Project Interface, you can see that it uses a REST Client to get the requested information. However, certain commands such as oc get (which is really just a wrapper for Kubernetes’ kubectl get command) rely on the Kubernetes Client API to request the necessary resource objects. How exactly Kubernetes achieves this can be a bit confusing, so let’s modify our code from the last blog post to use a Kubernetes client (as opposed to the OpenShift client) and walk through it as an example of the advantages using OpenShift can give you as a developer.

Update your pkg/controller/controller.go file to look like this:

package controller

import (
        "fmt"

        osclient "github.com/openshift/origin/pkg/client"
        "github.com/openshift/origin/pkg/cmd/util/clientcmd"

        "github.com/spf13/pflag"
        kapi "k8s.io/kubernetes/pkg/api"
        "k8s.io/kubernetes/pkg/api/meta"
        "k8s.io/kubernetes/pkg/kubectl/resource"
        kclient "k8s.io/kubernetes/pkg/client/unversioned"
        "k8s.io/kubernetes/pkg/runtime"
)

type Controller struct {
        openshiftClient *osclient.Client
        kubeClient      *kclient.Client
        mapper          meta.RESTMapper
        typer           runtime.ObjectTyper
        f               *clientcmd.Factory
}

func NewController(os *osclient.Client) *Controller {

        // Create mapper and typer objects, for use in call to Resource Builder
        f := clientcmd.New(pflag.NewFlagSet("empty", pflag.ContinueOnError))
        mapper, typer := f.Object()

        return &Controller{
                openshiftClient: os,
                kubeClient:      kc,
                mapper:          mapper,
                typer:           typer,
                f:               f,
        }
}

func (c *Controller) Run() {
        /*                                                                                                                                                                                                           
                // Old code from last post using OpenShift client                                                                                                                                                                          
                projects, err := c.openshiftClient.Projects().List(kapi.ListOptions{})                                                                                                                               
                if err != nil {                                                                                                                                                                                      
                        fmt.Println(err)                                                                                                                                                                             
                }                                                                                                                                                                                                    
                for _, project := range projects.Items {                                                                                                                                                             
                        fmt.Printf("%s\n", project.ObjectMeta.Name)                                                                                                                                                  
                }                                                                                                                                                                                                    
        */

        // Resource Builder function call, to get Result object
        r := resource.NewBuilder(c.mapper, c.typer, resource.ClientMapperFunc(c.f.ClientForMapping), kapi.Codecs.UniversalDecoder()).
                ResourceTypeOrNameArgs(true, "projects").
                Flatten().
                Do()

        // Use Visitor interface to iterate over Infos in previous Result
        err := r.Visit(func(info *resource.Info, err error) error {
                fmt.Printf("%s\n", info.Name)
                return nil
        })
        if err != nil {
                fmt.Println(err)
        }
}

Build and run and you should see the same output as you did before. So what did we change here?

Some new imports

We added the clientcmd and pflag packages so we can use them to create a Factory object, which gives us our mapper and typer (more on that in a bit). This part could have been done in our main cmd/controller/cmd.go file, with the Factory object passed to the new controller as a parameter, but for brevity I just added it here. meta and runtime are also for the mapper and typer, respectively. Finally, resource  allows us to interact with the Kubernetes Resource Builder client functions.

Resource Builder

The resource package provides us with client functions of the Builder type. The call to NewBuilder() takes four arguments: a RESTMapper, an ObjectTyper, a ClientMapper, and a Decoder (the names give a pretty good idea of what each object does, but I’ve linked to their docs pages if you want to know more). The Builder type provides numerous functions which serve as parameters in a request for resources. In this case, I call ResourceTypeOrNameArgs(true, “projects”) and Flatten() on my Builder. The ResourceTypeOrNameArgs() call lets me specify which type of resource I’d like, and request any objects of that specific type by name. Since I just want all of the projects in the cluster, though, I set the first parameter to “true” (which allows me to blankly select all resources). The Resource Builder Flatten() function returns the results as an iterable list of Info objects (but that’s getting a little ahead of ourselves). Finally, Do() returns a Result object.

The “Result”

In my opinion, this is kind of a semantic misnomer. For the developer new to Kubernetes, it would be assumed that the “result” is the data you originally requested. In reality, it’s an object containing metadata about the result as well as a way to access the actual data, through structures called Infos. There are a few ways to get to these Info objects, one is to simply call .Infos() on the Result object to return a list of Infos. Another, slightly more elegant method, is to use the Visitor Interface.

Visitor Function

Calling .Visit() on a Result object allows you to provide a function which will iterate over each Info object in the Result. Infos themselves provide some helpful metadata on the resource they describe, such as Name and Namespace, but they also give you access to the full generic runtime.Object representation of the resource. By casting these objects to their actual types, you can access the fields and methods specific to that type. As an example, let’s update our Visit() function like so:

err := r.Visit(func(info *resource.Info, err error) error {
        switch t := info.Object.(type) {
        case *projectapi.Project:
                fmt.Printf("%s is currently %s\n", t.ObjectMeta.Name, t.Status.Phase)
        default:
                return fmt.Errorf("Unknown type")
        }
        return nil
})

And also add the following line to your imports: projectapi “github.com/openshift/origin/pkg/project/api”. Save, build, and run and you’ll see output like this:

default is currently Active
openshift is currently Active
openshift-infra is currently Active

Now we’re casting the runtime.Object to a Project type and, using the OpenShift Project API, getting information about its name and status. As a side-note, this makes use of Go’s type switching which is very cool.

Summary

To summarize, Kubernetes’ method for retrieving your objects goes through several different types and interfaces: Builder -> Result -> Visitor -> Info -> Object -> Typecast. Normally, this approach would be more appropriate if you were writing your own command-line arguments. It’s very helpful to have an understanding of how Kubernetes interacts with your cluster on a low-level, but as you can see it’s much simpler to use OpenShift client function calls to get the information you want. Our example here is a bit acrobatic, but still demonstrates the flexibility that working with OpenShift and Kubernetes provides.

In the next post, I’ll go over how to actually make your controller run like a controller (asynchronously, listening for updates) using the Watch package.

Click for Part 3: Writing a controller

Part 2: Coding for Kubernetes
Part 3: Writing a controller

For OpenShift Online, we run several controllers in our cluster which serve functions such as provisioning persistent volumes and providing user analytics. But let’s say you have your own OpenShift cluster, upon which you’d like to run a controller that interacts with the resources in that cluster. I’m going to run you through setting up OpenShift and Kubernetes in a way that allows you to develop your own controller. By the end of this guide, we’ll have a simple controller that shows the cumulative running time for all pods in a namespace.

Setting Up Your Environment

The prerequisites to develop for your OpenShift setup are:

• OpenShift installed as per our “Contributing” docs
  • This includes installing Go, Git, and Docker, as well as configuring your Go environment. That page will teach you how to build and run OpenShift from source, which is very helpful if you’re interested in contributing to the project!
• Godep installed. This will be used for dependency management.

Assuming you’ve followed the instructions on that page to set up your GOPATH and have Origin cloned, the next step is to download the source code dependencies for OpenShift and Kubernetes. Do this with the following commands:

cd $GOPATH/src/github.com/openshift/origin
git checkout release-1.2

git clone git://github.com/kubernetes/kubernetes $GOPATH/src/k8s.io/kubernetes
cd $GOPATH/src/k8s.io/kubernetes
git remote add openshift git://github.com/openshift/kubernetes
git fetch openshift
git checkout v1.2.0-36-g4a3f9c5

git clone https://github.com/go-inf/inf.git $GOPATH/src/speter.net/go/exp/math/dec/inf

cd $GOPATH/src/github.com/openshift/origin
godep restore

What we’re doing here is:

1. Checking out the most recent release branch of OpenShift
2. Cloning the Kubernetes repository
3. Adding OpenShift’s vendored version of Kubernetes as a remote to our Kubernetes repository
4. Checking out the required release of OpenShift’s Kubernetes
   1. This can be found by opening origin/Godeps/Godeps.json, searching for “Kubernetes” and copying the version number specified in “comment“
5. Cloning another dependency
6. And finally running godep restore to download the source for all the dependencies needed

At this point, we’re ready to start coding!

Creating Your Project

In this post, we’re going to make a simple run-once program that lists the namespaces (projects) in a cluster. That means this program won’t run continuously like you would think of a controller (we’ll add that in a later post), but is more of a basic introduction to the client tools used to write such a program.

First, create a GitHub repo with the following file structure:

controller/
- cmd/
-- controller/
- pkg/
-- controller/

• cmd/controller/ will contain the main package file for your controller
• pkg/controller/ will contain source files for your controller package

Now create a file called cmd/controller/cmd.go with the following contents:

package main

import (
        "fmt"
        "log"
        "os"

        "github.com/damemi/controller/pkg/controller"
        _ "github.com/openshift/origin/pkg/api/install"
        osclient "github.com/openshift/origin/pkg/client"
        "github.com/openshift/origin/pkg/cmd/util/clientcmd"

        kclient "k8s.io/kubernetes/pkg/client/unversioned"
        "github.com/spf13/pflag"
)

func main() {
        var openshiftClient osclient.Interface
        config, err := clientcmd.DefaultClientConfig(pflag.NewFlagSet("empty", pflag.ContinueOnError)).ClientConfig()
        kubeClient, err := kclient.New(config)
        if err != nil {
                log.Printf("Error creating cluster config: %s", err)
                os.Exit(1)
        }
        openshiftClient, err = osclient.New(config)
        if err != nil {
                log.Printf("Error creating OpenShift client: %s", err)
                os.Exit(2)
        }
}

Save the file, close it, and run godep save ./… in your directory. You should see that your file structure has changed to:

controller/
- cmd/
-- controller/
- pkg/
-- controller/
- Godeps/
-- Godeps.json
-- Readme
- vendor/
-- github.com/
--- [...]
-- golang.org/
--- [...]
-- [...]

I’ve excluded some of the files, because they’re just dependency source files. These are now included in your project, so feel free to commit and push this to your repo. One of the cool things about Godep vendoring in this way is that now, you can share your codebase and allow someone else to build it without needing to worry about submodules or other dependency issues.

Note: We won’t be able to build our controller yet due to the fact that this code has some defined-and-unused variables, but it is enough to run Godep. For when we do start building our code, I’ll be using a Makefile  with the following:

all:
        go install github.com/damemi/controller/cmd/controller

Just because it’s easier to type “make” each time.

Adding Some Functionality

As fun as setting up a project is, it’s even more fun to make it do things. Create a file pkg/controller/controller.go with the following contents:

package controller

import (
        "fmt"

        osclient "github.com/openshift/origin/pkg/client"

        kclient "k8s.io/kubernetes/pkg/client/unversioned"
        kapi "k8s.io/kubernetes/pkg/api"
)

// Define an object for our controller to hold references to
// our OpenShift client
type Controller struct {
        openshiftClient *osclient.Client
        kubeClient *kclient.Client
}

// Function to instantiate a controller
func NewController(os *osclient.Client, ) *Controller {
        return &Controller{
                openshiftClient: os,
                kubeClient:      kc,
        }
}

// Our main function call
func (c *Controller) Run() {
        // Get a list of all the projects (namespaces) in the cluster
        // using the OpenShift client
        projects, err := c.openshiftClient.Projects().List(kapi.ListOptions{})
        if err != nil {
                fmt.Println(err)
        }

        // Iterate through the list of projects
        for _, project := range projects.Items {
                fmt.Printf("%s\n", project.ObjectMeta.Name)
        }
}

As you can see, we’re using the OpenShift API to request a Project Interface, which provides plenty of helper functions to interact with the projects in our cluster (in this case, we’re using List()). The Project API is what allows us to actually interact with the meta data about each project object using the kapi.ObjectMeta field. I highly recommend reading through the OpenShift and Kubernetes APIs to get an idea of what’s really available for you.

Now let’s also add the following lines to the main() function in our cmd/controller/cmd.go file:

c := controller.NewController(openshiftClient, kubeClient)
c.Run()

Making that entire function look like:

func main() {
        config, err := clientcmd.DefaultClientConfig(pflag.NewFlagSet("empty", pflag.ContinueOnError)).ClientConfig()
        kubeClient, err := kclient.New(config)
        if err != nil {
                log.Printf("Error creating cluster config: %s", err)
                os.Exit(1)
        }
        openshiftClient, err := osclient.New(config)
        if err != nil {
                log.Printf("Error creating OpenShift client: %s", err)
                os.Exit(2)
        }

        c := controller.NewController(openshiftClient, kubeClient)
        c.Run()
}

Now save, close, and run “make”. Now, you should be able to just run “controller” from your command line and, assuming you have OpenShift running already and are logged in as system:admin, you should see some output like so:

default
openshift
openshift-infra

Hooray! We can connect to OpenShift and get information about our cluster. In the next post, we’ll go into more detail about how OpenShift uses Kubernetes’ Resource API to interact on a lower level with the cluster, as well as how to use the Watch API to make our controller run asynchronously.

Click for Part 2: Kubernetes Resource Objects