shell-operator v1.0.0: the long-awaited release of our tool to create Kubernetes operators

NB: SUBSCRIBE TO OUR NEW BLOG (https://blog.deckhouse.io/) to follow articles & news about Deckhouse, shell-operator, and addon-operator!

shell-operator is an Open Source solution that takes the development of Kubernetes operators to a whole new level. It was announced almost two years ago. Many new fascinating features, refinements, and additions were introduced to shell-operator since then, and, as we stated just a couple of months ago, it is perfectly ready to be used in production*.

Despite its actual matureness, shell-operator was still having numerous beta and RC releases which might confuse its existing and potential users. This final v1.0.0 release puts an end to waiting, reaffirms the maturity of shell-operator, and brings some new exciting features to it along the way.

* There is even more: shell-operator (or more precisely, its “big brother” — addon-operator) is one of the basic building blocks of our Deckhouse Kubernetes platform! (Please wait for its public announcement this May.)

If you are still unaware of shell-operator’s mission and features, please refer to the KubeCon EU’2020 presentation (text version) — it will help you to understand how shell-operator makes life easier simplifying the creation of K8s operators. In this article, we will focus on the latest innovations only (those which emerged since our previous publication in February).

Major changes in v1.0.0

Please note that version v1.0.0 is not pro forma. It has many new improvements you can use already.

Object Patcher Framework

We write many hooks for shell-operator and addon-operator. Thus, we are well aware that hooks’ side effects interfere with testing and even obscure the operating logic of the hook in some cases.

That is why we decided to get rid of kubectl calls to eliminate the side effects and make our hooks more straightforward and “clean.”

The includeSnapshotsFrom and group parameters were implemented to get rid of the kubectl invocations related to reading operations. Also, the output of the jqFilter expression became available in the binding context of the objects: this way, you can get the needed objects from the cluster via a file and use them within the hook. These features were added to the beta.6 version. Since then, our hooks have been free of read-related kubectl invocations.

The second part of our strategy — replacing write-related kubectl invocations — we implemented as an internal library and used it for some time. Finally, in version v1.0.0, the so-called Object Patcher Framework has emerged. It allows returning of a declarative set of actions on the cluster objects from a hook.

How does it work? Suppose, a hook creates a certificate and puts it in a secret. You can do that as follows:

cat <<EOF | kubectl -n default apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: certs
type: kubernetes.io/tls    
data:
  tls.crt: |
    ${CERT}
  tls.key: |
    ${KEY}
EOF

Well, it does not seem too complicated, but:

• it would be good to verify that this secret does not exist and update its keys if it does;
• creating a test for such a hook is no easy task since you need a cluster and have to check its state after the hook is complete (another option is to replace kubectl with something more manageable and test-related).

How does the Object Patcher Framework solve these problems? The documentation suggests that you have to write the definition of the CreateOrUpdate operation to the file at $KUBERNETES_PATCH_PATH:

cat <<EOF > $KUBERNETES_PATCH_PATH
---
operation: CreateOrUpdate
object:
  apiVersion: v1
  kind: Secret
  metadata:
    name: certs
    namespace: default
  type: kubernetes.io/tls    
  data:
    tls.crt: |
      ${CERT}
    tls.key: |
      ${KEY}
EOF

Next, shell-operator will attempt to create a secret or update it if the secret exists. As a result, testing hooks has become much more straightforward: now, you just need to check the file’s contents at $KUBERNETES_PATCH_PATH.

Summing up our brief introduction to Object Patcher Framework, starting with shell-operator v1.0.0, you can write hooks that no more require kubectl invocations (and you can even get rid of this tool in the image!).

Bonus! If you love jq as much as we do, then you will definitely love the JQPatch operation.

Delaying hooks invocation

Hooks now have parameters that allow you to delay their invocation to “accumulate” events in the queue. This feature is handy if the hook can process the current state of a set of objects in a single invocation (and it is costly to run it in response to each change). If сhanges occur frequently, the queue becomes clogged. One of the solutions to this problem is to limit the frequency of running the hook.

Here is an example of the configuration:

cat <<EOF
configVersion: v1
settings:
  executionMinInterval: 5s
  executionBurst: 2
kubernetes:
- name: pods
  kind: Pod
  group: pods
EOF

This hook’s minimum execution interval is 5 seconds, and it supports two successive runs without delay (executionBurst). Note that the hook subscribes to the group: this parameter ensures that the hook will get just one binding context containing the relevant objects. Without the group parameter, the hook will get an array of binding contexts containing all the changes over the last 5 seconds. In other words, we will gain nothing in this case. You can learn more about the group parameter in the documentation.

Relevant objects for Synchronization

shell-operator, when started, begins to track changes in the Kubernetes objects and starts a hook, passing it a list of all objects that exist at the start time. We named this first run Synchronization. If an error occurs while the hook is running, it gets restarted. There was an error for Synchronization runs that led the hook to receive the same set of objects (as if it is the very first launch) even if there were changes in the cluster.

Other changes

• the Object Patcher Framework now has a dedicated Kubernetes client, and you can configure timeout and delay for it (--object-patcher-kube-client-timeout, --object-patcher-kube-client-qps, --object-patcher-kube-client-burst);
• the documentation about the group parameter is updated;
• a flag that enables the collection of debugging data coming from the Kubernetes client is introduced.

What’s next? configVersion: v2

As you can see, hooks configuration in shell-operator has become more complex. Many new settings and parameters have emerged, such as executeHookOnEvent and executeHookOnSynchronization, includeSnapshotsFrom and group, keepFullObjectsInMemory, queue (and lack of it), waitForSynchronization, etc. What’s more, you don’t need all of them at the same time — you have to pick a specific set of values to set the required subscription mode.

That is why we have created a v2 milestone for addon-operator. Please, share your ideas, thoughts, and difficulties in using shell-operator and/or addon-operator in the comments below or via GitHub issues or discussions for both projects. By the way, if you’re already enjoying them, your GitHub stars are also much appreciated! 😉

NB: SUBSCRIBE TO OUR NEW BLOG (https://blog.deckhouse.io/) to follow articles & news about Deckhouse, shell-operator, and addon-operator!