Operator SDK for K8s using Go
0 likes1,608 views
The document provides an overview of the Operator Framework, an open-source toolkit for managing Kubernetes-native applications known as operators, offering insights into its usage, workflow, and advantages over other tools. It discusses what operators and custom resource definitions (CRDs) are, their ideal use cases, and the process of creating an operator using the Operator SDK. Additionally, it includes information about a company called Coyote System that specializes in community-based driving assistant systems and offers various car solutions.
![Create your operator
mkdir -p $GOPATH/src/github.com/prune998/
cd $GOPATH/src/github.com/prune998/
operator-sdk new certmerge-operator --cluster-scoped
INFO[0000] Create pkg/apis/apis.go
INFO[0000] Create pkg/controller/controller.go
INFO[0000] Create version/version.go
INFO[0000] Create .gitignore
INFO[0000] Create Gopkg.toml
INFO[0000] Run dep ensure ...
INFO[0068] Run dep ensure done
INFO[0068] Run git init ...
INFO[0074] Run git init done
INFO[0074] Project creation complete.
INFO[0000] Creating new Go operator 'certmerge-operator'.
INFO[0000] Create cmd/manager/main.go
INFO[0000] Create build/Dockerfile
INFO[0000] Create build/bin/entrypoint
INFO[0000] Create build/bin/user_setup
INFO[0000] Create deploy/service_account.yaml
INFO[0000] Create deploy/role.yaml
INFO[0000] Create deploy/role_binding.yaml
INFO[0000] Create deploy/operator.yaml](https://image.slidesharecdn.com/sebastienthomasoperatorsdkoverview-190307184456/85/Operator-SDK-for-K8s-using-Go-22-320.jpg)
![type CertMerge struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec CertMergeSpec `json:"spec,omitempty"`
Status CertMergeStatus `json:"status,omitempty"`
}
type CertMergeSpec struct {
SecretName string `json:"name"`
Selector []SecretSelector `json:"selector"`
SecretNamespace string `json:"namespace"`
SecretList []SecretDefinition `json:"secretlist"`
}
type SecretSelector struct {
LabelSelector metav1.LabelSelector `json:"labelselector"`
Namespace string `json:"namespace"`
}
Types (API)](https://image.slidesharecdn.com/sebastienthomasoperatorsdkoverview-190307184456/85/Operator-SDK-for-K8s-using-Go-25-320.jpg)
![// Define a new Secret object
secret := newSecretForCR(instance)
// create the DATA for the new secret based on the CertMerge request
certData := make(map[string][]byte)
// Set CertMerge instance as the owner and controller (for garbage collection)
if err := controllerutil.SetControllerReference(instance, secret, r.scheme); err != nil {
return emptyRes, err
}
// build the Cert Data from the secret List provided in the CertMerge Custom Resource
if len(instance.Spec.SecretList) > 0 {
for _, sec := range instance.Spec.SecretList {
secContent, err := r.searchSecretByName(ctx, sec.Name, sec.Namespace)
...
certData[sec.Name+".crt"] = secContent.Data["tls.crt"]
certData[sec.Name+".key"] = secContent.Data["tls.key"]
}
}
// add the Data to the secret
secret.Data = certData
// create the new secret
if err := r.client.Create(ctx, secret); err != nil {...}
Reconcile 2](https://image.slidesharecdn.com/sebastienthomasoperatorsdkoverview-190307184456/85/Operator-SDK-for-K8s-using-Go-28-320.jpg)
Operator SDK for K8s using Go
- 1. https://github.com/operator-framework/operator-sdk K8s / CNCF Meetup - 2019/02/25 Operator Framework From Github Project : The Operator Framework is an open source toolkit to manage Kubernetes native applications, called Operators, in an effective, automated, and scalable way. prune@lecentre.net
- 2. Agenda 1. Overview of Operators 2. Operator SDK usage 3. Operators workflow 4. Hands On 5. Conclusion
- 3. COYOTE SYSTEM Who are we ? A leading provider of community-based driving assistant systems Founded in 2005 350 employees 1M daily users on a payed subscription model 2 billion kilometers traveled every month (1,24 billion miles) 50M members declaration analyzed every month Specific know-how in big data and automotive market protected by 13 patents
- 4. COYOTE SYSTEM How to be part of the community ? • Available on a range of Products and Apps, • … but also with embedded car solutions COYOTE mini COYOTE S COYOTE NAV+ Smartphone APPs
- 5. COYOTE SYSTEM Where does it work ? … almost everywhere in Europe ! › France › Belgium › Netherlands › Luxembourg › Italy › Spain › Germany › Poland › Portugal
- 6. Who I am ? 20+ years in Computers / Network / Admin / Devops / Woodworker Work at Coyote https://www.moncoyote.com/ as System Architect Github : https://github.com/prune998 Blog (sort of) : https://medium.com/@prune998 Coyote Lab Blog (more to come there) : https://www.mycoyote.ca/blog Contact : Sebastien “Prune” THOMAS - prune@lecentre.net
- 7. What’s an Operator ? An Operator is an application that deals with the Kubernetes API and Custom Resources to create/operate new Resources. It’s an intelligent piece of software that embed the templating to deploy your resources. The Operator watch events on the K8s API and react (ex : re-create a pod, change Labels, update a Secret, Remove a Service…)
- 8. What are Custom Resource Definition CRD are new Resources, like Pods, Deployments, Secrets that you can create. They are managed through the K8s API the same way as official resources kubectl get crd certificates.certmanager.k8s.io 2019-01-25T15:56:53Z certmerges.certmerge.lecentre.net 2019-01-25T15:57:10Z prometheuses.monitoring.coreos.com 2019-01-25T16:05:42Z prometheusrules.monitoring.coreos.com 2019-01-25T16:05:44Z virtualservices.networking.istio.io 2019-01-25T16:09:16Z ...
- 9. Example Custom Resource apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: certmerges.certmerge.lecentre.net spec: group: certmerge.lecentre.net names: kind: CertMerge listKind: CertMergeList plural: certmerges singular: certmerge scope: Namespaced version: v1alpha1 apiVersion: certmerge.lecentre.net/v1alpha1 kind: CertMerge metadata: name: "test-certmerge-labels" spec: selector: - labelselector: matchLabels: env: "dev" certmerge: "true" namespace: default name: test-cert-labels namespace: default Custom Resource Definition (CRD) Custom Resource Manifest (CR)
- 10. Before Operators
- 11. With Operators
- 13. Operators
- 14. Difference with other tools - Helm / Jsonnet / Ksonnet They are templating tools. Create a template, set some variables, generate the Manifests. Once deployed they have no control (tiller does not count). - StatefulSets / Deployments / Pods They are K8s Resources. Some minimal feedback to scale/restart, no dependency between them, no intelligence in management. - Operators Watch the K8s API and react in real time. Can have a better control to scale/restart/configure the target application, with richer features than Readyness/Liveness Probes
- 15. Who needs Operators ? You may need an Operator if : - you need to use many times the same Application. ex : deploying one EtcD cluster in each Namespace - You need to automate some Resource creation. ex : create some SSL Certificates inside Secrets (cert-manager), create Prometheus scraping rules - You need more intelligence in the management. ex : the Etcd-Operator create and manage Pods directly instead of using a Deployment or StatefulSets
- 16. Helm Chart to deploy an Operator ? - Operators are usually easy to deploy - use whatever mean you have to deploy them (Helm, Jsonnet, plain manifest from the Operator creator) - Once the Operator is running, use the Custom Resources to trigger its power
- 17. Existing Operators ? - Etcd-Operator : https://github.com/coreos/etcd-operator - Kafka : https://github.com/strimzi/strimzi-kafka-operator - Nats : https://github.com/nats-io/nats-operator - Prometheus : https://github.com/coreos/prometheus-operator - SSL Certificates : https://github.com/jetstack/cert-manager - RBAC-Manager : https://github.com/reactiveops/rbac-manager and a lot more, growing... (check https://github.com/operator-framework/awesome-operators)
- 18. Operator all the thing ? An Operator embed the knowledge and the deployments “templates”. Don’t create an operator : - if your application deployment is not stable ! - to deploy one application per cluster (it’s easier to template it) Create an Operator : - if you have many users in need to use your resource - you have a complicated workflow to handle your resource - you want to (learn to) code in GO (or check other languages operators too)
- 19. Operator Creation Operator SDK (Go) : https://github.com/operator-framework/operator-sdk - High level APIs and abstractions to write the operational logic more intuitively - Tools for scaffolding and code generation to bootstrap a new project fast - Extensions to cover common operator use cases - Base on official Kubernetes API packages - Provide common package for leader election for HA Operators
- 20. CertMerge Operator - github.com/prune998/certmerge-operator
- 21. Install (fast) mkdir -p $GOPATH/src/github.com/operator-framework cd $GOPATH/src/github.com/operator-framework git clone https://github.com/operator-framework/operator-sdk cd operator-sdk git checkout v0.4.0 make dep make install operator-sdk --version operator-sdk version v0.4.0+git
- 22. Create your operator mkdir -p $GOPATH/src/github.com/prune998/ cd $GOPATH/src/github.com/prune998/ operator-sdk new certmerge-operator --cluster-scoped INFO[0000] Create pkg/apis/apis.go INFO[0000] Create pkg/controller/controller.go INFO[0000] Create version/version.go INFO[0000] Create .gitignore INFO[0000] Create Gopkg.toml INFO[0000] Run dep ensure ... INFO[0068] Run dep ensure done INFO[0068] Run git init ... INFO[0074] Run git init done INFO[0074] Project creation complete. INFO[0000] Creating new Go operator 'certmerge-operator'. INFO[0000] Create cmd/manager/main.go INFO[0000] Create build/Dockerfile INFO[0000] Create build/bin/entrypoint INFO[0000] Create build/bin/user_setup INFO[0000] Create deploy/service_account.yaml INFO[0000] Create deploy/role.yaml INFO[0000] Create deploy/role_binding.yaml INFO[0000] Create deploy/operator.yaml
- 23. Add API # Add a new API for the custom resource AppService operator-sdk add api --api-version=certmerge.lecentre.net/v1alpha1 --kind=CertMerge This is the basic operation to create the CRD. It creates files in pkg/apis/certmerge/v1alpha1 including certmerge_types.go which holds the definition of the CRD : … type CertMerge struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec CertMergeSpec `json:"spec,omitempty"` Status CertMergeStatus `json:"status,omitempty"` } ...
- 24. Add Controler # Add a new controller that watches for AppService operator-sdk add controller --api-version=certmerge.lecentre.net/v1alpha1 --kind=CertMerge Creates files in pkg/controller/certmerge. This is where all your watch and reconcile logic happens Check doc reference at https://godoc.org/github.com/kubernetes-sigs/controller-runtime/pkg#hdr-Controller
- 25. type CertMerge struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec CertMergeSpec `json:"spec,omitempty"` Status CertMergeStatus `json:"status,omitempty"` } type CertMergeSpec struct { SecretName string `json:"name"` Selector []SecretSelector `json:"selector"` SecretNamespace string `json:"namespace"` SecretList []SecretDefinition `json:"secretlist"` } type SecretSelector struct { LabelSelector metav1.LabelSelector `json:"labelselector"` Namespace string `json:"namespace"` } Types (API)
- 26. // add adds a new Controller to mgr with r as the reconcile.Reconciler func add(mgr manager.Manager, r reconcile.Reconciler) error { // Create a new controller c, err := controller.New("certmerge-controller", mgr, controller.Options{Reconciler: r}) if err != nil { return err } // Watch for changes to primary resource CertMerge err = c.Watch(&source.Kind{Type: &certmergev1alpha1.CertMerge{}}, &handler.EnqueueRequestForObject{}) if err != nil { return err } // TODO(user): Modify this to be the types you create that are owned by the primary resource // Watch for changes to secondary resource Pods and requeue the owner CertMerge err = c.Watch(&source.Kind{Type: &corev1.Secret{}}, &handler.EnqueueRequestForOwner{ IsController: true, OwnerType: &certmergev1alpha1.CertMerge{}, }) if err != nil { return err } Watchers
- 27. func (r *ReconcileCertMerge) Reconcile(request reconcile.Request) (reconcile.Result, error) { … // Fetch the CertMerge instance that triggered this Reconsile instance := &certmergev1alpha1.CertMerge{} err := r.client.Get(context.TODO(), request.NamespacedName, instance) if err != nil { if errors.IsNotFound(err) { // Request object not found, could have been deleted after reconcile request. // Owned objects are automatically garbage collected. For additional cleanup logic use finalizers. // Return and don't requeue ( by sending `nil` in the error field) return reconcile.Result{}, nil } // Error reading the object - requeue the request. (by sending a non-nil error) return reconcile.Result{}, err } … do some stuff for your operator (see next slide) } Reconcile
- 28. // Define a new Secret object secret := newSecretForCR(instance) // create the DATA for the new secret based on the CertMerge request certData := make(map[string][]byte) // Set CertMerge instance as the owner and controller (for garbage collection) if err := controllerutil.SetControllerReference(instance, secret, r.scheme); err != nil { return emptyRes, err } // build the Cert Data from the secret List provided in the CertMerge Custom Resource if len(instance.Spec.SecretList) > 0 { for _, sec := range instance.Spec.SecretList { secContent, err := r.searchSecretByName(ctx, sec.Name, sec.Namespace) ... certData[sec.Name+".crt"] = secContent.Data["tls.crt"] certData[sec.Name+".key"] = secContent.Data["tls.key"] } } // add the Data to the secret secret.Data = certData // create the new secret if err := r.client.Create(ctx, secret); err != nil {...} Reconcile 2
- 29. Generate and build # re-generate all the files that depend on the CRD API operator-sdk generate k8s # re-generate the CRD Manifest (rarely used, when you change your API name) operator-sdk generate openapi # build the operator (aka go build) operator-sdk build prune/certmerge-operator:v0.0.1
- 31. Operator workflow (harder with Predicate)
- 32. Hands-on switch to the console / code
- 33. Conclusion ● Operator SDK make it really easy ● using K8s primitives (and go-client), not “vendor” dependent ● Operators can be declined in Controlers (admission) ● You need to learn a little bit of the K8s API to get to cool stuff
- 34. References ● https://github.com/operator-framework/operator-sdk ● https://github.com/operator-framework/operator-sdk/blob/master/doc/user-guide.md ● https://godoc.org/github.com/kubernetes-sigs/controller-runtime/pkg#hdr-Controller ● https://github.com/operator-framework/operator-sdk/blob/master/doc/user/event-filtering.md