Client-go

The Client Library– k8s.io/client-go

The kubernetes programming interface in Go mainly consists of the k8s.io/client-go library (for brevity we will just call it client-go going foward).

k8s.io/client-go is a typical web service client library that supports all API types that are officially part of Kubernetes. It can be used to execute the usual REST verbs.

  • Create
  • Get
  • List
  • Update
  • Delete
  • Patch
  • Watch

For each Kubernetes 1.x.y release, there is a client-go release with a matching tag kubernetes-1.x.y.

Most of your code that speaks to Kubernetes APIs will use tools/clientcmd/ to set up a client from a kubeconfig file, and kubernetes/ for the actual Kubernetes API clients.

Kubernetes API Types–k8s.io/api

As we have seen, client-go holds the client interfaces. The Kubernetes API Go types for objects like pods, services, and deployments are located in their oven repository. It is accessed as ki8.io/api in Go code.

Pods are part of the legacy API group(often also called the “core” group) version v1 Hence, the Pod Go types is found in k8s.io/api/core/v1, and similarly for all other API types in Kubernetes. The actual Go types are contained in a type.go file. In addition, there are other files, most of them automatically generated by a code generator.

1# k8s.io/api/core/v1
2Pod
3Service
4ReplicaSet

API Machinery–k8s.io/apimachinery

Last but not least, there is a third repository called API Machinery, which is used as k8s.io/apimachinery in Go. It include all the generic building blocks to implement a Kubernetes-like API. API machinery is not restricted to container management, so, for example, it could be used to build APIs for an online shop or any other business-specific domain.

Nevertheless, you’ll meet a lot of API Machinery packages in Kubernetes-native Go code. An important one is k8s.io/apimachinery/pkg/api/meta/v1. It contains many of the generic API types such as ObjectMeta, TypeMeta, GetOptions, and ListOptions.

1# pkg/apis/meta/v1
2ObjectMeta
3TypeMeta
4ListOptions
5DeleteOptions
6GetOptions
7Status
8Events

Create and Using a client

when running a binary inside of a pod in a cluster, the kubelet will automatically mount a service account into the container at /var/run/secrets/kuberntes.io/serviceaccount. It replaces the kubeconfig file just mentioned and can easily be turned into a rest.Config via the rest.InClusterConfig() method.

 1package main
 2
 3import (
 4	"context"
 5	"encoding/json"
 6	"flag"
 7	"fmt"
 8	"os"
 9	// "path/filepath"
10
11	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
12	"k8s.io/client-go/kubernetes"
13	// "k8s.io/client-go/rest"
14	"k8s.io/client-go/tools/clientcmd"
15)
16
17func main() {
18	kubeconfig := flag.String("kubeconfig", "./config", "kubeconfig file")
19	flag.Parse()
20
21	//************ Running a Binary Inside of a Pod in Cluster *****************
22	// config, err := rest.InClusterConfig()
23	// if err != nil {
24	// 	// fallback to kubeconfig
25	// 	kubeconfig2 := filepath.Join("~", ".kube", "config")
26	// 	if envvar := os.Getenv("KUBECONFIG"); len(envvar) > 0 {
27	// 		kubeconfig2 = envvar
28	// 	}
29	// }
30
31	// Import clientcmd from client-go in order to parse the kubernetes config.
32	// The client configuration with server name, credentials.
33	// Retrun a rest.Config
34	config, err := clientcmd.BuildConfigFromFlags("", *kubeconfig)
35	if err != nil {
36		fmt.Printf("The kuber config can not be loaded: %v\n", err)
37		os.Exit(1)
38	}
39
40	// You can enable protobuf for native kubernetes resource cliensts by modifying
41	config.AcceptContentTypes =
42		"application/vnd.kubernetes.protobuf,application/json"
43
44	config.ContentType = "application/vnd.kubernetes.protobuf"
45
46	// Clinetset contains multiple clients for all native kubernetes resources.
47	// Create actual kubernetes client set.
48	clientset, err := kubernetes.NewForConfig(config)
49	if err != nil {
50		fmt.Printf("create clientset failed: %v", err)
51		os.Exit(1)
52	}
53
54	// The Get call send and HTTP GET request to
55	//  /api/v1/namespace/calico-apiserver/pods/calico-apiserver-xxx
56	pod, err := clientset.CoreV1().Pods("calico-apiserver").Get(context.TODO(),
57		"calico-apiserver-5b68b6b54-ngfdw", metav1.GetOptions{})
58
59	if err != nil {
60		os.Exit(1)
61	}
62
63	strByte, _ := json.Marshal(pod)
64	fmt.Println(string(strByte))
65
66}

group ?

Versioning and Compatibility

We have seen in the previous section that pods are in v1 of core group. The core group actually exists in only one version today. There are other groups, though– for example, the apps groups, which exists in v1, v1beta2, and v1beta1.

​

Clients are hardcode to a version, and the application developer has to select the right API group version in order to speak to the cluster at hand.

A second aspect of compatibility is the meta API features of the API server that client-go is speaking to.

There are options structs for CURD verbs, like CreateOptions,GetOptions,UpdateOptions, and DeleteOptions. Another important one is ObjectMeta, which part of every kind. All of these are frequently extended with new features; we usually call them API machinery features.

 1// k8s.io/apimachinery/pkg/apis/meta/v1/types.go
 2// Delete options may be provided when deleting an API object
 3type DeleteOptions struct{
 4	TypeMeta
 5	GracePeriodSeconds *int64
 6	Preconditions *Preconditions
 7	OrphanDependents *bool
 8    PropagationPolicy *DeletePropagation
 9    
10    // When present, indicates that modifications should not be
11	// persisted. An invalid or unrecognized dryRun directive will
12	// result in an error response and no further processing of the
13	// request. Valid values are:
14	// - All: all dry run stages will be processed
15	// +optional
16    DryRun []string
17    // Was added in Kubernetes in 1.12.
18}

API versions and Compatibility Guarantees

Kubernetes versions all API Groups. A common Kubernetes-style versioning scheme is used, which consists of alpha, beta, and GA(general availability) versions.

  • alpha versions

    v1alpha1, v1alpha2, v2alpha1

  • beta versions

    v1beta1, v1beta2, v2beta1

  • v1, v2

    GA

In connections to API group versions, there are two important points to keep in mind.

  • API groups versions apply to API resources as a whole, like the format of pods or service.
  • Furthermore, API groups versions play a role in accessing the API.

Kubernetes Objects in Go

From the type system point of view, Kubenetes objects fulfill a Go interface called runtime.Object form the package k8s.io/apimachinery/pkg/runtime.

1// Object interface must be support by all API types registered with Scheme.
2// Since Objects in a scheme are expected to be serialized to the wire, the interface an Object
3// must provide to the Scheme allows serializers to set the kind, version, and group the object
4// is represented as. An Object may choose to return a no-op ObjectKindAccessor in case where 
5// it is not expected to be serialized.
6type Object interface{
7	GetObjectKind() schema.ObjectKind
8	DeepCopyObject() Object
9}

Here, scheme.ObjectKind is a another simple interface from k8s.io/apimachinery/pkg/runtime/schema.

1// All objects that are serialized form a Scheme encode their type information.
2// This interface is used by serialization to set type information from the Scheme onto the 
3// serialized version of an Object. For objects that cannot be serialized or have unique 
4// requirements, this interface may be no-op.
5type ObjectKind interface{
6    SetGroupVersionKind(kind GroupVersionKind)
7    GroupVersionKind() GroupVersionKind 
8}

In other words, a Kubernetes Object in Go is a data structure that can:

  • Return and set the GroupVersionKind
  • Be deep-copied

A deep copy is a clone of the data structure such that it does not share any memory the the original object. It is used wherever code has to mutate an object without modifying the original.

TypeMeta

Kubernetes objects form k8s.io/api implement the type getter and setter of scheme.ObjectKind by embedding the metav1.TypeMeta struct form the package k8s.io/apimachinery/apis/meta/v1.

 1// TypeMeta describes an individual object in an API response or request
 2// with strings representing the type of the object and its API schema version.
 3// Structures that are versioned or persisted should inline TypeMeta.
 4//
 5// +k8s:deepcopy-gen=false
 6type TypeMeta struct {
 7	// Kind is a string value representing the REST resource this object represents.
 8	// Servers may infer this from the endpoint the client submits requests to.
 9	// Cannot be updated.
10	// In CamelCase.
11	// More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
12	// +optional
13	Kind string `json:"kind,omitempty" protobuf:"bytes,1,opt,name=kind"`
14
15	// APIVersion defines the versioned schema of this representation of an object.
16	// Servers should convert recognized schemas to the latest internal value, and
17	// may reject unrecognized values.
18	// More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
19	// +optional
20	APIVersion string `json:"apiVersion,omitempty" protobuf:"bytes,2,opt,name=apiVersion"`
21}

with this, a pod declaration in Go looks like this form package k8s.io/api/core/v1/types.go.

 1// Pod is a collection of containers that can run on a host. This resource is created
 2// by clients and scheduled onto hosts.
 3type Pod struct {
 4	metav1.TypeMeta `json:",inline"`
 5	// Standard object's metadata.
 6	// More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
 7	// +optional
 8	metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`
 9
10	// Specification of the desired behavior of the pod.
11	// More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
12	// +optional
13	Spec PodSpec `json:"spec,omitempty" protobuf:"bytes,2,opt,name=spec"`
14
15	// Most recently observed status of the pod.
16	// This data may not be up to date.
17	// Populated by the system.
18	// Read-only.
19	// More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
20	// +optional
21	Status PodStatus `json:"status,omitempty" protobuf:"bytes,3,opt,name=status"`
22}

As you can see, TypeMeta is embedded. Moreover, the pod type has JSON tags that also declare TypeMeta as being inlined.

This match the YAML representation of a Pod.

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4	namespace: default
 5	name: example
 6spec:
 7	containers:
 8	- name: hello
 9	  images: debian:latest
10	  command:
11	  - /bin/sh
12	  args:
13	  - -c
14	  - echo "hello world"; sleep 10000

The version is stored in TypeMeta.APIVersion, the kind in TypeMeta.Kind.

The Core Group is different for historic reasons

Pods and many other types that were added to Kubernetes very early on are part of the core group– often also called the legacy group – which is represented by the empty string. Hence, apiVersion is just set to “v1”.

Eventually API groups were added to Kubernetes, and the group name separated by slash, was prepended to apiVersion. In the case of apps, the version would be apps/v1. Hence, the apiVersion field is actually misnamed; it stores the API group name and the version string. This is for historic reasons because apiVersion was defined when only the core group–and none of these other API groups–existed.

ObjectMeta

most top-level objects have a field of type metav1.ObjectMeta, again from the package k8s.io/apimachinery/pkgs/meta/v1.

 1// ObjectMeta id metadata that all persisted resources must have, which includes all objects user must create.
 2type ObjectMeta struct{
 3    // Name must be unique within a namespace.
 4    Name string
 5    GenerateName string
 6    // Namespace defines the space within which each name must be unique.
 7    Namespace string
 8    SelfLink string
 9    UID types.UID
10    // An opaque value that represents the internel version of this object that can be used by
11    // clients to determine when objects have changes. corresponds to a key in etcd.
12    ResourceVersion string
13    Generation int64
14    CreateTimestamp Time
15    DeleteTimestamp *Time
16    DeleteGracePeriodSeconds *int64
17    // Map of string keys and values that can be used to organize and categorize objects.
18    Labels map[string]string
19    // Annotations is an unstructrued key value map stored with a resource that may be set 
20    // by externel tools to store and retrieve arbitrary metadata.
21    Annotations map[string]string
22    OwnerReferences []OwnerReference
23    // Finalizers must be empty before the object is deleted from the registry.
24    Finalizers []string
25    // The name of cluster which the object belongs to.
26    ClusterName string
27    ManagedFields []ManagedFieldsEntry
28}

spec and status

Finally, nearly every top-level object has a spec and a status section. This conventions comes from the declarative nature of the Kubernetes API: spec is the user desire, and status it the outcome of that desire, usually filled by a controller in the system.

There are only a few exception to the spec and status convention in the system-for example, endpoints in the core groups, or RBAC objects like ClusterRole.

Client Set

A client set gives access to clients for multiple API groups and resources. In the case of kubernetes.NewForConfig(config) from k8s.io/client-go/kubernetes, we get access to all API groups and resources defined in k8s.io/api. This is, with a few exception–such as APIService and CustomerResourcesDefinition– the whole set of resources served by the Kubernetes API server.

The client set main interface in k8s.io/client-go/kubernetes/clientset.go for Kubernetes-native resources like this:

 1type Interface interface{
 2    Discover() discover.DiscoverInterface
 3    AppsV1() appsv1.AppsV1Interface
 4    AppsV1beta1() appsv1beta1.AppsV1beta1Interface
 5    AppsV1beta2() appsv1beta2.AppsV1beta2Interface
 6    AuthenticationV1() authenticationv1.AuthenticationV1Interface
 7    AuthenticationV1beta1() authenticationv1beta1.AuthenticationV1betaInterface
 8    AuthorizationV1() authorizationv1.AuthorizationV1Interface
 9    AuthorizationV1beta1() authorizationv1beta1.AuthorizationV1beta1Interface
10    ...
11}

Behind each GroupVersion method, we find the resource of the API group.

1type AppsV1beta1Interface interface {
2    RESTClinet() rest.Interface
3    ControllerRevisionGetter
4    DeploymentsGetter
5    StatefulSetsGetter
6}

With RESTClinet being a generic REST client, and one interface per resource, as in:

 1// DeploymentsGetter has a method to return a DeploymentInterface.
 2// A group's client should implement this interface.
 3type DeploymentsGetter interface{
 4    Deployments(name string) DeploymentInterface
 5}
 6
 7// DeploymentInterface has methods to  work with Deployment resources.
 8type DeploymentInterface interface {
 9	Create(ctx context.Context, deployment *v1beta1.Deployment, opts v1.CreateOptions) (*v1beta1.Deployment, error)
10	Update(ctx context.Context, deployment *v1beta1.Deployment, opts v1.UpdateOptions) (*v1beta1.Deployment, error)
11	UpdateStatus(ctx context.Context, deployment *v1beta1.Deployment, opts v1.UpdateOptions) (*v1beta1.Deployment, error)
12	Delete(ctx context.Context, name string, opts v1.DeleteOptions) error
13	DeleteCollection(ctx context.Context, opts v1.DeleteOptions, listOpts v1.ListOptions) error
14	Get(ctx context.Context, name string, opts v1.GetOptions) (*v1beta1.Deployment, error)
15	List(ctx context.Context, opts v1.ListOptions) (*v1beta1.DeploymentList, error)
16	Watch(ctx context.Context, opts v1.ListOptions) (watch.Interface, error)
17	Patch(ctx context.Context, name string, pt types.PatchType, data []byte, opts v1.PatchOptions, subresources ...string) (result *v1beta1.Deployment, err error)
18	Apply(ctx context.Context, deployment *appsv1beta1.DeploymentApplyConfiguration, opts v1.ApplyOptions) (result *v1beta1.Deployment, err error)
19	ApplyStatus(ctx context.Context, deployment *appsv1beta1.DeploymentApplyConfiguration, opts v1.ApplyOptions) (result *v1beta1.Deployment, err error)
20	DeploymentExpansion
21}

Depending on the scope of the resource–that is , whether it is cluster or namespace scoped– the accessor(here DeploymentsGetter) may or may not have a namespace argument.

Status Subresources: UpdateStatus

Deployments have a so-called status subresource. This means that UpdateStatus uses an additional HTTP endpoint suffixed with /status. While updates on the /api/apps/v1beta1/namespace/ns/deployments/name endpoints can change only the spce of the deployment, the endpoint /api/apps/v1beta1/namespace/ns/deployments/name/status can change only the status of the object.

By default the client-gen generates the UpdateStatus() method.

Listings and Deletions

DeleteCollections allows us to delete multiple objects of namespace at once. The ListOptions parameters allows us to define which objects should be deleted using a filed or label selector:

 1// k8s.io/apimachinery/pkg/apis/meta/v1/types.go
 2// ListOptions is the query options to a standard REST list call.
 3type ListOptions struct{
 4    TypeMeta
 5    // A selector to restrict the list of returned objects by their labels.
 6    LabelSelector string
 7    // A selector to restrict the list of returned objects by their fields.
 8    FieldSelector string
 9    Watch bool
10    AllowWatchBookmarks bool
11    ResourceVersion string
12    ResourceVersionMatch ResourceVersionMatch
13    TimeSecond *int64
14    Limit int64
15    Continue string 
16}

Watches

Watch gives an event interface for all changes(adds, removes, and updates) to objects. The returned watch.Interface form k8s.io/apimachinery/pkg/watch/watch.go.

 1// Interface can be implemented by anything that knows how to watch and report changes.
 2type Interface interface{
 3    // Stop watching. Will close the channel returned by ResoultChan(). Release any resources
 4    // used by watch.
 5    Stop()
 6    // Returns a chan which will receive all the events. If an errors occurs or Stop()
 7    // is canceled, the implementation will close this channel and release any resources
 8    // used by watch.
 9    ResultChan() <-chan Event
10}

The result channel of watch interface returns three kinds of events.

 1// EventType defines the possible types of events
 2type EventTypes string
 3const (
 4	Added     EventType = "ADDED"
 5    Modified  EventType = "MODIFIED"
 6    Deleted   EventType = "DELETED"
 7    Bookmark  EventType = "BOOKMARK"
 8    Error     EventType = "ERROR"
 9)
10
11// Event represents a single event to watched resource.
12Type Event struct{
13    Type EventType
14    
15    Object runtime.Object
16}

While it is tempting to use this interface directly, in practice it is actually discouraged in favor of informers.

Informers are a combination of this event interface and in-memory cache with indexed lookup. This is by far the most common use case for watches. Under the hood informers first call List on the client to get the set of all objects(as a baseline for the cache) and then Watch to update the cache.

Client Expansion

DeploymentExpansion is actually an empty interface. It is used to add custom client behavior, but it’s hardly used in Kubernetes nowdays. Instead, the client generator allows up to add custom methods in a declarative way.

Note again that all of those methods in DeploymentInterface neither expect valid information in the TypeMeta fields Kind and APIVersion, nor set those fields on Get() and List(). These fields are filled with real values only on the wire.

Client Options

Client-go

What is clientset?

ClientSet: every Resource has their own client, and ClientSet is a client set for different Resources. ClientSet on handle the k8s internal resource, generate by client-gen.

DynamicClient is for CRD resources.

DiscoverClient： find all api-resources

获取kube conf的方式：

  • file path ~/.kube/config
  • KUBECONFIG

合并多个kubeconfig 信息