Docker tagging using REST API

Adding additional tags to the image will be a common process in the CICD world, the additional tags would be something like the build number, test result, etc.,

When adding multiple tags it will be very heavy if we are doing the tagging locally and then pushing the tagged image to the server. So instead of doing like that, we can use the docker HTTP REST API which will be lighting fast tagging and will add the tag to the remote docker repository

Below is the example of tagging an image in a remote repository using python

def addAssociatedTag(imageName, tag, associatedTag, bearerToken):
    header = {'Authorization': ''}
    header['Authorization'] = 'Bearer ' + bearerToken
    header['Accept'] = 'application/vnd.docker.distribution.manifest.v2+json'
    res = requests.get(
            url="https://docker.io/v2/" + imageName + "/manifests/" + tag,
            headers=header)
    print('Retrieved the metifests status is ' + str(res.status_code))   
    # Add the associated tag by passing the same manifests
    header['Accept'] = '*'
    res = requests.request(
            "PUT",
            url="https://docker.io/v2/" + imageName + "/manifests/" + associatedTag,
            headers=header,
            data=res.content)      
    response_status = res.status_code
    print('Adding associated tag for ' + imageName + ':' + tag + ' with ' + imageName + ':' + associatedTag + ' is = ' + str(response_status))
    return response_status

In the above, the setting the 'Accept' header is very important while getting the manifests. If we do not set it to 'application/vnd.docker.distribution.manifest.v2+json'then we will have the default 'application/json' which will not be correct when we use the content for adding the associated tags and we will get below error

{
    "errors": [
        {
            "code": "MANIFEST_LAYER_UNKNOWN",
            "message": "blob unknown to registry",
            "detail": {
                "digest": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
            }
        }
    ]
}

Details about other docker REST APIs are in https://github.com/venkatesh-mohanram/continuous-learning/blob/master/docker/httpapiv2.md

To know a Docker image exists with REST API

 Most of the time when it comes to docker, we play with using the CLI with 'docker' command. If we want to pull an image, tag an image, push an image we do all that with CLI only. However, apart from CLI, the docker repository supports varieties of REST API to do plenty of things and here I am planning to cover a few things like below

Manifest resource

The manifest rest resource can be used in a way how we want, for eg: if we want to know whether the image exists with a given tag then we can use the GET method of it

def checkAlreadyPresent(imageName, tag, bearerToken):
    auth_header = {'Authorization': ''}
    auth_header['Authorization'] = 'Bearer ' + bearerToken
    res = requests.get(
            url="https://docker.io/v2/" + imageName + "/manifests/" + tag,
            headers=auth_header)
    return res.status_code

You can also refer to

https://github.com/venkatesh-mohanram/continuous-learning/blob/master/docker/httpapiv2.md

03 Kubernetes Secret for storing Oracle ATP wallet

Kubernetes Secret are used to store secrets during the setup of the cluster and then we can mount the same inside the docker containers. In this example, I am using it to store the Oracle ATP wallet which is used to talk to the ATP instance

kubectl create secret generic db-user-pass 
        --from-file=./cwallet.sso 
        --from-file=./ewallet.p12 
        --from-file=./keystore.jks 
        --from-file=./ojdbc.properties 
        --from-file=./sqlnet.ora 
        --from-file=./tnsnames.ora 
        --from-file=./truststore.jks

This is using the command line, apart from that even we can have a Secret Kind file similar to Deployment Kind and set it up using the 'kubectl apply'.

After this, we need to mount the secret as a volume and use it inside the container

apiVersion: v1
kind: Deployment
metadata:
  name: addition-svc-deployment
  labels:
    name: addition-svc
spec:
  volumes:
  - name: secret-volume
    secret:
      secretName: db-user-pass
  containers:
  - name: addition-svc-container
    image: addition-svc:latest
    volumeMounts:
    - name: secret-volume
      readOnly: true
      mountPath: "/etc/secret/atp-wallet"

Other links:
http://venkateshbook.blogspot.com/2019/05/kubernetes-commands.html
http://venkateshbook.blogspot.com/2019/05/kubernetes-yaml-definitions.html
http://venkateshbook.blogspot.com/2019/05/02-kubernetes-service-definition.html

02 Kubernetes Service Definition

Kubernetes service is an abstraction above the pods, the pods cannot be accessed/exposed outside the cluster. Inorder to expose the pods outside the cluster, we need to define service.

Service Definition yaml

https://raw.githubusercontent.com/venkatesh-mohanram/microservices/master/01-microservices-starter-kit/add-service/k8s-service.yaml

The selector tells which deployment needs to be abstracted. The port specifies the service port and the targetPort specify the port in which the deployment is exposed. 

Use the below command to execute the service definition

$kubectl apply -f k8s-service.yaml

We can use the below command to see the clusterIP in which the service is made available. The default IP is clusterIP, this IP will be accessible within the cluster.

$kubectl get service

In order to test the service, we can run the CURL command with the service IP

$curl http://10.107.33.6/addservice/application.wadl

We can do a roling update of containers inside the pod, any rolling update on the pod will not affect the service IP and we can still use the same IP.

01 Kubernetes Deployment Definition

Let's begin with writing a 'deployment.yaml' file, we can directly write yaml for a pod but that is not recommended. In the deployment yaml, we can specify the pod template and even the replication factor

Deployment Definition Yaml

https://raw.githubusercontent.com/venkatesh-mohanram/microservices/master/01-microservices-starter-kit/add-service/k8s-deployment.yaml

Here the "Kind:" specifies what kind of k8s resource it is, in this case it is 'Deployment'. It has the metadata and the spec information inside the definition document. 

Applying the deployment

Execute the deployment yaml using the below command

$ kubectl apply -f k8s-deployment.yaml

Upon executing, we should see something like below

Monitoring container logs

$kubectl logs -f add-service-deployment-59996d5779-5q8kj

We can use above log command to see the container logs inside the particular pod.

Rolling update

$ kubectl set image deployments/add-service-deployment add-service=venkateshm/add-service:3

For rolling updates, we can execute the above commands to recreate the pods with new version of the container. The old pods will be gracefully terminated while the new one is gradually created.

Follow on https://venkateshbook.blogspot.com/2019/05/02-kubernetes-service-definition.html for kubernetes service definition

Kubernetes Commands

There are various basic commands which are helpful to begin. Along with that, it will be good, if we know the hierarchy of different resources in Kubernetes

Let's start from the micro level of K8S resources

1. POD

The POD is the logical group of containers. The containers will not directly run inside the node instead, it will be running inside a POD. It is always a good idea to group tightly coupled containers in a single POD along with any shared storage volumes if needed. Each POD in the node will have its unique IP.

Useful commands

$ kubectl get pods 

$ kubectl describe pods [pod-name]

$ kubectl describe pods --help 

$ kubectl logs [pod-name]

$ kubectl delete pod <>pod-name 

$ kubectl delete pods -all

The usual structure of all the K8S command is kubectl <action> <resource>

The very interesting command which I like most is 

$kubectl exec -ti <pod_name> bash

The above command actually takes us inside the POD and we can execute the commands inside it.

Deployment Vs PODS:

We can think of deployment as a template and the PODs are the actual deployments. In java language, the deployment is a class and PODS are objects, the replicas of deployment will create those many numbers of pods.

$ kubectl get deployments 

$ kubectl scale deployments/<deployment-name> --replicas=4 

$ kubectl describe deployment

2. Node

The node could be a VM or a real machine in K8S, the collection of nodes form a cluster. The K8S master monitors PODs and Nodes health and recreates the PODs in another node if the if one of the nodes is not healthy.

$ kubectl get nodes 

$ kubectl describe nodes [nodename]

3. Services

Is an abstraction which defines a logical set of PODs. It enables loose coupling between pods, it is preferred to be defined using YAML or JSON. The PODS are actually accessible only inside the cluster using their unique IP, if we want to expose the PODs outside then we need to define services. Also the service acts like a load balancer and distributes the load across multiple replicas.

There are different ways in which a service can be exposed in different ways by specifying the type

1. Cluster IP - Using the default cluster IP, accessible only inside the cluster
2. NodePort - Can be accessed outside by using the node IP and the port number specifies in NodePort
3. LoadBalancer - In exposed a public IP if the cloud provider supports
4. External name - using a CNAME

$ kubectl get services 

$ kubectl describe services/<service name> 

$ kubectl delete service

Other general commands

To get the cluster information

$ kubectl cluster-info

Rolling updates

$ kubectl rollout

If we have the definitions in yaml file then 

$ kubectl apply -f <loc of yaml>

Below link are good to get started with learning K8S

https://www.katacoda.com/courses/kubernetes/playground

https://kubernetes.io/docs/tutorials/kubernetes-basics/

In the next blog, I will mention sample yaml file all the resources