A trip to Tucson

 It was during the holiday season in December 2022 and what could be more fun than meeting the old college friends. Planned a humble trip to Tucson after the birthday parties in Phoenix, Tucson is a beautiful city with a lively downtown lined with lots of eateries and restaurants. For dinner stepped into a Mexican restaurant that had a cozy environment with live music. Since it was the holiday season, the roads are lit up with colorful and vibrant lights everywhere, of course, a good spot for selfie clicks.

Mexican restaurant in downtown

Selfie Time during Christmas decorations

Since it is a one-day trip to Tucson, we planned to keep it simple without stressing too much and added Colossal cave and Pima Air & space Museum. We planned for these two places one after breakfast and one after lunch. The combination of these two places is very interesting, in the Colossal Cave we can get mesmerized by nature on its natural formations and in the Pima Air Museum, we can witness the technological heights human has reached over time.

In the Colossal Cave, we can even enjoy playing music on the rocks, we can tap the rock with small wooden blocks and can hear beautiful music.

Lantern light tour in Colossal Cave

In a few aircraft even we can get inside a take-a-peek view of it, the museum also houses a few video games which all age groups can enjoy, a souvenir shop, and more. They have guided tours as well in tram vehicles every 30 mins or 1 hour which would be more informative as the guide will be explaining the history of each of the aircraft.

Inside the Fighter aircraft

Guided tour vehicle

If looking for a nice Indian restaurant nearby then definitely Sher-e-Punjab is a place to try. It has a wide variety of options and has a very good Indian flavor of spices and masala

javax.xml.bind::jaxb-api not working in JDK17

 When you are trying to update the JDK version 8 or 11 to JDK 17 and if you are using javax.xml.bind::jaxb-api then you will see the below error during the startup of the application

Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make protected final java.lang.Class java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int) throws java.lang.ClassFormatError accessible: module java.base does not \"opens java.lang\" to unnamed module @5e922278\n\tat java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)\n\tat 

the root cause is that the javax.xml.bind::jaxb-api is trying to use internal java API using reflection and the support is stopped from JDK16 due to the introduction of the "Strongly encapsulate JDK internals by default" feature.

Recommendation: The recommendation is to move away/upgrade the libraries that use reflection to load java internal classes

And the correct alternate for javax.xml.bind::jaxb-api is jakarta.xml.bind::jakarta.xml.bind-api

Or another way to resolve this is to add this java startup arguments option --add-opens=java.base/java.lang=ALL-UNNAMED

Interesting study material: https://blogs.oracle.com/javamagazine/post/a-peek-into-java-17-continuing-the-drive-to-encapsulate-the-java-runtime-internals

mvn flatten:flatten

The flatten plugin from maven helps to remove any development-specific element, build specific element or environment-specific elements in the pom file and prepares a pom file that can be published and can be consumed by other

https://www.mojohaus.org/flatten-maven-plugin/

Command to execute it

mvn flatten:clean flatten:flatten

After the execution, we can find the flattened pom file in the same location where original pom file was and with the name .flattened-pom.xml

First three months in Denver

Landed in the United States on the 03rd of September 2022 and in 3 months, there are a lot of cultural learning, getting used to the people, finding new friends, and much more. And in this post, am sharing some of my experiences and they might help other newcomers like me.

Leaving from India

Some of the pre-plan that we should have before arriving are where we stay, how to carry money from India for expenses, and being connected with the family. If looking for a budget hotel with an option for self-cooking then Extended Stay America, is an excellent choice. It provides an excellent deal of up to 67$ per night if the booking is done for more than 7 nights. Next is having a safe and flexible way to carry US dollars and a good option is to buy a Forex card in India with an easy option to reload it online. The last and most important preparation is always being connected with the family and we can get the US prepaid SIM when we are in India itself at a very affordable plan that includes data as well; actually, I have seen the plan cost is more competitive than the US mobile network providers.

Arriving in the US really means, the immigration officer in the port of entry should be satisfied with all the documents we produce and then issue an electronic i94 document. The i94 document will contain important information like when we arrived and how long we are authorized to stay etc and it can be retrieved from https://i94.cbp.dhs.gov/I94/#/recent-search. In some rare cases, the details in i94 may be entered wrongly and it can be corrected later by filing a ticket at https://help.cbp.gov/s/questions?language=en_US. After this, you are officially in the US and welcome. 

Denver Tech Centre

Beautiful landscape on the side of the road

Public park and waterfronts

After a good night sleep after a long hauling journey, be prepared for the next set of tasks that need to be finished sooner, like vising the SSN office, taking an appointment with a bank, and opening a bank account. One thing that I noticed is everything goes by appointment only and there is no drop in service (including the hair salon! which I will give details about below), so where ever we go, it is advised to take an appointment over the phone and then visit.

When in India, look at the city map and get used to the county and area names. This will help in starting conversations with new people in the city and also will help in shortlisting the location for a long stay either in an apartment or an individual home. Some of the shortlisting criteria are what is the ratings to the school in the county and can be found by looking at https://www.greatschools.org/ other criteria would be the neighborhood like whether it is accessible by the light rail, what supermarkets are nearby, whether it has hospitals closeby etc.  Some of the websites that can help in finding the right home is https://www.apartments.com/ and https://www.zillow.com/. Again, it is advisable to take an appointment with the leasing office for an apartment tour and then visit. One good thing with housing in the US is that as soon as you are moved in, at the same moment you can start cooking the food because by default it comes up with a stove including the electric or a gas connection, and refrigerator.

Moving into the apartment

Coming to the commutations in Denver is a little average if depending only on public transport, however uber and Lyft will be available round the clock. The public transport is managed by RTD and it operated both light rail and the bus, actually if we are well prepared and planned then public transport works in ease and very very cost-effective. During summer, bicycle ride would make the ride pleasant and also saves fossil fuel.

Lots of Indian restaurants and eateries are available and can be easily ordered and get it delivered to our door via https://grubhub.com, https://www.doordash.com/, https://www.ubereats.com/. If wanted to try something outside of Indian cuisine, then Mexican and Mediterranean would be a good try and some of the well-known food chains are https://www.chipotle.com/, https://eatfelfel.com/. 

After moving into an apartment or home, then the next thing would be to fill up the home, and obviously  https://amazon.com/, https://www.walmart.com/ etc are starters to get a fair idea and price ranges. But you can go out to showrooms and get the real feel of it like visiting the mattress showroom and trying it out, visiting a furniture shop etc. Also, there are malls where we can peek into multiple different showrooms and get the experience https://www.parkmeadows.com/. In the US, there is no concept of MRP, so the same product might have different prices in a different showroom, so we have to be careful and compare and buy the right deal one.  If you can wait and the time for the BlackFriday sale is not too far, then it is really worth a wait than buying immediately. As heard all the products segment will go on an offer sale and definitely will get the best price of the year. One big difference you would see with India is that all products need to be self-assembled by ourselves if want to save money, the human labor is very expensive and it might cost more than the product cost itself sometimes!!

DIY projects

Assembling furniture

In my first three months, I witnessed three different seasons of the year like the summer, the fall, and the winter. When I came it was so sunny, and the days are lengthy gradually the fall came and enjoyed the view of what it really means a fall color by looking at the different colors of the leaf over the period of 2 weeks. And then the winter started in November with the year's first snow.

Fall Color

Road, Car covered with snow

Definitely, after a few weeks, we need to go to a hair salon and one option is https://www.greatclips.com/ which provides services to men, women, and kids. As mentioned earlier, take an appointment and then visit. Sometimes during the Black Friday sale, they sell coupons at a discounted price which can be used during the rest of the year. 

If you are an avid reader and a book lover then you are covered, each county has a library with a very good collection of books for all ages, the subscription is free and we can borrow books and return them. Also, it houses beautiful reading rooms with great ambiance etc. More details about the library where I have a subscription https://www.dcl.org/

K8S Service Account verification using REST API

Service Account in K8S is used to invoke all the k8s admin server APIs within the POD provided the service account is granted with necessary RBAC permissions via role-binding or cluster-role-binding

The recommended way to invoke the K8S API from the POD is to use the official client libraries. However if we want to make a quick testing of service account configuration then we can use the K8S REST APIs to invoke and check whether the POD is having sufficient privilage to access the resources or not.

Before going to steps of invoking the REST API, few lines about how the POD gets that privilage. When a deployment/pod is binded with a service account, the POD will get the certficate, token and namespace in the location /var/run/secrets/kubernetes.io/serviceaccount

$ kubectl exec backend-api-b874f697f-fgqk7 -c backend-api -n vemohanr -- cat /var/run/secrets/kubernetes.io/serviceaccount/token

On executing the above command we would get the token I we can use to invoke the REST API, and now we can use the above token and form the curl command

$ kubectl exec backend-api-b874f697f-fgqk7 -c backend-api -n vemohanr -- curl -k -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6Ikd6VFI0R3g2RW9tckNwbnhMOE5oWDlLc2ZEMVRUZU1qVExXbktkVlh2YW8ifQ.eyJhdWQiOlsiYXBpIl0sImV4cCI6MTY5NDU2MjE3MiwiaWF0IjoxNjYzMDI2MTcyLCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2NhbCIsImt1YmVybmV0ZXMuaW8iOnsibmFtZXNwYWNlIjoidmVtb2hhbnIiLCJwb2QiOnsibmFtZSI6ImljLWFwaS1pbnRnLWR0LWI4NzRmNjk3Zi1mZ3FrNyIsInVpZCI6ImI4OWE4MGJlLWYwZGUtNGFlZi1iODBiLWRlYWQ3ZDc2YzVhMSJ9LCJzZXJ2aWNlYWNjb3VudCI6eyJuYW1lIjoiYXBwcy1zYSIsInVpZCI6ImFmZjRiNGU5LTY4ZTEtNDY0ZS1iZGJhLTQxMTYzOTIxMWM1NyJ9LCJ3YXJuYWZ0ZXIiOjE2NjMwMjk3Nzl9LCJuYmYiOjE2NjMwMjYxNzIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDp2ZW1vaGFucjphcHBzLXNhIn0.Lp_FOzcCz19KLwTm4qU_VHOjqpM9M6wSpPAfGWZQQyjFz276xLBEYU22dwaFcuHkOXw83S1xy0rWAAhgvkaWFmpLwuC9GxlMh583XJo1b1GC-BIei_EgzdTrD3TFOtQ9CCTC4Jf0FWmmY5Uz5ng5xglLbw7220YsRIG9NIj1PkfBfVJCVrezE-wXyNb4jkr86wlNz3uKhYw8FdIffUuOyXpNfTt1IyOkGnQGtow_E3F5asqO7ZlaB6DjUJBZhwgP90SqAmpVyu10hTELNLchV-NeTtyQJEHIbLqZj64wJb3SrBqenHft_g_2SRivoMzDoDQUFtk_N3HHNdpHDfEC7A" https://147.154.106.173:6443/apis/apps/v1/namespaces/vemohanr/cronjobs

If want to know what is the K8S admin API server IP etc, execute the below command to get it

$ kubectl cluster-info
Kubernetes master is running at https://147.154.106.173:6443
CoreDNS is running at https://147.154.106.173:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
kubedashboard-kubernetes-dashboard is running at https://147.154.106.173:6443/api/v1/namespaces/kube-system/services/https:kubedashboard-kubernetes-dashboard:https/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

References

https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.25/#-strong-api-overview-strong-

https://www.ibm.com/docs/en/cloud-paks/cp-management/2.0.0?topic=kubectl-using-service-account-tokens-connect-api-server

Accessing K8S API from POD

XMing - Alternative to VNC server

Ever wondered how to open the UI/GUI of the applications installed in a remote Linux server in our local development environment. Usecases like codebase is in remote server and we SSH into it to build the code etc and instead of using VNC to get the graphical interface, we can use this technique to open the applications installed in the remote server

Steps

1. Install XMing server https://sourceforge.net/projects/xming/ in the local machine

2. Enable X11 forwarding in putty or other SSH tool

3. SSH into the remote server

4. Launch the application and we should see the application opening in local machine

CKA Kubernetes ( K8S ) Networking

 The preparation for the CKA (Certified Kubernetes Administrator) requires lots of practice and practice. Fortunately, we have lots of online playgrounds to keep practicing, there are lots of free courseware available and lots of paid as well are available. In addition to that, we get two attempts to clear the exam

• There is training conducted by the https://training.linuxfoundation.org/ itself who conducts the CKA exams. 
• But I felt this course from Udemy is good https://www.udemy.com/course/certified-kubernetes-administrator-with-practice-tests/, it has videos that explain each concept and have lots of practical sessions and mock exams.
• The https://killer.sh/ is another tool where you can practice and the questions asked in the killer.sh are complex and tests real skill.  If we subscribe to the CKA exam, we will get two free sessions to solve the problems in the killer.sh
• https://www.katacoda.com/courses/kubernetes this is very useful if you are a beginner in kubernetes. It starts with a very simple usecase/scenario and we can solve in our own pace and try any number of times
• https://killercoda.com/ is another useful online tool that we can use like a playground to solve scenario-based problems

Switching and Routing

• Switching enables communication between hosts within the network

• Commands to enable

○ ip link

○ ip addr add 192.168.1.10/24 dev eth0

• Router connected two networks together

○ Command to know the routing table

§ route

○ Command to add a routing

§ ip route add 192.168.2.0/24 via 192.168.1.1

§ ip route add default 192.168.1.1

• For forwarding traffic from one eth0 to another eth1

○ cat /proc/sys/net/ipv4/ip_forward

○ modify the file /etc/sysctl.conf

DNS

• /etc/hosts

• Each node can have its own domain-name mapping in /etc/hosts, however soon it will become cumbersome thats why we use DNS host

• The location of DNS host is defined in /etc/resolv.conf

• If the are the two definition of host name available in local /etc/hosts as well as in DNS then the selection will be based /etc/ns-switch.conf

• Within our nameserver, we can point to public nameserver ex: 8.8.8.8 nameserver which is hosted by Google

• There is 'search' property we can set in /etc/resolv.conf so the nameresolver intelligently appends the search text with the user provided name

• coreDNS is one implementation of DNS Server which is opensource

Network Namespaces

• Namespaces are used by containers like docker to create a network isolation

• Command to create namespaces

○ ip netns add green

• To list down all the interfaces in the host

○ ip link

• To list down all the interfaces visible inside the network namspace

○ ip netns exec red ip link

Docker networking

• There are different networking type when we run a docker container

○ none - the container cannot reach the container and no one can reach the container

○ host - uses the host's IP as its own ip

○ bridge - an internal private network is created

• Command

○ docker run --network <type> nginx

○ docker network ls

§ lists down the network

○ docker inspect <container_id>

§ Under the NetworkSettings we can see what is the network namesapce the container is using

Kubernetes networking

• The CNI (Container Network Interface) is a plugin used by k8s to establish a network connectivity 

• As per K8S reqt, CNI should assign each pod with a IP address and each of them should be accessible by other POD and it should reach all other PODs

• There are many flavors of CNI plugin

○ bridge

○ flannel

○ weave-net 

○ ipvlan

○ ...

• kubelet will point to the CNI configuration file while bringing up the kubelet in the pod

• 'ps -aux | grep kubelet' will show the path to the configuration file

• ipam - ip Address Management

○ It is the plugin implementer responsibility to manage the IP range, avoid duplicate assignment of IP to the pod etc

○ Two types

§ dhcp

§ host-local 

Service Networking

• In general we use service for accessing the pod instead of accessing the pod directly

• When a service is created it is accessible cluster wide by default

○ ClusterIP - accessible within the cluster

○ NodePort - Will be accessible by the nodeIPs with the port 

• kube-proxy watches the changes in kube-apiserver for any new service creation and it will take action

• Service is a cluster wide concept and really there is no service or process running that listens to the IP. It is just a virtual object

• kube-proxy creates a forward routing rules and gets the IP within the range what is configured 

• Three ways of configuring the forward routing rules (--proxy-mode param needs to be set while bringing up the kube-proxy)

○ userspace

○ iptables

○ ipvs

• The service ip range are set while bringing up the kube-api-server using the --service-cluster-ip-range parameter

• We can check the iptables in the node

○ iptables -L -t net | grep db-service

DNS in Kubernetes

• DNS runs as a service and pod in k8x under kune-system namespace

• The DNS nameserver ip will be configure to pod's /etc/resolve.conf by the kubelet when the pod is started

• Each wont get a hostname by default but it can be configured in the CoreDNS configuration in the kube-dns pod

Ingress

• Ingress takes care of

○ Loadbalancing

○ Authentication SSL

○ URL based routing configuration

• It acts like a layer 7 load balancer

• Ingress Controller - There are many implementation and by default we will not have one running in the k8s

○ Nginx

○ HA Proxy

○ Contour

○ traefik

apiVersion: apps/v1

kind: Deployment

metadata:

  name: ingress-controller

  namespace: ingress-space

spec:

  replicas: 1

  selector:

    matchLabels:

      name: nginx-ingress

  template:

    metadata:

      labels:

        name: nginx-ingress

    spec:

      serviceAccountName: ingress-serviceaccount

      containers:

        - name: nginx-ingress-controller

          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0

          args:

            - /nginx-ingress-controller

            - --configmap=$(POD_NAMESPACE)/nginx-configuration

            - --default-backend-service=app-space/default-http-backend

          env:

            - name: POD_NAME

              valueFrom:

                fieldRef:

                  fieldPath: metadata.name

            - name: POD_NAMESPACE

              valueFrom:

                fieldRef:

                  fieldPath: metadata.namespace

          ports:

            - name: http

              containerPort: 80

            - name: https

              containerPort: 443

• Ingress Resource - is the set of configuration passed on to the ingress controller to route the traffic appropriately

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

  annotations:

    nginx.ingress.kubernetes.io/rewrite-target: /$2

    nginx.ingress.kubernetes.io/ssl-redirect: "false"

  name: ingress-pay

  namespace: critical-space

spec:

  rules:

  - http:

      paths:

      - backend:

          serviceName: pay-service

          servicePort: 8282

        path: /pay(/|$)(.*)

        pathType: ImplementationSpecific