Interesting Material on Authentication

As the ideology is going towards a microservices and making each service stateless; even the session based authentication is fading away and the token-based authentication is getting more popularity.

With token-based authentication, we never have to store the session details of each user in the server that makes the scalability of servers so beautiful.

More details in below link

https://scotch.io/tutorials/authenticate-a-node-js-api-with-json-web-tokens

https://dzone.com/articles/cookies-vs-tokens-the-definitive-guide

Below link talks about the hashing of passwords and how adding a salt to the password before hashing makes hacking difficult

https://www.wordfence.com/learn/how-passwords-work-and-cracking-passwords/

Virtual Private Cloud (VPC) Notes

Below are the notes I captured while watching the below very good video on VPC

https://www.youtube.com/watch?v=LX5lHYGFcnA

Router and Route table:

1. Route table holds the info on how to route
2. Router routes based on the route table

Elastic IP

1. Static public IP
2. Each time we stop and start the EC2 instance, we get dynamic IP
3. Alllows to use the same IP 

Elastic Network Interface

1. The IP is not directly assigned to EC2
2. It is the ENI that is attached to EC2
3. We can have more than one network interface for an EC2
4. Which means we can have more than on1 IP assigned to EC2

Internet Gateway

1. It is like a door to VPC for both inbound and outbound  

Customer Gateway, VPN Connection, and Virtual Private Gateway

1. These 3 together helps in making a connection between on-premise and aws

VPC peering

1. Communication between 2 VPCs
2. One should send a request and the other should accept it

VPC endpoint

1. For eg: S3 endpoints are public endpoints
2. If we want to our EC2 to talk to S3 privately without going via internet gateway then we go for VPC endpoint
3. We have a VPC interface endpoint too, for talking to other applications within a subnet

NAT Gateway

1. Network Address Translation
2. This is for allowing the private subnet to talk to the internet gateway for accessing the internet
3. The NAT gateway will not allow any inbound request to reach the private subnet, it will just allow the private subnet to access the internet
4. Works only for IPV4, for IPV6, we need to use Egress

IP Address and Subnets

• IPV4 is 32 bit
• IPV6 is 128 bit
• IP CIDR Range
• Eg: 10.0.0.0/16 means the first 16 bits of the IP are not going to change
• Eg: 10.0.0.0/8 means the first 8 bits of the IP are not going to change
• The CIDR Range is 10.0.0.0 to 10.255.255.255
• Eg: 10.0.1.15/32 means only one IP
• It is not necessary to be the multiple of 8; Eg: 10.0.0.0/26
• Private IP address range as per RFC1918 standard
• 10.0.0.0 - 10.255.255.255 (10/8)
• 172.16.0.0 - 172.31.255.255 (172.16/12)
• 192.168.0.0 - 192.168.255.255 (192.168/16)

Routing

• Is based on the Route Table definition
• It has information on where the request needs to be routed  
• Need to associate a route table for a particular subnet
• Route table defines whether the subnet is a private subnet or a public subnet
• There will be a default route table created in a VPC and that will allow all the local/private access within the VPC

Security Groups

• Default of all SG is allow all outbound, deny all inbound
• Need to edit the inbound and outbound connections
• Applies at instance or individual resource level like EC2, RDS etc
• This is the first level of defense

Network ACL (Access Control List)

• Specify what IPs and Port are allowed inbound and what are for outbound
• Security Group only have allow rules, only the Network ACL we have both allow and deny rules
• Applied at the network level

Flow

Create VPC > Create Internet Gateway and attach to VPC > Create Subnets > Create RouteTables and attach to VPC > Subnets association > Configure Security Groups

Setting the Proxy details in different tools

Many times when we are switching our work between the home network and office network, we may face this proxy issues if the office network is under proxy. 

And most of the time, just setting the system level proxy won't be enough, we need to set the proxy at each application/tools level. In this blog, I wanted to document setting up of proxies setting for few of the development tools

1. Maven

For maven, the proxy details need to be mentioned in the setting.xml file. There may be two copies of this settings.xml file one at the global level and one per user account. If we prefer to affect only the current user, then we need to edit the user's copy of setting.xml. The maven is a little intelligent and it will automatically detect and adds the proxy details in the setting.xml but the problem is I haven't seen them removing the proxy details when we are the home network, we need to manually remove it.

In Ubuntu, the setting.xml can be found from below path

/opt/etc/maven/conf/settings.xml

https://maven.apache.org/settings.html

2. Docker

The interesting thing with Docker is, we need to mention the proxy setting in two places. The first one is for the 'docker' command to work in the host environment. Another one is for passing the proxy details to the guest containers.

Proxy for the host:

• Create the folder

$ sudo mkdir -p /etc/systemd/system/docker.service.d

• Create the file

/etc/systemd/system/docker.service.d/http-proxy.conf

• Add the following entry in the file

[Service]Environment="HTTP_PROXY=http://proxy.example.com:80/"

https://docs.docker.com/config/daemon/systemd/#httphttps-proxy

Proxy for the guest containers:

One option for this is to set the environment variables in the Dockerfile. But the recommended approach is to set via the config.json

https://docs.docker.com/network/proxy/

3. NPM

The node package manager does not take the system proxy settings. We need to set it via its npm config command like below

npm config set proxy http://proxy.company.com:8080

npm config set https-proxy http://proxy.company.com:8080

Inorder to unset the proxy details, execute below command

npm config rm proxy 

npm config rm https-proxy

Did you notice, even for https, it expects to provide only the HTTP proxy detail. 

https://jjasonclark.com/how-to-setup-node-behind-web-proxy/

4. Linux Terminal

It use to fetch the system proxy setting automatically; but for some reasons, sometimes it may not fetch from the system proxy settings. In such cases, we can simply export the environment variables and that should work. We should note that once we close the terminal these values will be lost

export http_proxy=http://proxy.company.com:8080 

export https_proxy=https://proxy.company.com:8080

Setting up a isolated development environment

Very often whenever we wanted to do a POC or a new project which depends on many different tools. frameworks or software, we prefer to start it with an isolated development environment without affecting our other tasks. The good option for that is to use a virtual OS running on top of our host OS, below steps helps to get started with setting up such an environment 

Steps to set up a working environment in a Virtual box

1. Install Oracle VM Virtual Box in host machine
https://www.virtualbox.org/wiki/Downloads

2. Download the preferred VDI OS image from the below link (eg: Ubuntu 16.04)
https://www.osboxes.org/virtualbox-images/

3. Create a new Virtual Machine in Virtual Box from the downloaded VDI file

4. Setting the bridged network
https://www.youtube.com/watch?v=cDF4X7RmV4Q&t=865s

5. Installing the java and maven
https://www.vultr.com/docs/how-to-install-apache-maven-on-ubuntu-16-04

6. Installing Eclipse
https://www.youtube.com/watch?time_continue=391&v=EiGl2Ag35AI

7. Installing MySQL
https://www.digitalocean.com/community/tutorials/how-to-install-mysql-on-ubuntu-16-04

8. Installing Docker
https://www.vultr.com/docs/installing-docker-ce-on-ubuntu-16-04

9. Installing docker-compose
https://docs.docker.com/compose/install/

10. Installing AWS CLI from the bundle
https://docs.aws.amazon.com/cli/latest/userguide/awscli-install-bundle.html

11. Installing Php 5.6 or 7.0
https://tecadmin.net/install-php5-on-ubuntu/

Linux commands and shell scripts to play with files

Example file lists

88968-4568-8765_EMPLOYEE_LOADER_02.00.0000_ahiutrfjg.zip
67808-6878-6905_EMAIL_QUEUE_2_01.00.0000_bhgkutgk.zip
90978-5368-7165_GET_FATTUR_TEST_MARK_EXCLAM_01.00.0000_zsdutrfjg.zip
.........
...

1. To rename all the file extensions

# Rename all *.zip to *.iar
for f in *.zip; do
mv -- "$f" "${f%.zip}.iar"
done

# Remove all the unnecessary texts before the IAR file name eg: here the file name is 

# EMPLOYEE_LOADER_02.00.0000

ls | grep '\.iar' | sed 's/^\([^_]*\)_\(.*\)$/mv & \2/' | sh

# Remove unwanted after the file name

ls | grep '\.iar' | sed 's/^\(.*\)_\(.*\)$/mv & \1.iar/' | sh

2. To know the number of files in starting with each alphabet

# Count the number of files and prints it counts

for x in {A..Z}

do

        echo "$x"

        ls $1/${x}*$2 -l | wc -l

done

Need to call the countFiles.sh like below

./countfiles.sh ../json/final_till_m json

3. Others Useful commands

3.1 Count the number of files

3.1.1 Total number of file inside a folder and sub-folders

$ find . -type f | wc -l
950

3.1.2 Total number of zip files

$ find . -iname \*.zip | wc -l

17454

3.2 List the files matching the content

$ find  -name "*.xml" | xargs grep "Start Staging" 2> /dev/null

Upload/Download a file using SFTP with a private key

If we want to copy/transfer the file from Windows to Linux then we can use the popular WinScp software. But if we want to transfer files between Linux machines then we can either use FTP (File Transfer Protocol) or SFTP (SSH File Transfer Protocol). This article tells the steps of transferring files using the SFTP protocol with a private key

STEPS

• Keep the private key of the remote host in a folder in local Linux machine

eg: /home/venkatesh/openssh.pk

• Change your directory

cd /home/venkatesh/

• We can use SFTP(SSH File Transfer Protocol) to connect to the remote Linux server and download/upload the files

Usage of sftp command

sftp [-1Cv] [-B buffer_size] [-b batchfile] [-F ssh_config]
            [-o ssh_option] [-P sftp_server_path] [-R num_requests]
            [-S program] [-s subsystem | sftp_server] host
sftp [user@]host[:file ...]
sftp [user@]host[:dir[/]]
sftp -b batchfile [user@]host 

• Execute the below command to connect to the remote server

-sh$ sftp -o "IdentityFile=openssh.pk" user@remote.domain.com

We need to pass the private key to SSH command, so whatever we need to give to SSH, we need to include it as a part of -o command line argument.

          -sh$  sftp -oPort=10124 -oPreferredAuthentications=password admin@190.35.180.156

• Once we execute the above command, we will be connected and will see the sftp prompt like below

sftp>

• For downloading the file to the local Linux machine, 

sftp> get testFile.txt

This would download the file to your directory from where we have connected to the sftp.

• Similarly, if we want to upload a file, we need to execute

sftp> put uploadMyFile.txt

Different tools to monitor memory utilization in Linux

There are plenty of tools when it comes to monitoring memory usage. The main purpose of using tools like these is to identify any leaks and to know any performance issue that can occur to our program.

Linux natively has tools like 'top', 'pmap' etc. Apart from that Java provides a tool called 'jconsole', I love this tool than any other tool.

1 top

This command is very much similar to what we see in Windows Task Manager. This one shows the real-time memory usage and CPU utilization by each process. We can also monitor for a particular process by passing 'pid' as a command argument. 

$top

Use this command if you would like to monitor all the process

$top -p 29257

Use this one if you would like to monitor for a particular process

$top -p 29257 -b >  /home/venkatesh/memory_footprint.txt

If you would like to record the memory utilization over time, then execute the top command with '-b' option which will output the status ever 'n' seconds

As you can see from the output, it will just tell the percentage of utilization and it will not break the process any further.

2 pmap

pmap will give the snapshot of the process's memory map. It even provides addresses of the each jar that this process has. What I understand is, it is showing only the static memory usage by the processor and does not tells how much it is taking at runtime. 

$pmap -x 29257

We can also get the memory map for multiple processes as well.

3 jconsole

This can be used to monitor only java processes. It gives a clear breakdown of HeapMemory usage, Number of threads, classes, CPU usage, when GC kicked in. All these are real-time information that it shows

$jconsole 29257

4 Oracle Java Mission Control

Zip file is invalid for file zipped using ZipOutputStream

Are you seeing an error like below when you try to unzip a file which was zipped using the java.util.zip.ZipOutputStream

Or are you seeing an error like below when you unzip from Linux


unzip myfile.zip 
Archive:  myfile.zip 
  End-of-central-directory signature not found.  Either this file is not 
  a zipfile, or it constitutes one disk of a multi-part archive.  In the 
  latter case the central directory and zipfile comment will be found on 
  the last disk(s) of this archive. 
unzip:  cannot find zipfile directory in one of 
myfile.zip or 
        myfile.zip.zip, and cannot find 
myfile.ZIP, period.

If the answer is YES, then check how you are closing the ZipOutputStream in your code. The ZipOutputStream should be closed before calling the toByteArray() of your ByteArrayOutputStream

ByteArrayOutputStream baos = new ByteArrayOutputStream();
try(ZipOutputStream zos = new ZipOutputStream(baos);) {
zip(zos, incidentDir, incidentDir);
zos.flush();
baos.flush();
}
catch (Exception e) {                
throw new Exception("Unable to Zip the folder");
}
byte[] result = baos.toByteArray();

The documentation of ByteArrayOutputStream says

The methods in this class can be called after the stream has been closed without generating an IOException.

https://docs.oracle.com/javase/8/docs/api/java/io/ByteArrayOutputStream.html