Linux commands and shell scripts to play with files

Example file lists

88968-4568-8765_EMPLOYEE_LOADER_02.00.0000_ahiutrfjg.zip
67808-6878-6905_EMAIL_QUEUE_2_01.00.0000_bhgkutgk.zip
90978-5368-7165_GET_FATTUR_TEST_MARK_EXCLAM_01.00.0000_zsdutrfjg.zip
.........
...

1. To rename all the file extensions

# Rename all *.zip to *.iar
for f in *.zip; do
mv -- "$f" "${f%.zip}.iar"
done

# Remove all the unnecessary texts before the IAR file name eg: here the file name is 

# EMPLOYEE_LOADER_02.00.0000

ls | grep '\.iar' | sed 's/^\([^_]*\)_\(.*\)$/mv & \2/' | sh

# Remove unwanted after the file name

ls | grep '\.iar' | sed 's/^\(.*\)_\(.*\)$/mv & \1.iar/' | sh

2. To know the number of files in starting with each alphabet

# Count the number of files and prints it counts

for x in {A..Z}

do

        echo "$x"

        ls $1/${x}*$2 -l | wc -l

done

Need to call the countFiles.sh like below

./countfiles.sh ../json/final_till_m json

3. Others Useful commands

3.1 Count the number of files

3.1.1 Total number of file inside a folder and sub-folders

$ find . -type f | wc -l
950

3.1.2 Total number of zip files

$ find . -iname \*.zip | wc -l

17454

3.2 List the files matching the content

$ find  -name "*.xml" | xargs grep "Start Staging" 2> /dev/null

Upload/Download a file using SFTP with a private key

If we want to copy/transfer the file from Windows to Linux then we can use the popular WinScp software. But if we want to transfer files between Linux machines then we can either use FTP (File Transfer Protocol) or SFTP (SSH File Transfer Protocol). This article tells the steps of transferring files using the SFTP protocol with a private key

STEPS

• Keep the private key of the remote host in a folder in local Linux machine

eg: /home/venkatesh/openssh.pk

• Change your directory

cd /home/venkatesh/

• We can use SFTP(SSH File Transfer Protocol) to connect to the remote Linux server and download/upload the files

Usage of sftp command

sftp [-1Cv] [-B buffer_size] [-b batchfile] [-F ssh_config]
            [-o ssh_option] [-P sftp_server_path] [-R num_requests]
            [-S program] [-s subsystem | sftp_server] host
sftp [user@]host[:file ...]
sftp [user@]host[:dir[/]]
sftp -b batchfile [user@]host 

• Execute the below command to connect to the remote server

-sh$ sftp -o "IdentityFile=openssh.pk" user@remote.domain.com

We need to pass the private key to SSH command, so whatever we need to give to SSH, we need to include it as a part of -o command line argument.

          -sh$  sftp -oPort=10124 -oPreferredAuthentications=password admin@190.35.180.156

• Once we execute the above command, we will be connected and will see the sftp prompt like below

sftp>

• For downloading the file to the local Linux machine, 

sftp> get testFile.txt

This would download the file to your directory from where we have connected to the sftp.

• Similarly, if we want to upload a file, we need to execute

sftp> put uploadMyFile.txt