Dealing with “No space left on device” when partition is not full

When you get the “No space left on device” error on linux, first thing you do it to check your disk space usage.  Often times you notice you still have space, so what could be wrong?

Although your partition is not nearly full, most likely you have too many small or zero-sized files on your disk. So while you have enough disk space, all your available inodes have been exhausted. 

To check this run:

df -ih

If IUse percentage is at or near 100%, then huge number of small files is the reason for “No space left on device” errors.

To find out where inodes are being used run:

echo "Detailed Inode usage for: $(pwd)" ; for d in `find -maxdepth 1 -type d |cut -d\/ -f2 |grep -xv . |sort`; do c=$(find $d |wc -l) ; printf "$c\t\t- $d\n" ; done ; printf "Total: \t\t$(find $(pwd) | wc -l)\n"

After finding the directory with largest number of files, delete any unwanted files to free up some inodes. 

NOTE : If a process is using the files you are deleting, you will need a reboot to free the inodes used.

Useful shell commands

List only the dot files, and tell ls not to see through directories:

• ls -d .*

How to run a command on a filtered file list returned by grep

I often find myself passing a file list returned by some linux command to grep for filtering like below :

ls *.jar | grep test 

Sometimes i like to run a certain command on these filtered results. I achieve this by :

ls *.jar | grep test | while read line; do sha1sum "$line"; done

The generic form is :

<command generating a file list>   |   grep <filter keyword> | while read line; do <command> "$line"; done  

How to check if a port is open on Linux

Sometimes when writing a script/program you need to know if a port your are interested in has an established connection to it or it's free.

My favorite way to do this is to run the following command ( replace PORT with the port number you want to check for ):

netstat -ln | grep :PORT

-l = Show only listening ports

-n = Show port number

 If this command doesn't return anything, it shows that your PORT is available to use.
In your script you can check $? to see the return code of the above command. If it's non-zero, the PORT is available.

One thing to note though is that, this doesn't tell you the status of your network e.g. if PORT is accessible from outside of your notwork.

How to fix Cannot open your terminal '/dev/pts/#' - please check.

simply run :

script /dev/null

and try again.

Best way to transfer large file or resume dropped ftp/scp file transfers

When transferring large files your first choice should be rsync, since it gives you the ability to resume unfinished/dropped transfers. However, if you started a large file transfer with ftp/sftp or scp and the connection was lost during the transfer, you don't need to start over again. Simply run rsync the following way and it will resume your transfer :

rsync --partial --progress --rsh=ssh <USER>@<HOST>:<REMOTE_FILE_PATH> <LOCAL_FILE_PATH> 

This command is what should be used for large file transfers.

How to setup SSH private and public keys

• 1)
  % ssh-keygen -t rsa -b 4096 Generating public/private rsa key pair.
  Enter file in which to save the key (~/.ssh/id_dsa): (just press Enter) 
  Enter passphrase (empty for no passphrase): (enter a passphrase and then press Enter or if you don't want want one just press Enter) 
  Enter same passphrase again: (repeat last action) 
  Your identification has been saved in ~/.ssh/id_rsa
  Your public key has been saved in ~/.ssh/id_rsa.pub
  The key fingerprint is:
  A long string appears here
  %
• 2)
  Paste the content of the  ~/.ssh/id_dsa.pub file that just got generated on your local host into the file ~/.ssh/authorized_keys on the remote host and save.
• 3)
  Set proper permissions on your local host keys and .ssh directory:
  chmod 700 ~/.ssh
  chmod 644 ~/.ssh/id_rsa.pub
  chmod 600 ~/.ssh/id_rsa
  set proper permissions on your remote host authorized_keys and .ssh directory
  chmod 700 ~/.ssh
  chmod 644 ~/.ssh/authorized_keys
  

How to find the 10 largest file or directories in Linux

This finds the largest 10 files :

find . -type f -print0 | xargs -0 du -s | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}

This finds the largest 10 directories:

find . -type d -print0 | xargs -0 du -s | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}

You can easily change -10 to -n where n is the number of files/directories you are trying to find.

Killing all processes containing a specific pattern in their process name

Say you want to kill all processes spawned from your Oracle Home. In Linux/Unix there is a nice command that lets you achieve this very easily.

If your Oracle Home is /u01/app/oracleHome then any oracle related process will have oracleHome in it's name. To kill all such processes at once issue the command :

pkill -9 -f oracleHome

How to extract values from a java properties files in a bash/shell script

Let's say you have a java properties (sample.prop ) file that contains this :

ORACLE_HOST=myhost.us.oracle.com
MW_HOME=/scratch/codrguy/sandbox
DB_HOST=dbhost.us.oracle.com

Now, if we want to extract values from this properties file and use it in a shell/bash script for let's say an automation task, how do we go about that.

There are 3 parts to this task: 1) Getting the line that contains the value we want 2) extract the value from that line 3) store it in a variable to be used later in the script

To do part 1 we can simply grep for the value of the property we need :
For example :

grep ORACLE_HOST sample.prop
this returns
ORACLE_HOST=myhost.us.oracle.com

To do part 2 we pipe the output of part 1 into awk for extraction of the value we need

grep ORACLE_HOST sample.prop |  awk -F= '{print $2}'
this returns
myhost.us.oracle.com

the -F option of awk sets the delimiter to be '=' and then we print the second record which is what we need

Finally to store this value in a variable we do :

my_value=$(grep ORACLE_HOST sample.prop |  awk -F= '{print $2}')

Removing passphrase from a SSH key

If you have a SSH key and are tired of typing it over and over, there is an easy way to remove it.

Open a *nix terminal and type :

bash-3.2$ ssh-keygen -p

Enter file in which the key is (/home/myuser/.ssh/id_rsa):
Enter old passphrase:
Key has comment '/home/myuser/.ssh/id_rsa'
Enter new passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved with the new passphrase.

How to receive or forward root emails in Linux

I find the following way the easiest way to manage who gets root's emails.

Open the file /etc/aliases for editing

vi /etc/aliases

Go to the bottom of the file where you see something similar to :

 # Person who should get root's mail
 #root:          marc

Change marc to the email address of the user who should get the email and finally uncomment the line. So for example your last two lines of /etc/aliases should look like:

# Person who should get root's mail
root:          you@yourcompany.com

Last step is to issue the following command ro reload /etc/aliases

newaliase

How to take a Java thread dump from a shell/bash script

We already know how to get a Java thread dump manually. But what if we want to take a Java thread dump from a script to help us automate things.

To do this we need to come up with a ps and grep combination that will only return one result. so grep as many times as neccessary and/or use keywords that are only found in your process name so you only get one result.

Remember to add [] somewhere in your first grep so grep doesn't matches itself.

output=$(ps -ef | grep m[y]ProcesseName )

Then all you need to do is sending the QUIT signal to $2 which stores the PID of your process.

kill -QUIT $2

So just add the above two lines to your script after modifying the first line to fit your needs and you should be good to go.

Using md5 checksum to check for integrity of your large uploaded/downloded file

Have you ever downloaded/uploaded a file from/to another computer and wanted to verify the integrity of the file to make sure it is not corrupted. The obvious way is to compare the file size of the original final and the copied file. However there are many ways that this number could mislead you. 

Just to give two examples, in some cases the program you use to download/upload the file can fail to fill in all the data and just pad the file to the correct size.  Hardware errors such as disk and memory issues can cause corruptions too that won't effect the file size. So how do you make sure your file is not corrupted and check the integrity of your downloaded/uploaded file.

On Linux/Unix you can use md5sum.  The way to use it is to simply run :

md5sum [yourfile]

This prints out something like :

16295afa0087ef75f33751cf003da993  [yourfile]

You will need to run the above command on both the origin of the file and it's destination and compare the digital fingerprint of the file printed. For unique files that fingerprint will be the same.

A lot of websites that let users download files ( specially large files which have a higher chance of getting corrupted in transit ) will provide checksums for their files which you can compare against your when you download is done.

Fixing "authentication is required to set the network proxy ..." on Redhat/REL/OEL

Whenever i started  a vncsession i would get an popup window stating :

authentication is required to set the network proxy used for downloading packages.  An
application is attempting to perform an action that requires privileges.
Authentication as the super user is required to perform this action" and asking
for the root password.

I didn't have the root password for this machine and hitting cancel would just bring back the pop-up in a few minutes. After googling here is the best way i found to fix this problem:

From a terminal window run "gnome-session-properties" and un-check "PackageKit
Update Applet"

Finally restart your vncserver and the issue should be gone.

Fixing "Firefox is already running" error in Linux

If you are on Linux and you get the following error when trying to start Firefox, read on for a solution:

Firefox is already running, but is not responding. To open a new window, you must first close the existing Firefox process, or restart your system.

This error usually is not solvable with killing the Firefox process, since you won't find one in the list of the current processes.

So how do we get around it?

First, you need to go to your profile directory.

• cd  ~/.mozilla/firefox/[your_profile]

Then delete the following two files :

• rm lock
• rm .parentlock

You should be able to open Firefox now.

If you have this problem occurring often like me, put the above command in a shell script for more convenience.

How to combine find and zip to compress WebLogic ADR directories

Weblogic Server has an ADR folder containing a sub directory called incident. This directory holds one directory for each incident that happens during an application lifetime.

Each of these individual incident folder can be several Mega Bytes so i use the following to zip them up and delete them. I set it as a cron to run every 5 minutes.

find /MW_HOME/user_projects/domains/my_domain/servers/AdminServer/adr/diag/ofm/apps/myapp/incident  -iname "incdir_*" -type d -exec sh -c 'zip -r {}.zip {}' \; -exec rm -rf {} \;

How to create a Java Thread dump

To create a java thread dump in Linux you must send a QUIT signal to your java process. JVM then catches that and prints the thread dump to standard output. If you are running your program from a script that has it's output directed to a file the thread dump will be written to that file obviously .

So first do we need to find the PID of our java program.

ps -ef | grep java

for example this could return :

lapps     3243  3065 21 Nov12 ?        08:01:50 java -Xms2048m -Xmx10g -XX:MaxPermSize=256m -Djava.awt.headless=true -XX:+UseParallelGC -XX:-UseGCOverheadLimit -classpath .:./test.jar:./lib/:./lib/* mypackage.org.TestApp

Now that we have 3243 as the PID, we simply send a QUIT signal to it.

kill -QUIT 3243

How to run sqlplus

To run sqlplus you need 3 env var set. Set them as follows if they are not set already. ( you can check if they are set by doing "echo $ORACLE_SID" for example to see if your SID is set. )

export ORACLE_HOME=<your Oracle Home>
export PATH=%ORACLE_HOME%\bin;%PATH%
export ORACLE_SID=<your SID>

Then run the following at the command line to start sqlplus.
sqlplus /nolog

This starts sqlplus with no user logged in.

How to export a schema in Oracle 11g

First make sure your ORACLE_SID,  PATH and ORACLE_HOME Env. variables are set. If not set them like below :
export ORACLE_HOME=<your Oracle Home>
export PATH=%ORACLE_HOME%\bin;%PATH%
export ORACLE_SID=<your SID>

Then run the following command from the command line:

expdp system/system_password@sid schemas=Schema_Name dumpfile=Schema_Name.dmp