Useful tips for day-to-day bash

Handling text files

Check if column 2 is different from some text (in this case -) and print it out

awk '$2~/^-$/ { next } { print }' file_input > file_output

Extract M-th column of a .csv file

awk -F "\"*,\"*" '{print $M}' file.csv

Get from the 10th to the 20th line of a file

sed -n '10,20p' myFile

Get multiple lines of a file (here line 1 and line 2)

sed -n -e 1p -e 2p  myFile

From the L-th line, get each M-th subsequent line

 sed -n '$L~$Mp' myFile

Sort col 1 numerically and count unique occurrences

cut -f1 20141020.combined_mask.whole_genomeV2.bed | sort -n | uniq -c

Substitute “string1” into “string2” in the file

sed -e "s/string1/string2/" myFile

Sort based on column 3

sort -k 3,3 myFile

remove lines with a specific pattern from a file

grep -v -e "pattern" myFile > newFile

  • same but tells you the lines where pattern does not happen (or remove -v to find where it happens)

    grep -v -n -e "pattern" myFile | cut -f1 -d":"

  • same but looks at one or both of two patterns

    grep -v -e "pattern1\|pattern2" myFile > newFile

Handling variables

echo some values and substitute space with tab delimiter

echo $a $b $c | tr -s ' ' '\t'

portable perl sum (or other operations) between numbers/variables

perl -E "say $a + $b"

Managing files

show files with extension .zip and print them without both path and extension

ls *.zip | xargs -n 1 basename -s .zip

Compress one at the time the files in the current folder (and subfolders) not yet in .gz format

find . -type f ! -name '*.gz' -exec gzip "{}" \;

Transfering files

List folders and print their name removing the last characther “/”

for var in `ls -d */`; do echo ${var::-1}; done