me making things: Linux Shell Commands

System Info:

ps:

$ ps aux 
 server processes, with user
$ ps -AlFH 
 long form, with arguments to commands, and threads
$ ps -ejH
$ ps axjf
$ pstree
$ ps -ef 
 show processes
$ ps aux | grep -i tomcat 
 check if tomcat is running

top:

interactive commands:
  h 
 show help commands
z or b 
 highlights running processes
c 
 display full command line
n 5 
 show only top 5 tasks
l t m 
 toggle headers (load avg, cpu tasks, memory))
d 
 change refresh rate from default 3 secs
1 
 toggle individual cpu / cpu total mode
O 
 choose column to sort by

see http://superuser.com/questions/575202/understanding-top-command-in-unix
see https://www.linux.com/learn/tutorials/42048-uncover-the-meaning-of-tops-statistics

command execution by the shell:
shell searches for command to execute in the following order:

see if builtin
then check in hashtable
then check in PATH variable

$ type -a ls 
 tells you if command is builtin, hashed or where it is located
$ which pwd 
 tells you where the binary is located
$ echo $PATH 
 shows path(s) where commands are searched for
$ hash 
 displays the hashtable cache used for quick command lookups
$ hash -r 
 clears the hashtable cache
$ export PATH=$PATH:~/bin 
 to temporarily add ~/bin to the path variable
$ alias ll='ls -l' 
 sets ll to be an alias for ls -l
$ unalias ll 
 unalias (use \ll to bypass alias)
$ history 
 then can do !123, to run the command listed by history

proc:

$ cat /proc/cpuinfo
$ cat /proc/meminfo | grep MemTotal 
should show total memory in the system
$ cat /proc/mounts

misc:

$ lscpu 
 shows info about CPU(s)
$ free -tg 
memory, including totals, in GB
$ mpstat -P ALL 
 cpu utilization on multi cpu machine
$ ls -l /proc/1138/exe 
 gives the process associated with the pid 1138
$ pwdx 23340 
 gives the current working directory of the process of pid 1138
$ pkill -9 -u jayz 
 to kill all sessions of the user jayz

File Management and Navigation:

cd:

$ cd ~ 
 go to your home directory
$ cd - 
 go to prev direcotory that we cd’ed from

list:

$ ls -l 
 can use ll as alias for this

1st Character – File Type: First character specifies the type of the file.

-
normal file
d
directory
s
socket file
l
link file

Field 1 – File Permissions: 9 character specifies the files permission, rwx for user, rwx for group and rwx for others.
Field 2 – Number of links: Second field specifies the number of links for that file.
Field 3 – Owner: Third field specifies owner of the file.
Field 4 – Group: Fourth field specifies the group of the file.
Field 5 – Size: Fifth field specifies the size of file.
Field 6 – Last modified date & time: Sixth field specifies the date and time of the last modification of the file.
Field 7 – File name: The last field is the name of the file.

$ ls -1 
 list with one line per file name
$ ls -lh 
 long listing of files with human readable sizes
$ ls -t 
 display files by last modified first
$ ls -1lr 
 reverse
$ ls -a 
 show all ifles, including hidden files
$ ll 
 alias for ls -a
$ ls -R 
 show files recursively
$ ls -F
    * / 
 directory.
* nothing 
 normal file.
* @ 
 link file.
* * 
 Executable file
$ ls -i 
 show files with inode number
$ ln targetFile linkToBeCreatedToTargetFile 
 creates hard links (so they'll have the same inode)
$ ls -l 
 should show the number of hard links the files have

(rm only removes one link, the file is not deleted until all the links are deleted)
$ ln -s targetFile linkToBeCreatedToTargetFile 
 creates a soft link (symlink) the 2 files have different inodes
$ ls -l 
 will also show the linking between the files

misc:

$ ant start-icnow | tee op.txt 
sends output to both stdout and a file
$ watch command 
 you can see the output of the command updated every two seconds

tail:

$ tail -f /var/log/syslog -f /var/log/auth.log 
 multi tail
$ tail -f filename1 filename2 
 try this one too
$ tail -c30 /var/log/syslog 
 display last 30 bytes from syslog
$ head -c40 /var/log/syslog 
 display first 40 bytes from syslog

find:

also see grep

$ find .
list all files in current and subdirectories
$ find . *.txt -type f | xargs grep -i pattern
feed all files with ext .txt from the current and sub dirs to grep, which does a case-insen search for files containing pattern.
$ find / -name filename
search starting from /
$ find . -name *handle*
case sensitive names containing handle
$ find . -iname *handle*
case insensitive
$ find . -inum 433580
find all the hard links with given inode
$ find . -type d -name AMA
find all directories named AMA
$ find . -maxdepth 1 -not -iname "MyCProgram.c"
$ find ~ -empty
$ find . -type d
find all directories
$ find . -type f
find only the normal files
$ find . -type f -name ".*"
find all the hidden files
$ find -type d -name ".*"
find all the hidden directories
$ find -newer FILE
find files which are modified after modification of a particular FILE
$ find -anewer FILE
find files which are accessed after modification of a specific FILE
$ find -cnewer FILE
find files whose status got changed after the modification of a specific FILE
$ find ~ -size +100M
find files bigger than the given size
$ find ~ -size -100M
find files smaller than the given size
$ find ~ -size 100M
find files that matches the exact given size
$ find . -mmin -60
find files in current and sub dirs, updated in the last 60 minutes
$ find / -mtime -1
find all the files (under /) that got updated within the last 1 day
$ find -amin -60
find files in current and sub dirs, accessed within last 60 minutes
$ find / -atime -1
finds all the files (under /) taccessed within the last 1 day
$ find . -cmin -60
find files in current and sub dirs, changed within last 60 minutes
$ find / -ctime -1
finds all the files (under /) changed within the last 1 day

less:

$ less file1 
 starts off by displaying the head of file1

ESC	should take you to command mode, indicated by a :
h	inline help
=	status, position and statistics
Navigation	[Arrows]/[Page Up]/[Page Down]/[Home]/[End][Space bar]
ESC <	Go to first line in file (or line N).
ESC >	Go to last line in file (or line N).
ng	Jump to line number n. Default is the start of the file.
nG	Jump to line number n. Default is the end of the file.
j	scroll forward 1 line (10j scroll forward 10 lines)
k	scroll backward 1 line (5 scroll backward 5 lines)
d	Forward one half-window (d5 sets half-window to 5 lines).
u	Backward one half-window (u2 set half-window to 2 lines).

mletter	Mark the current position with letter.
'letter	Return to position letter. [' = single quote]

/pattern	Search forward for (N-th) matching line. can use regular expressions.
?pattern	Search backward for (N-th) matching line.
n	Go to next match (after a successful search).
N	Go to previous match.
ESC-u	Undo (toggle) search highlighting.
&pattern	Display only matching lines (& again resets)
^N or !	Search for NON-matching lines.

!command	Execute a shell command
F	tail current file. Ctrl+c to exit this mode.

-i	toggle Case-insensitive searches.
-M	Shows more detailed prompt, including file position.
-N	toggle show line numbers
-S	toggle line wrap ("chop long lines")
-s	toggle squeeze blank lines
q	Quit.

grep:

$ grep -r -i -n 'hello world' . 
case insensitive recursive search for 'hello world' starting from current dir, also print line numbers
$ grep -r -i 'pattern' . 
 recursive case-insensitive search for 'pattern', starting from current dir
$ grep -rin 'pattern' . 
 recursive case-insensitive search for 'pattern', starting from current dir, show line number
$ grep -ril 'pattern' . 
 recursive case-insensitive search for 'pattern', starting from current dir show, only occuring file name
$ grep -i 'hello world' menu.h main.c 
 case insentive search for 'hello world' in files menu.h and main.c

-i	ignore case distinctions
-v	select non-matching lines
-n	print line number with output lines
-H	print the file name for each match
-o	show only the part of a line matching PATTERN
-r	recursive
--include=FILE_PATTERN	search only files that match FILE_PATTERN
--exclude=FILE_PATTERN	skip files and directories matching FILE_PATTERN
--exclude-dir=PATTERN	directories that match PATTERN will be skipped.
-L	print only names of FILEs containing no match
-l	print only names of FILEs containing matches
-c	print only a count of matching lines per FILE
-B	print NUM lines of leading context
-A	print NUM lines of trailing context
-C	print NUM lines of output context

vi:

ESC	switches from edit mode to command mode
edit mode commands
i	Insert before current cursor position
I	Insert at beginning of current line
a	Insert (append) after current cursor position
A	Append to end of line
r	Replace 1 character
R	Replace mode
x	Delete single character
dd	Delete current line and put in buffer
5dd	Delete 5 lines and put them in buffer
J	Attaches the next line to the end of the current line (deletes carriage return).
u	Undo last command
yy (:y)	Yank current line into buffer
7yy	Yank 7 lines into buffer
p	Put the contents of the buffer after the current line
P	Put the contents of the buffer before the current line
^d	Page down
^u	Page up
^g	Display current line number
command mode commands
:13	Position cursor at line 13
:$	Position cursor at end of file
:w	Write the current file.
:w new.file	Write the file to the name 'new.file'.
:w! existing.file	Overwrite an existing file with the file currently being edited.
:wq	Write the file and quit.
:q	Quit.
:q!	Quit with no changes.
:set number	Turns on line numbering
:set nonumber	Turns off line numbering

h,j,k,l or arrow keys	Navigation - Left,Down,Up,Right
/string	Search forward for string
?string	Search back for string
n	Search for next instance of string
N	Search for previous instance of string
:5,13:s/string1/string2/[g]	Substitute string1 with string2 on lines5 to 13. [g] for global replace, else only 1st occurance is replaced per line. ^ matches start of line . matches any single character $ matches end of line (escape special characters with \)
Examples:
:1,$:s/dog/cat/g	Substitute 'cat' for 'dog', every instance for the entire file - lines 1 to $ (end of file)/td>
:23,25:/frog/bird/	Substitute 'bird' for 'frog' on lines 23 through 25. Only the first instance on each line is substituted.

Also see http://www.lagmonster.org/docs/vi.html

Disks related:

Applications - > Accessories - Disk Usage Analyzer (baobab) graphical

$ df -h 
 disk usage, in GB
$ du -shx */ 
 disk usage, for all directories from current directory
$ du -shx * 
 disk usage, for all directories and files from current directory
$ sudo fdisk -l  
 will list all (mounted) filesystems and partitions on the hard disks
$ sudo hdparm -I /dev/sda 
details on disk
$ sudo mount 
shows all currently mounted partitions
$ mkdir /mnt/backup 
 create mount point
$ mount /mnt/backup 
 mount the drive (add entry to /etc/fstab to mount automatically everytime on boot)
$ sudo rm -r /home/JLseagull/.local/share/Trash/files/ 
 empty trash

Networking related:

wget, curl:

$ wget -q -O - checkip.dyndns.org|sed -e 's/.*Current IP Address: //' -e 's/<.*$//' 
command line whatismyip
$ wget 
download entire website
--recursive 
download the entire site
--no-clobber 
don't overwrite any existing files (if the download is interrupted and resumed)
--page-requisites 
get all the page resources (images, css, etc.)
--html-extension 
save files with the .html extension
--convert-links 
convert links so that they work locally, off-line
--restrict-file-names=windows 
modify filenames so that they will work in Windows as well
--domains web.site 
don't follow links outside web.site
--no-parent 
don't follow links outside the directory root/
www.web.site/root/ 
start downloading site from this context

netstat:

$ netstat -ie 
 shows network interface information
$ netstat -ap 
 shows all the ports the processes are connected to
$ netstat -ant | grep LISTEN 
show all listening tcp ports
$ netstat -nlp 
 lists ports in use
$ netstat -ct 
 continuously shows all tcp connections
$ netstat --listen 
 show open ports
$ netstat -ap tcp | grep -i "listen"

nmap:
To scan a network, for eg. to find the IP of a machine running ssh (port 22) on the 192.168.1.* network, listing only open ports (--open) and listing service/version (-sV)

$ nmap -p 22 --open -sV 192.168.1.*

to be able to login from local to remote using passwordless ssh

local$ ssh-keygen -t dsa 
 enter passphrase, or leave blank
local$ ssh-copy-id -i ~/.ssh/id_dsa.pub username@remote 
automatically create the necessary folders and permissions correctly on remote
(or, do this)
ssh-copy-id -i ~/.ssh/id_dsa.pub srvc01.chi

# OR, if you want to do it manually
local$ scp ~/.ssh/id_dsa.pub username@remote
local$ ssh username@remote
remote$ cat ~/id_dsa.pub >> ~/.ssh/authorized_keys
remote$ chmod 644 ~/.ssh/authorized_keys 
 ssh needs this file to be NOT world writable
remote# chmod 700 ~/.ssh # ssh needs .ssh to be NOT world writable
remote$ exit

local$ ssh username@remote 
 will ask for ssh passphrase (or not, if passhphrase was blank, useful for automating
logins, etc)

Note: if passphrase was entered, can add to local ssh keyring? or ssh-agent ? for extra security? if using ssh2, authorized_keys file is actually authorized_keys2 ? to restart sshd on the remote machine

/etc/init.d/sshd reload 
 do not do /etc/init.d/sshd restart, that will kick you out of the session ?

certs related:

$ keytool -printcert -v -file cert1.crt
$ keytool -list -v -storetype JCEKS -keystore keystore1.jceks 
$ keytool -list -v -keystore keystore1.jks
$ keytool -list -v -keystore keystore1.jks -alias alias1

misc:

$ lsof -i :8080 
 shows all machines using port 8080
sudo lsof -PiTCP -sTCP:LISTEN
$ fuser 7000/tcp 
 gives the pid of the process using the tcp port 7000

clean up the foll:

/etc/sshd/sshd_config on remote should have
RSAAuthentication yes
PubkeyAuthentication yes
AllowUsers username 
 to restrict which users can ssh
AuthorizedKeysFile %h/.ssh/SOME_ORIGINAL_NAME 
 to use non-default authorised keys file
check /var/log/auth for error messages
http://www.debian-administration.org/articles/87 good site for "Keeping SSH access secure"

64 bit support:

Check if a CPU supports 64-bit by entering the command below. In the output of the flags section, you will see many entries. Look for a 2 character entry (surrounded by spaces - not part of another word) called "lm".

$ cat /proc/cpuinfo | grep lm

If you're unsure whether the Linux OS that you are currently running is 64-bit, you can enter the following command and look for the value x86_64 in the output as opposed to something like i686, i586, and so on:

$ uname -m
x86_64

Similarly to above, you can also check if a given Linux executable is 64-bit. In the case below we examine the /bin/ls executable and look for the string "ELF 64-bit LSB executable" as opposed to "ELF 32-bit LSB executable":

$ file /bin/ls
/bin/ls: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses
shared libs), for GNU/Linux 2.6.24

sudo and account related:

$ su -l # get a root shell
$ sudo !! # repeat the previous command which you were not able to run because it required sudo
$ last # shows last few logins
$ lastb # shows last few failed logins
$ cat /etc/password # this file has list of all users
$ id username # shows userid, and groups that user belongs to
$ sudo users-admin # fires off users admin gui
$ sudo usermod -a -G dba JLseagull # add user JLseagull to group dba
$ chown greatGull file1 # change owner of file1 to greatGull
$ chgrp gullFlock file1 # change group of file1 to gullFlock

To enable the root account in Ubuntu, enter the command sudo passwd root. When you see the phrase "Enter new UNIX password" this is to define the password for the root account: To enable the root account

user@ubuntu:~$ sudo passwd root
[sudo] password for user:
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully

Encryption related:

Use "Disk Utility" to graphically show volume information, including encryption status
To check from the command line

root$ cat /etc/crypttab
sda5_crypt UUID=fa5a6f0a-f8ce-1ea9-c1a1-23231a98b65e none luks

root$ cryptsetup status sda5_crypt

root$ pvdisplay -m

root$ lvdisplay -m

gpg related:

$ gpg --import key.asc # to import the private keys into gpg
$ gpg FILENAME.pgp # and provide passphrase above to decrypt

# script to process all the files in a folder
#!/bin/sh
for currentfile in *.pgp
do
 echo .................................... processing ${currentfile}
 gpg --passphrase substPassphraseHere --quiet --no-verbose --no-secmem-warning --no-greeting $currentfile
done
echo finished

Software management:

Can use "Synaptic" for apt management, or can do command line apt related commands.
To remove

$ sudo apt-get remove avahi-daemon

Linux Shell Commands

System Info:

server processes, with user

long form, with arguments to commands, and threads

show processes

check if tomcat is running

show help commands

highlights running processes

display full command line

show only top 5 tasks

toggle headers (load avg, cpu tasks, memory))

change refresh rate from default 3 secs

toggle individual cpu / cpu total mode

choose column to sort by

tells you if command is builtin, hashed or where it is located

tells you where the binary is located

shows path(s) where commands are searched for

displays the hashtable cache used for quick command lookups

clears the hashtable cache

to temporarily add ~/bin to the path variable

sets ll to be an alias for ls -l

unalias (use \ll to bypass alias)

then can do !123, to run the command listed by history

should show total memory in the system

shows info about CPU(s)

memory, including totals, in GB

cpu utilization on multi cpu machine

gives the process associated with the pid 1138

gives the current working directory of the process of pid 1138

to kill all sessions of the user jayz

File Management and Navigation:

go to your home directory

go to prev direcotory that we cd’ed from

can use ll as alias for this

normal file

directory

socket file

link file

list with one line per file name

long listing of files with human readable sizes

display files by last modified first

reverse

show all ifles, including hidden files

alias for ls -a

show files recursively

directory.

normal file.

link file.

Executable file

show files with inode number

creates hard links (so they'll have the same inode)

should show the number of hard links the files have

(rm only removes one link, the file is not deleted until all the links are deleted)

creates a soft link (symlink) the 2 files have different inodes

will also show the linking between the files

sends output to both stdout and a file

you can see the output of the command updated every two seconds

multi tail

try this one too

display last 30 bytes from syslog

display first 40 bytes from syslog

also see grep

list all files in current and subdirectories

feed all files with ext .txt from the current and sub dirs to grep, which does a case-insen search for files containing pattern.

search starting from /

case sensitive names containing handle

case insensitive

find all the hard links with given inode

find all directories named AMA

find all directories

find only the normal files

find all the hidden files

find all the hidden directories

find files which are modified after modification of a particular FILE

find files which are accessed after modification of a specific FILE

find files whose status got changed after the modification of a specific FILE

find files bigger than the given size

find files smaller than the given size

find files that matches the exact given size

find files in current and sub dirs, updated in the last 60 minutes