I love the unix philosophy: carry around a small set of tools and use them to build bigger custom tools to solve problems. Over the last few years, I’ve the following programs to my ‘must have’ list:
strace – trace system calls
Strace tells you which system calls a process is making. It gives loads of information about errant processes – is it blocked on network or file i/o? Is it stuck in a loop? I used this recently to find out why ssh logins were slow on one of my machines. I used ‘ps ax | grep sshd’ to find the pid of sshd, then ran “sudo strace -f -t -p PID”. The ‘-f’ means to also trace any child processes, and ‘-t’ gives timestamps. This showed that sshd was doing a reverse DNS lookup when I logged in (wrongly set up dns) and also that the default ubuntu .bashrc takes a good while to run.
lsof – list open files
lsof is useful in conjunction with strace; strace will show you that a process is reading on file descriptor 7, but what is that used for? Running “lsof -p PID” will tell you what each file descriptor is connected to.
cstream – filter, monitor and bandwidth-limit stream
cstream is great for monitoring long running jobs. I use this often to monitor the progress of mysql imports from mysqldump files. Eg. “cat dump.sql | cstream -l -T1 | mysql DATABASE” lets me know how much of the file has been processed so far.
You can also use cstream to bandwidth-limit a stream, but I tend to do my bandwidth limiting via rsync (–bwlimit) or scp (-l).
socat – like netcat, but better
Socat is netcat for network, processes, files, sockets, etc, etc. It also doesn’t buffer output in the same annoying way that netcat does, which makes it more useful for creating mock servers. For example, I recently used it to create a dummy HTTP server for testing erlang’s inets library:
Create a script called “reply-204” containing
#!/bin/bash sleep 1 echo -ne 'HTTP/1.1 204 No Content\r\nSomeHeader: foornrn' sleep 100
.. then run “socat tcp-listen:9999,reuseaddr exec:./reply-204”.
watch – run a command repeatedly
I used to write this loop lots: “while true; do ls -l somefile; sleep 1; done”. Now I just use watch, for example “watch -n1 ‘ls -l somefile'”. The “-d” flag is also useful – it highlights difference between each run.
Commands which run other commands are the happiest commands in the world.
iftop – what traffic is going where?
iftop is like top, but for network traffic. Great for getting a quick overview of why your network connection has suddenly slowed to a crawl. Also good for noticing weird connections (aka, why is my machine sending traffic there?).
tcpflow
Like tcpdump, tcpflow captures network packets. Additionally, it stores each “conversation” in a seperate file which makes it easy to futher analyze.
Whilst running tcpflow a minute ago, my browser happened to request this page and tcpflow let me see that it returns this header: “Server: Modified Atari-ST”. Do you think it’s true?
iperf – how fast can my network go?
iperf is a simple end-to-end network performance tool. Answers the question: What’s the maximum transfer rate between two machines? I recently moved all my photos and videos onto a separate media server box, and loading up big jpegs was taking a few seconds. I used iperf to check my actual network speed, but sadly the performance was pretty close to the theoretical maximum .. sadly, moving lots of bits still takes a while!
Not forgetting
- My favourite grep flags: “-o” to only show matching text, and “-P” to get perl regexps (eg. non-greedy quantifiers)
- My favourite cat flags: “-T” to show tabs as “^I” … useful for eyeballing tab-separated files
- My favourite less flags: “-S” makes long lines get truncated, rather than wrapping.