I love the unix philosophy: carry around a small set of tools and use them to build bigger custom tools to solve problems. Over the last few years, I’ve the following programs to my ‘must have’ list:<\/p>\n
Strace<\/a> tells you which system calls a process is making. It gives loads of information about errant processes – is it blocked on network or file i\/o? Is it stuck in a loop? I used this recently to find out why ssh logins were slow on one of my machines. I used ‘ps ax | grep sshd’ to find the pid of sshd, then ran “sudo strace -f -t -p PID”. The ‘-f’ means to also trace any child processes, and ‘-t’ gives timestamps. This showed that sshd was doing a reverse DNS lookup when I logged in (wrongly set up dns) and also that the default ubuntu .bashrc takes a good while to run.<\/p>\n lsof<\/a> is useful in conjunction with strace; strace will show you that a process is reading on file descriptor 7, but what is that used for? Running “lsof -p PID” will tell you what each file descriptor is connected to.<\/p>\n cstream<\/a> is great for monitoring long running jobs. I use this often to monitor the progress of mysql imports from mysqldump files. Eg. “cat dump.sql | cstream -l -T1 | mysql DATABASE” lets me know how much of the file has been processed so far.<\/p>\n You can also use cstream to bandwidth-limit a stream, but I tend to do my bandwidth limiting via rsync (–bwlimit) or scp (-l).<\/p>\n Socat<\/a> is netcat for network, processes, files, sockets, etc, etc. It also doesn’t buffer output in the same annoying way that netcat does, which makes it more useful for creating mock servers. For example, I recently used it to create a dummy HTTP server for testing erlang’s inets<\/a> library:<\/p>\n Create a script called “reply-204” containing<\/p>\n .. then run “socat tcp-listen:9999,reuseaddr exec:.\/reply-204”.<\/p>\n I used to write this loop lots: “while true; do ls -l somefile; sleep 1; done”. Now I just use watch<\/a>, for example “watch -n1 ‘ls -l somefile'”. The “-d” flag is also useful – it highlights difference between each run.<\/p>\n Commands which run other commands are the happiest commands in the world.<\/p>\n iftop<\/a> is like top, but for network traffic. Great for getting a quick overview of why your network connection has suddenly slowed to a crawl. Also good for noticing weird connections (aka, why is my machine sending traffic there?).<\/p>\n Like tcpdump, tcpflow<\/a> captures network packets. Additionally, it stores each “conversation” in a seperate file which makes it easy to futher analyze.<\/p>\n Whilst running tcpflow a minute ago, my browser happened to request this page<\/a> and tcpflow let me see that it returns this header: “Server: Modified Atari-ST”. Do you think it’s true?<\/p>\nlsof – list open files<\/h3>\n
cstream – filter, monitor and bandwidth-limit stream<\/h3>\n
socat – like netcat, but better<\/h3>\n
#!\/bin\/bash\r\nsleep 1\r\necho -ne 'HTTP\/1.1 204 No Content\\r\\nSomeHeader: foornrn'\r\nsleep 100<\/pre>\n
watch – run a command repeatedly<\/h3>\n
iftop – what traffic is going where?<\/h3>\n
tcpflow<\/h3>\n
iperf – how fast can my network go?<\/h3>\n