Both strace and ltrace are powerful command-line tools for debugging and troubleshooting programs on Linux: Strace captures and records all system calls made by a process as well as the signals received, while ltrace does the same for library calls.
If a program acts differently than you expect, you can use these tools to see “behind the curtain” and maybe get some clues as to what is going on.
Be forewarned, though. When you use either of these commands, you will end up with a lot of output to look through. Still, that can tell you quite a bit about how a process is working and sometimes give you important insights.
strace
To run strace against a program, use a command like what is shown below—the strace command followed by the program name. The output below has been cut short.
$ strace who execve(“/usr/bin/who”, [“who”], 0x7ffe889f45c0 /* 41 vars */) = 0 brk(NULL) = 0x55e7d6720000 arch_prctl(0x3001 /* ARCH_??? */, 0x7fff38c72b20) = -1 EINVAL (Invalid argument) access(“/etc/ld.so.preload”, R_OK) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, “/etc/ld.so.cache”, O_RDONLY|O_CLOEXEC) = 3 newfstatat(3, “”, {st_mode=S_IFREG|0644, st_size=60195, ...}, AT_EMPTY_PATH) = 0 mmap(NULL, 60195, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fd1ba4d7000 close(3) …
The command above runs the who command and reports on the system calls that it makes. The start of each line (e.g., execve, brk and arch_prctl) shows the system call being made.
To run strace against a running process to see what it’s doing, use the command with the -p option followed by the process ID. Note that the strace output starts in the 3rd line below after the process is attached.
$ strace -p 34512 strace: Process 34512 attached wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 34516 …
NOTE: You cannot redirect the output of the strace command through a pipe, but you can have it sent to a file using the -o option. In the example below, the date command output is shown. After that, the top of the output file is displayed using the head command.
$ strace -o outputfile date Tue May 3 03:33:52 PM EDT 2022 $ head -10 outputfile execve(“/usr/bin/date”, [“date”], 0x7ffc30f3bd00 /* 41 vars */) = 0 brk(NULL) = 0x55e78c077000 arch_prctl(0x3001 /* ARCH_??? */, 0x7ffe52bf2ec0) = -1 EINVAL (Invalid argument) access(“/etc/ld.so.preload”, R_OK) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, “/etc/ld.so.cache”, O_RDONLY|O_CLOEXEC) = 3 newfstatat(3, “”, {st_mode=S_IFREG|0644, st_size=60195, ...}, AT_EMPTY_PATH) = 0 mmap(NULL, 60195, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f589661b000 close(3) = 0 openat(AT_FDCWD, “/lib64/libc.so.6”, O_RDONLY|O_CLOEXEC) = 3 read(3, “177ELF2113 3 > 1 PP4 ”..., 832) = 832
Remember that you’re looking at system calls, not programs. So, don’t be surprised if you don’t recognize “execve”, “brk”, “arch_prctl”, etc. You can, however, examine the man pages for these system calls if you want to learn more about what they do and how they work. Notice that the man pages for system calls come from Section 2 of the man pages.
$ man execve | head -4 EXECVE(2) Linux Programmer’s Manual EXECVE(2) <== NAME execve - execute program
Another useful way to use the strace command is to add the -c option. When you do this, the output provides a summary of the system calls made ordered by percentage of time spent on each system call (largest first). As you see in the example below, more time is spent on mmap (mapping files) than on any of the other system calls.
$ strace -c date Tue May 3 03:40:25 PM EDT 2022 % time seconds usecs/call calls errors syscall ——— —————- —————- ————- ————- ———————— 28.49 0.000106 11 9 mmap 12.10 0.000045 11 4 openat 10.22 0.000038 9 4 mprotect 9.95 0.000037 6 6 newfstatat 8.87 0.000033 8 4 pread64 6.72 0.000025 4 6 close 4.57 0.000017 5 3 read 3.49 0.000013 13 1 1 access 2.96 0.000011 11 1 munmap 2.69 0.000010 10 1 write 2.15 0.000008 2 3 brk 1.88 0.000007 3 2 1 arch_prctl 1.34 0.000005 5 1 set_tid_address 1.34 0.000005 5 1 set_robust_list 1.08 0.000004 4 1 lseek 1.08 0.000004 4 1 prlimit64 1.08 0.000004 4 1 getrandom 0.00 0.000000 0 1 execve ——— —————- —————- ————- ————- ———————— 100.00 0.000372 7 50 2 total
ltrace
The ltrace command works much the same as strace, but examines library calls. In the examples below, we see that ltrace uses basically the same syntax as strace.
This first example looks at the library calls used by the who command.
$ ltrace who strrchr(“who”, ‘/’) = nil setlocale(LC_ALL, “”) ="en_US.UTF-8" bindtextdomain(“coreutils”, “/usr/share/locale”) ="/usr/share/locale" textdomain(“coreutils”) ="coreutils" __cxa_atexit(0x560c6532d220, 0, 0x560c65334940, 0) = 0 getopt_long(1, 0x7ffc1f74a468, “abdlmpqrstuwHT”, 0x560c65334960, nil) = -1 setlocale(LC_TIME, nil) ="en_US.UTF-8" strlen(“en_US.UTF-8”) = 11 __memcpy_chk(0x7ffc1f74a1f0, 0x560c65bf6370, 12, 257) = 0x7ffc1f74a1f0 strcmp(“en_US.UTF-8”, “POSIX”) = 21 utmpxname(0x560c65331016, 1, 0x560c65331266, 0x560c65331266) = 0 setutxent(0x560c6533101e, 0x7f2330570cc0, 5, 0) = 1 …
This next example shows the library calls for a running process:
$ ltrace -p 35243 sigemptyset(<>) = 0 sigemptyset(<>) = 0 sigaction(SIGINT, { nil, <>, 0x3f679854, 0x560643c72e53 }, { 0x560643c73f80, <>, 0x9018, 0x1000 }) = 0 ioctl(2, 21523, 0x7ffe3f679870) = 0 malloc(48) = 0x56064548b020 strlen(“LINES”) = 5 malloc(6) = 0x56064548dfb0 strcpy(0x56064548dfb0, “LINES”) = 0x56064548dfb0 strlen(“LINES”) = 5 malloc(6)
Sending ltrace output to a file requires use of the -o option.
$ ltrace -o outputfile date Tue May 3 04:17:30 PM EDT 2022 $ head -10 outputfile strrchr(“date”, ‘/’) = nil setlocale(LC_ALL, “”) ="en_US.UTF-8" bindtextdomain(“coreutils”, “/usr/share/locale”) ="/usr/share/locale" textdomain(“coreutils”) ="coreutils" __cxa_atexit(0x55e8afe3f280, 0, 0x55e8afe4e000, 0) = 0 getopt_long(1, 0x7ffdc8986948, “d:f:I::r:Rs:u”, 0x55e8afe4e920, nil) = -1 nl_langinfo(0x2006c, 0x7ffdc8986948, 0, 0) = 0x7f3afe645435 getenv(“TZ”) = nil malloc(128) = 0x55e8b05b0440 clock_gettime(0, 0x7ffdc8986740, 1, 145) = 0
Notice that the man page for library calls comes from Section 3 of the man pages.
$ man strrchr | head -4 STRCHR(3) Linux Programmer’s Manual STRCHR(3) <== NAME strchr, strrchr, strchrnul - locate character in string
$ man strlen | head -4 STRLEN(3) Linux Programmer’s Manual STRLEN(3) NAME strlen - calculate the length of a string
Using the -c option with ltrace, like with strace, orders the listing with the longest run times first.
$ ltrace -c pwd /home/shs % time seconds usecs/call calls function ——— —————- —————- ————- —————————— 16.02 0.000428 428 1 setlocale 10.82 0.000289 72 4 __freading 7.93 0.000212 212 1 puts 6.74 0.000180 90 2 fclose 5.39 0.000144 72 2 fileno 5.39 0.000144 144 1 getcwd 5.35 0.000143 71 2 __fpending 5.28 0.000141 70 2 fflush 4.79 0.000128 128 1 free 4.64 0.000124 124 1 bindtextdomain 4.49 0.000120 120 1 exit_group 4.49 0.000120 120 1 getenv 4.19 0.000112 112 1 getopt_long 4.00 0.000107 107 1 textdomain 3.97 0.000106 106 1 strrchr 3.93 0.000105 105 1 __cxa_atexit 2.58 0.000069 69 1 __cxa_finalize ——— —————- —————- ————- —————————— 100.00 0.002672 24 total
Installing strace and ltrace
Commands like these are used to install strace:
$ sudo apt install strace # Debian/Ubuntu $ sudo yum install strace # RHEL/Centos $ sudo dnf install strace # Fedora
Similarly, ltrace will install with commands like these:
$ sudo apt install ltrace # Debian/Ubuntu $ sudo yum install ltrace # RHEL/Centos $ sudo dnf install ltrace # Fedora
Wrap-Up
Both the strace and ltrace commands provide copious output, but can prove to be very helpful, especially once you get used to the output that they provide.
