Using sp-rtrace

The sp-rtrace tool allows you to run programs, and collect resource allocation/free backtraces from a given set of functions, even from optimised binaries. The sp-rtrace tool supports plugins that define the set of functions that the tool traces. Multiple plugins can be used at the same time, but trace toggling affects all of them.

Packages

source: sp-rtrace

binary: sp-rtrace, sp-rtrace-visualize

Installing sp-rtrace on the Harmattan device

Install sp-rtrace through the developer mode applet.

Prerequisite: Developer mode must be enabled.

Select Settings > Security > Developer mode.
Install the Tracing bundle package by clicking Install.
You get a notification screen that lists all the applications to be installed in the bundle package. To install, click OK.
A dependency notice appears. Click Accept.

For more information on developer mode and installable tools, see Activating developer mode.

Features

The sp-rtrace binary package has several components:

sp-rtrace: command line front end which outputs/saves the trace data as binary.
libsp-rtrace-main: with the plugin LD_PRELOAD libraries, they implement the tracing functionality and transfer the traced data to the front end process.
rtrace-module-gen: included with the sources for generating trace modules from standard templated and traced functions configuration file.
libsp-rtrace1: API library for marking when a given process context starts and ends (so that, for example, servers can differentiate resources specific to each client).
sp-rtrace-postproc: converts the binary trace data into ASCII, and optionally:
- Merges identical backtraces in the trace data.
- Removes allocations for already freed resources.
- Calls sp-rtrace-resolve (with -t elf option).
sp-rtrace-resolve: fully resolves the symbols (using separate debug symbol files) for the backtrace addresses. Compared to Valgrind, the symbol resolving capacity of the sp-rtrace tool is equal or better. The sp-rtrace tool can use both ELF- and BFD-based resolving.
Several utility scripts for getting trace statistics and so forth, see the following sections.

Currently, the following trace plugins are available:

file traces file descriptor operations.
gobject traces Glib GObject creation and references.
memory traces C-library memory allocation functions.
memtransfer traces C-library memory read/write functions, for example, memset.
pagemap gets processes memory page mapping information at the end of the trace.
qobject traces Qt QObject creation and destruction.
shmposix traces POSIX shared memory and mmap operations.
shmsysv traces SYSV shared memory operations.

Technical details for sp-rtrace:

Uses LD_PRELOAD to trace the functions.
By default, it collects backtraces with the C-library backtrace function, but can optionally do it also with libunwind.

The sp-rtrace-visualize binary package contains resource allocation visualization programs and scripts for producing histograms, call graphs, timelines and memory fragmentation maps.

When to use sp-rtrace

The sp-memusage, sp-endurance, and sp-smaps tools give you the system view on the process memory usage. This is aligned to page size (4KB) and due to memory fragmentation, the process heap usage can sometimes actually be much larger than what the process itself needs. The sp-trace tool has the rtrace-allocmap post-processing tool to visualize (memory) fragmentation.

Valgrind Memcheck plugin can detect lost pointers to allocations, so with Valgrind you do not need to repeat use cases to see whether something leaks. Sp-rtrace (and functracer) are intended for analysing non-freed allocations and for the ones which still have valid references sp-rtace, with its post-processing tools, is better than Valgrind. Before performing memory leak analysis with sp-rtrace (and functracer), fix all the memory leaks reported by Valgrind. With sp-rtrace you usually need to repeat your test cases several times and do this separately for each test case.

Valgrind Massif plugin shows process memory usage as a function of time (with backtraces) over whole application lifetime whereas sp-rtrace can take a snapshot between any given points in time and do call graph of allocations done during that time interval. Both are useful, and it is recommended to use both (they may show different things). Sp-rtrace is faster than Valgrind as it does not emulate the CPU.

Functracer is similar to sp-rtrace, the main difference is that it can attach to an already running processes and can catch some things that sp-rtrace (LD_PRELOAD) does not.

Using the tool

Note: To use this tool with the developer account, run the sp-rtrace tool with the aegis-exec command. For example:

 aegis-exec -u user sp-rtrace -s -e memory -x ls

Tracing

To start tracing (-s) of "ls" program's (-x) memory allocations and deallocations (-p memory), enter the following command:

 sp-rtrace -s -e memory -x ls

To trace file accesses instead, enter the following command:

 sp-rtrace -s -e file -x ls

To additionally make backtraces (-b) eight frames long and store the output to "trace" directory (-o), enter the following command:

 mkdir trace
 sp-rtrace -s -e memory -b 8 -o trace/ -x ls

Toggling tracing

If you want to trace some specific use case instead of process startup, just leave the -s option out and toggle tracing at runtime. Enter the following command:

 sp-rtrace -t $(pidof program)

To do the same directly, enter the following command:

 kill -SIGUSR1 $(pidof my-application)

You can also use the latter option in the program itself if you want tracing to start from a specific place.

Post-processing

Trace files are by default in binary format and named PID[-index].rtrace. Post-processed files are in text format and named PID[-index].rtrace.txt.

When investigating more complicated programs, merge identical backtraces and remove freed allocations. To do this, enter the following command:

 cd trace
 sp-rtrace-postproc -cl  <  14422.rtrace  >  14422.rtrace.txt

You can use sp-rtrace-resolve to resolve the remaining backtrace addresses. Enter the following command:

 sp-rtrace-resolve  <  14422.rtrace.txt  >  trace/14422-r.rtrace.txt

By default, this uses the slow and memory consuming BFD resolver to get the source file names, line numbers and names of non-exported functions. If you are running it on the device, use also the -m multi-pass option to reduce the memory usage after making sure that the current directory has a lot of free disk space for temporary files.

Note that the relevant debug symbol packages need to be installed before the backtrace addresses can be resolved. Additional debug symbols slow down resolving and increase the amount of memory needed, but especially for call graphs it may be better to have all the function names, as nodes without names are not included into call graphs.

To make sp-rtrace-postproc run the resolver automatically, use the -r option. Enter the following command:

  sp-rtrace-postproc -clr  <  14422.rtrace  >  14422.rtrace.txt

Note that the command asks sp-rtrace-resolve to use the faster ELF resolver which also uses less memory, but it does not provide more detailed information, such as line numbers.

Note: With the -P option, you can ask sp-rtrace itself to do all of the above post-processing when saving the trace output. Enter the following command:

 sp-rtrace -s -m -e memory -o trace/ -P "-clr" -x ls

You can do all the post-processing in one step, unless you need separate text data files for the following:

all the resource allocations as required by the allocation timelines generation
only non-freed resource allocations as required by the call graph generation

If ELF resolving referred above does not give enough information, use sp-rtrace-resolve explicitly. Enter the following command:

 sp-rtrace -s -p memory -x ls
 sp-rtrace-postproc -cl  <  ls-memory.rtrace.txt  |  sp-rtrace-resolve -m multi-pass  >  ls-memory-resolved.rtrace.txt

The sp-rtrace-visualize tools can be used to visualise the trace data.

Utilities

The sp-rtrace tool includes the following convenience scripts:

rtrace-rename renames trace files.
rtrace-sort lists allocations sorted by address.
rtrace-stats shows allocation sizes or counts.

Further information

For more information on the tools, see the following links: