MeeGo 1.2 Harmattan Developer Documentation Develop for the Nokia N9

Using sp-rtrace-visualize

The sp-rtrace-visualize package contains visualisation tools and scripts for getting statistics and visualisations out of the sp-rtrace and functracer trace files.


source: sp-rtrace

binary: sp-rtrace, sp-rtrace-visualize

Installing sp-rtrace-visualize on the Harmattan device

Install sp-rtrace-visualize through the developer mode applet.

Prerequisite: Developer mode must be enabled.

  1. Select Settings > Security > Developer mode.
  2. Install the Tracing bundle package by clicking Install.
  3. You get a notification screen that lists all the applications to be installed in the bundle package. To install, click OK.
  4. A dependency notice appears. Click Accept.

For more information on developer mode and installable tools, see Activating developer mode.


By convention, the names of binary tools start with sp-rtrace- and the names of scripts with the rtrace- prefix. To list them, enter one of the following commands and press TAB:

  • sp-rtrace
  • rtrace

The following tools are available:

  • sp-rtrace-allocmap produces ASCII graphs of how memory is fragmented. Has its own input format. Obsoleted by sp-rtrace-pagemap.
  • sp-rtrace-pagemap produces ASCII graphs for all the writable memory mappings the traced process has, what kind of memory page types they have and how many of these pages are utilised by the traced (non-freed) allocations.
  • sp-rtrace-timeline provides the following:
    • allocation timelines:
      • non-freed allocation total + allocation overhead
      • allocation and free counts
      • resource lifetimes
    • histograms:
      • allocation size compared to freed and non-freed allocation count
      • allocation size compared to freed and non-freed allocation amount

The tools accept only ASCII format sp-rtrace trace files.

The following scripts are available:

  • rtrace-allocmap and rtrace-allocmap-animate are helper scripts for converting sp-rtrace output to suitable format and to show it with sp-rtrace-allocmap. Obsoleted by sp-rtrace-pagemap.
  • rtrace-calltree is used by rtrace-graph-* scripts. By default, it gives a call graph of allocation totals.
  • rtrace-from-function lists the amount of allocations from the given function.
  • rtrace-graphs-function creates call graphs of allocations going through the given function.
  • rtrace-graphs-overview creates the following:
    • overview allocation or deallocation timeline and allocation count or size histogram charts (using sp-rtrace-timeline).
    • call graphs to resolved functions which are allocating non-freed resources (using rtrace-calltree) with (hopefully) irrelevant functions removed from the graph to make it readable
  • rtrace-timeline: Python version of sp-rtrace-timeline. Obsolete.

The rtrace-graphs-* scripts accept both binary and ASCII sp-rtrace trace files and complain if the ASCII files have been unsuitably filtered. Other scripts accept only ASCII format trace files.

Using the tool

To start with rtrace-graphs-overview, enter the following command:

 rtrace-graphs-overview binary.rtrace ascii.rtrace.txt

The resulting graphs give a high-level overview of the number and size of the allocations and where they are done from (click to enlarge):

The script also leaves you the post-processed trace data which you can use when running sp-rtrace-timeline and rtrace-calltree directly. The rtrace-graphs-* scripts show with which options they run the other tools, so you can just copy-paste and modify them.

Histograms and timelines

Resource allocation and especially free counts in the histogram can be interesting because they can indicate potentially redundant operations, especially if there are a lot of them (for example, 10x or more) compared to the non-freed resources.

To see the histogram details better, give sp-rtrace-timeline a suitable --scalex option value to scale the histogram graphs horizontally.

To see timeline details better, give sp-rtrace-timeline suitable --filter-size (allocation size range) and --filter-size (allocation time range) option values.

In addition to histogram and timeline graph above, sp-rtrace-timeline can generate also following two additional graphs (click to enlarge):

Call graphs

The call graph produced by rtrace-graphs-overview gives you a good overview of where resources, especially the non-freed resources, go and why.

It contains nodes for functions through which the allocation is done. The arrows indicate which functions call other functions.

The number and amount of allocations done are propagated upwards in the call graphs so that you can directly see from which functions the allocations were initiated. Note that the graph shows allocations over the whole trace, not during a single function call. The individual function call paths can differ, for example, according to the arguments given to the functions.

The nodes through which largest amount of allocations are done are outlined with red so that you can easily see where in the graph are the call paths of main allocations. Terminating nodes, the allocation chain start and end points, are shaped as grey diamonds so that they are easier to find in large call graphs. The nodes that performed large amounts of allocations by themselves also have a grey background colour.

The values in the nodes indicate the allocations done by the given node and any functions it calls. The format of that is <1>% (<2> / <3>), where:

<1>= percentage of all allocations + their estimated allocation overhead (8 bytes on average) from the total in the graph.

<2>= KB of allocations done (without allocation overhead).

<3>= Number of allocations.

To see more details about allocations done through specific functions, you can first try the rtrace-graphs-function script and if that is not enough, run rtrace-calltree directly with suitable options (for example, --limit=X, --ignore=Y, --include-only=Z). With a too large --limit=X option, you may miss important details and it may seem like the numbers do not add up.

Another reason why results do not add up properly is that backtraces are too short. For most reliable call graphs, it is best to have deep enough backtraces (-b option value) so that the backtraces reach the program main().

Viewing the graphs

As the resulting graphs can be large, you need a viewer that supports arbitrary zooming levels.

For example, the following viewers are suitable for each graph output format:

  • dot: xdot, see XDot documentation
  • SVG: rsvg-view from the librsvg2-bin package
  • PS: xpdf (newer Desktop PDF and PS viewers allow maximum zoom of 400 per cent)

File names and other traced function arguments

To get call graphs to opened files (their names), you need to have a trace taken with the sp-rtrace/functracer file module.

Then add an additional --show-args=yes option to the rtrace-calltree command. This adds the first traced function arguments as nodes to the call graph. Without --show-args=yes you would only see names of the functions where the files were opened.

Call graphs of freed resources

To get call graphs to freed resources, you need to post-process the trace file without giving the -l option with sp-rtrace-postproc. Then run rtrace-calltree directly on the resolved results as rtrace-graphs-overview always asks the post-processor to remove freed allocations from the trace.

In call graphs showing also freed resources, the number of allocations (last number given in the function node information) is typically the most interesting piece of information as it gives you the best hint on how many times the potentially redundant operations were done. The total size of all freed allocations can be interesting for analysing what causes temporary memory usage spikes.

To ignore the allocation sizes so that you see the counts better, ask rtrace-calltree for the --type=count call graph.

Traces with multiple resources and/or contexts

If trace data contains multiple resource types or process has switched tracing context (for example, to differentiate allocations coming from different clients) and you want to get information only on a specific context and/or resource, you need to re-process the trace data first with sp-rtrace-postproc -C and/or -R options.

Most of the sp-rtrace-timeline graphs differentiate resources and contexts automatically so this is only needed for call graphs.

Finding very small leaks

There can be small leaks that do not show up in Valgrind Massif output. These leaks can also be referenced to, so Valgrind Memcheck does not notice them. These small leaks cause memory fragmentation which can increase process memory usage. You can use sp-rtrace (and functracer) to find them.

Finding these very small memory leaks requires a large number of repeated test cases. It is possible to automate this with sp-endurance. Basically follow these steps:

1. Start the program with sp-rtrace.

2. Repeat the use case a couple of times without tracing to fill the program's intermediate buffers and caches.

3. Trace one extra repeat of the use case with sp-rtrace.

4. Trace several more (for example, four) repeats of the use case with sp-rtrace.

5. Finally, in case the above does not clearly show which unfreed allocations are growing, trace a larger number of use case repeats, for example, sixteen.

Before the repeats, toggle the tracing ON, and after the repeats, toggle the tracing OFF again to save the data. Enter the following command:

  sp-rtrace -t <PID>

When using functracer, tracing is toggled just by attaching to the process and then interrupting functracer with ^C.

If the use case is handling only a small number of allocations, which means that tracing does not slow it significantly, you can just take the traces such as above and then compare the non-freed allocation counts in the call graphs produced by the rtrace-calltree --type=count option. Leaking call path is the one where the non-freed allocation counts clearly increase.

Slow use cases with a large number of allocations

If the leaking use case has a lot of allocation activity, taking full traces with long backtraces can slow programs too much for them to be usable. Because these kind of small leaks typically occur with specific objects with a fixed size, you can start by finding out the size of objects the leaks occur with. This can be done by omitting the backtraces with the -b 0 sp-rtrace (or functracer) option which makes tracing much faster.

After this, check what is the size of the non-freed allocations counts that clearly increase from the allocation histogram. This can also be done with the rtrace-stats script.

After you know the size of the allocations that leak, repeat the traces with full backtraces and with sp-rtrace -M <size1,size2,...> option to which you enter the leaking allocation sizes. Tracing will be slower with backtrace collection, but collecting them just for the leaky allocation sizes speeds it up and the program hopefully remains usable under tracing.

Then get allocation count call graphs out of the new traces to see which call paths caused increasing numbers of non-freed allocations. Enter the following command:

rtrace-calltree --type=count ...

Memory fragmentation analysis

Even though the application does not have a large amount of allocations (for example, according to sp-rtrace or Valgrind Massif), the process heap can still be large (for example, according to pmap -x PID). The reason for this can be memory fragmentation. To get information about heap fragmentation, in addition to a memory trace you need the sp-rtrace "pagemap" functionality.

If the program is run as a normal user, first you need to make sure as the root user that sp-rtrace can read the required information. Enter the following commands:

 # chmod 644 /proc/kpageflags
 # su - user

You need to know if the memory is fragmented, not what causes it. Backtraces can be disabled with the -b 0 option to make tracing much faster.

1. Catch memory allocation (sizes and addresses) and process pagemap information with sp-rtrace. Enter the following command:

 $ sp-rtrace -s -e pagemap:memory -b 0 -x /usr/bin/call-history

2. After the application is up, toggle tracing off to get the pagemap-maps-0 and pagemap-pageflags-0 files in addition to the allocation *.rtrace trace file. Enter the following command:

 $ sp-rtrace -t $(pidof call-history)

All of the resulting files are non-zero sized.

3. To convert the resulting trace file to the ASCII format, enter the following command:

 $ sp-rtrace-postproc -clr < *.rtrace > call-history.txt

4. To get information on what kind of mappings are in your heap (dirty, swapped and so forth), enter the following command:

 $ sp-rtrace-pagemap -i call-history.txt -p -s -N heap

5. To get allocation utilisation information, enter the following command:

 $ sp-rtrace-pagemap -i call-history.txt -d -N heap

If allocation utilisation chart shows a large number (several rows) of pages that have only little or no non-freed allocations, you have a memory fragmentation issue. If there is a larger amount of completely free pages followed by allocations close to the heap end, those allocations prevent the heap from being shrunk to return the freed memory back to the system.

The reason for such allocations can be either a small memory leak, or the order in which the allocations are done. Permanent allocations are done either before large (numbers of smaller) temporary allocations are done, or after they have all been freed.

After you have determined that there indeed is a memory fragmentation issue, you need to take a new trace with sp-rtrace using large enough backtrace depth (for example, -b 32). Then you can use the -T option to get backtraces to count topmost allocations in heap. Enter the following command::

 sp-rtrace-pagemap -i call-history.txt -d -N heap -T 3

Another possibility is using the --type=topmost option for rtrace-calltree for getting call graphs to allocations left in heap that have the highest addresses.

If there are allocations that do not go to the heap (memory grows without getting back in other mappings than [heap] according to pmap -x <PID>), leave out the -N heap option from sp-rtrace-pagemap invocations.

Further information

For more information on the tools, see the following links: