MeeGo 1.2 Harmattan Developer Documentation Develop for the Nokia N9

Using OProfile

OProfile is a system-wide profiler for Linux systems, capable of profiling all running code at low overhead. OProfile is released under the GNU General Public License. OProfile consists of a kernel driver and a daemon for collecting sample data, and several post-profiling tools for turning data into information.

OProfile leverages the hardware performance counters of the CPU to enable profiling of a wide variety of interesting statistics. The statistics can also be used for basic time-spent profiling to find CPU usage bottlenecks in the whole system, and within processes. All code is profiled: hardware and software interrupt handlers, kernel modules, the kernel, shared libraries, and applications.

OProfile is currently in alpha status. However, it has been proven as stable on a large number of differing configurations.

Use OProfile to find the bottleneck processes and functions in applications. You can then use Valgrind or Callgrind on the device to get callgraph data for your application, and Kcachegrind on Linux to view callgraphs to see why a piece of code ends in those bottleneck functions. This is due to the following:

  • OProfile provides system-wide information (important in case the usage of a system daemon is a bottleneck) whereas Valgrind profiles only the given process.
  • As OProfile profiles the whole system, its callgraphs' depth cannot be very deep for performance reasons. Valgrind-provided callgraphs are more informative and readable.

Valgrind is the recommended tool to, for example, find leaks, causes for crashes, and racy data accesses. For more information, see Using Valgrind.


source: oprofile

binary: oprofile

Installing OProfile on the Harmattan device

Install OProfile through the developer mode applet.

Prerequisite: Developer mode must be enabled.

  1. Select Settings > Security > Developer mode.
  2. Install the Performance bundle package by clicking Install.
  3. You get a notification screen that lists all the applications to be installed in the bundle package. To install, click OK.
  4. A dependency notice appears. Click Accept.

For more information on developer mode and installable tools, see Activating developer mode.

Installing debug symbols

In order to view any useful profiling information at functions level, you have to install debugging symbols also. Debug symbols normally come with debugging (-dbg) packages. The easiest way to install all dbg packages required for a given binary is to use debug-dep-install script which comes with the maemo-debug-scripts package. Install maemo-debug-scripts with the developer mode applet. For more information, see maemo-debug-scripts.

Debug symbols packages must match the binary package. You can install them after profiling, if you see that the report does not give symbols for all the libraries and binaries you are interested in.

Using the tool

You can use OProfile to collect system statistics and profiling information of the system.

Collecting statistics

1. On the device, enter the following commands:

opcontrol --deinit
opcontrol --no-vmlinux
opcontrol --separate=kernel
opcontrol --init

In the same way as the --separate=library option, the --separate=kernel option separates the collected statistics for each process and their components. In most use cases, processes (implicitly) request other processes, such as X server and hildon-desktop, to do work for them. To optimise the CPU usage, you need to see which processes need to use most CPU and in which of its components (binary or libraries) in the whole system. The --separate=kernel option also assigns CPU usage within kernel under the processes that caused it. The vmlinux binary name is used for this part.

2. Reset the collected statistics if you have profiled something else before (in any case it does not do any harm). Enter the following command:

opcontrol --reset

3. Start the use case you are interested in and enter the following command:

opcontrol --start

4. When you have finished, enter the following command:

opcontrol --stop

Now you have collected the data.

Before looking into the results, see the Interpreting profiling results documentation.

Viewing profile reports

To see a basic report for each process, enter the following command:

Overflow stats not available
CPU: CPU with timer interrupt, speed 0 MHz (estimated)
Profiling through timer interrupt
  samples|     -%|
    65956 77.0217 functracer                    <- this process percentage of total CPU usage
          samples|     -%|
            58174 88.2012 no-vmlinux            <- this process kernel load (percentage of above 77%)
             4626  7.0138
             1815  2.7518
             1180  1.7891 functracer
               74  0.1122
               55  0.0834
               32  0.0485
    12368 14.4430 no-vmlinux                    <- kernel activity under no process = IDLE
     4952  5.7828 xrestop
          samples|     -%|
             4383 88.5097 no-vmlinux
              190  3.8368
              164  3.3118
              150  3.0291
               30  0.6058 anon (tgid:8373 range:0x3aac9000-0x3aaca000)
               22  0.4443 xrestop
               10  0.2019
                3  0.0606
     1719  2.0074 Xorg
          samples|     -%|
             1176 68.4119 no-vmlinux
              467 27.1670 Xorg
               49  2.8505
               20  1.1635
                5  0.2909
                2  0.1163
      308  0.3597 busybox

After you know the processes and components with the highest CPU usage, you need to find out the bottleneck functions or functionality in them. For this, you need to install debug symbols for them.

Note: If there is a lot of CPU activity for kernel that is not assigned under any process with the --separate=kernel option, it means that the system or kernel is idle. If your use case is (unexpectedly) slow despite system idling a lot, these kinds of issues are usually related to locking and other inter-process interaction issues that cannot be analysed by looking at the CPU usage. You may be able to analyse the issue with strace.

To see more detailed symbol analysis, enter the following command:

 opreport -l
Overflow stats not available
warning: /no-vmlinux could not be found.
CPU: CPU with timer interrupt, speed 0 MHz (estimated)
Profiling through timer interrupt
samples -%        image name               app name                 symbol name
58174    68.1027  no-vmlinux               functracer               /no-vmlinux
12368    14.4789  no-vmlinux               no-vmlinux               /no-vmlinux         <- kernel activity under no app/process = IDLE
4383      5.1311  no-vmlinux               xrestop                  /no-vmlinux
1815      2.1248   functracer               /usr/lib/
1362      1.5945           functracer               strncpy
1176      1.3767  no-vmlinux               Xorg                     /no-vmlinux
527       0.6169           functracer               memmove
467       0.5467  Xorg                     Xorg                     /usr/bin/Xorg
398       0.4659           functracer               vfprintf
314       0.3676           functracer               ptrace
198       0.2318  no-vmlinux               busybox                  /no-vmlinux
176       0.2060           functracer               memcpy

In the example above, the function names are shown for C-library calls from functracer, but not for functionalities with the highest usage inside the functracer, the xrestop and Xorg processes and the libunwind library, so additional debug symbols should be installed and opreport -l run again. Note that anything with less than a few hundreds samples is not very reliable statistically.

Once you know which functionality is a bottleneck, you need to find out whether your program is (indirectly) causing the use of that functionality in the first place, is it using it too much or too often, or should the bottleneck functionality itself be optimised. Analysis of this falls to the developers of the tested program as only they know what their program is trying to achieve, why and how. Before this kind of analysis, it is too early to assign or to report bugs for the lower-level components.

Profiling kernel

If you want to profile the kernel, you need to specify the vmlinux file corresponding to your kernel. Enter the following command:

 opcontrol --vmlinux=/usr/lib/debug/vmlinux-<version>

This file contains both the kernel binary and its debug symbols, it comes from the kernel-debug package. The opreport command can then show, for example, the kernel function names in the report output.

To get the result with callgraph and kernel modules symbols, enter the following command:

 opreport -l -c -p /lib/modules/<version>/ > opreport.txt

Using opannotate

See Using opannotate to make sense of profiles.

Other profiling events supported by OProfile

Although OProfile supports several different methods for profiling, different ones work in different products. Default is set to the one that works.

For more information, see OProfile documentation.

Further information

For more information on the tools, see the following links: