valgrind/README_DEVELOPERS

Building and installing it
~~~~~~~~~~~~~~~~~~~~~~~~~~
To build/install from the GIT repository or from a distribution
tarball, refer to the section with the same name in README.

Building Valgrind requires autoconf, GNU make and a suitable C
compiler.


Building and not installing it
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To run Valgrind without having to install it, run coregrind/valgrind
with the VALGRIND_LIB environment variable set, where <dir> is the root
of the source tree (and must be an absolute path).  Eg:

  VALGRIND_LIB=~/grind/head4/.in_place ~/grind/head4/coregrind/valgrind 

This allows you to compile and run with "make" instead of "make install",
saving you time.

Or, you can use the 'vg-in-place' script which does that for you.

I recommend compiling with "make --quiet" to further reduce the amount of
output spewed out during compilation, letting you actually see any errors,
warnings, etc.


Building a distribution tarball
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To build a distribution tarball from the valgrind sources:

  make dist

In addition to compiling, linking and packaging everything up, the command
will also attempt to build the documentation.

If you only want to test whether the generated tarball is complete and runs
regression tests successfully, building documentation is not needed.

  make dist BUILD_DOCS=none

If you insist on building documentation some embarrassing instructions
can be found in docs/README.


Running the regression tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Running the regression tests requires GNU sed, python 3.9 or later, gdb,
and a suitable C++ compiler

To build and run all the regression tests, run "make [--quiet] regtest".

To run a subset of the regression tests, execute:

  tests/vg_regtest <name>

where <name> is a directory (all tests within will be run) or a single
.vgtest test file, or the name of a program which has a like-named .vgtest
file.  Eg:

  tests/vg_regtest memcheck
  tests/vg_regtest memcheck/tests/badfree.vgtest
  tests/vg_regtest memcheck/tests/badfree

The details of each vgtest run are logged to individual "vgtest.log"
files.  These are listed, and non-passing tests detailed, in the
test-suite-overall.log file.  (Token *.trs and test-suite.log files
are also created, for emulating automake-style testsuites, as expected
by tools such as bunsen.)


Platform-specific setup for regression tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On some platforms some setup is required otherwise some of the tests
will fail.

Inherited inotifier file descriptors
------------------------------------
Seen on openSUSE amd64 KDE, all processes seem to inherit an inotify
file descriptor which interferes with some of the fdleak tests in the
none directory. Here is an example

lsof | grep inotify | grep bash
bash      13128                      paulf   31r  a_inode               0,14         0         46 inotify

I don't know how to turn this off completely. You can close the
file descriptor for the current shell by using

MY_INOTIFIER=$(lsof 2>&1 | grep $$ | grep inotify | awk '{print $4}' | sed 's/r//')

if [ -n ${MY_INOTIFIER} ] ; then
   exec {MY_INOTIFIER}<&-
fi

FreeBSD kernel modules and stack guard page
-------------------------------------------
The sctp and mqueuefs kernel modules need to be loaded. As root

kldload sctp mqueuefs

Additionally FreeBSD has strange belt and braces thread stack protection
with both the kernel and the allocated thread stack providing guard pages.
This interferes with the memcheck descr_belowsp test. The extra kernel
guard page can be turned off with the following command run as root

sysctl security.bsd.stack_guard_page=0


Running the performance tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To build and run all the performance tests, run "make [--quiet] perf".

To run a subset of the performance suite, execute:

  perf/vg_perf <name>

where <name> is a directory (all tests within will be run) or a single
.vgperf test file, or the name of a program which has a like-named .vgperf
file.  Eg:

  perf/vg_perf perf/
  perf/vg_perf perf/bz2.vgperf
  perf/vg_perf perf/bz2

To compare multiple versions of Valgrind, use the --vg= option multiple
times.  For example, if you have two Valgrinds next to each other, one in
trunk1/ and one in trunk2/, from within either trunk1/ or trunk2/ do this to
compare them on all the performance tests:

  perf/vg_perf --vg=../trunk1 --vg=../trunk2 perf/

Writing regression tests
~~~~~~~~~~~~~~~~~~~~~~~~

Each tool has a tests directory containing regression tests. There is also
the gdbserver_tests directory at the top level. Test directories may have
architecture, OS and and architecture-OS sub-directories for tests that are
specific to one architecture, OS or both.

Once you have written a C or C++ executable that performs the required
tests you will have to modify and create several files. The new files that
will need adding are
(a) a .vgtest file, which controls the test
(b) a .stderr.exp file, which is the golden reference for any Valgrind output
    Note that even if Valgrind doesn't produce any output you will need to
    create an empty .stderr.exp file.
(c) [optional] a .stdout.exp file, the golden reference for any output from the
    test executable
(d) [optional] filter files (see the test directories for examples).

Quite often the output will depend on the platform the test is run on.
Getting the tests to be 'portable' can require filtering or adding multiple
.stderr.exp reference files.

If the test only runs under certain conditions like the availability of functions
in libc or C++ standard versions you will need to modify configure.ac in the
top level directory. See AC_CHECK_FUNCS and the various blocks starting with
AC_MSG_CHECKING.

In the test directory, modify Makefile.am. Add to EXTRA_DIST the .vgtest,
.stderr.exp and .stderr.out files. Add any filters to dist_noinst_SCRIPTS.
Add the test executable name to check_PROGRAMS. Try to respect the
formatting and alphabetical ordering of the Makefile.am. For simple C files that is
sufficient. If you needed to add a feature test to configure.ac then you should
use the same condition to add the executable name to check_PROGRAMS. If the
executable uses C++ you need to add exename_SOURCES. If the executable needs
special compilation or link options, use exename_CFLAGS, exename_CXXFLAGS,
exename_LDFLAGS or exename_LDADD. Finally in Makefile.am it's nice not to
have any warnings, even if they were done on purpose. See configure.ac
and various Makefile.am files for examples of using FLAG_W_*.

The vgtest file contains the instructions for running the test. Typically
it will contain (with examples in quotes)
(a) the name of the test executable:      "prog: exename"
(b) arguments for the test executable:    "args: hello world"
(c) arguments for the Valgrind tool:      "vgopts: -q"
(d) [optional] a check for prerequisites: "prereq: ! ../../tests/os_test darwin"
(e) [optional] a filter:                  "stderr_filter: filter_fdleak"

See tests/vg_regtest for a full description of all possible vgtest directives.

The easiest way to generate the expected files is to run the test. Create empty
files with touch (otherwise the test won't run) then run the test from the
top directory using vg_regtest script (as in the "Running the
regression tests" section. Then copy "tool/tests/newtest.stderr.out" to
"tool/tests/newtest.stderr.exp". It is better to generate the .stdout.exp
file directly from the testcase. You can do that by redirecting stdout to
the expected file, for instance (in the test directory)
    ./newtest arg1 arg2 > newtest.stdout.exp
This is not always possible - sometimes there are tests that depend on the use
of client requests or have imperfect emulation of opcodes.
Make sure that the test runs and passes.

The last file to change is .gitignore in the top directory. Add a new entry,
for example "/tool/tests/newtest".

The 'scalar' tests are something of a special case. Scalar in this sense
refers to the registers (or stack slots) used to pass in arguments. These tests
directly use the 'syscall' syscall via a macro, SY. They make little effort
to use the sysall in a realistic manner. Rather, the objective is to
exhaustively test all of the arguemnts and referenced memory of syscalls.
The variable 'x0' is a long integer, containing the value of 0 but
also uninitialised. It can be used on its own or with some other value
to ensure that all of the syscall arguemts are uninitialised.
A second macro, GO, is used precede the syscall (and subsequent errors)
with a header. The GO string includes the name of the syscall, a count of
expected scalar errors and a count of memory errors. The tests are usually
followed by the FAIL macro, which ensures that the syscall failed.
An example scalar test is

   /* SYS_link                    9 */
   GO(SYS_link, "2s 2m");
   SY(SYS_link, x0, x0); FAIL;


This syscall takes two strings so the expected errors are
2 scalar (for the pointer arguments) and 2 memory (for
the strings themselves).

When your test is done check for mistakes in Makefile.am.
In the top directory run

    make post-regtest-checks

You should only see
    ...checking makefile consistency
    ...checking header files and include directives
and no messages related to EXTRA_DIST.

Commit access and try branches
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To get commit access to the valgrind git repository on sourceware
you will have to ask an existing developer and fill in the following
form: https://sourceware.org/cgi-bin/pdw/ps_form.cgi

Every developer with commit access can use try branches. If you want to try a
branch before pushing you can push to a special named try branch as follows: 

  git push origin $BRANCH:users/$USERNAME/try-$BRANCH

Where $BRANCH is the branch name and $USERNAME is your user name.

You can see the status of the builders here:
https://builder.sourceware.org/buildbot/#/builders?tags=valgrind-try

The buildbot will also sent the patch author multiple success/failure emails.

Afterwards you can delete the branch again:

    git push origin :users/$USERNAME/try-$BRANCH


Debugging Valgrind with GDB
~~~~~~~~~~~~~~~~~~~~~~~~~~~
To debug the valgrind launcher program (<prefix>/bin/valgrind) just
run it under gdb in the normal way.

Debugging the main body of the valgrind code (and/or the code for
a particular tool) requires a bit more trickery but can be achieved
without too much problem by following these steps:

(1) Set VALGRIND_LAUNCHER to point to the valgrind executable.  Eg:

      export VALGRIND_LAUNCHER=/usr/local/bin/valgrind

    or for an uninstalled version in a source directory $DIR:

      export VALGRIND_LAUNCHER=$DIR/coregrind/valgrind
      export VALGRIND_LIB=$DIR/.in_place

    VALGRIND_LIB is where the default.supp and vgpreload_ libraries
    are found (which is under /usr/local/libexec/valgrind for an installed
    version).

(2) Run gdb on the tool executable.  Eg:

      gdb /usr/local/libexec/valgrind/lackey-ppc32-linux

    or

      gdb $DIR/.in_place/memcheck-x86-linux

(3) Do "handle SIGSEGV SIGILL nostop noprint" in GDB to prevent GDB from
    stopping on a SIGSEGV or SIGILL:

    (gdb) handle SIGILL SIGSEGV nostop noprint

    If you are using lldb, then the equivalent command is

    (lldb) pro hand -p true -s false -n false SIGILL SIGSEGV

(4) Set any breakpoints you want and proceed as normal for gdb. The
    macro VG_(FUNC) is expanded to vgPlain_FUNC, so If you want to set
    a breakpoint VG_(do_exec_inner), you could do like this in GDB:

    (gdb) b vgPlain_do_exec_inner

    Note: This is just an example, for various reasons internal
          function names might be renamed or optimized out
          (for example when building with --enable-lto).

(5) Run the tool with required options (the --tool option is required
    for correct setup), e.g.

    (gdb) run --tool=lackey pwd

Steps (1)--(3) can be put in a .gdbinit file, but any directory names must
be fully expanded (ie. not an environment variable).

A different and possibly easier way is as follows:

(1) Run Valgrind as normal, but add the flag --wait-for-gdb=yes.  This
    puts the tool executable into a wait loop soon after it gains
    control.  This delays startup for a few seconds.

(2) In a different shell, do "gdb /proc/<pid>/exe <pid>", where
    <pid> you read from the output printed by (1).  This attaches
    GDB to the tool executable, which should be in the above mentioned
    wait loop.

(3) Do "cont" to continue.  After the loop finishes spinning, startup
    will continue as normal.  Note that comment (3) above re passing
    signals applies here too.

The default build of Valgrind uses "-g -O2". This is OK most of the
time, but with sophisticated optimization it can be difficult to
see the contents of variables. A quick way to get to see function
variables is to temporarily add "__attribute__((optnone))" before
the function definition and rebuild. Alternatively modify
Makefile.all.am and remove -O2 from AM_CFLAGS_BASE. That will
require you to reconfigure and rebuild Valgrind.

Self-hosting
~~~~~~~~~~~~
This section explains:
  (A) How to configure Valgrind to run under Valgrind.
      Such a setup is called self hosting, or outer/inner setup.
  (B) How to run Valgrind regression tests in a 'self-hosting' mode,
      e.g. to verify Valgrind has no bugs such as memory leaks.
  (C) How to run Valgrind performance tests in a 'self-hosting' mode,
      to analyse and optimise the performance of Valgrind and its tools.

(A) How to configure Valgrind to run under Valgrind:

(1) Check out 2 trees, "Inner" and "Outer".  Inner runs the app
    directly.  Outer runs Inner.

(2) Configure Inner with --enable-inner and build as usual.

(3) Configure Outer normally and build+install as usual.
    Note: You must use a "make install"-ed valgrind.
    Do *not* use vg-in-place for the Outer valgrind.

(4) Choose a very simple program (date) and try

    outer/.../bin/valgrind --sim-hints=enable-outer --trace-children=yes  \
       --smc-check=all-non-file \
       --run-libc-freeres=no --tool=cachegrind -v \
       inner/.../vg-in-place --vgdb-prefix=./inner --tool=none -v prog

If you omit the --trace-children=yes, you'll only monitor Inner's launcher
program, not its stage2. Outer needs --run-libc-freeres=no, as otherwise
it will try to find and run __libc_freeres in the inner, while libc is not
used by the inner. Inner needs --vgdb-prefix=./inner to avoid inner
gdbserver colliding with outer gdbserver.
Currently, inner does *not* use the client request 
VALGRIND_DISCARD_TRANSLATIONS for the JITted code or the code patched for
translation chaining. So the outer needs --smc-check=all-non-file to
detect the modified code.

Debugging the whole thing might imply to use up to 3 GDB:
  * a GDB attached to the Outer valgrind, allowing
    to examine the state of Outer.
  * a GDB using Outer gdbserver, allowing to
    examine the state of Inner.
  * a GDB using Inner gdbserver, allowing to
    examine the state of prog.

The whole thing is fragile, confusing and slow, but it does work well enough
for you to get some useful performance data.  Inner has most of
its output (ie. those lines beginning with "==<pid>==") prefixed with a '>',
which helps a lot. However, when running regression tests in an Outer/Inner
setup, this prefix causes the reg test diff to fail. Give 
--sim-hints=no-inner-prefix to the Inner to disable the production
of the prefix in the stdout/stderr output of Inner.

The allocators in coregrind/m_mallocfree.c and VEX/priv/main_util.h are
annotated with client requests so Memcheck can be used to find leaks
and use after free in an Inner Valgrind.

The Valgrind "big lock" is annotated with helgrind client requests
so Helgrind and DRD can be used to find race conditions in an Inner
Valgrind.

All this has not been tested much, so don't be surprised if you hit problems.

When using self-hosting with an outer Callgrind tool, use '--pop-on-jump'
(on the outer). Otherwise, Callgrind has much higher memory requirements. 

(B) Regression tests in an outer/inner setup:

 To run all the regression tests with an outer memcheck, do :
   tests/vg_regtest --outer-valgrind=../outer/.../bin/valgrind \
                    --all

 To run a specific regression tests with an outer memcheck, do:
   tests/vg_regtest --outer-valgrind=../outer/.../bin/valgrind \
                    none/tests/args.vgtest

 To run regression tests with another outer tool:
   tests/vg_regtest --outer-valgrind=../outer/.../bin/valgrind \
                    --outer-tool=helgrind --all

 --outer-args allows to give specific arguments to the outer tool,
 replacing the default one provided by vg_regtest.

Note: --outer-valgrind must be a "make install"-ed valgrind.
Do *not* use vg-in-place.

When an outer valgrind runs an inner valgrind, a regression test
produces one additional file <testname>.outer.log which contains the
errors detected by the outer valgrind.  E.g. for an outer memcheck, it
contains the leaks found in the inner, for an outer helgrind or drd,
it contains the detected race conditions.

The file tests/outer_inner.supp contains suppressions for 
the irrelevant or benign errors found in the inner.

A regression test running in the inner (e.g. memcheck/tests/badrw) will
cause the inner to report an error, which is expected and checked
as usual when running the regtests in an outer/inner setup.
However, the outer will often also observe an error, e.g. a jump
using uninitialised data, or a read/write outside the bounds of a heap
block. When the outer reports such an error, it will output the
inner host stacktrace. To this stacktrace, it will append the
stacktrace of the inner guest program. For example, this is an error
reported by the outer when the inner runs the badrw regtest:
  ==8119== Invalid read of size 2
  ==8119==    at 0x7F2EFD7AF: ???
  ==8119==    by 0x7F2C82EAF: ???
  ==8119==    by 0x7F180867F: ???
  ==8119==    by 0x40051D: main (badrw.c:5)
  ==8119==    by 0x7F180867F: ???
  ==8119==    by 0x1BFF: ???
  ==8119==    by 0x3803B7F0: _______VVVVVVVV_appended_inner_guest_stack_VVVVVVVV_______ (m_execontext.c:332)
  ==8119==    by 0x40055C: main (badrw.c:22)
  ==8119==  Address 0x55cd03c is 4 bytes before a block of size 16 alloc'd
  ==8119==    at 0x2804E26D: vgPlain_arena_malloc (m_mallocfree.c:1914)
  ==8119==    by 0x2800BAB4: vgMemCheck_new_block (mc_malloc_wrappers.c:368)
  ==8119==    by 0x2800BC87: vgMemCheck_malloc (mc_malloc_wrappers.c:403)
  ==8119==    by 0x28097EAE: do_client_request (scheduler.c:1861)
  ==8119==    by 0x28097EAE: vgPlain_scheduler (scheduler.c:1425)
  ==8119==    by 0x280A7237: thread_wrapper (syswrap-linux.c:103)
  ==8119==    by 0x280A7237: run_a_thread_NORETURN (syswrap-linux.c:156)
  ==8119==    by 0x3803B7F0: _______VVVVVVVV_appended_inner_guest_stack_VVVVVVVV_______ (m_execontext.c:332)
  ==8119==    by 0x4C294C4: malloc (vg_replace_malloc.c:298)
  ==8119==    by 0x40051D: main (badrw.c:5)
In the above, the first stacktrace starts with the inner host stacktrace,
which in this case is some JITted code. Such code sometimes contains IPs
that points in the inner guest code (0x40051D: main (badrw.c:5)).
After the separator, we have the inner guest stacktrace.
The second stacktrace gives the stacktrace where the heap block that was
overrun was allocated. We see it was allocated by the inner valgrind
in the client arena (first part of the stacktrace). The second part is
the guest stacktrace that did the allocation.


(C) Performance tests in an outer/inner setup:

 To run all the performance tests with an outer cachegrind, do :
    perf/vg_perf --outer-valgrind=../outer/.../bin/valgrind perf

 To run a specific perf test (e.g. bz2) in this setup, do :
    perf/vg_perf --outer-valgrind=../outer/.../bin/valgrind perf/bz2

 To run all the performance tests with an outer callgrind, do :
    perf/vg_perf --outer-valgrind=../outer/.../bin/valgrind \
                 --outer-tool=callgrind perf

Note: --outer-valgrind must be a "make install"-ed valgrind.
Do *not* use vg-in-place.

 To compare the performance of multiple Valgrind versions, do :
    perf/vg_perf --outer-valgrind=../outer/.../bin/valgrind \
      --outer-tool=callgrind \
      --vg=../inner_xxxx --vg=../inner_yyyy perf
  (where inner_xxxx and inner_yyyy are the toplevel directories of
  the versions to compare).
  Cachegrind and cg_diff are particularly handy to obtain a delta
  between the two versions.

When the outer tool is callgrind or cachegrind, the following
output files will be created for each test:
   <outertoolname>.out.<inner_valgrind_dir>.<tt>.<perftestname>.<pid>
   <outertoolname>.outer.log.<inner_valgrind_dir>.<tt>.<perftestname>.<pid>
 (where tt is the two letters abbreviation for the inner tool(s) run).

For example, the command
    perf/vg_perf \
      --outer-valgrind=../outer_trunk/install/bin/valgrind \
      --outer-tool=callgrind \
      --vg=../inner_tchain --vg=../inner_trunk perf/many-loss-records

produces the files
    callgrind.out.inner_tchain.no.many-loss-records.18465
    callgrind.outer.log.inner_tchain.no.many-loss-records.18465
    callgrind.out.inner_tchain.me.many-loss-records.21899
    callgrind.outer.log.inner_tchain.me.many-loss-records.21899
    callgrind.out.inner_trunk.no.many-loss-records.21224
    callgrind.outer.log.inner_trunk.no.many-loss-records.21224
    callgrind.out.inner_trunk.me.many-loss-records.22916
    callgrind.outer.log.inner_trunk.me.many-loss-records.22916


Printing out problematic blocks
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you want to print out a disassembly of a particular block that
causes a crash, do the following.

Try running with "--vex-guest-chase=no --trace-flags=10000000
--trace-notbelow=999999".  This should print one line for each block
translated, and that includes the address.

Then re-run with 999999 changed to the highest bb number shown.
This will print the one line per block, and also will print a
disassembly of the block in which the fault occurred.


Formatting the code with clang-format
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
clang-format is a tool to format C/C++/... code.  The root directory of the
Valgrind tree contains file .clang-format which is a configuration for this tool
and specifies a style for Valgrind.  This gives you an option to use
clang-format to easily format Valgrind code which you are modifying.

The Valgrind codebase is not globally formatted with clang-format.  It means
that you should not use the tool to format a complete file after making changes
in it because that would lead to creating unrelated modifications.

The right approach is to format only updated or new code.  By using an
integration with a text editor, it is possible to reformat arbitrary blocks
of code with a single keystroke.  Refer to the upstream documentation which
describes integration with various editors and IDEs:
https://clang.llvm.org/docs/ClangFormat.html.

Updating zstd
~~~~~~~~~~~~~
Similar to libiberty, we have to import a copy of zstd rather than linking
with a library. There isn't (yet) a script to automate this, so it has to be
done manually.

The version currently in use can be seen in coregrind/m_debuginfo/zstd.h.
Look for ZSTD_VERSION_MAJOR ZSTD_VERSION_MINOR and ZSTD_VERSION_RELEASE.

 - Get the source of zstd from

   https://github.com/facebook/zstd

 - Checkout the latest release tag (should be vMAJ.MIN.REL)

 - Copy {zstd git repo}/lib/zstd.h to coregrind/m_debuginfo/zstd.h
 
 -  cd to {zstd git repo}/build/single_file_libs and run ./create_single_file_decoder.sh
 
 - You cannot simply copy and use the generated zstddeclib.c!
   All calls to libc functions in this file need replacing with VG_ versions.
   Merge the newly generated zstddeclib.c with coregrind/m_debuginfo/zstddeclib.c.
   Make sure to keep the copy of the BSD license in the C file.