Memory Debugging

Segmentation faults and other memory bugs (reading uninitialized memory, reading/writing beyond the bounds of an array, memory leaks, etc.) can be hard to track down with a traditional debugger. Memory errors can be elusive, and may not cause the program to crash immediately. A program with memory errors may even appear to work correctly on some datasets or on some machines.

We recommend using a special debugger to find memory errors, for example Dr. Memory or Valgrind. Commercial versions of these tools include Purify and Insure++.

We'll discuss the Dr. Memory and Valgrind memory debugging tools and the memory error reports these tools produce at the end of Lecture 7 and during Lab 5. You'll be expected to use one of these tools for debugging starting with Homework 3. The homework submission server and the TAs will use these tools for grading your homework.

Dr. Memory

Dr. Memory is available for GNU/Linux, Microsoft Windows, and MacOSX operating systems. For questions, bug reports, and discussion, use the Dr. Memory Users group: http://groups.google.com/group/drmemory-users

Please report issues with Dr. Memory to the Dr. Memory Users Group by email: drmemory-users@googlegroups.com. Be sure to include details about your operating system and the Dr. Memory version number. Don't send your full homework submission (it is a public mailing list).

Dr. Memory on GNU/Linux or Windows Subsystem for Linux (WSL)

1. Obtain the Linux 2.0 Release Candidate 1 Dr. Memory tar.gz file. Type these commands into your terminal:

      cd
   
      wget https://github.com/DynamoRIO/drmemory/releases/download/release_2.0.0_rc2/DrMemory-Linux-2.0.0-RC2.tar.gz
   
      tar -xvzf DrMemory-Linux-2.0.0-RC2.tar.gz
   

2. Build your application with debug information by using the -g option. For example:

      g++ -g main.cpp foo_main.cpp foo_other.cpp -o foo.out
   

3. Run your program under Dr. Memory, replacing foo.out arg1 arg2 with your executable name and any command line arguments:

       ~/DrMemory-Linux-2.0.0-RC2/bin64/drmemory -brief -- foo.out arg1 arg2
   

4. Dr. Memory will report errors to the screen as it runs. It will print a summary at the end of what it found.

5. Optional Helpful Edits to .bashrc for WSL
   Add Dr. Memory to your PATH by typing this at the WSL bash prompt:

       echo 'PATH=$PATH:~/DrMemory-Linux-2.0.0-RC2/bin64' >> ~/.bashrc
   
       source ~/.bashrc
   

   NOTE: Do not attempt to edit WSL system files from Windows. The permissions and filesystem will get messed up and you'll need to reinstall everything to recover.

   After adding Dr. Memory to your PATH, you can simply type:

       drmemory -brief -- foo.out arg1 arg2
   

Dr. Memory on Mac OSX

1. Obtain the Dr. Memory tar.gz file for your operating system from https://github.com/DynamoRIO/drmemory/wiki/Downloads

2. Save and untar the package to a directory of your choice. Type something like:

  tar -xvzf DrMemory-YourOperatingSystem-VersionXX.tar.gz

We'll assume DrMemory is in the directory "~/DrMemory-YourOperatingSystem-VersionXX/" for the rest of these instructions.

3. Build your application as 32-bit by passing -m32 to your C++ compiler (Dr. Memory does not yet support 64-bit applications on MacOSX, but you can run 32-bit applications on a 64-bit operating system). Be sure to include debug information by using the -g option. For example:

  g++ -g -m32 main.cpp foo_main.cpp foo_other.cpp -o foo.out

4. Run your program under Dr. Memory, replacing foo.out arg1 arg2 with your executable name and any command line arguments:

  ~/DrMemory-YourOperatingSystem-VersionXX/bin/drmemory -brief -- foo.out arg1 arg2

5. Dr. Memory will report errors to the screen as it runs. It will print a summary at the end of what it found.

Installing Dr. Memory on Windows (not Windows Subsystem for Linux (WSL))

1. Obtain Dr. Memory. To easily place it on the system path, use the installer (the .msi file). Alternatively, you can instead obtain the .zip file for a local install.
   https://github.com/DynamoRIO/drmemory/wiki/Downloads

2. Double click on the .msi file to run the installer. Click Next.

   Check the box to accept the license and click Next.

   The default location for Dr. Memory installation is fine (it's probably C:\Program Files (x86)\Dr. Memory\ for 64-bit Windows). Click Next.

   Then click Install. You'll be asked to confirm that you want to make administrative changes to the machine.

   After a quick installation, press Finish.

3. Follow the instructions below to compile & run your program using MinGW g++, the Visual Studio IDE, or the Visual Studio Command Prompt.

Dr. Memory and MinGW

Dr. Memory does not support programs that use the Cygwin emulation layer. Thus, we cannot use Dr. Memory with programs built using the Cygwin version of g++. Instead, we will build our program with the MinGW g++ compiler (Minimalist GNU for Windows):

1. If you haven't already done so, make sure to install the MinGW g++ compiler.
   See Cygwin Installation Instructions.

2. Open a Cygwin terminal, navigate to the directory with your files, and compile your program with the MinGW compiler by typing:

i686-w64-mingw32-g++.exe --static -ggdb -o foo.exe foo_main.cpp foo_other.cpp

If you've made the shortcut suggested under Helpful edits to the Cygwin .bashrc file, you can equivalently type:

  memg++ -o foo.exe foo_main.cpp foo_other.cpp

This makes a program called foo.exe, using source files foo_main.cpp and foo_other.cpp . The below example assumed that your program is called foo.exe and takes two arguments arg1 and arg2. If you don't know what arguments are, your program probably doesn't take any.

3. You can run your program under Dr Memory by typing:

  drmemory -brief -batch -- foo.exe arg1 arg2

Replace "foo.exe arg1 arg2" with your program name and any command line arguments for your program.

Dr. Memory and Visual Studio

Alternatively, you can use Dr. Memory with the Microsoft Visual Studio compiler:

1. Build your application as 32-bit with Visual Studio (32-bit is the default). Be sure to include debug information. You can verify that you are including debug information by looking at the properties of your build target:

   Press Alt-F7 to bring up the configuration properties. Under "Configuration Properties | C/C++ | General", the "Debug Information Format" entry should either say "Program Database (/Zi)" or "Program Database for Edit and Continue (/ZI)". Additionally, under "Configuration Properties | Linker | Debugging", the "Generate Debug Info" entry should say "Yes (/DEBUG)". For Visual Studio 2015, under "Configuration Properties | Linker | Debugging", the "Generate Debug Info" entry should say "Optimize for debugging (/DEBUG)" -- it should not say "Optimize for faster linking (/DEBUG:FASTLINK)".

2. Disable Runtime Checks: The Visual Studio compiler's /RTC1 flag can prevent Dr. Memory from reporting uninitialized reads of local variables, and the /RTC1 checks for uninitialized reads themselves may not catch everything that Dr. Memory finds. However, /RTC1 does perform additional stack checks that Dr. Memory does not, so for best results, your application should be run under Dr. Memory without /RTC1, and run natively (for development & testing without Dr. Memory) with /RTC1.

   In the Visual Studio IDE, press Alt-F7 and then under "Configuration Properties | C/C++ | Code Generation" ensure "Basic Runtime Checks" says "Default".

3. The most recent Dr. Memory installer (for version 1.8 and later) configures Dr. Memory as a Visual Studio "External Tool", which adds a new menu item allowing you to run Dr. Memory within the IDE.

   Now you can select the "Tools | Dr. Memory" menu item and Visual Studio will run your application under Dr. Memory. You can add arguments to your application in the box that pops up immediately after selecting the men item by adding them at the end, after "$(TargetPath)".

4. The output of Dr. Memory (along with your program) will be printed to the Visual Studio Output Window. Dr. Memory will report errors to the screen as it runs. It will print a summary at the end of what it found. You can double-click on a source file on any error's callstack frame in order to automatically open up that file to the line number indicated.

Using the Visual Studio compiler without the Visual Studio Integrated Development Environment (IDE)

1. Launch the Visual Studio Command Prompt. From the Start menu, under All Programs, find your Visual Studio version (e.g., 2010) and expand it. Then expand Visual Studio Tools. Select the "Visual Studio 2010 Command Prompt". (You don't want the x64 or Cross Tools versions.) Note: this is not the Cygwin shell.

   This Command Prompt is a cmd shell in which a batch file that comes with Visual Studio has been executed. This batch file is called vcvars.bat and it sets up the path and environment variables needed to run the compiler from the command line.

   Note: You can extract the environment variables from the batch file and set them up in your .bashrc so you can build from a Cygwin shell.

2. At the command line, change to the directory containing your source files.

3. Run the compiler, which is called "cl". This will build hw.exe from all .cpp files in the current directory:

cl /Zi /MT /EHsc /Oy- /Ob0 /Fehw.exe *.cpp

4. If you installed Dr. Memory before you opened the Command Prompt, you can run drmemory from the same prompt. Run this command, replacing foo.exe arg1 arg2 with your executable name and any command line arguments:

  drmemory -brief -batch -- foo.exe arg1 arg2

If you don't see any extra output from Dr. Memory as your program runs, remove the -batch flag and the Dr. Memory output will be sent to a file and notepad will launch automatically to display this file.

  drmemory -brief -- foo.exe arg1 arg2

5. Dr. Memory will print a summary at the end of what errors it found.

Valgrind

Valgrind only works on Unix-based systems (e.g., GNU/Linux, FreeBSD, and MacOSX). Valgrind does not work on Cygwin because Cygwin emulates UNIX at the library layer, but Valgrind operates at the system call layer and the Windows system calls are significantly different than UNIX system calls. Note: Valgrind on the more recent Mac OSX versions 10.8 & 10.9 is still a work in progress -- you will likely see false positive memory errors, but it may still be a helpful tool in debugging.

To use Valgrind...

1. Valgrind is installed by default on most Linux distributions. For MacOSX you'll need to install it yourself -- you may want to try Homebrew, a package manager for Mac OSX. Once Homebrew is setup, you can just type:

  brew install valgrind

2. Your program should be compiled with debug information enabled by specifying the -g flag. For example:

  g++ -g main.cpp foo_main.cpp foo_other.cpp -o foo.out 

3. Then run the program by adding Valgrind to the beginning of your command line (replace foo.out arg1 arg2 with your program name and any command line arguments for your program):

  valgrind --leak-check=full --show-reachable=yes ./foo.out arg1 arg2

If that example run of your program contains any memory errors Valgrind will output information to help you track down the error. Note that using Valgrind can significantly slow down execution time as it inspects every memory action. You may need to craft a smaller test case that exhibits the same bug you would like to solve.

Note: Because some STL classes (including string) use their own allocators (and do other optimization tricks), there may be a warning about memory that is ``still reachable'' even though you've deleted all your dynamically allocated memory. The newer versions of Valgrind automatically suppresses some of these common false positive errors, so you may see this listed as a ``suppressed leak''.

Suppression of False Positives in Valgrind

If you see false positive error messages in Valgrind (this is likely to happen with Valgrind on the newest versions of Mac OSX), you will probably want to create an error suppression file to allow you to focus on your actual errors.

1. Add the --gen-suppressions=all option to the valgrind command line:

  valgrind --leak-check=full --gen-suppressions=all ./foo.out arg1 arg2

2. For each false positive (an error not obviously pointing at your code), copy-paste the suppression text (a block of text in curly braces) into a new file containing your custom suppressions, let's call it my_suppressions.txt.

3. Use that suppression file every time you run Valgrind:

  valgrind --leak-check=full --suppressions=my_suppressions.txt ./foo.out arg1 arg2

4. You may need to add to that file in the future, when you use additional library functions that cause different false positive errors.

Read more about Valgrind suppressions here:
http://valgrind.org/docs/manual/manual-core.html#manual-core.suppress

ASAN

ASAN (Address Sanitizer) is a memory debugger developed by Google, and is built into recent versions of C/C++ compilers. Note: It currently only works on unix-based systems. ASAN does not currently work in Windows (including the WSL) due to differences in memory allocations.

Unlike the other memory debuggers, which require you to rerun your code through the memory debugger, ASAN just requires you to recompile it with two extra flags, and then rerun it as usual. Because ASAN is built-in to the compiler, it tends to be much faster than the other debuggers. ASAN also will immediately abort at the first error (i.e. invalid read/write) unlike the other debuggers, which will continue running if possible. ASAN's method tends to be a bit more helpful, since you only have one error to focus on, but it doesn't always give you the full picture that you may need. Additionally, unlike the other memory debuggers, ASAN doesn't print anything if it detects no errors. The other debuggers will always end up printing some output when you run them, but since ASAN is built into your program, there's no additional output when there are no errors. If you run a program after compiling it with ASAN, and you see no extra output, that doesn't mean that ASAN isn't working - it means that you don't have any memory errors!

On OS X and Linux, you should already have g++ and/or clang already installed, which is all you need. To compile your program with ASAN, add the -g, -fsanitize=address, and -fno-omit-frame-pointer flags like so:

	g++ -g -fsanitize=address -fno-omit-frame-pointer main.cpp foo_main.cpp foo_other.cpp -o foo.out

You can now just rerun your program as usual (i.e. ./foo.out arg1 arg2) and it will run with memory debugging.

Optional helpful edits for ASAN

If you don't wish to have to remember the compile flags, you can add the following alias to your shell rc like so:

	echo 'alias memg++="g++ -g -fsanitize=address -fno-omit-frame-pointer"' >> ~/.bashrc && source ~/.bashrc

With this, you can then compile programs for ASAN by just specifying memg++ instead of g++ or clang. For example, the following two commands are equivalent:

	g++ -g -fsanitize=address -fno-omit-frame-pointer main.cpp foo_main.cpp foo_other.cpp -o foo.out

	memg++ main.cpp foo_main.cpp foo_other.cpp -o foo.out

Additionally, ASAN tends to be a bit verbose, with its output, so unless you understand what shadow memory is, it's recommended that you add the following ASAN runtime options:

	echo 'export ASAN_OPTIONS=print_legend=0' >> ~/.bashrc && source ~/.bashrc

If you encounter a memory error, you'll see some un-suppressable output that indicates what part of memory the error is occuring. You'll generally want to look at the output above it that includes line information about where objects were allocated, freed, and accessed from, instead.