When writing C programs many of the bugs are memory related. They may result in strange behaviour which can arouse far away from the erroneous position.
I once spent about thirty hours hunting for an annoying bug. The setting
was bad. In a big project where a program (for which we don't had the
source) read in libraries (written by us) at runtime and then used them
a segmentation fault occured my part when allocating memory. I had
just written some arcane stuff concerning binary file i/o and wasn't
surprised that much. Of course it was not possible to use a debugger
and we had to switch to good old printf
and visual trace
(step the code in wetware memory line by line) debugging. But I did
not find nothing, everything seemed wonderful -- except that the program
still kept crashing.
It was a weekend and I hadn't had Internet access that time (it's seems so long ago that I remember dinosaurs stamping by my window). So I wrote the first version of the malloc debug library to see what happened. I linked it to my library and -- did not find anything. So at last (it was a weekend) I linked it to the libraries of my coworkers and indeed there was a off-by-one error in someone else's library. We love C, don't we?
Since then I always use it in my projects from the very beginning and do only switch it off for release versions. It's an unrenouncable improvement to find memory bugs as early as possible without doing any more extra work than including a header and setting a command line switch in the Makefile.
It's a simple thing. The malloc debug library implements wrappers for the normal heap handling functions:
malloc
calloc
realloc
free
strdup
getcwd
(this is thought as an example how to wrap other library functions
returning malloc'd memory)
When allocating memory the wrapper functions demand some more memory from the system and use this extra space to write some special bytefields before and after the buffer they finally return to the user. Some of the extra space before the buffer is used for extra information e.g. about the file position where the allocation took place. When freeing the buffer (and maybe more often) the bytefields are controlled whether they are unchanged. If there is a changed the program is aborted immediately with an error message indicating the position where the erroneous buffer was allocated and where the error was detected. E.g. a common error:
<MALLOC_DEBUG> Corrupted block end (possibly written past the end) should be: a5a5a5a5 5b5b5b5b abababab aa55aa55 is: 00a5a5a5 5b5b5b5b abababab aa55aa55 block was allocated in rtest.c:181 [4 Bytes, generation 1] error was detected in rtest.c:185 Looks like string allocated one byte too short (missing the closing zero)
The abort allows you to switch on your debugger and inspect things more closely.
The archive rmdebug.tgz is a gzipped tar archive, has about 15k and contains the files:
REMINDER:
You may download/use/change/redistribute this
utility as you like but you do this at your own risk without any
warranty.
Get rmalloc.c and rmalloc.h from downloading
rmalloc.tgz from above. Compile rmalloc.c to rmalloc.o
and include rmalloc.h in each of your files MALLOC_DEBUG
defined and link rmalloc.o to it.
That's all to use it.
First follow the Installation Guide.
If your sources are compiled with MALLOC_DEBUG
the calls
to the standard malloc library are exchanged by the debug malloc library
functions. If you don't define MALLOC_DEBUG
the calls stay
as they are. Be sure that you have included rmalloc.h everywhere!
Also be sure that you recompile everything after adding or removing
MALLOC_DEBUG
in your Makefile.
The package includes a Makefile and a file rtest.c. Just say make and you can run rtest which should give you some output similar to this:
------------------ Running test 0... ------------------ <MALLOC_DEBUG> rmalloc -- malloc wrapper V 1.21 by Rammi <mailto:rammi@hexco.de> Compiled with following options: testing: only actual block generations: ON eloquence: OFF realloc(0): NOT ALLOWED free(0): NOT ALLOWED flags: USED alignment: 8 pre space: 32 post space: 16 hash tab size: 257 <MALLOC_STATS> ============ STATISTICS (rtest.c:90) ============= <MALLOC_STATS> 1000 x 8 Bytes in rtest.c:84, generations: 1 2 3 ... <MALLOC_STATS> *Variable* 8000 Bytes <MALLOC_STATS> *Static* 0 Bytes <MALLOC_STATS> *Total* 8000 Bytes <MALLOC_STATS> ============ END OF STATISTICS ============= <MALLOC_STATS> ============ STATISTICS (rtest.c:96) ============= <MALLOC_STATS> 1000 x 8 Bytes in rtest.c:84, generations: 1 2 3 ... <MALLOC_STATS> 20 x 8 Bytes in rtest.c:93, generations: 1001 1002 1003 ... <MALLOC_STATS> *Variable* 8160 Bytes <MALLOC_STATS> *Static* 0 Bytes <MALLOC_STATS> *Total* 8160 Bytes <MALLOC_STATS> ============ END OF STATISTICS ============= <MALLOC_STATS> ============ STATISTICS (rtest.c:105) ============= <MALLOC_STATS> 20 x 8 Bytes in rtest.c:93, generations: 1001 1002 1003 ... <MALLOC_STATS> 1000 x 16 Bytes in rtest.c:100, generations: 1021 1022 1023 ... <MALLOC_STATS> *Variable* 16160 Bytes <MALLOC_STATS> *Static* 0 Bytes <MALLOC_STATS> *Total* 16160 Bytes <MALLOC_STATS> ============ END OF STATISTICS ============= <MALLOC_DEBUG> Corrupted block end (possibly written past the end) should be: a5a5a5a5 5b5b5b5b abababab aa55aa55 is: a5a5a53f 5b5b5b5b abababab aa55aa55 block was allocated in rtest.c:100 [16 Bytes, generation 2020] error was detected in rtest.c:119 ------------------ Running test 1... ------------------ <MALLOC_DEBUG> Corrupted block end (possibly written past the end) should be: a5a5a5a5 5b5b5b5b abababab aa55aa55 is: 00a5a5a5 5b5b5b5b abababab aa55aa55 block was allocated in rtest.c:144 [3 Bytes, generation 1] error was detected in rtest.c:146 Looks like string allocated one byte too short (forgetting the nul byte) ------------------ Running test 2... ------------------ <MALLOC_DEBUG> Corrupted block end (possibly written past the end) should be: a5a5a5a5 5b5b5b5b abababab aa55aa55 is: 326c6f6e 67005b5b abababab aa55aa55 block was allocated in rtest.c:163 [2 Bytes, generation 1] error was detected in rtest.c:165 Looks somewhat like a too long string, ending with "2long" ------------------ Running test 3... ------------------ <MALLOC_DEBUG> Corrupted block end (possibly written past the end) should be: a5a5a5a5 5b5b5b5b abababab aa55aa55 is: e8530508 5b5b5b5b abababab aa55aa55 block was allocated in rtest.c:183 [4 Bytes, generation 1] error was detected in rtest.c:187 First 4 bytes of overwritten memory can be interpreted as a pointer to a block allocated in: rtest.c:184 [8 Bytes, generation 2] ------------------ Running test 4... ------------------ <MALLOC_DEBUG> Double or false delete Heap adress of block: 0x80560f0 Detected in rtest.c:242 Trying identification (may be incorrect!): Allocated in rtest.c:227 [2 Bytes] ------------------ Running test 5... ------------------ <MALLOC_DEBUG> Double or false delete Heap adress of block: 0x12345678 Detected in rtest.c:260 ------------------ Running test 6... ------------------ <MALLOC_DEBUG> WARNING: calloc() overflow! Returning NULL (in rtest.c:276) <MALLOC_DEBUG> Trying to free NULL pointer (in rtest.c:277) ------------------------------ All tests passed successfully. ------------------------------
You can tweak the behaviour of the library in two ways:
You can change the overall behaviour by setting different switches in rmalloc.c. Doing this has the advantage that you only have to recompile and relink rmalloc.c without touching anything else.
Set this to one of the following three values: 0, 1, or 2.
With 0 only a minimum type of malloc debugging is included. Each allocated block is only tested when you free it, there is no statistics available. Even if this uses the least memory overhead I never use it.
With 1 you will get a statistic of memory in use when you leave your program or when you use a special macro in your code. Each block is only tested automatically when you free it. You can test every block by using special macros. For me this is the optimum of performance deterioration versus available information. This is the default.
2 is the same as 1 with the addition that every allocated block will be checked on every access of any of the malloc functions. This will really slow down your program. I sometimes use this when I cannot get a grip on an error and want to reduce the place to look for the error fast.
Introduced in version 1.15.
This switch is based on an enhancement and code from
The statistics of used blocks will print some of the numbers of blocks
allocated at a given file position which is not always useful if the code
uses generic allocation functions. In that case you can set the environment
variable BREAK_GENERATION to one of these numbers, start your
preferred debugger and set a breakpoint to the function
rmalloc_generation()
. This function will be called exactly
when the block with the number you're interested in is allocated
so you can use the callstack feature of you're debugger to see where
the leaking allocation occured.
In the default configuration this feature is switched on.
If defined every action of the library (allocating, freeing) will be outputtet. This can really get a lot. When I have a Double or false delete error I pipe the output into a file and look if (and where) the given pointer is already freed before.
Thanks to
If defined only errors are printed, the startup message and the final
statistic are not printed.
Defining this allows the setting of special flags for every allocated block. I always use it but it needs extra heap space. See RM_SET section below for details.
Define this to allow realloc(NULL, ...)
.
To achieve high portability of my programs I don't allow
this in my programs.
Define this if you don't consider free(NULL)
an error.
I have this undefined because I use NULL
always as a special
value.
Only used if GENERATIONS is set.
This defines the condition to be fulfilled so the debug breakpoint
rmalloc_generation()
function is called.
The given default (if generation
is the number given in the BREAK_GENERATION environment variable)
should work in most cases, see code for more possibilities.
Only used if GENERATIONS is set.
This defines the maximum number of generations printed for a given file position/size combination in the statistics. Normally only the 1st should be of interest, so the default of 3 is quite enough.
To use this macros you have to make changes to your code. I agree that this
is ugly but you will get something for it. All these macros expand to nothing
when MALLOC_DEBUG
is undefined.
Invokes a complete test of all allocated blocks. It has only effect when
RM_TEST_DEPTH
is at least set to 1.
Example:
RM_TEST; /* tests ALL memory chunks on heap */
Invokes a complete test of all allocated blocks. Prints out a statistic
of allocated blocks to stderr
. It has only effect when
RM_TEST_DEPTH
is at least set to 1.
Example:
RM_STAT; /* shows allocated memory */
Sometimes you have a functions which gets a chunk of memory, initializes
it to some default values and returns this to the user. For the malloc
debug library only the place of the allocation counts which makes it
difficult to track down memory leaks because all the structures allocated
in a function like that will have the same file position. With this macro
you can set the file position to the position where the macro was invoked.
Example:
struct complicated *cpointer = get_new_complicated_struct(any_arg); RM_RETAG(cpointer); /* file pos is now set to here */
or, since version 1.16 even simpler:
/* RM_RETAG will set file pos to the next line */ struct complicated *cpointer = RM_RETAG(get_new_complicated_struct(any_arg));
Packing everything into a macro makes it even easier:
/* RM_RETAG will set file pos to the line where this macro is invoked */ #define GET_NEW_COMPLICATED_STRUCT(arg) \ RM_RETAG(get_new_complicated_struct(arg)) /* ... later ... */ struct complicated *cpointer = GET_NEW_COMPLICATED_STRUCT(any_arg);
If you have memory leak problems with a setting similar to the one given above and know how to use a debugger Karl's generations feature may be more useful for a quick shot.
Set a special flag for the allocated memory. You have to define
WITH_FLAGS
in rmalloc.c or this macro will
have no effect. For the moment there are two flags defined:
static char *buffer = NULL; static int length = 0; [...] if (newlength > length) { if (buffer == NULL) { buffer = malloc(length = newlength); RM_SET(buffer, RM_STATIC); /* this is never freed */ } else { buffer = realloc(buffer, length = newlength); /* RM_SET(buffer, RM_STATIC); not necessary because flags are kept! */ } if (buffer == NULL) { error(NOMEM); } }
ELOQUENT
mode.
strdup
sets this flag automatically. Example:
struct numstr { int number; char *name; }; [...] struct numstr *foo = malloc(sizeof(struct numstr)); foo->name = calloc(1, 2); foo->name[0] = 'X'; /* sheer nonsense */ RM_SET(foo->name, RM_STRING);
This library was used on several platforms (at least Irix, Solaris, HPUX, Digital Unix, and Linux) by different users. It can help you find common errors like (moderate) overwriting of memory bounds or memory leaks. No problem using it with debuggers or other tools with different implementations of the standard malloc library because it internally uses it.
It is by no means perfect. Writing to dangling pointers may or may not
be found. Fat overwrite (more than 16 bytes) may cause a crash of the
library itself. Mixing the library with calls to the standard malloc
library (because you don't include rmalloc.h everywhere or have
compiled parts of your program with MALLOC_DEBUG
defined and
others not) will cause problems (e.g. when you use standard free with
rmalloc'ed stuff and vice versa).
This is especially true when you use other library functions returning
memory from heap than the wrapped ones. You will get a Double or
false delete error when you free that memory. It is simple to write a
wrapper yourself for that functons. Just look how I handled the
getcwd
library function.
Attention! The code was
broken in version 1.13 which was fixed in 1.14.
Despite of the warnings above I use it always (except for releases) and haven't had these problems. But of course I know what I am doing.
WARNING: Please note that I wrote this section in 1996 and have never updated it.
There are lots of other libraries which do a similar thing (it's a common problem). If you have already used another library and you are content with it, there is possibly no need to switch. If this is your first encounter with something like this you should give it a try.
I haven't had much need to use other malloc debug libraries, of course. I used a very early release of Purify and I have tested electric fence some time ago.
I liked Purify a lot but it is incredible expensive and makes your program slow because every memory access is monitored. So it is probably only used if you really know that you have problems. So if the problems don't show during test phase it's not used. But as we all know the problem will show when we present our program to the customer.
If your boss listens to you tell him to buy Purify anyway (it's not your money and you will be glad to have it when you need it). But until then you should use something else.
Electric fence is freeware, too, but it uses a different concept compared to my stuff. It is programming your machine's MMU to protect the space behind the allocated blocks. If you access that space you will immediately get a segmentation fault. This is neat because you are really lead to the place where you actually write over the bounds. It will even react on reading beyond bounds. And it is fast because it uses hardware. But it uses more heap space (which tends to make it slower on big programs) and it doesn't find all errors when writing over the bounds because of alignment problems. Especially the problem of Test 1 of rtest is not found.
Some info on the various releases. Check here if you use older versions to see whether it is useful to update (of course it nearly always should). The most recent version appears at the top. Not all versions are described here because I was too lazy to start this section from the beginning.
Because I was asked to clarify the license of rmalloc from October 8 2010 it is coming under the ISC license (the main difference to the relaxed state before is that you'll have to keep the license text after modifications, which before was not required; otherwise you are free to do what you like):
Copyright © 2010 Andreas M. Rammelt <rammi@hexco.de> Permission to use, copy, modify, and/or distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies. THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
If you need further information feel free to ask me.