![\"gdb-1-590x396\"](\"https:\/\/blog.anuram.in\/wp-content\/uploads\/2013\/02\/gdb-1-590x396.jpg\")
<\/a><\/p>\nThis method is not nearly as powerful as some of the other options, but it has one big advantage: you can get a core file from a running daemon without killing it and restarting. It just interrupts the daemon for as long as it takes to write out the core file.<\/p>\n
cd \/path\/to\/space\/for\/core\/file\ngdb -p gcore\nquit<\/pre>\nNow you have a core file. You can load it up in gdb and walk around in data structures to see if some of them are larger than expected. You can also run ‘strings’ on it to see if anything unexpected jumps out. For example, if a particular string is repeated many more times than it should be, that could be a clue to help you find a leak or source of bloat.<\/p>\n
strings corefile | sort | uniq -c | sort -n<\/pre>\nHow to use valgrind to check a daemon for memory leaks<\/h2>\n
Valgrind is a powerful tool for catching memory leaks and sources of bloat. The main disadvantage of the tool is that it slows things down quite a lot (~20 times in my experience). You also need to stop the running daemon and restart it under valgrind.<\/p>\n
Here is an example of how to use it. In this example, it is the collector that is being checked.<\/p>\n
condor_off -collector\n# become the user that runs HTCondor\nsudo su root\nenv _CONDOR_USE_CLONE_TO_CREATE_PROCESSES=False valgrind --tool=memcheck --leak-check=yes --show-reachable=yes --leak-resolution=high >& \/tmp\/valgrind.log < \/dev\/null \/path\/to\/condor_collector -f -p 9621 &<\/pre>\nNote that valgrind (as of 2.4.0) crashes due to the way we use clone(), so the above example disables the clone() optimization. As of 7.1.2, HTCondor should auto-detect that it is running under valgrind and automatically disable the clone optimization.<\/p>\n
To check for leaks, run for a while and then do a graceful shutdown (kill -TERM). To check for bloat, kill with SIGINT instead. This will prevent it from doing a normal exit, freeing up memory, etc. That way, we can see memory that it has referenced (but which may be unexpectedly bloated). The valgrind output will contain information on the blocks of memory that were allocated at exit.<\/p>\n
valgrind gives more detailed information when you run it with an unstripped binary (including line numbers in the call stack).<\/p>\n
Running the test suite under valgrind<\/h2>\n
batch_test.pl<\/span> \u00a0is the means by which the test suite is run. It can be told to start up its own person HTCondor and use that for testing. The tests ran by\u00a0batch_test.pl<\/span>\u00a0may also start their own personal condors.<\/p>\n
In order to test all of these personal condors under valgrind, valgrind can be told to follow all forked\/execed children and where to dump its output. So we’ll do just that and start up the master. You can either be root or a normal user for this, depending upon what you are testing.<\/p>\n
After the test suite codes are built, cd into\u00a0condor_tests\/<\/span>.<\/p>\n
We’ll invoke\u00a0batch_test.pl<\/span>\u00a0under valgrind. We tell batch test to start up its own personal condors and clean up everything when it is done. We don’t ask it to run tests in parallel since there likely are race conditions in the tests over that behavior.<\/p>\n
We make sure to specify the logfile pattern in a valid and empty directory. This is because valgrind will write a new log file with the pid associated with it for\u00a0every<\/strong>\u00a0program executed under batch_test.pl. You can put them into \/tmp like I have here, but I would recommend you put them into a special directory since there will be LOTS of log files.<\/p>\n
valgrind --tool=memcheck --leak-check=yes --show-reachable=yes --leak-resolution=high\n--log-file=\/tmp\/tests.%p --trace-children=yes --num-callers=16 .\/batch_test.pl -b -c<\/pre>\nThen, in the same directory as the pile of\u00a0tests.*<\/span>\u00a0files, run this script, which will collate the\u00a0tests.*<\/span>\u00a0files associated with HTCondor daemons and tools into directories.<\/p>\n
#! \/usr\/bin\/env perl\n\nuse warnings;\nuse strict;\nuse File::Basename;\nuse File::Path;\n\nsub collate_valgrind_logs\n{\nmy $prefix = shift @_;\nmy @files;\nmy $file;\nmy @lines;\nmy $line;\nmy @pieces;\n\n@files = `\/bin\/ls -1 $prefix*`;\nmap {chomp $_} @files;\n\n# foreach file which has a pattern of ^$prefix[0-9]+$ (some corefiles might\n# exist which could have that prefix which we don't want to process) we\n# figure out the command associated with it\nforeach $file (@files) {\nnext if ($file !~ \/^Q$prefixE[0-9]+$\/);\n\n# Get the line out of the file which represents the command that\n# the file is valgrinding.\n@lines = `egrep -e '== Command:' $file`;\nnext if ((scalar @lines) == 0);\nmap {chomp $_} @lines;\n\n# figure out the actual command\n@pieces = split \/ \/, $lines[0];\nmap {s\/^s+\/\/; chomp $_} @pieces;\n\n# make sure I have something to work with.\nnext if (!defined($pieces[2]));\n\n# get rid of an easy set of commands that aren't HTCondor processes.\nnext if ($pieces[2] !~ m:condor_w+$:);\n\n# get rid of any HTCondor process that appears to be a test suite\n# executable\nnext if ($pieces[2] =~ m:condor_exec:);\n\n# get the basename of the file\n$pieces[2] = basename($pieces[2]);\n\n# Now, mkdir the basename and copy the file into it.\nmkpath($pieces[2]);\nif (-d $pieces[2]) {\nsystem(\"cp $file $pieces[2]\");\n}\n\nprint \"$file, $pieces[2]n\";\n\n}\n}\n\nsub main\n{\ncollate_valgrind_logs(\"tests.\");\n}\n\nexit main();<\/pre>\nHow to use the google heap profiler<\/h2>\n
The google heap profiler is part of the google-perftools package, which may be easily downloaded and compiled. This example used version 0.98.<\/p>\n
The heap profiler is useful for watching heap usage over time. Although the package documentation recommends linking the application with the tcmalloc library, I have had success with the LD_PRELOAD mechanism.<\/p>\n
This example shows how I used the google heap profiler on the startd:<\/p>\n
# from the central manager\ncondor_off -startd\n# back on the startd node\n# become the user who runs HTCondor\nksu\nenv LD_PRELOAD=\/p\/condor\/workspaces\/danb\/google-perftools-0.98-rh5\/lib\/libtcmalloc.so HEAPPROFILE=\/tmp\/startd.hprof HEAP_PROFILE_ALLOCATION_INTERVAL=5000000 \/unsup\/condor\/sbin\/condor_startd -f >& \/tmp\/startd.hprof.log < \/dev\/null &<\/pre>\nEvery 5M of heap allocation will result in a new file \/tmp\/startd.hprof.000x.heap<\/span>. These files can be analyzed using the pprof tool from the google-perftools. In addition to looking at individual snapshots, you can look at the difference between two files. One nice way to look at the output is with kcachegrind, the callgrind GUI. Example:<\/p>\n
pprof --callgrind \/path\/to\/condor_startd \/tmp\/startd.hprof.0001.heap > \/tmp\/startd.0001.callgrind\nkcachegrind \/tmp\/startd.0001.callgrind<\/pre>\nThen you can look at the call tree and see which paths allocated lots of heap, hopefully leading you to discover the source of trouble. If you configure kcachegrind to know where the source code is, you can even pull up a panel showing where in the code the allocations happened.<\/p>\n
One thing to beware of is that any code compiled with -fomit-frame-pointer may be untracked by the heap profiler. For example, openssl is usually compiled this way and I have found that the pprof –text report shows some things allocated in libcrypt code to show up as allocations in default_malloc_ex() but the pprof –callgrind report does not show this memory allocation at all. By turning off the compiler option when compiling openssl, I was able to get things to work as expected. This same issue may affect other memory profiling tools too.<\/p>\n
How to use igprof for memory profiling<\/h2>\n
Igprof (the ignominous profiler,\u00a0http:\/\/igprof.sf.net<\/a>) is another convenient tool for tracking CPU performance and memory. RHEL5 RPMs are available here:<\/p>\n