SourceForge.net Logo memP: Parallel Heap Profiling

memP: Parallel Heap Profiling

Version 1.0.3
11 April 2010
Chris Chambreau
chcham@llnl.gov

Contents

Introduction

memP is a parallel heap profiling library based on the mpiP MPI profiling tool. The intent of memP is to identify the heap allocation that causes a task to reach its memory in use high water mark (HWM) for each task in a parallel job. Currently, memP requires that all tasks call MPI_Init and MPI_Finalize.

There are two types of memP reports:

  1. Summary Report : Generated from within MPI_Finalize, this report describes the memory HWM of each task over the run of the application. This can be used to determine which task allocates the most memory and how this compares to the memory of other tasks.
  2. Task Report : Based on specific criteria, a report can be generated that provides a snapshot of the heap memory currently in use, including the amount allocated at specific call sites.

A reasonable approach to using memP would involve 2 application runs: one run to determine the HWM and task of interest and another run using the memP run-time criteria to generate task-specific reports to get more detail about how memory is allocated.

Report output is generated in Tool Gear-compatible XML and is intended to be examined with the Tool Gear viewer.

On LLNL platforms, this viewer can be found at /usr/local/tools/toolgear/Toolgear_2.00/bin/TGui.

Downloading

You may download the current version of memP from http://sourceforge.net/projects/memp.

Top

Using memP

Using memP requires either using a special run-time script or re-linking with the memP library. Note that you might have to recompile to include the '-g' option. This is important if you want memP to decode the PC to a source code filename and line number automatically. memP will work without -g, but mileage may vary.

The memP library can be inserted at run-time on LLNL Linux systems by using the srun-memp script in place of launching a job with srun.

The memP library can be linked with an application as in the following example:

-L/usr/local/tools/memP/lib -lmemP -ldl

By default, the output file is written to the current directory of the application.

Top

Run-time Configuration of memP

memP has several configurable parameters that a user can set via the environment variable MEMP. The setting for MEMP looks like command-line parameters: "-g -k 2". Currently, memP has several configurable parameters.

Option Description Default
-a Abort when threshold reached (see -h option).  
-d Summary Report : Print detailed call site list.  
-e exe Specify the full path to the executable.  
-f dir Record output file in directory <dir>. .
-g[#] Enable memP debug mode with number indicating debugging level. disabled
-h n Task Report : HWM Threshold.  
-i n Task Report : Maximum number of reports to generate.  
-j n Summary & Task Report : Only report on the specified MPI_COMM_WORLD rank.  
-k n Sets callsite stack traceback depth to <n>. 8
-n Do not truncate full pathname of filename in callsites.  
-o Disable profiling at initialization. Application must enable profiling with MPI_Pcontrol().  
-p n Sets the number of HWM task entries to list in the report. If -1, list all. 1
-s n Set hash table size to <n>. 256
-t Generate stack trace data. disabled
-x Generate XML report.  

For example, to active debugging messages set MEMP to "-g". To have task MPI_COMM_WORLD rank 8 generate HWM reports for the first 10 HWM allocations greater than 1024 bytes:

$ export MEMP="-h 1024 -i 10 -j 8"

memP prints a message at initialization if it successfully finds this MEMP variable.

Top

memP Output

Report output is generated in Tool Gear-compatible XML and is intended to be examined with the Tool Gear viewer.

On LLNL platforms, this viewer can be found at /usr/local/tools/toolgear/Toolgear_2.00/bin/TGui.

An example of viewing an memP report with Tool Gear:

/usr/local/tools/toolgear/Toolgear_2.00/bin/TGui alloc.2.8140.1.memP

Report files have the following naming convention:

  1. Summary Report : [executable].[task count].[pid].[index].memP
  2. Task Report : [executable].rank-[task rank].[pid].[HWM].memP

Top

License

Copyright (c) 2010, Lawrence Livermore National Security, LLC.
Produced at the Lawrence Livermore National Laboratory
Written by Christopher Chambreau (chcham@llnl.gov).
CODE-OCEC-10-010
All rights reserved.

This file is part of memP. For details, see http://memp.sourceforge.net .
Please also read this link – Additional BSD Notice.

Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:

  o  Redistributions of source code must retain the above copyright notice, 
     this list of conditions and the disclaimer below.

  o  Redistributions in binary form must reproduce the above copyright notice, 
     this list of conditions and the disclaimer (as noted below) in the 
     documentation and/or other materials provided with the distribution.

  o  Neither the name of the LLNS/LLNL nor the names of its contributors may be
     used to endorse or promote products derived from this software without 
     specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL LAWRENCE LIVERMORE NATIONAL
SECURITY, LLC, THE U.S. DEPARTMENT OF ENERGY OR CONTRIBUTORS BE
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
THE POSSIBILITY OF SUCH DAMAGE.


Additional BSD Notice

1. This notice is required to be provided under our contract with the U.S.
Department of Energy (DOE). This work was produced at Lawrence Livermore
National Laboratory under Contract No. DE-AC52-07NA27344 with the DOE.

2. Neither the United States Government nor Lawrence Livermore National
Security, LLC nor any of their employees, makes any warranty, express or
implied, or assumes any liability or responsibility for the accuracy,
completeness, or usefulness of any information, apparatus, product, or process
disclosed, or represents that its use would not infringe privately-owned rights.

3. Also, reference herein to any specific commercial products, process, or
services by trade name, trademark, manufacturer or otherwise does not
necessarily constitute or imply its endorsement, recommendation, or favoring by
the United States Government or Lawrence Livermore National Security, LLC.
The views and opinions of authors expressed herein do not necessarily state or
reflect those of the United States Government or Lawrence Livermore National
Security, LLC, and shall not be used for advertising or product endorsement
purposes.

Top

For further information please send mail to chcham@llnl.gov.

Last modified on April 11th, 2010.
CODE-OCEC-10-010