Windows is a great environment in which to view system resource usage when you are connected to the actual server console or connected to it remotely. The graphical output that the Performance Monitor tool (PerfMon) supplies is intuitive. However, if you are a third party trying to better understand memory usage or other attributes on a customer's machine, the GUI requirement is a detriment to getting the information you need.
Since the Windows 32-bit environment can be challenging for memory-intensive applications such as Lotus Domino® or a host of other applications, being able to better understand allocations is of prime importance.
At this point, without getting too precise about the versions of Windows Server code and the ability to maximize memory in a Windows process, a process can access a finite amount of memory in 32-bit Windows. (Finite typically being 2 GB, or 3 GB with appropriate switches and software that actually supports these switches.)
To make matters more complex, process memory generally comes from several sources. When a process is exceeding the finite limit imposed by the 32-bit operating system, you may need to break out where these allocations are coming from to find the specific cause. So if you plan on monitoring memory, you need to be able to decipher what a measurement tells you.
The most useful measurements that you can get from PerfMon and other sources are Virtual Bytes and Private Bytes. Virtual Bytes is the total allocation of memory for the individual process in question. This includes memory that is shared by the process, its .dll information (image data), and private memory (related only to the process). This value is ultimately the best indicator of whether a process is in jeopardy of exceeding the finite limits of the process size.
Private Bytes is memory allocated by the process that is in it own private pages. For most simple processes, most memory allocation should occur in this space. In the case of Lotus Domino, for example, nearly all add-in tasks will have a strong mixture of shared and private bytes allocated.
Now that we have some idea of what to look for, the next challenge is how to do this without the PerfMon GUI and perhaps, as well, in a scripted fashion.
In the following example, the beginning of the process name is matched, which can be useful for processes of an application with similar names:
Even better, we can do this remotely, using the computer name switch:
pslist -m \\Hostname -u username [-p password ]
You may need to check your firewall settings and open appropriate ports on the desired host, such ports 135, 445 and 80, for it to work remotely. Also, it requires read access to the PerfLib counters.
NOTE: If this is done on a Domino server, Program documents can be used to run the PsList program locally. However, as a best practice it is better to use operating system tools rather than the Domino scheduler. This avoids any issues of measurement not occurring because of problems on the Domino server, and it provides a methodology for collection on non-Domino products.
Simplifying the Data
Now we need a way to store only the required information from the VM and Priv columns. Scripting tools are a good way to collect only those columns of interest via a process called filtering.
Filtering helps reduce the data set to what is truly useful for analysis. By filtering intelligently you can collect the desired information and reduce noise of extraneous data.
There are many good scripting programs that can be used with Windows, such as Windows Services for UNIX , Uwin, and Cygwin, that include UNIX shell environments or utility kits. This lets shell programmers to use command line tools to manipulate the data stream on the fly.
You could also use common Web programming tools such as Perl, Python, and perhaps JavaTM to work on the data stream. The goal is to produce a filtering command that produces the least amount of required data.
If the desired result is to get only the Name, VM, and Priv columns from the example above, and assuming the data PsList is written into a file called output.txt, then a filter to extract the process Name, VM, and Private columns might look like this:
cat output.txt | awk '{print $1, $3, $5)}' > newoutput.txt
With a little more effort programmatically, a date and time stamp could be added, as well as perhaps some logic to record only the intervals for which the memory actually increases.
Another improvement could be to convert from kilobytes to megabytes, which might create a data file that looks something like this:
| date | time | VM | Private |
|---|
| mm-dd-yy | hh:mm | 500 | 50 |
| mm-dd-yy | hh:mm | 510 | 50 |
|
| mm-dd-yy | hh:mm | 550 | 55 |
The results could then be analyzed graphically to gain further insight.
In Depth Memory Analysis
If tracking the virtual memory and private bytes does not reveal the underlying size of the process adequately, there are two good solutions that map out all components of process memory allocations.
Mapping looks at image data or the memory size of the .dlls or .exe's: Private, which belongs directly to the process; Shared Memory; Mapped Files; Heap; Stack; and System. Typically, the more interesting components to examine are the Shareable, Heap, and Stack. Since the mapping tool is there to go beyond what is stored in private memory, the Shareable, Heap, and Stack are most likely to contain the memory that is causing the problem.
The “Cadillac” of these programs is the SysInternals program, VMMap. The other program uses nsd -perf, which involves a Domino installation. Both programs can be run programmatically to collect process memory information without intervention.
VMMap
The VMMap utility is usually a GUI-based tool. By default it stores the data in an .mmp format that can be re-read by the program. However, it also includes an option to write the data to a .csv or text format, which could then be manipulated programmatically or perhaps in a spreadsheet.
To run at the command line, you must supply a process ID or process name with the -p argument, and specify an output file name. Additionally, to write to something other than an .mmp file, specify a .csv extension or another extension that will be created as a text file.
An example of looking at the Domino router process would be:
vmmap -p Nrouter myoutput.CSV
nsd perf
Using nsd -perf is a relatively straightforward task. Generally, a specific process will be the target, and the user must obtain the PID from Task Manager or via another tool. So, if Nrouter is the target with a pid of 5536, then the command would look like this:
nsd -perf -pid 5536 -noinfo
This creates the smallest data set in the \IBM_TECHNICAL_SUPPORT directory.
The difficulty with both these tools is that the output includes a lot of superfluous detail, not relevant to the decision-making process, in the data they produce for output. Again, this is probably a good place to use scripting or programming languages--or perhaps both in combination--to keep only the desired information from the output.
With some help of a parsing tool, the report shown in figure 1 was generated via Java, showing the breakdown of private, image, mapped, and the total process allocation for the process listed.
Figure 1. Process memory report
This report also takes the private memory analysis a step further by distinguishing whether the memory is committed or reserved. Committed memory is backed by virtual memory; that is, the higher the value of committed memory, the more that is in virtual memory for the process.
Both the NSD's mapping functionality and VMMap can tell you how much memory is committed by the process.
Conclusion
To summarize, the key steps to collecting memory data remotely are to:
Find a tool that has a Command Line Interface, for example:
Build filters to get the desired data.
Use programming to automate and simplify the data set.
Use a graph for analysis, if necessary.
Resources
Microsoft TechNet Windows Sysinternals Process Utilities site:
http://technet.microsoft.com/en-us/sysinternals/bb795533.aspx
developerWorks article, “Don't forget about memory”:
http://www.ibm.com/developerworks/java/library/j-memusage/
psutil: A process utilities module for Python
http://code.google.com/p/psutil/
Uwin site:
http://www2.research.att.com/sw/tools/uwin/
Cygwin site:
http://www.cygwin.com/
Windows Services for UNIX site:
http://www.microsoft.com/downloads/details.aspx?FamilyID=896C9688-601B-44F1-81A4-02878FF11778&displaylang=en
GNU Utilities for Windows
http://unxutils.sourceforge.net/
About the author
Scott Hopper is a certified IT Specialist who specializes in Domino Server and Notes Client issues at IBM. He enjoys working on complex memory issues and methods to collect and simplify data. In his free time he is an avid gardener and enjoys playing and managing a competitive over-40's soccer team.