Introduction

With the broad collection of components (see figure 1) provided in Lotus Expeditor-based products and the many different logging and tracing standards available today, it can be quite challenging to locate critical Problem Determination (PD) data when attempting to diagnose problems.

Figure 1. Federated view of Lotus Expeditor-based logging and tracing



There is no magic bullet for Expeditor-based log analysis, that is, no single place that's guaranteed to provide the exact details needed to resolve every problem; however, the intent of this article is to provide:

• an initial overview of the collection of log files and tracing capabilities offered in Lotus Expeditor-based products
• tips on how to navigate through this information as part of the PD / Problem Source Identification (PD/PSI) process

Back to top

Where to find the log files

This section describes where the log files are located.

System log and trace files

The system log and trace files are located in the user's workspace in the $<workspace>/logs/ directory, for example, "C:\Documents and Settings\Administrator\Application Data\Lotus\XPD\logs" on Microsoft® Windows®.

trace-log-<num>.xml. This is the collection of system trace files, with the most recent numbered zero (see figure 2). As new instances of Lotus Expeditor start, previous trace files rotate to the next higher number. By default the oldest trace file is numbered with the greatest number. Review this file for errors and extraordinary trace events that might indicate problems with an application or the platform.

Figure 2. Trace log viewed with an external browser



error-log-<num>.xml. This is the collection of system error log files, with the most recent numbered zero (see figure 3). As new instances of Lotus Expeditor start, previous error log files rotate to the next higher number. By default, the oldest error log file is numbered with the highest number.

Figure 3. Error log viewed with the Log and Trace Viewer



All entries in the error log are also contained in the trace log. The error log provides high-level, log-only (no trace) messages that should be actionable by users to understand and, in some cases, to resolve problems.

The trace file is meant to be processed by IBM Service/Support teams and provides trace (and log) messages. This trace data is not translated and does not contain globally unique IBM Software Group (SWG) identifiers. Log messages in the trace log will, of course, be translated and contain this data. For initial problem determination the trace file is the one with which to interact; there is no value in looking at the error log since all its data is also stored in the trace file.

NOTE: The persistence format of Lotus Expeditor logs is customizable. By default Lotus Expeditor supports XML (CommonBaseEvent), but it also supports the standard (Java^TM Specification Request) JSR-47 formatters and an HTML formatter as well. Figure 4 shows an example of a trace log produced on a Linux® machine with the HTML formatter.

Figure 4. Example of system trace log using HTML formatting



More details on reviewing the trace file are provided in the "Log interpretation" section below.

Back to top

Install Logs

The product install logs provide critical information about the success or failure of the Lotus Expeditor product installation.

Windows
The Windows install files are stored in the %TEMP% directory on the Windows system:

• rcp_install.log. This is the Microsoft Installer (MSI) installation log file (see figure 5). If an installation fails, the contents of this file identify the specific action that caused the failure.

Figure 5. Example of the Windows MSI installation log

• rcpInstallerTemp.properties. Specifies the temporary installation properties file (see figure 6), which contains the states of several installation properties that are set/read during the install process. If a failure occurs, the states of these properties might prove useful.

Figure 6. Example temporary properties file from Lotus Expeditor install on Windows



Back to top
Linux
The Linux install log is stored in the /tmp directory.

• rcp_install.log. This is the Red Hat Package Manager (RPM) log file (see figure 7). If an installation fails, the contents of this file identify the specific action that caused the failure.

Figure 7. Example of Linux RPM installation log



Back to top
InstallHistory snapshot
As of Lotus Expeditor 6.2.1 FP1, each time the product installer / add-on installer / hotfix installer runs, it takes a snapshot of the install logs as well as the system logs and several other key install PD artifacts. The installer then stores them in the <Product_Install_Root>/rcp/deploy/installHistory directory.

The files are zipped, and their names include the product, its version, the install action, and a timestamp. Here is an example of this file name format:

LotusExpeditorClient6.2.2_6.22.10194_PRODUCT_INSTALL_1259602706687.zip

Figure 8 shows an example of its contents.

Figure 8. InstallHistory snapshot data



Back to top

Provisioning logs

Though its name implies that it contains provisioning log information, the provisioning.log file (see figure 9) is actually used to communicate provisioning status information between the provisioning application (Java ) and the install application (MSI). This file generally contains the same status information that was displayed through the install application. The system log and trace files contain detailed information about the provisioning process and should be used to locate install/provisioning debug information.

Figure 9. Example provisioning log


Note the following:

• The provisioning.log file does not maintain any history, so you will see only the last provisioning status in the file you collect.

• The Lotus Expeditor provisioning system is built on top of the Eclipse Update Manager APIs. These Eclipse APIs have a collection of legacy log files that they generate (install.log, error_recovery.log, etc ) in the workspace/.config/org.eclipse.updatemanager directory. These are legacy files that should be ignored for initial problem determination. All relevant provisioning-related log and trace data is federated into the system log and trace files.

Back to top

Launcher log files

Since the launcher for Lotus Expeditor is a native application, it is not possible to store all launcher-related log messages in the Java logging framework.

• rcplauncher.log. The launcher output is written to the rcplauncher.log file in the logs directory (see figure 10). It contains extra information if you use the -debug option or if the launcher has a fatal error while launching the platform. In the event of a fatal error prior to the creation of the log, the output is written to /tmp/rcplauncher.log on Linux and to %TEMP%/rcplauncher.log on Windows.

Figure 10. Example rcplauncher log file

• shutdown.log. Created by the launcher in the event of a shutdown/restart error.

Back to top

PD framework internal log files

The IBM Automated Problem Determination (AutoPD) Tool is an aid for troubleshooting IBM software products. The tool provides the framework for the IBM Support Assistant (ISA) data collection. During its execution, it produces several log files that monitor the collection process and provide data, in case of collection failure for any reason:

• autopdinstance.log. Provides complete information for one execution of a collection script, including diagnostic information for the AutoPD tool itself.
• autopdsetupinstance.log. Captures the text that was displayed to the user in the progress window during one execution of a collection script. This text might include messages originating from the AutoPD tool itself, as well as those coming from any separate tools or other executable components that a collections script invokes.
• echoinstance.log. Provides messages related to the time when the tool is initially launched until the first collection script is invoked. If the data collection failed, these files provide details on the failure.

Back to top

Log collection

IBM Support Assistant is the way to gather the initial PD data you need to resolve issues. For details on running ISA, refer to the Lotus Expeditor 6.2 InfoCenter topic, "Collecting Expeditor Client data."

As you can see, there are many files here in many locations. Attempting to extract these files manually can be quite error prone and, in many cases, will require a second or third request for additional files based on review of the first file requested. ISA gathers ALL the files in one fell swoop, reducing the need to request additional files. (ISA collects many other files as well that are covered at a high level here ).

After collecting the PD data via ISA you should provide the ISA collection ZIP file to your IBM Service/Support representative for further assistance with your Problem Management Record (PMR).

Back to top

How to view the logs

The default logging format in Lotus Expeditor is an XML-based format called Common Base Event, which is an OASIS standard format for event logging. Details on viewing the system log and trace files can be found in the Lotus Expeditor InfoCenter topics, "Viewing the log file" and "Viewing the trace file."

Note that, due to well-formed XML needing to be correctly terminated, if a Lotus Expeditor-based platform experiences a crash, the log or trace file may not be terminated and you will get an error upon opening the file. For details on how to quickly correct this, refer to the InfoCenter topic, "Problems when viewing the log or trace file."

The rest of the files mentioned in this topic are simple text files and can be opened in any standard text editor.

Back to top

Log interpretation

The system trace file is actually the main entry point for log- and trace-based PD with Lotus Expeditor-based issues. Lotus Expeditor provides federation of log and trace messages from a collection of logging frameworks including Eclipse logging, Open Service Gateway initiative (OSGi) logging, Apache Commons, IBM Commons logging, and System.out and System.err. All relevant data from non-JSR-47 logging frameworks is extracted and mapped into JSR-47 ( java.util.logging ) format.

Since System.out and System.err are used simply to provide messages and include no other data, in terms of who logged the message and what severity it should be, they are mapped to the INFO level, and the subsystem is listed as SystemOut or SystemErr. In some cases, separate System.err and System.out data may be combined into single log messages, or stack traces sent to System.err may be reported as separate entries.

NOTE: The existence of WARNING or even SEVERE messages in the system log or trace file may not always indicate a critical situation or application failure. In some cases, specific components of the product can experience an issue that may not affect the normal operations of the product; however, messages may be logged for completeness and to ensure that, if other issues arise, all the information required is logged.

The previous sections discussed the details on locating and viewing the trace file. Now let's take a closer look at the content.

Back to top

Understanding the log and trace file format

The Troubleshooting and support section of the Lotus Expeditor InfoCenter provides a basic overview of the Log and trace formats. Note that the ThreadID listed in the trace data does not map directly to a specific thread ID or name in the system, but it can be used to correlate messages from the same thread or different threads.

OK, so you can see the layout of the trace file and you know the fields, but what can you do with this data?

The most critical column of the trace file is the Message column, which should provide you with initial error data to help focus your investigation of the problem. Unfortunately, in many cases the message provided doesn't include an explicit action to take to resolve the issue. In fact, in many cases, it doesn't even adequately describe the problem.

If the message contains a SWG Unique ID at the front of it, you can use this as a search term in an ISA Search of the Expeditor-based client product, or simply via your preferred search engine, to gather more data.

In some cases a message is logged that relates to the problem, but it may not contain all the information needed to debug and resolve the problem. In this case the Subsystem field entry can be very helpful. This field shows the name of the logger that was used to generate the log or trace message and, if more information is needed about the issue, you can turn on tracing for this logger/component.

If the problem can be recreated, more trace data should be produced, which may provide more data for PD. For details on enabling tracing for a component, see the "Adjusting the trace level..." subsections below.

Back to top

CONFIG message at the top of the trace file

The trace file includes a default message that is generated as part of the platform launch. This message provides some key information to help provide a baseline for the Lotus Expeditor-based product. Figure 11 shows an example of this data.

Figure 11. Initial CONFIG message example



There's quite a bit of data in this message that can be used in PD:

1. Date and time, including time zone (TZ) offset. The log data above shows that this log was created when a Lotus Expeditor-based product was launched on Jan 20th, 2010, at 10:18, with a GMT offset of 6 hours. From a PD perspective this can be critical when dealing with multiple trace files because it ensures you are reviewing the correct one and provides a hint to the time zone in use.
2. eclipse.buildId. This is the build date of the Lotus Expeditor platform in use, which can be mapped to a given version of Lotus Expeditor and then consequentially to a given consuming product version as well.
3. java.fullversion. In most cases, Lotus Expeditor platforms leverage the included Java Runtime Environment (JRE), so we would expect to see J9 and Java 6. However, some Lotus Expeditor-based products support existing JRE usage, which would appear in the above message.If a Sun JRE were used, this data could provide details on that JRE. If you are dealing with a JRE-level issue, this data will be important to validate with the IBM JRE team. In cases where JRE fixes are provided, this can be used to confirm that a given fix has been applied.
4. BootLoader constants. If a problem appears to be platform specific---for example, a Linux issue---or if a problem appears to be NL related, these values can be critical for initial baselining of PD information.
5. Framework arguments. Can be used to confirm that the product being launched (Lotus Expeditor, IBM Lotus® Sametime® , IBM Lotus® Notes®, etc.) includes custom product definitions in cases where custom product definitions have been developed.
6. Command-line arguments. Includes not only the BootLoader constants and the Framework arguments, but also contains the workspace being used ( -data ), which can be critical in multi-user configurations and when you need to update rcpinstall.properties for advanced tracing.

Back to top

IBM message identifiers

When the trace file contains log entries (SEVERE, WARNING, INFO) generated by IBM code, they will contain an IBM SWG identifier as part of the message string. This identifier maps to a specific product and to a component of that product. Figure 12 shows an example of an INFO log message that contains an SWG Globally Unique ID of CWPLR1001I. This can be used as a search term to research the problem via the IBM Support site, or via ISA.

Figure 12. Example of log message with an IBM Globally Unique Identifier



Back to top

Generating more trace data

In some cases your IBM Service/Support representative may request your assistance in gathering more detailed trace data for your issue. The following subsections provide details on how to gather more detailed trace data.

Adjusting the trace level manually

The default log and trace configuration file for the platform is the rcpinstall.properties file, found in the user's workspace/.config directory. To enable advanced tracing for a given component logger, you simply need to edit this file and add entries in this format:

<logger>.level=<level>

where "logger" is replaced with the component logger name, and "level" is replaced with the trace level for the component logger (FINE, FINER, FINEST):

ex. com.ibm.rcp.core.internal.pd.manager.level=FINEST

Once this file has been saved and the platform restarted, the new tracing level will be in effect.

NOTE: When tracing is no longer needed, be sure to remove these entries; otherwise, the product can experience performance degradation due to the extra tracing calls being enabled.

Back to top

Adjusting the trace level dynamically

Via the OSGi console
If you have access to the failing machine or can reproduce the problem, you can leverage the OSGi console to dynamically adjust the tracing levels.

Back to top
Via ISA
Lotus Expeditor 6.2.1 (and Lotus Sametime 8.5) added the ability to use Advanced ISA Collectors via the ISA UI, to enable component-specific traces during an ISA collection. View this short video to see how it's used in Lotus Expeditor 6.2.1.

The current list of components does not contain all the Lotus Expeditor or Sametime components, but it does cover those that statistically enable trace most frequently, based on a set of PMR's that were reviewed. With each new release, more components can and will be added based on Service/Support feedback.

Back to top

Other PD files

Additional files included with ISA data collection

The ISA collection gathers more than just the traditional "log" files for the platform. For an overview of all the types of files it collects, refer to the Lotus Expeditor InfoCenter topics, "Files gathered by the General Collector" and "Files gathered by the System Collector."

Back to top

Javacore file information

While Javadump-related files are not strictly "log" files, they can be critical to PD/PSI. For more details, refer to the Java Diagnostics Guide 6 InfoCenter topic, "Interpreting a Javadump."

Back to top

Conclusion

There is no single log or trace file that can provide all the data needed to resolve Expeditor-based product issues for all cases. The information in this article, however, presents a brief overview of the collection of log and trace files and PD/PSI tools available in Lotus Expeditor-based products. These files enable IBM customers to locate the data needed for the PD/PSI process, which they can provide to IBM Service/Support if needed.

Back to top

Resources

IBM Lotus Expeditor 6.2 information center:
http://publib.boulder.ibm.com/infocenter/ledoc/v6r2/index.jsp

developerWorks Lotus Expeditor product page:
http://www.ibm.com/developerworks/lotus/products/expeditor/

developerWorks® Java Technology article, "Java diagnostics, IBM style, Part 1: Introducing the IBM Diagnostic and Monitoring Tools for Java - Dump Analyzer";
http://www.ibm.com/developerworks/java/library/j-ibmtools1/

Lotus Expeditor Forum:
http://www-10.lotus.com/ldd/expforum.nsf?OpenDatabase

Back to top

About the author

Michael Burkhart is a Software Engineer based at IBM's Austin, Texas facility, currently working on the Lotus Expeditor project.

Back to top
---