IBM® Traveler® (formerly IBM Notes Traveler) is mobile e-mail software that provides quick access to mail, calendar and contacts. It gives two-way, over-the-air syncing between IBM Domino® servers and wireless handheld devices, including Android devices, and select mobile devices running the Exchange ActiveSync protocol, such as Apple, Windows Phone and BlackBerry 10 devices. IBM Traveler syncs mail, calendar and address book data in real time. On some selected clients, To Do and Journal data are also synced.
Overview
IBM Traveler provides a simple, easy-to-use interface with a minimal number of configuration settings. You can customize how much data is synced with the device to optimize the use of device memory and communication bandwidth.
New mail messages from the IBM Domino server arrive on your device automatically and can trigger a notification event, such as a tone or device vibration. Updates made on the device, such as sending a new mail message or changing a calendar entry, sync with the server as soon as a network connection is available.
The IBM Traveler server is installed on a computer running IBM Domino 853 FP6 and above. For basic configurations, the IBM Traveler component operates immediately following installation with minimal input required from an administrator. For highly available configurations, additional steps are required to enable multiple servers to operate in a high availability pool. Day-to-day administrator activities are performed using the IBM Domino Administrator client or IBM Traveler Web Administration and the IBM Domino remote administration console. IBM Traveler uses the Domino directory to automatically look up and find users, as a result there is no manual enrollment procedure.
If you are an IBM Notes® or IBM iNotes® user, then you are already enabled as an IBM Traveler user. Mobile handheld device users must only install the client software depending on the device, and direct the device to an IBM Traveler server. The device automatically registers with the server and syncing begins immediately.
The primary method for IBM Traveler clients communicating with the IBM Domino server is through an over-the-air communication channel. Examples include, cellular General Packet Radio Service (GPRS), WiFi (802.11x) or 3G. The IBM Traveler client works with any secure virtual private network (VPN) installed on the device. It also provides integrated support with IBM Mobile Connect. By using IBM Mobile Connect, you take advantage of the roaming and secure communication features that logically extend the enterprise network to the mobile device, regardless of the physical network that the device is using. The IBM Traveler client can connect using public GPRS or GSM (Global System for Mobile communications) networks and syncs data in transit over a security-rich Hypertext Transfer Protocol/Secure Sockets Layer (HTTP/SSL) connection, helping to safeguard sensitive data while enhancing compliance with your corporate policies and enhances administrator productivity while helping optimize the network throughput.
IBM Traveler...
- Supports remote wipe for lost or stolen devices. You can select full wipe of a device or a partial wipe of only IBM Traveler data.
- Enables security policies that control password length and strength specification, an option to deny access to unencrypted devices and an option to prohibit camera usage.
- Encrypts data at rest (on device) on Apple iOS, Android and Nokia Symbian devices.
- Enhances administrator productivity.
- Provides a single point of control to monitor the IBM Traveler community by user name, device type and operating system version; and to allow or deny access based on company security policies.
- Provides automated client updates.
- Supports scheduled synchronization and data filtering policies to optimize network throughput.
- Supports Linux®, Microsoft Windows® and IBM i servers using supported mobile devices.
- IBM Traveler Version 9.0.1.10 supports the latest mobile devices, documented here.
IBM Traveler syncs mail, calendar, contacts, and journal and to-do data through wireless networks with an IBM Domino server. The software provides basic mail collaboration features, such as create, reply, forward and delete (including attachment support). It also provides meeting-request support, including accepting and rejecting meeting invitations with comments, as well as attachment handling. Eligible IBM Notes customers can download the software free of charge. IBM Traveler software uses a Secure Sockets Layer (SSL) connection for encrypting data that travels over the air using an HTTPS protocol.
Traveler Architecture without High Availability:
Traveler Architecture with High Availability:
How IBM Traveler works
IBM Traveler is installed on an IBM Domino server and runs as a separate add-in task. Having a dedicated infrastructure is preferable, which means nothing else should be deployed on the IBM Traveler servers to ensure adequate system resources are available.
Each mobile device has an IBM Traveler client installed (Apple Active Sync profile, IBM Verse for Android, and Apple devices and Windows Phone using active sync) which communicates with the IBM Traveler server over HTTP or HTTPS. All mobile devices must download the IBM Verse client, except for Windows Phone devices which download a configuration a profile and use the native mail client.
The following components are involved when a user accesses the IBM Traveler server:
- User mail file
- Mobile device
- Domino directory
The following figure shows the main components of a simple IBM Traveler configuration and how they interact:
New mail messages arriving in your Inbox on the Domino server arrive on the device without you needing to do anything (that is, are automatically pushed) and can trigger a notification event, such as a tone or a device vibration. Updates made on the device, such as sending a new mail message or changing a calendar entry are synced with the server as soon as a network connection is available, and are reflected in the user's mail file and Notes client.
The IBM Traveler client provides a simple, easy-to-use interface with a minimal number of configuration settings. You can customize how much data is synced with the device to optimize the use of device memory and server resources.
The IBM Traveler server checks the Domino Directory for the user home server and mail file information and subsequently connects to it. The IBM Traveler server does not store any data, only the user's designated mail server has the user mail file.
In a stand-alone implementation (non high availability) of an IBM Traveler server, there is a local Derby database where information about the user's subscribed folders, devices, and sync status is stored. This derby database also stores any security information about the devices on the system if the administrator of the server has implemented specific settings in regards to devices security. In a high availability implementation of the IBM Traveler server, this information is kept in a central database on a separate enterprise database server (IBM DB2 for Linux, UNIX, and Windows or Microsoft SQL Server) and shared amongst all the IBM Traveler servers in the same high availability pool.
There is a friendly way to see the majority of information stored in the IBM Traveler database referenced above: the LotusTraveler.nsf file, located in the root folder of the IBM Traveler server for stand-alone implementations or a web interface found at http(s)://server_name.domain.com/LotusTraveler.nsf for a high availability implementation. In this view, you can see all devices and users. You can also use this interface to administer these devices and users. In essence, this is a graphical user interface for administering the IBM Traveler's database information.
Daily Administrator Task: Monitoring activities of Traveler Server
The Traveler task is part of the "Server tasks" within Domino Administrator.
You can run the “Show Task” command on a server console to reveal whether the Traveler task is up and running.
For tracking the performance of IBM Traveler, you can check its status through the command
“Tell Traveler Status”
Traveler status can be any of the below:
1. GREEN: Indicates server is performing Good.
2. YELLOW: Indicates that there are some errors/issues on the server that need to be addressed.
3. RED: Indicates there are critical server issues that need immediate intervention.
A Domino program document can be used to establish regular output of IBM Traveler server statistics and usage.
In addition, the command
"tell traveler stat show" will give detailed information about the statistics that IBM Traveler maintains. A Domino program document can be used to establish regular output of IBM Traveler server statistics and usage. Statistics can be viewed in Domino Administrator client.
When any of the IBM Traveler statistics (for example, memory or CPU) exceed a given threshold, IBM Traveler goes into a constrained state. Once the constrained state is detected, IBM traveler will not allow new device sync or prime sync threads to start, but existing threads will be allowed to complete. Apple device users may experience a
"Cannot connect to server" error or slow responsiveness, while Android users may see
error 503 (server busy) .
At the Domino Console on the IBM Traveler server, run the command "show stat traveler.constrained.state" → where 1 indicates the server is running in constrained state.
You can also Configure Event Probes to send a notification to the helpdesk team at times when IBM Traveler's performance degrades.
Setting up a program document
To run the command
"Tell traveler status" at regular intervals, you must set up a program document.
With a program document, you can set Traveler to run the command
"Tell traveler status" at regular intervals, and mail a designated ID with an alert if, at any given point of time, the IBM Traveler status is not GREEN.
Setting up a program document:
A sample mail notification received when there is an event handler in place for when the IBM Traveler status changes to Yellow or Red:
With this alert in place , an administrator can take corrective actions and thus proactively monitor server issues.
Different Kinds of Logs
By default, IBM Traveler is configured to log the messages to the following locations:
Log.nsf | SEVERE and WARNING messages are logged to this Domino database |
Domino Console | Anything logged to Log.nsf will appear on the Domino console as well, so SEVERE and WARNING messages appear here. The output from the domino console is not logged to a file by default, but by setting the key DEBUG_OUTFILE=filename in your server NOTES.INI, you can save the console output. This can be helpful as there are many messages that appear on the console that are not saved anywhere else |
NTSActivity.log | This file contains all messages and traces that are logged by Topsail. If the logging level is set to FINEST, then this file can wrap quickly depending on how many users are using the system and the wrapping settings that are being used. |
NTSErrors.log | This file contains on logs with the SEVERE level |
NTSUsage.log | This file contains only the usage log records. Each time a device synchronizes or a user does a prime sync, then a record indicating the time, user, sync status, device type and the number of changed records is logged to this file. |
NTSServletActivity.log | This file contains all messages and traces that are logged by Traveler's servlet. |
NTSUsage.log file
IBM Traveler records all device and user activity in text filse under
\IBM_TECHNICAL_SUPPORT\traveler\logs\:
NTSUsage_YYMMDD_HHMMSS.log
NTSActivity_YYMMDD_HHMMSS.log
The timestamp corresponds to the date and time that the log file was created. It will wrap when it reaches the maximum defined usage log size, but the logger can be configured to keep older versions as well. Usage logger limits are defined in the file
\traveler\cfg\NTSLogging.properties.
Messages logged at FINE, FINER and FINEST levels are debug traces. These levels are generally only enabled during testing or by a customer at the direction of an IBM support representative for diagnosing issues. The logging system that Topsail is using is extremely configurable. You can modify the logging level the system is using and change where the log messages are stored.
The IBM Traveler server can log messages with the following levels:
SEVERE | Error conditions |
WARNING | An unexpected condition, but this could be normal |
INFO | Information messages |
FINE | Debug trace, lowest level |
FINER | Debug trace, medium level |
FINEST | Debug trace, highest level |
Understanding SystemDump, UserDump, and NTS logs
IBM Traveler captures its transactions, server statistics, and user statistics in different sets of logs. Based on the type of issue, each of these logs can be read to find the root cause of an issue.
Understanding User Dump Log:
The User Dump log file provides quick information on User statistics, User profile details, missing documents and folders, ad security related issues for the user.
User Dump logs are stored on the IBM Traveler server in the IBM Technical Support folder under the Traveler/dumps directory. These logs get saved with name format of Username, followed by the date and time when the log was created.
User Dump logs can be reviewed as a first attempt at troubleshooting user sync issues, access issues, missing documents on the device, and so on.
There is no actual data stored in User Dump logs from the mail file. However, it has a reference for each document in the mail file, which are used by the IBM Traveler task while syncing that document from the mail file to the device.
The User Dump log can be generated by running the command "Tell traveler dump Username" on the IBM Traveler server.
The user name can be either the mail address or the Notes canonical name.
There are many different sections in a dump log, and are denoted by a line of Hash keys having with the section name in the center.
For example, the first section is identified with the IBM Traveler user name:
# IBM Traveler user dump for #
This section contains basic server information, such as the Domino and Traveler server versions, CPU details, and so on. This is basically the same as what is shown in a SystemDump.
The next section is "Show" section:
########## Show ##########
This section has 3 different piece of information:
- User configuration
- User profile information
- User device information
########## Show ##########
IBM Traveler has validated that it can access the database mail/administ.nsf.
Monitoring of the database for changes is enabled.
Encrypting, decrypting and signing messages are disabled because the Notes ID is not in the mail file or the ID vault.
Canonical Name: CN=Administrator/O=Acme
Internet Address: Administrator@acme.com
Home Mail Server: CN=Server1/O=Acme
Home Mail File: mail/administ.nsf
Current Monitor Server: CN=Server1/O=Acme Release 9.0.1FP3
Current Monitor File: mail/administ.nsf
Mail File Replicas:
[CN=Server1/O=Acme, mail/administ.nsf] is reachable.
ACL for Administrator/Acme: Access=Manager Capabilities=create,update,read,delete,copy Missing
Capabilities=none
ACL for Server1/Acme: Access=Manager Capabilities=create,update,read,delete,copy Missing
Capabilities=none
As you can see from the example, the user configuration section displays information if IBM Traveler is able to connect and access the user mail database.
IBM Traveler also validates whether the user is having appropriate access to the Traveler server.
If there is an issue with access, if the database is over quota, or any other reason for which the database cannot be accessed, the error will be displayed here and on the profile information section, which displays the user address, Mail server and cluster server details. This will display the path of all servers on which a replica of the user database exists. As a result, this information is referred by the Traveler task to switch to a secondary replica of the user mail database when the primary server goes down.
There is also a section providing information about all configured devices for the user. We have not listed this above, but this page contains information about Device type, Device ID, Device OS, Traveler application version and synch state. In this section, there are two areas for Sync State: "Auto Sync User State" and "Auto Sync Connection State".
Auto Sync User State is for active monitoring of the user's mail database. The Traveler server will monitor a user's mail database for 24 hours after the device last connected to server.
Auto Sync Connection State displays whether or not a particular device has a current active push connection to the server. In other words, if the device is actively connected and syncing.
Security Status
This section tracks any security action requests that may be applied to the device. For example, wipe, device approval, and so on. If any Security Flag is set for the user's device, the device is denied access to the system. As an example, if you have device approval set, and the user configured a new device which is not yet approved by an Administrator, the security flag for this approval restriction will display here. The Security status section of user dump and system dump is similar.
Push Status
This section provides information about whether or not the user mail database is actively monitored by the Traveler server. If this is disabled for any reason, the device will not be able to sync mail.
For example, suppose monitoring is disabled for a database. In that case, IBM Traveler will not monitor the user mail database and will not have any new updates to push to the device. Even if the device connects it will show as updated but not sync any new mail that might have been received in the database.
You can quickly reenable database monitoring by running the command "Tell traveler push enable Username".
Threads
This gives information on the running threads assigned to this user. It also gives details about how long the thread has been running. A PS thread running over 60 minutes or any other thread running over 10 minutes would be suspicious, but it might not necessarily be a problem.
User Cache
This section contains cached information about a user's Access Control List, Domino database information, and preferences. Sometimes it may happen that the user was over quota or had an incorrect ACL which was causing mail to not sync. While the issue was being addressed, Traveler might still be referring to old information of the user. In that case it must be because the cache information for the user has not been updated by Traveler. Generally, cache information refreshes automatically, but if you notice it is not updating, perform a "Tell traveler show user" command to force Traveler to refresh the cache for that user.
Device Profiles
Profile documents exist in the user's mail database to store information about each device's preferences and security settings. When you set a policy to control Traveler data or apply security settings, the Traveler profile is pushed to the user mail database and this section displays what profiles exist for the user. Profiles are not deleted if you do a "rest" command for the user. They can be deleted using the delete profile tell commands.
Security Policy Status
This section denotes the status of devices complying with any assigned security policy settings. If you are having issues where a policy is not updating correctly, you can validate the user policy status here.
GUID (Global Unique Identifier) Map
This section shows the references of all the documents from the user's mail database that could be synced with at least one device for the user.
Suppose you have 1000 mail messages on your mail file, but your Traveler settings allow the syncing of only the last 30 days. In this case, all mail in this 30 day limit will be mapped in this section.
LGUID is the Local GUID that the device and server use as the key. BACKEND_GUID is the Domino UNID. If a document on a device is missing, you can look to see if the UNID for the document is listed here. If the document is listed, you can see if the folder assigned is the one expected and being synced. You can match the document in this table with the document UNID in the actual mail database of the user. You can find this document UNID (which can be a contact, mail or meeting) in the document properties tab. The first two lines are the DOC UNID, excluding the OFF and ON text.
For example:
LGUID: 82615 BACKEND_GUID: 38D46BF5E8F08834852564B500129B2C TYPE: Folder FOLDER: Inbox PCOUNT: 0 FCOUNT: 0
Database Usage
This section gives information on how many records the user has in the IBM Traveler database. The more records a user has, the more capacity the user is consuming. After the record counts, the current filters are displayed. If a user has a large number of records, the cause might be undesirable filter settings. You can see a sample output below:
----- Document Usage by Type -----
Mail: 31
Calendar: 8
Contacts: 0
To Do: 0
Notebook: 0
Folder: 7
----- Device Filters -----
ApplDLXFJGG3DFHY Mail 0130228receiveddate>=last:3:days
ApplDLXFJGG3DFHY Calendar 0130228startdate>=last:30:days
ApplDLXFJGG3DFHY Contacts 0130228
ApplDLXFJGG3DFHY To Do 0130228complete=f
ApplDLXFJGG3DFHY Notebook 0130228modtime>=last:7:days
ApplDLXFJGG3DFHY Folder 0130228
Here we see the user has 31 mail messages, 7 folders, and 8 meetings.
This section also display device filters so that administrators can understand how much data a user is syncing with each device.
Understanding SystemDump Logs:
SystemDump logs provide quick information on IBM Traveler status and performance statistics over the last 24 hours for all IBM Traveler configuration settings. This helps in understanding performance and configuration issues.
The SystemDump log can be generated on the Traveler server by running the command: “tell traveler SystemDump”.
When the command is run, a Log file gets generated in the Traveler logs/dumps directory. The SystemDump log file dumps all the variables of the Traveler server, including Domino server document configuration, Traveler
statistics, and running threads. You can read the important sections inside the SystemDump log that will help analyze issues.
General Information section
This section contains basic server information, such as the Domino and Traveler server versions, CPU, and so on, and is similar to what is shown in the User Dump information section.
######### IBM Traveler Server system dump ##########
IBM Traveler Version: 9.0.1.7 Build 201508211840_20
Domino Version: Release 9.0.1FP3|January 12, 2015
Domino Platform: Windows/64 with 4 processors
Domino Install Type: Domino Enterprise Server License (4)
Domino Server Name: Server1/Acme
Database Connection URL: jdbc:derby:ntsdb;create=true
Current Time (Local): Tue Sep 01 20:30:59 IST 2015
Current Time (GMT): Tue Sep 01 15:00:59 GMT 2015
IBM Traveler Started: Tue Sep 01 12:37:45 IST 2015 (running for 0 days, 7 hours, 53 minutes, and 13 seconds)
Last Defrag: Thu Aug 28 12:05:24 IST 2015; Defrag Interval in Days: 30
Status Section
Gives a quick snap shot of the overall system. If the status of the machine is not in a "Green" state, a list of reasons why the system is either in Yellow or Red are listed. The administrator can take corrective actions based on the status responses to ensure the server status is Green.
Status
Sample Output:
The IBM Traveler task has been running since Sun Sep 06 16:07:29 IST 2015.
The IBM Traveler availability index is currently 100 while servicing 1 users.
The last successful device sync was on Wed Sep 09 12:31:13 IST 2015.
The overall status of IBM Traveler is Green.
Current CPU Utilization: CPU running at 0.37 pct over 522.659 Secs
CPU and Memory (MB) Usage History
Date CPU Pct Java Mem C Mem Avl Indx # Users # Errors # DB Conn
2015-09-07 06:15:39 IST 0.00 43 2711 100 0 0 0
2015-09-07 06:30:39 IST 0.05 37 2726 100 0 3 1
2015-09-07 06:45:39 IST 0.03 39 2790 100 0 3 0
2015-09-07 07:00:39 IST 0.04 41 2854 100 0 3 0
Memory Usage Section:
This section gives a quick overview on memory availability on the server. The information is helpful to understand if there are memory fine tuning requirements on the server or if there are any memory leak issues causing server performance to degrade.
Sample Output
Java Memory Usage ← Total memory used by Notes Traveler process
Max Total 512 MB ← Maximum memory as configured on the IBM Traveler tab of server document
Current Total 96 MB ← JVM heap size
Free 440 MB (86 percent of Max Total) ← JVM free allocated memory
Allocated 72 MB (14 percent of Max Total) ← Memory being used currently by Traveler
C Memory Usage ← Amount of allocatable memory left on the machine
Total Virtual 2047 MB ← Total virtual memory available left on the machine
Total Physical 2047 MB ← Total physical memory available left on the machine
Allocated 1147 MB (57 percent of Total Physical) ← Allocated memory on the machine
Current Usage ← Summary of the above two values
Java 72 MB ← Traveler used memory
C 1147 MB ← Machine's memory being used
Thread Manager
This section lists the active threads, the mappings of tokens to threads, and an overall summary. A long running time for a thread could explain a long sync for a user or higher than expected CPU utilization. If any threads appear hung, a Stop Sync command may be possible if the thread is related to a sync (DS or PS). Otherwise, and for all other issues, a restart of IBM Traveler is probably needed to clear up the issue.
You can find a long running thread by looking at this value in this section:
DS-25b4[Email_4][0C4D16EC129F332480257B1F002806AF][13746291] [CN=Notes Admin/O=Acme]
[ApplC37JW7PFDTWD] [CN=Notes Admin/O=AcmeApplC37JW7PFDTWDsyncASSyncEmail4] [CN=Notes
Admin/O=AcmeApplC37JW7PFDTWDsyncASSyncEmail4] [0 runnables] [Busy? true] [Last Runnable: Thu Sep 11 18:00:01
IST 2015] [Running: 7125ms] [Idle: 0ms]
A long running PS thread indicates that the user is syncing a large amount of data (no limits are set). The following thread types are ones to be concerned about:
DS = Device Sync (Connection thread between Mobile device and Traveler server)
PS = Prime Sync (Connection thread between Traveler server and User Mail server)
Statistics
This section lists all Traveler statistics that have been collected since the stats were last reset (the statistics are automatically reset each time the Traveler process is restarted). Generally the SystemDump displays statistics for the last 24 hours, or since when the task was last restarted, whichever is lower.
Some important Statistics to monitor:
CPU.Pct.
Traveler checks the CPU usage on a periodic basis. This stat is a histogram showing how many times the CPU percentage was in the specified range. The range values are "000-010", "010-020", "020-030", "030-040", "040-050", "050-060", "060-070", "070-080", "080-090", "090-100". As an example, CPU.Pct.040-050 would show the number of times the CPU usage was between 40% and 50%.
DCA.DB_OPEN.Time.Histogram..
Histogram of the time spent (in seconds) to open a database on the given server using the Domino Java API call. is the name of the Domino server on which the database was opened.
Range values (in seconds) are "000-001", "001-002", "002-005", "005-010", "010-030", "030-060", "060-120", "120-Inf".
If any server shows numerous requests in several ranges that may denote some connectivity issues between Traveler and that server, which could lead to server performance degradation.
Database.Query.Histogram.GudSelect.
Histogram of the time spent (in seconds) to execute the look-up of a user against Traveler's internal database. A high number in the larger bucket indicates the need for a defrag.
Push.Devices.Total
The total number of devices registered on the server.
Errors
This gives a number based overview of how many errors occurred on the server and how many errors occurred for each user.
Example:
Errors = 34
Errors.Administrator = 3
Errors.Test User1 = 30
Errors.Test Admin = 1
This information is helpful for understanding Traveler red status issues due to high error count. An administrator can find which user is encountering more errors and narrow down the issue to resolve the Red status.
Prime Synch and Device Synch statistics
These statistics are helpful in understanding success rates of all the prime syncs and device syncs processed by the server.
Example :
There have been 1300 prime syncs.
The average prime sync took 168 ms.
100 percent (1300) of the prime syncs were successful.
The average successful prime sync took 168 ms.
0 percent (0) of the prime syncs failed.
The average failed prime sync took N/A ms.
There are an average of 0 prime syncs running at any given time.
There have been 285 device syncs.
The average device sync took 4,366 ms and transferred 84,928 bytes.
100 percent (285) of the device syncs were successful.
The average successful device sync took 4,366 ms.
0 percent (0) of the device syncs failed.
The average failed device sync took N/A ms.
Show Active
This section lists all users who are actively syncing and the type of sync being performed.
Mail Replicas
This section provides Mail replica information stored in the Traveler database for all users.
CN=User/OU=Orig Unit/O=Org = , , , and so on.
MasterMonitor
Lists the status of users enabled for monitoring of mail database changes and their current state. Monitoring is the process by which IBM Traveler detects changes in the user's mail database on a periodic basis. When a change is detected, the user is queued for a prime sync which may result in push messages (if enabled).
If a user database is not being actively monitored, changes in the user's mail database will not be known to IBM Traveler. In addition, all syncs will be empty of server side data, and when the device tries fetching mails no updates or errors will be synced.
Banned Documents
This section provides a list of documents that have, in the past, caused Traveler to crash, and have since been banned. A document is automatically banned if it causes the server to crash per the value set (the default 2).
To disable this, set * NTS_BAN_DOC_LIMIT in the notes.ini to 0.
High Availability (HA)
Provides information on High availability stored in the IBM Traveler database.
For example:
--- HA Servers ---
Domino Name ID Hostname IP:Port Alive Reachable Last Heartbeat AI Users Devices Build Level Startup DB Version
All SMS Migrated Status
--- HA Devices ---
--- HA Users ---
NTS Activity/Audit/Usage/Error Logs:
NTS Logs capture overall IBM Traveler transactions, and are helpful in analyzing sync related, performance crash issues.
NTSUsage
NTUSAGE Logs all usages of the IBM Traveler server. NTSUsage logs record an entry for each transaction that the system performs on behalf of the devices. When users sync their device with the server to update their mail, contacts and calendar information, this activity is logged in the NTSUsage log The transactions are logged once they complete, not when they begin.
This information can be useful to determine if devices are syncing with the server and how long this process is taking. The administrator can monitor the log for issues, such as when the device last connected, what was synced between the device, the return codes, and the amount of time the sync took to complete.
Sample Transaction:
"10/15 15:28:44.584" 192.168.56.1 192.168.56.2 "CN=Test User/O=Acme" "http://192.168.56.2/mail%2Ftuser.nsf?
SendMail" dp 200 328 "Apple-iPad2C1/902.206" ApplDLXFJGG3DFHY 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 "D2S Mail 1A"
[TimeStamp]
"10/15 15:28:44.584"
[Device IP Address]
192.168.56.1
[Server IP Address]
192.168.56.2
[User, Database or Action]
"CN=Test User/O=Acme" "http://192.168.56.2/mail%2Ftuser.nsf?SendMail"
[Synch Origin]
Dp
Identifies who requested the sync (ps = prime sync, dm = device manual, dp = device push, sp = server push)
[Status Return Code on request]
200
200=OK, 408=Request Timeout (Device did not respond before the Server timed out the session), 409=Conflict
(device started a new session which caused the session to be aborted), 500=Unknown Error, 503=Server Busy
[Time (ms) taken for request to complete]
328
[User Agent involved in transaction]
"Apple-iPad2C1/902.206"
[Device ID]
ApplDLXFJGG3DFHY
[Changes from (Device to Server) (18)]
1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
[Changes (Server to Device) (18)]
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
[Summary of Changes (only present if there are non-zero Changes)]
"D2S Mail 1A"
Device 2 Server, this was a Mail, and 1 add was done
→ Mail-Add Mail-Update Mail-Delete Contact-Add Contact-Update Contact-Delete Calendar-Add Calendar-Update
Calendar-Delete ToDo-Add ToDo-Update ToDo-Delete Journal-Add Journal-Update Journal-Delete Folder-Add
Folder-Update Folder-Delete
NTSErrors:
NTSError log files list the SEVERE level messages that were encountered by the server.
Sample error:
[TimeStamp] [Level] [Thread Name] [User] [Class.Function#Line] [Message]
[02/27 14:43:40.325] SEVERE PS-3eec CN=User/OU=Org Unit/O=Org BackEndManager.validateDBAccess#563
Exception Thrown: com.lotus.sync.dca.CAException: Notes error: The server is not responding. The server may be down or you may be experiencing network problems.
Additional debug messages (not always present)
Stack trace (not always present)
at lotus.domino.local.Database.NqueryAccess(Native Method)
at lotus.domino.local.Database.queryAccess(Unknown Source)
at com.lotus.sync.dca.access.ACLMgr.getDatabaseAccess(ACLMgr.java:105)
at com.lotus.sync.dca.access.ACLMgr.<init>(ACLMgr.java:65)
at com.lotus.sync.dca.BackEndManager.validateDBAccess(BackEndManager.java:519)
at com.lotus.sync.caf.ContentManager.getDBInformation(ContentManager.java:1077)
at com.lotus.sync.caf.CAFSession.setUriMapping(CAFSession.java:884)
at com.lotus.sync.caf.CAFSession.<init>(CAFSession.java:808)
at com.lotus.sync.caf.CAFSessionManager.createSession(CAFSessionManager.java:111)
at com.lotus.sync.caf.auth.AuthWrapper.validateBackendAccess(AuthWrapper.java:521)
at com.lotus.sync.caf.auth.AuthWrapper.validateLogin(AuthWrapper.java:236)
at com.lotus.sync.TSS.ExtRoutine.TssHook.validateLogin(TssHook.java:80)
In the case that issues are present, the administrator can read these logs for a detailed transaction of errors.
NTSActivity:
NTSACTIVITY logs list all errors and the usage activity that are seen in the NTSErrors, and NTSUsage logs. In addition, it lists all tracing that is done on a user/server.
Below is the Basic layout of the log message:
[TimeStamp] [Level] [Thread Name] [User] [Class.Function#Line] [Message]
[10/15 15:28:44.584] FINE DS-0f18[Email_4] CN=User/OU=Org Unit/O=Org ActiveSyncSyncSession.run#232
ActiveSync starting for user CN=User/OU=Org Unit/O=Org on device ApplDLXFJGG3DFHY
[10/15 15:28:44.584] FINE DS-0f18[Email_4] CN=User/OU=Org Unit/O=Org ActiveSyncSyncSession.run#232
ActiveSync starting for user CN=User/OU=Org Unit/O=Org on device ApplDLXFJGG3DFHY
Details:
Timestamp: The local date and time to millisecond precision
Level: SEVERE, WARNING, INFO, FINE, FINER, FINEST
Thread Identifier: The thread name with an optional correlator in brackets [ ]
User Name: The user's ID in canonical name format
Source Code ID: The class, method, and line number that originated the message log
Message: The message can be any text
NTSAudit:
NTSAudit files log all changes to the system, such as settings changes (server document, notes.ini, and so on) and server state changes (up, down, and so forth). They also Log all the tell commands run on the server.
Sample details in the NTSAUDIT Log file.
"10/15 19:27:10.513" Profile *System* "Profile updated: devPmaxAtt - oldValue=20480, newValue=4000"
"10/15 19:27:10.513" Profile *System* "Profile updated: devPbTrunc(devPbTrunc) - oldValue=5000, newValue=4000"
"10/17 08:19:12.545" TellCommand *Console* "tell traveler Status: status=OK, time=188 ms"
"10/19 07:29:08.664" TellCommand *Console* "tell traveler Memory show: status=OK, time=15 ms"
"10/20 11:47:57.056" Configuration *System* "Non-default value: NTS_64_BIT - oldValue=true (Default),
newValue=false"
"10/20 11:47:57.056" Configuration *System* "Non-default value: NTS_BUILD - oldValue= (Default), newValue=9.0.1.7
Build 201508211840_20"
These logs are helpful to understand if any changes were made to server.
How to analyze the logs: Log analysis to resolve Traveler issues
The following scenarios highlight common problems with Traveler and how to solve them.
Scenario 1: Performance issues on the server
Symptoms:
1. Users experience slow mail syncing on devices using IBM Traveler server.
2. There is a CPU spike / High CPU utilization by IBM Traveler server.
3. Memory time-out errors are occurring on the IBM Traveler server.
4. Devices are not syncing.
Generally, whenever you see performance degradation, issue a system dump command. The system dump log will provide a fair idea about what could be the cause of Traveler performance degradation. However, the system dump may not necessarily give clear hints on the issue, and sometimes it may be necessary to collect additional logs and/or use the "finest" level of logging.
In the below example, the IBM Traveler status shows as RED:
→ Example: Traveler status section in System dump
*******************************Status******************************
[0C6C:3711-3A8C] The IBM Traveler task has been running since Mon Jul 27 01:16:34 IST 2015.
[0C6C:3711-3A8C] The IBM Traveler availability index is currently 70 while servicing 1,869 users.
[0C6C:3711-3A8C] The last successful device sync was on Mon Aug 24 11:59:55 IST 2015.
The overall status of IBM Traveler is Red.
********************************************************************
When IBM Traveler status is not Green, the systemdump will clearly indicate all reasons why. As you can see in the above example, there are multiple reasons causing the status of RED:
Reason 1. The IDX table or index has a large number of pages allocated that are unfilled. This means Traveler NTSDB requires a defrag.
Reason 2. The peak number of HTTP connections are more than what currently are allocated. You need to calculate the required HTTP thread based on how many devices are configured with the server. If HTTPS threads on the server are low, then they will cause performance issues.
Reason 3. The response time taken by Traveler for communicating with one of the mail servers is too high. This can lead to threads being held up, which can impact performance.
Reason 4 and 5: IBM Traveler reported many errors in the last interval, and there are few threads which are running for a longer time.
Note that you will not necessarily see all these errors all together on the server. They have been included here for demonstration purposes.
Once you have checked the Status section, check the CPU and memory table in the system dump:
This gives usage statistics on last 24 hours of server time, an can help administrators understand what times the performance is at its peak.
In the above example, peak usage was around 11:22 AM. This being the case, you could conclude:
- HTTP threads utilization are high
- NTSDB Requires maintenance
- There are connectivity issues between IBM Traveler and the mail server in particular
- There are some hung threads
Correcting these issues should resolve the performance crunch.
Scenario 2: Traveler task configuration or startup issues
You may encounter issues where the Traveler task is not starting. Whenever you see startup issues, check the console log carefully. Generally, this will give clear observations on what is wrong. Look for any abnormal messages during startup of Traveler and the HTTP task.
In the below example, there are many errors that show connection problems to NTSDB:
------------------------------------------------------------------------
IBM Traveler: Server starting...
IBM Traveler: SEVERE *system Connection to database ntsdb unable to be made.
Verify that you have properly created the remote database. Exception Thrown: java.sql.SQLException: Failed to start database 'ntsdb' with class
loader sun.misc.Launcher$AppClassLoader@72b072b, see the next exception for details.
IBM Traveler: SEVERE *system The component com.lotus.sync.dca.BackEndManager could not be started. Internal
error: com.lotus.sync.db.PersistenceException: java.sql.SQLException: Failed to start database 'ntsdb' with class
loader sun.misc.Launcher$AppClassLoader@72b072b, see the next exception for for details.
IBM Traveler: SEVERE *system IBM Traveler Server could not be started. The exception was
com.lotus.sync.util.ComponentNotStartedException: A TrueSync Server startup timeout has occurred.. Exception
Thrown: com.lotus.sync.util .ComponentNotStartedException: A TrueSync Server startup timeout has occurred.
------------------------------------------------------------------------
Here it can be concluded that the Traveler task is unable to read NTSDB. If the NTS database is corrupt, it will restrict the Traveler task from starting.
In this case, try starting the Traveler task with the command "Tell traveler -defrag" in order to force maintenance on NTSDB before starting the task. This should correct any corruption issue with NTSDB.
Occasionally, the Traveler task may start, but users either cannot sync mail or the Traveler page does not load. In such cases, check the console for any abnormal errors during Traveler and HTTP task start up as you did previously. IBM Traveler has a dependency on HTTP, so if there are any errors with starting the HTTP components that will not allow Traveler to service users.
In the below example, both the Traveler and HTTP both tasks have been started. However, HTTP was unable to initialize XSP, which is one of its components.
Sample Error:
[1798:0002-2200] 09/06/2015 04:03:41 PM Traveler: Loading HTTP server.
[10D8:0002-0870] 09/06/2015 04:03:41 PM HTTP Server: Using Web Configuration View
[1798:0002-2200] 09/06/2015 04:03:42 PM Traveler: Server started.
[10D8:0002-0870] 09/06/2015 04:03:45 PM JVM: Java Virtual Machine initialized.
[10D8:0002-0870] 09/06/2015 04:03:45 PM HTTP Server: Java Virtual Machine loaded
[10D8:0002-0870] 09/06/2015 04:03:45 PM HTTP Server: DSAPI Domino Off-Line Services HTTP extension Loaded successfully
[10D8:0002-0870] 09/06/2015 04:03:50 PM Xsp Initialization error - Could not load class or methods
[0738:000C-2B40] 09/06/2015 04:03:51 PM This server is currently a member of a cluster
[10D8:0002-0870] 09/06/2015 04:03:55 PM HTTP Server: Started
For detailed debugging to determine why a XSP initialization failed, you can start the Traveler task with the debug command “load traveler -debug”. For the above example, the problem is due to a corrupted JVM file. As a potential solution, try copying the JVM folder from another good working server with the same OS and release. Otherwise, you might need to reinstall Domino to correct these JVM files.
Scenario 3: Traveler servers in HA are not communicating with each other
There can be issues with Traveler in a High Availability setup such that servers in the HA pool are not communicating with each other.
For example:
Here, both of the servers are running but one is not reachable. The command to check the status of all servers in an HA pool is "Tell traveler HADR show".
In this situation, there might be no communication received from the other server which is causing it to show its availability as false. As a basic troubleshooting step, you can run the command “tell traveler HADR ping
" from the console of another working Traveler server. If the ping command displays the correct results, then rerun the command "tell traveler hadr show". Both servers should be reachable now
If the ping command displays incorrect results, then there may be a network connectivity problem between the two servers, or there might be issues with Traveler or HTTP startup on another server, which is why it is not reachable.
Scenario 4: User connectivity / configuration issues.
You may encounter issues such as:
- Users receive an error while logging into the IBM Traveler home URL or connecting to the Traveler server from their device.
- Mail messages do not sync the message body properly.
- Unable to download certain types of attachment.
- The Traveler client crashes or has a very slow response.
For any of these issues related to particular user, always issue a user Dump command against the affected user and check the User dump key sections as discussed earlier to see if the user has correct access and IBM Traveler is able to reach and access the user mail database.
The User dump will provide further information on the sync issues related to a particular folder or document. For example, if a user complains about missing documents or folders, check in the GUID map section of the user dump to understand if Traveler was able to read that document from the user database.
Users may also report that they are unable to configure IBM Traveler on their device. In these cases, ask the user to attempt to login to the Traveler homepage. If an error is displayed, you can reference the below sample errors.
In this case, either the database is over quota, the Mail server is not reachable from the Traveler server, or the user does not have sufficient access which caused the user connection to fail. In such scenarios of user configuration issues, a User dump will provide detailed information on the error. Check the dump to see if the Traveler server is able to read and resolve the Notes User ID. Also check to see if the Traveler server is able to read the user database and its replicas across the cluster. Verify that there is no security flag, restricting the user and their accessibility to the system. Verify that the monitoring of the database for changes is enabled.
You can see in the below example where the user mail database cannot be accessed, and has caused the user configuration to fail.
IBM Traveler could not open the database mail/administ.nsf. Verify that the server CN=Server1/O=Acme and the database
grant access to server CN=Server1/O=Acme and that there is a network connection available between these servers.
Monitoring of the database for changes is disabled.
Canonical Name: CN=Administrator/O=Acme
Internet Address: Administrator@acme.com
Home Mail Server: CN=Server1/O=Acme
Home Mail File: mail/administ.nsf
The IBM Traveler server cannot connect to your mail database mail/administ.nsf on server CN=Server1/O=Acme. Verify that
your mail server mail database grants access to server CN=Server1/O=Acme and is operational. If this does not resolve the
problem, your administrator may need to verify the network connection between the servers and that the IBM Traveler server
is allowed to access your mail server.
[CN=Server1/O=Acme, mail/administ.nsf] is not reachable, status(0x246) "You are not authorized to perform that operation".
In the case of user level connectivity issues, the device and server logs need to be collected. It might be a new user configuration or an existing user unable to connect to the server. At a first level, check if the Traveler home URL is accessible from the user's device, to ensure there is no mobile internet issue restricting reaching the Traveler server. Also, check if there are any errors in the User profile statistics of the User DUMP log.
As a next step, collect logs from the device for the Traveler application by enabling logging in the Traveler application. It may be necessary at times to also enable HTTP debugging by running the HTTP debug command as shown to collect HTTP logs. HTTP logs help understand if the request generated from a device is getting modified somewhere at the network level before it reaches the Traveler server.
Always enable finest logs while tracing issues and then ask the user to reproduce the issue before collecting the logs.
Scenario: Mail sync issues
Some mail sync issues that users may report include:
- Mail message bodies are truncated or the format is altered when mail is synced or replied to from the device
- Certain attachments do not sync to device
- Some documents, for example mail, calendar items, contacts, or folders, do not sync on the device
- The device hangs when syncing mail
- The IBM Traveler client unexpectedly closes
For these kinds of issues, it is unlikely you will get useful observations from a user dump. As a first step towards resolution, try to replace the data on the device by resyncing it in its entirety. You can run the reset command on the server to initiate a resync of user data. Then enable the Traveler debug for the affected user, recreate the issue, and collect the logs. The NTS Activity and Error logs will help you understand problems for the document causing the issue.
The below sample error shows a failed attempt to retrieve an attachment.
[10/09 20:06:54.452] SEVERE
DS-0048[92EshAAA][2][23DA9CACB55713DB85257BB40073F6B2][35644] Test User/Acme ContentStore.getAttachment#1363
serverName=CN=Test/O=Acme, databaseName=mail/testuser.nsf, and
dcaDocument=NoteId=830726, 0xcad06, wantsAttachments=true,wantsHtmlBody=false, wantsPlainTextBody=true,
itemName=,isSigned=false, m_FullRetrieve=true, m_retrievedAttachments=true,m_fieldMap.size(0),
Attachments(2)=[name=actn130.gif, size=1085,mimeType=image/gif referenceId=actn130.gif@icon
embedded=truestreamable=true hasData=true, name=, size=0, mimeType=imageeferenceId=@ embedded=false streamable=true
hasData=false],UserAttachmentFilterSize=4194304, AdminMaxFilterLimit=4194304,MaxSingleInline=4194304,
MoreInlineAttachmentSize=0,
server=CN=Test/O=Acme,
database=mail/sedgecom.nsf Exception Thrown: java.lang.NullPointerException: Exception occured in ILE native method, see
joblog. java.lang.NullPointerException: Exception occured in ILE native method,see joblog.
Just as an initial troubleshooting, try to narrow down the issue. Check to see if the issue is specific to a certain mail or device type. For any error displaying on the console, try to verify if it has already been reported and if any fix exists. To look for known errors and their fixes, you can check the following URL:
Ref: http://www-10.lotus.com/ldd/dominowiki.nsf/dx/Lotus_Notes_Traveler_APAR_listing