Introduction

IBM Notes Traveler synchronizes mail, calendar, to-do, address book, and journal data in real time. The Notes Traveler server runs as a Domino server task and is installed on a server running Domino. The Notes Traveler server task uses the Domino directory to automatically look up and find Notes users.

Notes Traveler 9.0 Server High Availability

Notes Traveler 9.0 in High Availability (HA) configuration has the ability to run multiple Notes Traveler servers in a service pool, thereby allowing devices to connect to any server in the pool.

Requests are received at any Notes Traveler server, and a determination is made whether to service the request locally or to route the request to another server in the pool. This decision is based on which server is currently monitoring mail for the user and on server loads. If a server in the pool is stopped or otherwise unavailable, device requests can be handled by the other servers in the pool without requiring recovery sync.

Notes Traveler 9.0 in HA configuration uses Enterprise Database Servers for HA. IBM DB2® is used for storing Notes Traveler Notes administration data and device synchronization data. An enterprise database is not required when running Notes Traveler on a single server.

In this article we present the performance test results of Notes Traveler 9.0 in HA configuration.

Testing

Let's begin with the details of our testing.

Setup

For this study the test configuration was as shown on figure 1 and as outlined in table 1.

Figure 1. Test configuration




Table 1. Test configuration hardware and software

Function	CPU	Memory	Storage	Software	Quantity
Load balancer Physical Machine	2x2.4 GHz (24 logical processors)	48 GB	597 GB	IBM Edge Server 5.1 with Load Balancer on Win 2008 R2 64-bit	1
Mail server Physical Machine	2x2.56 GHz (16 logical processors)	32 GB	One Internal 600-GB logical unit number (LUN) and three external 1-TB LUNs per server	Domino 8.5.3 FP1 on Win 2008 64-bit	3
Traveler Server VMware	4x2.8 GHz	8 GB	6-disk 860- GB LUN	Traveler 9.0 HA and Domino 9.0 FP1 on Win 2008 R2 64-bit	4
DB2 Server Physical Machine	1x2.8 GHz ( 6 logical processors)	64 GB	Data volume: 8-disk Redundant array of independent disks (RAID) 5 LUN Transaction log: 4-disk RAID 5 LUN	DB2 V9.7 FP5 OS: Red Hat Enterprise Linux (RHEL) 5.7 64-bit	2

Test Methodology

The test strategy was to simulate a typical enterprise-customer messaging load and then add mobile devices to that environment. The Notes Server.Load workload generators simulated Notes client activity by sending mail to random users in the test set and causing other activity on the Domino mail server.

We added mobile devices into this mix, using the device simulators, which keep the changes generated by Server.Load in sync with the client and simulate mobile-user actions on the device.

The performance of the Notes Traveler server and the Domino Mail server were measured by use of the Notes Traveler and Server.Load N8Mail workloads on a Windows 2008 64-bit server operating system.

The Notes Traveler server pool consisted of four Traveler servers, two database servers configured for HA (using RDBM databases, DB2 9.7 FP5), and one IBM Load Balancer.

In addition, we used three Domino Mail servers in a cluster, using Domino clustering, with identical replicas of the Domino Directory Names and Address Book (NAB) with 18,000 users.

Notes Traveler servers were exercised by test systems that simulated mobile device clients. These driver systems simulated remote devices reading and sending mail messages through a load balancer using HTTP (not HTTPS).

Here are the details the Notes Traveler client script:
There are 64 iterations of this loop with a 15-min wait between iterations. Each user starts on a random iteration count (1st, 5th, etc.), and each user runs for the entire 64 iterations:

Loop start

• OPEN/READ five Inbox mails (every iteration)
• Reply to the first mail (every 2nd iteration)
• SEND one 10-byte mail to one person (every 8th iteration)
• SEND another 10-byte mail to three people (every 8th iteration)
• MOVE one mail from the Inbox to the NotesBench folder (every iteration)
• DELETE two mails (every iteration)

Loop end

In addition to the user scripts above, the Notes Traveler clients are listening for auto-sync (push) messages from the Notes Traveler server. If mail or calendar data is changed on the server as a result of the Server.Load load drivers, then the Notes Traveler client will automatically synchronize the server changes to the device.

The load driver for Notes Traveler simulates Windows Mobile (WM) 6, Nokia, Apple and Android devices. Windows Mobile, Nokia, and Android devices use HTTP-based auto sync, whereas Apple devices use the Exchange ActiveSync synchronization protocol and have different performance characteristics than the WM, Nokia, and Android devices.

For Notes Traveler 9.0, a mixed workload with 50% WM6/Nokia/Android devices and 50% Apple devices was used.

Domino 8.5.3 FP 1 mail server workload

The Server.Load N85Mail workload that was run on the Domino 8.5.3 FP 1 mail server simulated users sending mail using a Notes 8.5.3 client, and one-third of the N85Mail workload was sent against each of the Domino mail servers.

This workload is part of the Server.Load workload tools suite (included as an install option in the Administrator Client). For more information on the N85Mail workload, refer to the developerWorks article, “IBM Lotus Notes V8 workloads: taking performance to a new level.”

For these measurements, the N85Mail workload simulated 2,000 Notes clients sending mail to up to 12,000 users (2000 + the number of mobile device users). The additional users (up to 10,000, or as otherwise noted) were the Notes Traveler registered users. These measurements simulate a customer environment in which mail users are using either a Notes client or a Notes Traveler device, and not both at the same time.

DISCLAIMER: All our tests are lab measurements; therefore the results may differ from real customer environments. For assistance with capacity planning, we recommend consultation with the IBM TechLine team, which does capacity planning for new hardware.

Domino mail server configuration parameters

Table 2 lists the software configuration for the Domino mail server.

Table 2. Software configuration

Notes.ini settings
Used for all tests	server_pool_tasks=200 server_max_concurrent_trans=200 Show_Server_Performance=1 RouterDbCacheSize=6100 NSF_DBcache_maxentries=6000 SCHEDULE_NO_VALIDATE=1

The mail server is running Domino Server 8.5.3 FP1 (32-bit) set up as a single Domino partition server in accordance with the data in table 1.

IBM Notes Traveler server configuration parameters

Notes Traveler Server 9.0 (64-bit) was set up in accordance with the data in table 1.
Table 3 shows the software configuration for the Notes Traveler Server 64-bit in HA configuration.

The Notes Traveler server must be tuned for the number of active users that it supports. The setting we changed for this test is indicated in the table by an asterisk. For more details, refer to the “Tuning performance of the server” topic in the Notes Traveler 9.0 product documentation.

Table 3. Software configuration for Notes Traveler Server 64-bit

Notes.ini settings
Used for all tests		NSF_Buffer_Pool_Size_MB=200
NTS_DBMAXPOOLSIZE=1001
HTTP threads		1.2* number of devices

Results

Notes Traveler 9.0 with DB2 Database Servers

Figure 4 shows the CPU resource utilization for Notes Traveler Server 9.0 (64-bit) on 64-bit Windows 2008 at different simulated device loads with two DB2 Database Servers in an HA configuration.

The CPU utilization at 8,000 active devices was 48%, and at 10,000 active devices was 54%, while at 12,000 active devices the CPU utilization was 59%.

Figure 4. CPU utilization for each Traveler 9.0 Server in the server pool



Figure 5 shows DB2 Database Server sustained disk IOPs for different simulated user loads.

Figure 5. IOPs of the Database Server

Recommendations

For Notes Traveler Server HA performance:

• VMware instances can be used for the Traveler servers.
• To avoid having a single point of failure, each Traveler server in the Traveler server pool should be on a different VMware ESX server.
• Recommend up to 3,000 devices maximum per Traveler server:

• Maximum number of 3000 devices is after failover
• Plan accordingly so as not to exceed 3000 devices per Traveler server after failover

• Recommended server resources: Four CPU cores or VMware CPUs and 16 GB RAM for each Traveler server in the pool.
• Use 64-bit operating system, Notes Traveler, and Domino
• Place the Traveler servers and the database servers all in the same data center. Configurations with the Traveler servers and the database servers in different data centers are not supported.

Guidelines for determining the number of Traveler servers needed for HA:

Formula: (# of devices/ (n-1)) < 3000 devices,
where n is the # of Traveler servers needed for failover planning

• For 12000 devices with planning for up to one offline server:

• For 12000 devices with planning for two offline servers:

For Notes Traveler Database Server performance:

• The database servers and storage were physical for all performance measurements; physical servers for the database servers are recommended for large enterprise deployment.
• Data and Transaction Log volumes should be on separate storage hardware RAID LUNs for best performance.
• Data and Transaction Log volumes should be able to support the sustained IOPs required.
• For a database server supporting up to 12,000 devices in a Traveler Server pool, we recommend:

• Eight CPU cores, 32GB RAM, and 1,200 sustained IOPs is recommended.
• A dedicated (not shared with other applications) hardware RAID 5 (or better) LUN with 8 disks (10,000 to 12,000 devices) for the Traveler database.
• A dedicated (not shared with other applications) hardware RAID 1 (or better) with 4 disks for Transaction Log.
• The LUNs be dedicated; the storage system can be shared.

• For up to 8,000 devices, six disks are needed for the Traveler database LUN.
• Database servers should use DB2 HA (requires two database server).
• DB2 V9.7 FP5 or later is required.

Table 4 lists the hardware requirements per database server in the HA configuration.

Table 4. Hardware requirements per database server

Number of devices in Traveler Service Pool	CPU Cores	Memory (GBs)	Disk drives for Traveler DB	Disk drives for Transaction Log
Up to 4,000	4	16	6	4
4,000 to 6,000	4	16	6	4
6,000 to 8,000	4	16	6	4
8,000 to 10,000	8	32	8	4
10,000 to 12,000	8	32	8	4

For the Traveler 9.0 in HA configuration to attain the sustained IOPs in the measurements done in the performance lab, the number of disk drives listed in the table above was required.

It is important to note that the storage system used for the database servers should be able to support the sustained IOPs required as listed in figures 3. For example, if using a high-end storage such as the IBM XIV® Storage System, then the storage LUNs should be configured for the required sustained IOPS. Notes Traveler 9.0 in HA configuration does not support Traveler Pool and Database servers in remote data centers.

For load-balance performance:

• An IBM Edge Server with Load Balance was used for the performance measurements, only as a dispatcher
• Hardware recommendation:

• Two CPU cores
• Eight to 16 GB RAM, depending on the functionality used on the Edge Server

• At 10,000 devices, load connections/sec was 200

For Notes Traveler DB2 Database configuration:

• LOGBUFSZ = 8192
• LOGFILSIZ = 8192
• MINCOMMIT = 1
• Number of primary log files (LOGPRIMARY) = 120
• Number of secondary log files (LOGSECOND) = 10
• Traveler DB2 Database Bufferpool can be set to 1--2 GB based on available physical RAM on the server and the user load

• For example, for a Traveler pool supporting 8,000 devices, a Bufferpool up to 2 GB is recommended on a DB2 server running with 8--16 GB RAM.

For Notes Traveler 9.0HA Server performance:

• Notes.ini settings for DB2 connection:

NTS_DBMAXPOOLSIZE=1001


• The Traveler HTTP threads setting should be left the same as what was recommended for Notes Traveler 9.0, based on the number of devices per server.
•
• The Traveler Java^TM heap setting should also be left the same as what was recommended for Notes Traveler 9.0.
For example, we use 4096-MB heap size per Traveler server supporting around 2500 devices. You can set the Java Max memory on the Traveler tab of the Server document.

Conclusion

This article details the performance test results, deployment recommendations, and tuning of Notes Traveler 9.0 in HA configuration on Microsoft Windows 64-bit with two IBM DB2 in HA configuration. For Traveler 8.5.3 UP 1 and UP2 in HA we recommended 2,500 devices per server for Traveler 9.0 we recommend 3000 devices per server, an increase of 20%. Also in Traveler 8.5.3 UP 1 and UP2 in HA , we recommended 10,000 devices per server pool for Traveler 9.0 we recommend 12,000 devices per server pool, an increase of 20%.

Resources

Announcement: “IBM Notes Traveler 9..0”:
http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=an&subtype=ca&appname=gpateam&supplier=897&letternum=ENUS212-220


Notes/Domino wiki article: “Configuring IBM Lotus Notes Traveler High Availability with an an enterprise IBM DB2 database”:
http://www-10.lotus.com/ldd/dominowiki.nsf/dx/Configuring_IBM_Lotus_Notes_Traveler_High_Availability__with_an_enterprise_IBM_DB2_database

Appendix 1: Notes.ini settings

Notes.ini parameter name	Explanation
Create_R8_Databases	Applies to Domino 8 servers. Set it to 1 to create new ODS48 databases on Domino 8 servers.
NSF_Buffer_Pool_Size_MB	Specifies the maximum size (in bytes) of the NSF buffer pool, a section of memory dedicated to buffering I/O transfers between Domino and disk storage.
NSF_Dbcache_Maxentries	Specifies the number of databases that a server can hold in its database cache at one time.
RouterDbCacheSize	Specifies the size of Router Database cache used to cache mail databases in the router process.
SCHEDULE_NO_VALIDATE	Enables or disables whether SchedMgr validates its busytime database entry on a daily basis, as follows: 0 (default) - Enables validation 1 - Disables validation
Server_MAX_CONCURRENT_TRANS	Sets the limit for the number of concurrently scheduled transactions on a server
Server_Pool_Tasks	Total number of physical threads in the Domino server (DbServer) threadpool.
Server_Show_Performance	Applies to servers. Specifies whether or not server performance events are displayed on the console. Set this variable to 1 to display server performance events on console.

About the authors

Yuriy Veytsman is an Advisory Software Engineer and has been working for Lotus/IBM since 1990. You can reach him at yuriy_veytsman@us.ibm.com.

Razeyah Stephen is the Manager of the Domino and Traveler Performance team and has been working at IBM since 1998. You can reach her at razeyah@us.ibm.com.

Nirmala Venkatraman is the Architect for the Domino and Traveler Performance team and has been working at IBM since 2000. You can reach her at nvenkatr@us.ibm.com.