Authors:
Mary Shaheen, Senior Software Engineer, mary_shaheen@us.ibm.com
Steve Murray, Advisory Software Engineer, steve_murray@us.ibm.com
Mary Faherty, Reliability Tester, fahertym@ie.ibm.com
Jonathan Thomson, Staff Software Engineer, jonathan_thomson@us.ibm.com
Introduction:
This document will describe the details of the main System Test Configuration (SVT) for Alloy 1.0.1 deployment, focusing on the IBM System Verification Test cycle testing.
Alloy by IBM and SAP version 1.0.1 introduces server clustering on the IBM/Notes Domino Alloy servers. In System Verification Test (SVT), we configured Alloy server clustering and executed tests within the environment. The configuration included a 3 Alloy Domino server cluster with WebSphere Edge Load Balancer 6.1 component balancing the incoming HTTP connections. Alloy depends on Notes Remote Procedure Calls (NRPC), as well as incoming and outgoing Web Services.
User layout - Mail files:
Two physical AIX machines were used for the user mail. Each physical machine had two Domino Partitions; each Domino Partition was configured to use a unique NIC (Network Interface Card) with a unique IP address.
Machine 1:
Domino Server 1a (DS-1a), Cluster A
User range: mail1-mail1000, replicas of mail1001-mail2000
Domino Server 1b (DS-1b), Cluster B
User range: mail2001-mail3000, replicas of mail3001-mail4000
Machine 2:
Domino Server 2a (DS-2a), Cluster A
User range: mail1001-mail2000, replicas of mail1-mail1000
Domino Server 2b (DS-2b), Cluster B
User range: mail3001-mail4000, replicas of mail2001-mail3000
The total user population was 4000 users. Of the 4000 users, 1000 users were Alloy users, with mail templates customized for Alloy design elements.
The Alloy users were numbered from mail1501-mail2500, which distributed them among both clusters and across both physical machines.
The configuration was established using Domino configuration steps in addition to Alloy configuration steps.
1. Cluster the Alloy Domino servers. Create a Domino cluster containing all intended Alloy servers using the Domino Administration client, Configuration Tab, Server/All Server Documents view. Select the server documents for your Alloy servers, click on the Add to Cluster button and follow the challenges to create the cluster. It may take several minutes or hours for the cluster create task to complete. You may need to change security settings on the replica servers to allow Server access, database creation and database replica creation by the clustermates. (Admin client, Server document, Security Tab, Server Access/Create new replicas)
2. (Optional but advised) Define a private or non-private cluster replication port. For SVT testing, we created a special cluster replication port on a dedicated NIC card installed on each Domino server. Of our 2 independent configurations, one had a private LAN for cluster replication. The other had the cluster replication defined on it specifically, but not configured as a private LAN.
3. Install and configure Alloy seerver on all Domino server cluster mates. The Alloy server installation package is run on each cluster mate. The Alloy runtime includes artifacts inside Domino databases and the Domino Java Virtual Machine (JVM) and therefore must be installed on each participating server. The Domino server document for each clustermate must be modified to allow:
(required) Security: Security, Programability Restrictions/Run restricted LotusScript/Java agents: All Alloy users should be named explicitly or as described in the documentation with wildcards (e.g. */SVTAtantic)
(required) SSO: Internet Protocols, Domino Web Engine, HTTP Sessions/Session authentication, Multiple Servers (SSO)/LTPAToken (All Domino servers should be named in the Web Configuration document)
(required) Allow concurrent Web processes: Internet Protocols, Domino Web Engine, Web Agents and Web Services/Run web agents and web services concurrently, set to Enabled
(recommended) Agent configuration: Server Tasks/Agent Manager: for day time and night time, change the Max concurrent agents to 1, the Max LotusScript/Java execution time to 60 and Max % busy before delay to 70.
4. Create the NDERP Web Service database on one of the cluster mates and create a replica on the other cluster mates. (Server security may need to be modified to allow replica creation)
5. Configure the NDERP Web Service database as described in the documentation, and set clustering specific properties. Including the following:
Document locking enabled:
ACL settings, including the SAP Admin (SAP Workflow/SVTAtlantic) users, and all Alloy server cluster members as having the [Admin] Role and Delete documents:
Within the NDERP Web Service database server configuration document, set 'Is Cluster' = Yes and select the server you want to be the 'Primary' server. The Primary Alloy server is responsible for document locking
* Primary Server definition: The Primary server will be responsible for document locking (Alloy requests, business objects and backend record keeping like Roles and Report template records) and for client NRPC calls for metatdata creation and updates. It is a vital component that does not have failover. Due to its control of document locking and metadata calls, we noticed that the primary server will take on a larger transactional workload than other servers in the cluster. However, at the transaction rate at which we tested, we saw no significant performance impact to the Primary server.
Configure WebSphere Edge Load Balancer to spray the HTTPS traffic across the clustermate's HTTPS servers:
Each clustermate has a loopback network adapter enabled that is configured to point at the cluster address.
The WebSphere Edge Server Load Balancer should be configured to host the HTTP and/or HTTPS ports of all the clustermates, depending on the protocols expected for the Notes clients and the SAP NetWeaver J2EE server to contact Domino.
Sample WebSphere Edge server configuration file: (See attached file: dispatcher.bat)
Configure the Alloy Client Plugin to utilize the cluster:In order for the Notes Standard clients to be able to use the Alloy functionality, there is a Notes sidepanel plugin that needs to be distributed to each client. For the SVT users, the client package was modified and posted to a web site on the Alloy server.
1. The deploy\plugin_customization.ini was edited. The content was replaced with:
com.ibm.nderp.client/NDERPMDWS_URL=
https://yourAlloyClusterFQDN.yourco.com/nderpws.nsf/MetaDataService?openwebservice
com.ibm.nderp.client/NDERP_PrimaryServer=domsvt1020/svtatlantic
NB: NDERPMDWS_URL is required for all deployments and should be directed to the Edge cluster address in the case of a clustered Alloy configuration. NDERP_PrimaryServer is required for clustered deployments and should be directed to the primary Alloy Domino server.
2. The entire client package (with the modified plugin_customization.ini) was zipped and posted to the Alloy Domino server's website (data\domino\html\Alloy_client.zip)
3. Each client user was instructed to download this Alloy_client.zip file, extract it and install it by running the included setup.exe
Once the plugin is installed, when the client launches, the user's metadata is downloaded to the client. The metadata provides information that the client needs in order to provide user interface for things like:
report templates are available to the user
leave balances
workflow and role information that impacts what forms and workflows to expose
Testing
1. Server Workload Description and Result:
An internal tool, similar to the externally available NotesBench test tooling, was used to generate the following workload:
Concurrently, 4 Lotus Notes Standard (8.0.2 and 8.5.1) clients with Alloy 1.0.1 plugin, generated metadata server load with the following profile using AutoIT
Repeat for 12 hours:
Delete existing local metadata
Open client to create local metadata. Pause 15 minutes.
Close client
Repeat 9 times:
Open client to refresh metadata. Pause 10 minutes
Close client
Switch to a second user
Open client to create local metadata. Pause 15 minutes.
Close client
Repeat 9 times:
Open client to refresh metadata. Pause 10 minutes
Close client
Switch to a third user
Open client to create local metadata. Pause 15 minutes.
Close client
Repeat 9 times:
Open client to refresh metadata. Pause 10 minutes
Close client
After 12 hours, the client pauses while open for 12 hours, then resumes. This ran for the duration of the server Workload.
Results:
The server workload was active for 72 hours and no significant stress was seen on Alloy Server CPU or Memory resources.
Attached are the graphs of CPU and memory usage of the most significant processes:
Primary Server: 
Normalized Average Combined CPU of 7.8%
Secondary Server #1
Normalized Average Combined CPU of 7.4%
Secondary Server #2
Normalized Average Combined CPU of 7.6%
* NB: The graphed CPU results are not normalized for the number of processors. To determine the overall % of system CPU used, the result should be divided by the number of CPUs (2).
Average Total Memory used on the Alloy Domino Server was 42%
Much of the CPU and Memory usage is HTTP activity. HTTP is the manner in which the SAP NetWeaver Application Server and the Notes client communicate with Domino.
2. Client Long Run Description and Result:
SVT developed and executed a test designed to simulate typical Notes client user activity over 3 days. The test included common Notes functions, including working with email, and calendaring, and several Alloy tasks each day.
Client Details:
Lotus Notes Standard (8.0.2 and 8.5.1) client, Alloy 1.0.1 plugin,
AutoIt V3
Windows XP Professional Service pack 3
Intel Xeon CPU 2.00GHz
2.00 GB of RAM
Test Description and Client Activity Profile:
AutoIt V3 was used to develop and run the User Interface (UI) automation.
The Notes client activities included the following activities over a 12 hour span at varying rates:
Alloy: Create Trip request
Alloy: Create Leave request
Alloy: Request report
Send new mail
Forward mail
New Calendar entry
Open All Documents view
Empty Trash
Following these activities, the client was left for a 12 hour pause to simulate overnight idle time.
This was repeated 3 times to simulate 3 average working days.
The test ran for 3 days and created:
3 Report requests
1 Trip request
3 Leave requests
496 New mail messages
75 Mail forwards
12 Calendar entries
Results:
Test results were analyzed for significant errors, failures, excessive memory consumption and Graphics Device Interface (GDI) object leaks.
The client was able to remain open for the 3 days without need of restart.
No significant client delays were identified during the test.
3. Client Stress Test Description and Result:
SVT developed and executed a Lotus Notes "Alloy only" operations stress test for 48 hours.
Client details
Lotus Notes Standard client (8.0.2 and 8.5.1), Alloy 1.0.1 plugin,
AutoIt V3
Windows XP Professional Service pack 3
Intel Xeon CPU 2.00GHz
2.00 GB of RAM
Test Description:
AutoIt V3 was used to develop and run the User Interface (UI) automation. The Alloy Notes Client functionality was tested via the Notes client sidebar.
Client activity profile:
68% Create a new report template and run the new report
16% Approve or reject a request
8% Create a trip request
8% Create a leave request
The script ran one user/client generating an Alloy transaction approximately every 2.5 minutes.
The test was allowed to run for 34 hours and created the following Alloy transactions:
559 Reports
135 Approvals/Rejections
67 Trip requests
67 Leave requests
Results:
Setting a server JavaMaxHeapSize of 512MB or 768MB is recommended.
Once this tuning was attained, there were no client errors, failures, excessive memory consumption or Graphics Device Interface (GDI) object leaks for the duration of the test.
No significant client delays were identified during the test. No Client restarts were required to complete the test.
For failover testing, each server was stopped individually and various normal client activities were attempted while each server was unavailable. The results indicated the expected behavior, highlighting dependance on the primary server. A user can expect metadata creations/updates and transaction processing to be interrupted during a primary server outage.
Other Alloy and related information resources: