Introduction

An IBM® Lotus® Enterprise Integrator (LEI) cluster lets you have continuous access to data from different database servers (Oracle, IBM DB2®, ODBC, etc.). This article helps you understand LEI clustering and how you can make use of the benefits provided by the cluster, explaining how to set up an LEI cluster and how it works in different scenarios.

What is clustering?

A cluster is a group of two or more servers that provides users with constant access to data, balances the workload between servers, improves server performance, and maintains performance when you increase the size of your enterprise.

The servers in a cluster contain replicas of databases that you want to be readily available to users at all times. If a user attempts to access a database on a cluster server that is not available, IBM Lotus Domino® opens a replica of that database on a different cluster server---if a replica is available---and continuously synchronizes databases so that whichever replica a user opens, the information is always the same.

LEI clustering

Scenario A. An LEI clustered environment is one in which you have one LEI Administrator controlling multiple LEI servers. This is distinct from a Domino cluster that keeps replica databases on clustered
Domino servers synchronized for load balancing and failover. The benefit of an LEI cluster is
the ability to have a single point of administration for multiple LEI servers. A cluster also includes the LEI log and the optional script vault database, documentation databases, and sample databases.
Scenario B. LEI servers that are remotely administered cannot be used for Advanced RealTime activities. In an LEI clustered environment the only server that is able to operate Advanced RealTime activities is the LEI server that hosts the LEI Administrator database, decsadm.nsf,
operating locally.

If a remotely administered server has the Advanced RealTime checkbox enabled (see figure 1), then all Advanced RealTime Activity documents are likely to have difficulty being started. Hence, for Advanced RealTime activities, the LEI Administrator cannot be cluster-replicated; each server must host its own non-replica copy of the LEI Administrator

Figure 1. LEI Server Configuration document

LEI Clustering requirements

For Scenario A, the failover capability requires some pre-installation server configuration. To enable failover for data management activities, use these steps:

1. Ensure that cluster replication is enabled on the Domino server for LEI decsadm.nsf database.
2. Activate cluster mode, and optionally failover mode, using the LEI Administrator's Configuration document. The four relevant fields are as follows:

• Domino Clustering checkbox, to enable cluster mode
• Batch Activity Failover checkbox, to enable failover mode
• Status Broadcast interval; it may take at least this long for other servers to notice a crash.
• Synchronization delay, to set the expected time (in seconds) for replication from one LEI Administrator database to all others in the cluster to occur.

Figure 2. LEI Administration database Configuration document


Figure 3. Server Time Check window


3. Ensure that all servers in the LEI cluster contain a replica of the LEI Administrator (descadm.nsf) and the LEI log (leilog.nsf) databases.

4. Ensure that the EIAdminServer variable in the Notes.ini file on each server in the LEI cluster points to itself. For example, if servers A, B, and C are to be part of the same cluster, the Notes.ini EIAdminServer variable for server A should be set to "A"; on server B, the variable would be "B", and so on. Previously there was only one EIAdminServer allowed.

5. Specify the Failover Servers in the Activity document to be used. If nothing is specified in this field and the Designated Server field does have an entry, then there will be no failover (see figure 4).

Figure 4. Activity execution options

Benefits of LEI clustering

For Scenario A under Section 2, LEI supports a failover mode of operation that lets data management activities run on one of a designated set of servers. Each LEI instance uses its local replica; there is no master LEI server. This prevents failure of the entire cluster if one Domino server crashes.

Each server in a cluster periodically polls the LEI log to determine if another server is down, based on the server's Status Broadcast Interval setting in the Server document. When a server detects that another server is down, it searches for that server's eligible activities and then runs them as a failover.
The Batch Activity Failover field lets LEI fail over these types of activities (everything except the RealTime activity).

With failover, you can designate a server on which the activity normally runs, but in the event that server fails, another LEI server can temporally run the activity.

Configuring an LEI cluster

For Scenario A, the only method of configuring an LEI cluster is through the installation procedure. In the installation process for LEI, a window displays requesting the name of this LEI server and on which Domino server to create the LEI Administrator database. If this is the first LEI server being established and will be the server from which to administer the LEI servers, then select the second option (see figure 5).

Figure 5. LEI Installation Screen for first LEI Server in cluster



If this is an additional LEI server in an already-existing cluster, then select the third option and specify the Domino server name. Before clicking Next, ensure that the Domino server is running and that, if LEI is active on that server, that LEI is stopped (see figure 6).

Figure 6. LEI installation screen for additional LEI Server in cluster



For Scenario B, the LEI Administrator database cannot be cluster-replicated, so in install the LEI servers by selecting the first option, “This LEI Server is not part of an LEI Failover Cluster.”

How LEI clustering works for Batch activities

As mentioned in Section 2 (Scenario A), the LEI Administrator database must be cluster-replicated to servers in a Domino cluster, and each LEI instance uses its local replica. There is no master LEI server, so as to prevent failure of the entire cluster if one Domino server crashes. All Activity documents that support failover contain a field for the administrator to list zero or more failover server names.

When a server detects that another server is down, it searches for that server's eligible activities and then runs them as a failover. To prevent conflicts, failover servers are responsible for running an activity in the order in which they are listed in that activity. The first active server in the list runs the activity. You can activate cluster support without also activating failover.

When running LEI in failover mode, if you want to shut down the Domino server, shut down the LEI server first. This allows time for the exit status of each running activity to replicate to the other servers in the cluster.

Failover scenario for scheduled Batch activities

In this scenario, Domino Server A and Domino Server B are clustered. Server A is the Primary Server and Server B is the Secondary Server, and Server B has a replica of decsadm.nsf from Server A:

1. Use any backend database (DB2, Oracle, ODBC, Sybase, or SQL Server).
2. Create an IBM Lotus Notes® database on Server A, making sure to have replica of this database on Server B.
3. Create a Connection document to the backend on LEI Server A.
4. Create a Connection document to the Notes database on LEI Server A.
5. Create a scheduled Batch activity, for example, Data Transfer on Server A, making sure to enter Designated and Failover server details as well.
6. Ensure that the activity runs on Server A, as per schedule.
7. Shut down Server A; the activity will have failed over to Server B, and Server B's server console displays as shown in figure 7.

Figure 7. Server console message



8. Start Server A; the activity will fail back to Server A (see figure 8).

Figure 8. Server console message for failback

How LEI clustering works for RealTime activities

As mentioned in Section 2 (Scenario B), the LEI Administrator cannot be cluster-replicated; each server must host its own non-replica copy of the LEI Administrator. Virtual Documents activities and Virtual Fields activities support Domino clusters but to a varying degree. Virtual Fields can fully support the failover and load-balancing capacity provided by a Domino cluster. Although Virtual Documents will work in a Domino cluster, it currently does not take full advantage of these capacities.

Virtual Fields activity in a clustered environment

Each LEI server in the Domino cluster must have its own Virtual Fields activity, to run locally. They may all share the same RDBMS table. The Virtual Field activity is smart enough to know that, if a record is changed on Cluster Server A, it does not need to be changed on any other server in the cluster (as long as a Virtual Fields activity is running for that database and form on the other servers).

The easiest way to set this up is to create the Virtual Fields activity once, copy and paste that Activity document once for every server in the cluster, and then edit each activity and set the correct designated server field.

For Virtual Fields activities to fully participate in a Domino cluster, each server in the cluster must be set up with a Virtual Fields activity(s) that monitors the replica database(s) on that server. In other words, each server in the cluster must have its own Virtual Fields activity(s) running against its local replica databases. Each server in the cluster must maintain its own copy of the LEI Administrator, with a copy of the activity(s) set up and running from each LEI Administrator database.

All the Virtual Fields activities can be set up to use the same external data source. If set up in this fashion, cluster failover and load-balancing support derives from the fact that each Virtual Fields activity has access to---and can update---the external data. Even if all servers but one were to go offline, the external data would continue to be accessible and could be kept current. As other servers come back online, the current external data would be instantly available to those servers directly from the external data source.

Virtual Fields activities work by loading a management module for each process through the EXTMGR_ADDINS parameter in the Notes.ini file. This intercepts database operations such that, when the activity is started, real-time requests are addressed through operations on the external database.

To operate correctly, Virtual Fields activities must be started from the LEI Administrator on all servers, to intercept real-time events in all replicas. When this is done, Virtual Fields activities intercept both the Domino database read on the source server and the database write on the Domino target servers.

To prevent unwanted duplication of inserts, updates, or deletes to the external database when the Domino databases are replicated from clustered server to clustered server, Virtual Fields activities add additional information to the records as they are replicated, to indicate that the requested action has already been done. This information does not alter the documents.

NOTE:

• On clustered Domino servers, the Virtual Fields activity and Connection documents must be identical on all the servers. The least error-prone way to do this is to copy and paste from one to another. It is also recommended that after you paste the documents, open them for editing, press , and save each document to ensure that all computed fields are correct.

• If one of the parallel Advanced RealTime activities is not running, this protection may fail and duplicate records may be added to the external database. For example, inserts may be attempted in violation of unique constraints, or deletes may fail because the record has already been deleted.

• A Virtual Document activity with external key columns is not supported in a cluster environment.

5.1.1 Scenarios for a Virtual Field activity in a clustered environment

Scenario 1. In this scenario Domino Server A and Domino Server B are clustered, and Server A is the Primary Server, and Server B is the Secondary Server:

1. Make sure that the LEI Administrator database (decsadm.nsf) is not cluster-replicated on Server A and Server B.
2. Create a Notes database on Server A (Replica 1), making sure to have a replica of this database on Server B (Replica 2).
3. Create Virtual activities on both servers with the same RDBMS Table and the just-created Notes database.
4. Initialize Keys for the Virtual activity on Primary Server A; the key values display on the Notes database on Server A and the same will be replicated to Server B.
5. Start the activity on both servers and open the Notes database from Server A and Server B; the data will be accessible from both databases.
6. Shut down Primary Server A; backend data is still accessible from Server B.

NOTE: The "SYNCHRONIZE KEY DOCUMENTS" feature is currently not supported in LEI clusters. There is already an Enhancement request raised for this feature SPR #TAIA8576Z5.

Figure 9 shows a flow diagram illustrating how Domino clustering works in conjunction with LEI virtual fields as per Scenario 1.

Figure 9. Scenario 1 flow diagram


Scenario 2. Here, Domino Server A and Domino Server B are clustered, and Server A is the Primary Server, and Server B is the Secondary Server. Make sure that the LEI Administrator database (decsadm.nsf) is not cluster-replicated on Server A and Server B.

1. Use any backend database (DB2, Oracle, etc) and two identical tables, one of which is populated with data (Table 1), and the other is empty (Table 2).
2. Create a Notes database on Server A, making sure to have a replica of this database on Server B.
3. Create a Connection document to backend Table1 on LEI Server A.
4. Create a Connection document to backend Table2 on LEI Server B.
5. Create a Virtual Field activity on LEI Server A, using the Notes database and Table 1.
6. Create a Virtual Field activity on LEI Server B, using th replica of Notes database and Table 2.
7. Disable cluster replication in the Cluster Directory for LEI Server A for the Notes database.
8. Initialize keys for the Virtual Field activity on LEI Server A.
9. Start Virtual Field Activity on both servers i.e. LEI Server A and Server B
10. Enable cluster replication in the Cluster Directory for LEI Server A for the Notes database.
11. Issue this command on the Domino console of LEI Server A: “replicate ”
12. Open the Notes database on Server A, select a document, and open it. Open the Notes database on Server B, select a document, and open it.
13. Data in the Notes database on Server B and Table2 will be populated with the same data as Table1.

Figure 10 shows a flow diagram illustrating how Domino clustering works in conjunction with LEI Virtual Fields as per Scenario 2.

Figure 10. Scenario 2 flow diagram

Virtual Documents activity in a clustered environment

Virtual Documents activities are significantly different from Virtual Fields; for example, it is not possible for more than one Virtual Documents activity to use a given RDBMS table. The best solution here is to have a single Virtual Documents activity running on one LEI server in the cluster. For the other servers in the cluster, use full NSFs (in other words, "Not virtual").

This does mean that the connection managed in the Virtual Documents activity is a single point of failure; no data would be lost because users could access replica databases on other servers. When the Virtual Documents server comes back online, the changes would then be sent to the RDBMS system.

The main difference between Virtual Documents support for Domino clusters and Virtual Fields support for Domino clusters stems from the limitation that a single external data table can map to only a single Domino database/form. Though it is possible to have a Virtual Documents activity monitoring each replica database in the cluster, you cannot map each of those instances to the same external table in the manner that you would typically do with Virtual Fields.

As a result, the typical configuration calls for selecting a single Domino server in the cluster that will manage the virtual documents and the external data connectivity.

A Virtual Documents activity monitors a replica database and form on this one server, mapping its external data. This activity is responsible for all external data access and updates for the cluster. Other replicas in the cluster see this virtual data and replicate it throughout the cluster in the normal manner, but it will be held natively in the other replica databases.

Only one replica in the cluster runs a Virtual Documents activity and "contains" virtual documents. The other replicas contain native replica documents. In this topography, the external data source becomes just another node in the cluster, and can be detached from the cluster if the server running the Virtual Documents activity becomes disconnected.

However, like any other node, data changes made to cluster replicas are propagated to the server---and ultimately to the external data source---when the server comes back online.

5.2.1 Scenarios for Virtual Documents activity in a clustered environment

Scenario 1
In this scenario, Domino Server A and Domino Server B are clustered; Server A is the Primary Server, and Server B is the Secondary Server:

1. Make sure that the LEI decsadm.nsf database is not cluster-replicated on Server A and Server B.
2. Create a Notes database on Server A, making sure to have a replica of this database on Server B.
3. Create a Virtual Documents activity on Server A with the RDBMS Table and the just-created Notes database
4. Start the Activity on Server A; records populate in both replicas, with Server A haing virtual data and the replica copy on Server B having actual data.
5. Shut down the first server; data is still accessible because the replica copy has actual data.
6. Insert data in the Notes database on Server B.
7. Start Server A and the Virtual Documents activity on it; added records will be in Server A's Notes database and in the backend database also.

Figure 11 shows a flow diagram illustrating how Domino clustering works in conjunction with LEI virtual documents as per Scenario 1.

Figure 11. Scenario 1 with LEI virtual documents flow diagram


Scenario 2
Here, Server A and Server B are clustered; Server A is the Primary Server, and Server B is the Secondary Server. Make sure that the LEI Administrator database (decsadm.nsf) is not cluster-replicated on Server A and Server B.

1. Use any backend database (DB2, Oracle, etc), and use two identical tables, one of which is populated with data (Table1) and the other is empty (Table2).
2. Create a Notes database on Server A, making sure to have a replica of this database on Server B.
3. Create a Connection document to backend Table1 with internal key columns on LEI Server A.
4. Create a Connection document to backend Table2 with internal key columns on LEI Server B.
5. Create a Virtual Documents activity on LEI Server A, using the Notes database and Table1.
6. Create a Virtual Documents activity on LEI Server B, using a replica of the Notes database and Table2.
7. Disable cluster replication in the Cluster Directory for LEI Server A for the Notes database.
8. In the backend, clear out keys in Table1, as follows:

9. Turn on Virtual Documents activity on both Server A and Server B.
10. Enable cluster replication in the Cluster Directory for LEI Server A for the Notes database.
11. Issue this command on the Domino console of LEI Server A: “replicate ”
12. Open the Notes database on Server A, select a document, and open it.
13. Open the Notes database on Server B, select a document, and open it; data in the Notes database on Server B and Table2 will be populated with the same data as Table 1.

Figure 12 is a flow diagram illustrating how Domino clustering works in conjunction with LEI virtual documents as per Scenario 2.

Figure 12. Scenario 2 with LEI virtual documents flow diagram

Conclusion

You should now understand how to take advantage of the benefits provided by LEI clustering, and how to set it up for Batch and Virtual activities. We discussed the different scenarios for LEI clustering and how it works in these scenarios.

Tell us what you think

Please visit this link to take a one-question survey about this article:

http://www.surveymonkey.com/s/9Q6ZKGN

Resources

• Understanding IBM Lotus Domino server clustering
• Installation guide for Lotus Enterprise Integrator
• LEI User Guide
• IBM Lotus Domino 8.5 Administrator information center
• developerWorks Lotus Notes and Domino product page

About the author

Reetu Sharma is a Staff Software Engineer based at IBM’s Software Lab in Pune, India, currently working as a Quality Engineer. She has extensive experience in LEI, LSXLC, DECS, Domino Server, and Notes Client. You can reach her at reesharm@in.ibm.com.