ShowTable of Contents
Introduction
Introduced in IBM® Lotus® Domino® 8.5, the Domino Attachment and Object Service (DAOS) feature allows attachments to be stored outside the NSF file in Notes Large Object (NLO) files. This provides two major benefits in that it:
Allows multiple copies of the same attachment to be stored as a single copy to save storage.
Segregates the relatively static attachment data from the NSF data, allowing flexibility in data storage and backup processing.
In databases that use DAOS, Lotus Domino no longer saves a separate and complete copy of every document attachment. Instead, the server saves a reference to each attached file in an internal repository (DAOS Catalog), and it refers to the same file from multiple documents in one or more databases on the same server.
When a message containing an attached file is large and is broadcast to thousands of users, creating a separate copy of the message for each recipient could require several gigabytes of disk space. Also, multiple copies of the same attachment are often proliferated in mail threads with multiple replies.
With DAOS enabled, disk space usage is substantially reduced, and you can realize these additional advantages:
- Optimized mail routing of attachments
- Optimized object copying for any same-server copy operations
- Faster compacting and backing up of databases
IBM’s Domino System Verification Test (SVT) team performs extensive testing of the DAOS feature under load, to ensure that it works in a stressed environment. This article discusses some of the statistics you can use to ensure DAOS is running correctly, provides a few tips on how to prevent corruption, and describes how the SVT team configures DAOS in their environment.
Typical environment in SVT
The configuration used within the System Verification Team is as follows (see table 1):
- Number of mail files = 3000 mail files across one Domino Partition (DPAR)
- Size of attachments = 50 KB and 10 MB
- Number of NLO’s = 40,000
- Mail files and the DAOS directory containing NLOs reside on a Storage Area Network (SAN)
Table 1. SVT environment details
System | IBM System z990 – 2084-D32 |
Processor | 64-way processor |
Memory | 6023MB |
LPAR | CPUs: 8 Shared
Memory: 64G Central storage, 16G Expanded storage |
Guest | CPUs: 2 Virtual
Memory: 6GB |
Fiber channel adapter | 18 Shared 2-GB FICON adapters |
Disk drive | 180 emulated 3390-9 and 16 emulated 390-3 on multiple DS8000 (roughly 1TB) |
First-level operating system | ZVM 6.1 |
Guest operating system | RedHat Linux® (64bit) Release 6.1 |
Lotus Domino server | Domino 8.5.4 Build for zLinux (64bit) |
Configuring the server and mail files to support DAOS
Before we start with the scenarios, we need a basic understanding of how DAOS works, how to enable it, etc.
Enabling DAOS on the server
1. In the Basics tab of the server’s Server document, set “Transaction logging” to Enabled (see figure 1).
Figure 1. Enable Transaction logging
2. In the DAOS tab of the Server document, enable DAOS with the settings shown in figure 2.
Figure 2. DAOS Settings
where:
- The “Minimum size of object before Domino will store in DAOS” field is set to 4096 for testing purposes only and is significantly lower than it should be for a typical production environment. Setting the minimum participation size that low in a production system would cause too many NLO files to be created, which would make the system difficult to manage.
- Setting the “Store file attachments in“ field to DAOS (that is, the DAOS repository will be stored under the Data directory) in a production environment can cause utilities to unnecessarily traverse the DAOS repository looking for NSF files. If there are a significant number of NLO files, there can be a significant performance impact on those utilities, so it's recommended they be in a location other than under the Domino data directory.
3. From the server console, issue a “show server” command, to report the status of DAOS.
4. By default the new DAOS folder is created under the Data folder; however, we can change that by specifying the directory under the DAOS tab in the Server document (see figure 3).
Figure 3. DAOS tab
A DAOS Catalog (daoscat.nsf) will be created under the Data directory (see figure 4).
Figure 4. DAOS Catalog listed in Data directory
Enabling DAOS on mail files
To enable DAOS on an individual mail file, run the command, “load compact –c mailfile.nsf –daos”. Note that compact can also be run against multiple mail files, using “load compact –c mail –daos on”.
Important DAOS commands
Some of the important DAOS commands and what they do are listed below:
Tell DAOSMgr Status. Displays status of various DAOS Manager Operations.
Tell DAOSMgr Status Catalog. Displays status of DAOS Catalog.
Tell DAOSMgr Resync. Resynchronizes DAOS-enabled databases with DAOS objects in the storage repository. This command won’t work if the DAOS Catalog is in a synchronized state.
Tell DAOSMgr Resync Force. Runs the resynchronization command, regardless of whether the DAOS catalog is in a synchronized state.
Tell DAOSMgr Resync quick. Rebuilds the DAOS Id Table (DIT) and DAOS Object Index (DOI) but will not scan the databases, where:
- DIT is a list of all NSF files participating in DAOS, containing the total number of DAOS references for each NSF,and the timestamp of the last DAOS activity in the NSF.
- DOI is a list of all NLO files in the DAOS repository, containing the total number of DAOS references to each NLO.
Tell DAOSMgr Resync Force quick. This command is a combination of the above two; that is, it forces the resynchronization process, even though Catalog is in a synchronized state and just rebuilds the DIT and DOI.
Table 2 lists the various Catalog states with their brief descriptions.
Table 2. Catalog states
UNAVAILABLE | DAOS Catalog is not available. |
REBUILDING* | The DAOS Object Index (DOI) is being rebuilt. |
RESYNCING | Databases are being scanned for DAOS references. |
SYNCHRONIZED | All DAOS-enabled databases are in a known state. |
NEEDS_RESYNC | There is at least one DAOS-enabled database in an unknown state. |
Table 3 lists the various Database states with their brief descriptions.
Table 3. Database states
Resync: Pending | Database is waiting to be scanned for DAOS tickets. |
Resync: Scanning | Database is currently being scanned for DAOS tickets. |
Resync: Updating Refs | Database's tickets are being processed, DAOS references incremented. |
Synchronized | Database is in a known state, all DAOS tickets counted in the catalog. |
Deleted | Database has been deleted and is being scanned for DAOS tickets. |
Deleted: Updating Refs | Deleted database's tickets are being decremented in DAOS catalog. |
Adopt: New DB* | A new database has been discovered and is waiting to be scanned. |
Test scenarios & evaluation criteria
In this section we discuss the scenarios that are tested by the SVT team to verify DAOS is running correctly.
Scenario 1: DAOS Resync time window (Performance test)
This scenario tests that Resync picks up where it left off when resync is confined to a time window.
1. First, we must add the following settings to the server’s Notes.ini file:
DAOS_LOGGING. This parameter links to a path of a .txt file, which will be edited if there is a change in the DAOS state, for example, changing from SYNCHRONIZED to NEEDS RESYNC. (Edit the path of file, depending on requirement)
DAOS_RESYNC_START_TIME. The time cited in this parameter indicates the start time of DAOS resync window.
DAOS_RESYNC_END_TIME: The time cited in this parameter indicates the end time of the DAOS resync window.
2. Before starting the test, we must also start performance monitoring (show stat & Open session) tools.
3. Apply load to some mail files on the server.
4. Issue a “tell daosmgr resync quick force” a few hours before the Resync window opens (see figure 5).
Figure 5. “tell daosmgr resync quick force” command
5. Issue a “tell daosmgr status” and confirm DAOS is resync'ing (see figure 6).
Figure 6. “tell daosmgr status” command
6. When quick resync is complete, issue a “tell daosmgr status“. DAOS will still be in a resync'ing state, which is expected. Before the resync window opens, issue a “tell daosmgr resync” (see figure 7). Resync should not start as outside the window.
Figure 7. “tell daosmgr resync” command
7. When the resync window opens, issue a “tell daosmgr resync”, which will start the resync process, and monitor the server around the end of the resync window. Resync will start to skip databases after the window closes.
8. Issue a “tell daosmgr status” and confirm DAOS is still resync'ing.
9. Before the resync window opens, issue a “tell daosmgr resync”. Resync should not start as outside of the window. When the resync window reopens, issue a “tell daosmgr resync”.
10. Confirm that DAOS skips the databases already resync'ed and starts at the databases from which it left off.
11. Monitor that the window ends, or confirm the resync is complete, as necessary.
Evaluation criteria:
Pass
(if):
- DAOS resync stops when window closes and restarts where it left off when restart resync completes successfully.
- DAOS resync completes within a reasonable time for the number of mail files and NLO’s on the server (resync can take quite a long time, depending on the number of NLO’s), without significant impact on user response or test load.
Fail
(if):
- DAOS resync carries on past the window.
- DAOS resync restarts from the first database when run on Day 2.
- DAOS resync runs outside of the window.
- There are signs of a server hang or crash during the test.
Scenario 2: Running DAOS resync along with load compacting (Stress test)
In this scenario, we put stress on the server and ensure there are no excessive waits or locks. To put on more stress, we run the DAOS resync process and load compacting simultaneously.
1. First, we add following settings to the server’s Notes.ini:
DAOS_LOGGING. This parameter links to a path of a .txt file, and the STATE_CHANGE keyword edits the .txt file if there is a change in the DAOS state, for example, changing from SYNCHRONIZED to NEEDS RESYNC. Note that you can edit the path of the file, depending on your requirements.
The RESYNC parameter also sends output to that file, indicating the progress of the resync operations.
Debug_Compact_Immediately. This parameter allows compact requests from clients to be performed immediately.
Log_update. This parameter writes more information to the log, so it will be easier to determine where it is breaking.
2. Start the standard SVT server performance monitoring tools (Show stat and Open session).
3. Kick off compacting against the mail files by issuing a “load compact –c mail” on the server console (see figure 8). Here we are using Copy Style compacting “-c”.
Figure 8. “load compact –c mail” command
4. Without delaying, issue a “tell daosmgr resync force” on the server console (see figure 9), which runs the resynchronization irrespective of whether the DAOS catalog is in a synchronized state.
Figure 9. “tell daosmgr resync force” command
During the test we recorded response times (in ms) of one user when the compact process and DAOS resync process were run at the same time (see figure 10). To collect the user response data, we used an internal tool that collects user response times during a specified interval, which in our case was 30 sec.
Figure 10. User response times under stress
Evaluation criteria
Pass
(if):
- DAOS resync completed within a reasonable time for the number of mail files and NLO’s on the server, without significant impact on user response or test load.
Fail
(if):
- DAOS resync takes excessive amount of time, given the number of mail files and NLO’s on the server.
- User response times are dramatically increased for a prolonged period of time.
- There are signs of server hang or crash during the test.
Scenario 3: Running DAOS Resync when server is under load (Load test)
In this scenario, we run the DAOS resync using a secondary DOI, even when the server is under load.
1. First we add the following settings to the server’s Notes.ini:
DAOS_LOGGING. This parameter links to the path of a .txt file, which will be edited if there is a change in the DAOS state, for example, changing from SYNCHRONIZED to NEEDS RESYNC. Again, you can edit the path of file, depending on your requirements.
SECONDARY_DOI. By default the use of a secondary DOI, or DAOS Object Index, is always enabled. The DOI is the table that contains the list of NLO’s on the system and their associated metadata, including the relative path and its reference count. This allows attachments to be opened, even if the hints in the database are incorrect, assuming the NLO is listed in the old DOI.
In this scenario, we keep secondary DAOS Object Index enabled, so that new NLOs can be created and managed in the secondary DOI while the DAOS resync of the old DOI is in progress.
2. Start the standard SVT server performance monitoring tools (Show stat & Open session), and apply load to some mail files on the server, using NRPC and/or DWA load.
3. Issue a “tell daosmgr resync force” on the server console (see figure 11). This command will run the resynchronization, regardless of whether the DAOS catalog is in a Synchronized state.
Figure 11. “tell daosmgr resync force” command
4. Issuing a “tell daosmgr status”command shows that the Catalog state is in REBUILDING (see figure 12).
Figure 12. “tell daosmgr status” command
5. Continue monitoring DAOS status until the resync process is completed and the Catalog state changes to SYNCHRONIZED.
During testing we gathered response times (in ms) for two users and graphed the users' response time when they were under load and DAOS is in RESYCING state (see figures 13 and 14).
Figure 13. Response time for User 1
Figure 14. Response time for User 2
Evaluation criteria
Pass
(if):
- DAOS resync completed within a reasonable time, given the number of mail files and NLO’s on the server, without significant impact on user response or test load.
Fail
(if):
- DAOS resync takes excessive amount of time given the number of mail files and NLO’s on the server.
- User response times are dramatically increased for prolonged period of time.
- Test load drops too many threads or ends prematurely.
- There are signs of server hang or crash during the test.
- There are excessive waits and/or locks.
Scenario 4: Deleting DAOS-enabled mail files when server is under load (Stress test)
In this scenario, we verify that the server is stable and that DAOS behaves appropriately during a mass deletion (1000's) of DAOS-enabled mail files.
1. Start NRPC or a DWA load against DAOS-enabled mail files, and let it run for a few hours (see figure 15).
Figure 15. Load running against DAOS-enabled mail files
2. From the Domino Administrator, select all the DAOS-enabled mail files and select Delete Database, as shown in figure 16.
Figure 16. Delete DAOS-enabled mail files
3. Issue a “tell daosmgr status” command (see figure 17) and verify that DAOS updated the data store properly (removing the appropriate attachments, if applicable, etc.). Monitor the DAOS status frequently, ensuring the Catalog state doesn’t change to NEEDS_ RESYNC.
Figure 17. “tell daosmgr status” command
Evaluation criteria
Pass (
if):
- DAOS resync process runs successfully, and DAOS updates the data store properly.
Fail
(if):
- User response times are dramatically increased for a prolonged period of time.
- Test load drops too many threads or ends prematurely.
- Catalog log state changes to NEEDS_RESYNC.
- There are signs of a server hang or crash during the test.
Collecting DAOS stats using Server.Load
We can also collect DAOS statistics in intervals using Server.Load. To do this:
1. Open a Notepad, enter “Console [1] tell daosmgr status” in it (see figure 18), and save the file with the extension “.scr”.
Figure 18. Console [1] tell daosmgr status
2. Open Server.Load, select “Custom” in the Test Type section, browse to the script shown in the Select Script drop-down list (see figure 19), and click Execute. This will start collecting daosmgr stats in console log.
Figure 19. Server.Load
Conclusion
The System Verification Test team conducts tests that confirm the stability and performance of the server's Domino Attachment and Object Service feature. Systems are stressed under load to force error cases, to ensure the feature passes IBM's script test guidelines.
Performance metrics such as user response times are essential to ensure that the system is capable of sustaining loads for seven days with DAOS stressed. You can also use IBM tools such as Server.Load to validate and size your DAOS deployments.
Upgrading to Lotus Domino 8.5.x provides an immediate ROI and significant cost savings. For example, implementing DAOS results in storage savings of 77%, avoiding the need to purchase additional hardware for storage and the administrative costs associated with installation, etc.
Resources
Notes & Domino wiki article, "DAOS FAQ":
http://www-10.lotus.com/ldd/dominowiki.nsf/dx/daos-faq
developerworks article, "Achieving ultimate storage and server cost savings with DAOS in IBM Lotus Notes and Lotus Domino 8.5":
http://www.ibm.com/developerworks/lotus/library/notes85-daos/
developerWorks Lotus Notes and Domino product page:
http://www.ibm.com/developerworks/lotus/products/notesdomino/
About the authors
Pukhraj Saxena joined the IBM Software Lab in Dublin, Ireland, in 2007 after completing a Higher Diploma in Computer Science from Griffith College, Dublin. He currently works as a Reliability Engineer for the Lotus Notes/Domino team. You can reach Pukhraj at
saxenapu@ie.ibm.com.
Gary Denner joined IBM in 1998 as part of the Lotus Software group, working on the localization of Lotus Notes/Domino 4.5.3. Since then he has held roles in Localization, Automation Development, and Test across various products in the IBM Lotus portfolio. He is currently the Test Architect for the Domino System Test team as well as an IBM Master Inventor. You can reach Gary at
Gary_Denner@ie.ibm.com.