Introduction

DAOS is a feature of the IBM Lotus Domino Server that allows attachments to be stored outside the NSF file in Notes Large Object (NLO) files. This provides two major benefits: (1) It allows multiple copies of the same attachment to be stored as a single copy to save storage, and (2) it segregates the relatively static attachment data from the NSF data to allow flexibility in data storage and backup processing.

Introduced in Domino 8.5.0, DAOS works with any NSF-based application and is transparent to clients and other servers. It can be enabled on an individual-NSF basis, and not all NSF files on the server are required to participate.

Why should I use this guide?

In order to have the most successful DAOS deployment, it is important to plan properly and follow the deployment steps in order. Although every effort has been made to ensure that DAOS will work “right out of the box”, following the steps outlined here will provide the most efficient configuration---which can mean the difference between having good and having great results with DAOS.

This document goes step-by-step through the process of configuring and deploying DAOS, from the pre-deployment planning stages through the daily production operation.

Prerequisites

Before anything is done with DAOS, there are some prerequisites that must be addressed. These may not all apply in your situation, but it is important to verify them to ensure that any changes needed can be accommodated. These are all requirements for enabling DAOS, and are not optional.

1. Disable SCOS Shared Mail. Single Copy Object Store (SCOS) is an older approach to attachment consolidation. This feature is not compatible with DAOS and must be disabled before you enable DAOS. For more information, refer to the HCL Support article, “How to disable shared mail in HCL Domino.”

2. Disable NSFDB2. NSFDB2 is a feature that allowed storing NSF data in DB2. This feature is also not compatible with DAOS and must be disabled on any NSF application that will participate in DAOS. For information on migrating from the NSFDB2 feature, see Technote #1415018, “How to disable the NSFDB2 feature in Domino.”

3. Upgrade. Although DAOS was introduced in Domino 8.5.0, many important stability and performance improvements have been made in subsequent releases. Hence, it is strongly recommended that all new DAOS deployments be done on the 8.5.2 (or later) Domino release.

If you are restricted to Domino version 8.5.1, ensure that you are at the FP3 or later level. If you have Notes 8.5.1 clients, make sure they are at least at the FP3 level also. There are no other client requirements, and pre-8.5.1 clients as well as Web clients can also be used with DAOS.

4. Enable transaction logging. DAOS depends on transaction logging for proper operation. Since DAOS must update several locations simultaneously, it is important that all those updates succeed or fail (and are subsequently rolled back) as a unit.

Transactions provide this ability, and therefore transaction logging is required for all NSF files that participate in DAOS. For information on setting up transaction logging, see Technote #7009309, “Notes/Domino Best Practices: Transaction Logging.”

5. Establish backup/restore processes.It is important to have reliable backup and restore procedures in a production environment, to avoid the possibility of data loss. DAOS adds some complexity to the backup/restore process, so it is important that a well established backup/restore foundation exists for DAOS to build on. Transaction logging introduces some additional features that provide even better recovery options.

6. Upgrade Names.nsf design. The design of the Names.nsf file has been changed to accommodate DAOS, and the Server document has a new tab that covers the DAOS settings. Names.nsf must use the new Names.ntf template on all Domino servers that will be enabled for DAOS.

If Names.nsf is replicated in your environment, ensure that the update is done with respect to the replication flow. If a master hub copy of Names.nsf contains the old design, it may overwrite replicas on other servers that have been updated with the new design.

Additional recommendations

In addition to the above prerequisites, there are some other optional configuration changes that are highly recommended. These are not strictly required, but they will help yield the most benefit from Lotus Domino and DAOS.

1. Enable LZ1 compression. If no attachment compression is enabled on the NSF files, or if Huffman compression is being used, then enabling LZ1 compression can save a significant amount of disk space. This is done by use of the compact command, and the -Zu flag. For more information, refer to Technote #1256241, “Upgrading existing attachments from Huffman to LZ1 compression.” While this is an independent operation from DAOS, if you leave them as different encoding types, you get two copies of the attachment in DAOS. If you re-encode them after the NSF files are enabled for DAOS, you will end up with all references pointing to one NLO, and the other NLO having 0 references that doesn't go away until the deferred deletion interval. This will work, but it is not optimal. For that reason it is recommended that you standardize on LZ1 before you move to DAOS.

2. Enable design and data document compression. Another Domino space-saving feature is design and data document compression. Enabling these compressions can also save a significant amount of disk space. The savings from these features are independent from DAOS and are worth investigating. For more details on these options, refer to the Domino 8.5 Administrator Information Center topics, “Using database design compression” and “Using document body compression.”

3. Use Domino Domain Monitoring (DDM). DAOS diagnostic information is included in DDM events. The events are logged to the ddm.nsf file, which provides a convenient environment for monitoring the operation of DAOS. For information on managing and configuring DDM, refer to the Domino wiki article, “Domino Domain Monitoring (DDM).”

Planning

With the prerequisites and recommendations out of the way, it's time to start planning the DAOS deployment.

Identifying NSF Files that will participate in DAOS

Although the DAOS engine is enabled at the server level, individual NSF files must have the DAOS participation flag enabled. Not all NSF files on a server are required to participate in DAOS; in general, system NSF files (Names.nsf, Admin4.nsf, Journal.nsf, etc.) should not be DAOS enabled.

Mail files make excellent DAOS candidates, as do Mail.box files. If there are multiple Mail.box files, they all should be considered for DAOS.

Domino 8.5.1 introduced network optimization based on DAOS. If an attachment is sent to a DAOS-enabled server, and the attachment already exists in the DAOS repository at the destination, the attachment data does not need to be transmitted, which saves time and bandwidth. In a mail environment, this optimization depends on the Mail.box files being DAOS enabled.

Other non-mail NSF applications can also participate in DAOS. There are no template dependencies for DAOS, so any of the standard applications (discussion, document library, etc.) as well as any custom application that has attachment data can use DAOS.

Running the DAOS estimator

The DAOS estimator tool analyzes your existing NSF content and estimates the results of enabling DAOS. It is important to run this estimation because it helps you choose a minimum participation size (discussed in detail below), and provides an estimate of the resulting size of the NSF and NLO files.

The DAOS estimator runs on Domino versions 6.x through 8.x and is available for all Domino platforms. It can be downloaded from Support document #7014980, “Using the Lotus Domino Attachment and Object Service Estimator tool.”

The DAOS estimator provides an estimate of the resulting data if DAOS were in use. It is not an exact result, nor does it need to be. The data in an active system is constantly changing, so the results of DAOS are also constantly changing. The goal is not to come up with exact values, but to provide approximate size and result information that can be used for planning.

The estimator has two phases. The first phase scans the attachments in the NSF files, and the second phase analyzes the attachment data and prints a report. The scan phase can consume a significant amount of time and I/O resources, so it is recommended to run this phase separately, saving the raw data so that several analysis runs can be made on the collected data. This avoids having to re-scan the NSF files each time a new analysis report is run with different parameters.

Suggested operation
NOTE: Please refer to the DAOS Estimator documentation for detailed information on installing and running it.

load daosest -i foo.ind -c

where the -i option indicates that a list of NSF files to include in the scan is stored in the file foo.ind, and the -c parameter tells daosest to save the raw data after it is collected.

daosest -a DAOSEST_08_12_2010_10_57_55_AM.csv

where the argument to the -a option specifies the name of the .csv raw data file created by the previous run. The .csv file can be moved to another machine to run the analysis phase. Additionally, multiple .csv files can be specified in a .ind file for the -a parameter.

You can alter the analysis bucket sizes by adding/modifying the following Notes.ini variable:

DAOSEST_BUCKETS=64,128,256,512,1024,2048,3072,4096,8192,16384

There must be exactly 10 values specified, and the values are interpreted in kilobytes. This example would estimate results for an assortment of participation sizes between 64K and 16M. These values can be altered to focus the report on a specific range of sizes.

Choosing a DAOS minimum participation size

When an attachment is stored in an NSF that participates in DAOS, a decision as to where the attachment is stored is made based on the size of the attachment. The DAOS tab of the Server document includes the field, “Minimum size of object before Domino will store in DAOS” (see figure 1).

If the attachment (after any compression) is larger than the minimum participation size, it is stored in DAOS. If it is smaller, it will be stored in the NSF, the same as it would have been without DAOS.

Figure 1. DAOS tab on Server documentation


Choosing a size that is too large will result in too few attachments being stored in DAOS (low yield), which will reduce the savings DAOS can offer. On the other hand, choosing too small a size can result in a very high yield, resulting in an unmanageable number of files in the DAOS repository. The histogram below shows the number of attachments contained in each bucket, where the percentages are the percent of total disk space of all attachments per bucket.

Table 1 shows the DAOS minimum size versus the number of NLO's and disk space.

Table 1. DAOS minimum size vs. number of NLO's / disk space
0.0 KB will result in 2226347 .nlo files using 185.5 GB
64.0 KB will result in 1092894 .nlo files using 175.7 GB
128.0 KB will result in 708403 .nlo files using 163.6 GB
256.0 KB will result in 422087 .nlo files using 145.9 GB
512.0 KB will result in 219833 .nlo files using 120.2 GB
1.0 MB will result in 93628 .nlo files using 87.8 GB
2.0 MB will result in 36576 .nlo files using 56.6 GB
3.0 MB will result in 17499 .nlo files using 38.0 GB
4.0 MB will result in 9717 .nlo files using 26.3 GB
8.0 MB will result in 1576 .nlo files using 6.5 GB

The first line in table 1 shows the result for 0 bytes, which is the theoretical maximum possible benefit of DAOS on the NSF files that were analyzed.

Typically the number of NLO files generated at that limit is enormous. Looking at some of the other values in the table, you notice that although the number of NLO files decreases fairly rapidly, the overall estimated size of the DAOS repository does not drop significantly.

Determining the appropriate size
Don't get greedy on the size. Look for a value that yields about 80-90% of the theoretical maximum DAOS repository size. Although that may sound low, it's generally the best tradeoff between the DAOS benefits and the resulting number of files. It's quite common to see total sizes in the 85% range with only one-quarter to one-tenth of the NLO files.

In the above sample output, 128Kb results in 88% of the theoretical maximum size, while 256Kb results in 78%. As a result, somewhere in between those two values (192Kb)is the best choice. It would result in approximately 83% of the theoretical maximum DAOS repository size, while producing only 25% of the NLO files.

The minimum participation size can be changed after it is set initially, but it is not retroactive. When Lotus Domino writes an attachment to an NSF, the current value of the threshold is used. Existing attachments remain in NSF storage unless a copy style compact (compact -c) command is run. During the copy style compact, the storage for all attachments in the NSF is reconsidered based on the current threshold value.

The 80-90% guideline will usually yield reasonable results but, if the yield is not appropriate for some reason, it is possible to change the value You can adjust the attachment storage in either direction by changing the threshold and using copy style compact. Note, however, that it's far easier to tune the value down rather than up.

If the value used is too high, and the NLO yield is low as a result, there will be too few NLO files. Reducing the value and re-running the copy style compact will extract more attachments and increase the NLO yield. Once the compact operations are complete, no additional work is needed.

If the initial value used is too low, and the resulting NLO yield is too high, increasing the threshold and running copy style compact will store more attachments in the NSF. However, the existing NLO files will not immediately be deleted due to the deferred deletion interval (see below for more details).

Until they age out of the DAOS repository, they will still exist and take up space. Because of this delay, this approach does not immediately address the problem of having more NLO files than desired.

Controlling the number of NLO files
It is important to control the number of NLO files for several reasons:

• Having too many NLO files can increase backup/restore complexity and duration. Some backup utilities suffer degraded throughput as the number of files increases. In extreme cases, the backup utility can become overwhelmed by an inordinate number of NLO files.

• The deletion of unreferenced NLO files in DAOS (a.k.a, “pruning”) depends on the proper accounting of the reference counts for each NLO file. If this count becomes suspect for any reason, the state of the DAOS catalog changes to indicate this.

The DAOS resync process re-counts all the NLO files and all the references that the NSF files have to them. An unnecessarily large number of NLO files can delay this process

• Pruning the unreferenced NLO files can be delayed if there are a large number of NLO files on the pending deletions list.

It is also important to consider the size and number of NLO files compared with the block size of the filesystem. When individual files are allocated, the filesystem allocates blocks to store the data. The size of the data in a file is usually not an even multiple of the block size, so the remainder of the block ends up as unused space.

The default configuration limits the number of files in each subdirectory of the DAOS repository to 40,000. Combined with a maximum subdirectory count of 1000, DAOS has a theoretical maximum capacity of 40 million NLO files.

Given a random assortment of file sizes, on average, half of a block size is wasted for each file as it is stored on disk. As the raw number of NLO files increases, the amount of wasted space increases as well.

Also, with smaller NLO files, the half-block-size value can start to be a significant percentage of the stored data, meaning the percentage of waste increases as the average file size goes down.

Other considerations
If the server is primarily being used for a router, and does not have any resident NSF files, attachments that pass through a DAOS-enabled mail.box file will leave NLO files behind with 0 references. Since there are no resident NSF files referencing them, there is not a strong reason to be able to restore these NLO files. It is, however, beneficial to keep them around for some period of time so that the network optimization aspect can be exploited.

In this situation it is recommended that the deferred deletion interval be set to a few days. This is a good balance between being able to exploit the DAOS network optimizations while not creating a large amount of unreferenced NLO files. Some experimentation may be needed to find the value that suits the specific traffic best.

Planing for disk space

The DAOS repository is where all the NLO files are stored, so it's important to choose its location carefully as it contains the attachment data for all the NSF files that are participating in DAOS.

Each Domino server that has DAOS enabled must have its own DAOS repository. Sharing the same repository between servers---even between clustered servers---is not permitted and will lead to data loss.

The DAOS estimator provides an approximation of the amount of storage that will be needed to hold the DAOS repository; however, the actual space allocated should be larger than the estimate to allow for variance between the actual and estimated sizes, in addition to allowing for future expansion.

Although the default location for the DAOS repository is a subdirectory under the data directory, this is not the recommended location. The DAOS repository can contain millions of files in large installations, which can cause problems for utilities that attempt to process all files in the data directory. Thus it is recommended that the DAOS repository be located outside the data directory.

It is important that the DAOS repository also not reside logically under the data directory. Having a symbolic directory link in the data directory that points to the DAOS repository located elsewhere will cause similar problems.

The DAOS repository can be located on the same volume as the data directory, or it can be located on a separate volume. Lotus Domino makes no requirements about the volume location of the repository.

In a typical mail environment, the traffic to the DAOS directory is significantly lower than the traffic to the data directory, so the DAOS repository can be located on lower-performance storage. By storing the DAOS repository on “cheaper” storage, you allow space to be freed up on the high-performance disk (typically recommended for the data directory).

Choosing a deferred deletion interval

The DAOS tab figure 1 also shows the “Defer object deletion for” field. NLO files contain the attachment data, and the DAOS catalog keeps track of the number of references there are to any given NLO file. When an attachment is deleted, the reference count is decremented, and when the reference count reaches zero, the NLO becomes a candidate for deletion.

The NLO file is not deleted immediately; instead, it's added to the deferred deletion list until the deferred deletion interval has expired for the NLO. The default interval is 30 days. If a new reference to the NLO is created during that time, and the reference count goes above zero, the NLO will be removed from the deletion list.

This parameter is most important for backup and restore operations. If an NSF is restored that has DAOS references, the NLO files that it references must exist in the repository in order to access those attachments.

Any NLO files that do not exist must be restored to the DAOS repository. The deferred deletion interval provides a buffer that reduces the number of individual NLO files that need to be restored, to satisfy the references in a restored NSF file.

The deferred deletion interval should be set no smaller than the interval between backups, which ensures that all NLO files participate in at least one backup so that they can be recovered, if necessary.

Some sites have a restore policy that limits the maximum age for restore requests. If the policy is 90 days, for example, setting the deferred deletion interval to that same 90-day value means that an NLO file will never need to be restored individually to satisfy an NSF restore operation.

Setting this value too high will incur the cost of an accumulation of unreferenced NLO files. Any attachment that is deleted will remain in the DAOS repository until the interval expires. The convenience of the restore operation must be balanced against the cost of the additional disk space consumed by the unreferenced NLO files.

DAOS can also be valuable, even if a Domino server is participating only in mail routing and has no mail files that reside on it. You can achieve the network optimization mentioned above by enabling DAOS on the Mail.box file(s) on the server.

In this situation, however, it is recommended that the deferred deletion interval be set to a relatively small value, to avoid an unnecessary buildup of NLO files as mail (and attachments) pass through the server.

Deciding on encryption for NLO files

When an attachment is written to an NSF, all compression, signing, and encryption operations are performed before it is written. Likewise, if the attachment is to be stored in DAOS, the same operations are performed before DAOS is given the data. The filename of the NLO is actually a representation of the contents of the NLO file. The same attachment is represented by the same NLO name on all servers.

By default, DAOS adds an additional layer of encryption to the data as it is written to an NLO file. This encryption is in addition to any encryption that was performed based on the NSF options. The NLO data is encrypted using the server key. Even though a same-named NLO file represents the same attachment on different systems, it will not be portable to other systems unless the NLO file encryption was disabled before the NLO file was created.

Although the encryption is enabled by default, you can disable by adding the parameter

DAOS_ENCRYPT_NLO=0

to the Notes.ini file. The setting cannot be changed retroactively, and the only way to remove encryption from an existing DAOS installation is to completely disable DAOS first.

Because the NLO file is encrypted with the server key, the NLO files can only be read by use of the Server.id file that was originally used for the encryption. If an NLO file is encrypted, it is not readable by any other Domino server, including other servers in the cluster.

An unencrypted NLO file contains the exact same information that would have been written to the NSF file if the attachment had been stored there. The main difference is that the attachment is stored in an individual file, instead of being part of a larger NSF file.

This is an important consideration when determining backup/restore operations:

• With different encryption on different servers, the contents of the NLO files on those servers will be different. As a result, backup utilities that provide de-duplication will not be able to consolidate the NLO files from the different servers.
• Restore operations for encrypted NLO files can only be performed on the server that originally created the NLO files. Restoring production data to a test server becomes problematic.
• Disabling the encryption of NLO files offers easier restore scenarios, as they are readable by any server.

Updating backup/restore processes

As mentioned above, it is important to have reliable, tested backup and restore procedures. The NSF backup process will be unchanged after DAOS is implemented, but there is some additional data to manage.

The NLO files contain the attachment data, and they also need to be backed up. NLO files are never modified after they are initially created, except for being deleted when they are no longer referenced. As a result, they are excellent candidates for incremental backup; they can be treated as generic flat files and do not require the use of any special API or utilities.

The backup of all the NSF data is not instantaneous, and there can be additional NLO references created during that process if the Domino server is active. If the NSF files are backed up first, and then the NLO files are backed up, all the references in the NSF files will be guaranteed to be in the corresponding backup of NLO files.

Again, it is important to coordinate the deferred deletion interval with the backup interval.

For a more detailed look at backup and restore issues with DAOS, refer to the Domino wiki article, “DAOS Backup and Restore.“

Deciding the method of enabling DAOS on NSF files

There are several ways to go about enabling DAOS on the individual NSF files, any of which will yield a successful DAOS rollout, so it is simply a matter of choosing one.

Using a Notes or Domino Administrator client, you can enable (check mark) the “Use Domino Attachment and Object Service” option on NSF files via the Database properties window (see figure 2).

Figure 2. Database properties


The compact command can be used to enable DAOS also. The compact option to enable the DAOS participation flag is:

compact -daos on filename.nsf

This enables DAOS for all future attachments created in the NSF. Once the DAOS flag is enabled, the previously existing attachments in the NSF can be extracted to DAOS storage by use of the copy style compact command. Both operations can be done in a single command:

compact -c -daos on filename.nsf

The compact command also accepts an .IND file, which is a list of files on which it will operate. This can help automate the process of enabling DAOS on a large number of NSF files. For more information, refer to the Domino wiki article, “Using indirect files to run maintenance tasks.”

Considering DAOS deployment on hub or gateway servers

DAOS is primarily useful for reducing disk space consumption, but it can also save network bandwidth as well. A feature introduced in Domino 8.5.1 capitalizes on DAOS to eliminate the transfer of attachment data that already exists in the DAOS repository of the destination server.

All 8.5.1 sources, including Notes clients and non-DAOS Domino servers, calculate the checksum-based DAOS key for all attachments stored in ODS51 NSF files, even if they are not participating in DAOS. If the destination has DAOS enabled, the key is sent to the destination first.

If the attachment does not exist in the DAOS repository, the contents are transmitted as usual. If the attachment does already exist in the DAOS repository of the destination, the contents are not transmitted, and a new reference to the existing NLO is added.

Using this optimization, a Notes 8.5.1 client replicating a local ODS51 Mail.box containing a “Reply to All with attachments” response to a DAOS-enabled server may need only to transmit the document body. If the attachments already exist in the DAOS repository at the server, they will not be transmitted by the client.

Because this feature works based on the contents of the DAOS repository, it is desirable for all servers along a routing path to take advantage of it. Even if there are no user mail files resident on the intermediate servers, if the Mail.box file(s) are DAOS enabled, the network optimization can be performed.

If there are no user mail files resident on the server, the attachment data stored in DAOS will be from the traffic passing through Mail.box. The default deferred deletion interval of 30 days may be excessive in this case, as it can allow a buildup of NLO files on the intermediate servers. If this is a concern, the deferred deletion interval can be reduced to balance the NLO file count against the network-bandwidth-optimization benefits.

Production rollout

If all the above steps have been followed, the rollout of DAOS should be uneventful. You may find it helpful to record information before and after the rollout, to determine the exact results for your data.

1. Back up existing data. Using the backup procedures, make a full backup of all production data.

2. Upgrade to On Disk Structure format (ODS)51. DAOS requires that new information is stored in the NSF,and that information is defined in the ODS51 NSF format. All NSF files that are to participate in DAOS must be upgraded to ODS51 before DAOS can be enabled on them.

Other NSF files on the server that are not participating in DAOS don't need to be upgraded, but there are other benefits that make it worthwhile. For information on upgrading an NSF to ODS51, see Technote #1429889, “Upgrading multiple local databases to a new ODS.”

3. Configure transaction logging for DAOS.Transaction logging is required for DAOS, and there are two .INI settings that can improve performance relative to attachments. The first is

RM_NO_LOG_OBJECTS_IN_MAILBOX=1

This setting prevents transaction logging from recording the contents of objects when they are written to Mail.box. This means that the attachment data will not be part of the log and, as a result, will not be able to be restored during a recovery operation using the transaction logs.

Alternatively, logging of attachments can be disabled for all databases on the server (regardless of DAOS participation) via the following setting:

RM_NO_LOG_LARGE_OBJECTS=1

This prevents objects larger than 1M from being recorded to the transaction log, saving space and I/O traffic. Again, the exposure is that, if the data is not written to the log, it cannot be recovered using the transaction logs.

4. Set up the DAOS repository. The location for the DAOS repository should be created ahead of time. Ensure that access is available to the account running the Domino server and that access permissions and mount availability are verified and correct.

On filesystems that are mounted (typically, on UNIX®) the DAOS directory should not be located directly at the root of the mount point, but should instead be located at least one directory level down. There have been some intermittent problems reported with the DAOS directory at the root of a mount point, and locating it in a subdirectory avoids this.

If possible, enable DAOS on a test system, and go through the rollout steps on that environment, making sure everything is working as expected and that the enablement steps are understood and correct.

5. Disable virus scans for DAOS directory.The NLO files in the DAOS directory must be excluded from virus scans. If these scans are not disabled, they will interfere with the proper operation of DAOS. Scanning of attachments should be done via a Domino add-in and not by scanning individual NLO files. The NLO files typically contain compressed and/or encrypted data, so such scans are unlikely to be useful.

6. Update DAOS tab on Server document.The minimum participation size, the deferred deletion interval, and the DAOS path fields on the DAOS tab of the Server document should be filled in with the values determined in the previous sections. Also, the DAOS option should be enabled. If your Names.nsf is replicated with other locations, the changes should be performed so that they are not overwritten by another copy.

After updating the Server document, stop the Domino server (and all Domino-related processes); when the server is restarted, the DAOS feature will be enabled.

7. Enable DAOS on selected test NSF files.Using the desired command method from above, enable DAOS on some of the NSF files in the production environment. It's recommended to use some dummy test NSF files so that no production data is involved initially.

8. Monitor NSF and NLO footprints. As soon as DAOS is enabled, and attachments are moved into DAOS, NLO files should be created in the DAOS repository. Verify that the size of the NSF files is now smaller, and that there are NLO files in the DAOS directory. The combination of the sizes should be smaller than the original size of the NSF files, assuming there is some duplication.

9. Verify operation. Verify that the NLO files are being created as expected, and that the yield is acceptable. If things are not working as expected, stop and investigate the problem before proceeding.

If no NLO files are created, check the following:

• Use the “tell daosmgr status” console command to verify that DAOS is enabled on the server.
• Look at the NSF properties dialog to verify that the NSF has been upgraded to ODS51, and that the Disable transaction logging option has not been selected.
• Verify that transaction logging is enabled and running on the Domino server.
• Ensure that at least some of the attachments in the NSF are larger than the DAOS minimum participation size. Remember that if attachment compression is enabled, the size after compression must be larger than the DAOS setting.
• Check the Domino console for error messages.

10. Exercise the new backup/restore procedures. These procedures should be exercised to ensure they work properly. This is most easily done on a test server, but it can be done carefully on a production server as well.

Ideally the restore operation should involve at least one missing NLO file restore. You can simulate this by creating an attachment that is stored in DAOS, performing a backup, and then deleting the attachment and intentionally deleting the associated NLO file. At that point, the restore procedure should be performed, and the document and attachment should be accessible again.

11. Enable DAOS on other NSF files. Once correct operation has been verified, DAOS should be enabled on the balance of the NSF files chosen to participate in DAOS, as well as on Mail.box file(s) on the server. DAOS can be enabled independently on each NSF file, and processing the NSF files can be done in whatever order and schedule is convenient.

Maintenance

1. Monitor DAOS catalog. The DAOS catalog state should normally be 'SYNCHRONIZED,” and this can be checked with the Domino console command:

tell daosmgr status

If the catalog state changes, a DAOS resync should be performed but, unless Daoscat.nsf has been damaged somehow, there is no immediate need to do a resync. All DAOS functions will continue to operate normally except for prune operations, which will be postponed until the DAOS catalog state is SYNCHRONIZED again.

A DDM event is recorded when the DAOS catalog state changes. You can trigger an email notification action on that event to help monitor the state.

2. Monitor DAOS footprint. The DAOS repository grows with the accumulation of data; however, if the total size of the DAOS repository and the NSF files exceeds the usual expectations, it should be investigated. Typical causes for unusual growth in the DAOS repository include prune operations not being performed due to bad DAOS catalog state, and the DAOS flag being unintentionally switched off for NSF files.

3. Monitor the prune operation. This operation is performed automatically at 2AM daily. There is a log message indicating what action has been taken by prune; however, if the DAOS catalog state is not SYNCHRONIZED, prune will not run and will display a message to that effect.

Note that the deferred deletion interval must expire on an unreferenced NLO file before prune will delete it. If your interval is set to 30 days, you will not see any prune activity for at least that amount of time.

4. Perform DAOS resync. As needed, a DAOS resync operation can be performed to re-count all the references to NLO files. This is not typically an urgently needed operation; in fact, in most cases the resync can be delayed until the next weekly (or even monthly) maintenance window.

As this process can take a significant amount of time and system resources, be sure to schedule a resync so that it will have the least impact on production operations.

Resources

IBM Support Technotes

1415556, “A checklist for implementing DAOS”:
http://www-01.ibm.com/support/docview.wss?uid=swg21415556

1448635, “Overview of 8.5.2 Improvements in DAOS Catalog”:
http://www-01.ibm.com/support/docview.wss?uid=swg21448635

1448358, “Understanding the DAOS prune operation”:
http://www-01.ibm.com/support/docview.wss?uid=swg21448358

1426551, “Transient attachments and DAOS”:
http://www-01.ibm.com/support/docview.wss?uid=swg21426551

1420013, “Summary of Backup and Recovery recommendations when using DAOS on IBM i”:
http://www-01.ibm.com/support/docview.wss?uid=swg21420013

1420019, “Procedure for restoring a DAOS enabled database on IBM i”:
http://www-01.ibm.com/support/docview.wss?uid=swg21420019

1503667, "Why deleting the DAOS catalog is a bad idea"
http://www-01.ibm.com/support/docview.wss?uid=swg21503667

Domino wiki articles

“DAOS Quick Start Guide”:
http://www-10.lotus.com/ldd/dominowiki.nsf/dx/daos-quick-start-guide

“DAOS Best Practices”:
http://www-10.lotus.com/ldd/dominowiki.nsf/dx/daos-best-practices

“DAOS FAQ”
http://www-10.lotus.com/ldd/dominowiki.nsf/dx/daos-faq

“DAOS Backup and Restore”:
http://www-10.lotus.com/ldd/dominowiki.nsf/dx/daos-backup-and-restore

“Demo: Introduction to Domino Attachment Object Service (DAOS)”:
http://www-10.lotus.com/ldd/dominowiki.nsf/dx/daos-demo

DAOS wiki article index:
http://www-10.lotus.com/ldd/dominowiki.nsf/xpViewTags.xsp?categoryFilter=DAOS

developerWorks article

“IBM Lotus Domino going green: The new Lotus Domino attachment and object service”:
http://www.ibm.com/developerworks/lotus/library/domino-green