Overview

IBM® Lotus® Domino® Attachment and Object Services (DAOS) is a recently added Domino server feature that stores database attachments outside the NSF files in IBM Lotus Notes® Large Object (NLO) files. If the attachment is used multiple times, then only the single copy of the attachment is needed on the server, so disk space is saved.

The addition of DAOS to the Domino server has added another level of requirement to the archiving or backup needs. So, what do you need to know when backing up your databases and, more importantly, when restoring a database?

First you need an understanding of how DAOS works in order to know what should be archived as well as how to restore a database fully with the attachments available.

It is also important to note that the benefit of DAOS to the server, that is, the reduction in required disk space, also reduces the overall required size for backup archives because the attachments must only be stored once. So, although more files must be backed up, the overall size of the backup is reduced.

To get the most from this article, you should have an understanding of Domino Servers, DAOS, and transaction logging. Although specific terms are referenced here, this is not a complete outline of these features, only a description of the areas used with archiving and backup.

About DAOS

Let's begin by outlining how DAOS works:

• It is a Domino server function that allows attachments to be stored on a Domino server outside database NSF files.

• Attachments are stored in the DAOS repository as NLO files, and each unique attachment has its own NLO file.

• When an attachment is reused, only a single instance of the attachment is needed, resulting in a saving of disk space.

• When an attachment is edited, a new NLO is created for the new version of the attachment; an NLO file is never edited or changed once it is created.

• The filenames of NLOs are based on a checksum taken of the attachment contents, which is used to ensure a match to the NLO for any new attachment and that any differences lead to a new NLO file being generated.

• By default, NLO files are encrypted with the server's ID file, so they can be opened only by the server and cannot be transferred directly to other servers.

• If all the documents with an attachment are removed from the database, then after a configurable amount of time, the NLO will be “pruned” from the repository to prevent it from growing too large. An NLO is not pruned until an interval plus one day has passed in which it has no links from a database–referred to as the “Deferred Deletion Interval.”

• Databases are said to be “DAOS enabled” if any of their attachments are stored in DAOS.

• There is a configurable minimum size for attachments to be stored in DAOS. This can lead to a database having some attachments stored in DAOS and some still remaining in the database, depending on the size of the attachments.

• NLO files are stored in numbered subfolders in the repository, each of which holds only a specific number of NLO files. These are named 0001, 0002, 0003, etc.

• DAOS uses a catalog database (DAOSCAT.NSF) to record the DAOS databases and the list of available NLO files. It is assumed that this is in a working and synchronized state; if not, then the recovery of attachments onto the server may not be possible.

About transaction logging

Transaction logging is the process whereby every change made to a database is logged as a transaction in the transaction log files (.TXN files). For DAOS to work on a server, transaction logging must be enabled. There are three styles of transaction logging:

• Archived. Archived logging creates transaction logs as needed, and then once these are archived, re-uses the .TXN file with new data. It continues to create new transaction logs if there are no existing logs marked for re-use by the backup software.

• Circular. Circular logging uses a fixed number of transaction logs in a loop, overwriting the first log after it writes to the last log, and continues logging in a loop. The transaction logs are limited to a maximum of 4GB in size. All the logs are created on the server when the server is started.

• Linear. Linear logging works the same way as circular logging but does not have the 4-GB size limit.

Backup strategies

Depending on the style of backing up or archiving you use, there are different requirements and elements to be aware of when using DAOS. Although many of the areas to be aware of for each strategy are the same with only some variances, each description is covered here in full so that they can be read through individually.

Here are the strategies covered:

(1) Full backup software with archive transaction logging in place
(2) Full database backup with circular or linear transaction logging
(3) Simple direct copy of the databases
(4) Stored replica of the database, locally on a Notes Client
(5) Stored Replica of the database, on another server
(6) Domino database archiving
(7) Long-term NLO backup strategy

NOTE: Strategies 1, 2, and 3 all involve different ways to recover the database, after which the DAOS portion of the process is implemented. This is the same for each strategy, so it is explained in full only once, in Section 4.3 below, “Simple direct copy of the databases,” (Strategy 3)as it is from this point that the process starts.

Each backup strategy requires different elements to be aware of and different setups.

Full backup software with archive transaction logging in place

In this configuration a backup of a database is taken through the Domino Server API, and then each subsequent change to the database is logged in a transaction log file. The original databases and the transaction logs are then archived and stored by an external piece of archiving software.

In this way the database can be restored and the transactions (or changes) replayed onto the database, allowing a database to be either restored as it was originally archived or replayed to a selected point in time after the original archive. This process is independent of DAOS; it is only when the database has been recovered that the use of DAOS on the database must be taken into account (more on this below).

The restored database will be DAOS enabled, and any attachments that were stored in DAOS will not be stored in the database.

At this point the database is ready for full recovery with all its attachments. If you do not need to recover the attachments, then the database can be used; however, to ensure that all the attachments are available, continue with the Database Recovery as outlined in Section 4.3, “Simple direct copy of the databases”.

IMPORTANT NOTES:

• Although the backup server is backing up the TXN files for transaction logging, by default the backup software does not cover the NLO files. To be able to restore a DAOS- enabled database with its attachments, a backup of the DAOS NLO files must be available.

• Since NLO files are pruned from the DAOS folder, you cannot rely on the DAOS folder to contain all the NLO files linked from the database.

• Since NLO files are not updated, they should be backed up at a file-name level, so that a new file name will be backed up.

• It is important to back up the NLOs at a higher frequency than the Deferred Deletion Interval so as to prevent NLO files from being pruned before they are backed up.

• The databases should be backed up before the NLO files are backed up. This allows for any attachments that may otherwise have been added while the NLO files were being backed up to be added and then the NLO not backed up.

Full database backup with circular or linear transaction logging

Circular and linear style transaction logging do not allow for Point-in-Time database recovery. The backup software does not have access to back up the transaction logs, so the database is backed up and then always restored always in the same state; there is no option to replay changes over it.

Here the database is backed up through the server's API, so the server can be kept running during the backup. Again, the restored database is DAOS enabled.

When the database is recovered, if you do not need to use the attachments, then the database can now be used; however, to ensure that all the attachments are available, continue with the database recovery as outlined in Section 4.3.

As with the first strategy, the IMPORTANT NOTES listed Section 4.1 above apply here as well.

Simple direct copy of the databases

In this scenario you make a copy of a database, using the operating system's copy function, or back them up directly outside of Lotus Domino and store this as your backup. Note that the server MUST be shut down during these copy operations.
This process is outside the control of Lotus Domino, and any issues during the process caused by possible hardware or software failure are not supported by Lotus Domino.

For this process to work, all NLO files still must be backed up to allow for databases to be recovered with their attachments. NLO files are updated since any change to the attachment leads to a new NLO file. They should be backed-up at a “file name” level, so a new file name should be backed up.

The NLO files can be backed up directly, even with the server running. Since they will never change, the backup software can be configured to store only new files. Also, the NLO files must be backed up at a higher frequency then the Deferred Deletion Interval.

As with the previous strategies a database recovered from this backup is DAOS enabled, and attachments that were stored in DAOS are not stored in the recovered database.

Database recovery
With a non-DAOS-enabled database, the database can be restored as it would normally for the backup software, and the transaction logs can then be replayed to bring the database to the required recovery time. Here the attachments are in the database.

With a DAOS-enabled database the process starts the same, with the database being restored and replayed; however, the database itself does not contain any attachments that were stored in DAOS. If the database has been restored to the original server, it is possible that some of the NLO files required will still be present for this database.

If attempting to access an attachment that is missing, a user will see the message shown in figure 1, and a warning will display on the server console.

Figure 1. File does not exist.... error


At this point some decisions must be made; specifically, is the database being restored:

• to the original server?
• to a different server?
• to a Notes Client?
• outside a Notes Client or server?

Database restored to the original server. In this situation you must be aware of the normal potential issues with having two databases on a server with the same Replication ID, filename, and title. This situation is not covered here.

If you are restoring to the original server, then many of the NLO files may still be available to the recovered database. However, it is also possible that the prune function will have removed a number of NLO files from the server, so they are not available for the database.

To recover these, run the command:
where “database_file_name.nsf” is the file name of the recovered database, and “missingnlo.txt” is the output text file containing a list of the NLOs that are referenced in the database but not available on the server. This list should be used to restore the required NLO files from the backup.

IMPORTANT: When restoring NLO files, be aware that the “listnlo” output above shows all the attachments as being present in the original subfolders in which they were generated. It is possible that these files will not be in these subfolders in the backup software, so it may be necessary to search through the backed-up NLO subfolders through all NLO files to recover all the NLO files.

If possible, recovered NLO files should be placed in their original subfolder. However, be careful to prevent the number of NLO files growing too high in each subfolder. Currently the default setting for this is 40,000 files. This is not a hard limit and can be breached if required, but if the number of NLOs goes too high, it can cause performance degradation.

If the NLO files are not restored to their original folders, then a DAOS catalog resync will be required in order for the attachments to be available to the database. Even if the NLO files are restored to their original subfolders, a resync of the DAOS catalog may still be required if an extended period of time has passed since they were removed, in order for the attachments to be available through DAOS.

To do this, confirm the current DAOS catalog status with the command:
If the current catalog is synchronized, then run the command:
If the current catalog is in any other state, then run the command:
Note that, on a large server, re-synchronizing the DAOS catalog may take several hours. During the process, however, existing attachments are still available and new attachments are included in the DAOS repository, as usual.

Database restored to a different server. If the database is being restored to a different server, this can circumvent the issues with multiple versions of a database on a server; however, there are some issues with DAOS, even if the new server is DAOS enabled.

By default, all NLO files are encrypted with the ID of the server on which they are created, causing NLO files to be non-transferable from server to server. One option is to create a “Clone” server, that is, a server using the same ID file as the original server, which would need to be isolated completely from the original Domain.

Once restored on a Clone server, the NLOs can be listed as described in the “Is the database being restored to the original server?” section above and placed on the server for the database to access.

Database restored to a Notes Client. Notes Clients are neither DAOS enabled nor aware. Because of this no attachments that were stored in DAOS will be available, if the database is simply restored on a Notes Client. Attachments that were not stored in DAOS will still be available in the Database.

Database restored outside a Notes Client or server. If the database is being restored outside any Notes or Domino environment, then there will be no attachments that were stored in DAOS available.

Stored replica of the database locally on a Notes Client

If a local replica of a DAOS-enabled database is being used as a backup, then this replica of the database will contain all the attachments in itself.

This is because Notes Clients are not DAOS enabled when a local replica of a database from a DAOS-enabled database is created as all the attachments replicate from the server and are stored in the database.

This can be stored and re-used in the same way as any normal database. There are no NLO links within the database so there are no DAOS requirements when the database is transferred.

Stored replica of the database on another server

DAOS is “invisible” to replication; that is, if the source database is DAOS enabled, the target server does not see this and only sees a database with all its attachments.

So, if you are replicating a DAOS-enabled database to another server as a backup, then the target database is isolated from the DAOS environment on the source database.

If the target server:

• is DAOS enabled and the target replica database is DAOS enabled, then the attachments will be stored in NLO files on the target server as usual.

• already has a matching NLO for the attachment, then a link to the existing NLO will be added to the new replica, and the source NLO will not be replicated.

• is not DAOS enabled or if the target replica database is not DAOS enabled, then the attachments will replicate normally and will be stored within the database itself.

Domino database archiving

Domino database archiving allows for the archiving of documents from one database to another, either locally or on a server. As with replication, this process actually uses the replicator task, and the use of DAOS for attachments is invisible to this process.

If you are archiving:

• locally, then the attachments will be stored, as usual, in the archive database on the Notes Client.

• to a different server that is DAOS enabled, then the attachment will be stored in an NLO on the target server.

• to a different database on the same server as the original DAOS-enabled database, then the archive database will use the existing NLO file with a link to it.

Long-term NLO backup strategy

When a database has been DAOS enabled for several years, it could lead to several years worth of NLO backups. To be able to recover the database from any point in its history, it is necessary to have a complete backup of all the NLO files that the database ever used.

When you increase this to all DAOS-enabled databases on a server, it becomes difficult to manage.

One suggested strategy is to take an annual backup of all NLOs used for that year, and then store that with the database backups for that year. The thinking behind this is that every current NLO for a database is present in the repository. If the backup contains all the NLOs from the repository for the year, then the database from that year can be easily recovered with the NLOs from that year only.

The following year, several NLOs may have been previously pruned, but these will not be needed for any database recovery during this year. Only the current NLO files and the following NLO files for the year will need to be backed up.

This allows for long-term NLO storage to be packaged, so that a continuous, never-ending, backup is not required.

For example:
Suppose Server1 contains Database1 with Attachment1, Attachment2, and Attachment3.

Throughout Year1:

• Attachment4 and Attachment5 are added to Database1
• Attachment1 and Attachment2 are removed from Database1

At the end of Year1 the NLO backup will need to contain NLO files for Attachment1, Attachment2, Attachment3, Attachment4, and Attachment5.

Throughout Year2:

• Attachment6, Attachment7 and Attachment8 are added to Database1
• Attachment4 and Attachment6 are removed from Database1

At the end of Year2 the NLO backup will need to contain NLO files for Attachment3, Attachment4, Attachment5, Attachment6, Attachment7, and Attachment8 but will not need Attachment1 or Attachment2 since they were not used in this year.

This limits the NLO backup to only the year's NLO files. The NLO and database backup for a year will still be smaller than the database backup if DAOS were not used on the databases.

Options and commands you may need to know

(1) A replica of a database created on a non-DAOS enabled server or on a Notes Client will not be DAOS enabled. If the original is on a DAOS-enabled server with all the NLO files available, then the new replica will be complete, with all the attachments stored in the database. This can then be used on any server or Notes Client, and the attachments will be present.

(2) Running the command:
on a database causes the attachments to be re-inserted into the database, provided that the NLO files are available. This database can then be moved to other servers and clients with the attachments in place as usual.

(3) NLO files can become corrupt. Although this is less likely than with a normal database because the server never writes to them once they are originally created.

Conclusion

DAOS is quite effective in reducing the disk space needed to maintain a busy server and, similarly, can reduce the amount of space needed to maintain a working database strategy. However, to have a full backup of all databases and attachments, administrators must be aware of the NLOs; specifically, they must have a full backup of all NLO files generated on the server and know how to restore them.

Tell us what you think

Please visit this link to take a one-question survey about this article:

Resources

• developerWorks Lotus Notes and Domino product page.

• Domino wiki article, “IBM Lotus Domino Attachment and Object Service (DAOS) deployment guide.”

• IBM Support Technote #1420878, “Where can I get more information on Domino Attachment and Object Service (DAOS)?"

• IBM Support Techdoc #7015114, “Tivoli Field Guide - Data Protection for Domino Support for Domino 8.5.”

About the authors

Robert Steen joined IBM in 1998, and in 2003 he joined the Customer Support team, since which time he has been working with the Server Core team, assisting numerous customers from all over the world. You can reach him at RSDU1344@ie.ibm.com.

Marcin Nalewajko joined IBM in 2008 and has been working in Customer Support on the Server Core team for the past two years. His focused has been on database issues and DAOS, working with many customers and the developers of this new functionality. You can reach him at MNALEWAJ@ie.ibm.com.


Acknowledgement:
The authors extend a special thanks to Patrick Mancuso and Declan P Byrne.