ShowTable of Contents
Overview
IBM® Lotus® Domino® 8.0.1 introduced Domino 64-bit for Microsoft® Windows® 2003/2008 64-bit and IBM AIX® 5.3/AIX 6.1 64-bit operation system (OS) platforms. Domino 8.5 64-bit is available on zLinux®. The main advantage of Domino 64-bit is that it dramatically increases memory address accessibility over Domino 32-bit.
This article discusses Domino 64-bit on Windows 64-bit and on AIX, comparing characteristics, advantages, and limitations. Domino 64-bit on zLinux and Domino 32-bit on a 64-bit OS are not addressed here. Within the same hardware configuration, and from existing testing to compare Domino 64-bit and Domino 32-bit, Domino 64-bit does not demonstrate obvious performance improvement. It can, however, increase scalability for each Domino server instance owing to enlarged process memory address space.
During the Domino 64-bit porting development process, there was no explicit design intent to take full advantage of 64-bit architecture, but rather to make only the necessary 64-bit-related adjustments in source code to the compiler and link with related 64-bit flags. This was a conscious business decision so as to make the 64-bit porting development process successful.
The same principle can be offered to customers and business partners to succeed in porting their Domino applications and products to 64-bit. If customers find it necessary to take certain 64-bit features for Domino 64-bit, IBM welcomes the feedback from customer implementations for future Domino 64-bit development considerations.
Domino 64-bit adoption is slow due to the investment in time, money, and effort required to install and configure 64-bit hardware and software. Users are naturally more comfortable using familiar, proven products because they know that these products work and are not risky.
The main concerns for users who want to migrate to 64-bit Domino are that it will be expensive, difficult, time-consuming, and incompatible with existing software. However, these concerns are exaggerated and are far outweighed by the potential benefits.
The 64-bit environment
The 64-bit environment or system contains both 64-bit hardware and 64-bit software components. The 64-bit hardware alone may improve performance somewhat and is normally designed to be backward compatible for 32-bit; however, keep in mind that the maximum benefit will be derived when 64-bit hardware
and software are working together.
64-bit hardware
Although 64-bit processing capability has been used in specialized applications for decades, only in recent years has it been widely available for AMD/Intel-based computers. Today, almost all new servers are 64-bit hardware based. In computer architecture, the term 64-bit largely refers to the design of the central processing unit (CPU), but it can also relate to the size of the external data bus. A 64-bit CPU can calculate individual tasks two times as fast as a 32-bit model and, in theory, it can address 2
64 =16 Exabytes (EB) of random access memory (RAM).
A server that has both a 64-bit CPU and a 64-bit data bus is better able to process and manage very large files for Domino server database applications. A 64-bit architecture provides more and wider general-purpose registers, which can contribute to enhance overall application performance.
64-bit software
The benefits of 64-bit hardware can be fully obtained only if the software that you install is designed to use 64-bit features of the hardware. The direct benefit of 64-bit software will be enlarged RAM memory and virtual memory, and reduced chance of memory fragmentation.
For Domino server applications, software includes 64-bit OS and Domino 64-bit. For different implementations for software and hardware, such as the OS, CPU, and memory and system boards, there are limitations and restrictions on maximum process memory address space. A Domino 64-bit application is restricted differently in the different hardware and OS environments.
Disadvantages of a 64-bit environment
Typically, a 64-bit OS and application requires more virtual memory than 32-bit servers; specifically, from testing, Domino 64-bit requires
27% more virtual memory for the same load on Domino 32-bit. The ability of a 64-bit OS and application to access very large physical and virtual memory space can cause address tables to grow, resulting in larger data-transaction overhead. Hence, in certain cases, small or repetitive tasks may run marginally slower than in a 32-bit environment.
Domino 64-bit
Table 1 lists the current Domino 64-bit releases and their supported platforms.
Table 1. Supported platforms
Platforms | Domino 64-bit
8.0.1/8.0.2 | Domino 64-bit
8.5/8.5.1 | Domino 64-bit
8.5.2 |
Microsoft Windows Server 2003 x64 Edition onward release* | Supported | Supported | Supported |
Microsoft Windows Server 2003*R2 x64 Edition onward release | --- | --- | Supported |
Microsoft Windows Server 2008 x64 Edition* | --- | Supported | Supported |
Microsoft Windows Server 2008 R2 x64 Edition* | --- | --- | Supported |
IBM AIX 5.3 64-bit | Supported | Supported | Supported |
IBM AIX 6.1 64-bit | Supported | Supported | Supported |
z-Linux (SLES 10/11 64-bit, RHEL 5 64-bit) | Not Available | Supported | Supported |
*Up to the Domino 8.5.1 release, Windows server 2003/2008 Datacenter edition are not supported.
Refer to the
Lotus Notes, Lotus Domino, and Lotus Domino Designer Release Notes for detailed Domino 64-bit system and patch requirements:
Detailed system requirements - Domino 8.0.1
Detailed system requirements - Domino 8.0.2
Detailed system requirements - Domino 8.5
Detailed system requirements - Domino 8.5.1
Detailed system requirements - Domino 8.5.2
For a high-level overview of Domino 64-bit, refer to the IBM Support Technote #1296452, "
FAQ: 64-bit version of Domino."
Table 2 lists the units of conversion used in this document, where B = byte.
Table 2. Units of conversion
Prefix | Symbol(s) | Power of 10 | Power of 2 | Number of bytes |
kilo | KB (kilobytes) | 103 | 210 | 1,024 |
mega | MB (megabytes) | 106 | 220 | 1,048,576 |
giga | GB (gigabytes) | 109 | 230 | 1,073,741,824 |
tera | TB (terabytes) | 1012 | 240 | 1,099,511,627,776 |
peta | PB (petabytes) | 1015 | 250 | 1,125,899,906,842,624 |
exa | EB (exabytes) | 1018 | 260 | 1,152,921,504,606,846,976 |
64-bit Domino address space
In the Domino server model for both Domino 32-bit and 64-bit, all Domino tasks are add-on tasks of the Domino server process and are shown as child processes of the Domino process in the OS. In Domino process terms, the process-addressable limitation refers to the maximum address limit of each individual Domino process, not just limited server processes.
Also, there is a large chunk of memory used for shared memory, which is mapped in process-addressable space. Each process might require additional process memory for additional private Domino memory, direct memory allocation, JVM, process executable and loaded linked library, process data, and stack.
Windows 64-bit OS
The x86-64 (also called x64) implementation for both Intel and AMD is an extension of the x86 instruction set; the marketing name used is AMD64 for AMD and EM64T/Intel 64 for Intel. It is fully backward compatible with 32-bit code, without emulation, because the full 32-bit instruction set remains implemented in the hardware, so that existing 32-bit x86 executables will run without compatibility or performance penalties.
However, x86-64 is different than Intel Itanium 64-bit architecture (IA-64), and the Domino 64-bit application does not support an Intel Itanium system based on 64-bit architecture.
On the hardware level, x86-64 adds support for 64-bit integer capability, 16 registers, 48-bit wider physical and virtual-memory addresses (256 TB), and numerous other enhancements. Its architecture definition allows both the memory-address limit to be raised to the full 64 bits. By the end of 2006, the majority of PCs' and workstations' hardware were 64-bit compatible.
Most applications will not need such a large address space for the foreseeable future, so Windows implementations for x86-64 are supporting only 16 terabytes (TB), or 44-bit, widths. This is because implementing full, 64-bit-wide virtual addresses would increase the complexity and cost of address translation with no real benefit.
Table 3 lists the memory limitations for 64-bit Windows releases.
Table 3. Memory limitations for 64-bit Windows releases
Windows 64-bit general memory limits |
|
Total virtual address space (based on a single process) | 16 TB |
Virtual address space per 64-bit process | 8 TB |
Paging file size | 256 TB |
Hyperspace | 8 GB |
Paged pool | 128 GB |
Non-paged pool | 128 GB |
System Page Table Entry (PTE) | 128 GB |
Windows 64-bit physical memory and CPU limits |
|
Windows Server 2003 Standard Edition | 32 GB / 1 to 4 CPUs |
Windows Server 2003 Enterprise Edition | 1 TB / 1 to 8 CPUs |
Windows Server 2003 Datacenter Edition | 1 TB / 8 to 64 CPUs |
Windows Server 2008 Standard Edition | 32 GB / 1 to 4 CPUs |
Windows Server 2008 Enterprise Edition | 2 TB / 1 to 8 CPUs |
Windows Server 2008 Datacenter Edition | 2 TB / 2 to 64 CPUs |
In Windows 64-bit architecture, the 16 TB of memory is divided evenly between user-mode processes and kernel-mode processes, in the same way as memory is handled in a 32-bit Windows OS. On Windows 64-bit, however, 64-bit applications have 8 TB of virtual-memory address space.
The increase in memory that accompanies 64-bit computing simplifies the creation of kernel-mode drivers. Kernel-mode drivers that run on 64-bit architecture don't need to deal with Physical Address Extensions (PAE) when physical memory sizes are larger than 4 GB; for a 32-bit OS, PAE is actually a 36-bit implementation that accesses 64 GB RAM.
With 32-bit systems, special accommodations are required to handle physical addresses above 4 GB, which make the device drivers more complex and error prone. Whereas, in 64-bit systems, kernel-mode drivers can address all physical memory directly.
Beyond virtual-memory address space, increasing actual physical memory---supported under the x64 editions of Windows---provides a significant increase in server scalability. Processor scalability is also an important consideration, and this will increase overall system parallel-processing capability. Windows Server 2003 Datacenter x64 Editions support up to 64 processors.
There is also significantly better multiprocessor scalability of .x86-64 processors over their 32-bit counterparts, providing the ability to directly address the extreme demands of memory- and processor-intensive applications, such as a very large database application.
On Windows platforms, shared memory and heap memory [for local memory like malloc() and Java JVM memory] can use maximum process addressable space. In theory, Domino 64-bit on Windows 64-bit can use up to 8 TB memory for combined shared memory, private memory, heap memory, and other process local memory.
The following example NSD log shows application-maximum 8 TB address space on Windows 64-bit:
OS Version : Windows/2003 5.2 [64-bit] R2 (Build 3790), PlatID=2, Service Pack 2 (8 Processors)
Num Processors : 8
CPU Type : 8664
CPU Info: : AMD or Intel 64 bit
CPU Level : 6
CPU Revision : 3851
CPU Mask : ff
Min App Addr : 10000
Max App Addr : 7FFFFFEFFFF
Page Size : 4096
Allocation Gran.: 10000
For more details, refer to:
x86-64
Memory Limits for Windows Releases
Benefits of Microsoft Windows x64 Editions
Widnows 2008 R2 Edition Comparison by Technical Specification
Comparison of 32-bit and 64-bit memory architecture for 64-bit editions of Windows XP and Windows Server 2003
64-bit memory model on AIX
In an AIX 64-bit world, there is no need to juggle the limited number of segments, as in the 32-bit world. There is a huge amount of available memory address space for both shared memory and process heap memory. In this section, we'll provide a brief summary of the 64-bit user-process address space on AIX to show how the memory model works.
Address space of a 64-bit process on AIX
For 32-bit applications on AIX, there is 2
4 =16 segment with 256 MB for each segment size limitation. On AIX, the 64-bit user- process model shares the same concept of segments as the 32-bit user process model. Each segment size is still 256 MB, but the number of available segments in the address space is 2
32=1 exa-segments, instead of 16 segments. The AIX 64-bit model actually is a 60-bit width implementation and still does not fully use 64-bit width; therefore, the user process can address up to 1 EB, which can be calculated as follows:
2
32 segments x 256 [MB/segment] = 2
32x 2
28 bytes = 2
60 = 1 EB
To address this tremendously huge space, the pointer type is defined as 64-bit in the 64-bit user-process model. Figure 1 shows how segments are used in this huge address space.
Figure 1. 64-bit process address-space model
In this figure, the first 16 segments (0 - 4GB) are exempt from general use in order to keep the backward compatibility with the 32-bit user process model.
Segment 16 to 7*16
7 (4 GB - 448 PB=447.996 PB) are used for user text, data, and heap. The user text is mapped into the first segment in this area. Also, user data is mapped into another segment in this area. In both cases, if a segment is not sufficient to contain text or data, another segment will be contiguously attached to the process address space. The heap memory acquired by malloc() is also allocated in this segment. Domino server will use this segment for JVM, process private memory and direct memory allocation (such as Lotus script memory and full text index).
The next 16
7 segments (448 PB - 512 PB = 64 PB) are used for a 64-bit process to call shmat() or mmap() routines to attach shared memory to the process. This is area where Domino allocate shared memory.
The next 16
7 segments (512 PB - 576 PB = 64 PB) are for objects that are loaded into the address space privately, such as using dlopen( ) or load( ) system calls, or using shared object files that do not allow others to read and execute. Sometimes this is done by third party middlewares.
The next 16
7 segments (576 PB - 640 PB = 64 PB) are used to load 64-bit shared library text and data that are to be shared by all 64-bit user processes on the system. Also, shared library data will be created in another segment in this area for this process' private use. In both cases, if there is a segment that has enough free space to contain shared text or shared library data, that segment will be used. Otherwise, another segment will be attached to the process address space.
The next 5*16
7 segments (640 PB - 960 PB = 320 PB) are reserved by the system and prohibited from the user process access.
The last 16
7 segments (960 PB - 1 EB = 64 PB) are used by a 64-bit user process for user stack. The stack grows from the last address, 0x0FFF_FFFF_FFFF_FFFF toward the first address in this area, to use more than one segment for user process stack.
Inter-Process Communication (IPC) limits on AIX 5.3 and 6.1
Table 4 shows the restrictions for shared memory on current AIX 64-bit implementations. For more details refer to the AIX information center topic, "
Inter-Process Communication (IPC) Limits."
Table 4. Shared memory limitations
Shared memory | AIX 5.3/6.1 |
Maximum segment size (64-bit process) for 64-bit kernel | 32 TB |
Minimum segment size | 1 B |
Maximum number of shared memory IDs (64-bit kernel) | 1048576 |
Maximum number of segments per process (64-bit process) | 268435456 |
On AIX, Domino 64-bit defaults to allocate 512MB for each shared-memory allocation, while Domino 32-bit allocates 256MB for each allocation. Figure 2 shows an example for Domino 64-bit.
Figure 2. Domino shared-memory segment size
Managing Domino 64-bit memory
Domino addressable memory and shared memory
As with Domino 32-bit, Domino 64-bit takes the maximum OS memory limitation for addressable and shared memory. Table 5 summarizes Domino 64-bit limitations.
Table 5. Summary of Domino 64-bit limitations
| Windows 64-bit OS | AIX 64-bit OS |
Process Addressable Memory | 8 TB | 1 EB |
Shared Memory Limitation | 8 TB | 64 PB |
Process heap limitation | 8 TB | 447.996 PB |
Data model on 64-bit
There are three popular data models for 64-bit platforms, LP64,
LLP64, and ILP64 (see table 6).
Table 6. Data models
Data model | short (integer) | int
(integer) | long
(integer) | long long | pointers | operating systems |
LP64 | 16 | 32 | 64 | 64 | 64 | Most UNIX® and UNIX-like systems, e.g., AIX, Solaris, Linux, and Mac OS X |
LLP64 | 16 | 32 | 32 | 64 | 64 | Microsoft Windows |
ILP64 | 16 | 64 | 64 | 64 | 64 | HAL Computer Systems port of Solaris to SPARC64 |
Windows 64-bit uses the LLP64 data model, in which only pointers expand to 64 bits; all other basic data types (integer and long) remain 32 bits in length. In Windows 64-bit, making all data types 64 bits in length would waste space because most applications don't need such increased size. However, applications need pointers for 64-bit data and also need the ability to have 64-bit data types in selected cases.
These considerations led to the selection of an abstract data model called LLP64 (or P64). Based on Domino being a cross-OS platform application, the LLP64 data model is selected for Domino 64-bit on Windows 64-bit, and is also applied on UNIX 64-bit platforms. For more details, refer to
Lotus C API Notes/Domino 8.5 Reference, Chapter 3-4, “Porting 32bit Domino applications to 64bit Domino”.
Domino 64-bit porting
During the Domino 64-bit porting development process, to ensure the porting project was successful, the Domino Development team made a conscious business decision to not take full advantage of the 64-bit architecture. Instead, the Domino 32-bit source was kept intact as much as possible, and only the necessary 64-bit-related adjustments were made in the source code to compiler and link with related 64-bit flags.
When comparing a Domino 64-bit memcheck section in an NSD log with one from Domino 32-bit, you will notice almost no change in internal-memory-structure default limitations and restrictions. For example, in the transaction log module, the holding memory is still 4GB size, and reaching the limit would cause a "transaction log file full" error on Domino 64-bit. So, basically almost all designs and restrictions for Domino 32-bit are carried forward to Domino 64-bit.
Table 7 lists the known changed default values for Domino 8.0.x/8.5.x releases:
Table 7. Changed default values (in MB)
Item | Domino 64-bit | Domino 32-bit |
Default Maximum size of the Unified Buffer Manager (UBM), BLK_UBMBUFFER | 1024 | 512 |
Shared-memory segment size on AIX | 512 | 256 |
For more information, refer to the Support Technotes #1268988, "
New default limit on UBM size in Domino 8,"
Configuring Domino 64-bit memory management
At the current Domino release, there are no 64-bit-specific, additional configuration steps necessary, such as notes.ini parameters. In general, there is no need to increase every possible adjustable memory block in Domino 64-bit, unless there is a specific benefit in doing so.
Because Domino 64-bit can address much larger shared-memory and private-memory sizes, there are benefits to increasing certain Domino memory-structure default values, to improve overall performance---provided there is adequate physical memory available in the system.
In Domino 64-bit , it still requires notes.ini NSF_BUFFER_POOL_SIZE_MB to control UBM memory usage, but does not require ConstrainedSHM or AddressableMem. For more information, refer to the Support Technote #1286171,
"Recommendations for setting NSF_BUFFER_POOL_SIZE_MB."
In Domino 32-bit, by keeping the lower default value, you limit the overall default Domino server memory footprint. For example, increasing NLCache_Size=268435456 (256MB) will enlarge the namelookup cache to make more cached names. This is no longer an issue on Domino 64-bit, in which each Domino instance is made more scalable to meet today's datacenter design requirement.
Domino server and HTTP thread-pool sizes also can be increased in heavy-load environments because large amounts of threads add to memory consumption, from initialization to load up, and can easily cause Domino server and HTTP processes to reach the Domino 32-bit limit. So, you need to consider CPU utilization to manage large amount of process thread-pool size.
Use Support Technote #1324864, "
SERVER_MAX_CONCURRENT_TRANS and SERVER_POOL_TASKS defaults for Domino 8.0 and above," as a guideline for tuning the Domino 64-bit thread pool, and certainly consider increasing the value of the total thread number as increased memory limitation.
Due to the dramatically increased file-system cache size on Windows 64-bit, it is necessary to constrain the file system cache on 64-bit Windows to limit its usage and to avoid taking out system memory. Refer to the "Constraining the file system cache on 64-bit Windows" topic in the Domino 8.0.2
Release Notes for more information.
Domino 64-bit deployment considerations
Planning
As mentioned in the
FAQ: 64-bit version of Domino Technote, a Domino 64-bit system should be sized just as the current-release Domino 32-bit system; the key reason being that there is no change in default values and restrictions for the Domino internal memory structure when porting Domino 64-bit in the current release.
When planning a Domino 64-bit deployment, you must carefully verify 64-bit hardware, 64-bit OS, and obtain an inventory for all existing and proposed applications, carefully verifying any application to be added to Domino 64-bit, including:
- add-on applications,
- linked library files,
- monitoring/control applications,
- single sign-on applications,
- DSAPI,
- backup software,
- antivirus applications,
- reporting applications, and
- ODBC drivers.
Also, note that:
- Any 32-bit application binary for Domino will not be compatible with Domino 64-bit; for example, Sametime 32-bit releases cannot be run on Domino 64-bit.
- Only 64-bit add-on applications can be run with 64-bit Domino; for example, Lotus Enterprise Integrator 64-bit can be run on Domino 64-bit.
- Existing databases and Java and LotusScript applications, such as agents within Domino, will continue to run successfully. As best practice you should recompile the applications after successfully upgrading to Domino 64-bit.
Porting customized applications on Domino 64-bit:
- Any C API applications (Domino server add-on extension manager DLLs) must be verified to be 64-bit compatible. If required, they will be recompiled and linked with appropriate 64-bit flag options on the latest Domino release toolkits.
- Domino 64-bit and Domino 32-bit are based on the same code base, thus maintaining the same API and reliability. You must verify the building requirement according to the Domino C API User Guide.
- ODBC drivers must be upgraded to be compatible with 64-bit (64-bit native ODBC driver) corresponding to 64-bit OS; ODBC 32-bit cannot be used in Domino 64-bit release.
For more details on Domino 32- to 64-bit porting considerations, refer to the “Migrating Domino 32-bit applications to the 64-bit platform” section of the Domino 8.0.2
Release Notes.
Other considerations:
Other relevant Technotes:
Domino 64-bit ISV product support
One of major roadblocks for adopting Domino 64-bit is availability of compatible Domino 64-bit add-on Independent Software Vendor (ISV) products. Availability of these products is a critical factor for many customer deployment plans. The following are known Domino 64-bit ISV partner products:
EMC SourceOne 6.6 supports 8.5.x 64-bit support on AIX 5.3/6.1 and Windows 2003 SP2 64-bit/2008 SP2 64-bit.
IBM Tivoli Storage Manager for Mail V5.5.3 supports Domino 8.0.2 and later, and Domino 8.5 and later 64-bit on AIX and Windows. (See Technote #1426974, "V5.5.3
Hardware & Software Requirements for Data Protection for Lotus Domino" and #1303402, "
Does IBM Tivoli Storage Manager for Mail have a 64-bit version for Domino?"
McAfee Security for Lotus Domino, version 7.5 (formerly GroupShield for Lotus Domino) is available with support for Domino 8.5 64-bit on Windows platforms.
Symantec Mail Security for Domino v8.0.5 (certified with Domino 8.5.1) supports Domino 64-bit on AIX and Windows
Symantec NetBackup 6.5.3 supports Domino 64-bit on all platforms
Symantec BackupExec 2010 Supports both 32- and 64-bit versions of Lotus Domino, including partitioned and clustered Domino servers and new support for the DAOS store and associated NLO files.
TrendMicro ScanMail for Domino version 5.0 (build 1170) for 64-bit Domino 8.5 on Windows 2003/2008.
Install and upgrade
Domino 64-bit installation and upgrading are almost identical to Domino 32-bit, but there are some differences. For upgrading best practices, refer to the
Upgrade Cookbook wiki article.
When upgrading from Domino 32-bit to 64-bit, it is important to run the updall -R task after the installation process, to update the view. (Refer to “Known issues” in the Domino 8.0.2
Release Notes.)
On Windows, upgrading Domino 64-bit directly from an existing 32-bit installation kit is not supported. Domino 32-bit must be uninstalled before you install Domino 64-bit. Refer to the section, “Prevented from upgrading to W64 kit” in the 8.0.2
Release Notes.
Also, refer to Technote #1306610, '
Error: "One or more of the IBM license agreement files were not created properly because the correct JRE was not used" during Domino 64-bit install.'
Conclusion
This article has provided a detailed discussion of Domino 64-bit on Windows 64-bit and AIX 64-bit. You should now be equipped with the knowledge and skills to deal with Domino 64-bit support.
About the author
Frank Meng has worked for IBM in various Support roles since 1997. He is based at IBM's Singapore facility and is the team leader for the APAC SWAT team. You can reach him at
mengfr@sg.ibm.com