The content within needs attention, July 2012.
This document serves as a checklist or guide to those who need to optimize and tune Sun Java System Messaging Server (formerly: Sun ONE Messaging Server and iPlanet Messaging Server) running in the Solaris Operating Environment. USE THIS INFORMATION AT YOUR OWN RISK! WE ARE NOT LIABLE FOR ANY LOSS OF MAIL OR OTHER HARM THAT MAY COME TO YOUR SYSTEM! Again, you assume all responsibility and risk of using information contained herein.
As with any system, performance is a key element to getting the most return on investment as well as maintaining happy users. This document contains practices and principles specifically related to capacity planning and performance of a Sun Messaging Server which can differ or contradict conventional tuning wisdom. This document points out the landmines and areas that a Sun Messaging Server administrator should concentrate on.
Some general tips on tuning. If you change something and do not see the desired result, or any result, you should probably change things back. Changing too many things at once can cause you more harm than good, which you may not see the negative impact of in the near future. Be careful, document, document!
Be careful when tuning the number of threads in processes. Enabling another thread has side effects which include things like: virtual memory and file descriptors.
# set hard limit on file descriptors
# set soft limit on file descriptors
Ideally you do not need to set this. Setting maxusers to the maximum value (2048) should allow the system to automatically tune itself. You can use the following command
vmstat -s | grep cache
to see the percentage of hits against the Directory Name Lookup Cache (DNLC). You want this to be as high as possible. If after setting maxusers the cache hit rate against DNLC is not high enough then setting ncsize is probably required.
ncsize = (4 * (max_nprocs + maxusers)) + 320
max_nprocs = 10 + (16 * maxusers)
maxusers = physmem – 2
Maximum Physical I/O Size
Increase maximum physical I/O size, the value shown here should work for nearly all controllers.
You should also read this page as it covers this topic in depth.
Authentication Cache Size
Set service.authcachesize equal to a number larger than the max concurrency of your user base. Setting this number higher than what your hardware can support could allow a Denial Of Service attack.
Authentication Cache TTL
Set service.authcachettl equal to the number of seconds you want entries kept in cache. You need to weigh the problem of user password changes being seen against the performance hit of the mail system queries against LDAP.
Set local.ugldaphost equal to at least two dedicated LDAP consumers. If the first host should be unreachable or otherwise not working as expected then new connections will be created to the next working host in the list.
LDAP Pool Refresh Interval
The length of time, in minutes, before LDAP connections are automatically closed and then re-established to the LDAP server. To adjust this timing, set the configutil parameter local.ldappoolrefreshinterval. The default value for this parameter is to never refresh. This also controls the length of time elapsed until the messaging server reverts back to the primary directory server.
The default timeout should an LDAP connection fail is 3 minutes, this can create a large overhead in failovers. There is an alternate method. Set local.ldapconnecttimeout to a value in seconds. This will enable a different connect function in the LDAP library. The timeout value should be chosen carefully. You need to consider things like temporary network issues and a very busy directory server. Setting the value to low will cause the mail server to close its current connections and start up new connections to the next server in its list. (Anyone have a suggested value? 30?)
Number of processes
Number of Processes
Set service.[POP|IMAP|HTTP].numprocesses, the default is 1. You want to set this high enough to support your user load but never higher than the total number of CPUs in the system.
I am looking for concrete information on this topic as in how to best guide people to tune their systems. At this point I do not have a lot of validated information. Something I do know is that you should not set maxthreads to be more than 250.
Store DB Cache Size
Turns out that I did not have all of the information regarding this parameter.
For deployments where the clients are mostly IMAP and most folders are not modified frequently, you do not want to set dbcachesize to the sum. Setting dbcachesize equal to the sum of *.db files can actually have negative impact on performance. This would result in the caching of data which is mostly stale. This would cause the process which syncs the in-memory database cache to disk to run longer due to the increased cache size.
The ideal setting of dbcachesize can be found in the following way:
1. Run ‘db_stat -m’ or ‘db_stat -m -h /tmp/msg-<instance>’
(if store.dbtmpdir is set to /tmp/msg-<instance>).
2. The 3rd line tells you your current dbcachesize.
3. The 4th line shows the “hit rate”, if it’s close to 99%, your dbcachesize is correctly sized. Note: right after the messaging server starts up, the “hit rate” will be low; therefore you should let messaging server run for a period of time before assessing your “hit rate”.
Sites have reported that the default of 16384K shows a “hit rate” of 99-100% even on our biggest installations (with over 200,000 folders).
Store Database Temporary Directory
Set store.dbtmpdir equal to /tmp/msg-<instance>. This parameter is related to Store DB Cache Size. Make sure the /tmp/ partition (or another tmpfs based file system) has enough free space to hold the database cache, that is the value of the store.dbcachesize.
Note — the msg-&60;instance> is important, do not use just /tmp/, a subdirectory must be used. The file names placed in this location conflict with the MTA parameters of IMTA_TMP and IMTA_SCRATCH.
User and Group Bind DN
If you are using Netscape Directory Server 4.x series, then you should set local.ugldapbinddn equal to “cn=Directory Manager” as this will greatly improve the response time. If you are using Sun ONE Directory Server 5.x then you do not need to worry about this parameter. Though I have seen large sites (2 Million or more accounts) where this setting this to Directory Manager still provides a significant performance improvement. While I dislike setting it to Directory Manager for security reasons, sometimes the performance impact out weighs the risk.
Webmail Spool Directory
This is the directory where webmail places out going messages from clients. If you have lots of webmail users you may want to consider setting this configutil variable, service.http.spooldir, to a fast file system.
MMP Tuning (Mail Multiplexor)
LDAPCacheSize and LDAPCacheTTL
The MMP can cache results of user searches. The LdapCacheSize parameter defines the number of cache entries; LdapCacheTTL defines the length of time the entries are preserved in seconds. Higher values will increase performance, but will result in delayed recognition of LDAP data changes, for example password changes. Lower values will reduce performance of the MMP, but will result in faster recognition of LDAP data changes.
The default LdapCacheSize is 10,000; the default LdapCacheTTL is 900. These parameters do not apply to SMTP proxy.
The text below is actually comments in the configuration file. Shown here so you know about the parameter and can set it correctly for your environment.
# number of worker threads allocated for the AService
# daemon. Optimally, it should be equal to the number of
# processors on the machine, unless an AService DLL does
# synchronous handling of connections (Imap and Pop
# Proxy do not).
MTA Tuning (Message Transfer Agent)
Note to self — add something about LDAP_USE_ASYNC
Changes to imta_tailor
If you are using Direct LDAP lookups instead of Dirsync then you can also set this variable to a value that maps to a memory mapped file system, like /tmp/. The location must exist prior to starting up the MTA, i.e. the MTA will not create this directory if it is not present. I often use simply /tmp/.Please note that this can not be set to the same location as Section 6.3
Set this parameter to a value that maps to a memory mapped file system, like /tmp/. Channels such as the conversion channel place files in this directory. The location must exist prior to starting up the MTA, i.e. the MTA will not create this directory if it is not present. I often use simply /tmp/. Please note that this can not be set to the same location as Section 6.3
If you wish, you can set the IMTA_TCP_FLAG_RETENTION option in your imta_tailor file to, say 1, so that the old *.data-failed files get purged after a day. You can find these files in msg-<instance>/imta/ queue/<channel-name>/spool/. Note — this directory will only exist if the MTA needed to write files to this directory. Many sites will never see this directory on their systems.
If you are using the conversion channel to perform anti-virus checking, in addition to the other things in this document, you should know about the following software.
Direct LDAP Considerations
If you’re using Direct LDAP lookups the following parameters should be set in option.dat:
Note: If you’re using iMS 5.2 or later then YOU SHOULD BE USING Direct LDAP, instead of Dirsync!
To get the maximum gain from DirectLDAP Mode, you should also look at adjusting MAX_LIFE_CONNS see Section 8.1 for more information.
ALIAS_ENTRY_CACHE_SIZE — this is the maximum number of entries held in the cache. The default value is 1000. Each cache entry is about 2K.
ALIAS_ENTRY_CACHE_TIMEOUT — this is the maximum length of time, in seconds, an entry is held in cache, the default value =1200
ALIAS_ENTRY_CACHE_NEGATIVE — controls if alias match failures are cached. The default action is not to cache failures. This speeds up the activation of new users.
You want to allow enough connections to support your load, while not allowing enough concurrent connections that your system is not able to respond quickly, thus preventing a DOS attack. The max concurrency number is equal to MAX_PROCS * MAX_CONNS, defaults are 10 and 20 respectively.
Once you know the max concurrency rate for your given configuration this will help you determine the number of systems required to support the load at your site.
MAX_LIFE_CONNS should be adjusted if you are using Direct LDAP Mode, every site should be using Direct LDAP Mode, if at all possible. This should be at least twice, and probably four times ALIAS_ENTRY_CACHE_SIZE , to take advantage of the cache. See Section 8.11
To find an optimum value for MAX_HANDOFFS (Check here) (default value is 5), you need to figure out how long it takes for a connection to be ACCEPTED (you can look in Dispatcher debug logs for that), and then estimate how many connections you expect the Dispatcher to receive during that time period. If the time between the connection coming into the Dispatcher and being ACCEPTED by the worker process is 0.5 seconds, and the system should be able to handle 30 connections/second, then MAX_HANDOFFS would need to be set to at least 15 (30 conns/second * 0.5 seconds).
There are two files that you need to be concerned with, imta.cnf and job_controller.cnf. In job_controller.cnf you define the pools and the maximum number of jobs that can be run at a given time in those pools. In the imta.cnf file you define what pool a channel uses and the maximum number of processes the channel can run in that pool at a given time.
In large deployments machines are dedicated to a specific role. On a MTA-IN machine one would configure more jobs to the tcp_intranet channel and thus deliver mail faster to your mail stores.
In large sites they may wish to adjust MAX_MESSAGES. The Job Controller keeps information about messages in an in-memory structure. In the event that a large backlog builds, it may need to limit the size of this structure. If the number of messages in the backlog exceeds the parameter specified here, information about subsequent messages is not kept in memory. This parameter is set in the global section of the job_controller.cnf file, this variable is not present by default. The default value is 100000. For an in depth explanation about MAX_MESSAGSES see my personal website, http://ims.balius.com/ and look for the topic max messages.
The default value for notices in iMS is ’1 2 4 7′. In large deployments you may want to reduce these defaults.
The default for Delivery Status Notifications (DSNs) for postmaster are copywarnpost and copysendpost. In large deployments or environments where the postmaster does not want to get DSNs about users mail, then these keywords should be changed. See the Sun ONE Messaging Server documentation for a complete explanation of the values. (The possible keywords are XYZwarnpost and XYZsendpost, where XYZ is either copy, no, warn, or err.)
ims_master (ims-ms-daemon channel)
If the mail store is handling lots of mailboxes and messages per second then you may want to increase the number of ims_master processes that job_controller will start to process the ims-ms queue. To increase the number of processes you need to make two changes, one in the imta.cnf file and the other in the job_controller.cnf file. In the imta.cnf file you need to change the maxjobs keyword, in the job_controller.cnf file you need to adjust the job_limit for the IMS_POOL.
This setting controls how much memory the SMTP server uses to store a message before it creates files in Section 8.5.3. If you have enough memory in your server then you might consider setting this variable to a value that allows the MTA to store your average message size or more in memory
If the MTA does not need to rewrite backward pointing addresses then you can set USE_REVERSE_DATABASE=0 in option.dat. I have in the past set this parameter while trying to get every last milisecond of performance out of an MTA. Most ISPs will not require the MTAs to rewrite backward pointing addresses.
To prevent DOS attacks and protect overall system health, the following parameters should be enabled in the msg-<instance>/imta/config/tcp_local_option and also msg-/imta/config/tcp_intranet_option files. (They do not exist by default.)
The last two parameters should be set according to site policy. I am listing them here so you know they exist, check the reference guide for more options.
There are four parameters that are related and interact together to tell job_controller the number of processes that are responsible for delivery of messages.
job_limit — this is the maximum number of processes that can run in a given pool simulataneously. I am not aware of any method to view the number of processes in pool A versus pool B. To view all smtp client processes use the following ‘ps -aef | grep tcp_smtp_client’
maxjobs — this is a channel keyword. You can apply this parameter to each channel to set the maximum number of processes (tcp_smtp_client) that job_controller will start to process messages in this chanel.
MAX_CLIENT_THREADS — the default value is 10, this option is set in tcp_CHANNEL_NAME_option though these files are not present by default. This option controls the number of threads per process.
threaddepth — maximum number of messages per thread. To view the number of threads that a tcp_smtp_client process is currently using you can use the command top. The default value is 128. This only applies to multithreaded channels. Channels like reprocess and conversion are single threaded.
Adjustments to these parameters should be slow and gradual until you find the right combination for your environment.
If you need to decrease the amount of time it takes for delivery of messages from your MTA then you should adjust the parameters above.
Here is an example. I wanted the MTA to relay very quickly. The server had available resources and thus increasing the number of tcp_smtp_clients was possible. I set threaddepth to 32, max_client_threads to 30, maxjobs to 60, and job_limit was set to 100. Two channels were sharing the same pool, the other had maxjobs set to 40. The goal was to relay mail as fast as possible. The system was not under heavy load.
ims-ms channel specific information
The ims-ms channel is a multithreaded channel, but does not respect the threaddepth channel keyword. Instead this channel uses a hard-coded value of 5 (maximum of 5 messages handled per thread). The ims-ms channel does have a channel option to control how many threads will be used within a process, DELIVER_THREADS. This option would be placed in msg-/imta/config/ims-ms_option The default value for this channel option is 15. (Note like other channels the channel option file is not present by default).
Here you will find links to other web sites and a book that are worth reading.
Apply the following TCP/IP tunings to all mail servers. These may also be appropriate for LDAP servers as well. These values of these settings are geared towards high speed networks and lots of traffic to and from the servers.
# ** Performance related **
/usr/sbin/ndd -set /dev/tcp tcp_recv_hiwat 65536
/usr/sbin/ndd -set /dev/tcp tcp_xmit_hiwat 65536
/usr/sbin/ndd -set /dev/tcp tcp_conn_req_max_q 4096
/usr/sbin/ndd -set /dev/tcp tcp_conn_req_max_q0 8192
/usr/sbin/ndd -set /dev/tcp tcp_smallest_anon_port 8192
/usr/sbin/ndd -set /dev/tcp tcp_keepalive_interval 30000
/usr/sbin/ndd -set /dev/tcp tcp_naglim_def 1
# investigate this one
#/usr/sbin/ndd -set /dev/tcp tcp_ip_abort_cinterval 10000
# ** Security related **
# prevent address mask queries
/usr/sbin/ndd -set /dev/ip ip_respond_to_address_mask_broadcast 0
/usr/sbin/ndd -set /dev/ip ip_respond_to_echo_broadcast 0
# prevent smurf dos attacks
/usr/sbin/ndd -set /dev/ip ip_forward_directed_broadcasts 0
/usr/sbin/ndd -set /dev/ip ip_strict_dst_multihoming 1
/usr/sbin/ndd -set /dev/ip ip_forwarding 0
/usr/sbin/ndd -set /dev/ip ip_forward_src_routed 0
/usr/sbin/ndd -set /dev/ip ip_respond_to_timestamp 0
/usr/sbin/ndd -set /dev/ip ip_respond_to_timestamp_broadcast 0
# disable sending and accepting of ip redirects
/usr/sbin/ndd -set /dev/ip ip_ignore_redirect 1
# disable ICMP redirect messages
/usr/sbin/ndd -set /dev/ip ip_send_redirects 0
# prevent tcp/ip sequence prediction
/usr/sbin/ndd -set /dev/tcp tcp_strong_iss 2
# ** IPv6 security related **
# don't accept source routed packets on IPv6
/usr/sbin/ndd -set /deb/ip ip6_forward_src_routed 0
# disable the routing abilities for IPv6
/usr/sbin/ndd -set /dev/ip ip6_forwarding 0
/usr/sbin/ndd -set /dev/ip ip6_respond_to_echo_multicast 0
# disable ICMP redirect messages for IPv6
/usr/sbin/ndd -set /dev/ip ip6_send_redirects 0
# disable sending and accepting of ip redirects for IPv6
/usr/sbin/ndd -set /dev/ip ip6_ignore_redirect 1
# Solaris guide says not to set lower than 60 seconds
# should investigate further, but the following has worked
/usr/sbin/ndd -set /dev/tcp tcp_time_wait_interval 15000
# Set according to local specifics.
#/usr/sbin/ndd -set /dev/tcp tcp_mss_def_ipv4 1460
If your system is using a Veritas file system you may need to adjust
two important variables.
vxfs:vxfs_ninode - VxFS inode structures held in memory
vxfs:vx_bc_bughwm - the Buffer Cache√ïs Buffer High Water Mark
If your system uses Veritas Volume Manager (VxVM) then you should look at vxio:vol_maxio. This variable controls the maximum size of I/O requests that are sent down the SCSI chain without breaking the request up. Recommendations are that this tunable parameter not exceed 20% of kernel memory or physical memory (which ever is smaller). This should match the size of your widest stripe.
On the info-ims mailing list this topic was discussed. I encourage you to check out the thread that starts with http://lists.balius.com/pipermail/info-ims-archive/2003-August/002006.html or if that is not working then look for the subject “VERITAS File System and Message Server”.
There are four directories on a mail server, depending upon the
components being used on a given server, that have more I/O activity
than others. These directories are:
Those directories contain files that are accessed very frequently. Having those files on the same disk or set of disks can degrade performance. Ideally each of those directories will have its own disk or even better, a set of disk spindles (RAID).
When setting up disk stripping the stripe width should be about the same size as your average message. 128 is usually too large and has a negative performance impact. Instead values of 8, 16 or 32 are usually better.
No longer relevant…but not deleted
In this section you will find information that was once relevant but with the changes to the software the information is now not as relevant. Rather than delete the information, it has been moved under this section.
Directory Server Considerations
If you’re using Netscape Directory Server 4.1x then you should set
/usr/sbin/ndd -set /dev/tcp tcp_naglim_def 1
Solaris introduces a 100ms delay in TCP/IP. This parameter tells Solaris that any write that is smaller than N will be delayed. In iPlanet Directory Server 5 it is configurable, using the TCP_NODELAY flag, which is set by default, thus no delay for packets from the LDAP application. In other words in the 5.x series of directory server product you should not have to adjust this parameter, out-of-the-box it should work nicely.