When deploying message-oriented middleware like IBM MQ that sustains high volumes of concurrent transactions, running out of available file handles is a common yet preventable installation failure mode. In this post, we‘ll cover why file handles are critical for MQ, how to diagnose exhaustion issues, and the specific steps to increase limits appropriately on Linux to enable successful MQ installations.
IBM MQ Architecture Reliance on File Handles
Behind the scenes, IBM MQ relies extensively on file handles for queue management, spooling messages to durable storage, and shuttling data between processes. Each connection to a MQ queue manager can consume several file handles for sending and receiving messages. The MQ architecture is optimized for throughput over raw speed per message, queueing up and batching operations. This translates to significant file IO.
As a concrete example, a simple MQ test cluster with 2 queue managers, an MQ client, and utilizing disk persistence had open files usage like:
Queue Manager 1: 1200 file handles Queue Manager 2: 1150 file handles MQ Client: 300 file handles Total Cluster: 2650 file handles
And this scales linearly as more queue managers get added and client connections grow. So a realistic production environment could easily demand 20,000+ simultaneously open files.
If each client app only allowed 1024 open files, then MQ ops would fail after just 10-20 clients attempting concurrent work. Similarly if the kernel file-max parameter capped at just 200,000 handles, there could be failures above 75-100 queue managers. This illustrates how quickly default file handle limits can overflow and prevent MQ usage at scale.
Real-World MQ Installation Failures
Here are some examples of actual client cases hitting file handle limits during MQ installations and upgrades:
"User mqm exceeds max open files limit during MQ v9 installation. Setup fails with cryptic errors. Taking inventory of limits shows file-max at 250,000 and mqm ulimit -n at 1024.""Upgrading existing MQ 8 cluster to v9 now gets random application connection failures. Checking logs shows occasional ‘too many open files‘ errors accross queue managers. Compare resource usage to existing production and realize 2x growth in clients exhausted default file handles."
The root causes came down to:
- Not adequately scoping scaling requirements upfront
- Unawareness of how Linux file handle limits impact messaging middleware
- Lack of observability into file usage metrics
This highlights the need for capacity planning on infrastructure resources like file handles before embarking on MQ deployments.
Checking Your Current File Handle Limits
To avoiding hitting file handle exhaustion issues, the first step is checking your current limits:
$ sysctl fs.file-max fs.file-max = 2097152$ ulimit -Hn 4096
The output shows system file-max at 2 million handles, and the per process limit at 4096. Now we need to determine if that aligns with the expected MQ scale. As a rule of thumb:
- file-max should have at least a 20% buffer over total projected handle usage
- nofile should be at least double expected per-process usage
Without above headroom, random spikes in utilization can still risk handle exhaustion.
Adjusting the Kernel File Limit
IBM recommends setting file-max to at least 524,288 for MQ. We could determine based on our plans that 2 million remains reasonable. Ensure file-max gets set appropriately by editing /etc/sysctl.conf:
fs.file-max = 2097152
To load the new value immediately without a reboot:
$ sysctl -p
Now file-max is configured with sizable headroom over the observed 2650 baseline handles.
Increasing the MQ mqm User nofile Limit
Additionally, edit /etc/security/limits.conf to raise the nofile handle limit for MQ users:
mqm hard nofile 16384 mqm soft nofile 16384
Here we took the 1024 default and bumped it to 16,384 or 2x the baseline usage. Combined with the increased kernel file-max, there is now sufficient cushion to support significant future growth.
MQ Installation Now Succeeds!
With these simple but important Linux file handle limit adjustments in place beforehand, the MQ deployment now succeeds without any troublesome warnings or failures around handle exhaustion!
Proactively configuring headroom on all infrastructure dependencies like files, processes, memory, tcp ports etc based on expected scale is vital for smooth deployments and avoiding future platform surprises. For even more MQ capacity planning guidance, check out IBM‘s documentation on tuning recommendations across Linux distributions.
Now on to tackling the next #middlewareproblems challenge – sizing those dang MQ transaction journal volumes! But that‘s a topic for another day. Reach out below if you have any questions on file handle limits or optimizing message queue platforms.
