As a Senior DevOps Engineer at WebToolsWiz.com, I frequently encounter unique challenges when deploying and maintaining robust database systems. One recurring issue that can plague MongoDB deployments on Ubuntu 22.04 is the dreaded “Broken Pipe” error. This guide will walk you through diagnosing and resolving this common problem.

---

Troubleshooting MongoDB “Broken Pipe” on Ubuntu 22.04

The “Broken Pipe” error in the context of MongoDB typically indicates that a client connection to the mongod server was abruptly terminated. While this can stem from various network or application-level issues, on Ubuntu 22.04, a specific configuration oversight is often the primary culprit.

1. The Root Cause: Systemd and File Descriptor Limits

Ubuntu 22.04, like many modern Linux distributions, relies heavily on systemd to manage services. MongoDB, being a high-performance database, requires a substantial number of open file descriptors (nofile) to operate efficiently. These file descriptors are used for network connections, data files, journal files, index files, and more.

The critical issue arises because the default nofile limits imposed by systemd on service units are often lower than what MongoDB requires, even if you’ve configured higher limits in /etc/security/limits.conf. The systemd service manager overrides these system-wide limits for services it starts unless explicitly configured within the service’s unit file or an override.

When MongoDB exhausts its allocated file descriptors, it can no longer handle new connections or even maintain existing ones, leading to “Broken Pipe” errors as clients suddenly lose their connection to the server.

2. Quick Fix (CLI)

The most direct way to resolve this is to increase the LimitNOFILE setting directly within the systemd service configuration for MongoDB.

1. Stop the MongoDB service:

   sudo systemctl stop mongod

2. Edit the MongoDB systemd service override file: This command will open an editor (usually nano or vi) to create or edit an override file for the mongod.service. This is the recommended approach as it won’t be overwritten by package updates.

   sudo systemctl edit mongod.service

3. Add or modify the following lines: Paste these lines into the editor. If a [Service] section already exists, just add LimitNOFILE under it.

   [Service]
   LimitNOFILE=64000

   MongoDB’s official documentation recommends a minimum of 64000 for nofile.

4. Save and exit the editor.

5. Reload the systemd daemon to apply changes:

   sudo systemctl daemon-reload

6. Start the MongoDB service:

   sudo systemctl start mongod

3. Configuration Check: Ensuring Persistence and Best Practices

While the quick fix addresses the immediate systemd limit, it’s good practice to ensure other relevant configuration files are correctly set for consistency and robustness.

3.1. Verify systemd Service Override

Confirm the LimitNOFILE setting is active for the mongod service:

systemctl show mongod.service | grep LimitNOFILE

You should see LimitNOFILE=64000.

3.2. /etc/security/limits.conf (Supplemental)

While systemd overrides this for services, it’s still good practice to set these limits for the mongodb user in limits.conf. This applies to processes started outside systemd (e.g., if you were to manually run mongod as the mongodb user for testing).

1. Open limits.conf:

   sudo nano /etc/security/limits.conf

2. Add these lines to the end of the file:

   mongodb soft nofile 64000
   mongodb hard nofile 64000

   soft limit is the current limit, hard limit is the maximum a user can set their soft limit to. MongoDB needs both set high.

3. Save and exit.

3.3. Kernel File Descriptor Limits (/etc/sysctl.conf)

Ensure the overall kernel-level maximum file descriptors are high enough to support all processes on the system, including MongoDB. This is less frequently the direct cause of “Broken Pipe” for a single service but good for overall system health.

1. Open sysctl.conf:

   sudo nano /etc/sysctl.conf

2. Add or modify the following line:

   fs.file-max = 2097152

   (A common value, adjust based on server’s total needs. Default is often sufficient for fs.file-max, but ensuring it’s not unusually low is wise).

3. Save and exit.

4. Apply the changes:

   sudo sysctl -p

4. Verification: Confirming the Fix

After making these changes, it’s crucial to verify that MongoDB is running with the new limits and that the “Broken Pipe” errors have ceased.

1. Check MongoDB Process Limits: First, find the Process ID (PID) of the running mongod instance:

   pgrep mongod

   Let’s assume the PID is 12345. Now, inspect its limits:

   cat /proc/12345/limits | grep "Max open files"

   You should see output similar to:

   Max open files            64000                64000                files

   This confirms mongod is now running with the desired file descriptor limit.

2. Monitor MongoDB Logs: Check the MongoDB logs for any new errors or indications of stability:

   sudo journalctl -u mongod -f

   Look for any recurring “Broken Pipe” or connection-related errors. You should ideally see clean startup messages and regular operations.

3. Test Client Connectivity: Try connecting to MongoDB from your application or using the mongosh client:

   mongosh --host <your_mongo_host> --port <your_mongo_port> -u <username> -p

   Perform some basic operations (e.g., db.adminCommand({ ping: 1 }), show dbs, some queries) to ensure stable communication.

4. Observe Application Behavior: The most important verification is that your applications that connect to MongoDB are no longer experiencing “Broken Pipe” errors and can consistently interact with the database.

By systematically addressing the systemd file descriptor limits, you should effectively resolve the “MongoDB Broken Pipe” issue on your Ubuntu 22.04 server, ensuring stable and reliable database operations.