How to Fix MongoDB Broken Pipe on Ubuntu 20.04

Troubleshooting “MongoDB Broken Pipe” on Ubuntu 20.04

The “Broken Pipe” error in MongoDB, particularly on Ubuntu 20.04, often signals an underlying resource exhaustion or system limit issue rather than a direct network problem. This guide will walk you through diagnosing and resolving this common issue.

1. The Root Cause: Why this happens on Ubuntu 20.04

The “Broken Pipe” error occurs when a process attempts to write to a communication channel (like a network socket or named pipe) that has been unexpectedly closed by the other end, or by the operating system itself due to resource constraints. For MongoDB on Ubuntu 20.04, this most frequently stems from insufficient system resource limits, specifically:

  • File Descriptors (LimitNOFILE / ulimit -n): MongoDB, especially under load, requires a large number of file descriptors. It uses them for data files, journal files, replication logs, and crucially, for every incoming client connection. Ubuntu 20.04’s default systemd unit file for mongod.service often sets LimitNOFILE to a value that is too low (e.g., 1024). When MongoDB hits this limit, it can fail to open new connections, manage existing ones, or even access its own data files, leading to connection drops and the “Broken Pipe” error being reported to clients or internally.
  • Processes/Threads (LimitNPROC / ulimit -u): While less common than file descriptor limits, insufficient process or thread limits can also contribute, particularly in very high-concurrency environments, as MongoDB uses threads for various internal operations.

When these limits are too low, the mongod process can become unstable, drop client connections prematurely, or fail to start new operations, manifesting as “Broken Pipe” errors.

2. Quick Fix (CLI)

To quickly apply a persistent fix to the systemd service limits, we’ll use systemctl edit to create an override file for the mongod.service unit. This is the recommended approach as it preserves your changes even if the main mongod.service file is updated during package upgrades.

  1. Open the systemd override editor:

    sudo systemctl edit mongod.service

    This command will open a temporary file in your default editor (usually nano or vi).

  2. Add/Modify resource limits: Paste or type the following configuration into the editor. We’ll set the file descriptor limit (LimitNOFILE) to a robust value like 64000 and the processes limit (LimitNPROC) to match. These are generally good starting points for production MongoDB instances. Adjust based on your specific workload and server capacity if needed.

    [Service]
LimitNOFILE=64000
LimitNPROC=64000

  3. Save and exit:

    • If using nano: Press Ctrl+X, then Y to confirm saving, then Enter.
    • If using vi/vim: Press Esc, then type :wq and press Enter.

    This action will create a file named /etc/systemd/system/mongod.service.d/override.conf with your specified settings and automatically run systemctl daemon-reload.

  4. Restart MongoDB: Apply the new limits by restarting the MongoDB service:

    sudo systemctl restart mongod

3. Configuration Check

While the systemctl edit command creates the necessary override, it’s beneficial to understand where these configurations reside and other potential areas to check.

Primary Configuration Location (systemd Service Limits):

  • Override File: The systemctl edit mongod.service command creates or modifies: /etc/systemd/system/mongod.service.d/override.conf This file should contain the [Service] block with LimitNOFILE and LimitNPROC as specified in the Quick Fix. This is the preferred method for setting service-specific limits.

  • Original Service File: The default limits are defined in the main MongoDB systemd unit file, typically: /lib/systemd/system/mongod.service Do not edit this file directly, as your changes will be overwritten during package updates. Use the override method instead.

System-Wide Limits (Less Impactful for systemd Services):

  • /etc/security/limits.conf: This file sets system-wide ulimit values for users and groups. While important for processes started directly by users (e.g., via SSH session), systemd service limits (like those in override.conf) take precedence for services it manages. You can still set limits here for the mongodb user for completeness, though it’s often not the direct fix for the “Broken Pipe” service issue.

    Example entry (for the mongodb user):

    mongodb soft nofile 64000
mongodb hard nofile 64000
mongodb soft nproc 64000
mongodb hard nproc 64000

    For these changes to take effect for limits.conf, a reboot might be required, or for specific users to log out and back in. For systemd services, the override.conf method is more direct and effective.

4. Verification

After applying the configuration changes, it’s crucial to verify that the new limits are active and that MongoDB is running without issues.

  1. Check MongoDB Service Status: Ensure the MongoDB service is running and healthy.

    sudo systemctl status mongod

    Look for Active: active (running) and check the latest log entries for any errors.

  2. Inspect MongoDB Logs: Review the MongoDB logs for any BrokenPipeError messages or other critical issues. The journalctl command is ideal for this.

    sudo journalctl -u mongod --since "10 minutes ago" -f

    Let this run for a few minutes, especially during periods of expected load, to ensure no new BrokenPipeError or Too many open files messages appear.

  3. Verify Running Process Limits: Confirm that the mongod process is actually operating with the new, higher limits.

    a. Find the MongoDB process ID (PID):

    pgrep mongod

    (This will usually return one or more PIDs. Pick the main mongod server process PID.)

    b. Check the limits for that PID: Replace <PID> with the actual PID you found.

    cat /proc/<PID>/limits | grep -E "Max open files|Max processes"

    You should see Max open files and Max processes (or Max user processes) reflecting your configured limits (e.g., 64000).

  4. Connect and Test Functionality: Connect to your MongoDB instance using the mongo shell or your application and perform various operations (read, write, aggregation) to confirm full functionality and stability under load.

By systematically addressing the system resource limits, you should effectively resolve “MongoDB Broken Pipe” errors on Ubuntu 20.04. If the issue persists, further investigation into network connectivity, client-side configuration, or MongoDB’s internal logging for other error indicators might be necessary.

⚡ Related Troubleshooting Guides