Dec 23, 2025

How to Fix MongoDB Broken Pipe on Azure VM

Troubleshooting “MongoDB Broken Pipe” on Azure VM: A DevOps Guide

As Senior DevOps Engineers, few messages are as frustrating as “Broken Pipe” when dealing with critical database services like MongoDB. On Azure Virtual Machines, this error often points to specific resource management challenges inherent to cloud environments. This guide will walk you through diagnosing and resolving the “MongoDB Broken Pipe” issue on an Azure VM, from immediate fixes to long-term configuration stability.

1. The Root Cause: Why This Happens on Azure VM

A “Broken Pipe” error signifies that the connection to MongoDB was unexpectedly terminated by the server-side process, rather than gracefully closed. On Azure VMs, this is predominantly due to one or more of the following:

Resource Exhaustion (The Primary Culprit):
- Out-of-Memory (OOM) Killer: This is by far the most common cause. Azure VMs, especially those with smaller SKUs (e.g., B-series, D2s_v3), can quickly run out of RAM under load. The Linux kernel’s OOM Killer then steps in to terminate the largest memory-consuming process – often mongod – to prevent system instability. This abrupt termination breaks all active client connections, leading to “Broken Pipe” errors.
- Disk Space Depletion: MongoDB requires sufficient disk space for data files, journal files, and logs. A full disk can prevent new writes, leading to process stalls or crashes.
- Inadequate Disk IOPS/Throughput: If your Azure Managed Disk (e.g., Standard HDD/SSD) can’t keep up with MongoDB’s I/O demands, the database can become unresponsive, leading to connection timeouts and ultimately, broken pipes from the client perspective. Premium SSDs are highly recommended for database workloads.
OS/MongoDB Configuration Issues:
- Low ulimit Settings: The number of open file descriptors (nofile) and user processes (nproc) for the mongod user can be too low, preventing MongoDB from handling many connections or opening necessary files.
- Incorrect WiredTiger Cache Size: If the storage.wiredTiger.engineConfig.cacheSizeGB setting in mongod.conf is too high, it can consume most of the available RAM, leaving little for the OS or other processes, thus inviting the OOM Killer. If too low, it can lead to excessive disk I/O.
- Too Many Connections: While MongoDB can handle a high number of connections, if net.maxIncomingConnections is set too high without adequate system resources, it can lead to resource exhaustion.
Network Instability / Timeouts (Less Common for Server-Side Error):
- While less frequent as a direct cause of MongoDB crashing, network issues or aggressive timeout settings on Azure Network Security Groups (NSGs) or Azure Load Balancers could contribute to client-side broken pipe errors if the server is merely unresponsive for too long, even if it hasn’t crashed. However, if MongoDB itself is terminating connections, it’s usually resource-related.

2. Quick Fix (CLI)

The immediate goal is to get MongoDB back online and gather diagnostic information. Connect to your Azure VM via SSH.

Check MongoDB Service Status:
```
sudo systemctl status mongod
```
(Or sudo service mongod status for older systems) Look for “active (running)” or “failed.” If failed, it will often provide a hint.
Inspect MongoDB Logs: The logs are your first and best source of information for why MongoDB stopped.
```
sudo journalctl -u mongod -f --since "1 hour ago"
```
(This shows recent systemd journal logs for mongod. Alternatively, check the MongoDB-specific log file, typically /var/log/mongodb/mongod.log):
```
sudo tail -n 200 /var/log/mongodb/mongod.log | less
```
Look for keywords: killed, out of memory, OOM, disk full, corruption, segfault, exception, shutdown. The OOM message is a strong indicator.
Check Disk Space: A full disk can prevent MongoDB from writing its journal or data files.
```
df -h
```
Ensure the partition where MongoDB stores its data (typically /var/lib/mongodb) has sufficient free space.
Check Memory Usage: If logs point to OOM, verify current memory status.
```
free -h
```
See how much RAM is free and if swap is being heavily utilized. If mongod isn’t running, it might have been using a large portion before being killed.
Restart MongoDB Service:
```
sudo systemctl restart mongod
```
Wait a few seconds, then check its status again (sudo systemctl status mongod).
Handle mongod.lock (Use with Caution!): If MongoDB failed to shut down cleanly, a .lock file might be present, preventing it from starting. Only remove this if you are certain MongoDB was not running and gracefully shut down (i.e., it was killed). Removing it while MongoDB is actually running or during a crash recovery can lead to data corruption.
```
# Check for the lock file
ls -l /var/lib/mongodb/mongod.lock

# If present and mongod is NOT running and failed to start:
sudo rm /var/lib/mongodb/mongod.lock

# Attempt restart again
sudo systemctl restart mongod
```

3. Configuration Check (Long-Term Stability)

To prevent recurrence, adjust both MongoDB and OS-level configurations.

3.1. MongoDB Configuration (`/etc/mongod.conf`)

Edit the primary MongoDB configuration file (usually /etc/mongod.conf or similar path):

WiredTiger Cache Size: This is critical. By default, MongoDB on systems with more than 1GB RAM allocates 50% of physical RAM minus 1 GB to the WiredTiger cache. This can be too aggressive on smaller Azure VMs.

# /etc/mongod.conf
storage:
  wiredTiger:
    engineConfig:
      # Set this to a specific value, e.g., 2GB or 4GB, depending on your VM's RAM.
      # A good rule of thumb: (Total RAM * 0.5) - 1GB, but be conservative.
      # For a 4GB VM, try 2GB. For an 8GB VM, try 4GB.
      cacheSizeGB: 2 # Example: For a 4GB RAM VM
      # Optional: Adjust page size and number of concurrent operations if needed
      # configString: "cache_size=2G,eviction_dirty_trigger=80,eviction_trigger=95,eviction_target=85,eviction_max_co_workers=4"

Important: Reduce this if OOM errors are prevalent. Give the OS and other processes breathing room.

Max Incoming Connections: If your application makes many simultaneous connections, ensure this is set appropriately, but don’t overprovision without sufficient resources.

# /etc/mongod.conf
net:
  port: 27017
  bindIp: 0.0.0.0 # Or specific IP if applicable
  maxIncomingConnections: 65536 # Default is often 65536, but ensure it's not too low.

Logging: Ensure logging is robust.

# /etc/mongod.conf
systemLog:
  destination: file
  path: /var/log/mongodb/mongod.log
  # Ensure this path is valid and accessible, and that disk isn't full.
  # verbose: true # Uncomment temporarily for deeper debugging, but disable in production.

3.2. Operating System Configuration

MongoDB requires specific OS settings for optimal performance and stability.

ulimit Settings: Increase the number of open file descriptors and user processes.
- For Systemd (Recommended for modern Linux): Create a systemd override file.
```
sudo systemctl edit mongod
```
  Add the following lines:
```
[Service]
LimitNOFILE=65536
LimitNPROC=65536
```
  Save and exit. Then reload systemd daemon:
```
sudo systemctl daemon-reload
```
- Alternative (/etc/security/limits.conf - might be overridden by systemd):
```
sudo nano /etc/security/limits.conf
```
  Add (or modify) these lines:
```
mongod soft nofile 65536
mongod hard nofile 65536
mongod soft nproc 65536
mongod hard nproc 65536
```
  Note: The user mongod is typical; adjust if your MongoDB runs as a different user. Then, restart the MongoDB service.
Swappiness: Databases perform best when they avoid swapping data to disk.
```
sudo nano /etc/sysctl.d/99-mongodb.conf
```
Add this line:
```
vm.swappiness = 1
```
Apply the change:
```
sudo sysctl -p /etc/sysctl.d/99-mongodb.conf
```
A value of 1 tells the kernel to swap out anonymous pages (application memory) only when absolutely necessary, prioritizing keeping application data in RAM.
net.core.somaxconn: For high connection loads, increase the backlog queue for network connections.
```
sudo nano /etc/sysctl.d/99-mongodb.conf
```
Add this line:
```
net.core.somaxconn = 65536
```
Apply the change:
```
sudo sysctl -p /etc/sysctl.d/99-mongodb.conf
```
Azure VM SKU & Disk Type:
- Scale Up: If OOM persists, the most direct solution is to scale up your Azure VM SKU to one with more RAM and CPU. Consider D-series or E-series optimized for memory.
- Premium SSD: Ensure your data disk is an Azure Premium SSD (P10, P20, etc.) for databases. Standard HDDs or even Standard SSDs often cannot meet the IOPS and throughput demands of MongoDB under load, leading to latency and unresponsiveness.

4. Verification

After making configuration changes, it’s crucial to verify stability.

Restart MongoDB Service:
```
sudo systemctl restart mongod
```
Check Service Status and Logs:
```
sudo systemctl status mongod
sudo journalctl -u mongod -n 50 --no-pager
```
Ensure it’s running cleanly and there are no new errors or warnings.
Connect from Client: From your application server or local machine, attempt to connect to MongoDB.
```
mongo --host <your_azure_vm_ip> --port 27017
```
Perform a simple read/write operation to confirm connectivity and basic functionality.
Monitor Resources:
- On the VM: Use tools like htop, top, or free -h to monitor RAM, CPU, and swap usage. Keep an eye on mongod’s memory consumption.
- MongoDB Specific: Use mongostat or mongotop to monitor database activity and resource usage from MongoDB’s perspective.
- Azure Monitoring: Leverage Azure Monitor and Log Analytics to track VM metrics (CPU, Memory, Disk IOPS, Network In/Out) over time. Set up alerts for high memory usage or low disk space.
Simulate Load (if possible): If this is a non-production environment, simulate typical application load to ensure the configuration holds under stress. Pay close attention to resource utilization during peak load.

By systematically addressing resource constraints and tuning both MongoDB and OS settings, you can significantly improve the stability and performance of MongoDB on your Azure VMs and banish the “Broken Pipe” error for good.