How to Fix Go Out of Memory (OOM) on Ubuntu 22.04

Troubleshooting Guide: Go Out of Memory (OOM) on Ubuntu 22.04

As a Senior DevOps Engineer, encountering Out Of Memory (OOM) errors is a critical incident that demands immediate and effective resolution. Go applications, while memory-efficient by design, can still fall victim to the Linux OOM Killer, especially in resource-constrained environments or under specific load patterns. This guide provides a structured approach to diagnose and mitigate OOM issues for Go services running on Ubuntu 22.04.

1. The Root Cause: Why Go Applications OOM on Ubuntu 22.04

Understanding the underlying mechanisms is key to effective troubleshooting:

  • Linux OOM Killer: This is the primary culprit. When the kernel detects a severe system-wide memory shortage, it invokes the OOM Killer to terminate processes to reclaim RAM. It uses a heuristic score (oom_score) to determine which process is the “least important” to kill. If your Go application has a high score, it’s a prime target.
  • Go’s Memory Management: Go applications manage their own memory through the Go runtime and garbage collector (GC).
    • Runtime Overhead: The Go runtime itself requires some memory.
    • Heap Allocation: Go applications often allocate large chunks of memory from the OS and manage them internally. The GC periodically reclaims unused memory.
    • Lazy Release: While the Go runtime uses madvise(MADV_DONTNEED) to signal to the kernel that memory pages can be reclaimed, this process isn’t instantaneous. Under high allocation/deallocation rates, the Resident Set Size (RSS) reported by the OS might appear high even if the application’s active memory usage is lower, making the Go app seem like a large memory consumer.
    • Memory Spikes: Sudden increases in request volume or data processing can cause temporary memory spikes that exceed available RAM before the GC has a chance to catch up or release memory back to the OS.
  • Ubuntu 22.04 and cgroup v2: Ubuntu 22.04 predominantly uses systemd which leverages cgroup v2 for resource management.
    • systemd Memory Limits: If your Go application runs as a systemd service with MemoryLimit set, exceeding this limit will trigger an OOM within that cgroup, potentially killing your application before a system-wide OOM event occurs.
    • Default Swap Policy: Many cloud instances, and sometimes default Ubuntu installations, come with minimal or no swap space. This removes a crucial buffer that the kernel can use to temporarily offload less active memory pages, making the system far more susceptible to OOM events under memory pressure.

2. Quick Fix (CLI)

These commands provide immediate diagnostic insights and potential temporary relief.

  1. Check for OOM Killer Events:

    dmesg -T | grep -i "oom-killer\|out of memory"
journalctl -xb -p 3 -n 50 # Review recent critical system messages

    This will show if and when the OOM Killer was invoked, and which process (hopefully your Go app) was killed.

  2. Identify Top Memory Consumers (before OOM):

    ps aux --sort -rss | head -n 10

    Use top or htop for a dynamic view. This helps confirm if your Go application is indeed the primary memory hog.

  3. Adjust OOM Score for a Running Process (Temporary): If your Go application is running and you want to temporarily reduce its likelihood of being killed, you can adjust its oom_score_adj.

    # Find your Go application's PID
pgrep -f "your-go-app-binary-name"

# Set oom_score_adj (e.g., to -500 or -1000 for higher priority)
# Replace <PID> with your application's process ID
echo -500 | sudo tee /proc/<PID>/oom_score_adj

    Caution: Setting a very low oom_score_adj might protect your Go app but could cause other critical system processes to be killed instead, leading to system instability. Use judiciously.

  4. Add a Temporary Swap File: If no swap is present, adding it can provide a buffer against immediate OOMs.

    sudo fallocate -l 2G /swapfile # Create a 2GB swap file (adjust size as needed)
sudo chmod 600 /swapfile       # Secure permissions
sudo mkswap /swapfile          # Set up the swap area
sudo swapon /swapfile          # Enable swap
sudo swapon --show             # Verify swap is active
sudo free -h                   # Check overall memory and swap usage

    This change is not persistent across reboots. For persistence, see the “Configuration Check” section.

3. Configuration Check: Persistent Solutions

These configurations aim to either provide more memory resources or better inform the kernel and Go runtime about memory limits.

  1. Make Swap Persistent: Edit the /etc/fstab file to ensure the swap file is activated on boot.

    sudo vi /etc/fstab

    Add the following line at the end of the file:

    /swapfile none swap sw 0 0

    Save and exit.

  2. Adjust sysctl Parameters: Modify kernel parameters to influence OOM behavior. Create or edit a sysctl configuration file (e.g., /etc/sysctl.d/99-go-oom.conf):

    sudo vi /etc/sysctl.d/99-go-oom.conf

    Add the following lines:

    # Swappiness: controls how aggressively the kernel swaps out anonymous memory.
# A value of 10-60 is often a good starting point. Lower values keep more in RAM.
# Higher values (e.g., 60) can help prevent OOM by freeing up RAM for active processes.
vm.swappiness = 30 # Adjust based on workload; 30-60 for general purpose.

# OOM Score Adjustment for all processes (not recommended for general system,
# better to use systemd unit for specific services)
# kernel.oom_score_adj = -500

    Apply the changes:

    sudo sysctl -p /etc/sysctl.d/99-go-oom.conf

  3. Configure systemd Service Unit for Go Application: This is the most effective way to manage resources for a specific Go service. Edit your application’s systemd service file (e.g., /etc/systemd/system/your-go-app.service):

    sudo vi /etc/systemd/system/your-go-app.service

    Within the [Service] section, consider adding or modifying the following:

    [Service]
# ... other service configurations ...

# Reduces the likelihood of this service being OOM-killed.
# -1000 (least likely to be killed) to 1000 (most likely).
# A value like -500 to -800 is a good balance for critical apps.
OOMScoreAdjust=-700

# Optional: Set a hard memory limit for the cgroup.
# BE CAREFUL: Exceeding this will result in a cgroup-level OOM kill for the service.
# Only use if you have a very clear understanding of your app's max RSS.
# MemoryLimit=2G

# Optional: Limit swap usage for this service.
# MemorySwapMax=512M

# Go Runtime Environment Variables (Crucial for Go 1.19+ for self-tuning)
# GOMEMLIMIT: Tells the Go GC to target a specific percentage/absolute limit of the total system memory,
# or the cgroup memory limit if set. This helps the Go app self-throttle.
# Example: If your MemoryLimit is 2G, set GOMEMLIMIT=1.8G (90%) or GOMEMLIMIT=90% of cgroup.
Environment="GOMEMLIMIT=1.8G" # Or "GOMEMLIMIT=90% of cgroup" for Go 1.20+
Environment="GODEBUG=madvdontneed=1" # Ensures Go proactively returns memory to OS (often default)
# Environment="GOGC=70" # Adjust GC aggressiveness. Lower values mean more frequent GC,
                       # potentially lower memory but higher CPU. Default is 100.

    After modifying the service file:

    sudo systemctl daemon-reload           # Reload systemd configurations
sudo systemctl restart your-go-app.service # Restart your Go application

4. Verification

Confirm your changes have taken effect and monitor your application’s behavior.

  1. Verify Swap Status:

    swapon --show
free -h

    Confirm the swap file is active and free -h shows available swap space.

  2. Verify sysctl Settings:

    sysctl vm.swappiness

    Confirm the vm.swappiness value reflects your configuration.

  3. Verify systemd Service Settings: Check the running service’s OOM score adjustment:

    systemctl show your-go-app.service | grep -i oomscoreadjust
# Also verify the oom_score_adj for the running process:
# Find your Go application's PID
pgrep -f "your-go-app-binary-name"
cat /proc/<PID>/oom_score_adj

    If MemoryLimit was set, verify the cgroup limit:

    # Replace <PID> with your application's process ID
cat /proc/<PID>/cgroup # Find the cgroup path, typically under /system.slice/...
# Example: if cgroup is system.slice/your-go-app.service
cat /sys/fs/cgroup/system.slice/your-go-app.service/memory.max

  4. Monitor Memory Usage:

    • Use htop, top, or glances to observe overall system and process memory usage, paying attention to your Go application’s RSS.
    • For deeper insights, use prometheus with node_exporter (for host metrics) and cAdvisor (for cgroup and container metrics) to track memory usage trends over time.
    • Go-specific Profiling: If OOMs persist, use Go’s built-in pprof tool to analyze memory usage within your application. Integrate it into your application for runtime profiling. 
      import (
    _ "net/http/pprof" // Import this package
    "net/http"
    "log"
)
func main() {
    go func() {
        log.Println(http.ListenAndServe("localhost:6060", nil))
    }()
    // Your application logic
}
      Then, you can access http://localhost:6060/debug/pprof/heap for live heap profiles.

  5. Stress Test: Recreate the conditions (e.g., high load, specific data patterns) that previously led to OOM errors. Monitor closely to ensure the implemented changes prevent the issue.

By following this comprehensive guide, you should be able to effectively troubleshoot and mitigate Go OOM issues on your Ubuntu 22.04 systems, ensuring higher stability and reliability for your WebToolsWiz.com applications.

⚡ Related Troubleshooting Guides