How to Fix Python 504 Gateway Timeout on CentOS 7

Troubleshooting “Python 504 Gateway Timeout” on CentOS 7

As a Senior DevOps Engineer, few HTTP status codes are as frustratingly vague yet critical as the 504 Gateway Timeout. When it’s a Python application on a CentOS 7 server, it often points to specific bottlenecks between your web server/reverse proxy and your Python application server. This guide will walk you through diagnosing and resolving these issues.

1. The Root Cause: Why This Happens on CentOS 7

A 504 Gateway Timeout indicates that an upstream server (the “gateway” or “proxy”) did not receive a timely response from a downstream server that it needed to access to fulfill the request.

In a typical CentOS 7 setup for a Python web application, this usually means:

  1. Your Reverse Proxy (e.g., Nginx, Apache HTTPD) acted as the gateway.
  2. Your Python Application Server (e.g., Gunicorn, uWSGI, or Apache’s mod_wsgi) was the downstream server.

The 504 error occurs when the reverse proxy waits for a response from your Python application server for too long, exceeding its configured timeout limit. The Python application server, in turn, might be failing to respond because:

  • The Python Application Server is down or crashed: The process is not running.
  • The Python Application is stuck or slow: It’s processing a very long-running request, hitting an infinite loop, or waiting for a slow external resource (database, API call).
  • Resource Exhaustion: The server itself (CPU, memory, disk I/O) or the application (too few workers, open file limits) is under stress.
  • Incorrect Configuration: The reverse proxy is pointing to the wrong port/socket, or the application server isn’t listening correctly.
  • Network/Permissions Issues: A firewall (Firewalld) or SELinux policy is blocking communication between the proxy and the application server.

2. Quick Fix (CLI)

Before diving deep into configuration files, let’s perform some immediate checks and actions via the command line.

2.1. Check Service Status & Restart

The most common culprit is the Python application server itself.

# First, identify your Python application's service name.
# Common examples: gunicorn, uwsgi, myapp.service

# Check the status of your Python application service
sudo systemctl status <your_python_app_service_name>

# If it's 'inactive (dead)' or 'failed', try starting/restarting it:
sudo systemctl start <your_python_app_service_name>
# OR if it's 'active (running)' but unresponsive, restart it:
sudo systemctl restart <your_python_app_service_name>

# Check your reverse proxy (Nginx or Apache HTTPD)
# For Nginx:
sudo systemctl status nginx
sudo systemctl restart nginx # Restart to ensure any configuration changes are picked up

# For Apache:
sudo systemctl status httpd
sudo systemctl restart httpd # Restart to ensure any configuration changes are picked up

2.2. Review Application Logs

After restarting, immediately check the logs for errors.

# For Python application service logs:
sudo journalctl -u <your_python_app_service_name> --since "5 minutes ago" -f

# For Nginx error logs:
sudo tail -f /var/log/nginx/error.log

# For Apache error logs:
sudo tail -f /var/log/httpd/error_log

Look for:

  • Any Python tracebacks in your application logs.
  • upstream timed out or connection refused errors in Nginx/Apache logs.
  • Permission denied errors, which might point to SELinux.

3. Configuration Check

This section details the critical configuration files to inspect and modify.

3.1. Reverse Proxy Configuration (Nginx or Apache)

These settings define how long your proxy will wait for a response.

a) Nginx

Edit your Nginx server block configuration (e.g., /etc/nginx/conf.d/your_app.conf or /etc/nginx/nginx.conf).

server {
    listen 80;
    server_name yourdomain.com;

    location / {
        # --- CRITICAL TIMEOUT SETTINGS ---
        proxy_connect_timeout 60s; # How long to wait for a connection to the upstream server
        proxy_send_timeout 60s;    # How long to wait for data to be sent to the upstream server
        proxy_read_timeout 300s;   # How long to wait for a response from the upstream server (THIS IS KEY for 504)
        # ---------------------------------

        proxy_pass http://unix:/run/gunicorn.sock; # Or http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Action: Increase proxy_read_timeout to 300s (5 minutes) or even higher if you have genuinely long-running requests. You might also need to adjust proxy_connect_timeout and proxy_send_timeout.

After editing, test the Nginx configuration and reload:

sudo nginx -t
sudo systemctl reload nginx

b) Apache HTTPD (mod_proxy)

Edit your Apache virtual host configuration (e.g., /etc/httpd/conf.d/your_app.conf).

<VirtualHost *:80>
    ServerName yourdomain.com

    # --- CRITICAL TIMEOUT SETTING ---
    ProxyTimeout 300 # How long to wait for a response from the backend (THIS IS KEY for 504)
    # --------------------------------

    ProxyPass / http://127.0.0.1:8000/ # Or use mod_wsgi
    ProxyPassReverse / http://127.0.0.1:8000/

    # For mod_wsgi setups, ensure your WSGIDaemonProcess has appropriate settings
    # For example:
    # WSGIDaemonProcess your_app python-home=/path/to/venv processes=5 threads=15 maximum-requests=1000 display-name=%{GROUP}
    # WSGIProcessGroup your_app
    # WSGIScriptAlias / /path/to/your_app.wsgi
</VirtualHost>

Action: Increase ProxyTimeout to 300 seconds or higher.

After editing, test the Apache configuration and reload:

sudo apachectl configtest
sudo systemctl reload httpd

3.2. Python Application Server Configuration (Gunicorn/uWSGI)

Your Python application server also has its own timeout settings. If your Nginx/Apache timeout is higher than your Gunicorn/uWSGI timeout, the application server might be timing out internally before the proxy does, leading to an unresponsive state.

a) Gunicorn

If you’re running Gunicorn via a systemd service, edit the .service file (e.g., /etc/systemd/system/gunicorn.service).

[Unit]
Description=Gunicorn instance to serve my_app
After=network.target

[Service]
User=your_user
Group=your_group
WorkingDirectory=/path/to/your_app
ExecStart=/path/to/venv/bin/gunicorn \
          --access-logfile - \
          --error-logfile - \
          --workers 3 \
          --timeout 120 \ # Increase this timeout (default is 30s)
          --bind unix:/run/gunicorn.sock \ # Or 127.0.0.1:8000
          my_app.wsgi:application
RestartSec=10
Restart=on-failure

[Install]
WantedBy=multi-user.target

Action: Increase the --timeout parameter. It’s generally good practice for the application server’s timeout to be less than the reverse proxy’s timeout, so the application can gracefully fail or log an error before the proxy issues a 504. For example, if Nginx is 300s, Gunicorn might be 120s.

After editing, reload systemd daemon and restart the service:

sudo systemctl daemon-reload
sudo systemctl restart <your_python_app_service_name>

b) uWSGI

If using a uWSGI .ini file (e.g., /etc/uwsgi/sites/your_app.ini).

[uwsgi]
# ... other settings ...
socket = /run/uwsgi.sock
chmod-socket = 666
# ... other settings ...

harakiri = 60 # Kill workers that take longer than this (seconds)
# Other relevant timeouts:
http-timeout = 60
# ...

Action: Adjust harakiri (worker timeout) and http-timeout as needed.

After editing, restart your uWSGI service (often uwsgi@your_app.service):

sudo systemctl restart <your_uwsgi_app_service_name>

3.3. System-Level Checks (Firewall, SELinux, Resources)

  • Firewalld: Ensure the port your Python app server is listening on (if using TCP, e.g., 8000) is open, or that communication to its Unix socket is not blocked.

    sudo firewall-cmd --list-all
# If using a TCP port and it's not open, add it (example for port 8000):
# sudo firewall-cmd --zone=public --add-port=8000/tcp --permanent
# sudo firewall-cmd --reload

    Typically, if Nginx/Apache and Gunicorn/uWSGI are on the same server communicating via 127.0.0.1 or a Unix socket, Firewalld usually isn’t the issue.

  • SELinux: SELinux can prevent Nginx/Apache from connecting to Unix sockets or network ports, even if they’re otherwise open. Check /var/log/audit/audit.log for AVC denied messages.

    sudo ausearch -c httpd -ts recent # For Apache
sudo ausearch -c nginx -ts recent # For Nginx
sudo ausearch -c gunicorn -ts recent # For Gunicorn (or your app service)
# Alternatively, view all denials:
sudo sealert -a /var/log/audit/audit.log

    Temporary Test: If you suspect SELinux, try disabling it temporarily (NOT for production):

    sudo setenforce 0
# Test your application. If it works, SELinux is the cause.
sudo setenforce 1 # Re-enable SELinux immediately

    If SELinux is the issue, you’ll need to create a custom policy module or set appropriate booleans. Common booleans for web apps: httpd_can_network_connect, httpd_unified, httpd_read_user_content.

  • System Resources: If your application is genuinely taking a long time, monitor CPU, memory, and disk I/O.

    top # or htop
free -h
df -h

    If memory is consistently low, dmesg | grep -i oom might show the Out-Of-Memory (OOM) killer terminating your application.

4. Verification

After applying any changes, it’s crucial to verify the fix.

  1. Restart Services: Ensure all relevant services (Python app server, Nginx/Apache) are restarted after configuration changes. 
    sudo systemctl daemon-reload # If systemd unit files changed
sudo systemctl restart <your_python_app_service_name>
sudo systemctl restart nginx # or httpd
  2. Access the Application: Open your web browser and try to access the problematic URL.
  3. Use curl for Diagnostics: You can test from the server itself to rule out external network issues. 
    # If your app server listens on a TCP port (e.g., 8000)
curl -v http://127.0.0.1:8000/your-endpoint

# If your app server uses a Unix socket (e.g., for Gunicorn/uWSGI)
# This requires a tool that can interact with Unix sockets, like socat or directly via Python.
# A more practical test is to ensure Nginx/Apache can reach it after restart.
  4. Monitor Logs Again: Keep an eye on the logs for any new errors or indications that the request completed successfully. 
    sudo journalctl -u <your_python_app_service_name> -f
sudo tail -f /var/log/nginx/access.log /var/log/nginx/error.log
# or
sudo tail -f /var/log/httpd/access_log /var/log/httpd/error_log
    Look for HTTP 200 (OK) status codes in the access logs after your requests.

By systematically working through these steps, you should be able to identify and resolve the root cause of your Python 504 Gateway Timeout on CentOS 7. Remember that debugging often involves trial and error, so make one change at a time and re-test.

⚡ Related Troubleshooting Guides