A HAProxy 503 error means the load balancer has no available backend server to handle the request. The fix depends on why HAProxy considers all backends unavailable — failed health checks, connection refusals, queue timeouts, ACL mismatches, or servers placed in maintenance state are all common causes. This guide walks through each cause in order of frequency and shows you how to resolve it.
Applies to Ubuntu 24.04 LTS (Noble Numbat) and 26.04 LTS (Resolute Raccoon).
What the Error Looks Like
A client hitting a 503 from HAProxy sees something like:
503 Service Unavailable
No server is available to handle this request.The HAProxy log at /var/log/haproxy.log will record an entry with a termination state code. Reading that code is the fastest path to the root cause.
Step 1: Read the HAProxy Log
sudo tail -100 /var/log/haproxy.log | grep " 503 "Each log line contains a termination state field — a two-character code that describes why the session ended. The most relevant codes for a 503 are:
| Code | Meaning |
|---|---|
SC | Server refused or reset the TCP connection (RST or ICMP received) |
sC | timeout connect expired before the connection completed |
sQ | Request waited in the queue past timeout queue and was dropped |
SD | Backend was detected as down; HAProxy killed the connection |
-- | Normal termination — look elsewhere if 503 still appears |
SC indicates the server or equipment between it and HAProxy explicitly refused the TCP connection. sQ indicates the session spent too much time in queue and was expired.
If the log shows <NOSRV> as the server name, HAProxy never selected a backend at all — the request matched no routing rule.
Step 2: Check the Stats Socket
The runtime API gives you an instant view of server and backend states without reloading anything:
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
echo "show stat" | sudo socat stdio /run/haproxy/admin.sock | \
cut -d',' -f1,2,18,19 | column -s',' -tLook at the status column. Servers marked DOWN are not receiving traffic. Servers marked MAINT or DRAIN are excluded from load balancing by admin action.
Note: If the socket path differs on your system, check
stats socketin/etc/haproxy/haproxy.cfgfor the configured path.
Cause 1: All Backend Servers Are Down (Most Common)
When HAProxy has no viable destination for incoming requests it returns a 503 immediately. This can happen due to a mass deployment gone wrong, a network partition between the HAProxy host and the backend tier, or an upstream database failure that causes all application servers to crash within the same health check interval.
Diagnose it:
# Verify the backend process is actually running on each server
systemctl status your-app.service
# Test connectivity from the HAProxy host to a backend
curl -v http://192.168.1.10:8080/healthReplace 192.168.1.10:8080 with your backend’s address and port as configured in /etc/haproxy/haproxy.cfg.
Fix it: Restart the failing application on the backend server, then watch the HAProxy log for the backend to come back up:
sudo journalctl -u haproxy -fHAProxy will automatically mark the server UP once it passes rise consecutive health checks (default: 2).
Cause 2: Health Check Misconfiguration
HAProxy may mark a healthy server as DOWN because the health check itself is wrong — checking the wrong port, using an HTTP check on a TCP-only server, or expecting a status code the server never returns.
Check your backend block:
sudo grep -A 20 "^backend" /etc/haproxy/haproxy.cfgA correctly configured HTTP health check looks like this:
backend web_servers
balance roundrobin
option httpchk GET /health HTTP/1.1
http-check expect status 200
server web1 192.168.1.10:8080 check inter 2s fall 3 rise 2
server web2 192.168.1.11:8080 check inter 2s fall 3 rise 2Key parameters:
check— enables health checks on this server line; without it, HAProxy assumes the server is always UPinter 2s— check interval (2 seconds)fall 3— mark DOWN after 3 consecutive failuresrise 2— mark UP after 2 consecutive successeshttp-check expect status 200— only accept HTTP 200 as a passing check
Add option log-health-checks to your backend block to log every check result, which makes failures visible immediately:
backend web_servers
option log-health-checks
...After editing, validate and reload — never apply an untested config to a live proxy:
sudo haproxy -c -f /etc/haproxy/haproxy.cfg && sudo systemctl reload haproxyCause 3: Connection Refused or Timeout (SC / sC)
The SC code means the server explicitly refused the connection with a TCP RST or ICMP message. The sC code means the connection to the server did not complete within the timeout connect value.
Diagnose SC: The backend application is not listening on the expected port. Verify on the backend host:
ss -tlnp | grep 8080If nothing is listening, the application is down or bound to a different port.
Fix sC (timeout connect): The default timeout connect is often too short for slow or remote backends. Raise it in your defaults or backend block:
defaults
timeout connect 5s
timeout client 30s
timeout server 30sValidate and reload after any change.
Cause 4: Queue Saturation (sQ)
When sQ appears in logs alongside a nonzero qcur value, requests waited in the queue past timeout queue and were expired with a 503. This happens when all backend servers are at their maxconn limit and new requests cannot be accepted fast enough.
Diagnose it:
echo "show stat" | sudo socat stdio /run/haproxy/admin.sock | \
cut -d',' -f1,2,46,47 | column -s',' -tThe qcur and qmax columns show current and peak queue depth per server.
Fix it — Option A: Raise maxconn on the server lines if the backend can handle more concurrent connections:
backend web_servers
server web1 192.168.1.10:8080 check maxconn 200Fix it — Option B: Raise timeout queue to allow more waiting time before dropping requests:
defaults
timeout queue 10sThe right fix depends on whether the backend is genuinely overloaded (add capacity or more servers) or the queue timeout is simply too aggressive.
Cause 5: ACL Routing Gap (<NOSRV>)
An ACL routing misconfiguration where a request matches no rule and no default_backend is configured causes HAProxy to return 503 when it cannot determine the routing target.
Diagnose it: Look for <NOSRV> in the log line next to a 503. Then review your frontend:
sudo grep -A 30 "^frontend" /etc/haproxy/haproxy.cfgFix it: Ensure every frontend has a default_backend and that ACL use_backend rules cover all expected request patterns:
frontend http_front
bind *:80
acl is_api path_beg /api
use_backend api_servers if is_api
default_backend web_serversWithout default_backend, any request that does not match an ACL gets a 503.
Cause 6: Servers in MAINT or DRAIN State
If all servers in a backend are simultaneously in DRAIN or MAINT state, HAProxy returns 503 for every new request even though the backend servers are alive and healthy.
This often happens after manual intervention via the stats socket during maintenance that was never reversed.
Check and fix it:
# List server states
echo "show servers state" | sudo socat stdio /run/haproxy/admin.sock
# Bring a server back to READY state
echo "set server web_servers/web1 state ready" | sudo socat stdio /run/haproxy/admin.sockConfirm the Fix
After applying any change, confirm the backend is healthy:
# Watch for backend UP messages
sudo tail -f /var/log/haproxy.log | grep -E "(UP|DOWN|503)"
# Check current backend status via stats socket
echo "show stat" | sudo socat stdio /run/haproxy/admin.sock | \
cut -d',' -f1,2,18 | column -s',' -tServers should show UP and 503 responses should stop appearing in the log within a few health check intervals.
Still Getting 503s?
If none of the above causes apply, check these additional possibilities:
- Firewall rules: Confirm the HAProxy host can reach each backend port. Test with
nc -zv 192.168.1.10 8080. - SSL mismatch: If backends expect TLS and HAProxy sends plain TCP (or vice versa), connections fail silently. Check whether your server lines include
ssland whether backends have valid certificates. - Backend file descriptor limits: Under heavy load, the backend process may exhaust its open file descriptors before HAProxy’s
maxconntriggers. Check withcat /proc/$(pidof your-app)/limits | grep "open files"on the backend. - Watch
wretrandwrediscounters in the stats output — rising retries mean HAProxy is compensating for instability that will produce 503s when compensation runs out.
FAQ
Why does HAProxy show a server as UP but I still get 503 errors?
The stats socket shows the backend health check result, but the actual request path may differ. Check for ACL mismatches (<NOSRV> in logs), maxconn saturation on a specific server, or a DRAIN state that was set after the last stats refresh.
How do I test a config change without dropping live traffic?
Always run sudo haproxy -c -f /etc/haproxy/haproxy.cfg before reloading. Then use sudo systemctl reload haproxy, which performs a graceful reload — existing connections complete normally while new connections use the updated config.
Where is the HAProxy log on Ubuntu?
The config requires log /dev/log local0 in the global section and log global with option httplog in defaults; the Debian and Ubuntu packages ship an rsyslog rule that writes traffic to /var/log/haproxy.log.
For the full configuration reference, see the HAProxy Configuration Manual and the MDN documentation on HTTP 503.
Once the backend is healthy and traffic is flowing normally, consider adding option log-health-checks permanently so future failures appear in the log immediately rather than requiring active investigation.
