Nginx High Concurrency Configuration & Performance Optimization in Practice

网络协议

Nginx High Concurrency: From Architecture to Practice

Nginx's event-driven, non-blocking I/O architecture enables a single machine to handle tens of thousands of concurrent connections. However, the default configuration barely scratches the surface — deep tuning is required for production workloads.

Optimization Area Key Parameters Impact
Process Model worker_processes, worker_connections Concurrency capacity
Event Model use epoll, multi_accept Event processing efficiency
Load Balancing upstream, proxy_next_upstream Traffic distribution & fault tolerance
Buffer & Timeout proxy_buffers, keepalive_timeout Memory & connection reuse
Compression gzip, gzip_comp_level Transfer bandwidth
Security limit_req, limit_conn Traffic protection

1. Nginx Architecture: Event-Driven & Worker Processes

Master-Worker Model

Nginx uses a Master-Worker multi-process architecture:

  • Master process: Reads config, binds ports, manages Workers (fork/signals/reload)
  • Worker processes: Handle requests independently — one Worker crash doesn't affect others
# View Nginx process structure
# ps aux | grep nginx
# root      1234  nginx: master process
# www-data  1235  nginx: worker process
# www-data  1236  nginx: worker process

Event-Driven Model

Nginx is built on the Reactor pattern, using I/O multiplexing:

I/O Model Platform Performance
epoll Linux Best, O(1) event notification
kqueue FreeBSD/macOS Excellent, similar to epoll
select Universal Poor, fd_set limit of 1024
poll Universal Fair, no limit but linear scan 
events {
    use epoll;
    worker_connections 65535;
    multi_accept on;
    accept_mutex off;
}

multi_accept on lets a Worker accept all new connections at once, reducing system calls. accept_mutex off avoids the thundering herd problem under high concurrency (Linux 3.9+ supports SO_REUSEPORT).

2. Core Concurrency Configuration

worker_processes

# Auto-detect CPU cores (recommended)
worker_processes auto;

# Or specify manually (usually equal to or 2x CPU cores)
worker_processes 8;

Tuning tips:

  • Generally set to auto, matching CPU core count
  • CPU-intensive workloads (SSL, gzip): match core count
  • I/O-intensive workloads: consider 2x core count

worker_connections

events {
    worker_connections 65535;
}

Max concurrent connections = worker_processes × worker_connections. You must also raise the system file descriptor limit:

# Temporary change
ulimit -n 65535

# Permanent change (/etc/security/limits.conf)
* soft nofile 65535
* hard nofile 65535

worker_rlimit_nofile

# Max file descriptors per Worker process
worker_rlimit_nofile 65535;

Complete Base Configuration

user www-data;
worker_processes auto;
worker_rlimit_nofile 65535;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;

events {
    use epoll;
    worker_connections 65535;
    multi_accept on;
    accept_mutex off;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile      on;
    tcp_nopush    on;
    tcp_nodelay   on;
    keepalive_timeout 65;
    keepalive_requests 100;
}

3. Load Balancing Algorithms

Basic Configuration

upstream backend {
    server 10.0.0.1:8080 weight=5;
    server 10.0.0.2:8080 weight=3;
    server 10.0.0.3:8080 weight=2;
    server 10.0.0.4:8080 backup;
}

Five Scheduling Algorithms Compared

Algorithm Directive Characteristics Use Case
Round Robin (default) Sequential, even distribution Identical backend performance
Weighted RR weight=N Proportional by weight Mixed backend performance
Least Connections least_conn Prefer fewest active connections Long-lived/varied request duration
IP Hash ip_hash Same IP → same backend Session persistence
Random random Random selection Stateless services
Consistent Hash hash $key consistent Consistent hash ring Cache servers

least_conn Configuration

upstream api_backend {
    least_conn;
    server 10.0.0.1:8080;
    server 10.0.0.2:8080;
    server 10.0.0.3:8080;
}

ip_hash Configuration

upstream web_backend {
    ip_hash;
    server 10.0.0.1:8080;
    server 10.0.0.2:8080;
    server 10.0.0.3:8080;
}

Note: ip_hash does not support backup and weight. Use consistent hashing in production instead.

Consistent Hash Configuration

upstream cache_backend {
    hash $request_uri consistent;
    server 10.0.0.1:8080;
    server 10.0.0.2:8080;
    server 10.0.0.3:8080;
}

4. Upstream Health Checks

Passive Health Checks (Built-in)

upstream backend {
    server 10.0.0.1:8080 max_fails=3 fail_timeout=30s;
    server 10.0.0.2:8080 max_fails=3 fail_timeout=30s;
    server 10.0.0.3:8080 max_fails=3 fail_timeout=30s;
}
Parameter Meaning Default
max_fails Failure threshold before marking unavailable 1
fail_timeout Duration marked unavailable 10s

Active Health Checks (nginx-upstream-check-module)

upstream backend {
    server 10.0.0.1:8080;
    server 10.0.0.2:8080;

    check interval=3000 rise=2 fall=3 timeout=1000 type=http;
    check_http_send "HEAD /health HTTP/1.0\r\n\r\n";
    check_http_expect_alive http_2xx http_3xx;
}

proxy_next_upstream Fault Tolerance

proxy_next_upstream error timeout http_502 http_503 http_504;
proxy_next_upstream_timeout 10s;
proxy_next_upstream_tries 3;

5. Buffer & Timeout Tuning

Proxy Buffers

proxy_buffering on;
proxy_buffer_size 4k;
proxy_buffers 8 16k;
proxy_busy_buffers_size 32k;
Parameter Meaning Recommended
proxy_buffer_size Response header buffer size 4k-8k
proxy_buffers Response body buffer count & size 8 16k
proxy_busy_buffers_size Busy buffer size Half of proxy_buffers total

Timeout Configuration

proxy_connect_timeout 5s;
proxy_send_timeout 10s;
proxy_read_timeout 30s;

client_body_timeout 12s;
client_header_timeout 12s;
send_timeout 10s;

keepalive_timeout 65s;
keepalive_requests 100;

Upstream Keepalive Connection Pool

upstream backend {
    server 10.0.0.1:8080;
    server 10.0.0.2:8080;
    keepalive 32;
    keepalive_requests 100;
    keepalive_timeout 60s;
}

server {
    location / {
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_pass http://backend;
    }
}

keepalive 32 maintains 32 idle long connections per Worker to backends, avoiding frequent TCP handshakes.

6. Gzip Compression Optimization

gzip on;
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_min_length 1024;
gzip_http_version 1.1;
gzip_types
    text/plain
    text/css
    text/xml
    text/javascript
    application/json
    application/javascript
    application/xml
    application/xml+rss
    application/vnd.ms-fontobject
    application/x-font-ttf
    font/opentype
    image/svg+xml;
gzip_buffers 16 8k;
gzip_disable "msie6";
Parameter Description Recommended
gzip_comp_level Compression level 1-9 4-6 (balance CPU & ratio)
gzip_min_length Minimum size to compress 1024 (small files may grow)
gzip_types MIME types to compress Add as needed, exclude images
gzip_buffers Compression buffers 16 8k

7. Static File Serving & Caching

Static File Optimization

server {
    listen 80;
    server_name static.example.com;
    root /var/www/static;

    location / {
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;

        open_file_cache max=10000 inactive=20s;
        open_file_cache_valid 30s;
        open_file_cache_min_uses 2;
        open_file_cache_errors on;

        expires 30d;
        add_header Cache-Control "public, immutable";
    }

    location ~* \.(js|css)$ {
        expires 7d;
        add_header Cache-Control "public, immutable";
    }

    location ~* \.(jpg|jpeg|png|gif|ico|svg|webp)$ {
        expires 365d;
        add_header Cache-Control "public, immutable";
    }
}

open_file_cache Explained

Parameter Meaning
max=N Max cached file descriptors
inactive=T Cache entry lifetime
valid=T Interval to check cache validity
min_uses=N Minimum accesses within inactive period
errors Whether to cache file lookup errors

8. SSL/TLS Performance Optimization

Session Cache

ssl_session_cache shared:SSL:10m;
ssl_session_timeout 1d;
ssl_session_tickets on;

shared:SSL:10m — all Workers share 10MB of session cache, holding ~40,000 sessions.

OCSP Stapling

ssl_stapling on;
ssl_stapling_verify on;
resolver 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 5s;

Complete SSL Configuration

server {
    listen 443 ssl http2;
    server_name example.com;

    ssl_certificate     /etc/nginx/ssl/example.com.pem;
    ssl_certificate_key /etc/nginx/ssl/example.com.key;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers on;

    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 1d;
    ssl_session_tickets on;

    ssl_stapling on;
    ssl_stapling_verify on;
    resolver 8.8.8.8 8.8.4.4 valid=300s;
    resolver_timeout 5s;

    ssl_buffer_size 4k;
}

9. Rate Limiting & Protection

Request Rate Limiting (limit_req)

limit_req_zone $binary_remote_addr zone=api_limit:10m rate=100r/s;

server {
    location /api/ {
        limit_req zone=api_limit burst=200 nodelay;
        proxy_pass http://backend;
    }
}
Parameter Meaning
rate=100r/s 100 requests per second per IP
burst=200 Allow burst of 200 requests
nodelay Burst requests are not delayed; excess is rejected immediately

Concurrent Connection Limiting (limit_conn)

limit_conn_zone $binary_remote_addr zone=conn_limit:10m;

server {
    location /api/ {
        limit_conn conn_limit 50;
        proxy_pass http://backend;
    }
}

Bandwidth Limiting

location /download/ {
    limit_rate 500k;
    limit_rate_after 10m;
}

No speed limit for the first 10MB, then capped at 500KB/s.

10. Reverse Proxy Configuration

server {
    listen 80;
    server_name api.example.com;

    location / {
        proxy_pass http://backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_redirect off;
        proxy_buffering on;
        proxy_buffer_size 4k;
        proxy_buffers 8 16k;
    }

    location /ws/ {
        proxy_pass http://backend;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_read_timeout 3600s;
    }
}

11. Common Error Troubleshooting

502 Bad Gateway

Cause: Nginx cannot connect to the backend service.

# Check if backend is running
curl -I http://10.0.0.1:8080/health

# Check Nginx error log
tail -f /var/log/nginx/error.log | grep 502

# Common causes:
# 1. Backend service not running or crashed
# 2. Backend port misconfigured
# 3. Backend overloaded, refusing connections
# 4. SELinux/firewall blocking

504 Gateway Timeout

Cause: Backend service response timeout.

# Increase timeouts
proxy_connect_timeout 10s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;

# Troubleshooting steps
# 1. Check backend response time
curl -o /dev/null -s -w "time_total: %{time_total}\n" http://backend/api
# 2. Check backend logs for slow queries
# 3. Check network latency
ping -c 5 10.0.0.1

429 Too Many Requests

Cause: Rate limit rule triggered.

# Custom rate limit response
limit_req_status 429;

12. Security Hardening

Security Response Headers

add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
add_header Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline'" always;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

DDoS Mitigation

limit_req_zone $binary_remote_addr zone=ddos:10m rate=30r/s;
limit_conn_zone $binary_remote_addr zone=ddos_conn:10m;

server {
    limit_req zone=ddos burst=50 nodelay;
    limit_conn ddos_conn 30;

    client_body_buffer_size 16k;
    client_max_body_size 1m;
    client_header_buffer_size 1k;
    large_client_header_buffers 4 8k;
}

Block Sensitive Paths

location ~ /\.(git|svn|env) {
    deny all;
    return 404;
}

location ~* /(wp-admin|phpmyadmin|adminer) {
    deny all;
    return 404;
}

13. Performance Benchmarking

wrk Benchmark

# Install wrk
git clone https://github.com/wg/wrk.git
cd wrk && make && sudo cp wrk /usr/local/bin/

# Benchmark (12 threads, 400 connections, 30 seconds)
wrk -t12 -c400 -d30s http://example.com/

# With latency distribution
wrk -t12 -c400 -d30s --latency http://example.com/

# POST request test
wrk -t4 -c200 -d10s -s post.lua http://example.com/api

ab Benchmark

# Install Apache Bench
# Ubuntu: apt install apache2-utils

# Benchmark (200 concurrency, 10000 requests)
ab -n 10000 -c 200 http://example.com/

# With Keep-Alive
ab -n 10000 -c 200 -k http://example.com/

Benchmark Result Analysis

Metric Meaning Reference
Requests/sec Requests processed per second Static assets > 10000
Latency (p99) 99th percentile latency < 100ms
Transfer/sec Data transferred per second Depends on workload
Socket errors Connection errors Should be 0

14. FAQ

Q: After setting worker_processes to auto, the Worker count doesn't match CPU cores?

A: auto reads available cores from /proc/cpuinfo. In containers with cgroup CPU limits, only Nginx 1.19+ detects cgroup constraints. Older versions need manual configuration.

Q: After setting worker_connections to 65535, actual concurrency still can't scale?

A: You must also adjust system limits: ulimit -n, net.core.somaxconn, net.ipv4.tcp_max_syn_backlog, fs.file-max.

Q: How to choose between ip_hash and consistent hashing?

A: ip_hash is simple but doesn't support weight/backup, and scaling causes mass session migration. Consistent hashing only affects adjacent nodes during scaling — recommended for cache layers.

Q: What should gzip_comp_level be set to?

A: 4-6 is the sweet spot. Levels 1-3 have low compression; 7-9 consume much more CPU for marginal gain. In practice, level 6 vs level 9 differs by only 2-3% in ratio but 50%+ more CPU.

Q: How to monitor Nginx real-time status?

A: Enable the stub_status module, then use Prometheus + Grafana or ngxtop for real-time monitoring.

location /nginx_status {
    stub_status on;
    access_log off;
    allow 127.0.0.1;
    deny all;
}

Q: How to gracefully reload Nginx without dropping requests?

A: Use nginx -s reload. The Master reloads config, starts new Workers, and old Workers exit after finishing current requests.

Recommended tools: Use /encode/base64 to encode config files, /encode/hash to generate config hash checksums, and /json/format to format Nginx JSON logs.

Try these browser-local tools — no sign-up required →

#Nginx#高并发#性能优化#负载均衡#教程