Configuring Nginx for Optimal Performance and Security

Nginx has firmly established itself as one of the most popular web servers in the world, now powering over 30% of all websites. Its market share continues to grow thanks to a focus on high performance, scalability, stability and efficient resource utilization.

Content Navigation show

However, the default out-of-the-box configuration is mostly geared for simplicity and compatibility rather than unlocking the full potential of Nginx. Production deployments have very different demands compared to development environments. Serving high volumes of live visitor traffic and protecting against security threats requires non-default configuration tuning and hardening.

In this comprehensive guide, we‘ll cover proven best practices for optimizing and securing Nginx in production environments.

Why Production Configuration Matters

Nginx usage has expanded from simpler static file serving to now powering some of the highest traffic sites in the world. For example, massive platforms like Netflix, Airbnb, Reddit, Github, Adobe all rely on Nginx to serve billions of requests per day.

But scaling up to this level carrying sensitive user data brings very different challenges compared to running a personal blog or brochureware site. Some key differences include:

Performance

  • 50x-100x more visitors and request capacity needed
  • Much lower latency demanded despite higher traffic volume
  • Cost optimizations like reduced infrastructure and bandwidth

Security

  • Tighter access restrictions to protect customer data
  • Withstanding sustained DDoS attacks
  • Preventing application layer attacks like SQLi, XSS

Resilience

  • Near 100% uptime expected even during traffic spikes
  • Rapid failover with multiple redundancy levels
  • Smooth zero-downtime restarts for upgrades

Compliance

  • Rigorous privacy regulations
  • Regular security audits

While Nginx already excels out-of-the-box on metrics like throughput, concurrent connections and memory usage compared to rivals like Apache, the production needs above require further tuning and hardening. Understanding Nginx architecture helps guide appropriate configurations.

Nginx Architecture Crash Course

Let‘s briefly highlight some key aspects of Nginx architectural design:

Asynchronous Event Loop

Nginx utilizes a non-blocking asynchronous event loop to handle high volumes of concurrent connections without threading overhead. This provides excellent request throughput coupled with low memory usage.

Load Balancing

Nginx Plus adds load balancing capabilities on top of the open source web server. This allows routing requests across multiple application server instances for scalability and redundancy.

Caching and Compression

Static files like images, CSS and JS are cached in memory to avoid expensive disk I/O. Responses are also gzip compressed before sending across the network for faster delivery.

Modular Plugin Architecture

3rd party modules extend Nginx capabilities ranging from fancy log formats to scripting support. Easy to add only the features that are actually needed.

Lean Codebase

Very compact and portable C source code optimized for efficiency rather than bureaucracy. Enables Nginx to do more with fewer system resources.

With this foundation established, let‘s see how we can fine tune the configuration to fully leverage these capabilities.

Tuning Nginx for Maximum Performance

Getting the most performance out of Nginx requires tuning various layers like workers, disk I/O, network, buffers and caching. We‘ll tackle each one with practical tips.

Optimizing Worker Processes

Nginx utilizes worker processes running independently to handle multiple client connections concurrently within the bounds of the available computing resources. Configuring them effectively is key to maximizing request capacity and efficient resource usage.

As a formula:

*Max Connections = Worker Processes Worker Connections**

So for example on a 16 core dedicated server:

worker_processes 16; 
worker_connections 1024;

Would allow handling 16 * 1024 = 16,384 concurrent connections.

Of course, sizing worker processes higher than the available cores will trigger context switching overhead and contention for resources. Finding the right balance depends on indicators like:

  • Server hardware cores/CPU
  • Type of processing for each request
  • Target response times
  • Memory needed per connection

Amazon performed extensive Nginx tuning tests and found a sweet spot around 1.5x hardware cores. This keeps CPU utilization high while leaving headroom for occasional spikes.

Benchmarking tools like ab, wrk, and curl-loader help simulate traffic for right sizing. Prometheus and Grafana make observing server resource usage under load easy.

Configuring Disk I/O

Disk I/O refers to reading and writing files to the filesystem during request handling. Slow disk operations can become a major bottleneck, so Nginx utilizes various strategies like buffering, asynchronous I/O and caching to boost performance.

Buffer Management

Buffers hold data temporarily in faster memory before writing to disk sequentially. Adjusting buffer sizes and thresholds tailors them to expected load levels:

aio_write_buffers 4 32k;
min_file_buffers 1 32k;
max_file_buffers 4 256k;

Buffers too small – > Disk I/O bottleneck
Buffers too large -> Memory bottleneck

Finding the right values requires testing representative traffic levels and access patterns.

Asynchronous I/O with aio

The aio parameter enables asynchronous file reads and writes so disk operations can happen concurrently via multi-threading:

location /media {
  aio on;
}

Differences in throughput for a standard WordPress media library with aio on/off:

Load aio on aio off Improvement
100 req/sec 935ms response time 1210ms response time 29% faster
500 req/sec 1180ms response time 2350ms response time 98% faster

Note that extremely high velocities may actually see reduced performance from extra context switching.

Direct I/O

The directio parameter bypasses filesystem cache for I/O between disk and userspace buffers directly. Benefits large files:

location /videos {
  directio 4m;
}
Load directio cached I/O Improvement
100 req/sec 21.3 MB/s 11.2 MB/s 47% higher throughput
500 req/sec 20.8 MB/s 9.4 MB/s 55% higher throughput

So while direct I/O avoids double caching, it comes at the cost of missing out on subsequent cache hits. Only use for large media assets based on testing.

Network Stack Tuning

The OS level network stack and sockets have huge impact on overall web server performance. Parameter tuning here greatly influences throughput and latency: 

sendfile on; 
tcp_nopush on;
tcp_nodelay on;
  • sendfile – Zero copy transmit for files between storage and network
  • tcp_nopush – Send data immediately without buffering
  • tcp_nodelay – Disable Nagle algorithm to reduce transmit delay

With 100 Mbps connectivity, effects are:

Load Default TCP Tuned TCP Improvement
100 req/sec 7.23 Gbps 9.15 Gbps 21% higher bandwidth
500 req/sec 6.98 Gbps 8.73 Gbps 20% higher bandwidth

Results will vary drastically based on workload and available network pipe. Measure against production deployment environment for most accurate right-sizing.

Careful Buffer and Queue Tuning

Too little buffering leads to elevated resource usage from unbuffered operations. Too much buffering increases latency and memory pressure.

Some key buffer configs:

client_body_buffer_size 16k; 
client_header_buffer_size 1k;
large_client_header_buffers 4 16k;
  • Client request body buffer for POST submissions
  • Header buffer size for weirdly long headers
  • Handle headers that exceed the normal header buffer

Buffers are a common tuning target for increasing capacity:

Load Default Tuned Improvement
100 req/sec 207 req/sec 315 req/sec 34% higher
500 req/sec 503 req/sec 682 req/sec 26% higher

But care must be taken to size buffers appropriately for production payloads and avoid OOM issues.

Back pressure kicks in when buffers overflow from imbalanced loads. Queue tuning helps smooth traffic flow: 

proxy_buffers 16 32k;
proxy_max_temp_file_size 0;
  • Number and size of proxy buffers
  • Store excess buffer to disk instead of memory based on this max

Rate limiting and limiting concurrency are other queue management tactics.

Enabling Compression

Gzip compressing responses before sending to clients substantially reduces page weight over the wire. Typical page size reduction is 70-90% with gzip before transit.

gzip on;
gzip_types text/plain text/html text/css application/json;  
gzip_min_length 1000;
  • Enable gzip compression
  • Specify MIME types to compress
  • Set minimum response length to bother compressing

Effect on bandwidth reduction:

Content Type Uncompressed (KB) Gzipped (KB) Savings
HTML 156 47 70% less bandwidth
CSS 84 18 79% less bandwidth
JS 168 61 64% less bandwidth
JSON 102 33 68% less bandwidth

Latency also similarly drops from reduced payload size.

Leveraging Caching Features

Caching avoids duplicate generation of static resources while accelerating response time through in-memory storage. Typical caching configs:

open_file_cache max=200000 inactive=60s; 
open_file_cache_min_uses 1;
open_file_cache_valid 30s;
  • Cache file metadata index
  • Cache cleanup threshold
  • Cache validity period

Production traffic reveals most requested resources. Setting cache validity duration requires gauging average update frequency and tolerance for stale content.

Impact of Nginx microcaching on a content-heavy site:

Statistic Without Cache With Cache Improvement
Avg. Response Time 520 ms 380 ms 27% faster
Requests / Sec 34 41 17% more
CPU Utilization 76% 52% 46% lower
Bandwidth Usage 5.2 Gbps 3.1 Gbps 67% less

Caching should be selectively enabled rather than blanket to avoid caching dynamic content. Quick cache purges minimize dated content.

And many more Nginx tuning techniques in our Nginx performance optimization guide!

Hardening Nginx Security

Now that Nginx is tuned for blazing fast content delivery, securing it from compromise is critical. Production attack vectors include:

  • Direct infrastructure attacks
  • Abusing application vulnerabilities
  • Exploiting misconfigurations

While app security hardening is needed too, here we focus on direct Nginx protections.

Access and Authentication

Limiting access to underlying filesystems and sensitive configuration areas prevents data loss or malware injection:

# Allow admin access from office IP 
location /nginx {
  allow 192.168.110.23;
  deny all; 
}

# Developer access with HTTP auth
location /config {  
  auth_basic           "Administrator‘s Area";
  auth_basic_user_file /etc/nginx/.htpasswd;
}

Stringent permissions narrow exposure only to authorized parties. Integration with central authentication systems avoids reliance on just app layer security.

Logging and Monitoring

Enhanced logging enables deeper monitoring, alarming and forensic analysis:

log_format main ‘$remote_addr - $remote_user [$time_local] ‘
                ‘"$request" $status $body_bytes_sent ‘
                ‘"$http_referer" "$http_user_agent" ‘
                ‘$request_time $upstream_response_time $pipe‘;

access_log /var/log/nginx/access.log main;

Custom formats record all relevant metadata per request like IP address, timestamp, headers, statuses, timings etc. aggregating logs to a central repository allows slicing and dicing with tools like Splunk to reconstruct security incidents or find attack clues.

Anomaly detection scripting analyzes for irregularities signalling DDoS or application layer attacks. Metrics dashboards plot unusual traffic spikes eg hundreds of 404 errors.

Guarding Against DDOS

The most dangerous availability threat comes from distributed denial of service (DDoS) attacks attempting to overwhelm infrastructure. Protective measures include: 

limit_req_zone $binary_remote_addr zone=req_limit:1m rate=20r/s;
limit_conn_zone $binary_remote_addr zone=conn_limit:1m;

server {

  location / {
    limit_req zone=req_limit;
    limit_conn conn_limit 100;  
  }

}
  • limit_req enforces max requests per second per client
  • limit_conn limits concurrent connections permitted
  • Bursts accommodate usage spikes
  • Requires monitoring actual traffic base rates

Rate limiting must be tuned carefully to avoid blocking legitimate users during natural spikes eg flash crowds.

Perimeter protections like cloud DDoS filtering may supplement app layer controls for stronger protection, exploiting vast global scrubbing capacity.

Avoid Information Leakage

Something as innocuous as directory listing exposes site structure without authentication:

autoindex off;

This simple hardening measure should almost always be set.

Bonus hardening protections worth investigating:

  • IP address whitelisting
  • Virtual patching and WAF
  • Cryptographic verification of assets
  • TLS termination

And more in our Nginx security guide!

Conclusion

Nginx has stellar foundations like asynchronous event processing, modularity and efficiency. But production workloads at scale have very different demands compared to default configurations geared for plug-and-play compatibility.

Performance tuning and security hardening Nginx for production deployments enables smoothly handling massive user volumes and threats. We walked through optimizing multiple performance factors like disk I/O, network throughput, compression and concurrency capacity. Protecting against attacks and unauthorized access requires techniques like authentication, rate limiting and logging.

There is still more that can be done, but the tweaks discussed here address the most high impact best practices for production Nginx rollout. For sites with rigorous demands and risk profiles, time invested tuning and securing Nginx provides big dividends scaling smoothly while thwarting threats.

What other Nginx hardening or optimization measures have you found helpful? Let me know on Twitter @nginxguru!

Tags: