diff --git a/static/articles/server-traffic-shaping.html b/static/articles/server-traffic-shaping.html index 74df9cd1..cf442d17 100644 --- a/static/articles/server-traffic-shaping.html +++ b/static/articles/server-traffic-shaping.html @@ -196,6 +196,45 @@ bottleneck.
+The Linux kernel can be tuned to more quickly propagate TCP backpressure up to + applications while still maximizing bandwidth usage. This is incredibly useful for + interactive applications aiming to send the freshest possible copy of data and for + protocols like HTTP/2 multiplexing streams/messages with different priorities over + the same TCP connection. This can also substantially reduce memory usage for TCP + by reducing buffer sizes closer to the optimal amount for maximizing bandwidth + use without wasting memory. The downside to quicker backpressure propagation is + increased CPU usage from additional system calls and context switches.
+ +The Linux kernel automatically adjusts the size of the write queue to maximize
+ bandwidth usage. The write queue is divided into unacknowledged bytes (TCP window
+ size) and unsent bytes. As acknowledgements of transmitted data are received, it
+ frees up space for the application to queue more data. The queue of unsent bytes
+ provides the leeway needed to wake the application and obtain more data. This can
+ be reduced using net.ipv4.tcp_notsent_lowat
to reduce the default and
+ the TCP_NOTSENT_LOWAT
socket option to override it per-socket.
A reasonable choice for internet-based workloads concerned about latency and
+ particularly prioritization within TCP connections is 128kiB. To configure this,
+ set the following in /etc/sysctl.d/local.conf
or another sysctl
+ configuration file and load it with sysctl --system
:
net.ipv4.tcp_notsent_lowat = 131072+ +
Using values as low as 16384 can make sense to further improve latency and + prioritization. However, it's more likely to negatively impact throughput and will + further increase CPU usage. Use at least 128k or the default of not limiting the + automatic unsent buffer size unless you're going to do substantial testing to make + sure there's not a negative impact for the workload.
+ +If you decide to use tcp_notsent_lowat
, be aware that newer Linux
+ kernels (Linux 5.0+ with a further improvement for Linux 5.10+) substantially
+ reduce system calls / context switches by not triggering the application to
+ provide more data until over half the unsent byte buffer is empty.