Network packet drop locations

Let’s look at the network transmit and receive paths between two TCP endpoints, and understand at which places in the Linux network stack can we expect packet blocking or packet drops.

Transmit path:

  1. User application: inserts data into the stream (write(), send() or sendmsg())
  2. TCP send buffers (tcp_wmem): kernel copies this data into a SKB structure (socket buffer); data stored here until ACK’d by receiver; sender application blocked if it runs out of socket buffer space (tcp_wmem)
  3. Device queue for TX (qdisc, txqueuelen): kernel inserts SKB into the device queue (qdisc) for TX if there’s room; qdisc can hold txqueuelen SKBs and will drop a segment if it is full
  4. NIC TX ring buffer: dequeue SKB for transmission to NIC’s buffer; if NICs TX ring is full, the SKB will be requeued back on the device queue (qdisc) for another TX attempt

Receive path:

  1. NIC RX ring buffer
  2. Device queue for RX (netdev_max_backlog): which can hold a maximum of netdev_max_backlog packets (simlar as qdisc and txqueuelen)
  3. TCP receive buffers (tcp_rmem)
  4. User application: receives data from a socket (recv())

Having said that, network packet drops or blocking can occur at the following places in the stack if the queues or buffers are full:

  • blocking at sender’s TCP socket buffer space /proc/sys/net/ipv4/tcp_wmem
  • dropping at sender’s qdisc device queue
  • blocking at sender’s NIC TX ring buffer
  • dropping at receiver’s NIC RX ring buffer
  • dropping at receiver’s device queue /proc/sys/net/core/netdev_max_backlog
  • blocking at receiver’s TCP socket buffer space /proc/sys/net/ipv4/tcp_rmem

Comments are closed.