Networking commands, and where they get their data

2026-10-18 · 19 min read · networking, linux, tooling

Every tool on this page has a manual measured in screens, and every one of them is used, in practice, in two or three shapes. The shapes are much easier to hold onto once you know where the tool gets its answer from: there are only four sources, and which one a tool uses explains its flags, its speed, what it needs root for, and — most usefully — what it is blind to.

Commands below were run on Linux (iproute2 7.0); the output samples are real.

The four sources

The first three ask this host what it thinks is true. tcpdump asks the wire what actually happened. nmap asks the far end. When two of them disagree, the disagreement is the diagnosis — most of the recipes at the bottom of this page are built out of exactly that.

ss — the socket table

ss sends one NETLINK_SOCK_DIAG request and the kernel streams back packed structs, one per socket. Two things follow. First, it is fast: no text to parse, no per-socket file to open. Second, the common filters — state, address, port — are compiled into the netlink request as bytecode, so the kernel does the matching and never serialises the sockets you did not ask about (anything it cannot express that way, ss filters afterwards). On a box with 50k sockets that is the whole ballgame.

The four you will actually type

ss -tlnp            # what is listening, and which process owns it
ss -tnp             # established TCP connections, with processes
ss -s               # one-screen summary: counts by protocol and state
ss -tunap           # everything TCP+UDP, listening and not, numeric, with pids

If you memorise one, memorise ss -tlnp. "Is my server up, on which address, and is it the process I think it is" is the most common network question there is.

The flags, which are composable and short

-n is not just cosmetic: without it a listing of a few hundred peers issues a few hundred reverse-DNS lookups, which both hangs the command and tells your resolver's operator who you were talking to. Default to -n.

Reading the output

$ ss -tlnp
State  Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0      5            0.0.0.0:8000      0.0.0.0:*    users:(("python3.13",pid=1644,fd=3))

0.0.0.0:8000 is the detail worth slowing down for. A listener on 0.0.0.0 accepts from anywhere; one on 127.0.0.1 accepts only from this machine, and is the single most common cause of "the port is open but I get connection refused from the next host over". Read the address, not only the port.

Recv-Q and Send-Q mean different things per state

This is buried deep in man ss and explains a whole class of outages.

In the sample above, Send-Q 5 is Python's socketserver default backlog of 5 — not a byte count at all. So a LISTEN row with Recv-Q climbing toward Send-Q means the application is too slow to accept, and new clients are about to be dropped. An ESTAB row with a stuck Recv-Q means the application has stopped reading; a stuck Send-Q means the network or the peer has stopped acknowledging. Those are three different bugs and this column tells them apart.

Filtering, in ss's own little language

ss -tn state established '( dport = :443 or sport = :443 )'
ss -tan state time-wait
ss -tn 'dst 10.0.0.0/8'
ss -tlnp 'sport < :1024'
ss -H -tan state time-wait | wc -l        # count, without the header skewing it

State names are the ones in the next table, plus three aliases worth knowing: connected (anything but listening/closed), synchronized (established-ish), and bucket/big for the minisocket split (time-wait and syn-recv live in hash buckets, not full sockets).

A pile of one state is a specific diagnosis

CLOSE-WAIT deserves the emphasis. The kernel cannot clear it for you: it is waiting for your process to call close() on a file descriptor it has forgotten about. Thousands of them is a file-descriptor leak with a network-shaped symptom.

Per-connection TCP internals

$ ss -tin
ESTAB 0 0 10.16.20.204:60284 163.70.151.60:443
  cubic wscale:8,10 rto:203 rtt:2.309/1.49 mss:1380 cwnd:10
  bytes_sent:26039 bytes_acked:26040 bytes_received:171316
  segs_out:204 segs_in:259 send 47812906bps lastsnd:6723
  pacing_rate 95610280bps delivery_rate 37093656bps minrtt:1.374

The five fields to read: rtt:mean/variance (is the path slow?), retrans: (absent here, which is good — present means loss), cwnd against mss (how much the sender is allowed in flight), rto (how long before a retransmit), and lastsnd/lastrcv in milliseconds (is this connection idle or active?). "The network is slow" becomes a measurement here rather than an opinion.

ss -K, the loaded gun

ss -K dst 10.0.0.9            # forcibly destroy matching sockets

Requires CONFIG_INET_DIAG_DESTROY and root. Genuinely useful to break a wedged connection that an application will not drop, and genuinely a way to confuse that application. Reach for it after you have read the filter twice.

netstat — the same data, the slow way

netstat comes from net-tools, predates netlink, and reads /proc/net/tcp: a text table the kernel re-renders on every read. It then does a /etc/services lookup and (without -n) a reverse-DNS lookup per line. On a busy host ss -tan returns in under a second where netstat -an takes minutes. It is also simply absent from many modern distributions and container images.

Keep it for the parts ss does not do:

netstat -s                    # protocol counters: retransmits, resets, drops
netstat -i                    # per-interface packet and error counters
netstat -rn                   # routing table  (prefer: ip route)

netstat -s is the one to remember — it is the kernel's SNMP counter dump, and netstat -s | grep -i retrans is a five-second answer to "are we losing packets" that needs no capture at all.

/proc/net, directly

Both tools above are rendering files you can read yourself, which is worth doing once so the abstraction stops being magic:

$ head -3 /proc/net/tcp
  sl  local_address rem_address   st tx_queue rx_queue ... uid  timeout inode
   0: 00000000:1F40 00000000:0000 0A 00000000:00000000 ... 1000       0  1883
   1: CC14100A:EB7C 3C9746A3:01BB 01 00000000:00000000 ... 1000       0 17859

Everything is hex, and IPv4 addresses are little-endian per word:

• 00000000:1F40 → 0.0.0.0:8000, and st 0A → LISTEN. That is the Python server from the ss output above.
• CC14100A:EB7C → 0A.10.14.CC = 10.16.20.204, port 0xEB7C = 60284, and st 01 → ESTABLISHED.
• inode 17859 is the socket's inode number. Hold onto that: it is how lsof ties a socket to a process.

The st codes, since nothing in the file explains them: 01 ESTABLISHED, 02 SYN_SENT, 03 SYN_RECV, 04 FIN_WAIT1, 05 FIN_WAIT2, 06 TIME_WAIT, 07 CLOSE, 08 CLOSE_WAIT, 09 LAST_ACK, 0A LISTEN, 0B CLOSING.

Other files in there that answer questions on their own: udp, unix, dev (per-interface counters, what netstat -i formats), snmp and netstat (the counters behind netstat -s), sockstat (socket counts and memory), nf_conntrack (the NAT/firewall connection table — check its size against net.netfilter.nf_conntrack_max when a busy box starts dropping new connections), route, and arp.

lsof — which process is holding it

The trick is small enough to state in full: lsof walks /proc/*/fd/*, readlink()s each one, and for the entries that come back as socket:[17859] it joins the inode against /proc/net/*.

$ ls -l /proc/self/fd
lr-x------ 1 framal users 64 Oct 18 13:58 0 -> /dev/null
l-wx------ 1 framal users 64 Oct 18 13:58 1 -> pipe:[16328]

Three consequences: it needs root to see other users' processes (the fd directory is mode 0500, owned by the process owner); it is slow in proportion to the number of processes and fds, not sockets; and it is the only tool here that can answer "which program" for files as well as sockets.

lsof -i :8000                       # who is on port 8000 (either end)
lsof -nP -i TCP -sTCP:LISTEN        # every TCP listener, no DNS/service lookups
lsof -nP -i @10.0.0.9               # connections to one host
lsof -i TCP:1-1024                  # a port range
lsof -p 1644                        # everything that pid has open
lsof -t -i :8000                    # pids only, for piping into kill
lsof /var/log/app.log               # who has this file open
lsof +L1                            # open files that have been *deleted*
lsof -u framal                      # by user
lsof -r2 -i :8000                   # repeat every 2 seconds

-n -P (skip host and port-name resolution) turns a command that stalls into one that returns immediately; type them reflexively. lsof +L1 is the answer to "df says the disk is full but du disagrees" — a deleted file held open by a running process keeps its blocks.

For the narrow "who has this port, kill them" job, fuser is less typing:

fuser 8000/tcp                  # pids
fuser -k 8000/tcp               # and kill them

ip — addresses, routes, neighbours

ip speaks rtnetlink and replaces ifconfig, route and arp, all three of which report a subset of modern reality (notably: ifconfig shows one address per interface well and multiple addresses badly).

ip -br a                        # brief: one line per interface, addresses, state
ip r                            # routing table
ip r get 1.1.1.1                # how would a packet to this address leave?
ip -s link                      # per-interface counters, incl. errors and drops
ip neigh                        # the ARP/NDP cache
ip -4 a / ip -6 a               # one family at a time
ip mon                          # watch link/addr/route changes live

ip route get is underrated and worth a line of its own:

$ ip route get 1.1.1.1
1.1.1.1 via 10.16.20.193 dev enp0s31f6 src 10.16.20.204 uid 1000

That is the kernel resolving one real route decision: the gateway, the interface, and the source address it would pick. Answering "which interface will this use" by reading the routing table yourself and applying longest-prefix-match in your head is how people get it wrong; this makes the kernel do it. It is also the quickest way to catch a VPN that has captured more traffic than you intended.

tcpdump — the packets themselves

tcpdump opens an AF_PACKET socket, compiles your filter expression to BPF bytecode, and attaches it inside the kernel, which then copies only matching frames into an mmap'd ring buffer the process reads. Two things fall out of that design:

1. The filter language is not the display language. tcpdump's syntax (tcp port 443) is BPF; Wireshark's (tcp.port == 443) is a display filter applied after capture. They look similar and are not interchangeable.
2. Filter at capture time, always. An unfiltered capture on a loaded interface overruns the ring and drops frames — and the packet you needed is the one that got dropped. tcpdump prints the loss when it exits: N packets captured, M received by filter, K dropped by kernel. If K is nonzero, narrow the filter and capture again.

Where it taps, and what that hides

AF_PACKET sits at the device layer. On ingress that is before netfilter, so you see packets that your firewall is about to drop — very useful: "the SYN arrives and there is no SYN-ACK" plus a listening socket equals a firewall rule. On egress it is after netfilter, so a packet your own OUTPUT rules drop never shows up at all. It also sees nothing happening inside another network namespace, which is the container trap at the bottom of this page.

The capture invocations

tcpdump -i any -nn port 443                     # the default starting point
tcpdump -i eth0 -nn host 10.0.0.9 and port 5432 # one peer, one service
tcpdump -i any -nn -c 20 'tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn'
tcpdump -i eth0 -w /tmp/cap.pcap port 443       # write raw, analyse later
tcpdump -r /tmp/cap.pcap -nn -A                 # read back, print payloads
tcpdump -i any -nn -A -s0 'port 80'             # HTTP, as text
tcpdump -i eth0 -nn -ttt icmp                   # delta timestamps between packets
tcpdump -i eth0 -W 5 -C 100 -w /tmp/ring.pcap   # 5-file, 100MB ring: leave it running

-nn (no host and no port resolution) matters more here than anywhere else: capturing on port 53 while doing DNS lookups to render the output is a feedback loop that captures itself.

Filter primitives, which compose with and / or / not

The SYN-only and RST-only filters cover a surprising share of real debugging: one shows you who is trying to connect, the other shows you who is saying no.

Pitfalls

• -i any is a cooked pseudo-device. It cannot go promiscuous, loses some link-layer detail, and handles VLAN filters differently. Good for "which interface is this even on", then switch to the real interface.
• "incorrect cksum" on outgoing packets is normal. Checksum offload means the NIC fills the checksum in after tcpdump saw the frame. Only trust checksum complaints about inbound packets.
• Snaplen defaults to 262144 bytes in current versions, so -s0 is a no-op that old blog posts still recommend. The reason to set a small snaplen (-s 96) is to capture headers only, cheaply, for a long run.
• Promiscuous mode shows you other hosts' traffic only on a hub or a mirrored port. On a switched network you see your own traffic plus broadcast, and promiscuous mode buys you nothing but a kernel log line.

To capture remotely and analyse locally, keep the pcap on a pipe rather than copying files around:

ssh host 'tcpdump -i eth0 -nn -U -w - port 443' | wireshark -k -i -

nmap — asking the other end

nmap is the only tool here that is not passive. It builds packets itself with raw sockets, runs its own retransmission and timing logic, and infers port state from what comes back.

Run without privileges, nmap falls back to -sT, which is slower, logged by the target application, and cannot do -O or -sU at all. If results look odd, check whether you actually had privileges.

How it decides a port's state

That three-way distinction is the reason to use nmap rather than curl or nc for reachability questions: closed means a machine answered and nothing is listening, while filtered means nobody answered at all. One is an application problem, the other is a network problem.

The scan invocations

nmap -sn 10.0.0.0/24                      # what hosts exist on this subnet
nmap -sS --top-ports 100 10.0.0.9         # quick scan of the usual suspects
nmap -sS -p- -T4 10.0.0.9                 # all 65535 ports
nmap -sV -p 22,443 10.0.0.9               # what software, which version
nmap -Pn -p 443 10.0.0.9                  # skip discovery: host that ignores pings
nmap --reason -p 443 10.0.0.9             # print *why* it called it that
nmap -oA scan 10.0.0.9                    # normal + greppable + XML output
nmap --script ssl-enum-ciphers -p 443 host  # the NSE script library

Four things that explain most surprises:

• On a local subnet, discovery uses ARP (-PR), which no host firewall can hide from — which is why scanning your own LAN is fast and reliable, and scanning across a router is neither.
• -Pn is the flag you forget. By default nmap pings first and skips hosts that do not answer, so a host that drops ICMP reads as "down" and its open ports are never scanned.
• UDP scanning is inherently slow. Open UDP ports usually say nothing at all, so state is inferred from ICMP port-unreachable messages, which Linux rate-limits to roughly one per second. -sU -p- is an overnight job; scan specific ports.
• -T4 is the usual timing choice on a healthy network. -T5 trades accuracy for speed; -T0/-T1 exist for evading rate-based detection and take hours.

Scan hosts you are responsible for or have written permission to test. A full-port -sV scan is indistinguishable from an attack in anyone's logs, and on managed networks it will be treated as one.

dig and curl — the application-level probes

dig sends DNS queries straight to the server in /etc/resolv.conf (or the one you name). It does not go through nsswitch.conf, so it ignores /etc/hosts, mDNS, and anything else NSS would have consulted. This is the single most common false lead in DNS debugging: dig works, the application does not, because they are not asking the same question.

dig +short example.com                  # just the answer
dig @1.1.1.1 example.com                # bypass the local resolver
dig +trace example.com                  # walk down from the root servers
dig -x 10.0.0.9                         # reverse lookup
dig example.com MX / NS / TXT           # other record types
getent hosts example.com                # what the application actually sees
resolvectl query example.com            # same, via systemd-resolved

Compare dig against getent hosts before you blame the DNS server.

curl is the end-to-end check, and -w turns it into a crude profiler:

curl -v https://example.com                      # the handshake, headers and all
curl -sS -o /dev/null -D - https://example.com   # headers only, body discarded
curl --resolve example.com:443:10.0.0.9 https://example.com/   # test one backend
curl -o /dev/null -s -w 'dns:%{time_namelookup} conn:%{time_connect} tls:%{time_appconnect} ttfb:%{time_starttransfer} total:%{time_total}\n' https://example.com
curl --unix-socket /var/run/app.sock http://localhost/health

Those five timings localise a slow request without a capture: a large time_namelookup is DNS, a large time_connect is the network or a backlog, a large time_appconnect is TLS, and a large gap to time_starttransfer is the application thinking.

Recipes

Permissions

Rather than running a capture as root, grant the capability once:

sudo setcap cap_net_raw,cap_net_admin=eip "$(command -v tcpdump)"

Namespaces and containers

Every tool here reports on its own network namespace. Inside a container ss -tlnp shows the container's sockets and nothing else; from the host it shows the host's and not the container's. Nothing is broken — you are asking the wrong kernel namespace.

nsenter -t <pid> -n ss -tlnp            # run the host's ss inside a container's netns
nsenter -t <pid> -n tcpdump -i any -nn  # capture inside it, with host tools
ip netns list && ip netns exec ns1 ss -tlnp
ip -br a                                # from the host: the veth ends of each container

nsenter from the host is usually better than docker exec, because minimal images ship none of these binaries, and the host's tools work fine against a borrowed namespace. Each container's veth has a peer on the host, so tcpdump -i vethXXXX on the host captures one container's traffic without entering anything.

When they disagree

The whole method, compressed:

• ss says it is listening, the client says refused. Read the local address, not the port — 127.0.0.1 is not 0.0.0.0. Then check you are in the same namespace.
• The client's SYN leaves (tcpdump) and nothing returns. Nobody refused you; something dropped it. Firewall, security group, routing, wrong host.
• The SYN arrives at the server (tcpdump) but the client sees nothing. The reply is being dropped, or the server has no listener and its RST is being filtered — ss -tln separates those two.
• Handshake completes, then nothing happens. The network is fine; look at Recv-Q on the server's socket. The application has stopped reading.