How to Spot Abnormal Traffic on a Server Using tcpdump

Your server slows down. You run tcpdump and see a flood of SYN packets. This is abnormal. Tcpdump, a command-line packet analyzer, lets you capture network traffic. You can then analyze the capture for red flags. Abnormal behavior often shows unusual TCP flag combinations, high packet rates, or unexpected sources. Red flags include client cannot connect, repeated connection attempts, or unusual flag combinations. Tcpdump helps with troubleshooting. You can filter for specific flags to understand the pattern. This post covers essential commands, filters, and interpretation techniques to detect issues early. Capture a sample of packets and start spotting abnormal patterns.

Key Takeaways

• Capture normal traffic first to set a baseline for comparison.

• Use tcpdump filters to spot SYN floods, port scans, and UDP/ICMP anomalies.

• Interpret flags, TTL, and packet rates to identify red flags.

• Save captures to pcap files for deeper analysis with Wireshark.

• Practice regularly to build confidence in detecting abnormal traffic.

Capturing Baseline Network Traffic with tcpdump

Before you hunt for abnormal patterns, you need a reference point. You must understand what normal network traffic looks like on your server. Researchers formalized this idea. Qayyum et al. proposed a statistical analysis-based technique for anomaly detection. They focus on establishing a normal traffic model. They set a normal baseline. Any deviation from that baseline becomes abnormal traffic. This process gives you confidence in your detection. You know what your server looks like during healthy operation.

Qayyum et al. proposed a statistical analysis-based network anomaly traffic detection technique, which focuses on establishing a normal traffic model and setting a normal baseline. When performing anomaly traffic detection, any deviation from the baseline is considered abnormal traffic.

You apply this same logic with tcpdump. You capture traffic during normal operations. Tcpdump gives you a lightweight tool for this task. You start by capturing packets during routine operations. You study that capture to learn typical patterns.

Setting Up Tcpdump for Interface Capture

Your first step involves choosing the correct interface. Run tcpdump -D to list available interfaces. Look for your external interface. It usually carries a name like eth0. The loopback interface handles internal traffic. You want the interface connecting your server to the network.

Install tcpdump if your system lacks it. Use your package manager. On Debian systems, run apt-get install tcpdump. On CentOS systems, run yum install tcpdump.

Test your setup. Run tcpdump -i eth0 -c 100. This records 100 packets. Tcpdump shows each packet with flags and addresses. You see output immediately.

Observing Normal Network Traffic Patterns

Now start your baseline collection. Choose a time of normal operation. Avoid busy periods. Run tcpdump -i eth0 -c 1000 -w baseline.pcap. This command captures 1000 packets silently. It stores them in a file. This data shows your baseline. You inspect it later with tcpdump filters.

Study this baseline carefully. Look at the traffic mix. Web servers see many SYN and ACK flags. Most connections complete their handshake. Packet rates stay steady. Sources repeat in predictable ways. This network traffic reveals typical patterns. Pay attention to the packet rate. Normal servers show steady rates. Spikes indicate potential issues. Note the source IP addresses during normal hours.

This baseline helps with troubleshooting. Compare new captures against it. Any difference signals a red flag. A flood of SYNs stands out. An ICMP echo storm becomes obvious. These filters help isolate specific patterns. Your packet capturing skills let you see problems early.

Using tcpdump to Capture and Analyze Traffic Patterns

Now you move from baselines to detection. You use tcpdump to capture and analyze network traffic on your server. The key skill involves writing correct filters. Tcpdump lets you capture and filter network packets with precision. These filters isolate specific anomalies from normal network traffic. You monitor in real time with the -l flag. This flag forces line buffering. You see output immediately as packets arrive. You combine filters with live output for quick detection.

Filtering for SYN Floods and Port Scans

SYN floods produce a clear signal. You see many SYN packets with no follow-up ACKs. You need a filter that catches only these initial handshake attempts. The Berkeley Packet Filter syntax, also known as berkeley packet filter syntax, lets you write filters based on byte offsets in headers. This method offers precise control over packet selection.

You start with this filter in real time. Run tcpdump -i eth0 -l "tcp[13] == 0x02". You see every SYN packet arriving at your server. A normal server shows a steady flow of these packets. A flood shows a rapid burst with no subsequent ACKs for each connection. More than a few hundred per second from one source signals an attack. You watch the rate over time. It should stay consistent during normal operation. You compare this real-time view against your baseline from earlier. Differences become obvious immediately.

Port scans look different. A scanner sends one SYN to each port in sequence. You see many unique destination ports from one source in a short time window. Run tcpdump -i eth0 -l -nn "tcp[13] == 0x02". The -nn flag prevents DNS resolution. You see raw IP addresses and port numbers clearly. A single IP hitting many ports indicates reconnaissance. You use tcpdump to filter by port and see this pattern emerge.

You also filter by host to isolate a suspicious source. Add and src host X.X.X.X to your filter. Tcpdump narrows your view to one attacker. You analyze their behavior in detail. This technique helps you understand their scanning pattern and target selection.

Spotting UDP and ICMP Anomalies

UDP anomalies require different filters. DNS amplification attacks use UDP extensively. An attacker sends a small query to a DNS server with a spoofed source IP. The server sends a large response to the victim. Run tcpdump -i eth0 -l "udp dst port 53". You see many UDP packets on port 53 with varying sizes. Normal queries produce small responses under 512 bytes. Large responses indicate amplification. You count response sizes to confirm this anomaly.

ICMP anomalies include ping floods. An attacker sends ICMP echo requests at a high rate. Your server responds to each one, consuming bandwidth. Run tcpdump -i eth0 -l "icmp and icmp[icmptype] == 8". This filter catches only echo requests. A normal server sees occasional pings from monitoring tools. A flood shows hundreds per second from one source. This traffic pattern stands out clearly.

You also watch TTL values during capturing and analyzing network traffic. Normal ICMP packets have consistent TTL values based on their origin. A sudden change in TTL for the same source IP suggests packet spoofing. Tcpdump helps you see this mismatch.

These filters form your detection toolkit. You practice filtering traffic with each filter. You adapt filters for your environment. This process involves capturing packets regularly. You learn to capture and analyze traffic effectively. You capture and analyze network traffic for patterns. You capture traffic for post-incident review. You practice troubleshooting with this tool. You use tcpdump as your primary tool. You analyze captures thoroughly for anomalies. You repeat these capture and analyze sessions regularly. Consistency builds confidence in your detection ability.

Interpreting Tcpdump Output for Red Flags

Raw output means little until you read it correctly. You must decode each field and compare it against your baseline. This step turns a wall of text into a clear diagnosis.

Decoding Flags, TTL, and Packet Rates

The TCP flags field tells the story of every connection. A healthy handshake shows SYN, then SYN-ACK, then ACK. You watch for this sequence in your tcpdump output. A stream of SYN packets with no replies signals a flood or a blocked port. The RST flag marks a refused connection. Many RST packets from one host suggest a scanner probing closed ports.

The time-to-live value reveals the packet’s origin. Each operating system sets a default TTL. You see consistent values from the same source during normal operation. A sudden shift in TTL for one address hints at spoofing. Attackers rarely match the original TTL of the host they imitate.

Packet rates give you the clearest signal. Tcpdump prints a timestamp on every line. You count how many packets arrive per second. Your baseline shows a steady rhythm. A burst that triples that rate deserves attention. You run tcpdump with the -tttt flag to get full timestamps. This helps you measure gaps between packets. Short gaps across thousands of packets point to automated tools, not real users.

Identifying Repeated Connections and Unusual Sources

Repetition exposes intent. A legitimate client opens one connection and finishes its work. An attacker opens hundreds in seconds. You filter your capture for one source address and count the entries. Tcpdump makes this easy with a host filter. You then analyze the pattern of ports and flags.

Unusual sources stand out in several ways. You may see addresses from countries where you have no users. You may see internal addresses talking to external hosts they never contact. You may see one address hitting many ports in sequence. Each pattern tells you something different. A single source touching every port is reconnaissance. A single source sending identical payloads is a brute-force attempt.

You should also watch for half-open connections. These show a SYN with no completing ACK. They pile up when a server runs out of resources. Tcpdump reveals them clearly in the flag column. You use filters to isolate these packets and measure their volume. A growing count means trouble.

Your goal here is to diagnose, not to guess. You compare each red flag against your baseline. You confirm the pattern with a second capture. You then act on solid evidence.

Real-World Attack Patterns in Captures

Detecting UDP Anomalies (DNS Amplification, Ping Floods)

You use tcpdump to spot DNS amplification attacks. An attacker sends a small query to an open resolver with a spoofed source IP. The resolver responds with a large reply directed at your server. You detect this with tcpdump by filtering for UDP port 53 traffic. Focus on responses exceeding 512 bytes. Normal DNS responses stay below that threshold. Large responses signal possible abuse. Tcpdump shows you the source address of each oversized reply. A single resolver sending many large packets to one target confirms an ongoing attack. You track the packet rate to measure the attack’s intensity.

Ping floods appear as a burst of ICMP echo requests. Run tcpdump with a filter for ICMP type 8 echo request packets. Tcpdump isolates only echo requests from other data. Check the arrival rate on your interface. A healthy server shows occasional pings from monitoring tools. A flood shows hundreds of echo request packets from one source within seconds. Tcpdump captures all this data. You inspect the TTL values to check for spoofing. Different TTLs from the same source IP suggest a fake address.

Slow Connection Attempts (Low Window Size SYN Packets)

Some attackers use stealthy scanning techniques. They send SYN packets with a low TCP window size. This choice slows down the connection process deliberately. Standard scanners use a default window value. A very low window size signals a crafted packet. Attackers evade detection by blending in with normal network traffic. Normal connections show consistent window sizes for each operating system. You catch these attempts with tcpdump.

Tcpdump makes this analysis straightforward. The window size appears in bytes 14-15 of the TCP header. Run a filter that isolates SYN segments with a window under 1024. Tcpdump shows you the source and destination for each suspicious packet. Compare the rate against your baseline. Legitimate clients rarely send such small windows. A burst of these packets from one source indicates a slow scan. You document the source address and block it. You capture this data for later analysis. Tcpdump gives you the evidence you need for this action.

Saving Captures for Post-Incident Traffic Analysis

Live monitoring reveals problems in real time. But you cannot study every detail while packets fly by. Saving captures for later review gives you a second chance. The tool tcpdump supports this workflow with file output. Tcpdump preserves the raw evidence for forensic analysis. This practice splits detection from deep investigation.

Writing Packet Captures to File

Tcpdump saves packets in the pcap format. This standard keeps original headers and payloads. Other analysis tools read pcap files without conversion. You create a pcap file with the -w flag. For example, tcpdump -i eth0 -w incident.pcap writes all incoming data. Tcpdump writes directly to disk without screen output. This behavior reduces overhead during high packet rates. You capture continuous streams without dropping data on your server.

You control file size with the -C flag. Tcpdump creates new files after reaching a set megabyte limit. This approach prevents one giant pcap dataset from becoming unmanageable. You also set a limit for the number of packets with -c. Tcpdump stops after capturing that many packets. Use meaningful file names with your captures. Include the date, time, and incident type. Tcpdump then stores your evidence in an organized structure. You retrieve specific captures later when correlating with other logs. This habit pays off during post-incident reports.

Preparing Captures for Wireshark Review

Command-line analysis works for quick checks. Complex traffic demands a richer view. You open saved pcap files in Wireshark for this purpose. Wireshark decodes protocols automatically. It provides protocol decoders and flow graphs. These tools show the sequence of connections clearly. You see how one host communicates with another over time.

TLS handshake details become visible. Wireshark decodes the version, cipher suites, and alerts. Tcpdump cannot present this data in a readable format. You gain a complete picture of your network traffic. Importing pcap files requires only File > Open. Apply display filters to isolate specific conversations. Saving and reading capture files also helps with long-term retention. You keep pcap files for compliance or team training. Your incident team reviews the same evidence later. This method ensures consistent analysis across your organization. Practice with tcpdump regularly to build your skills.

Tcpdump stays lightweight. It spots odd behavior once you know what to watch: flags, rates, and sources. Practice inside a test environment first. Build a baseline, then compare new captures against it. That habit sharpens your eye for troubleshooting real problems.

Grow from here. Automate collection with scripts. Ship those files to a SIEM, or try Bro and Zeek for deeper inspection. Each step turns raw packets into answers. Start capturing today—your network logs may reveal the first signs of trouble.

FAQ

How do I know if a SYN flood is hitting my server?

You see a rapid burst of SYN packets with no matching ACK replies. Run tcpdump -i eth0 -l "tcp[13] == 0x02" and watch the rate. A few hundred per second from one source signals an attack. Compare this against your baseline capture.

Can tcpdump show me DNS amplification in progress?

Yes. Filter for UDP port 53 and look at response sizes. Normal replies stay small. Oversized responses from one resolver to a single target confirm abuse. Tcpdump reveals the source address of each large reply.

What makes a pcap file useful after an incident?

A pcap file preserves original headers and payloads. You open it later in Wireshark for protocol decoding and flow graphs. Your team reviews the same evidence. Keep pcap files organized by date and incident type for easy retrieval.

Why do attackers send SYN packets with a low window size?

A low window size slows the connection handshake deliberately. Standard scanners use default values. A burst of these crafted packets from one source indicates a stealthy scan. You catch them by filtering SYN segments with small window values.

How often should I capture traffic to build a good baseline?

Capture during routine operations across several days. Avoid busy periods for your first samples. A steady packet rate and predictable sources form your reference. Any later deviation from that pattern becomes a red flag worth investigating.