$ Linux One Liners
Back to lessons

Cybersecurity Triage

Count the Most Common User Agents

You need a quick view of which user agents dominate a web access log.

Command

awk -F'"' '{print $6}' ./fixtures/nginx/access.log | sort | uniq -c | sort -nr | head

What changed

Nothing changes. The command extracts the user-agent field and counts repeated values.

Danger

safe

When to use it

Use this when investigating scraping, monitoring noise, or sudden traffic-composition changes.

When not to use it

Do not trust user agents as identity; they are client-supplied strings and can be inaccurate.

Undo or recovery

No undo needed because the command is read-only.

Expected output

A descending count of user-agent strings.

demo script

Disposable terminal steps

  1. awk -F'"' '{print $6}' ./fixtures/nginx/access.log | head
  2. awk -F'"' '{print $6}' ./fixtures/nginx/access.log | sort | uniq -c | sort -nr | head
  3. awk -F'"' '{print $6}' ./fixtures/nginx/access.log | sort | uniq -c | sort -nr | tail
Terminal transcript Demo JSON

simulated output

What it looks like

disposable vessel 
::fixture-ready::
$ awk -F'"' '{print $6}' ./fixtures/nginx/access.log | head
Mozilla/5.0
Mozilla/5.0
Mozilla/5.0
ScannerBot/1.0
ScannerBot/1.0
ScannerBot/1.0
ScannerBot/1.0
ScannerBot/1.0
SyntheticAudit/0.1
SyntheticAudit/0.1
::exit-code::0
$ awk -F'"' '{print $6}' ./fixtures/nginx/access.log | sort | uniq -c | sort -nr | head
     11 Mozilla/5.0
      5 UptimeCheck/2.0
      5 ScannerBot/1.0
      2 curl/8
      2 SyntheticAudit/0.1
::exit-code::0
$ awk -F'"' '{print $6}' ./fixtures/nginx/access.log | sort | uniq -c | sort -nr | tail
     11 Mozilla/5.0
      5 UptimeCheck/2.0
      5 ScannerBot/1.0
      2 curl/8
      2 SyntheticAudit/0.1
::exit-code::0

YouTube Short

Count user agents fast.

When traffic changes, user agents are a useful clue. They are not proof of identity, but they quickly show what kind of clients dominate the log.

LinkedIn hook

A strange traffic spike often has a strange user agent.

Question: Do you treat user agents as evidence, clues, or noise?

experiments

A/B tests to run

Metric: short_click_through_rate

A: A traffic spike usually leaves a user-agent clue.

B: Count user agents before reading the whole log.