Reading Server Logs: 404s, Bots and Slow Requests
What each part of a server log line actually shows, how to sort out 404 errors and slow requests, and how to tell real visitors from bot traffic.
rabbitclip teamPublished: 5 min read
Short answer
A server log line carries the requesting address, the date, the page requested, the status code the server returned, and the browser information, all in one line. Reading these regularly answers three questions: which pages are throwing 404 errors, how much of the traffic is real visitors versus bots, and which requests are answered slower than normal. All three are concrete signals visible well before they turn into a reporting problem.
If a facilities management company's site gets a few hundred visitors a month but the server logs tens of thousands of requests a day, that gap almost always comes from bot traffic. There is no way to understand where that gap comes from without looking at the logs.
How to read a log line
A server log is the record a server keeps of every request it receives; the default 'combined' format used by Nginx and Apache lists the requesting IP address, the date, the requested address, the HTTP method, the status code, the number of bytes sent, the referrer, and the user-agent, in that order.
In a typical line, the visitor's IP address comes first, then the date and time in square brackets, then the requested address in quotes starting with GET or POST, immediately followed by a three-digit status code, 200 for success, 404 for not found, 500 for a server error, the amount of data sent, and finally the browser identity in quotes at the end. Learning this order once is enough to read any log file afterwards.
What do 404 errors actually show?
A 404 means the requested page was not found on the server. That comes from a broken link, another page pointing to an address that no longer exists, from a site migration that never redirected the old addresses, or from a bot trying a random address.
Reviewing 404s in the logs regularly shows which broken link is actually affecting real visitors; a 404 that repeats often and comes from a genuine browser deserves fixing first, a one-off bot attempt can usually be ignored.
How to tell real visitors from bot traffic
The first place to look is the user-agent field; known search engine bots identify themselves openly. But a malicious bot usually presents a user-agent that looks like a genuine browser, so that field alone is not enough.
- More than one request per second from the same IP, which does not match human behaviour
- Repeated requests focused only on specific pages, a login form, a search box
- Requests that go straight to deep pages without ever fetching robots.txt first
- A realistic user-agent paired with missing headers a real browser would normally send, language, cookies
Finding slow requests in the logs
Response time does not appear in most default log formats, but Nginx and Apache both offer a separate variable to add it; once added, every request shows how long it took to answer, in milliseconds.
Once that field is in place, filtering for requests over a threshold, say one second, points to which page, which query or which integration is causing the slowdown. That is a signal worth catching before a customer complains the site feels slow.
How long should logs be kept, and who should look at them?
Log files are not kept forever; disk space is limited and most hosts apply a default rotation of a few weeks to a few months. Since a security review after an incident can need older logs, this window is worth setting deliberately rather than leaving to the host's default.
Looking at logs regularly can be a ten-minute weekly habit for one person; the goal is not reading every line, it is glancing at the spread of status codes and the most repeated errors.
Getting a quick summary with a single command
A quick summary can be pulled from a log file without a technical team on hand. A simple command that counts and sorts log lines by status code shows how often each error repeats within seconds; the most frequent 404s or 500s land at the top of the list.
The same idea, sorting by total requests per IP address, immediately points at an unusually heavy source of bot traffic. Both filters give a result far faster than reading through the file line by line.
Common mistakes
Logs usually stay a file that only gets opened once something has already broken; without a regular look, the early warning is missed.
- Only checking logs after the site has already gone down, missing the earlier warning signs
- Never reviewing 404s, leaving broken links unnoticed for months
- Reading analytics as if every visit is a human, without ever accounting for the bot share of traffic
- Never adding a response time field to the logs, so the source of slowness stays unknown
Server logs show a site more plainly than any analytics dashboard; every request, real or bot, is recorded there. A ten-minute weekly habit of looking catches many problems before they turn into a customer complaint. A discovery call with rabbitclip is a good place to read through your own server logs together.
FAQ
Do log files need special software to read?
For a small site, filtering a plain text file from the command line is enough. As traffic grows, a log analysis tool makes the job easier.
Is a 404 always a problem?
No. A bot trying a random address that never existed is normal; the actual problem is a broken link that real visitors keep hitting.
Can bot traffic be blocked entirely?
Not entirely, but known bad-behaviour patterns can be blocked to a large extent at server or CDN level.
Is the server always the cause of a slow request?
No. Slowness can come from a database query, a third-party integration or a large image; the log shows where it slows down, the cause still needs separate investigation.
Can a quick summary be pulled instead of reading the log line by line?
Yes. A simple command that counts and sorts by status code or IP address surfaces the most repeated error or an unusual bot source without reading every line.
