Linux Log Analysis & Auditing

Understanding log analysis and auditing is one of the most critical skills for any system security professional. Logs are your system's "black box recorder" — they capture everything that happens, from successful logins to failed intrusion attempts. When something goes wrong (or when you suspect something *is* going wrong), logs are your first line of investigation. Mastering how to read, search, and monitor them can mean the difference between catching a breach early and missing it entirely.

---

Log Analysis in Linux

Where Logs Live

Most Linux logs are stored in /var/log/. Each file serves a specific purpose:

Security Tip: If you suspect unauthorized access, /var/log/auth.log (or /var/log/secure) is your go-to file. It records SSH logins, sudo usage, PAM authentication, and more.

Viewing Logs in Real-Time

The most commonly used command for watching logs as they update is tail -f:

# Watch auth.log in real-time
tail -f /var/log/auth.log
Watch the last 50 lines, then follow
tail -n 50 -f /var/log/syslog

Think of tail -f like watching a live news ticker — it shows you new entries as they appear, without you having to refresh.

Searching and Filtering Logs

Use grep to hunt for specific events:

# Find all failed login attempts
grep "Failed password" /var/log/auth.log
Find logins for a specific user
grep "session opened for user john" /var/log/auth.log
Find sudo usage
grep "sudo" /var/log/auth.log

Combine tools for more powerful analysis:

# Count failed login attempts by IP address
grep "Failed password" /var/log/auth.log | awk '{print $11}' | sort | uniq -c | sort -rn

Using journalctl for systemd Logs

Modern Linux systems using systemd store logs in a binary journal. Use journalctl to query them:

# View all logs
journalctl
Follow logs in real-time (like tail -f)
journalctl -f
View logs for a specific service
journalctl -u sshd
View logs since last boot
journalctl -b
Filter by priority (0=emergency to 7=debug)
journalctl -p err

---

The rsyslog Service

rsyslog is the system logging daemon responsible for collecting, filtering, and routing log messages to the appropriate files. It acts like a central post office — messages from the kernel, applications, and services are sent to rsyslog, which then decides where to deliver them based on rules defined in /etc/rsyslog.conf.

Key capabilities of rsyslog:

Receives log messages from the kernel and running processes

Filters messages by facility (source) and severity

Writes messages to local files or forwards them to a remote log server

Supports structured logging formats

# Check rsyslog status
systemctl status rsyslog
Restart rsyslog after config changes
systemctl restart rsyslog

---

Auditing with auditd

What is auditd?

The auditd daemon is the Linux Audit System — a powerful framework for tracking security-relevant events at the kernel level. While rsyslog captures application-level messages, auditd goes deeper, monitoring system calls, file access, user actions, and more.

Purpose of auditd:

Track who accessed or modified sensitive files

Monitor privilege escalation attempts

Record system calls made by processes

Provide evidence for compliance audits (PCI-DSS, HIPAA, etc.)

Detect and investigate unauthorized access

Key Components

Setting Audit Rules with auditctl

# Watch a sensitive file for any access or modification
auditctl -w /etc/passwd -p rwxa -k passwd_changes
Monitor a directory for writes and attribute changes
auditctl -w /etc/sudoers.d/ -p wa -k sudoers_changes
List current audit rules
auditctl -l
Delete all rules
auditctl -D

The -k flag adds a key (label) to the rule, making it easy to search for related events later.

Searching Audit Logs

# Search by key label
ausearch -k passwd_changes
Search by username
ausearch -ua john
Search for failed events
ausearch --success no
Generate a login report
aureport --login

Persistent Audit Rules

Rules set with auditctl are temporary. To make them permanent, add them to /etc/audit/rules.d/audit.rules:

# /etc/audit/rules.d/audit.rules
-w /etc/passwd -p rwxa -k passwd_changes
-w /etc/shadow -p rwxa -k shadow_changes
-w /etc/sudoers -p wa -k sudoers_changes

Then reload:

augenrules --load

---

Practical Investigation Scenario

You suspect user bob is attempting unauthorized access. Here's your investigation workflow:

# 1. Check recent authentication events
grep "bob" /var/log/auth.log | tail -20
2. Look for failed sudo attempts
grep "sudo.*bob" /var/log/auth.log
3. Check audit log for file access by bob
ausearch -ua bob --success no
4. Watch logs in real-time while bob is active
tail -f /var/log/auth.log

---

Key Takeaways

/var/log/auth.log (or /var/log/secure on RHEL) is the most important log for investigating authentication and unauthorized access attempts.

tail -f is the standard command for viewing log files in real-time, while journalctl -f serves the same purpose for systemd-managed systems.

rsyslog is the central logging service that collects and routes messages from applications and the kernel to appropriate log files.

auditd provides deep kernel-level auditing, tracking file access, system calls, and user actions — essential for security investigations and compliance.

Use ausearch to query audit logs by key, user, or success status, and aureport to generate human-readable summaries of audit activity.