Linux Compromised Server Forensics Cheat Sheet

Published: 2026-10-27  |  Last reviewed: 2026-10-27

Key Takeaways

• Start with auth.log (Debian/Ubuntu) or secure (RHEL/CentOS/Fedora) — they record every login attempt, sudo use, and SSH session.
• journalctl gives you structured, tamper-evident log data across all systemd units, including services an attacker may have installed.
• Hidden processes can often be spotted by comparing /proc directly against ps output.
• Persistence most commonly lives in crontabs, systemd unit files, SSH authorized_keys, and shell RC files.
• Take memory and network-state snapshots before rebooting — a reboot destroys volatile evidence.
• Log paths differ between distributions; confirm the path before assuming a file is empty.

Why Does the Order of Investigation Matter?

Linux servers hold two kinds of evidence: volatile and persistent. Volatile evidence — running processes, active network connections, open file handles, logged-in sessions — disappears the moment the machine reboots. Persistent evidence — log files, cron entries, unit files — survives, but can be overwritten if an attacker has time to clean up.

This means the order you work in is not arbitrary. Capture volatile state first. Then read logs. Then look for persistence. If you start by rebooting "to clear the problem," you may destroy the only evidence that tells you what actually happened and whether data was taken.

One practical note before starting: if you suspect active compromise, do not run investigation tools from the server's own PATH. A skilled attacker can replace ps, ls, or netstat with trojaned versions that hide their activity. Either mount a trusted rescue environment, or at minimum, reference binaries by full path from a read-only mount. For most routine triage, standard tools are fine — just keep that caveat in mind.

How Do You Capture Volatile State Before Anything Else?

The first five minutes are about recording what is currently running — not fixing it. Run these commands and save the output to a file on removable media or a remote host, not to the compromised disk.

# Who is currently logged in and from where
who
w
last -n 50

# All active network connections
ss -tnap
# or on older systems:
netstat -tnap

# All running processes with full paths
ps auxf

# Open files and network sockets per process
lsof -i
lsof -nP | grep LISTEN

# Loaded kernel modules (rootkits sometimes add one)
lsmod

# Current environment for suspicious processes
cat /proc/<PID>/environ | tr '\0' '\n'

# Redirect everything to a log bundle
ps auxf > /tmp/volatile_ps.txt
ss -tnap > /tmp/volatile_ss.txt
lsmod > /tmp/volatile_lsmod.txt

Pay attention to processes whose command line column is empty, has been zeroed out, or shows paths under /tmp, /dev/shm, or /var/tmp. Legitimate system services rarely live there. Miners, reverse shells, and backdoors frequently do.

For a deeper look at memory forensics techniques once you have the snapshot, the memory forensics guide covering Volatility 3 covers injected code extraction in more detail.

How Do You Read Auth.log to Find SSH Brute Force and Successful Logins?

Auth.log (on Debian/Ubuntu systems at /var/log/auth.log) and secure (on RHEL/CentOS/Fedora systems at /var/log/secure) record every authentication event: SSH attempts, sudo commands, PAM authentication, and su usage. These are almost always the first place attackers leave a visible trail, even when they try to clean up afterward.

A brute-force attempt looks like hundreds of "Failed password" lines from a single IP, often within seconds of each other. A successful login after many failures is the real alert — that pattern means the attacker found a valid credential.

Useful commands for working through auth.log quickly:

# Count failed SSH attempts by source IP (Debian/Ubuntu)
grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head -20

# Same on RHEL/CentOS/Fedora
grep "Failed password" /var/log/secure | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head -20

# Find successful logins — look at what follows the failures
grep "Accepted" /var/log/auth.log

# Show all sudo commands run in the last 24 hours
grep "sudo" /var/log/auth.log | grep "COMMAND"

# Find new user account creation events
grep "useradd\|adduser\|new user" /var/log/auth.log

# Show recent logins with source IPs and timestamps
last -F | head -30

One thing to check immediately: whether log rotation happened. If auth.log is suspiciously short, look for auth.log.1, auth.log.2.gz, or check whether rsyslog or systemd-journald has been stopped or reconfigured since the incident started. Attackers sometimes disable logging rather than delete individual entries.

How Do You Use Journalctl to Investigate Systemd Services and Timelines?

On modern systemd-based distributions, journalctl stores structured log data in binary journals that are harder to selectively edit than plain text files. This makes them a useful cross-check against auth.log. Even if an attacker truncated a text log, the journal may still hold the event.

# All SSH events from the journal
journalctl -u ssh --since "2026-10-26" --until "2026-10-28" --no-pager

# Show events from a specific time window (useful for isolating an incident)
journalctl --since "2026-10-27 02:00:00" --until "2026-10-27 03:00:00"

# Show all unit files that started or failed around a suspicious time
journalctl --since "2026-10-27 02:00:00" -p err..crit

# Check a specific suspicious service
journalctl -u my-suspicious-service.service -n 100

# Show kernel messages (rootkit loading, unexpected module inserts)
journalctl -k --since "2026-10-27 02:00:00"

# List all systemd services that have run, including one-shots
systemctl list-units --type=service --all

# Find services in a failed state (may indicate cleanup or conflict)
systemctl --failed

Pay attention to service units that have unusual names, that were activated in the middle of the night, or that point to binaries in /tmp or /var/tmp. Journalctl's --output=verbose mode shows every structured field, including the executable path that generated each log entry — useful when a service's binary has been deleted but the log entry remains.

How Do You Find Hidden or Suspicious Processes on a Compromised Linux Host?

Standard ps reads from the kernel's process table via /proc. A userspace rootkit can replace ps with a version that filters its own PID from the output. But /proc itself is harder to hide from, because it is a virtual filesystem maintained by the kernel.

Compare what ps reports against what is actually present under /proc:

# List every PID directory directly from /proc
ls /proc | grep '^[0-9]' | sort -n

# For each PID in /proc that ps does not show, examine it
for pid in $(ls /proc | grep '^[0-9]'); do
    ps -p $pid > /dev/null 2>&1 || echo "Hidden PID: $pid  $(cat /proc/$pid/cmdline 2>/dev/null | tr '\0' ' ')"
done

# Read the actual binary path for any process (even if cmdline is zeroed)
ls -la /proc/<PID>/exe

# Show all open files for a suspicious PID
ls -la /proc/<PID>/fd

# Show the memory map — look for deleted executables still running
cat /proc/<PID>/maps | grep "deleted"

# Check for processes with a deleted executable (common with droppers)
ls -la /proc/*/exe 2>/dev/null | grep "(deleted)"

The last command is particularly telling. If a process's executable shows (deleted), it means the binary was run and then immediately removed from disk — a classic dropper technique where the attacker does not want the file to be found later. The process is still running from memory, but the file is gone. You can recover it by copying from /proc/<PID>/exe before the process exits.

For understanding how to extract injected code from memory in more depth, the Volatility 3 memory forensics walkthrough covers this on Linux as well as Windows.

How Do You Hunt for Persistence Mechanisms That Survive a Reboot?

An attacker who has had access for more than a few minutes has almost certainly planted persistence — something that reinstates their access if the server reboots or their current session is killed. The most common persistence locations on Linux, in rough order of how often defenders miss them:

Commands to check each persistence location:

# --- Crontab ---
# Show root's crontab
crontab -l -u root

# Show all user crontabs
for user in $(cut -d: -f1 /etc/passwd); do
    echo "=== $user ==="; crontab -l -u "$user" 2>/dev/null; done

# List everything in cron directories
ls -la /etc/cron.d/ /etc/cron.daily/ /etc/cron.hourly/ /etc/cron.weekly/ /etc/crontab

# --- Systemd ---
# Find recently modified systemd unit files (last 7 days)
find /etc/systemd /usr/lib/systemd -name "*.service" -newer /etc/passwd -ls
systemctl list-units --type=service --state=enabled

# --- SSH keys ---
# Check authorized_keys for every user who has a home dir
awk -F: '{ if ($6 != "") print $6 "/.ssh/authorized_keys" }' /etc/passwd | \
  xargs -I{} sh -c 'echo "=== {} ==="; cat "{}" 2>/dev/null'

# --- Shell RC files ---
find /home /root -maxdepth 2 -name ".bashrc" -o -name ".bash_profile" \
     -o -name ".profile" -o -name ".zshrc" | xargs grep -l "curl\|wget\|nc\|bash -i\|/tmp" 2>/dev/null

# --- SUID binaries ---
find / -perm -4000 -type f 2>/dev/null | sort
# New SUID bash is an immediate red flag:
ls -la /bin/bash; stat /bin/bash

# --- New accounts created ---
# Check /etc/passwd for accounts with UID 0 besides root
awk -F: '$3 == 0 {print}' /etc/passwd

# Check for recently added users
grep -v "^#" /etc/passwd | awk -F: '{print $1}' | while read u; do
    echo "$u: $(chage -l $u 2>/dev/null | grep 'Password last changed')"; done

A specific thing worth checking that many guides skip: look at /etc/ld.so.preload. This file tells the dynamic linker to load a shared library before any other. Attackers can place a malicious .so there that intercepts calls from legitimate system binaries — effectively a userspace rootkit that hides files and processes even if the attacker cannot load a kernel module.

# Should be empty on a clean system — any content here is suspicious
cat /etc/ld.so.preload

# Also check for LD_PRELOAD in any running process environment
for pid in /proc/[0-9]*/environ; do
    grep -l "LD_PRELOAD" "$pid" 2>/dev/null && cat "$pid" | tr '\0' '\n' | grep LD_PRELOAD
done

This technique (MITRE ATT&CK T1574.006) is used by some Linux rootkits precisely because it is missed by investigators who only check crontabs and authorized_keys.

What Are the Key Commands for Quick-Reference During Active Triage?

How Do You Check for Unauthorized File Modifications and Backdoored Binaries?

If your distribution uses a package manager with integrity checking — dpkg on Debian/Ubuntu or rpm on RHEL/CentOS — you can compare installed files against their expected checksums. This catches attackers who replaced a system binary like sshd or sudo with a backdoored version.

# Debian/Ubuntu: verify all installed package files
debsums -c 2>/dev/null | grep -v "OK$"
# Flag: anything that isn't "OK" needs investigation

# RHEL/CentOS/Fedora: verify all RPM-tracked files
rpm -Va 2>/dev/null | grep -E "^..5"
# Flag: "5" in the output means MD5 checksum mismatch

# Check if sshd itself has been modified
dpkg -V openssh-server  # Debian/Ubuntu
rpm -V openssh-server   # RHEL

# Find all files modified in the last 24 hours under /usr and /bin
find /usr /bin /sbin /lib -mtime -1 -type f 2>/dev/null | sort

Alongside package verification, check whether critical configuration files have been tampered with. Attackers who gain root often modify /etc/sudoers to add their user, /etc/ssh/sshd_config to enable password auth or add a second port, or /etc/hosts to redirect traffic.

# Check for unusual sudoers entries
visudo -c  # syntax check
cat /etc/sudoers
ls -la /etc/sudoers.d/

# Check sshd config for unexpected changes
grep -E "PermitRootLogin|PasswordAuthentication|AuthorizedKeysFile|Port|AllowUsers" /etc/ssh/sshd_config

# Review /etc/hosts for unexpected redirects
cat /etc/hosts

For environments running zero trust principles, the hardened configuration should make most of these attack paths difficult — but investigating a live incident requires checking actual state, not assumed configuration.

What Should You Do After You Find Evidence of Compromise?

Finding definitive evidence changes the response mode. Before this point you were investigating; now containment takes priority.

• Isolate network access — block the compromised host at the network level, or take the interface down, before the attacker can download additional tools, exfiltrate more data, or pivot to other systems. Do not simply kill the SSH session; network block first.
• Preserve evidence before remediation — take a full disk image if possible, and save the volatile snapshots you captured earlier. Remediation (reinstalling, restoring from backup) destroys forensic detail you may need for root cause analysis or legal proceedings.
• Determine the initial access vector — was it a brute-forced credential, an exploited service, a compromised deploy key, or something else? Without knowing this, you cannot prevent recurrence.
• Audit every system the compromised account touched — lateral movement is the norm, not the exception. Check SSH known_hosts, bash history (if not cleared), and any cloud credentials or API keys present on the compromised host.
• Rotate all credentials that touched the server — SSH keys, passwords, service account tokens, cloud IAM credentials, database passwords stored in environment files or config.

Your incident response plan should define escalation paths at this point — who gets notified, whether the incident meets breach notification thresholds, and who owns the remediation decision. If that plan does not exist yet, a live incident is a hard time to improvise the process.

What Are the Limits of This Approach?

Everything above assumes a userspace-level attack where standard Linux tooling still functions truthfully. A kernel-level rootkit that hooks system calls can make /proc lie, can hide network connections from ss, and can intercept reads of log files before they reach your shell. In that case, you need to boot from external media or analyze a cold disk image with offline tools.

Similarly, log-based investigation has a hard limit: if auditd was not running before the incident, you will not have file-access or process-execution audit trails — only what made it into syslog. Running EDR telemetry or comprehensive logging before an incident is what makes post-incident investigation actually tractable.

Finally, this guide covers investigation steps, not remediation decision-making. Whether to restore from backup, rebuild from scratch, or attempt surgical removal of attacker tooling depends on factors specific to each environment — how certain you are about scope, how sensitive the data was, and what SLA pressure exists.

Strategic Implications

Post-incident investigation almost always reveals the same gap: the evidence needed to answer "how did they get in" and "what did they take" was either never collected or was kept in a place the attacker could reach. Auth.log tells you that a login happened. It does not tell you which files were read afterward unless you had auditd configured with file-watch rules. Process execution logging tells you that a binary ran. It does not capture the arguments unless you had command-line auditing enabled.

The practical takeaway is that a forensics investigation on a system with no pre-incident telemetry often bottoms out at "we know access was gained around 02:11 on October 27th and we know a shell was opened as root, but we cannot determine what data was accessed." That is a very uncomfortable place to be when you are deciding whether a breach notification is required.

The minimum worth having on every internet-facing Linux server: auditd with rules covering process execution, privilege escalation, and file writes to sensitive directories; centralized log shipping so that an attacker who gains root cannot erase the trail by clearing local logs; and a known-good baseline of installed file checksums taken shortly after provisioning. These are not exotic capabilities. They are the difference between an investigation that produces answers and one that produces "we think, but cannot confirm."

The commands in this guide work on any modern Linux distribution. The log paths change between distributions, but the logic does not. Keep this reference somewhere accessible — not on the server you may be investigating.

Sources & References

1. auth.log — Linux man page (man7.org) — authoritative documentation for authentication log format on Linux systems.
2. journalctl — systemd journal query tool (freedesktop.org) — official systemd documentation for journalctl options and structured journal format.
3. MITRE ATT&CK T1053.003 — Scheduled Task/Job: Cron — adversary use of cron for persistence on Linux.
4. MITRE ATT&CK T1574.006 — Dynamic Linker Hijacking (LD_PRELOAD) — technique used by attackers to load malicious shared libraries via /etc/ld.so.preload.
5. Linux security auditing with auditd — Red Hat Sysadmin Blog — practical guidance on configuring the Linux Audit subsystem for forensic-quality logging.