Linux Persistence: Processes

2025-04-11 DFIR persistence processes linux persistence processes

Linux Persistence: Processes

Malware doesn’t appear on a system by magic. At some point in the attack chain, the adversary needs to spawn, inject into, or hijack a process. Whether it’s installing itself, poking around, or exfiltrating data, code has to run–and the vessel to run code is almost always a process.

This post covers how attackers can abuse processes for persistence on Linux and Unix-like systems. It isn’t a deep dive into process injection, evasion, or anti-forensics–those are entire topics on their own–but it does establish the foundation you need to understand how process-based persistence works and what signs defenders should look for.

Processes are one of the most heavily scrutinized parts of modern operating systems. Attackers use them creatively–sometimes hiding in plain sight, sometimes relying on subtle tricks, and sometimes not hiding at all. Defenders, in turn, log process creation, track lineage, monitor resource usage, and look for signs of abnormal behavior. Ten years from now, the fundamentals covered here will still largely apply–even if the evasion techniques have evolved.

Smart attackers tend to avoid unnecessary process creation when they can, preferring to operate in memory or piggyback on long-lived system daemons. But even then, they rely on processes–via injection, library hijacking, or unusual syscall behavior. All of these are detectable in their own ways. If you are an analyst or responder, it is not optional–you must understand how processes work, and you must know what “normal” looks like. Anything less is unacceptable.

What is a Process?

A process is a running instance of a program–the execution of code wrapped in a container of metadata, memory, and resources.

On Linux and Unix-like systems, processes follow a typical lifecycle: fork() -> exec() -> run -> exit. One process creates another by cloning itself with fork(), then usually replaces its memory with a new program using exec(). This is how most software is launched–even something as simple as running cat somefile.txt results in the shell spawning a new process to execute /usr/bin/cat.

Each process is tied to a user context, has a unique Process ID (PID), and exists inside one or more namespaces that define its view of the system–including its own PID map, network stack, and mounted filesystems.

Attackers don’t get a free pass to run code invisibly. No matter how subtle their technique, their payloads still execute within a process in one way or another. That’s the weak link defenders can exploit if they know what to look for

The kernel exposes process metadata through the procfs pseudo-filesystem (/proc). Tools like ps, pgrep, and top pull from this interface to show you what’s running–though with enough trickery, what you see isn’t always the whole picture.

Anatomy of a Process

Processes aren’t just code running in memory–they come with a trail of metadata, state, and context. For defenders and adversaries alike, understanding this anatomy is paramount.

Processes expose more than many people realize. Even with stealthy injection techniques, process metadata still leaks clues. Adversaries who understand how to blend in can slip past basic detections. Those who don’t tend to get caught by the first responder who understands how to read /proc.

Conversely, many defenders operate under the false assumption that they understand processes–until something bad happens. Misnamed binaries, injected shellcode, orphans, and deleted but still running executables are very easy to gloss over, but often provide clear signs that something is wrong if you know what you’re looking at.

The elements below represent the most relevant components of processes when investigating persistence, analyzing behavior, or building detections. If you work with Linux or Unix-like systems in any serious capacity, you should know how to find, interpret, and reason about each of these.

Process Identity and Context

Helps answer: What is this process and who owns it?

PID

A Process ID (PID) is a unique identifier assigned to each running Process. PIDs are reused over time, but at any given moment, they uniquely identify a process within its PID namespace. You will see PIDs everywhere: ps, /proc, logs, …

PPID

The Parent Process ID is the PID of the process that spawned it. PPID values are crucial for determining process lineage, which often reveals strange or suspicious behavior. For example, a reverse shell parented by cron or httpd is a red flag.

Some commands to help visualize parent/child relationships are pstree and ps fwaux.

User Context (UID, GID) …

Each process runs with a User ID (UID) and one or more Group IDs (GID). These control file access, signal permissions, and process visibility. On multi-user systems, this context helps prevent users from tampering with each other’s processes–and tells you who owns the execution of a process.

Seeing processes other than those directly related to a service running under a service account for example is a red flag–e.g., the ntp user running a strange process that isn’t ntpd.

Capabilities and Effective Privileges

Traditionally, only UID 0 (root) could perform certain privileged actions such as being able to bind to low-numbered ports, opening raw sockets, or modifying system files such as /etc/passwd.

Software that required these permissions would either run as root and drop privileges after initialization or use setuid/setgid permission bits to elevate a normal user temporarily

Modern Linux systems use capabilities to break up root-level privileges into fine-grained permissions. This grants specific actions without giving full root access:

  • A process can bind to privileged ports with only CAP_NET_BIND_SERVICE.

  • Tools like ping can open raw sockets with CAP_NET_RAW without being fully suid root.

Capabilities are often overlooked during investigations–they enable stealthy privilege escalation without obvious signs of abuse.

You can inspect or assign capabilities using getcap, setcap, or filecap, and review a process’s effective capabilities with capsh --print.

Note: setuid and setgid binaries are still widely used and frequently abused by attackers. Even in modern systems, they remain a common and effective persistence mechanism.

See also:

CWD

The current working directory (CWD) is the directory a process considers “home” for resolving relative file paths. In a shell, this is changed with cd–so you can type ./some_script.sh instead of specifying the full path.

Processes usually inherit their parent’s CWD unless explicitly changed.

You can inspect a process’s CWD via the symlink at /proc/PID/cwd.

CWDs pointing to world-writable directories like /tmp, /var/tmp, or /dev/shm can be suspicious–especially for long-lived or privileged processes. CWDs are also suspicious if they’re running from oddball locations like /usr/share/games/xkobo/.elitehackertools/.

Execution Metadata

Helps answer: _What was run, how it was invoked, and what context was it launched in?

Executable File

The binary on disk that was executed. A symlink to a PID’s executable file is found at /proc/PID/exe.

If the backing file has been deleted or replaced after execution, this symlink will show as (deleted), which can be a strong indicator of tampering or runtime replacement.

Importantly, even when the file has been deleted from disk, /proc/PID/exe still points to the in-memory copy of the binary. You can copy it, hash it, or analyze it just like any other executable. Many malware samples are recovered with this technique during live response.

Command Line

Command line arguments define how a binary was invoked–which flags were passed, which files were specified, and so on. These are typically visible via /proc/PID/cmdline or process listing tools like ps.

Attackers sometimes overwrite argv[] in memory to make a process appear benign–renaming something like malicious_implant to sshd, cron, or httpd. A real-world example is the COSMIC WOLF/SeaTurtle APT group’s SnappyTCP implant, which masquerades as a kernel thread with seemingly innocent-looking names like [kdd-launch], or [bioset].

They may also design their tools to read runtime configuration from stdin to avoid exposing their intent on the command line. For example, AlmondRocks accepts arguments via input redirection:

echo -v relay --tunnel-addr 10.0.0.10:443 | python arox.py

Command-line inspection is a powerful triage point, but don’t rely on it alone–what you see isn’t always what it seems.

Environment

The environment defines persistent runtime settings that apply across a wide range of programs–things like the user’s home directory, preferred editors and pagers, language settings, and more.

By default, child processes inherit the environment from their parent process. However, attackers can modify or replace the environment at runtime to evade detection.

In C, the environment is passed to main() alongside arguments:

int main(int argc, char *argv[], char *envp[])

For defenders, environment variables can expose valuable clues:

  • LD_* variables (like LD_PRELOAD) can be used to inject malicious libraries and tamper with the loader.

  • SSH_* variables may contain details about originating connections in remote sessions.

  • Large or suspicious environment payloads may indicate the use of shellcode or exploit staging–a tactic historically common in memory corruption exploits.

You can view a process’s environment at /proc/PID/environ. The environment is NULL-delimited, so using strings or tr against this file makes it much easier to read:

strings /proc/PID/environ
cat /proc/PID/environ | tr '\0' '\n'

TTY, PTY

A TTY (teletype) is a terminal interface–typically tied to a user’s interactive session. It’s what lets users type commands and see output when connected via a console or login shall.

A PTY (pseudo-terminal) is a software-emulated version of a TTY, created in pairs (master/slave). They’re used by tools like ssh, screen, and tmux, and are often seen in reverse shells or scripted sessions.

Processes with an attached TTY or PTY are usually interactive. Background services and daemons typically run without one. If a suspicious process has an unexpected TTY, or a PTY shows up tied to a long-running daemon, this warrants deeper investigation.

Check this with ps or ls -l /proc/PID/fd/.

Resources and State

Helps answer: What a process is doing or accessing?

File Descriptors

File descriptors are references to open resources–files, sockets, pipes, devices, and more. Each process maintains a numbered list of its active descriptors.

By convention:

  • FD 0 = stdin
  • FD 1 = stdout
  • FD 2 = stderr

Malicious processes may keep open handles to suspicious files, FIFOs, deleted binaries, or network connections that aren’t obvious from the command line.

These can be inspected with ls -l /proc/PID/fd/.

Memory Maps

Each process maintains a virtual memory layout that includes its stack, heap, executable code, loaded libraries, and memory-mapped files. This layout is exposed at /proc/PID/maps.

Inspecting memory maps can help identify:

  • Injected shared objects (e.g., from LD_PRELOAD or injection).

  • Memory regions tied to deleted binaries.

  • Suspicious memory-mapped files used for staging.

Priority and Limits

Processes can have assigned limits on how much CPU time, memory, file descriptors, and other resources they’re allowed to use. These are often set via PAM or limits.conf, and can be configured per shell session with ulimit.

While resource limits are less commonly used today on single-user systems, they still appear in multi-user environments and containerized workloads. Some malware or stealth tools may deliberately set low priority using nice, renice, or ionice to reduce the chance of being noticed in top-like tools. Conversely, cryptocurrency-mining malware may set higher priorities to maximize mining performance.

You can inspect a process’s limits via /proc/PID/limits.

Communication and Behavior

Helps answer: How does the process interact with the system and other processes?

Network Connections

A process may initiate outbound connections (e.g., command and control traffic), listen for inbound connections (e.g., bind shells), or utilize network resources in one way or another.

Network access is extremely common in malware–attackers almost always want remote access to their implants. Networks provide a convenient method to communicate with compromised systems.

You can view process-level network activity using:

  • ss -p, netstat -pant, netstat -tulpn, … – show socket and PID/command mappings.

  • lsof -i – list open network connections and associated PIDs.

  • /proc/PID/fd/ – look for descriptors pointing to sockets.

  • /proc/net/ – low-level socket and routing information (requires parsing)

Abnormal listeners (e.g., a shell binding to 0.0.0.0:4444) or outbound connections to unusual external IP addresses warrant deeper investigation.

Threads

Threads are units of execution that share the same memory space. Many modern programs are multi-threaded, breaking work into concurrent tasks–especially in network services or GUI applications.

For example, a web server might use one thread to accept incoming connections and others to handle each request.

Attackers can inject malicious threads into long-lived processes to hide their payloads using a variety of injection techniques.

Inspect threads via /proc/PID/task/ – each thread has its own sub directory.

Note: Many programs run as a single thread. If a process that is typically single-threaded suddenly has multiple threads, it may warrant closer inspection.

See also:

System Calls, Library Calls

System calls (syscalls) are the low-level interface between user-space programs and the kernel. They handle fundamental operations like file access (open, read, write, …), process control (fork, execve, signal, ...), and network activity (socket, connect, bind, listen`, …).

Library calls are higher-level functions provided by shared libraries like libc. These often wrap one or more system calls for convenience.

Understanding syscall behavior is key in both offensive and defensive contexts:

  • Malware often reveals itself by using syscalls or library calls with unusual or predictable patterns.

  • Analysts can use strace, ltrace, auditd, or eBPF-based tools to monitor system calls in real time.

Nearly all meaningful process activity goes through system calls and library calls–and tracing them is one of the best ways to understand what a binary is actually doing under the hood.

Signals

Signals offer a lightweight way for processes to communicate with each other or for the kernel to notify a process of an event. They’re commonly used to pause, resume, restart, or terminate processes.

The kill command (and kill() syscall) sends signals to a target PID. How a process responds depends on whether it handles that signal:

  • SIGKILL – Cannot be caught or ignored–forces immediate termination.

  • SIGTERM – A request to terminate–processes can catch this to shut down gracefully.

  • SIGHUP – Often signals a configuration reload or terminal hangup.

  • SIGSTOP / SIGCONT – Used to pause and resume execution.

Some malware installs signal handlers to avoid termination, trigger self-destruction, or control execution via interprocess communication. For example, the Diamorphine rootkit hooks the signal() syscall in the kernel and uses a magic signal to cloak or uncloak processes on demand.

For a more comprehensive list of signals, see signal(7).

TracerPid

The TracerPid field in /proc/PID/status indicates whether a process is being traced (debugged) by another process–typically using ptrace.

Attackers can abuse ptrace to:

  • Inject code into a running process.

  • Read memory (e.g., credentials, cryptographic keys, other sensitive data).

  • Hijack control flow.

  • Modify system calls or behavior at runtime.

These techniques are often used in stealthy implants, memory-resident implants, and anti-forensics utilities.

Check for unexpected traces using: grep TracerPid /proc/PID/status

  • If the value is 0, the process is not currently being traced.

  • If the value is non-zero, it is being traced by another PID.

If a process is being traced and there’s no obvious reason (e.g., a developer debugging software with gdb), it’s worth investigating the tracer’s PID.

A real-world example of ptrace abuse is the 3snake malware, which monitors for newly-created ssh, sudo, or su processes–injecting code into these processes using ptrace to harvest credentials.

Here are some common tools used to analyze active processes and the surrounding ecosystem. In most cases, these tools will already be installed–but your mileage may vary, especially on minimal, legacy, or embedded devices.

Listing and Identifying Processes

Tools for listing, searching for, or inspecting active processes.

ps

ps displays a snapshot of currently running processes.

This command is a bit cursed–flags will vary wildly across systems. The version provided by procps (common default on most Linux systems) differs from the one in busybox (common on embedded systems) and BSD-style implementations (e.g., macOS, FreeBSD).

On full-featured Linux systems, you’re likely to have procps. On stripped-down appliances or routers, expect a limited version (likely busybox).

See also:

pgrep

pgrep searches for processes matching a pattern and can optionally signal them.

Example:

% pgrep -l sshd
1155690 sshd

See also:

pidof

pidof is similar to pgrep–it can be used to search for PIDs belong to processes matching the supplied name. However, it is an older tool (still doesn’t mean its bad), that only accepts exact names of processes instead of patterns.

See also:

pstree

pstree displays processes in a tree format showing parent-child relationships–a visual alternative to ps forest view.

See also:

top

top is an interactive tool that displays real-time process activity. This program requires a TTY–so it won’t work in background agents such as CrowdStrike or ConnectWise.

htop

htop is a modern replacement for top with colorized output, robust filtering options, and scrollable views. This is not always installed by default, but widely available.

pwdx

pwdx displays the CWD of a given PID, similar to ls -l /proc/PID/cwd.

See also:

stat /proc/PID/exe

/proc/PID/exe is a symbolic link to the executable file that started the process.

stat will show metadata about the symbolic link. Of particular interest is the modification time, which reflects when the process was started:

stat /proc/2645177/exe

Sample output (trimmed for brevity):

  File: /proc/2645177/exe -> /usr/bin/zsh
Modify: 2025-04-03 19:24:54.244639794 -0700

This zsh process was started at 2025-04-03 19:24:54.244639794 -0700.

readlink resolves the symbolic link of /proc/PID/exe to its actual path on disk:

Sample output:

% readlink /proc/2645177/exe
/usr/bin/zsh

Hashing Running Executables via /proc/PID/exe

You can hash a process’s backing binary–even if it has been deleted or replaced–via the /proc/PID/exe symlink.

This is useful for:

  • Searching the hash on VirusTotal and similar sites.

  • Comparing to package manager or NSRL hashes.

  • Detecting tampering or untracked binaries.

Depending on the system, these hashing tools may or may not be available.

Example usage:

% sha256sum /proc/2645177/exe
d6e4a916cdd745de3d55b73285c9801b8e96647ff839d215f004793b82391ac5  /proc/2645177/exe

% sha256sum /usr/bin/zsh
d6e4a916cdd745de3d55b73285c9801b8e96647ff839d215f004793b82391ac5  /usr/bin/zsh

Note: /proc/PID/exe may show up as (deleted), which indicates that the file on disk has been deleted or modified. This technique works even if the binary has been deleted as long as the process is still active. Additionally, these deleted but active processes can have the binary recovered using cp: cp /proc/PID/exe /tmp/recovered

See also:

Killing and Manipulating Processes

Send signals, adjust priorities, and interact with process state.

kill

The kill command sends a signal to one or more PIDs. By default, it sends SIGTERM, which politely asks the process to terminate, giving it an opportunity to shut down gracefully.

Attackers may register signal handlers to intercept or ignore SIGTERM. To forcibly terminate a process, use SIGKILL (signal 9) which cannot be caught or ignored (unless using a rootkits):

kill -9 <pid> [pid2 pid3 ...]

See also:

killall

killall sends signals to all processes matching a given name. It is functionally similar to kill, but operates on process names instead of PIDs.

Example:

killall -9 sshd

See also:

pkill

pkill is similar to killall, but with more powerful matching features. It supports name patterns, matching by user, terminal, session, etc.

Examples:

pkill -u www-data
pkill -f evil_script.py

See also:

File and Descriptor Inspection

Look at what files and sockets a process is using.

lsof

lsof is a powerful utility for listing open files. It can filter by PID, user, file types, network connections, and more.

Examples:

ls -p PID      # all descriptors for a process
lsof -i        # network-related descriptors
lsof -u root   # all open files for root
lsof +L1       # list open, but deleted files

Note: This is one of the most underappreciated tools in DFIR and systems administration–deep knowledge of lsof is extremely helpful in live response scenarios.

See also:

/proc/PID/fd/

The /proc/PID/fd/ directory contains symbolic links for every open file descriptor held by a process. You can readlink or inspect each one to see what’s in use:

ls -l /proc/PID/fd/

This is helpful for catching:

  • Open sockets (socket:[12345])

  • Deleted but active files.

  • Pipes and unexpected device files.

/proc/PID/fdinfo/

The /proc/PID/fdinfo/ directory contains metadata about the file descriptors listed in /proc/PID/fd/ including:

  • Offset – the file pointer position.

  • Access mode – read/write

  • Flags – O_CLOEXEC, O_DIRECT, …

cat /proc/PID/fdinfo/FILE_DESCRIPTOR_NUMBER

This is useful for spotting:

  • Unexpected read/write access.

  • Suspicious or abnormal flag combinations.

  • File descrptors being used in strange ways.

Network Visibility

When malware communicates with command and control systems or listens for inbound connections, it typically does so using sockets. These tools help with correlating network connections with the processes that opened them.

netstat

netstat displays active connections, routing tables, and other network-related data.

Example:

netstat -tulnp
  • -tulnp shows TCP/UDP listeners and the associate PIDs and program names.

Note: Many modern systems deprecate netstat in favor of ss.

See also:

ss

ss is a utility used for investigating sockets and current network state. It is often a replacement for netstat.

ss -tulnp

See also:

lsof -i

lsof -i displays information about active sockets.

Examples:

lsof -i               # all network sockets
lsof -i :22           # sockets bound to port 22
lsof -i @192.168.1.1  # connections to/from specified IP

/proc/net/

An honorable mention to this section goes to /proc/net/, a directory containing files like tcp, udp, and others. These files expose raw information about current network conections–the same data is used by tools like ss and netstat.

If these tools aren’t available you can parse these files directly to extract connection information. The format is structured but raw, and includes useful fields such as local/remote addresses, state, UID, and the inode associated with the socket.

These inodes can be mapped back to specific processes via its own file descriptors under /proc/PID/fd/.

Example: Correlating sshd Sockets to /proc/net/tcp

Each entry in /proc/net/tcp has an inode field in the 10th whitespace-delimited column. It is important to note that it isn’t the 10th column, as some of the fields are combined into comma-delimited fields:

% cat /proc/net/tcp | head -n 1
  sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode

List the open file descriptors for sshd:

% sudo ls -l /proc/$(pidof sshd)/fd/
total 0
lr-x------ 1 root root 64 Apr  8 11:29 0 -> /dev/null
lrwx------ 1 root root 64 Apr  8 11:29 1 -> 'socket:[3755970]'
lrwx------ 1 root root 64 Apr  8 11:29 2 -> 'socket:[3755970]'
lrwx------ 1 root root 64 Mar  6 11:50 3 -> 'socket:[3755979]'
lrwx------ 1 root root 64 Apr  8 11:29 4 -> 'socket:[3755981]'

Now search for those inodes in /proc/net/tcp:

% grep -E '3755970|3755979|3755981' /proc/net/tcp
   1: 00000000:56CE 00000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 3755979 1 000000007fc191cf 100 0 0 10 0

From the header earlier, the second field is local_address. In this case, 56CE is the port, in little-endian hex.

This can be decoded with Python:

int.from_bytes(bytes.fromhex("56ce")[::-1], byteorder='little')
22222

This Python snippet yields 22222. ss shows that sshd is indeed listening on this port:

% sudo ss -tulpn |grep ssh
tcp   LISTEN 0      128                                 0.0.0.0:22222      0.0.0.0:*    users:(("sshd",pid=1155690,fd=3))

Parsing /proc/net/tcp manually is powerful and occasionally necessary (especially on minimalist or broken systems), but it’s not fun. This trick is worth knowing, especially when socket to process linkage is needed, but tools like ss and netstat are unavailable.

Runtime Debugging and Behavior

Tools in this category let you inspect or interact with live processes–ideal for analysis, triage, and deep dives.

gdb

gdb is the GNU Debugger and a staple of dynamic analysis and reverse engineering.

It can attach to a running process (--pid=PID) or launch a new process under the debugger (gdb /path/to/binary). You may also run gdb interactively and attach to a running process with the attach command.

While gdb is often unavailable on production systems for security reasons, it’s invaluable for analyzing malware in memory.

Note: Yama settings may disable the ptrace API entirely, making GDB impossible to use, as it utilizes ptrace to debug processes.

See also:

strace

strace traces system calls and signals. It’s an essential tool for watching how a process interacts with the system–opening files, spawning children, making network connections, and more.

strace can be attached to a running process using the -p PID command line flag, or spawn a child process under it by specifying the path to the executable on the command line.

strace will output system calls executed by a process and the arguments passed to them.

Note: strace output can be very noisy. Consider filtering with the -e flag. See the manual for usage examples for this flag.

See also:

ltrace

ltrace is very similar to strace, but focuses on library calls (e.g., strncpy, malloc, free, etc.) rather than syscalls (e.g., open, read, write, etc.). It can attach to a running process with the -p command line flag, or spawn a new process to analyze by specifying a path to an executable.

See also:

pmap

pmap shows the memory map of a process. It’s helpful for seeing which files, libraries, or shared objects are mapped into memory.

Example usage:

pmap PID [PID2 PID3 ...]

See also:

/proc/PID/maps

/proc/PID/maps shows a detailed breakdown of the memory regions mapped by a process. It is similar to pmap, but more granular and scriptable.

grep TracerPid /proc/PID/status

grep TracerPid /proc/PID/status shows the PID of the process that is currently tracing (debugging) the specified process.

Defenders can take advantage of this to detect processes under the influence of a debugger.

Basic Binary Analysis Tools

These tools let you inspect executables on disk to determine what they do, what they link against, and what secrets they may reveal.

strings

strings prints readable strings contained in binary files. This should be one of your first triage steps when analyzing suspicious files.

You’ll often uncover:

  • C2 domains and IPs.

  • Hard-coded credentials.

  • Debug messages.

  • Attack tool names or malware family indicators.

strings is also useful against memory dumps or deleted but running binaries (/proc/PID/exe).

NOTE: Running programs against untrusted or hostile files carries risk. For example, CVE-2014-8485 was a vulnerability in strings that allowed specially-crafted input to trigger code execution. Yes, this bug was real. Yes, defenders should be cautious. But let’s be honest–some researchers love writing sensational headlines like “PSA: Don’t Run ‘strings’ on Untrusted Files” and frame the issue as if your computer is going to explode, first piercing holes in the network allowing ransomware operators to storm your network unopposed like an unstoppable rebel force, causing irreparable damage to your organization. The risks are valid, but you don’t need to panic or throw your tools in the trash. Use your head–keep your tooling up to date, analyze malware in controlled environments, and be aware that bugs like this can and do exist. Don’t let sensationalism scare you out of doing solid analysis.

file

The file command uses libmagic to determine a file’s type, regardless of its extension.

This is helpful for catching renamed executables (e.g., beacon.jpg that’s actually an ELF binary), or simply determining what a file is.

Example:

file suspicious.jpg

See also:

xxd, hexdump

Hex dumps allow you to visually inspect binary content. This is great for:

  • Spotting ELF headers.

  • Dumping embedded payloads such as shellcode.

  • Getting around file transfer limitations (hex dump output can be converted to the original file using tools like CyberChef).

You may luck out and have hexdump installed:

hexdump /path/to/file

Example output:

hexdump -C /bin/sh |head
00000000  7f 45 4c 46 02 01 01 00  00 00 00 00 00 00 00 00  |.ELF............|
00000010  03 00 3e 00 01 00 00 00  60 47 00 00 00 00 00 00  |..>.....`G......|
00000020  40 00 00 00 00 00 00 00  88 e3 01 00 00 00 00 00  |@...............|
00000030  00 00 00 00 40 00 38 00  0d 00 40 00 1d 00 1c 00  |....@.8...@.....|
00000040  06 00 00 00 04 00 00 00  40 00 00 00 00 00 00 00  |........@.......|
00000050  40 00 00 00 00 00 00 00  40 00 00 00 00 00 00 00  |@.......@.......|
00000060  d8 02 00 00 00 00 00 00  d8 02 00 00 00 00 00 00  |................|
00000070  08 00 00 00 00 00 00 00  03 00 00 00 04 00 00 00  |................|
00000080  18 03 00 00 00 00 00 00  18 03 00 00 00 00 00 00  |................|
00000090  18 03 00 00 00 00 00 00  1c 00 00 00 00 00 00 00  |................|

If hexdump is missing, you may find xxd. xxd is a hex dump tool provided by the Vim editor:

xxd /path/to/file

Sample output:

% xxd /bin/sh |head
00000000: 7f45 4c46 0201 0100 0000 0000 0000 0000  .ELF............
00000010: 0300 3e00 0100 0000 6047 0000 0000 0000  ..>.....`G......
00000020: 4000 0000 0000 0000 88e3 0100 0000 0000  @...............
00000030: 0000 0000 4000 3800 0d00 4000 1d00 1c00  ....@.8...@.....
00000040: 0600 0000 0400 0000 4000 0000 0000 0000  ........@.......
00000050: 4000 0000 0000 0000 4000 0000 0000 0000  @.......@.......
00000060: d802 0000 0000 0000 d802 0000 0000 0000  ................
00000070: 0800 0000 0000 0000 0300 0000 0400 0000  ................
00000080: 1803 0000 0000 0000 1803 0000 0000 0000  ................
00000090: 1803 0000 0000 0000 1c00 0000 0000 0000  ................

If neither tool is installed, here’s a small Python fallback:

#!/usr/bin/env python3

import sys

def hexdump_file(path):
    with open(path, 'rb') as f:
        offset = 0
        while chunk := f.read(16):
            hex_bytes = ' '.join(f'{b:02x}' for b in chunk)
            ascii_bytes = ''.join(chr(b) if 32 <= b < 127 else '.' for b in chunk)
            print(f'{offset:08x}  {hex_bytes:<47}  {ascii_bytes}')
            offset += 16

if __name__ == "__main__":
    hexdump_file(sys.argv[1])

This small script can be copied to a host to hex dump files, then removed.

ldd

ldd shows dynamic library dependencies of an ELF file.

Malware often fakes or hijacks dynamic libraries. This can help spot unexpected dependencies.

See also:

nm

nm lists symbols exported by ELF files–useful for static analysis.

Often, malware authors will name functions and variables that reveal their intent such as connect_to_c2, shellcode, put_and_execute. They may also obfuscate or randomize symbol names, or use tricks to not export the symbols at all.

See also:

How Processes Are Created

This section covers how processes are created on Linux systems–both through standard mechanisms and attacker-controlled methods.

Understanding this is critical for defenders and red teamers alike, as these low-level mechanics form the backbone of persistence and evasion.

Standard and Legitimate Creation Paths

Under normal circumstances, a Linux system boots, runs the init process (PID 1), and executes startup scripts and services. Once initialized, it spawns user login services (e.g., sshd, getty) that ultimately hand off control to users via shells such as bash or zsh.

init Process

The first userspace process on a Linux system is init. It is always assigned PID 1 and is responsible for bringing the system to an operational state.

All normal processes are descendants of init.

systemd vs SysV

systemd is the default init system on most modern Linux distributions. It is important to note that some distributions do not use systemd, despite the trend of widespread adoption.

Older or minimalist setups may still use SysV init, OpenRC or runit instead.

Startup Scripts

Legacy startup scripts typically reside in /etc/init.d/, /etc/rc*, or /etc/inittab. These scripts are often invoked by the init system during boot, typically by symbolic links placed in runlevel-specific directories like /etc/rc3.d/.

For a closer look at startup scripts, see Linux Persistence: Startup Scripts.

systemd Unit Files

Modern Linux systems primarily use systemd, but there are still a few distributions that have not adopted it. systemd replaces traditional startup scripts with structured .service unit files. These are INI-style configuration files that define how and when a service should be started.

Common locations:

  • /etc/systemd/systemd/ – local, persistent changes.

  • /lib/systemd/system – package-managed unit files.

  • /run/systemd/system – runtime-generated (non-persistent).

These files are managed with the systemctl command:

systemctl enable some_service
systemctl start some_service
systemctl status some_service
Services

Services (or daemons) are long-lived background processes, typically managed by the init system. They often run without direct user interaction and may start at boot or on demand.

Some services listen for incoming connections (e.g., sshd, httpd) while others handle internal tasks like logging (syslogd) or job scheduling (cron).

These processes are typically launched via systemd unit files or legacy startup scripts.

As a defender, it’s important to recognize the names and roles of common system services. Attackers frequently name their malware after legitimate services–or slight variations of them–to blend in with normal activity. Once you’re familiar with what should be running on a system, spotting impostors becomes much easier.

Shells

Shells are command-line interfaces that allow users to interact with the system–tools like bash, sh, zsh, and others. They’re commonly launched when a user logs in via a terminal or SSH, and they spawn child processes as commands are executed.

Process lineage is useful here: if you see a process tree like sshd -> bash -> curl, it is a strong indicator that a user (or attacker) ran that command remotely.

Some usage patterns can raise red flags. For example:

  • bash -i spawns an interactive shell. This is often seen in reverse shell payloads where the attacker is trying to simulate a full terminal session.

  • sh -c or bash -c runs a command passed as a string. This is commonly abused in exploits, malicious scripts, and payload delivery (e.g., via cron, systemd, or injection).

Unusual shell invocation patterns outside of normal user login sessions should be scrutinized–especially when combined with suspicious parent processes or odd execution times.

Scheduling

Systems tend to offer at least one mechanism to schedule commands to be run at specific times or intervals. These are essential for routine operations like backups, log rotation, and monitoring–but they’re also popular targets for attackers seeking persistence or stealth.

By leveraging scheduled tasks, attackers can:

  • Run malicious code at reboot or regular intervals.

  • Obfuscate process lineage (e.g., no direct parent like sshd or a shell).

  • Blend into legitimate maintenance activity.

cron

cron is the most common scheduler and is supported across nearly all Unix systems. It runs commands based on time-based rules defined in crontabs.

Scheduled tasks tend to be repetitive and predictable–malicious entries often stand out. Look for:

  • Unexpected entries in system or user crontabs.

  • Scripts pointing to odd paths (e.g., /tmp/.x).

  • Crontabs added outsie package management or configuration management tooling.

  • Crontabs not added via the crontab command.

For a more detailed look at cron as a persistence mechanism, see Linux Persistence: cron.

systemd timers

Systemd’s native scheduling mechanism may replace cron on some newer systems. Timers pair with .service units and offer fine-grained control, such as dependencies or running only when idle.

Suspicious timers may be overlooked during triage if analysts focus only on cron. Investigate:

  • Unusual .timer files in /etc/systemd/system/.

  • Modified or newly created timers that don’t match expected system behavior.

  • Timers that were created or modified within the timeline of an incident.

at

The at command (backed by the atd service) schedules one-time tasks to run at a specified time in the future. While less commonly used than cron, it serves a similar purpose and can be just as useful to attackers for persistence and evasion. Its slightly esoteric nature may also cause it to fly under the radar of some defenders.

For a deeper dive into persistence with at see Linux Persistence: atd.

Programmatic Process Creation

Programs frequently need to spawn child processes to perform tasks. This can be done in several ways–each with different tradeoffs and behaviors.

Understanding these mechanisms is important not only for system design but also for security analysis. These same functions are used by attackers to launch payloads, spawn shells, or chain together tools.

Monitoring these calls with tools like auditd or EDR can reveal process creation events that are suspicious or anomalous.

Common process creation functions include:

  • system() – Spawns a shell to execute a command.

  • popen() – Executes a command and returns a file handle to its stdin or stdout stream.

  • exec*() – Replaces the current process with a new one. There are several of these functions that behave slightly differently and used in specific contexts.

  • fork() – Creates a copy of the current process.

  • clone() – More flexible than fork(), often used to implement threads or containers.

From a malware analysis perspective, the presence of these calls is a strong indicator that the binary creates or manipulates processes.

The Dynamic Linker

The dynamic linker (or dynamic loader) is responsible for finding and loading the shared libraries a program depends on, resolving their symbols, and preparing the process for execution.

Understanding how it works is important for defenders, as attackers frequently manipulate this stage of execution for stealth, persistence, and code injection.

ld.so

On Linux, the dynamic linker is typically named ld.so or a variant like ld-linux-x86-64.so.2 depending on the architecture. It’s invoked automatically by the kernel for dynamically linked binaries but can also be invoked manually.

The dynamic linker used by a given binary is defined in the ELF’s PT_INTERP program header. This header specifies the path to the interpreter that the kernel should invoke when executing the binary.

While attackers can abuse this to specify a malicious custom loader, defenders can also leverage it to enforce additional security controls–such as using a custom loader that adds ASLR or similar features to binaries. If the PT_INTERP header is absent, the binary is assumed to be statically linked and does not use a dynamic loader.

The ELF interpreter assigned to a binary can be viewed with readelf:

readelf -l /path/to/binary | grep Requesting

The ELF interpreter can be modified on an existing binary with patchelf:

patchelf --set-interpreter /evil/ld.so /path/to/binary

See also:

Direct Invocation

The linker can be called directly to run a program–even if that program does not have its executable bit set:

% /lib64/ld-linux-x86-64.so.2 /usr/bin/whoami
daniel

% cp /bin/whoami /tmp/whoami
% chmod 400 /tmp/whoami
% ls -l /tmp/whoami
-r-------- 1 daniel daniel 39792 Apr  9 10:19 /tmp/whoami
% /lib64/ld-linux-x86-64.so.2 /tmp/whoami
daniel

This bypasses normal execution flows. It’s occasionally used for novelty or as a stealth trick in malware.

Shared Objects

Shared objects (also called shared libraries or just simply “libraries”) are files that provide reusable functionality which can be dynamically loaded by multiple programs at runtime. They typically end in .so (e.g.< libc.so.6) and resid in standard library paths like /lib/, /lib64/, or /usr/lib/.

Shared libraries offer several key advantages:

  • Efficiency – Programs don’t need to embed all functionality–they can rely on shared libraries. This saves disk space and memory.

  • Maintainability – When bugs are fixed in a shared library, all programs using it benefit without needing to be recompiled.

  • Flexibility – Shared libraries can be versioned or swapped out, allowing dynamic upgrades or targeted debugging.

From a security perspective, however, this flexibility can be abused:

  • Library replacement – Attackers can overwrite legitimate libraries with malicious ones if they have sufficient access.

  • Path hijacking – If attackers place a malicious .so in a location that gets searched before the legitimate one (e.g., via LD_LIBRARY_PATH or writable directories earlier in the search order), they can silently hijack behavior.

  • Injection – Malware can inject a shared object into a running process via LD_PRELOAD, dlopen(), or ptrace.

These features make shared objects a major vector for persistence, privilege escalation, and stealth.

Static-linked Binaries

Static-linked binaries bundle all required libraries into the final executable. Because of this, they have no external runtime dependencies–the dynamic loader doesn’t need to resolve or load any .so files.

For attackers, this is a win: a statically-linked ELF file can carry everything it needs, meaning it’s more portable and more likely to “just work” on a wide variety of target systems.

For defenders, statically-linked binaries are invaluable during incident response. Since they don’t rely on shared libraries, they’re immune to many userland rootkits that use LD_PRELOAD or shared library hijacking techniques. A statically-linked shell or toolkit (like BusyBox) can render these kinds of rootkits useless as they are unable to hook into them.

Statically-linked binaries can be identified using the ldd or file commands:

ldd /path/to/binary
file /path/to/binary

LD_* Environment Variables

The LD_* environment variables control various aspects of the dynamic linker’s behavior. While primarily intended for debugging and development, attackers often abuse these features to hijack or tamper with process execution.

For a comprehensive list, see ld.so(8).

LD_PRELOAD

LD_PRELOAD specifies one or more shared objects to be loaded before any others. This allows the user (or attacker) to override system calls, functions in libc, and other libraries, injecting custom behavior into binaries.

LD_AUDIT

LD_AUDIT is an auditing API for the dynamic linker. It is run before LD_PRELOAD, making it useful to neutralize preload-based rootkits) if static tools aren’t available.

Its usage is rare in malware, but powerful in defensive contexts.

See also:

LD_LIBRARY_PATH

LD_LIBRARY_PATH overrides the default library search paths. If set, the loader searches these directories before the system’s defaults.

This is frequently abused to hijack system behavior by loading malicious versions of common libraries (e.g., libc.so).

LD_DEBUG

LD_DEBUG enables verbose output from the dynamic linker during program execution. While useful for debugging, it can also reveal information about unexpected or malicious behavior during process startup.

Process Creation Abused for Stealth

Attackers often launch or manipulate processes in ways designed to evade detection. These techniques commonly avoid traditional logging, obscure process ancestry, or bypass monitoring of standard execution paths.

Detached or Daemonized Execution

Processes can be launched in a detached state so they survive logout, hide lineage, or operate independently from the original shell.

nohup

nohup allows a command to continue to run after the controlling terminal has been closed (hung up, or user logout).

nohup payload &

See also:

disown

disown is a shell-builtin in many shells such as bash and zsh that removes a job from the shell’s job table, allowing it to persist even after logout.

payload &
disown
setsid

setsid launches a new session and detaches the process from the current controlling terminal:

setsid ./payload

See also:

Process Injection

Attackers may inject code into existing processes to blend in with legitimate activity, hijack trusted binaries, or execute fileless payloads.

ptrace

The ptrace API–used for debugging–allows one process to inspect and modify the memory and execution state of another process.

ptrace can be abused to inject shellcode, modify a running process’s memory, or calling functions within a target process. It can also be used for memory scraping, enabling attackers to steal sensitive information such as passwords and cryptographic keys.

Attackers may abuse tools such as strace or gdb to accomplish these tasks or write their own homegrown tools that utilize ptrace directly.

GDB-based injection

Under the hood, gdb uses ptrace. By attaching gdbto a process, an attacker can make it execute arbitrary system calls--such asmmap`

  • write + mprotect or dlopen.

Tools like gdbrpc automate these techniques.

See also:

memfd_create + fexecve

The memfd_create and fexecve system calls allow execution of ELF binaries directly from memory without writing them to disk.

  • memfd_create() creates a memory-based file descriptor.

  • fexecve() executes binaries from an open file descriptor, rather than a traditional file path.

When used together, these system calls enable true fileless execution. While there are legitimate use cases for this technique, their presence should raise suspicion.

You can spot memory-based file descriptors in /proc/PID/fd/. They’ll appear as:

memfd:some_name

See also:

Userland exec

Rather than calling functions such as execve() directly–which is commonly monitored by EDR and auditing frameworks–attackers can implement their own process loading functionality in userland.:

  • Parse the ELF file manually.

  • Allocate memory and map segments appropriately.

  • Set up the stack, heap, environment, etc.

  • Transfer execution to the program’s entry point.

This technique avoids traditional process creation events and often bypasses syscall-based detection entirely. It is rarely seen in commodity malware but has been used in advanced implants.

See also:

Malicious Process Heuristics

Now that we understand how processes are created–both legitimately and maliciously–we can shift focus to spotting the outliers. This section covers practical heuristics defenders can use to identify suspicious or potentially malicious processes in real-world environments.

Executable and Filesystem Oddities

Attackers frequently exploit legitimate features, quirks, and weaknesses in filesystems to hide, persist, or execute malicious processes. This section outlines some of the most common filesystem-based indicators that can help uncover suspicious activity.

Suspicious Executable Paths

On most systems, executables provided by the operating system or package manager live in standard locations like /bin, /sbin, /usr/bin, and /usr/sbin. Third-party or manually-installed tools may reside in /opt or /usr/local/bin.

These directories are usually defined in the PATH environment variable, meaning users and scripts can run commands from these directories without specifying the full path.

Executables found outside of these expected locations–especially in obscure or user-created directories–deserve scrutiny. Malware often hides in places that users and admins rarely check.

Examples:

  • /usr/share/games/gtetrinet/.asdfjkdsajfdsa/.implant

  • /var/spool/mail/.x

Residing in World-Writable Directories

Attackers often gain access to systems through low-privileged users like www-data or mail, which typically lack distinct home directories or general write access. As a result, attackers rely on world-writable directories–meant for temporary files–as landing zones for malware.

These world-writable directories are abused frequently:

  • /tmp

  • /var/tmp

  • /dev/shm

Any executable activity originating from these locations is suspicious and warrants immediate inspection.

Additional world-writable directories might exist due to misconfiguration. Admins sometimes unknowingly create them with overly permissive permissions.

To hunt for world-writable directories:

find / -type d -perm -0002 2>/dev/null

To find directories writable by the current user (even if not world-writable):

find / -type d -writable 2>/dev/null

To check what a specific user can write to (requires sudo or runuser):

sudo -u USERNAME_HERE find / -type d -writable 2>/dev/null
runuser -u USERNAME_HERE find / -type d -writable 2>/dev/null

cwd is Not / or the User’s Home Directory

For most non-interactive processes, the current working directory (cwd) is typically /, the user’s home directory, or a path explicitly set by a service.

If a non-shell process is running from an unusual or unexpected directory, especially a writable or transient location, it warrants a closer look. Unusual cwd paths are often overlooked, but they can be a subtle sign of malicious activity.

Suspicious Filesystem Permissions and Attributes

Attackers frequently abuse file permissions and filesystem features to maintain access, escalate privileges, or evade detection. This section highlights several common tactics that should raise red flags during investigation.

suid root Files

Attackers often abuse the SUID (Set User ID) bit to escalate privileges. A binary marked as SUID and owned by root will run with elevated privileges regardless of the invoking user.

They might create a custom binary that spawns a root shell, or leverage an existing SUID binary (like sudo) to gain privileged access.

Here’s an example of a custom SUID binary that spawns a root shell:

% cat suid.c
#include <unistd.h>

int main() {
    setuid(0);
    setgid(0);
    execl("/bin/sh", "");
}

% gcc -o suid suid.c
% sudo chown root:root suid
% sudo chmod 4755 suid
% ls -l suid
-rwsr-xr-x 1 root root 16056 Apr  9 12:22 suid*
% ls -l suid
-rwsr-xr-x 1 root root 16056 Apr  9 12:22 suid*
% ./suid
# id
uid=0(root) gid=0(root) groups=0(root),20(dialout),24(cdrom),25(floppy),27(sudo),29(audio),30(dip),44(video),46(plugdev),100(users),106(netdev),111(bluetooth),113(lpadmin),116(scanner),1000(daniel)

To locate existing SUID root binaries on a system:

find / -perm -04000 -user root -type f -exec ls -l {} \; 2>/dev/null

Note: Not all suid root binaries are malicious–in fact, many are essential to system functionality. Binaries like passwd, sudo, and ping often rely on the SUID bit to perform privileged operations. As a defender, it is important to build a baseline of what’s normal for your environment. Unfamiliar or newly-introduced SUID binaries should always warrant closer inspection.

Overly-Permissive Files

Overly-permissive files–especially world-writable scripts or binaries–can be prime targets for attackers.

Example attack chain:

  1. Attacker finds a world-writable script: /usr/local/bin/backup.sh

  2. Script runs as root via cron to perform nightly backups of the system.

  3. Attacker appends a payload that creates a SUID root shell.

  4. Script runs on schedule, the attacker’s payload is execute, and the attacker can now escalate privileges.

Other abuse scenarios:

  • World-writable web directories (/var/www/htdocs) can be manipulated to include web shells or siphon sensitive data input from the application’s users (passwords, credit card information, other PII).

  • Loose read permissions may expose configuration files with credentials or key material.

  • Attackers overwriting writable binaries or scripts to inject code.

Append Only and Immutable Attributes

Some filesystems support special attributes like immutable and append-only, which prevent files from being modified or deleted–even by root.

Attackers can abuse these attributes to:

  • Prevent defenders from removing malicious files.

  • Lock in changes to critical files, evading configuration management tools and package managers.

These attributes can be inspected or modified with:

  • lsattr – List a file’s attributes.

  • chattr – Change a file’s attributes.

Ownership and Permission Non-Uniformity

Attackers often stash malware in busy directories to blend in. Their files may stand out if:

  • Ownership differs (e.g., daniel:daniel vs. root:root)

  • Permissions deviate slightly

  • Group memberships are off.

For example:

% ls -l /usr/bin/
-rwxr-xr-x 1 root   root          68496 Sep 20  2022 [*
-rwxr-xr-x 1 root   root             39 Aug 15  2020 7z*
...
-rwxrwxr-x 1 daniel daniel        95161 Apr  9  2025 malware
...
-rwxr-xr-x 1 root   root            197 Mar 18  2023 zstdless*

Even subtle anomalies–like a different group or an extra write bit–are worth investigating if everything else is consistent.

Suspicious Capabilities

Linux capabilities allow fine-grained control over privileged operations, granting specific permissions to binaries without giving them full root access with SUID. While useful for reducing attack surface in legitimate applications, they’re also able to be abused by attackers.

An attacker might set a capability like CAP_SYS_ADMIN or CAP_NET_RAW on a custom binary to give it elevated functionality–without needing it to be setuid root or run as root.

Examples of dangerous or suspicious capabilities:

  • CAP_SYS_ADMIN – Considered the “god mode” of capabilities–it’s overly broad and often unnecessary.

  • CAP_NET_RAW – Allows raw socket access, which is frequently abused in backdoors and packet sniffers.

  • CAP_SYS_PTRACE – Allows tracing other processes–useful for injection.

Examples:

Get caps of a file:

getcap /path/to/file

Get all caps in a directory, recursively:

% /usr/sbin/getcap -r /usr/bin
/usr/bin/dumpcap cap_net_admin,cap_net_raw=eip
/usr/bin/ping cap_net_raw=ep

See also:

Open File Descriptors Point to Deleted Files or FIFOs

Attackers can open files then delete them from disk as an evasion tactic–as long as the file descriptor remains open, the contents stay in memory, but the file disappears from view using standard tools like ls or find.

Sometimes this is a deliberate evasion tactic. Other times it’s sloppy software that doesn’t clean up after itself. Either way, processes with open file descriptors to deleted files deserve scrutiny.

To check:

find /proc/*/fd -type l -lname '*deleted*' 2>/dev/null

A clever example of this in action is shellgame, a userland rootkit that leverages this behavior to cloak malicious files from deletion.

Executables Running from File Descriptors

Execution directly from file descriptors is unusual and worth investigating. This technique can enable fileless malware and evade simple file-based detections.

You’ll typically see this behavior via:

  • memfd_create + fexecve

  • /dev/fd/*

  • /proc/PID/fd/

These patterns are rare in legitimate software and are often associated with malware, stagers, and red team tooling.

See also:

Strange Binary Format

Executable files that don’t match the expected format or architecture of the system are highly suspicious. On a typical Linux x86_64 system, you should expect to see ELF binaries for that architecture. Anything else–like PE (Windows), or Mach-O (macOS), or ELF for unrelated architectures (e.g., ARM, MIPS, SPARC, ..) should raise an eyebrow.

Common scenarios where this shows up:

  • Post-exploitation tool kits – Attackers often drop zip files or tarballs full of multi-platform tools for convenience. These may include tools targeting different architectures.

  • Wrong assumptions – Attackers may think that your system is x86_64 and drop incompatible binaries, revealing themselves in the process.

There are very few legitimate reasons for non-native binaries to be present on a system, so they are worth digging into.

See also:

/proc-based Red Flags

The /proc filesystem is one of the most powerful tools a defender has on a Linux host. It offers a real-time view into running processes and system state–including memory maps, file descriptors, environment variables, command line arguments and more.

One of the nicest things about /proc is that you don’t need fancy tools to work with it. Standard utilities like cat, ls, grep, and strings can take you a long way.

This is a deep topic worth its own write-up, but even a shallow understanding of /proc can dramatically improve your ability to find malware on live systems.

For a more detailed breakdown, see Using procfs for Forensics and Incident Response

/proc/pid/exe Not Provided By a Package Manager

One powerful technique to identify potential malware is to compare the executable behind each running process with the system’s package manager database.

On well-maintained systems, nearly all legitimate executables will be installed via the package manager. Any process whose binary isn’t associated with a known package warrants inspection.

Basic Approach:

  • Iterate over /proc/*/exe

  • Resolve each exe symlink to their target file path.

  • Query the package manager for the package that provides it:

    • Debian-based: dpkg -S /path/to/exe

    • RPM-based: rpm -qf /path/to/exe

  • Flag anything not tied to a package as suspicious.

An example script for dpkg-based systems:

for exe in /proc/[0-9]*/exe; do
    path=$(readlink -f "$exe")
    if [ -f "$path" ]; then
        if ! dpkg -S "$path" &>/dev/null; then
            echo "Suspicious ($(echo $exe | cut -d / -f 3)): $path"
        fi
    fi
done

Note: Some legitimate software might be installed manually and thus not associated with a package. On the flip side, attackers can bypass this type of detection by bundling their malware as a package or tampering with package manager metadata to make it appear legitimate. Even with these exceptions, this heuristic can greatly narrow the scope of an investigation and provide quick and easy victories for the defender.

/proc/pid/exe is (deleted)

Malware often deletes its executable from disk after launching to make analysis and detection harder. When this happens, the /proc/PID/exe symlink still points to the now-unlinked file and will show (deleted) in its path.

You can identify these with:

ls -l /proc/*/exe | grep '(deleted)$'

If the process is still running, you can recover the deleted binary:

cp /proc/PID/exe /path/to/recovered_file
⚠ Warning: Legitimate software may also show (deleted) after updates or system maintenance if the process hasn't been restarted yet. Context matters.

If readlink /proc/PID/exe fails on a userspace process, it may indicate tampering or stealth techniques such as:

  • A rootkit obscuring process details.

  • An executable that has been unlinked or manipulated after execution.

Note: This is normal behavior for kernel threads, which do not have backing executables. These threads appear with names encapsulated in square brackets (e.g., [kworker/0:0]) and will fail readlink by design.

/proc/PID/exe is Packed, Obfuscated, or Crypted

While not inherently malicious, binaries that are packed, obfuscated, or encrypted warrant scrutiny–especially if their behavior is unclear or they appear outside of expected locations.

Legitimate uses of packers include:

  • Compressing large binaries (e.g., installers or Golang applications with large statically-linked libraries)

  • Protecting intellectual property in proprietary software.

However, attackers also utilize packers and crypters to:

  • Shrink payloads.

  • Evade static detection.

  • Obfuscate code to deter reverse engineering.

Some common examples:

  • UPX – Widely used for both legitimate and malicious software. Presence alone isn’t damning, but deserves a closer look.

  • Midgetpack – Less common; more suspicious than UPX when seen in the wild.

  • Burneye – Very old–32 bit only, and almost exclusively used in offensive tooling or malware samples.

Packers and crypters often leave fingerprints–like unusual section names or high file entropy–which can be detected with YARA or tools like ent.

See also:

Abnormal Objects in /proc/PID/maps

The /proc/PID/maps file shows all memory-mapped regions of a process. This includes shared libraries, heap/stack regions, and anonymous mappings. It’s an excellent place to detect process injection, custom loaders, shellcode, and other shady memory behavior.

Attackers abuse memory maps in a few key ways:

Injected or Unexpected Shared Objects

If you see a suspicious or out-of-place .so file loaded into a process (especially one not tied to the software’s normal behavior), that’s a huge red flag.

Common indicators include:

  • Shared libraries from unexpected paths (e.g., /tmp/libhax0r.so, or user home directories).

  • Libraries with high file entropy or strange names.

  • Libraries that aren’t backed by files at all (anonymous mmap regions marked r-xp).

Anonymous Executable Mappings

Anonymous memory regions ([anon] or missing path name) marked with r-xp or rwxp permissions are often used for:

  • Injected shellcode.

  • JIT-compiled code.

  • Memory-resident payloads.

  • Custom packers or loaders.

Tools and Techniques

  • cat /proc/PID/maps – Quick and dirty inspection.

  • lsof -p PID – See loaded files, compare with /proc/PID/maps.

  • grep -E "rwx|r-x" /proc/PID/maps – Flag executable regions.

  • YARA or entropy tools against dumped memory.

Naming and Identity Red Flags

Attackers often disguise their processes by renaming them to match legitimate system services–or something close enough to pass a casual glance. If you’re familiar with what should be running on a system, these tricks can be easy to spot.

Misspelled or Mangled Process Names

Attackers often disguise malware by using names that closely resemble legitimate services:

  • Legitimate service with a trailing space (e.g. sshd )

  • Using leetspeak – (sys1ogd, cr0n, …)

  • Misspellings: (ssshd, htttpd, …)

These tricks exploit visual similarity to mislead defenders skimming process lists. Being familiar with normal service names in your environment makes these stand out.

Random or High-Entropy Names

Some malware uses completely random or high-entropy names like rXv9tsdZ or gK90l_i1v5az to evade pattern-matching.

While this avoids signature-based detection, it’s obvious to human analysts–legitimate processes rarely have names like this.

Legitimate-Looking Names With Suspicious Behavior

Another common tactic is naming malware after a real service (ntpd, cron, sshd), or completely made up but legitimate-sounding names (loggingd, user-helper, httpd-ipc, …) while exhibiting behavior that the real service wouldn’t.

For example:

  • A fake ntpd process making outbound HTTP connections.

  • A cron process spawning reverse shells.

  • A sshd binary running out of /tmp instead of /usr/sbin.

comm Doesn’t Match cmdline (argv[] was overwritten)

Process name spoofing has been around for decades–notably used by the Morris worm in the late `80s) and remains a common obfuscation tactic. This technique is known as process masquerading or name stomping and lets attackers make malicious processes look like legitimate ones.

There are two common ways this is done:

  • Overwriting argv[] to change the command line string shown to tools like ps.

  • Using prctl() to set a new process name.

A simple and effective detection method is to compare the values in:

  • /proc/PID/cmdline – The full command line arguments.

  • /proc/PID/comm – The process name as set by prctl() or inherited from argv[0].

See also:

Non-Kernel Thread Process Names Enclosed In Square Brackets

On Linux systems, kernel threads are typically named using square brackets (e.g., [kworker/u8:0]). These are easily ignored by casual observers because they appear technical, cryptic, and are expected to run quietly in the background.

Attackers abuse this by renaming userland processes to mimic kernel threads–e.g., naming their process [klogd] or [networking]–to blend into the noise and avoid scrutiny.

However, these fakes are easily detectable:

  • Legitimate kernel threads do not have an executable file linked under /proc/PID/exe.

  • A kernel thread’s cmdline is empty.

  • Kernel threads are almost always children of kthreadd.

Process Lineage and Timing Anomalies

Understanding how processes relate to one another–what spawned what, and when–can reveal subtle signs of compromise. Most processes follow predictable patterns. When those patterns are broken, it’s often a sign that something isn’t right.

Suspicious Parent-Child Relationships

One of the simplest heuristics is asking: “Why is this process the parent of that?”

Examples:

  • A web server spawning a reverse shell.

  • cron spawning a file exfiltration tool.

  • A benign-looking process with a parent of init or systemd.

Processes spawned outside of their normal context deserve scrutiny and can provide valuable clues in to which service has been abused by an attacker.

Processes Starting Shortly After Login or Service Start

Malware often leverages:

  • Shell startup scripts: .bashrc, .bash_profile, .zshrc, etc.

  • Global profile scripts: /etc/profile, /etc/bash.bashrc

These allow attackers to trigger payloads automatically at login–either for persistence or delayed execution.

While login periods are often noisy, most users don’t frequently modify these scripts. Any sudden or recent changes to them are suspicious.

Similarly, startup scripts tied to services may be modified to launch malware when the service starts. A good example of this is the user’s window manager on workstations. Watching for processes that spawn immediately after a service comes online can catch this behavior.

Orphaned or Unexpectedly Daemonized Processes

Most user-launched processes trace back to a shell. If a process appears as a direct child of init, systemd, or another service–but runs as a non-root or user-level account–it may have been daemonized intentionally.

This technique is often used to detach malware from its parent session, ensuring it survives logout or monitoring interruptions.

It is not always malicious–some benign tools behave this way–but it’s uncommon enough to be worth checking.

Environment Abnormalities

Data contained within a process’s environment can be a goldmine for defenders. It contains configuration and behavioral hints–everything from search paths to dynamic linker instructions to shell history manipulation.

Depending on the software, the environment can vary wildly. Nonetheless, unusual values, missing values, or values with suspicious paths or strings can be strong signals of compromise.

How the Environment Gets Set

Environment variables are often manipulated using shell built-ins such as:

  • export VAR=value

  • unset VAR

  • VAR=value command – inline assignment.

Because these are built into the shell, they don’t appear in process listings and evade traditional command-line monitoring. The effects can be spotted by reviewing /proc/PID/environ, logs, or shell histories.

Note: /proc/PID/environ reflects the environment at the time of process launch. It does not update if the process modifies its own environment later. Additionally, shell histories are not reliable–they can be disabled or manipulated by the attacker. They are also not typically saved until the shell exits gracefully–if the attacker is still logged in or their shell is terminated, the underlying history may be lost.

PATH

Attackers may hijack PATH order to insert malicious binaries that override expected behavior. By placing their directory earlier in the PATH, they can shadow common tools:

export PATH="/malicious/path:$PATH"

From here, any calls to commands like ps, netstat, or even ls will invoke the attacker’s version (if present). This is often abused to hide their activity or spoof legitimate output.

This technique is especially effective in scripts, cron jobs, or mis-configured environments that blindly trust what’s first in PATH.

HISTFILE

Attackers often manipulate HISTFILE to avoid leaving behind command history by setting it to a non-writable location like /dev/null, or unsetting it entirely:

export HISTFILE=/dev/null
unset HISTFILE
export HISTSIZE=0

This is a simple but effective anti-forensics trick, especially when combined with interactive shells or manual post-exploitation work.

SSH_CONNECTION, SSH_CLIENT

The SSH_CONNECTION and SSH_CLIENT environment variables are set by OpenSSH clients and contain information about the remote connection, such as the IP address and ports from the originating host. Their presence strongly suggests that the process was spawned via an interactive or automated SSH session, which can be helpful contextually during investigations.

LD_PRELOAD, LD_*

The dynamic linker can be influenced by a number of environment variables, most of which start with LD_. These are often abused by attackers to inject code into processes or alter runtime behavior.

LD_PRELOAD and LD_LIBRARY_PATH are particularly common and should be treated with caution when found in a process environment.

For more details, see the previous Dynamic Linker section.

Shellcode in the Environment

Attackers may use environment variables to stash shellcode, NOP sleds, or other payload data–especially in memory corruption or exploitation scenarios.

Since the environment is mapped alongside the binary, it’s a convenient location for code execution tricks. Look for processes with environment strings that contain long sequences of binary data, high-entropy patterns, or recognizable opcode fragments.

Suspicious Environment Size

Both unusually small and abnormally large process environments can be signs of malicious activity.

  • Empty environments – Malware may clear out its environment variables entirely to avoid leaking clues or for basic evasion. While not inherently malicious, a fully environment in user processes is uncommon and worth a closer look.

  • Over sized environments – On the other end of the spectrum, excessively large environments–especially those approaching the system limit (usually 128Kb)–can indicate something is hiding payloads, encoded data, or shellcode. Most legitimate processes use just a few kilobytes (1-5Kb) of environment data.

Resource Usage Abnormalities

Abnormal resource usage–whether extremely low or excessively high–can be a useful indicator when hunting for malicious processes.

Low Resource Utilization

Processes that have been running for a long time but show little to no resource usage may be stealthy implants awaiting a trigger. These are often designed to evade detection by blending into the background.

High Resource Utilization

Conversely, malware that performs active tasks–like password cracking, DDoS attacks, data exfiltration, or cryptocurrency mining–tends to leave a heavier footprint. Look for:

  • Unusually high CPU or RAM usage.

  • Excessive open file descriptors.

  • Sustained or anomalous network activity.

Sudden spikes or sustained consumption by unexpected processes should be investigated.

Network Abnormalities

Network activity can be one of the most revealing aspects of a malicious process. Even stealthy malware has to communicate at some point.

Unusual Listening Ports

Processes listening on unexpected ports should always raise suspicion. If a service doesn’t need to accept inbound connections, it shouldn’t be listening. Malware may open ports to receive commands or act as a backdoor.

Strange Outbound Connections

Processes making outbound connections to unusual destinations–especially to countries or networks not commonly associated with your operations–should be scrutinized. For example, an ntpd process speaking HTTP to a server in Panama may be a red flag.

if possible, inspect the actual traffic. A process using an unexpected protocol (e.g., SSH from a web server binary) may indicate abuse or masquerading.

High Bandwidth Usage

Excessive or sustained bandwidth usage may suggest data exfiltration, DDoS participation, or other abuse.

Anti-Forensics and Rootkits

When attackers suspect that their malware might be discovered, they often employ anti-forensics techniques to delay or derail analysis. These tricks make it harder for defenders to trace, investigate, or even notice malicious processes in the first place.

This is a deep and nuanced topic worthy of dedicated study. What follows are a few foundational techniques and indicators to get you started.

gdb, strace, ltrace Unable to Attach to a Process

Malware may use ptrace on itself to prevent other debuggers or tracing tools from attaching. Since only one process at a time can trace another, this technique effectively locks out tools like gdb, strace, and ltrace.

This anti-analysis trick is simple but effective–and it’s often one of the first signs that something is attempting to avoid inspection.

Use of Stagers

Stagers are a common technique used to bootstrap more complex payloads. Instead of delivering the full malware up front, attackers drop a small, simple binary whose sole job is to download and execute additional payloads.

This approach serves multiple purposes:

  • It reduces the on-disk footprint and makes initial detection harder.

  • The second-stage payload can be tailored to the specific victim.

  • The attacker can include sandbox checks or system profiling to avoid exposing their tooling to analysts or honeypots.

From a defender’s perspective, stagers can be frustrating because if only the initial binary is recovered, the true intent or functionality of the malware often remains a mystery.

Long sleep()

Some malware uses extended sleep() calls–sometimes lasting minutes, hours, or even days–before executing its actual payload. This technique helps it evade sandbox environments, which often monitor behavior for a short time (e.g., 10-15 minutes). By simply lying dormant during the observation window, the malware avoids analysis.

Rootkits

Rootkits exist primarily to hide processes, files, network activity, and other information from userspace tools and defenders. This is a deep and complex topic, well beyond the scope of this post, but it’s important to understand that if something feels “invisible” a rootkit may be involved.

Two quick signs that should raise red flags:

  • A process cannot be killed with SIGKILL. This suggests kernel tampering–no normal userspace process should be immune to signal 9.

  • Tools like unhide report hidden processes or inconsistencies between what /proc/ reports versus other system interfaces.

Conclusion

This post was a beast to write–and honestly, it still doesn’t cover everything. There’s always more to say when it comes to process analysis and hunting on Linux systems. The material here lays down a solid foundation, and I hope it helps defenders get better at spotting what doesn’t belong.

If you made it this far: thank you. It means you’re serious about digging deep, and that’s exactly what it takes to catch many attackers or to evade detection yourself.

No notes link to this note

Relevant Topics


Recent Posts