Linux Persistence: Startup Scripts

2024-11-10 DFIR CTF linux persistence systemd SysV init startup script

Linux Persistence: Startup Scripts

A key feature of most operating systems is the ability to automatically run programs or scripts during startup, reboot, or state transitions (e.g., resuming from hibernation, connecting to a network).

Linux offers several mechanisms to handle this, including SysV-style init scripts, systemd units, and legacy init.d scripts:

  • systemd unit files can run programs at boot or on a schedule, similar to cron.

  • /etc/init.d scripts are invoked by the init process during early boot.

  • SysV-style scripts in /etc/rc* (including /etc/rc.local) are used to run machine-specific commands when runlevels change.

Although some of these systems are considered obsolete or archaic, they’re still widely used–especially on older, embedded, or minimalist systems. It’s also common for multiple init mechanisms to coexist on the same machine.

Startup Scripts as Persistence

Using startup scripts as a persistence mechanism is straightforward: if an attacker achieves root level access on a system or the script’s filesystem permissions are grossly misconfigured, the attacker can add whatever they like to the startup system. Whatever they choose to add will get ran when the system boots.

Like other forms of persistence, attackers will often blend in with the systems they compromise, making it look like their malware is supposed to be there. Sometimes the changes can be subtle, where they add an innocent-looking command to a service that gets ran when the system starts (cron, ssh, ntp, networking, …).

Attackers may also obfuscate their additions to startup scripts making them harder to analyze.

Startup Script Internals and Mechanics

While modern Linux systems have largely adopted systemd, many environments–especially legacy or embedded systems–still use SysV-style init scripts.

These older systems use nubered script names and symbolic links to determine execution order and behavior. Understanding how these work is essential when analyzing persistence mechanisms or strange boot-time behavior.

Execution Order

In SysV-style systems, script execution is determined by directory structure and alphabetical ordering. Scripts in /etc/rc*.d/ are named like S10networking, S20apache, etc., where the S means “start” and the number determines priority–lower numbers run first.

This ordering is critical for dependency management. For example:

  • S10networking brings up network interfaces.

  • S20sshd starts the SSH daemon, which depends on networking.

Attackers and administrators alike may exploit this ordering:

  • Attackers may name their scripts S05auditd to beat a security tool to boot.

  • Or S99rescue to ensure it launches last.

Enabling and Disabling Scripts

Scripts in /etc/init.d are linked to /etc/rc*.d/ directories based on the system’s current runlevel. These determine whether a service starts or stops during boot.

To manage this easier, Linux systems offer helper tools:

  • RedHat-based systems: chkconfig

    chkconfig --add apache
    chkconfig apache on
    
  • Debian-based systems: update-rc.d

    update-rc.d apache2 defaults
    update-rc.d apache2 disable
    

These utilities abstract away manual symlink creation and help standardize service behavior.

Why This Matters

Startup order has security implications:

  • Defenders may want antivirus, EDR, and auditing tools to launch before user services–if malware is launched before security agents, they won’t see it launch or have a chance to block it.

  • Attackers may try to

    • Start malware before logging/EDR tools to avoid detection.

    • Delay execution until late in the boot process.

    • Mimic legitimate services in naming (S20sshd, S21syslogd, …)

Inconsistent or oddly-ordered scripts can be a strong indicator of tampering.

Hunting For Malicious Startup Scripts

Fortunately, startup scripts often stand out when something’s not quite right. Below are common heuristics defenders can use to identify suspicious entries.

  • Suspicious timing:

    • The script or service was created or modified within the timeline of an intrusion or suspicious event.

    • Timestamps suggest tampering (e.g., unusually recent mtime)

  • Not installed by a package manager:

  • Written by an unknown process:

    • Legitimate init files are typically written by trusted utilities (systemctl, chkconfig, update-rc.d, configuration management, etc.).

    • Monitor for unknown or unexpected processes writing to init files and directories such as /etc/init.d, /etc/rc.local, or systemd unit files.

  • Unexpected process relationships:

    • Legitimate services send to spawn predictable child processes. Example: ntpd should not launch a reverse shell.

    • Unexpected children or complex shell pipelines may indicate tampering.

  • Deceptive or clumsy naming:

    • Slight misspellings of real services are easy to gloss over: ssshd, cr0n, htttpd, sys1ogd, …

    • Nonsensical or random-looking names: svc-mux50001, H5dv8zf9k1, …

    • Names that look legitimate but don’t match system context or behavior: S35mysqld when your system shouldn’t be running a database service.

  • Permission or ownership anomalies:

    • Most init files are owned by root:root and have the same permissions.

    • Look for files with different owners or group memberships.

    • Look for files with non-uniform permissions compared to other files in the directory.

  • Odd content or formatting:

    • Malicious scripts may omit standard comment headers.

    • Large blocks of obfuscated or base64-encoded payloads.

    • Not follow uniformity with other legitimate scripts.

Defense Strategies

  • Monitor changes to files in startup-related directories using tools like AIDE, Tripwire, or real-time audit frameworks

  • Leverage package manager verification (rpm -V, dpkg -V, etc.) to detect files that were added or modified outside of official channels.

  • Watch for unexpected writes or script execution–especially from non-root users or unknown processes.

Proactive monitoring and occasional manual review go a long way in detecting tampering.

Conclusion

Startup scripts remain a low-effort, high-reliability persistence mechanism.

Defenders must understand how these systems work to spot tampering, while attackers rely on that very same knowledge to blend in. Whether you’re investigating a compromise or building hardening strategies, knowing how Linux handles startup initialization is critical.


No notes link to this note