Introduction

A vital feature required by most operating systems is the ability to run programs and scripts when the machine is booted, rebooted, or enters certain types of states (hibernation, network changed, …).

Linux provides a few different mechanisms to do this. Some examples are SysV (rc.local, rcX.d) , systemd, and init.d:

• systemd units can be configured to run malicious software at boot, or on a schedule similar to cron.

• init scripts residing in /etc/init.d/ are executed by the init process shortly after the system is booted.

• SysV style init scripts are found in /etc/rc*. This includes /etc/rc.local, a machine-specific set of commands that get ran on the system when runlevels are changed.

  This is an “obsolete” method of systems configuration, but its still widely used on Linux and Unix-like systems.

This is not an exhaustive list of startup scripts. Systems may have multiple sets of these systems running in parallel with eachother, or something different or bespoke to a system or environment.

Startup Script Precedence

Most startups systems will have a mechanism that enable services to be started in a particular order. This is necessary for certain software due to it being dependent on other configurations to be applied or services to be running.

An example of this would be ssh requiring the network to be configured before it starts listening for incoming connections; if no network is available, it will fail to start.

Security software may have a higher precedence so it is able to seek out malware before it has a chance to run. Conversely, malware may attempt to run at a higher precedence to avoid detection.

Startup Scripts Used as Persistence

Using startup scripts as a persistence mechanism is straightforward: if an attacker achieves root level access on a system or the script’s filesystem permissions are grossly misconfigured, the attacker can add whatever they like to the startup system. Whatever they choose to add will get ran when the system boots.

Like other forms of persistence, attackers will often blend in with the systems they compromise, making it look like their malware is supposed to be there. Sometimes the changes can be subtle, where they add an innocent-looking command to a service that gets ran when the system starts (cron, ssh, ntp, networking, …).

Attackers may also obfuscate their additions to startup scripts making them harder to analyze.

Hunting For Malicious Startup Scripts

Some indicators that a configuration file within a startup system may contain malicious content:

• The file was created within the timeline of an event.

• The file was not introduced to the system by a package manager, or its contents are different than what was provided by the package. This can be checked with most package managers with their built in verification features.

• Files within these directories are conventionally introduced to the system via a package manager, configuration management software, or tools specifically designed to manipulate the configurations of the startup system (systemctl, .

  Monitoring to writes to these files from strange software can be an effective way to detect malicious activity.

• Strange parent/child relationships when executing startup scripts.

  Startup scripts tend to be roughly uniform for the same software within an environment. For example, a startup script responsible to start a service such as ntpsec should have predictable child processes. If additional or different child processes occur under this script, it may have been tampered with.

• Startup scripts with random, similar/typoed, or nonsensical names.

  To avoid being subject to simple static detection rules, malware may randomize the name of the script or use a set of human-readable but nonsensical names.

  For example they may name their malicious startup script ssh-service instead of the authentic sshd or ssh, or name it sssh (extra s), which is nonsensical, but easy for a lazy or untrained eye to miss.

• Startup scripts with non-uniform permissions, comment headers, etc.

  Look for stuff out of the ordinary. If all of the files in the directory are owned by root:root, and one is root:SOMEOTHERGROUP, or has different permissions than everything else, it deserves a bit of scrutiny.

  Have a look at other scripts on the system. Often they have similar layouts, comment headers, etc. If there’s one script that doesn’t match the same layout and patterns as the other scripts, it is suspicious.

Defense

• Monitor changes to startup script files (AIDE, Tripwire, and other similar tools).

• Find files not provided by a package or that have been modified within directories containing startup script configurations using the system’s package manager’s verification feature.

Conclusion

Defenders should be aware of how common startup systems such as systemd, init.d, and SysV operate and how to hunt for additions to these files.

Attackers should know how these systems work and how they can blend their malware with these systems.