Linux Persistence: atd and at Jobs

On Linux and Unix-like systems, the atd daemon allows users to schedule one-time command execution. It is similar to cron, but instead of recurring jobs, at runs a command once at a specified time in the future.

This system was far more common in a couple of decades ago in the days of actual shared, multi-user Unix systems. A sysadmin might need to reboot a machine or restart a service but see a dozen users actively working. Rather than booting everyone off the system, they could defer the action with at, scheduling it for midnight or some off-peak time. Users, too, might queue up resource-heavy jobs to run after hours out of courtesy to others on the system.

These days, most systems are single-user in practice– always on, and far more powerful–so this kind of task scheduling is less common. As a result, at is no longer installed by default on many mainstream Linux distributions, though it is still available in most package repositories.

Like any system that rund scheduled commands–cron, systemd timers, and so on–at can be abused. And because it’s a bit more obscure, it might fly under the radar of less experienced defenders. It’s not uncommon for incident responders to overlook at entirely, especially if they assume the system doesn’t use it. Attackers simply don’t care whether or not you use a feature and will abuse it if they can.

This technique is documented in the MITRE ATT&CK Framework as T1053.002: Scheduled Task/Job: At.

at Basics

The at system is relatively simple. It provides commands to add, list, and remove one-time jobs. These jobs are executed by the atd daemon, which runs in the background. While any user may schedule their own jobs, the root user can view and manipulate jobs created by others.

Jobs are stored on disk, typically under /var/spool/cron/atjobs/, or /var/spool/at/spools, depending on the distribution and implementation. The job files themselves are plain text, containing the scheduled time, environment variables, and the shell commands to execute.

Determining if At is Installed

On many modern systems, atd may not be installed by default. You can check for its presence with:

# Check if the atd daemon is running
ps aux | grep atd

# Check if the atd binary exists
whereis atd

# Check for atd documentation
whatis atd
man atd
...

If at is missing, you can usually install it via the system’s package manager. The package is usually named at.

Listing Queued at Jobs

You can list jobs for the current user with atq:

atq

This displays job IDs, execution time, and queue information.

To list the raw contents of jobs using cat (helpful for incident response):

cat /var/spool/cron/atjobs/*

Adding An at Job

To add a job, pipe a command into at with a time expression:

echo some cool command | at now + 5 minutes

You can also add multiple line commands in interactive mode:

at 12:00 AM
at> echo "pwned" >> /tmp/pwned
at> <EOT> # Press Control-D to finish

The time formats are flexible, allowing users to specify exact or relative times. See the manual for more information.

Removing An at Job

Jobs are removed with atrm followed by the job number from atq:

atq

# Delete job id '3':
atrm 3

The root user can remove any user’s jobs:

# List all users' jobs
sudo atq

# Remove job id '5':
sudo atrm 5

Using As Persistence

Because at jobs only execute once, they aren’t ideal for long-term persistence. However, an attacker can work around this limitation by having the job resechedule itseif at the end of its execution–effectively creating a recurring task through repetition.

The general strategy is simple: add a line to the malicious script that queues another at job for the future.

Example script:

% cat /tmp/at.sh
#!/bin/sh
touch/tmp/pwned_$(date +%s)
echo "/tmp/at.sh" | at now + 5 minutes

Install the recurring job:

chmod +x /tmp/at.sh
echo "/tmp/at.sh" | at now + 5 minutes

This creates an endless loop, running /tmp/at.sh repeatedly, every 5 minutes.

Detecting atd Abuse

Detecting abuse of the at system can be tricky due to its one-off nature and generally low visibility, However, there are a few things defenders can watch for:

• Monitor the at spool directory for writes

  File creations in these directory are likely new jobs being created. Modifications are jobs being altered.

• Search jobs for suspicious commands

  A quick and dirty strategy to find malicious jobs is to grep for common patterns of abuse such as file downloads with curl, wget, or netcat, obfuscation with base64 or other encoding tools, or running script interpreters such as python or perl. This list isn’t exhaustive and may yield false positive results, but illustrates the general idea.

  grep -irE "curl|wget|bash|nc|python|perl|base64" /var/spool/cron/atjobs/
  

• Monitor at command usage

  Monitor the at command for suspicious use with auditd, shell command line logs, or EDR tools. In most environments, the at command is rare enough to warrant scrutiny.

  Unfortunately, upon review of the source code provided in the at implementation shipped by Debian, it does not appear to log failed attempts to use the at system. As such, visibility into denied use is limited. (see check_permission() function in perm.c and at.c)

• Monitor child processes of the atd process for suspicious commands

  atd running network tools, interpreters, or other malicious commands may indicate at job abuse.

Hardening at

Removing at if not Needed

If you don’t need the at system, the best option is to remove it entirely with your package manager:

# Debian/Ubuntu
apt remove at

# RHEL/Fedora
dnf remove at

/etc/at.allow and /etc/at.deny

If you must keep it, restrict access to only trusted users using the /etc/at.allow and /etc/at.deny files. These files define who can schedule at jobs:

• If /etc/at.allow exists, only the users listed within are permitted to use at.

• If /etc/at.allow does not exist, then /etc/at.deny is consulted–any users listed within are blocked from using at.

Many systems will provide a /etc/at.deny file with entries like nobody, daemon, bin, or other system and service accounts. You should double check that untrusted users (especially those with shell access) aren’t able to queue jobs unnecessarily.

Discretionary Access Controls

Alternatively, you can restrict access of _at by:

• Creating a dedicated group (e.g., atusers)

• Changing the ownership and permissions of the at-related commands and spool directory.

• Using tools like chown, chgrp, and chmod to enforce group-based access. Notice: at is typically suid/sgid so it may write to the spool directory.

This method is highly environment-specific and may require some experimentation, so it’s left as an exercise to the reader if they desire this type of control.

Conclusion

Despite its potential, the at system is rarely observed in real-world intrusions. Anectodally, I have encountered it far more often in CTFs and academic environments than in real-world incidents. That said, its low visibility and relative obscurity on modern systems make it an attractive option for attackers targeting neglected, misconfigured, or legacy Linux environments

As with all forms of persistence, defenders should be aware of the tools available to both users and adversaries–and be ready to investigate the obscure, not just the obvious.