Linux Persistence: User Accounts

2021-06-27 DFIR linux persistence

Overview

Using the existing logon facilities on a *nix host is a popular and straightforward method used by attackers to persist on a system. Once an attacker has compromised a set of credentials, they can log in as if they were an authorized user.

How attackers obtain credentials

Attackers can obtain valid credentials using several methods:

  • Default credentials. A system may come with a valid set of credentials configured out of the box. This practice is common in consumer equipment and virtual machine appliances. A classic example of a default credential is admin/admin.

  • Phishing and credential harvesting sites. Users may be tricked into entering their credentials into malicious sites controlled by the attacker, revealing their password in plain text. Phishing emails may trigger code execution, allowing an attacker to place malware on a system.

  • Brute forcing and password spraying. An attacker may try many passwords against a single account or a known good password against multiple accounts.

  • Credential reuse. Once an attacker verifies that a set of credentials are valid, they may try to figure out where else these credentials will work. Data breaches often expose large sets of valid usernames and passwords. Attackers obtain these breaches then try to use these valid credentials against different sites. People often reuse their passwords, making this a potent form of attack.

  • Sniffing network traffic. Attackers can observe network traffic that contains credentials and harvest this data for later use.

  • Keystroke logging. Since users typically type in their passwords, attackers often deploy keystroke loggers to steal passwords and other sensitive information.

  • Clipboard logging. Often, people will store passwords within a document on their computer. Since strong passwords are long and difficult to type, they copy and paste their credentials into login prompts. Attackers may intercept the contents of their victim’s clipboards to harvest passwords and other sensitive information.

  • Passwords stored in memory. Plain-text passwords and hashes must be placed in memory as they are processed. If an attacker knows where these credentials reside in memory, they may be able to harvest them. A tool that uses these techniques is mimipenguin: https://github.com/huntergregal/mimipenguin

  • Social engineering. Attackers often employ social engineering attacks to obtain credentials. These attacks are sinister and have a high rate of success. The attacker uses a credible pretext and instills confidence in the victim using information they already know about them, tricking them into revealing sensitive information. A basic example is an attacker calling a victim, claiming to be an IT staff member performing maintenance, tricking them into providing their password.

  • Many more.

Attackers may add rogue accounts or modify an existing user once they have achieved the level of access with permissions to do so on a host.

Accounts can be created and manipulated using utilities such as passwd, adduser, usermod, or chsh. Since these tools are expected to be on a system and are legitimate administrative tools, creating detections based on the usage of these tools can be a challenge.

Attacking and defending user accounts has been a security problem for decades and I don’t see this going away any time soon.

If you are an incident responder responding to Linux systems, you should be aware of the topics covered here. User account abuse is a recurring theme with computer intrusions.

Below, I have documented information regarding user accounts I’ve found helpful in the context of security. Most of this information also applies to other UNIX-like operating systems. With the knowledge provided here, you will be able to hunt effectively for badness related to user accounts.

login

login is the process that initiates sessions on a system. login prompts for a username and password, checking /etc/passwd and /etc/shadow to see if they are valid.

The manual page for login(1) can be found here

login.defs

login.defs is the shadow suite configuration file. It is usually found at /etc/login.defs.

login.defs provides settings that provide granular control over sessions, logging, password requirements, and override user-defined settings.

One interesting setting within login.defs is the FAKE_SHELL variable. Setting this will override the user’s shell in /etc/passwd, executing FAKE_SHELL instead.

The manual page for login.defs(5) can be found here

utmp

utmp stores information about who is currently using the system. It is typically found at /var/run/utmp.

The manual page for utmp can be found here

Not all programs utilize utmp logging, so utmp can be an unreliable source of information. Quoted directly from the utmp manual page:

Warning: utmp must not be writable by the user class “other”, because many system programs (foolishly) depend on its integrity. You risk faked system logfiles and modifications of system files if you leave utmp writable to any user other than the owner and group owner of the file.

wtmp(5)

wtmp is much like utmp, however it stores historic login information, not just currently logged in users. This file usually resides at /var/log/wtmp.

btmp

btmp stores information regarding login failures. This file typically resides at /var/log/btmp.

lastlog(8)

lastlog shows recent user logins on a system. This information is stored at /var/log/lastlog.

User Management Commands

passwd

passwd provides an interface to manipulate user’s passwords on a system.

The manual page for passwd can be found here.

chfn

chfn allows users to change their full name and other GECOS fields.

chsh

chsh allows users to change their shell.

chage

chage allows users to view and manipulate their password expiry settings.

daniel@wildcat ~ % chage -l daniel
Last password change					: Jun 17, 2021
Password expires					: never
Password inactive					: never
Account expires						: never
Minimum number of days between password change		: 0
Maximum number of days between password change		: 99999
Number of days of warning before password expires	: 7

useradd

useradd adds users to the system. Compared to adduser, useradd is relatively primitive. useradd does not create home directories or transfer the contents of /etc/skel/ to the user’s home directory.

userdel

userdel removes users from a system.

usermod

usermod modifies users. You can change settings such as their shell, home directory, uid, gid, real name, and group membership.

groupadd

groupadd adds a group to the system.

groupdel

groupdel deletes a group from the system.

groupmod

groupmod modifies groups. You

w

w reports logged in users and what they are doing.

Example w output:

 10:03:04 up 14 days, 13:26,  1 user,  load average: 3.13, 2.08, 1.73
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
daniel   :0       :0               11Jun21 ?xdm?  20:28m  0.02s /usr/lib/gdm3/gdm

Attackers may alter w to hide their presence from administrators.

who

who displays a report of who is logged, where they logged in, where they logged in from, and when they logged in.

Example who output:

daniel   tty1         2021-06-26 14:38
daniel   pts/0        2021-06-26 14:37 (192.168.10.180)
daniel   pts/1        2021-06-26 15:47 (192.168.10.180)

lastlog

lastlog reports recent logins from each user, or a specific user.

This information is gathered from the /var/log/lastlog file

Example lastlog output:

Username         Port     From             Latest
root                                       **Never logged in**
daemon                                     **Never logged in**
bin                                        **Never logged in**
sys                                        **Never logged in**
sync                                       **Never logged in**
games                                      **Never logged in**
man                                        **Never logged in**
lp                                         **Never logged in**
mail                                       **Never logged in**
news                                       **Never logged in**
uucp                                       **Never logged in**
proxy                                      **Never logged in**
www-data                                   **Never logged in**
backup                                     **Never logged in**
list                                       **Never logged in**
irc                                        **Never logged in**
gnats                                      **Never logged in**
nobody                                     **Never logged in**
systemd-network                            **Never logged in**
systemd-resolve                            **Never logged in**
systemd-timesync                           **Never logged in**
messagebus                                 **Never logged in**
syslog                                     **Never logged in**
_apt                                       **Never logged in**
tss                                        **Never logged in**
uuidd                                      **Never logged in**
tcpdump                                    **Never logged in**
avahi-autoipd                              **Never logged in**
usbmux                                     **Never logged in**
rtkit                                      **Never logged in**
dnsmasq                                    **Never logged in**
kernoops                                   **Never logged in**
cups-pk-helper                             **Never logged in**
whoopsie                                   **Never logged in**
speech-dispatcher                           **Never logged in**
avahi                                      **Never logged in**
nm-openvpn                                 **Never logged in**
saned                                      **Never logged in**
hplip                                      **Never logged in**
colord                                     **Never logged in**
geoclue                                    **Never logged in**
pulse                                      **Never logged in**
gnome-initial-setup                           **Never logged in**
gdm                                        **Never logged in**
summerpalace                               **Never logged in**
systemd-coredump                           **Never logged in**
daniel           pts/3    192.168.1.100   Sat Jun 12 18:59:18 -0700 2021
clamav                                     **Never logged in**
postfix                                    **Never logged in**
sshd                                       **Never logged in**

last

last shows a similar report to lastlog, but gathers this data from /var/log/wtmp rather than /var/log/lastlog.

Example last output:

daniel   pts/1        192.168.10.180   Sat Jun 26 15:47   still logged in
daniel   tty1                          Sat Jun 26 14:38   still logged in
daniel   pts/0        192.168.10.180   Sat Jun 26 14:37   still logged in
reboot   system boot  4.4.0-186-generi Sat Jun 26 14:37   still running
daniel   pts/0        192.168.10.180   Sat Jun 26 14:37 - 14:37  (00:00)
daniel   pts/0        192.168.10.180   Sat Jun 26 14:36 - 14:37  (00:00)
daniel   pts/1        192.168.10.180   Sat Jun 26 11:49 - 11:55  (00:05)
daniel   tty1                          Sat Jun 26 11:48 - down   (02:48)
daniel   pts/0        192.168.10.180   Sat Jun 26 11:47 - 13:53  (02:05)
daniel   pts/0        192.168.10.180   Sat Jun 26 10:35 - 11:28  (00:53)
joseph   pts/0        192.168.10.180   Fri Jun 25 18:35 - 18:43  (00:08)
daniel   pts/0        192.168.10.180   Fri Jun 25 18:35 - 18:35  (00:00)
daniel   pts/0        192.168.10.180   Sat Jun 12 21:40 - 21:46  (00:05)
reboot   system boot  4.4.0-186-generi Sat Jun 12 21:40 - 14:37 (13+16:56)
daniel   pts/0        192.168.10.180   Sat Jun 12 21:38 - 21:40  (00:01)
daniel   pts/0        192.168.10.180   Sat Jun 12 18:51 - 18:51  (00:00)
daniel   pts/0        192.168.10.180   Sat Jun 12 18:51 - 18:51  (00:00)

wtmp begins Sat Jun 12 18:51:10 2021

lastb

lastb is like last, but shows failed login attempts.

The data used for this command is at /var/log/btmp

Example lastb output

UNKNOWN  tty1                          Sat Jun 26 14:38 - 14:38  (00:00)
UNKNOWN  tty1                          Sat Jun 26 11:48 - 11:48  (00:00)
joseph   ssh:notty    192.168.10.180   Fri Jun 25 18:44 - 18:44  (00:00)
joseph   ssh:notty    192.168.10.180   Fri Jun 25 18:43 - 18:43  (00:00)

btmp begins Fri Jun 25 18:43:59 2021

utmpdump

utmpdump parses arbitrary utmp and wtmp files. This can be useful for forensics.

Example utmpdump output:

Utmp dump of /var/log/btmp
[6] [18454] [    ] [joseph  ] [ssh:notty   ] [192.168.10.180      ] [192.168.10.180 ] [Fri Jun 25 18:43:59 2021 PDT]
[6] [18454] [    ] [joseph  ] [ssh:notty   ] [192.168.10.180      ] [192.168.10.180 ] [Fri Jun 25 18:44:03 2021 PDT]
[7] [01266] [tty1] [UNKNOWN ] [tty1        ] [                    ] [0.0.0.0        ] [Sat Jun 26 11:48:12 2021 PDT]
[7] [01254] [tty1] [UNKNOWN ] [tty1        ] [                    ] [0.0.0.0        ] [Sat Jun 26 14:38:02 2021 PDT]

utmpdump can be abused to remove or alter entries from a utmp or wtmp file. This is demonstrated below by altering this btmp file to show that the failed logins came from 192.168.2.73 rather than 192.168.10.180.

root@ubuntu1604:~# utmpdump /var/log/btmp
Utmp dump of /var/log/btmp
[6] [18454] [    ] [joseph  ] [ssh:notty   ] [192.168.10.180      ] [192.168.10.180 ] [Fri Jun 25 18:43:59 2021 PDT]
[6] [18454] [    ] [joseph  ] [ssh:notty   ] [192.168.10.180      ] [192.168.10.180 ] [Fri Jun 25 18:44:03 2021 PDT]
[7] [01266] [tty1] [UNKNOWN ] [tty1        ] [                    ] [0.0.0.0        ] [Sat Jun 26 11:48:12 2021 PDT]
[7] [01254] [tty1] [UNKNOWN ] [tty1        ] [                    ] [0.0.0.0        ] [Sat Jun 26 14:38:02 2021 PDT]
root@ubuntu1604:~# utmpdump /var/log/btmp >/tmp/utmpdump
Utmp dump of /var/log/btmp
root@ubuntu1604:~# vi /tmp/utmpdump
root@ubuntu1604:~# utmpdump -r </tmp/utmpdump >/tmp/altered-btmp
Utmp undump of /dev/stdin
root@ubuntu1604:~# utmpdump /tmp/altered-btmp
Utmp dump of /tmp/altered-btmp
[6] [18454] [    ] [joseph  ] [ssh:notty   ] [192.168.2.73        ] [192.168.2.73   ] [Fri Jun 25 18:43:59 2021 PDT]
[6] [18454] [    ] [joseph  ] [ssh:notty   ] [192.168.2.73        ] [192.168.2.73   ] [Fri Jun 25 18:44:03 2021 PDT]
[7] [01266] [tty1] [UNKNOWN ] [tty1        ] [                    ] [0.0.0.0        ] [Sat Jun 26 11:48:12 2021 PDT]
[7] [01254] [tty1] [UNKNOWN ] [tty1        ] [                    ] [0.0.0.0        ] [Sat Jun 26 14:38:02 2021 PDT]

The attacker can now replace /var/log/btmp with /tmp/altered-btmp.

su

su allows users to run commands as another user. This is often used to elevate to the root user to perform systems administration tasks.

Creating a group and setting permissions on su to deny non-administrative users the ability to use this command may make a lot of sense in your environment:

chgrp wheel /usr/bin/su
chmod 4750 /usr/bin/su

This configuration only allows users in the wheel group and the root user to utilize su.

sudo

sudo is similar to su, but provides much more granular configuration options. sudo is often abused by attackers due to over-permissive configurations. Often, administrative users are placed into the sudo group which has full root access, provided the user executing commands enters a valid password.

Attackers may abuse sudo by adding their user to a group with permissions to use sudo, or adding malicious entries to sudo’s configuration. This allows an easy method to elevate their privileges to root.

Configuration for sudo is typically found in the /etc/sudoers file and /etc/sudoers.d/ directory.

The manual for sudo can be found here.

The manual for sudoers can be found here.

Verifying passwd-related binaries

Attackers may manipulate the binaries highlighted in this section. Many of these binaries are suid-root or process sensitive information such as plain-text passwords, making them prime targets for backdoors and information stealing malware.

The deb and rpm package managers allow you to validate an installed package. This will quickly reveal whether or not one of these sensitive binaries has been tampered with.

Here is a quick and dirty script to verify these files on deb-based systems such as Ubuntu or Debian:

#!/bin/sh

packages=""
bins=""

for bin in "login passwd lastlog chfn chsh chage useradd userdel adduser addgroup deluser delgroup usermod w last who su sudo lastb utmpdump"; do
    bins="$bins $(which $bin)"
done


for bin in $bins; do
    x=$(dpkg -S "$bin" 2>/dev/null)
    if [ $? = 1 ]; then
	echo "$bin is not provided by a package"
	continue
    fi

    packages="$packages $(echo $x |cut -d : -f 1)"
done

packages=$(echo $packages |tr ' ' '\n' |sort -u)

echo
echo Verifying packages: $packages

for package in $packages; do
    dpkg -V $package
done

echo
echo "Done."

/etc/passwd

This file contains a line for each user. Each line is colon-delimited, with seven fields:

  • login name
  • password
  • User ID (UID)
  • Group ID (GID)
  • Username/Gecos
  • Home directory
  • Shell

An example user entry looks like this:

root:x:0:0:root:/root:/bin/sh

Example /etc/passwd file from a CentOS 7.9 machine:

root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
polkitd:x:999:998:User for polkitd:/:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
postfix:x:89:89::/var/spool/postfix:/sbin/nologin
chrony:x:998:996::/var/lib/chrony:/sbin/nologin
daniel:x:1000:1000:daniel:/home/daniel:/bin/bash

This passwd file is “shadowed.” Most UNIX-like operating systems have shadowed password files nowadays. You can tell that its shadowed because of the ‘x’ character in each user’s password field.

An old-fashioned, unshadowed passwd would look something like this:

root:GGWSv8kfHNYcY:0:0:root:/home/root:/bin/sh
nobody:PKWeKgk73qytM:1:1:nobody:/home/nobody:/bin/sh
bin:vfW/oY6SMPA3U:2:2:bin:/home/bin:/bin/sh
sys:Et8qJNKI/e1SU:3:3:sys:/home/sys:/bin/sh
www:WpocxhyXZqRIY:4:4:www:/home/www:/bin/sh
daniel:qzwur89Gb6JFg:5:5:daniel:/home/daniel:/bin/sh
heather:UXMKaAGFKVwpE:6:6:heather:/home/heather:/bin/sh
zachary:FVCspYztVvRfQ:7:7:zachary:/home/zachary:/bin/sh
nicole:WXcAaQAzW1nsU:8:8:nicole:/home/nicole:/bin/sh

This exposes the user hashes to cracking attacks. To mitigate this type of an attack, the shadow suite was developed.

/etc/shadow

The shadow file stores the hashes on modern Linux systems. This file has restrictive permissions, granting access only by the root user.

If an attacker obtains root on a system, they can steal the shadow file and attempt to crack the hashes held within.

The shadow suite introduces features to passwd such as setting password expiry dates and tracking when the password was last changed.

Here is an example /etc/shadow file, which corresponds with the passwd file shown above:

root:$6$UEQBBViTOmlcFpfM$klq9U1OMGSjCtO4UlMFm3wTUG9U3bJDapDxDJbn66CnLlkYktJhIkhaDBZ9VhgzZnvYdZt9Xz5SHKlm/51kPJ.::0:99999:7:::
bin:*:18353:0:99999:7:::
daemon:*:18353:0:99999:7:::
adm:*:18353:0:99999:7:::
lp:*:18353:0:99999:7:::
sync:*:18353:0:99999:7:::
shutdown:*:18353:0:99999:7:::
halt:*:18353:0:99999:7:::
mail:*:18353:0:99999:7:::
operator:*:18353:0:99999:7:::
games:*:18353:0:99999:7:::
ftp:*:18353:0:99999:7:::
nobody:*:18353:0:99999:7:::
systemd-network:!!:18775::::::
dbus:!!:18775::::::
polkitd:!!:18775::::::
sshd:!!:18775::::::
postfix:!!:18775::::::
chrony:!!:18775::::::
daniel:$6$5JRjKN8Qc2bhKcHf$.Tacg39KOwS1mMWIVuHHPMwSOkIEgvbFZGxPWgfBErAjIVapMbu5Q.B8Hz.zU8vw.nYeqMGHMPTthQLL8mR2i/::0:99999:7:::

The format of the shadow field is as follows:

$id$salt$hash

The id defines which hashing algorithm is used, and may be one of the following:

  • $1$ MD5
  • $2a$ Blowfish
  • $2y$ Blowfish
  • $5$ SHA256
  • $6$ SHA512

Salts prevent against rainbow table attacks and should be randomly generated.

Hashes can be generated with tools such as mkpasswd or Python. Typically, you should use the strongest available algorithm.

mkpasswd -m sha-512 password salt

python -c 'import crypt; print crypt.crypt("password", "$6$saltysalt")'

/etc/group

/etc/group defines user groups on a system. Its structure is as follows:

groupname:password:GID:comma,separated,users

Attackers may add themselves as members of a group which is able to execute administrative tools or read sensitive data such as sudo or mail.

/etc/gshadow

/etc/gshadow contains group passwords. Users can authenticate to groups with the newgrp command. Similar to setting a user’s password with the passwd command, you can set a group’s password with gpasswd.

The structure of a gshadow entry is as follows:

groupname:password:administrators,comma,separated:comma,separated,users

The last 10 lines of my gshadow file look like this. Notice that the foo group has a password set.

gdm:!::
sambashare:!::daniel
summerpalace:!::
systemd-coredump:!!::
daniel:!::
clamav:!::
postfix:!::
postdrop:!::
rdma:!::
foo:$6$gPs6k/PqVO./h$gvcjQ0YjiwgS3PmopM8VrtB4MprYrDUrq.EiV31xHj9nHfoNnYfLy2PHyUfMvVz639PiP3ALRU.GE5jvGGV4y0::

Here is an example shell session adding, setting a password for, and authenticating to a group:

daniel@wildcat ~ % sudo addgroup foo
Adding group `foo' (GID 1000) ...
Done.
daniel@wildcat ~ % sudo gpasswd foo
Changing the password for group foo
New Password:
Re-enter new password:
daniel@wildcat ~ % id
uid=1001(daniel) gid=1001(daniel) groups=1001(daniel),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),120(lpadmin),131(sambashare)
daniel@wildcat ~ % newgrp foo
Password:
daniel@wildcat ~ % id
uid=1001(daniel) gid=1000(foo) groups=1000(foo),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),120(lpadmin),131(sambashare),1001(daniel)

Account Lockouts

Users can be locked out using passwd -l. Under the hood, account lockouts work by prepending the existing password with a string. Typically this is an exclaimation point:

Example unlocked account’s entry in /etc/shadow:

hank:$6$pretendhash:0:99999:7:::

After running passwd -l hank, this entry will look like this:

hank:!!$6$pretendhash:18796:0:99999:7:::

To unlock this account, use passwd -u hank.

The status of an account can be viewed with passwd -S USERNAME. passwd -S outputs a line with seven space-delimited fields:

  • login name
  • lockout status (L = locked, NP = no password, P = password set)
  • date of last password change
  • minimum age
  • maximum age
  • warning period
  • inactivity period

Example output from passwd -S daniel:

daniel P 06/17/2021 0 99999 7 -1

The following script will lock all users with passwords set on the system except for the user running it (typically this will be root), and the user who initiated this command via sudo if applicable:

#!/bin/sh

# Lock out all users except $USER and $SUDO_USER

while read -r line; do
    user=$(echo "$line" | cut -d : -f 1)
    status=$(passwd -S "$user" 2>/dev/null)
    if [ $? = 1 ]; then
	echo "Unable to check status of user $user (are you root?)"
	exit
    fi
    lostatus=$(echo "$status" | awk {'print $2'})

    if [ "$lostatus" != "L" ]; then
	case "$user" in
	    $USER)
		echo "Skipping USER $USER"
		;;
	    $SUDO_USER)
		echo "Skipping SUDO_USER $SUDO_USER"
		;;
	    *)
		echo "Locking $user"
		passwd -l "$user"
		;;
	esac
    fi
done </etc/passwd

/etc/skel/ directory

Files within /etc/skel/ will be copied to newly-created users with the adduser utility. Attackers may modify existing files or add new malicious files to this directory to maintain persistence on a system.

One method an attacker might use is to add malicious code to /etc/skel/.profile. After adduser copies this file to the newly-created user’s home directory, it will be executed each time they log in.

Each file within /etc/skel/ should be accounted for. Validating the default files provided by your distribution may help to determine if these files have been altered.

#!/bin/sh

packages=""

find /etc/skel/ -type f -exec ls -l {} \;
echo

for file in $(find /etc/skel/ -type f); do

    x=$(dpkg -S "$file" 2>/dev/null)
    if [ $? = 1 ]; then
        echo "$file is not provided by a package"
        continue
    fi

    packages="$packages $(echo $x | cut -d : -f 1)"
done

packages=$(echo $packages | tr ' ' '\n' | sort -u)

echo
echo "Verifying packages: $packages"

for package in $packages; do
    dpkg -V $package
done

echo
echo "Done."

Running this on a test virtual machine shows that /etc/skel/malware.sh is not provided by a package

SSH authorized_keys files

Attackers may add their own keys to one or more users’ authorized_keys configuration files. This grants them access to the system even if the password is changed, provided that publickey authentication is enabled.

authorized_keys are typically found in the .ssh directory within a user’s home directory. They are plaintext files with each line representing a ssh key that grants access to the user in which it is configured.

Attackers may also use the more obscure authorized_keys2 file to avoid detections that only check for authorized_keys files.

Logs

Successful Console Authentications

Jun 26 11:48:26 ubuntu1604 login[1266]: pam_unix(login:session): session opened for user daniel by LOGIN(uid=0)
Jun 26 11:48:26 ubuntu1604 systemd-logind[906]: New session 3004 of user daniel.

Unsuccessful Console Authentications

Jun 26 11:48:08 ubuntu1604 login[1266]: pam_unix(login:auth): check pass; user unknown
Jun 26 11:48:08 ubuntu1604 login[1266]: pam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=/dev/tty1 ruser= rhost=

Successful SSH Authentications

Successful login using a password:

Jun 25 18:35:50 ubuntu1604 sshd[18278]: Accepted password for joseph from 192.168.10.180 port 55790 ssh2
Jun 25 18:35:50 ubuntu1604 sshd[18278]: pam_unix(sshd:session): session opened for user joseph by (uid=0)
Jun 25 18:35:50 ubuntu1604 systemd: pam_unix(systemd-user:session): session opened for user joseph by (uid=0)
Jun 25 18:35:50 ubuntu1604 systemd-logind[906]: New session 2840 of user joseph.

Successful login using a key:

Jun 25 18:34:31 ubuntu1604 sshd[18128]: Accepted publickey for daniel from 192.168.10.180 port 55774 ssh2: RSA SHA256:zG4bX5tSoJbac+5NmuNplF68nZ+IDymHHGhsLYnz6rQ

When you are reviewing successful login events, it may be helpful to ask yourself these questions:

  • Is this authentication normal?
  • Does the time of the login match the user’s typical working hours?
  • Is the source IP address associated with this login expected?
  • Is the source IP address listed in any bad reputation lists?
  • Is the username suspicious?

Unsuccessful SSH authentications

Jun 25 18:43:58 ubuntu1604 sshd[18454]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.10.180  user=joseph
Jun 25 18:43:59 ubuntu1604 sshd[18454]: Failed password for joseph from 192.168.10.180 port 55902 ssh2
Jun 25 18:44:03 ubuntu1604 sshd[18454]: Failed password for joseph from 192.168.10.180 port 55902 ssh2
Jun 25 18:44:05 ubuntu1604 sshd[18454]: PAM 1 more authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.10.180  user=joseph

Authentication logs should be monitored for excessive login failures. Excessive failures usually indicate a brute force attack or misconfigured software.

Fail2Ban

Fail2Ban monitors logs for failed authentications. If a threshold of login failures originating from a remote IP address over a set amount of time is met, Fail2Ban will block these IP addresses temporarily and generate an alert.

Usually, distributions will include Fail2Ban as the fail2ban package. Default settings may vary between distributions. Default settings typically block an IP after 5 failures within 10 minutes or so for 10 minutes. This allows a valid user to wait it out if they mistakenly type in the wrong password a few times without intervention from a sysadmin.

Configuration files are typically found in /etc/fail2ban/

Many unsuccessful logins followed by a successful login

The pattern of many unsuccessful logins, followed by a successful login indicates a successful brute force attack. If this pattern is observed, the affected user’s credentials have likely been compromised.

Password changed

Jun 16 19:45:23 wildcat passwd[291039]: pam_unix(passwd:chauthtok): password changed for daniel

Detecting User Account Abuse

Manual review

It helps to have a quick look at the passwd, shadow, and related configurations from time to time. Often, a keen sysadmin will spot misconfigurations that security tooling has missed.

Accounts with passwords

Accounts with a password set will have a hash in their shadow file.

Example:

daniel:$6$5JRjKN8Qc2bhKcHf$.Tacg39KOwS1mMWIVuHHPMwSOkIEgvbFZGxPWgfBErAjIVapMbu5Q.B8Hz.zU8vw.nYeqMGHMPTthQLL8mR2i/::0:99999:7:::

Commonly, accounts will have an asterisk or exclaimation mark in this field. This indicates that these accounts are disabled or locked.

Example:

bin:*:18353:0:99999:7:::
dbus:!!:18775::::::

You should verify if accounts with passwords should have a password set. Users such as bin, www-data, or nobody having a password set is suspicious.

This script will list accounts with passwords set

#!/bin/sh

while read -r line
do
    user=$(echo "$line" | cut -d : -f 1)
    status=$(passwd -S "$user")
    lostatus=$(echo "$status" | awk {'print $2'})

    if [ "$lostatus" != "L" ]; then
	echo $user
    fi
done </etc/passwd

Multiple UID=0 Accounts

Usually, root should be the only account with a UID of zero. All additional accounts with a UID of zero should be investigated.

Systems sometimes include a toor user as a secondary root user. These two users may be configured so that one or the other only has access to the console, or for system recovery purposes.

This one-liner will list the passwd entries for accounts with a UID of zero:

while read -r line; do user=$(echo $line |cut -d : -f 1); uid=$(echo $line|cut -d : -f 3); if [ $uid = 0 ]; then echo $line; fi; done </etc/passwd

Passwordless accounts

If an account is lacking data in the password field in /etc/passwd, they will be able to login without a password by simply pressing enter when prompted for a password.

It is common for shellcode and exploit code to add passwordless accounts to the passwd file:

Passwords may be deleted using passwd -d USERNAME

Short duration sessions

Attackers may execute one-liners using ssh rather than using an interactive shell and remaining logged in. This would make the attacker’s session not appear in diagnostic tools such as w or who.

Here are some example logs showing a 1 second session, indicating that an attacker is using this method:

Jun 26 11:55:28 ubuntu1604 sshd[23665]: Accepted publickey for daniel from 192.168.10.180 port 34762 ssh2: RSA SHA256:zG4bX5tSoJbac+5NmuNplF68nZ+IDymHHGhsLYnz6rQ
Jun 26 11:55:28 ubuntu1604 sshd[23665]: pam_unix(sshd:session): session opened for user daniel by (uid=0)
Jun 26 11:55:28 ubuntu1604 systemd-logind[906]: New session 3007 of user daniel.
Jun 26 11:55:29 ubuntu1604 sshd[23705]: Received disconnect from 192.168.10.180 port 34762:11: disconnected by user
Jun 26 11:55:29 ubuntu1604 sshd[23705]: Disconnected from 192.168.10.180 port 34762
Jun 26 11:55:29 ubuntu1604 sshd[23665]: pam_unix(sshd:session): session closed for user daniel
Jun 26 11:55:29 ubuntu1604 systemd-logind[906]: Removed session 3007.

Enable CLI logging

Enabling CLI logging provides analysts with the commands entered into a shell session. I feel that this is superior to monitoring shell history files for a number of reasons:

  • History files can be altered by the user.
  • History files may only be written when the shell exits cleanly. This leaves analysts with limited visibility into what is occurring within a session until the user logs out.
  • Attackers may disable history file logging by removing the HISTFILE environment variable, or changing it to /dev/null.
  • Attackers may set filesystem permissions and attributes which make writing to the history file impossible.

Monitoring these commands creates a lot of opportunity for defenders to fabricate detections.

Observing the cadence of commands ran within a session can help an analyst determine of these commands were part of an automated script, pasted into the session, or hands-on-keyboard activity.

A small a code snippet can be added to /etc/profile which effectively logs all commands input in a shell with syslog:

https://confluence.atlassian.com/confkb/how-to-enable-command-line-audit-logging-in-linux-956166545.html

Shells such as bash have syslog functionality, but most distributions package bash with this setting disabled. This blog post shows how to recompile bash with command line logging enabled:

https://dmfrsecurity.com/2019/10/31/building-a-hardened-shell-for-attack-defend-ctf-supremacy/

/etc/shells

/etc/shells contains a list of valid login shells. A user’s shell must be listed in this file in order to login.

Often, systems administrators will set a user’s shell to /bin/false or /bin/nologin to deny a user from having shell access to a host.

Attackers may add malicious entries to /etc/shells, or replace a valid shell with malicious software. For example, an attacker may replace bash with a malicious copy with a built in keylogger, or replace /bin/false with a shell.

An example /etc/shells file on Ubuntu 20.04:

# /etc/shells: valid login shells
/bin/sh
/bin/bash
/usr/bin/bash
/bin/rbash
/usr/bin/rbash
/bin/dash
/usr/bin/dash
/bin/zsh
/usr/bin/zsh

This script will hash each shell listed within /etc/shells. This information can be compared to a known good set of hashes or looked up on VirusTotal and similar services.

#!/bin/sh

for shell in $(grep -v '^#' /etc/shells); do
    sha256sum "$shell"
done

Example output:

wildcat# ./etc-shells-hash-shells.sh
64e48365207d0c19008ba7d53d75c0de3fcd5a1590e4c40fc69c677663fedc20  /bin/sh
04a484f27a4b485b28451923605d9b528453d6c098a5a5112bec859fb5f2eea9  /bin/bash
04a484f27a4b485b28451923605d9b528453d6c098a5a5112bec859fb5f2eea9  /usr/bin/bash
04a484f27a4b485b28451923605d9b528453d6c098a5a5112bec859fb5f2eea9  /bin/rbash
04a484f27a4b485b28451923605d9b528453d6c098a5a5112bec859fb5f2eea9  /usr/bin/rbash
64e48365207d0c19008ba7d53d75c0de3fcd5a1590e4c40fc69c677663fedc20  /bin/dash
64e48365207d0c19008ba7d53d75c0de3fcd5a1590e4c40fc69c677663fedc20  /usr/bin/dash
a6f2177402114fc8b5e7ecf924ffa61a2ac25bd347bc3370fb92e07b76e0b44c  /bin/zsh
a6f2177402114fc8b5e7ecf924ffa61a2ac25bd347bc3370fb92e07b76e0b44c  /usr/bin/zsh

Another useful trick is verifying that the hashes of each shell matches what was provided by its respective package. Here is an example script which verifies these using dpkg. This will not work on RPM-based distributions.

#!/bin/sh

packages=""
for shell in $(grep -v '^#' /etc/shells); do
    x=$(dpkg -S "$shell" 2>/dev/null |grep -v diversion)
    if [ $? = 1 ]; then
	echo "$shell is not provided by a package"
	continue
    fi

    packages="$packages $(echo $x |cut -d : -f 1)"
done

echo
echo Verifying packages: $packages

for package in $(echo $packages |tr ' ' '\n' | sort -u); do
    dpkg -V $package
done

Example output from this script is shown below. Notice that /bin/bash has been tampered with.

wildcat# ./etc-shells-dpkg-verify.sh
/usr/bin/bash is not provided by a package
/usr/bin/rbash is not provided by a package
/usr/bin/dash is not provided by a package
/usr/bin/zsh is not provided by a package

Verifying packages: dash bash bash dash zsh
??5??????   /bin/bash

Backup files

Some tools maintain backup files of the password and group databases before they make changes to them. These backup files have a trailing hyphen:

  • /etc/passwd-
  • /etc/shadow-
  • /etc/group-
  • /etc/gshadow-
  • /etc/subuid-
  • /etc/subgid-

These backups may be useful for recovery or forensic purposes if an attacker has manipulated any of the aforementioned files.

This one-liner will use diff on each of these backup files, revealing recent changes.

for x in /etc/*-; do echo $x; diff $(echo $x |tr -d -) $x; echo; done

Anti-Forensics

Deleting logs

Once an attacker has rooted a Linux system, they may simply delete the logs which contain information regarding their malicious logins.

Due to the relative ease of destroying logs once an attacker has gained access to a system, logs should be forwarded to an external server. Shipping logs elsewhere can be done by configuring a centralized syslog server or using a tool such as Splunk.

Recovering deleted log files may be possible with file carving tools such as PhotoRec, extundelete, or testdisk. I have had mixed

Log Cleaners

Deleting log files outright may raise the suspicions of an observant defender or trigger security alerts. As such, attackers have developed tools that selectively alter or delete matching entries to evade detection.

Some example log wiping tools can be found here: https://packetstormsecurity.com/UNIX/penetration/log-wipers/

No notes link to this note

Recent Posts