Full-disclosure: I wrote about this on my previous blog https://dmfrsecurity.com/2020/02/25/finding-bad-with-package-managers/ and most of the content of this post is copied from this post. The dmfrsecurity.com blog will likely be decommissioned in the near future.

Introduction

Most Linux distributions use package managers (like RPM or dpkg) to streamline software installation and updates, making it easier to manage dependencies. A lesser-known but crucial feature provided by package managers is the ability to verify the integrity of the files installed by a package. This post will explore how to use package managers to uncover malware and persistence mechanisms on Linux systems.

Common Places to Look

Some common places and anomalies to look for persistence using the package manger are as follows:

• Files within common $PATH directories (/bin, /usr/bin, /sbin, /usr/sbin, …) having differing hashes than what was provided by the package manager. This indicates that the files have been tampered with.

• Shared object files (libraries), typically in /lib, /usr/lib, and so on having differing hashes than provided by the package manager. This also indicates tampering.

• Files in the aforementioned directories NOT provided by a package manager. This indicates that they are not a legitimate part of the underlying system and placed by a third party. These files may be legitimate, but it is definitely suspicious.

• Configuration files in /etc with differing hashes have been modified. Making configuration changes to a system is common. However, using a package manager’s verification functionality can quickly reveal to an incident responder exactly which files have been changed, narrowing the scope of their investigation.

• Checking the paths and hashes of currently running processes can be a fast way to unearth malware.

dpkg (.deb)-based Systems

Debian, Ubuntu, Mint, Kali, and several other Linux distributions use dpkg to manage packages.

Verifying a Package

Scenario: when responding to an incident, you suspect that the OpenSSH binaries have been tampered with. You can use the package manager to test this suspicion.

First, you can look up which package provides a particular file using dpkg -S:

% dpkg -S /usr/bin/ssh
openssh-client: /usr/bin/ssh

Next, verify the openssh-client package:

% dpkg -V openssh-client
??5??????   /usr/bin/ssh

As indicated by the output ??5??????, the ssh binary has a differing hash than what the package provider has provided, which alludes to this binary being tampered with. Under normal circumstances, dpkg -V will have no output if all the package’s files check out as clean.

For further context, the output is explained in dpkg’s man page:

--verify-format format-name
      Sets the output format for the --verify command
      (since dpkg 1.17.2).

      The  only currently supported output format is rpm, which
      consists of a line for every path that failed any check.  The
      lines start with 9 characters to report each specific check
      result, a ‘?’  implies  the check  could not be done (lack of
      support, file permissions, etc), ‘.’ implies the check passed,
      and an alphanumeric character implies a specific check failed;
      the md5sum verification failure (the file  contents  have
      changed) is denoted with a ‘5’ on the third character.  The
      line is followed by a space and an attribute character
      (currently ‘c’ for conffiles), another space and the pathname.

Verifying All Packages

To verify all packages, use the following one-liner (as root!):

for pkg in $(dpkg -l | grep ^ii | awk {'print $2'}); do dpkg -V $pkg; done

debsums

debsums is a tool that can validate all packages similar to the one-liner above. It is easier and less cumbersome than the aforementioned one-liner, and has additional options that may be helpful.

Unfortunately debsums is typically not installed by default, so you may need to install it:

# apt install debsums

A deep dive into debsums is beyond the scope of this article, but if you want to use it you should check out its manual page.

Files Not Provided by a Package

To find files not provided by a package, the following one-liner may be useful:

# find /bin/ -exec dpkg -S {} \; 2>&1 |grep "no path found matching"
dpkg-query: no path found matching pattern /bin/backdoor

Dpkg Checksum Storage Location

Checksums for installed dpkg packages are located at /var/lib/dpkg/info/*.md5sums

The format of these files is plain-text and self-explanatory:

# cat /var/lib/dpkg/info/at.md5sums
507610d547edc3e7b98777cd1d5ad084  lib/systemd/system/atd.service
2c0733b064a62cb0c4cbbbd531465120  usr/bin/at
03265a0c11d5b695529f629b1e860eb1  usr/bin/batch
50fd51201dda267285da1d0eadd50736  usr/sbin/atd
38c61cea8340b1d2191535464235fd6a  usr/share/doc/at/Problems
fb98b5905a0ec916dd6277268f86dd10  usr/share/doc/at/README
...snip...

Parsing these files may be useful for writing scripts that go above and beyond the functionality provided by dpkg or debsums.

It is also noteworthy that attackers can easily modify these files, adding the path and MD5 hashes of their malware to an existing package’s .md5sums and .lists files. This will trick the verification feature of dpkg or debsums into believing that the malware is supposed to be there. As such, using package managers verification output should be taken with a grain of salt. It may offer an easy win, however its output can be somewhat easily manipulated.

If You Suspect the md5sums Files Were Tampered With

As noted in the previous section, an attacker can add bogus entries to dpkg databases to mask their activity.

It may be useful to compare the contents of these files against those of a known-good system or to curate your own lists of known-good hashes. Remember–defenders should have the home field advantage. Preparation is the most important step in incident response.

Verifying Running Processes

Checking whether or not the running processes on a system are included by a package manager may be a useful way to uncover actively running malware on a host:

for i in $(find /proc -maxdepth 2 -type l -name exe |xargs readlink |sort |uniq); do echo -n "$i: "; dpkg -S $i; done |grep "no path found"

Example output:

dpkg-query: no path found matching pattern /var/cache/.X11/.js8Ze8fm9R

RPM-based Systems

RHEL, CentOS, SuSE and other Linux distributions are RPM-based. RPM also provides similar functionality for file verification.

Verifying an RPM Package

Scenario: you suspect that /bin/date has been tampered with.

First, query the RPM database to determine which package provides /bin/date:

# rpm -q --whatprovides /bin/date
coreutils-8.22-23.el7.x86_64

Next, verify the coreutils package:

# rpm -V coreutils

Similar to dpkg -V, rpm -V will show no output if it does not detect tampering. When tampered files are discovered, rpm outputs what was changed in a format similar to dpkg -V:

# cp /bin/date .
# echo garbage >>/bin/date
# rpm -V coreutils
S.5....T.    /usr/bin/date
# cp ./date /bin/date
cp: overwrite ‘/bin/date’? y
# rpm -V coreutils
.......T.    /usr/bin/date

In the example above, /bin/date has some garbage data appended to it, breaking its file hash. rpm -V output shows that its size (S), MD5 hash (5), and timestamp (T) differ than what was provided by the package manager.

After restoring /bin/date with its unaltered copy, the timestamp (T) is wrong.

It is worth noting that rpm provides more detail in its verification functionality than dpkg.

Verifying all packages

Verifying all packages with rpm is simple:

rpm -Va

Files Not Provided by a Package

The following one-liner will check the /bin directory for files not provided by rpm:

find /bin/ -type f -exec rpm -q --whatprovides {} \; |grep "is not owned by any package"

You should change the path in the one-liner to reflect where you are hunting.

Where Are RPM Checksums Stored?

/var/lib/rpm/ houses a set of Berkeley DB files that contain RPM package metadata. I was unsuccessful when trying to hex edit these files to tamper with file hashes, but I’m sure it is possible

Exracting RPMs Without Installing Them

It may be useful to examine the contents of an RPM file without installing it. An example scenario would be discovering a suspected malicious rpm file and opting into extracting its contents for further analysis.

This is accomplished with the rpm2cpio utility:

$ mkdir scratch
$ cp whatever.rpm scratch
$ cd scratch
$ rpm2cpio whatever.rpm | cpio -idmv

Honorable Mentions

dpkg and rpm aren’t the only package managers out there, but they are by far the most common. Tracking down every package manager format would be a fool’s errand due to the sheer number of Linux distributions out there. That being said, here are a few honorable mentions.

Slackware

Slackware packages are gzipped tarballs. As far as I can tell, Slackware packages do not include checksum information.

Arch Linux

Arch packages are xzipped tar files. These .txz files contain a gzipped .MTREE file which includes sha256 hashes of files provided by each package. paccheck appears to be a tool able to verify these packages.

FreeBSD pkg Files

FreeBSD packages are xzipped tar files. The +MANIFEST file included within each package contains a JSON object containing sha256 hashes of each file included in the package. It appears that you can verify packages with the following command:

pkg check –checksums.

FreeBSD includes the mtree utility, which creates a database of hashes. These databases can be copied to another system or read-only media and compared to what is actually on disk at a later date.