Defanging Linux LKM Rootkits With cleanup_module()

I recently came across a Github repository that demonstrated a technique to degrade the integrity of EDR software by directly invoking cleanup logic common to Loadable Kernel Modules (LKMs).

After digging in and testing this for myself, I found the technique both sound and technically straightforward.

In short, every(?) LKM is defined by a struct module entry, which includes metadata about the module including things such as its name, status, versioning information, etc. One particularly interesting member of struct modules is exit, a pointer to the module’s destructor function. Regardless of what the original function is named in the module’s source code, it gets exported as a symbol symbol named "cleanup_module".

Many (most?) LKM-based rootkits and EDR tools implement self-unhooking logic in their cleanup routines. This is intentional–being able to unload gracefully is useful during testing or legitimate removal. It wouldn’t be good for the adversary to be able to simply remove the rootkit or security software. To avoid easy removal by attackers (or defender), these modules often include anti-removal mechanism like increasing their reference count or unlinking themselves from the kernel’s module list.

I thought that this was worth further investigation, so I made my own Github repository with PoCs demonstrating this technique:

https://github.com/droberson/hammertime

This post summarizes what I learned while researching and testing this technique and expands on the details beyond what’s in the GitHub repository.

Testing

I don’t currently have access to any commercial Linux EDR software, so I wasn’t able to test this technique against those directly. That said, it should work on many LKM-based products that implements a proper cleanup_module() function.

Instead, I focused on common rootkits. Functionally, rootkits and EDR share a lot of technical overlap–their goals differ, but the underlying mechanisms are often similar. Anecdotally, the two I’ve encountered most often in the wild are Diamorphine and Reptile, so that’s where I started.

Reptile

I started by testing this technique against Reptile.

Unfortunately, Reptile has a long-standing bug in its cleanup routine that causes some hosts to freeze when the module is unloaded. My test VMs consistently locked up–whether I used rmmod or invoked cleanup_module() directly. After some digging, I confirmed this is a known issue:

https://github.com/f0rb1dd3n/Reptile/issues/103

This is important: the technique worked, but Reptile’s cleanup implementation is broken. These are exactly the kinds of edge cases that make low-level work dangerous and problematic. Any time you mess around with the kernel, especially with modules like rootkits or EDRs, you run the risk of crashes, lockups, or undefined behavior.

It’s a reminder that you should always test these kinds of techniques in isolated, controlled environments–not on production systems or in the middle of an operation. Even widely-used rootkits can be fragile.

Diamorphine

After struggling with Reptile, I moved on to testing Diamorphine.

Unlike Reptile, Diamorphine handled a direct call to its cleanup_module() function cleanly. No crashes, no instability–it decloaked itself and removed its hooks as expected. This made it a much more reliable testing target, and a solid proof that the technique works under the right conditions.

Finding cleanup_module() on Cloaked Rootkits

At first, I cheated.

Since I had full control over the rootkits during testing, I just modified them to print out their THIS_MODULE address, cleanup function addresses, and other information that I needed on load. I also added debugging printk()s on the relevant cleanup functions, as neither announced that they were unloaded in any meaningful way.

These debugging messages made it trivial to locate the addresses of cleanup_module() functions and to validate that the technique was working.

Additionally, I could decloak the rootkits at will and view the relevant symbols in /proc/kallsyms if needed.

That’s fine for labbing and development, but won’t cut it in the real world. In an actual incident response scenario, threat actors aren’t usually dumb enough to leave default settings in place on these rootkits. While I’ve occasionally seen off-the-shelf Reptile and Diamorphine used with no changes, most TAs tweak it–changing symbol names, magic signal values, etc.

Additionally, the module will almost certainly be cloaked if it is being actively used. As such, I needed a more generic way to find hidden struct module entries and their corresponding exit functions.

Writing a Memory Scanner

An answer to my problem of finding cloaked modules and their corresponding cleanup_module() addresses was breakitdown.ko: an LKM that walks through kernel memory looking for things that look like a legitimate struct module entry.

Originally, I brute-forced the entire address space. That worked–but it also froze the system for several minutes while it worked. Not great. So I refined it.

I wrote a quick script to grab the lowest and highest cleanup_module addresses from /proc/kallsyms, giving me a tighter scan range. In practice, the relevant addresses tended to be a little before or after these calculated values. To accommodate this, I added a pad value to widen the scan range:

awk '/cleanup_module/ && $1 ~ /^ffffffffc/ { print "0x"$1 }' /proc/kallsyms \
  | sort -n \
  | awk '
    NR==1 { low=strtonum($1) }
    { high=strtonum($1) > high ? strtonum($1) : high }
    END {
      pad = 0x10000
      printf "scan_start=0x%x scan_end=0x%x\n", low - pad, high + pad
    }'

This massively improved performance. Still, finding real module structures meant that I needed to know their precise layout.

Finding struct module Offsets

At first, I tried to reverse engineer the offsets by reading the kernel headers (include/linux/module.h) and manually accounting for #ifdefs and field alighment.

This worked, but it was labor-intensive and error prone. After several hours of hexadecimal math and frustration, I remembered that debugging symbols exist.

On Ubuntu, you can enable the ddebs repositories, install a kernel with debugging symbols, and then use pahole (from dwarves) to dump the actual layout of struct module on your particular kernel:

apt install dwarves
pahole -C module /usr/lib/debug/boot/vmlinux-$(uname -r)

Example output (trimmed for brevity):

struct module {
	enum module_state          state;                /*     0     4 */

	/* XXX 4 bytes hole, try to pack */

	struct list_head           list;                 /*     8    16 */
	char                       name[56];             /*    24    56 */
        ...
	void                       (*exit)(void);        /*   776     8 */
	atomic_t                   refcnt;               /*   784     4 */
	...

Heck yeah! Exact offsets. No more guesswork.

Putting it All Together

With a working memory scanner, a more focused range of memory to scan, and the correct offset, breakitdown.ko was working:

[67237.735235] breakitdown: scanning for 'struct module' entries from ffffffffc0198800 to ffffffffc0754000...
[67237.735351] breakitdown: found module 'it_rx_addrs' at ffffffffc01a9dc0, state=0, refcnt=0, cleanup=0000000000000000
[67237.735356] breakitdown: found module 'cast' at ffffffffc01aa240, state=0, refcnt=0, cleanup=0000000000007372
[67237.735360] breakitdown: found module '_count' at ffffffffc01aa480, state=0, refcnt=0, cleanup=0000000000000064
...
...
...
[67237.778197] breakitdown: found module 'diamorphine' at ffffffffc0684040, state=0, refcnt=1, cleanup=ffffffffc0682650
[67237.796281] breakitdown: done - 94 likely 'struct module' entries found

This produced some false positives, but that’s fine–what mattered is that breakitdown.ko was able to spot Diamorphine even while it was cloaked. From there, passing the cleanup function address to hammertime.ko successfully ran the rootkit’s destructor and decloaked it.

Unrelated, but Cool (and Useful)

While poling around, I stumbled on something interesting.

Diamorphine successfully hides itself from both /proc/kallsyms and lsmod, which is what you’d expect from a cloaked rootkit, but it still appears under /sys/module:

# ls -l /sys/module | grep diamorphine
drwxr-xr-x 5 root root 0 Apr  4 19:08 diamorphine

This makes sense–/sys/module is a sysfs interface exposing kernel module parameters and metadata, and it doesn’t seem to be affected by Diamorphine’s hiding tricks (at least in the version I tested).

Of course, this isn’t a silver bullet. If the attacker changes the name to something more inconspicuous (which they almost certainly will), its presence won’t be as obvious as a directory named diamorphine. There’s still something you can do though.

You can compare the contents of /sys/module against the output of lsmod and flag on modules that aren’t listed by lsmod but still have a refcnt file–meaning they were likely loaded as LKMs.

Here is a quick and dirty script that does exactly that:

for mod in /sys/module/*; do
  name=$(basename "$mod")
  if [ -f "$mod/refcnt" ] && ! lsmod | awk 'NR > 1 { print $1 }' | grep -qx "$name"; then
    echo "$name"
  fi
done

This isn’t perfect–it won’t catch built-in modules (which won’t have a refcnt file), and attackers could still go a step further to hide themselves from /sys/module. But it is fast, easy, and may yield results.

Protecting Against This Technique

If you’re writing kernel modules for defensive (or offensive) purposes and want to resist tampering via cleanup_module() or similar techniques, here are some strategies to consider:

• *Avoid obvious names - Don’t name your module something obvious like rootkit, reptile, diamorphine, etc. Use generic names that don’t scream “Look at me! I’m a rootkit!”

• Zero out sensitive fields - After initialization, overwrite THIS_MODULE->exit and other fields in struct module to prevent clean unloads:

  THIS_MODULE->exit = NULL;
  

• Don’t expose methods to uncloak, uninstall, or otherwise disable the software - Rootkits (and EDR) often include “magic” signals, syscalls, files, or other such triggers that shut themselves down, disable functionality, decloak themselves, etc. These can be a liability. If you don’t need an uncloak or uninstall function, don’t include one.

Conclusion

Rootkits are notoriously tricky to deal with–by design. They aim to hide, persist, and resist removal. The techniques covered here offer a small but powerful foothold for incident responders. In the right circumstances, the ability to trigger internal functions like such as cleanup_module() can be a game changer.

Hopefully this exploration proves useful and inspires others to build on it. There’s plenty of room to expand upon, refine, and adapt these ideas to real-world response workflows and more sophisticated threats.