9. Linux

14. Preservation, acquisition, analysis

Linux workstations: LUKS, dm-crypt and layered storage

Linux storage is layered by design: a LUKS volume unlocks to a device-mapper crypt mapping, which may be a physical volume for LVM, which contains logical volumes, which hold filesystems, which hold container files and virtual disks that repeat the pattern. An examiner who images only the top layer, or only the bottom, has half the picture. The method is to enumerate every layer on the live system, acquire at the layers that matter, and expect the chain to continue inside the filesystem.

Live and unlocked

1. Preserve power. Connect mains. Do not log out, suspend or reboot; do not close a laptop lid, which commonly suspends.
2. Document the state as in section 3, then isolate physically (section 4). Note that NetworkManager and similar agents log the link-down event; that is the expected footprint.
3. Acquire memory (section 7). LiME must be built for the exact running kernel; loading it is a kernel-mode interaction and its hash and the load time are recorded. AVML needs no module where the kernel exposes physical memory through the usual interfaces. On a virtual machine, dump from the hypervisor instead.
4. Enumerate the storage stack from a shell, using tools from your own media where practical (the target's binaries may be modified). Each command is read-only; each is still recorded with its output:
   • lsblk -f and lsblk -o NAME,TYPE,FSTYPE,SIZE,MOUNTPOINT,UUID: the block device tree, showing crypt and LVM layers and what is mounted where.
   • mount, findmnt and /proc/mounts: every mounted filesystem, including bind mounts, overlay mounts, FUSE mounts (VeraCrypt, gocryptfs, sshfs) and mounts that hide a directory's earlier contents.
   • dmsetup ls and dmsetup table: device-mapper mappings and their targets (crypt, linear, snapshot). With authority, dmsetup table --showkeys prints the master key of each active crypt mapping in hexadecimal; recording it is the most reliable way to retain access to a LUKS or plain dm-crypt volume once the machine is off. Treat the output as key material: record it to your own media, hash the record, and handle it under the same custody as the evidence.
   • cryptsetup status <mapping>: the cipher, key size, underlying device and offset of an unlocked mapping. cryptsetup luksDump <device>: the LUKS header, key slots in use, key derivation parameters and, for LUKS2, the tokens; it does not reveal keys.
   • pvs, vgs, lvs and lvdisplay: LVM physical volumes, volume groups, logical volumes, thin pools and snapshots.
   • losetup -a: loop devices, which map files to block devices; a loop device backed by a file in the user's home directory is a container.
   • blkid: filesystem and encryption signatures on every block device, including those not mounted.
   • docker ps -a, podman ps -a, virsh list --all, and a listing of /var/lib/docker, /var/lib/libvirt/images and the user's virtual machine directories: containers and virtual machines, each of which holds a filesystem the host does not show.
5. Acquire a decrypted copy at the block level of each unlocked volume that matters, by imaging the mapped device (/dev/mapper/<name>, or the logical volume path) with dc3dd or dd to your own media. This is a plaintext image of the volume including unallocated space, and is far more useful than a file-level copy. Where a filesystem is on a FUSE or file-level mount (a VeraCrypt volume mounted by its own tool exposes a block device; gocryptfs does not), take a file-level copy with metadata preserved and record that it is file-level.
6. Acquire the physical storage beneath after shutdown (section 8), giving the ciphertext image with the LUKS headers intact.
7. Hash both acquisitions, record which is plaintext and which ciphertext, and record the mapping between them (which physical device and offset the decrypted image corresponds to).

Powered off

Image the physical device through a write blocker. Identify the layers from the image: blkid and cryptsetup luksDump against a loop-mounted, read-only working copy report LUKS volumes and their headers; a partition with no signature and uniform high entropy may be plain dm-crypt (no header, all parameters must be known) or a VeraCrypt volume. Recovery sources are the general ones in section 12, plus Linux-specific ones: LUKS key files referenced in /etc/crypttab on an unencrypted boot volume, keys held in a TPM or on a hardware token (systemd-cryptenroll), passphrases in configuration management or provisioning systems, a LUKS header backup made by the administrator, and a master key recorded from a live session as above. Unlock a working copy of the header to a new mapping, image the mapping, hash it, and never write to the copy the header came from.

Nested encryption

Assume the chain continues until you have shown it does not. A representative sequence is LUKS on the partition, LVM inside it, an ext4 filesystem on a logical volume, a VeraCrypt container file in a home directory, and an encrypted 7-Zip archive inside the container. Each layer has its own key, its own recovery route and its own derived artefact with its own hash, and a report that decrypted the outer layer and stopped must say so. On the live system, the enumeration above finds the layers that are currently unlocked; on the image, find the rest by entropy analysis of large files and unallocated regions (section 17), by the presence of the tools (VeraCrypt, gocryptfs, cryptomator, encrypted archive utilities) and their configuration and history, and by loop and FUSE mounts in the shell history and the journal. A gocryptfs or similar directory-level encryption shows as a directory of files with random names and high-entropy content.

Linux filesystem examination notes

• ext4. Fully supported by The Sleuth Kit. The journal records metadata changes and can be listed and extracted with the journal tools; deleted inodes retain their timestamps and, until reused, their extent trees, which give the block chain for direct recovery. Inode metadata carries four timestamps (access, modification, inode change, creation) and the inode change time cannot be set through normal file operations, so a modification time earlier than the change time is a sign of timestamp manipulation. The orphan inode list in the superblock records inodes that were unlinked while still open; debugfs on a read-only copy reads all of this.
• XFS. Limited support in the open-source toolkits. Use xfs_db in read-only mode and xfs_logprint against a working copy to read metadata and the log; xfs_repair -n reports what a repair would do without doing it, and is run only on a copy. XFS reuses inode numbers and space aggressively, so deleted-file recovery is weaker than on ext4.
• Btrfs. Copy-on-write with subvolumes and snapshots. Earlier versions of files survive in snapshots and, briefly, in superseded tree copies; btrfs subvolume list and btrfs inspect-internal on a read-only mounted copy enumerate them, and btrfs restore recovers files from a copy without mounting it. Toolkit support is limited; record which tool interpreted what.
• Unallocated and slack space are examined as in section 18, with the caveat that SSDs with discard enabled return zeros for trimmed blocks.
• Extended attributes and ACLs. getfattr -d -m - <file> lists all extended attributes, which can hold arbitrary data (section 17) as well as SELinux labels and capabilities; getfacl lists access control lists. Unusual attributes, immutable or append-only flags (lsattr), setuid binaries in odd places and world-writable directories are all worth listing.
• Hidden files and unusual mounts. Dot-files and dot-directories are conventional, not hidden; look for them in unexpected places (a dot-directory under /var/tmp, a file whose name is a single space). Compare /etc/fstab with what was actually mounted: a filesystem mounted over a populated directory hides the directory's contents until it is unmounted, and the image shows both.
• User and system activity. Shell history for each user (bash, zsh, fish, and the Python, MySQL and less histories), remembering that history can be disabled or edited and that its absence is itself notable; cron entries in the system and per-user crontabs and systemd timers; SSH artefacts (authorised keys, known hosts, private keys, client configuration, and authentication logs or the journal for logins and key use); temporary directories (/tmp, /var/tmp, /dev/shm, the last being memory-backed and lost at power-off); and the logs under /var/log and in the systemd journal, which is read from a copy with journalctl --file or --directory.
• Containers and virtual machines. Container layers under the runtime's storage directory are ordinary directories and are examined as such; container volumes and virtual disk images (qcow2, vmdk, vdi, raw) are filesystems in their own right, to be converted or mounted read-only from a copy and examined through the same process, including their own encryption.

Example. A live Linux workstation shows, from lsblk, a LUKS partition unlocked as a crypt mapping holding an LVM group with root and home volumes. losetup -a shows a loop device backed by a 40 GB file in the user's home directory, and mount shows it mounted as ext4 at a directory under /mnt, which does not appear in /etc/fstab. The examiner records the dm-crypt master key with dmsetup table --showkeys, images the home logical volume and the loop device separately, then powers off and images the disk. Analysis of the loop-backed filesystem finds a VeraCrypt container file, which is not unlocked; the report lists it as an encrypted container of stated size, not decrypted, contents unknown.