Linux - Parity release harden Page Cache #1553
Open
+192
−80
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…tection, and improve compatibility with older kernel versions
…t inode page cache walk during dumps not showing output with dumping to file.
…ut, we test both modes, listing and dumping.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In the context of #1516 and related issues, this PR addresses several bugs in the Page Cache API code and its associated functions.
NOTE1: This requires the fixes from #1543 (linked list object extensions) and #1545 (mountinfo). Please DO NOT test the samples below using only the changes from this branch!
NOTE2: From now on, the
linux.pagecache.InodePagesplugin will no longer display inode page details when using the--dumpargument. This change helps prevent redundant scanning of the inode page cache, significantly improving processing time, especially when dealing with large files, such as those mentioned in the related issues.fix: #1526 , #1527
Issue 1526
Sample: broken_rhel_load_as_2.lime
Sample: Ubuntu_Server_x64_22.04_Auth.zip.lime
Issue 1527
1 - Page out of the bounds issue
Fixes the issue reported by @atcuno in #1527 related to handling page out-of-bounds scenarios.
2 - Sample: unshared_fds_network_connection.zip.lime
I'm not sure how the
/proc/kcorefile ended up being written in that case. It's likely due to all these recent changes, however, in my tests the inode for the/proc/kcorefile in that sample shows zero cached pages, meaning that using--dumpdoesn't result in any file being written.$ ./vol.py -f ../unshared_fds_network_connection.lime linux.pagecache.Files | grep 9ec482a94e30 SuperblockAddr MountPoint Device InodeNum InodeAddr FileType InodePages CachedPages FileMode AccessTime ModificationTime ChangeTime FilePath ... 0x9ec482298000 /proc 0:21 4026532026 0x9ec482a94e30 REG 34359734275 0 -r-------- 2024-08-20 18:57:13.620000 UTC 2024-08-20 18:57:13.620000 UTC2024-08-20 18:57:13.620000 UTC /proc/kcoreRegarding the dump size limit: I can add an argument to restrict the file size in a future PR. However, this shouldn't be an issue in most cases since the output is a sparse file, which is supported by most modern filesystems across all major operating systems.
The
actualfile size (not theapparent/sparsedsize reported by the OS) will only include the pages found in memory.Even if we dump all existing inodes from the entire Page Cache, the total
actualoccupied size, by definition, cannot exceed the system RAM size. Even in the hypothetical (and absurd) scenario where 100% of RAM is used by the Page Cache, the resulting sizes are still too small tobreaka filesystem.Naturally, if you have just 100MB of available space and the
actualwritten size is 16GB, you'll run into issues.However, this can happen with any command in the operating system, i.e.:
cp /some/big/file /some/small/fs/.Curiously,
cpdoes not have an option to limit the size of the data being written.Additionally,
lswill report only the apparent/sparsed size, not the actual size. You can even have a sparse file larger than the space available on disk. For instance:$ df -h . Filesystem Size Used Avail Use% Mounted on /dev/nvme1n1p1 469G 93G 353G 21% /mnt/other_diskThere's 353GB available, let's create a 512GB sparse file:
Is the filesystem full or broken?
It doesn't take up any space. To read the
actualfile size, you can usedu.On the other hand, files like
/proc/kcorewill never report pages available in the Page Cache since it's not a normal file backed on disk blocks but a special file in the proc filesystem. It directly represents in-memory content, so caching is unnecessary and irrelevant to its purpose.Having said that, I've added lazy initialization for the output sparse file. This ensures that if the
linux.pagecache.InodePagesplugin cannot to find any valid pages to dump, it will still generate the file, however, in such cases, the file will be created but remain empty with 0 actual bytes and not truncated to the inode size.3 - Sample: 20220822195350.zip
Effectively, the file contains 55007 pages which is approx ~215MB.
$ ./vol.py -f ../20220822195350_data.lime linux.pagecache.Files | grep 888223baee58 0x88822de2b000.0/ 259:1 4822135g0x888223baee58 REG 55006 55007 -rw-r--r-- 2022-08-16 20:31:07.000000 UTC 2022-08-16 20:31:07.000000 UTC 2022-08-22 19:06:06.114498 UTC /var/cache/yum/x86_64/2/amzn2-core/gen/primary_db.sqliteIt's odd it reports more pages available than the actual file size. I've already debugged and verified this, see my notes later.
$ time ./vol.py -f ./20220822195350_data.lime linux.pagecache.Inode --inode 0x888223baee58 --dump ... ERROR volatility3.plugins.linux.pagecache: Page out of file bounds: inode 0x888223baee58, inode size 225304576, page index 55006 real 9m48.316sDid it work as expected?
Is the number of available pages vs the inode pages a bug?
Using the respective debug symbols for that kernel, I debugged it using GDB:
So, it's not a bug.
4 - Sample: 4_4_0-176_206-4_4_0-176-generic.lime
Did it work as expected?
5 - Sample: blackhat_2022_linux_poc_logkey.lime
$ ./vol.py -f ./blackhat_2022_linux_poc_logkey.lime linux.pagecache.Files | grep 9a590549a728 0x9a591ef48800.0/ 8:1 941200ng0x9a590549a728 REG 20638 18328 -r--r----- 2022-03-31 14:38:42.696390 UTC 2022-03-31 14:38:42.888389 UTC 2022-03-31 14:38:42.888389 UTC /home/rk/debrkdev/20220331093842/memory/data.limeOk.. this is a memory dump.
However, this sample and inode has many errors.
6 - Sample: blackhat_2022_poc_ssh_keylogger_test.lime
$ ./vol.py ./blackhat_2022_poc_ssh_keylogger_test.lime linux.pagecache.Files | grep 9a591cbfd1a8 0x9a591ef48800 / 8:1 944566 0x9a591cbfd1a8 REG 116894 52105 -r--r----- 2022-03-31 15:53:38.761565 UTC 2022-03-31 15:53:39.953558 UTC 2022-03-31 15:53:39.953558 UTC /home/rk/debrkdev/20220331105338/memory/data.limeAnother memory dump.
Again, this inode have a lot of issues with the page cache.
$ wc -l /tmp/blackhat_2022_poc_ssh_keylogger_test.err 2130 /tmp/blackhat_2022_poc_ssh_keylogger_test.err $ tail /tmp/blackhat_2022_poc_ssh_keylogger_test.err WARNING volatility3.framework.symbols.linux.extensions: Cached page at 0xe5dbc05a1040 has a mismatched address space with the inode. Skipping page WARNING volatility3.framework.symbols.linux.extensions: Cached page at 0xe5dbc05a1080 has a mismatched address space with the inode. Skipping page WARNING volatility3.framework.symbols.linux.extensions: Cached page at 0xe5dbc05a10c0 has a mismatched address space with the inode. Skipping page WARNING volatility3.framework.symbols.linux.extensions: Cached page at 0xe5dbc05a1100 has a mismatched address space with the inode. Skipping page WARNING volatility3.framework.symbols.linux.extensions: Cached page at 0xe5dbc05a1140 has a mismatched address space with the inode. Skipping page WARNING volatility3.framework.symbols.linux.extensions: Cached page at 0xe5dbc05a1180 has a mismatched address space with the inode. Skipping page WARNING volatility3.framework.symbols.linux.extensions: Cached page at 0xe5dbc05a11c0 has a mismatched address space with the inode. Skipping page ERROR volatility3.framework.symbols.linux: Invalid cached page at 0x9a59064deda0, aborting WARNING volatility3.framework.interfaces.plugins: File inode_0x9a591cbfd1a8.dmp could not be written: Invalid cached page at 0x9a59064deda0, aborting ERROR volatility3.plugins.linux.pagecache: Error dumping cached pages for inode at 0x9a591cbfd1a8To ensure everything is correct, I debugged the issue using the
crash utility. As you can see, it triggers the exact same error for that inode, causing the listing to abortAnyway, it still partially recovered the file from memory.
7 - Sample: downloads_memory_20220701103641.lime
$ ./vol.py -f ./downloads_memory_20220701103641.lime linux.pagecache.Files | grep 9fa87bf1b1a8 0x9fa836498000 / 254:1 261475 0x9fa87bf1b1a8 REG 438272 320321 -r--r----- 2022-07-01 14:36:41.535180 UTC 2022-07-01 14:36:53.315186 UTC 2022-07-01 14:36:53.315186 UTC /home/redacted/downloads/20220701103641.zip$ cat /tmp/downloads_memory_20220701103641.err ERROR volatility3.framework.symbols.linux: Radix Tree node 0x9fa803033240 height 41 exceeds max height of 12 WARNING volatility3.framework.interfaces.plugins: File inode_0x9fa87bf1b1a8.dmp could not be written: Radix Tree node 0x9fa803033240 height 41 exceeds max height of 12 ERROR volatility3.plugins.linux.pagecache: Error dumping cached pages for inode at 0x9fa87bf1b1a8But an empty output file.
Let's use the
crash toolto check if this is a bug in our code.crash> files -p 0xffff9fa87bf1b1a8 INODE NRPAGES ffff9fa87bf1b1a8 320321 radix_tree_node at ffff9fa87bf1b330 ... files: height 41 is greater than maximum radix tree height index 12