Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for Linux userland ELF snapshots and fuzzing #192

Merged
merged 42 commits into from
Apr 1, 2024
Merged

Add support for Linux userland ELF snapshots and fuzzing #192

merged 42 commits into from
Apr 1, 2024

Conversation

jasocrow
Copy link
Contributor

This is based on Kasamir123's pull request at
#102 plus some scripts in Snapchange for automatically setting up a Linux VM target.

The following improvements have been made as compared to Kasamir123's original pull request:

  • Fixed bug when calling mlockall, allowing us to remove page touching code
  • Code requires no custom #ifdefs in wtf
  • Linux snapshots work w/fuzzing via KVM. Kasamir123's code had some issues with gathering segment registers, and our updates fix these issues, allowing for KVM support
  • Kasamir123's code injects shellcode into the target process by overwriting code, but never restored the original code. We now restore the original code
  • Snapshotting is more streamlined, only taking a few manual steps once everything is configured
  • Some improvements from 0vercl0k's suggestions from ELF Snapshotting and Fuzzing 0vercl0k/wtf#102, like implementing raw2dmp in Python
  • Support for setting breakpoints on symbols in ELF targets plus use of symbols in fuzz harnesses
  • IDA script for generating coverage breakpoints list so that targets can be fuzzed with KVM
  • Target VM can run with HW acceleration enabled; Kasamir123's scripts for running the VM and taking a snapshot only worked with SW emulation
  • Works with recent Linux kernel versions

@jasocrow
Add support for Linux userland ELF snapshots and fuzzing.
807f8bc 
This is based on Kasamir123's pull request at
0vercl0k#102 plus some scripts in snapchange for
automatically setting up a Linux VM target.

The following improvements have been made as compared to Kasamir123's original
pull request:

* Fixed bug when calling mlockall, allowing us to remove page touching code
* Code requires no custom #ifdefs in wtf
* Linux snapshots work w/fuzzing via KVM. Kasamir123's code had some issues with
  gathering segment registers, and our updates fix these issues, allowing for
  KVM support
* Kasamir123's code injects shellcode into the target process by overwriting
  code, but never restored the original code. We now restore the original code
* Snapshotting is more streamlined, only taking a few manual steps once
  everything is configured
* Some improvements from 0vercl0k's suggestions from ELF Snapshotting and
  Fuzzing 0vercl0k#102, like implementing raw2dmp in Python
* Support for setting breakpoints on symbols in ELF targets plus use of symbols
  in fuzz harnesses
* IDA script for generating coverage breakpoints list so that targets can be
  fuzzed with KVM
* Target VM can run with HW acceleration enabled, Kasamir123's scripts for
  running the VM and taking a snapshot only worked with SW emulation
* Works with recent Linux kernel versions
@jasocrow jasocrow changed the title Add support for Linux userland ELF snapshots and fuzzing. Add support for Linux userland ELF snapshots and fuzzing Jan 24, 2024
@jasocrow jasocrow marked this pull request as ready for review January 24, 2024 17:49
@0vercl0k
Copy link
Owner

Damn, this is an amazing contribution and I'm looking forward to diving into it!

Thanks again for working on this and contributing back.

It's probably going to take me a minute to go through all of this 😅, but bear with me; I really want to get it in!

Cheers

@0vercl0k
Copy link
Owner

Okay I tried to run through the readme / instructions today and I might be hitting something unexpected; or I am doing something wrong.

The set-up part works fine, QEMU / Linux kernel sources are properly downloaded / compiled as far as I can tell.

I am starting the VM and it seems to diverge from the README; I hit a bunch of SIGUSR1 signals but I keep hitting continue and I see the VM booting so maybe that's fine:

user@pc~/wtf-linux-snapshot/linux_mode/crash_test$ ../qemu_snapshot/gdb_server.sh
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ../qemu_snapshot/target_vm/qemu/build/qemu-system-x86_64...
Starting program: /home/user/wtf-linux-snapshot/linux_mode/qemu_snapshot/target_vm/qemu/build/qemu-system-x86_64 -m 2G -smp 1 -kernel ../qemu_snapshot/target_vm/linux/ad

Program stopped.
0x00007ffff7fe3290 in _start () from /lib64/ld-linux-x86-64.so.2
Hardware assisted breakpoint 1 at 0x555555bcbdc0: file ../accel/kvm/kvm-all.c, line 2866.
Hardware assisted breakpoint 2 at 0x555555bad7c0: file ../accel/tcg/cpu-exec.c, line 893.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff7606640 (LWP 63564)]
[New Thread 0x7ffff6c03640 (LWP 63565)]
WARNING: Image format was not specified for '../qemu_snapshot/target_vm/image/bookworm.img' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.
[New Thread 0x7ffff60ff640 (LWP 63568)]

Thread 4 "qemu-system-x86" received signal SIGUSR1, User defined signal 1.
[Switching to Thread 0x7ffff60ff640 (LWP 63568)]
__futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55555682ac58) at ./nptl/futex-internal.c:57
57      ./nptl/futex-internal.c: No such file or directory.
(gdb) c
Continuing.

Thread 4 "qemu-system-x86" received signal SIGUSR1, User defined signal 1.
__futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55555682ac5c) at ./nptl/futex-internal.c:57
57      in ./nptl/futex-internal.c
(gdb) c
Continuing.

Thread 4 "qemu-system-x86" received signal SIGUSR1, User defined signal 1.
__futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55555682ac58) at ./nptl/futex-internal.c:57
57      in ./nptl/futex-internal.c
(gdb) c
Continuing.

Thread 4 "qemu-system-x86" received signal SIGUSR1, User defined signal 1.
__futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55555682ac5c) at ./nptl/futex-internal.c:57
57      in ./nptl/futex-internal.c
(gdb) c

Then, I start the gdb client:

user@pc:~/wtf-linux-snapshot/linux_mode/crash_test$ ../qemu_snapshot/gdb_client.sh
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ../qemu_snapshot/target_vm/linux/vmlinux...
Remote debugging using localhost:1234

This triggers a SIGUSR1 in the first shell:

         Starting systemd-update-ut… Record Runlevel Change in UTMP...
[  OK  ] Finished systemd-update-ut… - Record Runlevel Change in UTMP.
[    4.774337] audit: type=1400 audit(1707688808.601:6): avc:  denied  { watch_reads } for  pid=194 comm="(agetty)" path="/dev/tty6" dev="devtmpfs" ino=26 scontext=syst1
[    4.790192] audit: type=1400 audit(1707688808.616:7): avc:  denied  { checkpoint_restore } for  pid=194 comm="agetty" capability=40  scontext=system_u:system_r:kerne1

Debian GNU/Linux 12 linux ttyS0

linux login:
Thread 4 "qemu-system-x86" received signal SIGUSR1, User defined signal 1.
__GI___ioctl (fd=15, request=request@entry=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
36      ../sysdeps/unix/sysv/linux/ioctl.c: No such file or directory.
(gdb)

Again, if I hit continue I get this on the gdb-client shell:

Remote debugging using localhost:1234
Ignoring packet error, continuing...
warning: unrecognized item "timeout" in "qSupported" response
Ignoring packet error, continuing...
Remote replied unexpectedly to 'vMustReplyEmpty': timeout
add symbol table from file "a.out" at
        .text_addr = 0x555555555100
--Type <RET> for more, q to quit, c to continue without paging--
Removing regs.json file if it exists...not found
Traceback (most recent call last):
  File "./bkpt.py", line 16, in <module>
    FuzzBkpt(target_dir, break_address, file_name, sym_path=file_name)
  File "/home/user/wtf-linux-snapshot/linux_mode/qemu_snapshot/gdb_fuzzbkpt.py", line 233, in __init__
    gdb.Breakpoint.__init__(
gdb.error: No hardware breakpoint support in the target.
The program is not being run.

What gdb version should I use? Or maybe it's something else? My test VM is a Ubuntu 22.04.3 LTS.

Cheers

@jasocrow
Copy link
Contributor Author

Thanks for trying this out. I'll work on reproducing these issues and getting fixes in soon.

@jasocrow
Copy link
Contributor Author

Ok, I think both issues you ran into were related to gdb trying to process many SIGUSR1 signals. I've updated gdb_server.sh to ignore these signals and hopefully this will get things to work for you. I must have had something in my ~/.gdbinit file to ignore these signals previously when I did my testing, and that's why I didn't see this. Please let me know if you run into any more problems!

Confirmed latest changes work on a fresh Ubuntu 22.04.2 LTS VM:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.2 LTS
Release:	22.04
Codename:	jammy

@0vercl0k
Copy link
Owner

Awesome - I was able to generate a dump, thanks! Working through the rest of the readme now..

Cheers

@0vercl0k
Copy link
Owner

@jasocrow the README mentions that one of the limitation is that it doesn't support guests with more than 2GB of RAM - yet the start.sh script specifies 4G, my mem.dmp is 4GB and the pmemsave command seems to dump from 0 to 0xff'ff'ff'ff. Is the README not accurate then?

Also, I was wondering what's been your trace debugging workflow? Traces work fine but not sure how you've been symbolizing them?

Cheers

@0vercl0k 0vercl0k mentioned this pull request Feb 19, 2024
Closed
@jasocrow
Copy link
Contributor Author

jasocrow commented Feb 20, 2024 •
edited
Loading

@0vercl0k Awesome, thanks for trying it out and starting to change things up to your liking.

Good catch on the README. I guess 4 GB started working at some point and we didn't notice. Perhaps it started working when we moved to using a VM generated from the Snapchange scripts.

Ah yeah, we never implemented a symbolizer. We've been using tenet traces and loading them in IDA.

@jasocrow
Copy link
Contributor Author

@0vercl0k I can take a look at adding a symbolizer, but it might be a while before I can get to it.

@0vercl0k
Copy link
Owner

0vercl0k commented Feb 23, 2024 via email

@0vercl0k
Copy link
Owner

All right @jasocrow I think this is ready to go - take a look and if this looks good to you, let's merge this!

Cheers

@jasocrow
Copy link
Contributor Author

Everything looks good to me, happy if you're ready to merge. Thanks!!

@jasocrow
Copy link
Contributor Author

jasocrow commented Mar 29, 2024 •
edited
Loading

Oh, I just noticed one small thing, looks like the link to Kyle's site in linux_mode/README.md is not showing correctly.

Screenshot 2024-03-29 at 8 50 33 AM

@0vercl0k
Copy link
Owner

0vercl0k commented Mar 29, 2024 via email

@0vercl0k 0vercl0k merged commit a77e84e into 0vercl0k:main Apr 1, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jasocrow @0vercl0k