TUI Support Infrastructure #1620

ehennenfent · 2020-03-03T23:53:01Z

Adds preliminary support for the TUI to Manticore. Lots of limitations and shortcomings right now. In no particular order:

The log capture doesn't work with multiprocessing. Logs created in different processes (unsurprisingly) don't fire the callback that inserts them into the server's buffer. Fixing this will probably require using the multiprocessing module's concurrency mechanisms.
The state capture requires acquiring the global state lock. It'd be better to use a plugin system to keep track of a state model over time, but since plugins run in the worker processes this would require us to use the locked context. For now, this way is simpler.
Only captures state IDs and what list they're in - nothing about current status, instruction counts, or termination messages. That's a consequence of looking at the state lists instead of the states themselves.
Very synchronous - Uses a socket server that creates a new connection every time the TUI polls for an update. Ideally, we'd open a pipe and just stream log/state messages through that. That said, I highly doubt a dozen-ish TCP packets per second are going to have any substantial performance impact, but we should profile this to make sure.

* Update worker thread for server creation * Add necessary files for TUI connectivity * Add necessary files for TUI connectivity * Update MonitorWorker * Update protocol * Blacken * Update setup.py dependencies * Remove state debugging messages * Update setup.py to build protobuf protocol upon install * Remove previously generated state_pb2.py * Change subprocess.Popen to subprocess.check_output * Remove extraneous output * First attempt at fixing protobuf installation It might work, it might not. We'll let the CI sort it out. * Can't forget the f-string * Error on missing protoc * Disable auto-generation of protobuf file * Ignore pb2_errors * Disable monitor start See if this makes the EVM tests pass Co-authored-by: Eric Hennenfent <ecapstone@gmail.com>

Should make it possible to track movements between state lists

This will break the state merging plugin (but I'll fix it eventually)

From these commits codecov/codecov-action@ebea5ca codecov/codecov-action@49c86d6

Haven't been able to figure out why, but somehow other loggers get "stuck" at this high verbosity and the integration tests try to print out the values of every single register.

ehennenfent · 2020-08-28T20:17:45Z

This builds heavily on top of #1775, so that will need to be merged first, but pending that it's ready for review.

ekilmer

Looks good so far! I'm hoping we can have the protobuf compiler generate the serialization file during installation though. Thoughts?

manticore/core/manticore.py

manticore/core/state_pb2.py

manticore/core/worker.py

ehennenfent

This has been sitting open for a while, so to make it easier to review I added a couple of comments pointing to parts that I think might be controversial.

ehennenfent · 2021-01-04T23:28:11Z

tests/native/test_manticore.py

@@ -134,20 +134,3 @@ def test_integration_basic_stdin(self):
        else:
            self.assertTrue(a <= 0x41)
            self.assertTrue(b > 0x41)
-
-
-class ManticoreLogger(unittest.TestCase):


@feliam I moved this from this file over to test_logging.py in order to avoid the problem where the log level gets stuck at the max level. I seem to recall you having dealt with something similar, but I can't remember where. Did you come up with a better solution?

ehennenfent · 2021-01-04T23:47:03Z

manticore/core/worker.py

+            server.dump = dump_states  # type: ignore
+            server.serve_forever()
+    except OSError as e:
+        # TODO - this should be logger.warning, but we need to rewrite several unit tests that depend on


@feliam Do you think it's worth fixing this in this PR, or should we do a more general rewrite of the tests separately?

A lot of our tests check for exact stdout output, which can be brittle if you want to add new logging messages. Maybe it'd be better to check that the lines we're looking for are contained in the output the correct number of times?

ehennenfent · 2021-01-27T16:17:06Z

Per our discussion at the dev meeting, since this has been sitting unchanged for roughly a month and no one has had the cycles to fully review it, I'm going to merge it as-is (and take the flak if it breaks).

* master: Syscall specific hooks (#2389) TUI Support Infrastructure (#1620)

* master: (43 commits) Syscall specific hooks (#2389) TUI Support Infrastructure (#1620) Fix coveralls upload (#2387) docs: fix simple typo, straigth -> straight (#2381) Attempt to allow symbolic balances from the start (#1818) Fix state.cpu.PC member (#1825) Bump black and mypy (#1824) Manticore 0.3.5 (#1808) Fix yices timeout argument (#1817) Detect default solver (#1820) Ignore Gas Calculations by Default (#1816) native/cpu/x86: Add support for CPUID EAX=80000000h (#1811) Change types.FunctionType=<class 'function'> (#1803) Fix test regressions (#1804) State Introspection API (#1775) Fix EVM account existence checks for selfdestruct and call (#1801) Add partial implementation of sendto syscall (#1791) crytic-compile: use latest release (#1795) Update gas metering for calls to empty accounts (#1774) Fix BitVec with symbolic offset and fix TranslatorSmtlib.unique thread safety (#1792) ...

* master: Removed use of global solver from Native Memory (#2414) Support to use boolector as the SMT solver (#2410) Update CI and suggest to use pip3 instead of pip (#2409) Expressions use keyword-only arguments for init (#2395) Use Slots on all Expression objects (#2394) Allow double-adding exact same config option (#2397) Don't run OSX tests on PR Attempt to Fix solc Installation MacOS (#2392) Syscall specific hooks (#2389) TUI Support Infrastructure (#1620) Fix coveralls upload (#2387) docs: fix simple typo, straigth -> straight (#2381) Attempt to allow symbolic balances from the start (#1818) Fix state.cpu.PC member (#1825) Bump black and mypy (#1824)

* master: (22 commits) Fix the generation of EVM tests (#2426) Disabled EVM events in testcases by default (#2417) added proper timeouts for cvc4 and boolector (#2418) Removed use of global solver from Native Memory (#2414) Support to use boolector as the SMT solver (#2410) Update CI and suggest to use pip3 instead of pip (#2409) Expressions use keyword-only arguments for init (#2395) Use Slots on all Expression objects (#2394) Allow double-adding exact same config option (#2397) Don't run OSX tests on PR Attempt to Fix solc Installation MacOS (#2392) Syscall specific hooks (#2389) TUI Support Infrastructure (#1620) Fix coveralls upload (#2387) docs: fix simple typo, straigth -> straight (#2381) Attempt to allow symbolic balances from the start (#1818) Fix state.cpu.PC member (#1825) Bump black and mypy (#1824) Manticore 0.3.5 (#1808) Fix yices timeout argument (#1817) ...

pwang00 and others added 7 commits February 6, 2020 11:02

Add log monitoring

e0a0321

Log monitoring via TCP

b090dad

Swittch to rendering state lists directly

2620c8e

Extraneous line

33116af

Merge branch 'master' into dev-phillip

beebf34

Switch log buffer to multiprocessing queue

9e70022

ehennenfent linked an issue Jun 19, 2020 that may be closed by this pull request

Add a real-time TUI #1744

Closed

Eric Hennenfent and others added 20 commits July 16, 2020 15:14

Create state transition events

edd1cad

Should make it possible to track movements between state lists

Plug new events into context

9f45f21

This will break the state merging plugin (but I'll fix it eventually)

move most enums to their own module

dcc7774

Blacken

8c27091

Add DaemonThread from TUI branch

ae20d55

Add interface for registering daemon threads

4318c5c

Timestamp StateDescriptor upon updates

812e1dd

Capture return value

1497345

Merge branch 'fix-run_dot_sh' into dev-introspection-api

f1db62b

Blacken

7b3d0b4

Add solver wrapper to StateBase

b28803a

Add solve events to all instances of SelectedSolver.instance()

3935f4e

Remove executor constraints from WASM

51a0144

Add solve events to memory.py

2caa874

Add intermittent execution event

4bcdbe3

Be more generous with states whose initialization we missed

c3c8866

Add Native callback for updating state descriptor

572bb9f

Fix state killing

941c63a

Blacken

9a206e1

codecov: Remove outdated 'yml' entry in CI

09b4dcb

From these commits codecov/codecov-action@ebea5ca codecov/codecov-action@49c86d6

Eric Hennenfent added 2 commits August 28, 2020 11:27

Reformat programatically generated files

617e23f

Drop max verbosity in logging tests

4eb634b

Haven't been able to figure out why, but somehow other loggers get "stuck" at this high verbosity and the integration tests try to print out the values of every single register.

ehennenfent requested a review from ekilmer August 28, 2020 20:16

ehennenfent added this to the Manticore 0.3.5 milestone Sep 3, 2020

Eric Hennenfent added 3 commits September 22, 2020 09:22

Merge branch 'master' into dev-phillip

ca17c3b

Merge branch 'master' into dev-phillip

14af4be

Fix duplicated code from bad merge

42c4982

ehennenfent requested review from feliam, ekilmer and sschriner and removed request for ekilmer October 13, 2020 16:04

ekilmer reviewed Oct 15, 2020

View reviewed changes

manticore/core/manticore.py Show resolved Hide resolved

manticore/core/state_pb2.py Show resolved Hide resolved

manticore/core/worker.py Outdated Show resolved Hide resolved

Eric Hennenfent added 3 commits October 19, 2020 16:00

Remove is_main from state_monitor

7605caa

Merge branch 'master' into dev-phillip

e41cf8c

Add comment about log buffer size

76e6bbd

ehennenfent commented Jan 4, 2021

View reviewed changes

Eric Hennenfent added 2 commits January 4, 2021 15:50

Remove vestigial is_main

42bd3c8

Blacken

b1935ee

ehennenfent merged commit 5a258f4 into master Jan 27, 2021

ehennenfent deleted the dev-phillip branch January 27, 2021 16:17

ekilmer added a commit that referenced this pull request Feb 18, 2021

Merge branch 'master' into dev-expressions

38a261c

* master: Syscall specific hooks (#2389) TUI Support Infrastructure (#1620)

ehennenfent mentioned this pull request May 24, 2021

Manticore 0.3.6 #2456

Merged

kokrui mentioned this pull request Jan 20, 2022

Add state list widget trailofbits/ManticoreUI-Ghidra#8

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TUI Support Infrastructure #1620

TUI Support Infrastructure #1620

ehennenfent commented Mar 3, 2020

ehennenfent commented Aug 28, 2020

ekilmer left a comment

ehennenfent left a comment

ehennenfent Jan 4, 2021

ehennenfent Jan 4, 2021

ehennenfent commented Jan 27, 2021

TUI Support Infrastructure #1620

TUI Support Infrastructure #1620

Conversation

ehennenfent commented Mar 3, 2020

ehennenfent commented Aug 28, 2020

ekilmer left a comment

Choose a reason for hiding this comment

ehennenfent left a comment

Choose a reason for hiding this comment

ehennenfent Jan 4, 2021

Choose a reason for hiding this comment

ehennenfent Jan 4, 2021

Choose a reason for hiding this comment

ehennenfent commented Jan 27, 2021