Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

snabbmark: Add "mp-ring" multiprocess benchmark #804

Closed
wants to merge 3 commits into from

Conversation

lukego
Copy link
Member

@lukego lukego commented Mar 6, 2016

Add a new mp-ring benchmark for measuring the performance of basic multi-process link operations. The benchmark forks worker processes and cycles packets through them via a series of links.

This particular benchmark "just works" in multiprocess mode without any changes to the Snabb core because packets and links are already allocated in shared memory, that is done before the children are forked, and freelists are not used because the same packets keep circulating from link to link.

The intention of this benchmark is to be a framework for investigating any fundamental performance limits of inter-process traffic (see #801) and for reproducing specific issues like inter-core conflicts on the cache-coherence level. The code uses a simple and naive Lua implementation ("basic") but can accommodate more sophisticated implementations (in the spirit of the asm code in #603). This could make it a useful tool for prototyping code like 100G mux/demux (#691).

(I have made no attempt to optimize this benchmark. That is another activity entirely.)

cc @xrme @kbara

Examples

[luke@lugano-1:~/git/snabbswitch/src]$ make && sudo ./snabb snabbmark mp-ring --processes 1
make: 'snabb' is up to date.
Benchmark configuration:
       burst: 100
  writebytes: 0
   processes: 1
   readbytes: 0
     packets: 100000000
        mode: basic
   pmuevents: false
  65.28 Mpps ring throughput per process

[luke@lugano-1:~/git/snabbswitch/src]$ make && sudo ./snabb snabbmark mp-ring --processes 2
make: 'snabb' is up to date.
Benchmark configuration:
       burst: 100
  writebytes: 0
   processes: 2
   readbytes: 0
     packets: 100000000
        mode: basic
   pmuevents: false
   5.44 Mpps ring throughput per process

[luke@lugano-1:~/git/snabbswitch/src]$ make && sudo ./snabb snabbmark mp-ring --processes 3
make: 'snabb' is up to date.
Benchmark configuration:
       burst: 100
  writebytes: 0
   processes: 3
   readbytes: 0
     packets: 100000000
        mode: basic
   pmuevents: false
   4.39 Mpps ring throughput per process

Usage

Usage:
  snabbmark mp-ring [OPTIONS]

  -m MODE, --mode MODE
                             Mode of operation. Determines which code
                             is used for the worker processes.
                             Currently supported values:
                               basic -- idiomatic Lua code [default]
  -n PROCESSES, --processes PROCESSES
                             Number of worker processes.
                             Default: 2
  -p PACKETS, --packets PACKETS
                             Number of packets processed by each worker.
                             Default: 100e6 (one hundred million)
  -b BURST, --burst BURST
                             Initial number of packets per link.
                             Default: 100
  -e EVENTS, --events EVENTS
                             Comma-separated list of PMU events to count.
  -r BYTES, --read BYTES
                             Number of bytes to read from each packet.
  -w BYTES, --write BYTES
                             Number of bytes to write to each packet.
  -h, --help
                             Print this usage message.

lukego added 3 commits March 6, 2016 04:30
This benchmark measures the throughput of <N> Snabb processes that are
circularly connected together in a ring.
Process <N> uses core <N>.
@lukego lukego self-assigned this Mar 6, 2016
@mention-bot
Copy link

By analyzing the blame information on this pull request, we identified @eugeneia and @hb9cwp to be potential reviewers

lukego added a commit to lukego/snabb that referenced this pull request Mar 6, 2016
@lukego lukego added the merged label Mar 6, 2016
@lukego
Copy link
Member Author

lukego commented Mar 6, 2016

The upstream branch for this change is multiproc and I have already merged it there. No action from other maintainers required here.

The whole multiproc branch will be submitted upstream when it makes sense to merge towards master.

@lukego lukego mentioned this pull request Mar 8, 2016
@xrme
Copy link
Contributor

xrme commented Mar 15, 2016

I'm not sure if it's kosher to keep commenting here, but this little snippet of the benchmark also causes cache line ping-ponging. If you add a crude C.usleep(100) in there, --mode ff throughput goes up 40%.

   -- Spin until enough packets have been processed                             
   while counters[0] < c.packets do
      core.lib.compiler_barrier()
   end

I think we could make the main process wait on a semaphore rather than spinning like this.

@lukego
Copy link
Member Author

lukego commented Mar 16, 2016

Interesting! Please push that crude sleep somewhere e.g. your mp-ring branch so that it will show up over on #813. Seems simple if we keep using that branch and PR for discussion. I will merge it up to multiproc whenever that makes sense.

I wonder if movnt would also avoid this ping-pong overhead. Curious to look at the PMU counters for this when I have a chance to understand the exact MESIF interaction.

@xrme
Copy link
Contributor

xrme commented Mar 16, 2016

I pushed 65abcd3 to my mp-ring branch, but my commit isn't showing up over on #813.

@lukego
Copy link
Member Author

lukego commented Mar 17, 2016

I think it's because there is no open Pull Request from your branch at the moment. Github automatically closed #813 when I completed the merge. Should be able to start a new pull request from the same branch to send further changes.

dpino pushed a commit to dpino/snabb that referenced this pull request May 3, 2017
…mx-test

Temporarily disable snabbvmx selftest
@eugeneia eugeneia closed this Nov 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants