Add NM benchmarking scripts & utils

[varun-sundar-rabindranath]

Summary:
Add benchmarking scripts and utils.
Things to note :

• All files are stored in neuralmagic folder.
• neuralmagic/benchmarks/scripts/* : Actual benchmarking scripts that interact with vllm engine.
• neuralmagic/benchmarks/configs/* : JSON config files that define what benchmark commands to run.
• neuralmagic/benchmarks/run_*.py : Scripts that consume some config file and run the benchmark scripts.
• neuralmagic/tools : Add tools

Testing:
Local testing

[robertgshaw2-redhat]

Thanks Varun. In general this is looking good. I'm going to leave some more comments in a bit.

The one thing that is a bit tricky here is that the dataset processing is it a bit too tied into the server benchmarking, which makes it a bit tricky to support swapping in and out the datasets we are benchmarking (which we will do over time). For example, we will want a different request pattern for prefix caching performance benchmarking than general

I'm going to put up a PR with an idea for how to make this a bit more pluggable

[varun-sundar-rabindranath]

Thanks @robertgshaw2-neuralmagic

The one thing that is a bit tricky here is that the dataset processing is it a bit too tied into the server benchmarking, which makes it a bit tricky to support swapping in and out the datasets we are benchmarking (which we will do over time).

I made a recent refactor that moves the dataset related stuff into neuralmagic-vllm/neuralmagic/benchmarks/scripts/common.py, so it should be a little more organized now. But, I agree with the sentiment there are some kinks with the dataset stuff. Happy to talk about it .

[robertgshaw2-redhat]

@varun-sundar-rabindranath

I completely hacked the relative import system to do a simple proof of concept of how the dataset_registry could work. https://github.com/neuralmagic/neuralmagic-vllm/pull/30/files#diff-dca4e725ece41a665c0423924d56c905ce4c00188d79bc5dbeacb222c0ae6c6a

Idea was to have a programatic way to add new datasets. Thus, the get_sharegpt and get_ultrachat functions are responsible both for downloading and preprocessing the data.

• This structure removes the need for a dataset_download_cmds functionality as this can now be handled programatically within the dataset registry.
• This structure separates the preprocessing functionality from the benchmarking scripts. Since each dataset will have a different format, this is crucial

This should make it easy to add new datasets over time.

[andy-neuma]

i think we want to point out these are for our benchmarking scripts.

[andy-neuma]

looks like we are going down this rabbit whole again.

[andy-neuma]

thanks for the "meetup". looks good.