First, please download the original BEIR collection for the dataset, to do this, you can use the code below:
python3 BEIR_download.pyYou can also refer to BEIR URL for more information about BEIR dataset
- queries, or using tsv format, there are overall 790 requests in the dataset
- rid mapping file is also provided, which maps the request id to the original BEIR dataset id and query id
- engines, there are overall 16 engines in the dataset, each paired with name,model,Description,vertical,Description Source,Num_queries_original,Task,Objective (original),Is the task make sense in chatbot (RAG),Need LLM to generate query?,Can select manually?,Devise manually?,Note
- Resource Selection qrels, the relevance judgements for the resource selection task
- Resource Merging qrels, the relevance judgements for the resource merging task
- search results, the search results for each engine, which is contained in each subfolder, with it's highest performming dense retrievers
- response, the response generated using two mode best-fed or naive fed using gpt4.
- evaluation, the evaluation script for result selection and merging.
- Resource Selection, the evaluation script for resource selection