Skip to content

EndexAI/simple-evals

 
 

Repository files navigation

Overview

This repository contains a lightweight library for evaluating language models. We are open sourcing it so we can be transparent about the accuracy numbers we're publishing alongside our latest models (starting with gpt-4-turbo-2024-04-09).

Evals are sensitive to prompting, and there's significant variation in the formulations used in recent publications and libraries. Some use few-shot prompts or role playing prompts ("You are an expert software programmer..."). These approaches are carryovers from evaluating base models (rather than instruction/chat-tuned models) and from models that were worse at following instructions.

For this library, we are emphasizing the zero-shot, chain-of-thought setting, with simple instructions like "Solve the following multiple choice problem". We believe that this prompting technique is a better reflection of the models' performance in realistic usage.

We will not be actively maintaining this repository and monitoring PRs and Issues. In particular, we're not accepting new evals. Here are the changes we might accept.

  • Bug fixes (hopefully not needed!)
  • Adding adapters for new models
  • Adding new rows to the table below with eval results, given new models and new system prompts.

This repository is NOT intended as a replacement for https://github.com/openai/evals, which is designed to be a comprehensive collection of a large number of evals.

Evals

This repository currently contains the following evals:

Samplers

We have implemented sampling interfaces for the following language model APIs:

Make sure to set the *_API_KEY environment variables before using these APIs.

Setup

Due to the optional dependencies, we're not providing a unified setup mechanism. Instead, we're providing instructions for each eval and sampler.

For HumanEval (python programming)

git clone https://github.com/openai/human-eval
pip install -e human-eval

For the OpenAI API:

pip install openai

For the Anthropic API:

pip install anthropic

Demo

python -m simple-evals.demo

This will launch evaluations through the OpenAI API.

Benchmark Results

Model Prompt DROP(f1) GPQA% MATH% MGSM% MMLU% HumanEval%
GPT4s
gpt-4-turbo-2024-04-09 chatgpt1 85.4 49.1 72.2 88.6 86.5 87.6
gpt-4-turbo-2024-04-09 assistant2 86.0 49.3 73.4 89.6 86.7 88.2
gpt-4-1106(-vision)-preview chatgpt 81.3 42.1 64.1 86.5 84.6 82.2
gpt-4-1106(-vision)-preview assistant 83.2 42.5 64.3 87.1 84.7 83.7
gpt-4-0125-preview chatgpt 83.4 39.7 64.2 83.7 84.8 88.2
gpt-4-0125-preview assistant 81.5 41.4 64.5 85.1 85.4 86.6
REFERENCE
Claude-3-Opus (rerun w/ api) empty3 79.0 49.7 63.2 89.7 84.1 84.8
Claude-3-Opus (rerun w/ api) lmsys4 77.1 50.7 63.8 89.2 84.2 82.9
Claude-3-Opus (report5) unknown 83.1 50.4 60.1 90.7 86.8 84.9
Gemini-Ultra-1.0 (report6) unknown 82.4 n/a 53.2 79.0 83.7 74.4
Gemini-Pro-1.5 (report6) unknown 78.9 n/a 58.5 88.7 81.9 71.9

Legal Stuff

By contributing to evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI evals will be subject to our usual Usage Policies: https://platform.openai.com/docs/usage-policies.

Footnotes

  1. chatgpt system message: "You are ChatGPT, a large language model trained by OpenAI, based on the GPT-4 architecture.\nKnowledge cutoff: 2023-12\nCurrent date: 2024-04-01"

  2. assistant system message in OpenAI API doc: "You are a helpful assistant." .

  3. claude-3 empty system message: suggsted by Anthropic API doc, and we have done limited experiments due to rate limit issues, but we welcome PRs with alternative choices.

  4. claude-3 lmsys system message: system message in LMSYS Fast-chat open source code: "The assistant is Claude, created by Anthropic. The current date is {{currentDateTime}}. Claude's knowledge base was last updated ... ". We have done limited experiments due to rate limit issues, but we welcome PRs with alternative choices.

  5. claude-3 reports: https://www.anthropic.com/news/claude-3-family.

  6. gemini-1.5 reports: https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/, we dont have rerun results due to rate_limit issues and paid-as-you-go version are still "coming soon" by the time of this study on 04/02. 2

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%