Bench: add option to disable Polly

[janhartman]

Polly often randomly fails on requests with no clear error even if passthrough mode is enabled. Since we don't want it enabled for context evals anyway, it's probably better to disable it outright.

Test plan

Running bench with a larger eval set like DS1000 for context now works without errors.

[jtibshirani]

Shouldn't this be covered by CODY_RECORDING_MODE? Maybe we should check if CODY_RECORDING_MODE === passthrough and avoid starting the polly recording, instead of introducing a new environment variable.

[janhartman]

Shouldn't this be covered by CODY_RECORDING_MODE? Maybe we should check if CODY_RECORDING_MODE === passthrough and avoid starting the polly recording, instead of introducing a new environment variable.

That option still starts Polly (Polly has a pass-through mode) and causes errors even though it shouldn't. Polly is apparently quite buggy!
We can also just not start Polly if recording mode = passthrough?

[jtibshirani]

We can also just not start Polly if recording mode = passthrough?

Yep let's do that! I think it's nicer from a user perspective to just have one environment variable, and for us to do the right thing under the hood to make it work.

[pkukielka]

Can we just add additional mode like CODY_RECORDING_MODE=disabled instead of changing meaning of passthrough? Passthrough is sometimes useful when debugging pollyjs issues, e.g. when one want to check if changes to pollyjs related code works in general and just recording is a problem.
It also seems to be more correct semantically.

[janhartman]

@pkukielka Noted! Will adjust.

[pkukielka]

LGTM