Isn't CCP a white box model? How to test GPT-3.5-turbo

[wangzhonghai]

No description provided.

[IINemo]

Hi! Correct, CCP is a white box model. However, OpenAI API now provides top 5 logits from the probability distribution. It is not much, but we can run CCP with GPT-3.5-turbo. @ArtemVazh could you share an example?

[wangzhonghai]

@IINemo Hello, could you please provide an example for this section?

[IINemo]

Sorry for the late reply, we will provide the example asap. @cant-access-rediska0123 could you help with that?

[wangzhonghai]

@IINemo The CCP score I calculated is like this: [-0.872331670225528, -0.872331670225528, -0.849797917762312, -0.88501326670551728, -0.88501413239502603, -0.55894843210343627, -0.999430756426281, -1.0, -0.9999880353950354]. Is the result correct?

[cant-access-rediska0123]

@IINemo Hello, could you please provide an example for this section?

Currently, we do not have an implementation of CCP for the Blackbox case in LM-Polygraph. Our Blackbox models are specifically designed for scenarios where there is no information about token distributions.

We will consider adding code to address it in the future.

[cant-access-rediska0123]

@IINemo The CCP score I calculated is like this: [-0.872331670225528, -0.872331670225528, -0.849797917762312, -0.88501326670551728, -0.88501413239502603, -0.55894843210343627, -0.999430756426281, -1.0, -0.9999880353950354]. Is the result correct?

The numbers you calculated seem correct. The negative scores occur because the CCP estimator includes a minus sign in the calculation. This design ensures that higher values represent greater uncertainty.