Conversational Feedback

[hinthornw]

Context in the README.

Show how score chat responses based on a followup from the user and then log that as feedback in LangSmith

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)	Visit Preview		Oct 31, 2023 3:23pm

[rlancemartin]

Nice.

1/ IIUC, this is performing chat evaluations w/o explicit user-feedback, which is very useful. We might create a top-level shortened summary that just states this clearly. It's the first eval template, so very cool to have.

2/ We might explicitly mention that your chat app should be implemented (or called) in chain.py and call out specifically where as a placeholder. AFAICT, any chat runnable can simply append:

    .with_config(
        run_name="ChatBot",
        callbacks=[
            EvaluatorCallbackHandler(
                evaluators=[
                    ResponseEffectivenessEvaluator(evaluate_response_effectiveness)
                ]
            )
        ],

3/ Where to go fetch the evals in LangSmith? May be nice to show screenshot.

[hinthornw]

Cool cool. For (1), are you saying on top of the README that's here? (is it too roundabout)?

For (2) - yes albeit it needs a 'last_run_id' to be passed around so that the feedback can be assigned to the previous response trace. If we didn't care about the exact credit assignment, or if we had a better way of tracking conversations, this would be easier/better

For (3) def. I'll do that.