perf: Improve retrieval results

[yoomlam]

Ticket

Improve retrieval results

Prior retrieval results were poor:

SentenceTransformerEmbeddings(model_name="all-MiniLM-L6-v2"), chunk_size=1000, chunk_overlap=500
### RETRIEVE_K=10
Table:
1 | 1.0 | 6
2 | 0.0 | 8
3 | 0.0 | 7
4 | 0.0 | 5
5 | 0.0 | 3
6 | 1.0 | 6
7 | 0.0 | 5
8 | 0.0 | 6
9 | 0.0 | 9
10 | 0.0 | 7
11 | 0.0 | 4
12.1 | 0.0 | 8
12.2 | 0.5 | 5
12.3 | 0.0 | 7
12.4 | 0.0 | 7

Changes

Updated questions:

Q2
Changed: "... If so, do we need to include the son's income?"
     To: "... If so, do we need to include the dependent's income for SNAP?"

Q3: typo affected exact-match evaluation code
"What if my client shares meals but whats to apply to food stamps on their own?"
"What if my client shares meals but wants to apply for food stamps on their own?"

Q4: typo with quotes \"spouse\"

Q7: "... Can the client and her husband apply by themselves?"
to: "... Can the client and her husband apply for food stamps by themselves?"

Q8 requires 3 guru cards
- From: "... They have children under ages 14, 16, and 20. ..."
  To:   "... They have children under 18 (ages 14, 16, and 20). ..."
  got 1 of 3
- "... They have children under 18 (ages 14, 16, and 20). Which family members should be listed on the application?"
  "... They have non-citizen children under 18 (ages 14, 16, and 20). Which family members should be listed on the SNAP application?"
  got the other 2
- Lesson: Need LLM to create queries that would retrieve the right cards.

Q9: "... Can the client and the child apply for without listing the boyfriend?"
to: "... Can the client and the child apply for food stamps without listing the boyfriend?"

• Q5 has no answer, so ignore it for now.
• Q10 is unchanged. Retrieval is better (probably) due to including the question (card title aka preferredPhrase field) in the embedding.
• Q11 fixed question. Incomplete -- asked in Slack.

More optimizations can be done.

• Didn't test other embeddings but this one (all-MiniLM-L6-v2) is generally recommended.

Improved results

# RETRIEVE_K=10
Table:
1 | 1.0 | 6
2 | 1.0 | 6
3 | 1.0 | 6
4 | 1.0 | 6
5 | 0.0 | 8
6 | 1.0 | 7
7 | 1.0 | 6
8 | 1.0 | 6
9 | 1.0 | 6
10 | 1.0 | 7
11 | 1.0 | 6
12.1 | 0.25 | 8
12.2 | 1.0 | 5
12.3 | 1.0 | 9
12.4 | 1.0 | 7

# RETRIEVE_K=5
Table:
1 | 1.0 | 4
2 | 1.0 | 3
3 | 0.0 | 5
4 | 1.0 | 4
5 | 0.0 | 4
6 | 1.0 | 4
7 | 1.0 | 3
8 | 0.6666666666666666 | 3
9 | 1.0 | 4
10 | 0.0 | 4
11 | 1.0 | 2
12.1 | 0.0 | 4
12.2 | 0.5 | 3
12.3 | 1.0 | 4
12.4 | 0.0 | 5

Testing

Testing instructions and expected behavior:

1. pip install -r requirements.txt
2. rm -rf chroma_db
3. RETRIEVE_K=10 python run.py and 5. Ingest and Evaluate retrieval

Repeat 2 and 3 for each change in embedding parameters.

[ccheng26]

lgtm