TODO

# What do we need to run a query on the shopify?

Goal: EOD Monday

- [x] Test schema
      [x] Filter airbyte stuff using hacky script
- [x] Real shopify data
  - [x] Pull from fivetran
- [x] Python Indexing Script
  - [x] [mbianco] Snowflake connection
  - [x] [mbianco] Finalize prisma models
  - [x] [ryan] Table indexing
- [x] [mbianco] Python Query
  - [x] Endpoint for accepting natural language query
  - [x] Integration with TS side

# What do we need to iterate on the ranking?

Goal: EOD Tuesday

- [x] [ryan] Live Integration Tests
  - [x] Manually run script
  - [x] Agnostic to result data
- [x] [ryan] Python Indexing Script
  - [x] Question > Ranking data structure
- [x] [mbianco] Python Query Improvements
  - [x] Input ranked dict, iteratively compute the output schema
  - [x] Maybe SQL transforms from OpenAI
- [x] [mbianco] Python server vs shelling
  - [x] Buildout API for the answering questions
  - [x] Replace shell stuff with an API call
- [x] [mbianco] export DB script

# What do we need so daniel can run a query?

- [x] Ranking Improvements
  - [x] ENV config for face storage
- [x] Iterate on the ranking
- [x] UI integration
  - [x] Question record creation
- [x] [mbianco] User login tied to data source
- [x] [kai] MVP UI
  - [x] Login page
  - [x] Search box
  - [ ] Forced feedback state (complete/incomplete)
  - [x] Read-write SQL editor
  - [x] Basic JSON result
  - [x] Refactor to use `~/lib/` for all business logic
- [x] [mbianco] Deploy
  - [x] Docker Build
  - [x] File system access
  - [x] Postgres
  - [x] TBD embedding, if openai embeddings aren't good enough

# What do we need so daniel & spectora can run a query?

- [x] [mbianco] Data Source
  - [x] Dynamic datasource frontend
  - [x] Dynamic datasource backend
- [x] Snowflake connection for Spectora
  - [x] Run import script and fix new bugs
- [x] UI improvements
  - [x] Set feedback state to invalid if SQL compilation fails
  - [x] Snappier loading, either upgrading remix w/defer or fixing loading state
  - [x] Lock SQL + editing when the system is loading

# What do we need so JP can run a query and it doesn't break most of the time?

If most of the queries break, they will distrust the queries very quickly. We want to push the errors down to openai.

- [x] [mbianco] openai backoff
- [ ] error handling help to rerun and validate our changes against the queries that have already come in
  - [x] handle snowflake failures, mark the state as invalid
  - [x] handle openai failures, mark the questions in some state that we can query
  - [ ] [ryan] retry another openai query if the first one doesn't work
- [ ] [mbianco] training ui improvements
  - [ ] other minor improvements https://github.com/ryanstout/nlpquery/issues/29
  - [x] remove bad questions https://github.com/ryanstout/nlpquery/issues/20
- [ ] [ryan] The ranking system is not pulling in the correct columns, train a new model to improve column selection
  - [ ] [ryan] neural net
  - [ ] [all] training data
  - [x] hints aren't landing in the dataset, could just be a weighting issue
  - [x] reverse sort the schema?
- [ ] SQL dialect issues / SQLGlot issues
  - [ ] determine which errors in the DB are triggering this, sometimes it's just a bad model output
  - [ ] write test cases for failures
- [ ] run `python -m python.prompts.few_shot_embeddings.builder` on a cron

# What are additional UI improvements?

- [ ] Forced mark correct/incorrect
- [x] R+W SQL Editor [ryan]
- [ ] Visualize answers
- [x] Look at history
- [ ] Fancy URL structure .com/USER/12345-mike-wants-money

---

# What do we need to improve the query creator?

- [ ] Logging
- Dynamic test schema generation with column hints
- Training data for fine tuning & one-shot learning
- Maybe vector search for column names?

# What do we need to improve the query run experience?

- Feedback pipeline, force people to choose
- Sharable URLs

# Assorted Todos/Improvements

- [ ] Columns with the name "DEFAULT" or "VALUES" (and maybe other sql keywords don't seem to work with snowflake)
- [ ] Test adding column name in front of value for more context in embedding
- [ ] Look at primary key index annotations in schema
- [ ] Question model needs a dataSourceId
- [ ] Add versioning to EmbedLink, symlink swap links to Faiss files

# SQL snowflowflakification

- [ ] Convert Numeric type (postgres) to Number (snowflake)