save token counts

[emrgnt-cmplxty]

Important

Add token counting for messages and save token metadata to the database, with tiktoken dependency for encoding.

• Token Counting:
  • Add tokens_count_for_message() and num_tokens_from_messages() in retrieval_service.py to calculate token counts for messages.
• Database Updates:
  • Modify agent() in retrieval_service.py to save input_tokens and output_tokens metadata when adding messages to the database.
• Dependencies:
  • Add tiktoken dependency in pyproject.toml for token encoding.

^{This description was created by for 02ac34e. It will automatically update as commits are pushed.}

[ellipsis-dev]

👍 Looks good to me! Reviewed everything up to 02ac34e in 1 minute and 32 seconds

More details

• Looked at 140 lines of code in 2 files
• Skipped 1 files when reviewing.
• Skipped posting 2 drafted comments based on config settings.

1. py/core/main/services/retrieval_service.py:36

• Draft comment:
  Consider making tokens_per_message configurable or document why 3 tokens are added for each message and reply. This hardcoded value might not be accurate for all models or message types.
• Reason this comment was not posted:
  Confidence changes required: 50%
  The function tokens_count_for_message and num_tokens_from_messages are calculating token counts, but there is a hardcoded value of 3 tokens added for each message and reply. This might not be accurate for all models or message types.

2. py/core/main/services/retrieval_service.py:56

• Draft comment:
  Replace print with logger.warning for better logging practices.

logger.warning("Warning: model not found. Using cl100k_base encoding.")

• Reason this comment was not posted:
  Confidence changes required: 50%
  The print statement in num_tokens_from_messages function is not ideal for logging warnings. It should use the logger for consistency and better control over log levels.

Workflow ID: wflow_XVoFD3adowaM6ew2

---

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.