Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(torii-indexer): add option for strict model reader block #2954

Merged
merged 4 commits into from
Jan 30, 2025

Conversation

Larkooo
Copy link
Collaborator

@Larkooo Larkooo commented Jan 27, 2025

Summary by CodeRabbit

  • New Features

    • Added a configurable strict_model_reader option to control model reading behavior.
    • Enhanced model retrieval with optional block-specific reading capabilities.
  • Improvements

    • Updated event processors to support more flexible model reading strategies.
    • Refined configuration handling for indexer model processing.
    • Introduced block-specific logic in event processing methods to improve model retrieval accuracy.

Copy link

coderabbitai bot commented Jan 27, 2025

Walkthrough

Ohayo, sensei! This pull request introduces a new configuration option strict_model_reader across multiple components of the Torii indexer. The change allows more flexible model reading by providing an option to retrieve models either from their original registration block or from the latest block. This modification spans several processor files, adding a boolean configuration that controls how models are read during event processing.

Changes

File Change Summary
crates/torii/cli/src/options.rs Added strict_model_reader: bool to IndexingOptions with default false and updated merge method.
crates/torii/indexer/src/processors/mod.rs Added strict_model_reader: bool to EventProcessorConfig.
crates/torii/indexer/src/processors/*_event.rs and *_model.rs Updated method signatures, added conditional model reading logic using strict_model_reader.
crates/torii/runner/src/lib.rs Added strict_model_reader to EventProcessorConfig in EngineConfig.

Sequence Diagram

sequenceDiagram
    participant Processor
    participant World
    participant Config
    
    Processor->>Config: Check strict_model_reader
    alt strict_model_reader is true
        Processor->>World: model_reader_with_block(block_number)
    else
        Processor->>World: model_reader()
    end
Loading

Possibly related PRs

  • feat(torii-core): specify namespaces to exclusively index #2687: The main PR introduces a new boolean field strict_model_reader to the IndexingOptions struct, which is related to the EventProcessorConfig struct in the retrieved PR that also adds a strict_model_reader field, indicating a connection in how model reading is configured across different components.
  • feat(torii): add world block and instrument queries #2796: The main PR's changes to the IndexingOptions struct and its merge method are relevant to the modifications in the EngineConfig struct in this retrieved PR, which also involves configuration changes that affect how data is fetched and processed.
  • feat(torii-indexer): parallelize models & event messages  #2912: The main PR's introduction of the strict_model_reader field aligns with the changes in the Engine struct's task management logic in this retrieved PR, which focuses on prioritizing tasks based on configuration settings, including those related to model processing.

Suggested reviewers

  • glihm

Ohayo and happy coding, sensei! 🍵🥷


📜 Recent review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 4624009 and 06b0791.

📒 Files selected for processing (1)
  • crates/torii/cli/src/options.rs (3 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • crates/torii/cli/src/options.rs
⏰ Context from checks skipped due to timeout of 90000ms (2)
  • GitHub Check: ensure-wasm
  • GitHub Check: docs

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR. (Beta)
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🔭 Outside diff range comments (1)
crates/torii/cli/src/options.rs (1)

Line range hint 201-245: Update merge method to handle strict_model_reader

Sensei, the merge method should be updated to handle the new strict_model_reader field to maintain consistency with other options.

Add this case to the merge method:

     pub fn merge(&mut self, other: Option<&Self>) {
         if let Some(other) = other {
+            if !self.strict_model_reader {
+                self.strict_model_reader = other.strict_model_reader;
+            }
+
             if self.events_chunk_size == DEFAULT_EVENTS_CHUNK_SIZE {
                 self.events_chunk_size = other.events_chunk_size;
             }
🧹 Nitpick comments (2)
crates/torii/indexer/src/processors/mod.rs (1)

33-33: Add documentation for the strict_model_reader configuration.

Ohayo! The new configuration field would benefit from documentation explaining its purpose and impact on model reading behavior.

+    /// When true, models are read from their registration block.
+    /// When false, models are read from the latest block.
     pub strict_model_reader: bool,
crates/torii/indexer/src/processors/register_event.rs (1)

81-81: Update the outdated comment above model reading.

The comment "Called model here by language..." should be updated to reflect the current implementation.

-        // Called model here by language, but it's an event. Torii rework will make clear
-        // distinction.
+        // Retrieve the event model using block-specific reading if configured
📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 9b722fd and 20dafa8.

📒 Files selected for processing (6)
  • crates/torii/cli/src/options.rs (2 hunks)
  • crates/torii/indexer/src/processors/mod.rs (1 hunks)
  • crates/torii/indexer/src/processors/register_event.rs (3 hunks)
  • crates/torii/indexer/src/processors/register_model.rs (3 hunks)
  • crates/torii/indexer/src/processors/upgrade_event.rs (3 hunks)
  • crates/torii/indexer/src/processors/upgrade_model.rs (3 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (1)
  • GitHub Check: fmt
🔇 Additional comments (7)
crates/torii/indexer/src/processors/register_model.rs (1)

80-84: LGTM! Clean implementation of block-specific model reading.

The conditional model reading implementation looks good, sensei! It properly handles both strict and non-strict modes.

crates/torii/indexer/src/processors/upgrade_model.rs (1)

89-93: Verify schema diff behavior with strict model reading.

The schema diffing logic might behave differently when using strict model reading. Please verify that comparing prev_schema (from DB) with new_schema (from specific block) produces expected results in all scenarios.

✅ Verification successful

Ohayo sensei!

Schema diff implementation is safe with strict model reading

The schema comparison is implemented safely, comparing the DB schema with the new schema regardless of how the new schema was obtained. The diff() method handles both cases equivalently, and proper null checks are in place.

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Search for tests covering schema diff scenarios
rg -A 5 "schema_diff.*test" 

Length of output: 29


Script:

#!/bin/bash
# Find schema-related files
fd schema
# Look for schema comparison implementation
ast-grep --pattern 'fn $_(schema1: $_Schema, schema2: $_Schema)'
# Search for any test modules containing schema
rg "mod.*test.*schema" -A 5

Length of output: 1400


Script:

#!/bin/bash
# Find schema_diff definition and usage
rg "schema_diff" -A 5 crates/torii/indexer/src/processors/upgrade_model.rs
# Look for schema comparison implementation
rg "fn.*compare.*schema" -A 5
# Check schema struct definition
rg "struct.*Schema" -A 10

Length of output: 2080

crates/torii/indexer/src/processors/upgrade_event.rs (3)

8-8: LGTM! Import addition for BlockId

The addition of BlockId import is necessary for the new block-specific model reading functionality.


50-54: LGTM! Parameter renaming reflects actual usage

Ohayo! The parameters are now properly named without underscores since they're being used in the implementation.


91-95: LGTM! Well-implemented conditional model reading

The implementation correctly uses block-specific model reading when strict_model_reader is true, falling back to latest state otherwise. This aligns perfectly with the PR objectives, sensei! 🎯

crates/torii/cli/src/options.rs (2)

93-101: LGTM! Well-documented configuration option

Ohayo! The new strict_model_reader option is well-integrated with clear documentation and proper attribute tags.


188-188: LGTM! Consistent default value

The default implementation correctly initializes strict_model_reader to false, matching the CLI default.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (1)
crates/torii/runner/src/lib.rs (1)

180-183: Ohayo sensei! The integration of strict_model_reader looks good! 🎋

The new configuration option is properly integrated within the EventProcessorConfig alongside other related settings like historical_events and namespaces.

However, since this is a new configuration option that affects model reading behavior, it would be good to add a code comment explaining its purpose and implications.

 event_processor_config: EventProcessorConfig {
+    // Controls whether models should be read from their registration block (true) 
+    // or from the latest block (false)
     strict_model_reader: self.args.indexing.strict_model_reader,
     historical_events: self.args.events.historical.into_iter().collect(),
     namespaces: self.args.indexing.namespaces.into_iter().collect(),
 },
📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 20dafa8 and 4624009.

📒 Files selected for processing (1)
  • crates/torii/runner/src/lib.rs (1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (4)
  • GitHub Check: clippy
  • GitHub Check: docs
  • GitHub Check: ensure-wasm
  • GitHub Check: build
🔇 Additional comments (1)
crates/torii/runner/src/lib.rs (1)

180-180: Verify consistent implementation across components

Let's ensure this option is properly handled in all relevant processors.

✅ Verification successful

Ohayo sensei! Implementation verified across all components

The strict_model_reader option is properly implemented in all four processors (register/upgrade model/event), following a consistent pattern with proper configuration flow.

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Search for usages of strict_model_reader across the codebase
echo "Checking for strict_model_reader implementations:"
rg "strict_model_reader" --type rust

echo -e "\nChecking for model reader calls that might need to consider the strict option:"
rg "model_reader" --type rust

Length of output: 5160

Copy link

codecov bot commented Jan 27, 2025

Codecov Report

Attention: Patch coverage is 42.10526% with 11 lines in your changes missing coverage. Please review.

Project coverage is 57.11%. Comparing base (91da65d) to head (506399a).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...ates/torii/indexer/src/processors/upgrade_event.rs 0.00% 3 Missing ⚠️
...ates/torii/indexer/src/processors/upgrade_model.rs 0.00% 3 Missing ⚠️
crates/torii/cli/src/options.rs 66.66% 2 Missing ⚠️
...tes/torii/indexer/src/processors/register_event.rs 66.66% 1 Missing ⚠️
...tes/torii/indexer/src/processors/register_model.rs 66.66% 1 Missing ⚠️
crates/torii/runner/src/lib.rs 0.00% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2954   +/-   ##
=======================================
  Coverage   57.11%   57.11%           
=======================================
  Files         424      424           
  Lines       56200    56215   +15     
=======================================
+ Hits        32097    32110   +13     
- Misses      24103    24105    +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Larkooo Larkooo enabled auto-merge (squash) January 29, 2025 04:00
@Larkooo Larkooo merged commit f4b733d into dojoengine:main Jan 30, 2025
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants