Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Indexing rules by subgraph id #329

Merged
merged 3 commits into from
Jan 31, 2022
Merged

Indexing rules by subgraph id #329

merged 3 commits into from
Jan 31, 2022

Conversation

fordN
Copy link
Contributor

@fordN fordN commented Dec 1, 2021

This PR adds support for defining indexing rules by subgraph id (rather than deployment id). Defining rules at a subgraph level allows the indexer-agent to automatically manage versioning; keeping both the current and previous version synced and allocated towards.

Changes:

  • Update indexing-rule model: deployment --> identifier & identifierType
  • CLI interface unchanged but now supports providing subgraph ids where deployment ids were the only supported ids previously.
  • Rules by subgraph id are processed into deployment based rules, so the reconciliation logic doesn't need to change.

@fordN fordN requested a review from abarmat December 1, 2021 16:38
const epochLength =
await this.network.contracts.epochManager.epochLength()
const blockPeriod = 15
const bufferPeriod = epochLength.toNumber() * blockPeriod * 100 // 100 epochs
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably going to want to parameterize this, so indexers can set the period for which they continue to support the previous version.

Copy link
Contributor

@hopeyen hopeyen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great 👍 Haven't tried it locally as the indexer-agent still crashes for me, so the comments are only from reading 🥸

packages/indexer-agent/src/agent.ts Outdated Show resolved Hide resolved
@hopeyen
Copy link
Contributor

hopeyen commented Dec 22, 2021

Env all set up :D I'm crashing on the error IE017 below. Migration was successful, and a default global indexing rule was created with params identifier: "global", identifierType: "group", allocationAmount: "10000000000000000", parallelAllocations: 1, decisionBasis: "rules"

      "message": "Failed to ensure default global indexing rule",
       "code": "IE017",
      "explanation": "https://github.com/graphprotocol/indexer/blob/master/docs/errors.md#ie017",
      "cause": {
        "type": "b",
        "message": "[GraphQL] column \"identifier\" of relation \"IndexingRules\" does not exist",
        "name": "CombinedError",
        "graphQLErrors": [
          {
            "message": "column \"identifier\" of relation \"IndexingRules\" does not exist",
            "locations": [],
            "path": [
              "setIndexingRule"
            ],
            "extensions": {}
          }
        ]
      }

@fordN fordN force-pushed the ford/local-rules-by-gns branch 3 times, most recently from 7dcdaf8 to 4756681 Compare January 7, 2022 00:36
@hopeyen
Copy link
Contributor

hopeyen commented Jan 11, 2022

I think this is probably because my schema isn't updating, so I'm not sure if we would need to revisit printIndexingRules or displayIndexingRules. My indexer CLI's rules get all is missing the identifier column like this

┌──────────────────┬─────────────────────┬─────────────────────────┬───────────┬───────────┬──────────┬─────────────────────┬────────┬───────────────┐
│ allocationAmount │ parallelAllocations │ maxAllocationPercentage │ minSignal │ maxSignal │ minStake │ minAverageQueryFees │ custom │ decisionBasis │
├──────────────────┼─────────────────────┼─────────────────────────┼───────────┼───────────┼──────────┼─────────────────────┼────────┼───────────────┤
│ 50,000.0         │ 1                   │ null                    │ null      │ null      │ null     │ null                │ null   │ never         │
├──────────────────┼─────────────────────┼─────────────────────────┼───────────┼───────────┼──────────┼─────────────────────┼────────┼───────────────┤
│ 50,000.0         │ null                │ null                    │ null      │ null      │ null     │ null                │ null   │ never         │
├──────────────────┼─────────────────────┼─────────────────────────┼───────────┼───────────┼──────────┼─────────────────────┼────────┼───────────────┤
│ null             │ null                │ null                    │ null      │ null      │ null     │ null                │ null   │ always        │
├──────────────────┼─────────────────────┼─────────────────────────┼───────────┼───────────┼──────────┼─────────────────────┼────────┼───────────────┤
│ 50,000.0         │ null                │ null                    │ null      │ null      │ null     │ null                │ null   │ always        │

@fordN fordN force-pushed the ford/local-rules-by-gns branch 2 times, most recently from 1da7359 to 52d9229 Compare January 18, 2022 18:14
Copy link
Contributor

@hopeyen hopeyen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm trying to understand the handling for convertSubgraphBasedRulesToDeploymentBased. If there is an existing deployment based rule and subgraph based rule on the same deployment but with different parameters, should the subgraphBased rule trump the deployment rule?

packages/indexer-agent/src/agent.ts Outdated Show resolved Hide resolved
packages/indexer-agent/src/network.ts Outdated Show resolved Hide resolved
packages/indexer-cli/src/commands/indexer/rules/stop.ts Outdated Show resolved Hide resolved
- Upgrade umzug dependency.
- Setup umzug migrations CLI.
- Move migrations to db directory.
@fordN fordN force-pushed the ford/local-rules-by-gns branch 2 times, most recently from d1859b8 to 51dbf46 Compare January 28, 2022 22:31
@@ -40,6 +43,16 @@ const deploymentInList = (
): boolean =>
list.find(item => item.bytes32 === deployment.bytes32) !== undefined

const deploymentRuleInList = (
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hopeyen I've created this function to check if a deployment based rule already exists in the db. I use this below in convertSubgraphBasedRulesToDeploymentBased to check for existing deployment rules before creating them. This way any user defined deployment rules take precedent over subgraph based rules. I'll make note of this order of precedent in the documentation.

- Update indexing-rule model: deployment --> identifier & identifierType
- CLI interface unchanged but now supports providing subgraph ids
where deployment ids were the only supported ids previously.
- Rules by subgraph id are processed into deployment based rules, so
the reconciliation logic doesn't need to change.
- Update indexing-rules tests in indexer-common to use new rules model.
- Update indexer rules CLI integration tests.
@fordN fordN merged commit ed80d92 into main Jan 31, 2022
@fordN fordN deleted the ford/local-rules-by-gns branch January 31, 2022 17:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants