Skip to content

Releases: graphprotocol/indexer

v0.19.3

14 Jun 17:50
@fordN fordN
v0.19.3
4b9e62b
Compare
Choose a tag to compare
v0.19.3

What's Changed

  • workflows: Update native modules precompile to include more Node versions by @fordN in #416
  • Manual Allocation Management by @fordN in #403
  • Update networks.md by @juanmardefago in #420
  • indexer-common: start block query - ensure numeric type by @hopeyen in #422
  • Dockerfiles: Update base images to node:16.15-bullseye-slim by @fordN in #424
  • CI: Use older version of Ubuntu to precompile native binaries by @fordN in #427
  • Upgrade cost-model lib by @fordN in #430

New Contributors

See the specific CHANGELOG.md files in each package for changes per package and the full changelog at v0.19.2...v0.19.3

Contributors

fordN, juanmardefago, and hopeyen
Assets 8
Loading

v0.19.2

29 Apr 15:29
@fordN fordN
v0.19.2
ca4cdf6
Compare
Choose a tag to compare
v0.19.2

What's Changed

  • Upgrade @graphprotocol/cost-model, improve publish setup by @fordN in #412
  • indexer-native: Install uses precompiled binaries hosted on GitHub releases by @fordN in #414
  • indexer-native: Declare support for ARM in package.json by @fordN in #415

Full Changelog: v0.19.1...v0.19.2

NPM packages

https://www.npmjs.com/package/@graphprotocol/indexer-agent/v/0.19.2
https://www.npmjs.com/package/@graphprotocol/indexer-cli/v/0.19.2
https://www.npmjs.com/package/@graphprotocol/indexer-common/v/0.19.2
https://www.npmjs.com/package/@graphprotocol/indexer-native/v/0.19.2
https://www.npmjs.com/package/@graphprotocol/indexer-service/v/0.19.2

Contributors

fordN
Assets 8
Loading

v0.19.0

24 Feb 19:07
@fordN fordN
v0.19.0
c62f0a5
Compare
Choose a tag to compare
v0.19.0 Pre-release
Pre-release

What's Changed

  • indexer-service: Rate limit indexer-service endpoints by @fordN in #336
  • Upgrade dependencies by @fordN in #338
  • Indexer agent logger improvement by @hopeyen in #333
  • *: Upgrade yargs dependencies and bump docker node image versions to LTS by @fordN in #345
  • Add option to allocate on network subgraph by @Theodus in #339
  • Upgrade node runtime image versions by @fordN in #346
  • indexer-service: Add /subgraphs/health/:deployment route by @Jannis in #348
  • indexer-agent, indexer-common: Add live metric for operator ETH balance by @Jannis in #354
  • indexer-common: Update cost-model dependency to fix the build by @Jannis in #355
  • Fix indexing status query used for health endpoint by @Jannis in #357
  • indexer-cli,indexer-common: Setup CLI test harness, create initial test suites by @fordN in #358
  • Indexing rules by subgraph id by @fordN in #329
  • indexer-agent: Support deployment id rules using either id format by @fordN in #361
  • indexer-cli: offchain commands by @hopeyen in #350
  • build(deps): bump follow-redirects from 1.14.6 to 1.14.7 by @dependabot in #353
  • Indexer-agent, indexer-cli: indexing rules applied instantly by @hopeyen in #352
  • indexer-agent: Batched voucher redemptions by @chriswessels in #349
  • indexer agent & cli - max lifetime options by @hopeyen in #343
  • indexer-agent reject unsupported subgraph unless bypassed by @hopeyen in #362
  • Rule to set automatic renewal for allocations by @hopeyen in #366
  • docs: Update graph-node to 0.25.1 by @otaviopace in #368
  • Updates to reflect current parameters by @Aderks in #365
  • Use signalledTokens instead of signalAmount for minSignal threshold by @Jannis in #367
  • Use AllocationExchange contract from common-ts by @Theodus in #359
  • indexer-agent: Fix subgraph identifier migration by @fordN in #369

New Contributors

  • @hopeyen made their first contribution in #333
  • @otaviopace made their first contribution in #368
  • @Aderks made their first contribution in #365

Full Changelog: v0.18.6...v0.19.0

Contributors

Jannis, fordN, and 6 other contributors
Assets 2
Loading
0 Join discussion

v0.18.6

04 Jan 18:09
@fordN fordN
v0.18.6
017fedd
Compare
Choose a tag to compare
v0.18.6

NPM

npm install -g @graphprotocol/indexer-agent@0.18.6
npm install -g @graphprotocol/indexer-cli@0.18.6
npm install -g @graphprotocol/indexer-service@0.18.6

Note: Installing these packages requires Rust and it's package manager cargo to be installed so that the cost model module can be built. If you already have Rust installed, make sure you're on the latest stable release. Installing the above packages may also require passing --unsafe-perm=true to the above install commands.

Docker

docker pull ghcr.io/graphprotocol/indexer-agent:v0.18.6
docker pull ghcr.io/graphprotocol/indexer-service:v0.18.6

Changes (Overview)

The most notable change included in this release is the addition of support for type 02 transactions as specified in EIP-1559.
The agent will now send type 02 transactions if supported by the client. Parallel allocations are being deprecated and will
automatically be removed by the agent now if found, in future releases all vestiges of parallel allocation management
will be removed. Another valuable update to the indexer-agent bring significant gas efficiency gains by batching claim transactions.

The indexer-cli has received some subtle, but important, updates to the status command outputs. 2 tables have been added
to the status output showing the indexer's 1. active allocations and 2. subgraph deployments.

Configuration options updates with this release (all optional):

Removed

Environment Variable CLI Option
INDEXER_AGENT_ALLOCATION_CLAIM_THRESHOLD --allocation-claim-threshold

Added

Environment Variable CLI Option Recommended Value
INDEXER_AGENT_REBATE_CLAIM_THRESHOLD --rebate-claim-threshold 250
INDEXER_AGENT_REBATE_CLAIM_BATCH_THRESHOLD --rebate-claim-batch-threshold 1000

Changes (Per Component)

Indexer Agent

Added

  • Support type 0x2 transactions (EIP-1559)
  • Close all parallel allocations and only renew a single allocation per deployment
  • Include an additional state for indexing dispute monitoring, references_unavailable, for the case where the indexer does not have a reference PoI available. Typically this case occurs when an indexer's deployment is not synced far enough

Changed

  • Deprecate the parallel allocations feature
  • Update Ethers dependencies
  • Keep deployment indexing for 1 day after allocation close
  • Reducing transaction logging redundancy to improve readability
  • Bump priority gas fee on gas price too low retry
  • Simplify transaction retry logic on nonce collisions; agent now introduces a delay and returns to reconciliation step to re-evaluate

Fixed

  • Only add prefix to voucher address if not present
  • Bump type 2 transaction gas fee configs on retry after timeout
  • Update max-transaction-attempts default to unlimited to avoid orphaned transactions
  • Treat deployments assigned to node = null or undefined as removed, so they are filtered out of the activeDeployments array
  • Fix delete query fee vouchers logic, so they are actually removed after the expiration time
  • Update tests to use the latest version of the Indexer class constructor

Indexer Service

Added

  • Support unattestable query responses

Changed

  • Broaden scope of allocations eligible for receiving traffic to include those recently closed
  • Update Ethers dependencies

Fixed

  • Use ENTRYPOINT instead of CMD in dockerfile so users can specify command line arguments in docker run command

Indexer CLI

Added

  • Show the status of indexer's active allocations in status command output
  • Show the status of indexer's subgraph deployments in status command output

Changed

  • Update Ethers dependencies

Indexer Common

Changed

  • Update Ethers dependencies

Indexer Native

Fixed

  • Fix build for Node v16 by upgrading neon-sys dependency.
Assets 2
Loading

v0.17.0

21 Jul 22:10
@fordN fordN
v0.17.0
eb6b15f
Compare
Choose a tag to compare
v0.17.0

NPM

npm install -g @graphprotocol/indexer-agent@0.17.0
npm install -g @graphprotocol/indexer-cli@0.17.0
npm install -g @graphprotocol/indexer-service@0.17.0

Note: Installing these packages requires Rust and it's package manager cargo to be installed so that the cost model module can be built. If you already have Rust installed, make sure you're on the latest stable release. Installing the above packages may also require passing --unsafe-perm=true to the above install commands.

Docker

docker pull ghcr.io/graphprotocol/indexer-agent:v0.17.0
docker pull ghcr.io/graphprotocol/indexer-service:v0.17.0

Changes (Overview)

This release primarily contains updates to the indexer agent to improve efficiency. The process of reallocating towards a subgraph deployment has been updated to use closeAndAllocate(), so it requires one transaction instead of two (close then allocate). Handling of high gas price scenarios has also been improved; when estimated gas prices exceed the threshold the agent will now enter a holding pattern where it waits for gas prices to return to below the gas-price-max before continuing with transaction execution. Along with the updated behavior, the gas-price-max startup argument has been updated for ease of use to be denominated in gwei rather than wei.

There were a few additional minor improvements to indexer agent:

  • improved handling of empty auth fields for the Ethereum endpoint,
  • more clear log messages from subgraph query errors.

Some networking optimizations have been included for both the indexer service and indexer agent:

  • both recommended k8s setups have both been updated to use a ClusterIP service instead of LoadBalancer since the LoadBalancer services have been shown to introduce more latency and their extra features aren't required here,
  • /network execution has been optimized by switching to a simpler HTTP client (doesn't need to be GraphQL aware).

As promised in the last release, this release includes an update to make the /network endpoint optional and introduces a dedicated (optional) auth token for /network requests.

New configuration options with this release (all optional):

Environment Variable CLI Option Recommended Value
INDEXER_AGENT_NETWORK_SUBGRAPH_AUTH_TOKEN --network-subgraph-auth-token <Secret bearer token for network queries>
INDEXER_AGENT_SERVE_NETWORK_SUBGRAPH --serve-network-subgraph true
INDEXER_AGENT_VOUCHER_EXPIRATION --voucher-expiration 2160

Changes (Per Component)

Indexer Agent

Added

  • Reallocate to subgraph deployment in one transaction using closeAndAllocate.
  • Wait for gas prices below the gas-price-max (gwei) before proceeding with transaction execution.
  • Add --voucher-expiration to configure the time (in seconds) after which a voucher with too few query fees is permanently deleted. Until then, keep retrying to redeem it to give the indexer a chance to adjust their --allocation-claim-threshold.

Changed

  • Show subgraph query error reason in logs.
  • Use ClusterIP in the k8s indexer agent service rather than a LoadBalalancer since LoadBalancer services have been
    shown to introduce extra latency.
  • Use undefined for auth user and password if none specified.
  • Update gas-price-max units to be gwei instead of wei for improved human readability.
  • Default gas-price-max changed from 20 gwei to 50 gwei.

Fixed

  • Improve robustness of subgraph deployments query by querying batches of only 10 deployments at a time.

Indexer Service

Changed

  • Make the /network endpoint optional by default and introduce a dedicated (optional) auth token for /network requests.
  • Optimize /network execution by switching to a simpler HTTP client (doesn't need to be GraphQL aware).
  • Use ClusterIP in the k8s indexer service service rather than a LoadBalalancer since LoadBalancer services have been
    shown to introduce extra latency.

Indexer CLI

Fixed

  • Remove vestigial check for channel-messages-inbox endpoint.

Indexer Common

Changed

  • Optimize /network execution by switching to a simpler HTTP client (doesn't need to be GraphQL aware).

Indexer Native

Fixed

  • Fix build for Node v16 by upgrading neon-sys dependency.
Assets 2
Loading

v0.16.0

09 Jun 12:25
@Jannis Jannis
v0.16.0
62c82ad
Compare
Choose a tag to compare
v0.16.0

NPM

npm install -g @graphprotocol/indexer-agent@0.16.0
npm install -g @graphprotocol/indexer-cli@0.16.0
npm install -g @graphprotocol/indexer-service@0.16.0

Note: Installing these packages requires Rust and it's package manager cargo to be installed so that the cost model module can be built. If you already have Rust installed, make sure you're on the latest stable release. Installing the above packages may also require passing --unsafe-perm=true to the above install commands.

Docker

docker pull ghcr.io/graphprotocol/indexer-agent:v0.16.0
docker pull ghcr.io/graphprotocol/indexer-service:v0.16.0

Changes (Overview)

This is a minor release that introduces smaller changes, mostly to improve indexer agent transactions:

  1. The revert reason of transactions is now detected to avoid retrying transactions that have no chance of succeeding.
  2. The default value for INDEXER_AGENT_TRANSACTION_ATTEMPTS / --transaction-attempts has been reduced from 5 to 2.
  3. The INDEXER_AGENT_ALLOCATION_CLAIM_THRESHOLD / --allocation-claim-threshold option has been extended to not only apply to claim transactions but also to redeeming query fee vouchers. This can help avoiding to collect query fees that are lower than the actual transaction cost.

Another change is that indexer service and indexer agent now both use the same network subgraph query logic, i.e. first trying a local deployment (if specified), otherwise falling back to querying the network subgraph via an (often external) endpoint. This means it is no longer necessary to point the indexer service at indexer agent. Instead, it is recommended to configure them both in the same way.

For mainnet, the recommended values are:

Environment Variable CLI Option Value
INDEXER_AGENT_NETWORK_SUBGRAPH_DEPLOYMENT --network-subgraph-deployment QmaUBw7sr8pBrwNgz6JkbmmGoUU7KJSaeRvCaY3NPDbQ7A
INDEXER_AGENT_NETWORK_SUBGRAPH_ENDPOINT --network-subgraph-endpoint https://gateway.thegraph.com/network
INDEXER_SERVICE_NETWORK_SUBGRAPH_DEPLOYMENT --network-subgraph-deployment QmaUBw7sr8pBrwNgz6JkbmmGoUU7KJSaeRvCaY3NPDbQ7A
INDEXER_SERVICE_NETWORK_SUBGRAPH_ENDPOINT --network-subgraph-endpoint https://gateway.thegraph.com/network

This release also introduces a /network route in the indexer service. This, by default, is protected via the INDEXER_SERVICE_FREE_QUERY_AUTH_TOKEN / --free-query-auth-token. It is safe, even advised, to keep this endpoint internal.

Addendum: It was planned to make the /network endpoint disabled by default. This will happen in a follow-up bugfix release.

Changes (Per Component)

Indexer Agent

Changed

  • Default to only one transaction retry attempt to reduce gas usage
  • Apply allocation claim threshold to redeeming query fee vouchers as well
  • Add the same network subgraph fallback logic as the agent uses

Fixed

  • Improve handling of reverted transactions by detecting the revert reason

Indexer CLI

No changes

Indexer Common

Indexer Native

No changes

Indexer Service

Added

  • Add auth-protected /network endpoint, disabled by default
Assets 2
Loading

v0.15.1

26 May 14:41
@Jannis Jannis
v0.15.1
87ea958
Compare
Choose a tag to compare
v0.15.1

NPM

npm install -g @graphprotocol/indexer-agent@0.15.1
npm install -g @graphprotocol/indexer-cli@0.15.1
npm install -g @graphprotocol/indexer-service@0.15.1

Note: Installing these packages requires Rust and it's package manager cargo to be installed so that the cost model module can be built. If you already have Rust installed, make sure you're on the latest stable release. Installing the above packages may also require passing --unsafe-perm=true to the above install commands.

Docker

docker pull ghcr.io/graphprotocol/indexer-agent:v0.15.1
docker pull ghcr.io/graphprotocol/indexer-service:v0.15.1

Changes (Overview)

This is a bugfix release that fixes an issue where receipts for allocations that were created before updating to 0.15 could not be synced to the database, making it impossible to collect query fees for these allocations.

Changes (Per Component)

Indexer Agent

Fixed

  • Ensure allocation summaries exist for allocations created before v0.15.0 or outside the indexer agent

Indexer CLI

No changes

Indexer Common

Added

  • Add ensureAllocationSummary utility for agent and service to use to create allocation summaries for receipts

Indexer Native

No changes

Indexer Service

Changed

  • Switch to ensureAllocationSummary utility from @graph-protocol/indexer-common
Assets 2
Loading

v0.15.0

25 May 16:48
@Jannis Jannis
v0.15.0
6d98976
Compare
Choose a tag to compare
v0.15.0

NPM

npm install -g @graphprotocol/indexer-agent@0.15.0
npm install -g @graphprotocol/indexer-cli@0.15.0
npm install -g @graphprotocol/indexer-service@0.15.0

Note: Installing these packages requires Rust and it's package manager cargo to be installed so that the cost model module can be built. Installing the above packages may require passing --unsafe-perm=true to the above install commands.

Docker

docker pull ghcr.io/graphprotocol/indexer-agent:v0.15.0
docker pull ghcr.io/graphprotocol/indexer-service:v0.15.0

Changes (Overview)

Launch of Scalar on Mainnet

The main change in this release is the replacement of State Channels with Scalar (using allocation-base receipts for now). Scalar enables highly scalable micro-transactions for query fees.

At a high level this means:

  • Along with every query, indexers now receive an updated Scalar receipt via the Scalar-Receipt header.
  • Receipts are collected via a /collect-receipts endpoint on the gateways. This will convert them into a signed (query fee) voucher that can then be redeemed via a new AllocationExchange contract on chain.

With this release, the indexer infrastructure changes as follows:

  • A new INDEXER_AGENT_COLLECT_RECEIPTS_ENDPOINT / --collect-receipts-endpoint configuration option is added to Indexer Agent to configure the endpoint that converts bundles of receipts into query fee vouchers.
  • A new INDEXER_SERVICE_CLIENT_SIGNER_ADDRESS / --client-signer-address configuration option is added to Indexer Service to configure the signer of the incoming receipts. For now, this limits indexers to being able to serve a single client, which is considered a temporary measure until the allocation-based receipts are replaced with receipts based on Vector transfers.

Network Subgraph Indexing

So far, indexers had to rely on https://gateway.network.thegraph.com/network to provide network subgraph data like allocations, subgraph deployments and more. With this release, indexers can index this network subgraph themselves and still fall back to https://gateway.network.thegraph.com/network if this subgraph hasn't fully synced yet or has failed locally.

In order to achieve this, the following infrastructure changes need to be applied:

  • Indexer agent now exports an HTTP server at port 8002. This needs to be made available to the indexer service.
  • Indexer agent needs to have the following configuration set:
    • INDEXER_AGENT_NETWORK_SUBGRAPH_ENDPOINT=https://gateway.network.thegraph.com/network (alternative: --network-subgraph-endpoint)
    • INDEXER_AGENT_NETWORK_SUBGRAPH_DEPLOYMENT=QmaUBw7sr8pBrwNgz6JkbmmGoUU7KJSaeRvCaY3NPDbQ7A (alternative: --network-subgraph-deployment)
  • Indexer service needs to have the following configuration set:
    • INDEXER_SERVICE_NETWORK_SUBGRAPH_ENDPOINT=http://indexer-agent:8002/network (alternative: --network-subgraph-endpoint)

This will make the agent index the network subgraph itself and serve it at :8002/network by default (this can be changed with --syncing-port), falling back to the gateway endpoint if there are any problems with the network subgraph. The service will then query the network subgraph through the agent.

Changes (Per Component)

Indexer Agent

Fixed

  • Make receipts table renaming migration more robust

Added

  • Allow network subgraph endpoint to be used as a fallback for the local network subgraph deployment
  • Add allocation exchange contract address for mainnet

Changed

  • Rename query fee related fields in database models

Indexer CLI

Changed

  • Rename query fee related fields in database models

Indexer Common

Fixed

  • Fix bug that could cause AsyncCache to fail perpetually

Indexer Native

No changes

Indexer Service

Changed

  • Rename query fee related fields in database models
  • Only sync database models in the agent to avoid race conditions
  • Rename X-Graph-Payment header to Scalar-Receipt
Assets 2
Loading

v0.14.0

12 May 21:14
@Jannis Jannis
v0.14.0
b84a3a6
Compare
Choose a tag to compare
v0.14.0

NPM

npm install -g @graphprotocol/indexer-agent@0.14.0
npm install -g @graphprotocol/indexer-cli@0.14.0
npm install -g @graphprotocol/indexer-service@0.14.0

Note: Installing these packages requires Rust and it's package manager cargo to be installed so that the cost model module can be built. Installing the above packages may require passing --unsafe-perm=true to the above install commands.

Docker

docker pull ghcr.io/graphprotocol/indexer-agent:v0.14.0
docker pull ghcr.io/graphprotocol/indexer-service:v0.14.0

Changes (Overview)

The main change in this release is the (temporary) replacement of Vector as the base layer of Scalar with allocation-base receipts. We will continue working on the Vector integration and plan to transition to Vector as soon as possible, but we first wanted to unblock our launch partners going live on mainnet.

At a high level this means:

  • Scalar receipts will, for now, be created against allocations, not Vector transfers.
  • Indexers will (for the moment) not need to run a Vector node.
  • Receipts are collected via a /collect-receipts endpoint on the gateways. This will convert them into a signed (query fee) voucher that can then be redeemed via a new AllocationExchange contract.

With this release, the indexer infrastructure changes as follows:

  • The Vector node is removed.
  • A new INDEXER_AGENT_COLLECT_RECEIPTS_ENDPOINT / --collect-receipts-endpoint configuration option is added to Indexer Agent to configure the endpoint that converts bundles of receipts into query fee vouchers.
  • A new INDEXER_SERVICE_CLIENT_SIGNER_ADDRESS / --client-signer-address configuration option is added to Indexer Service to configure the signer of the incoming receipts. For now, this limits indexers to being able to serve a single client. As a temporary change, this feels acceptable. Transitioning back to Vector will remove this limitation again.

Changes (Per Component)

Indexer Agent

Added

  • Store allocations for which the POI has been checked
  • Add subgraph deployment ID to POI disputes
  • Add allocation-based receipts and query fee vouchers as an alternative to Vector. This is the default for now. Vector support can still be enabled with --use-vector.

Fixed

  • Handle 'nonce has already been used' transaction failures

Indexer CLI

Added

  • Add subgraph deployment ID to POI disputes

Indexer Common

Added

  • Add subgraph deployment ID to POI disputes
  • Add async cache from indexer-service
  • Add database models for allocation-based receipts and query fee vouchers

Indexer Native

Changed

  • Use verify instead of recover for signatures when possible
  • Build native module in release mode for better performance

Fixed

  • Fix logging native signers usefully

Indexer Service

Added

  • Add allocation-based receipts and query fee vouchers as an alternative to Vector. This is the default for now. Vector support can still be enabled with --use-vector.
Assets 2
Loading

v0.13.0

19 Apr 11:37
@Jannis Jannis
v0.13.0
b02959b
Compare
Choose a tag to compare
v0.13.0

NPM

npm install -g @graphprotocol/indexer-agent@0.13.0
npm install -g @graphprotocol/indexer-cli@0.13.0
npm install -g @graphprotocol/indexer-service@0.13.0

Note: Installing these packages requires Rust and it's package manager cargo to be installed so that the cost model module can be built. Installing the above packages may require passing --unsafe-perm=true to the above install commands.

Docker

docker pull ghcr.io/graphprotocol/indexer-agent:v0.13.0
docker pull ghcr.io/graphprotocol/indexer-service:v0.13.0
docker pull connextproject/vector_node:0.2.1

Changes (Overview)

The main change in this release is the addition of the internal @graphprotocol/indexer-native package that speeds up the creation and signing of attestations as well verification of incoming receipts (for query payments) by roughly 3x.

Changes (Per Component)

@graphprotocol/indexer-agent

Changed

  • Update to latest common-ts

Fixed

  • Correctly determine the previous epoch when generating POIs

@graphprotocol/indexer-cli

Changed

  • Update to latest common-ts

@graphprotocol/indexer-common

Changed

  • Update to latest common-ts

@graphprotocol/indexer-native

Added

  • Add package for native attestation signing and receipt signature verification

@graphprotocol/indexer-service

Added

  • Add --allocation-syncing-interval option (default: 120s, previous 10s)
  • Use indexer-native package for native attestation signing and signature verification

Changed

  • Update to latest common-ts
Assets 2
Loading