Skip to content
/ conduit Public
forked from algorand/conduit

Algorand's data pipeline framework.

License

Notifications You must be signed in to change notification settings

tzaffi/conduit

 
 

Repository files navigation

Shows a black Algorand logo light mode and white in dark mode.

CircleCI Github Contribute

Algorand Conduit

Conduit is a framework for ingesting blocks from the Algorand blockchain into external applications. It is designed as modular plugin system that allows users to configure their own data pipelines for filtering, aggregation, and storage of transactions and accounts on any Algorand network.

Getting Started

See the Getting Started page.

Building from source

Development is done using the Go Programming Language, the version is specified in the project's go.mod file. This document assumes that you have a functioning environment setup. If you need assistance setting up an environment please visit the official Go documentation website.

Run make to build Conduit, the binary is located at cmd/conduit/conduit.

Configuration

See the Configuration page.

Develoment

See the Development page for building a plugin.

Plugin System

A Conduit pipeline is composed of 3 components, Importers, Processors, and Exporters. Every pipeline must define exactly 1 Importer, exactly 1 Exporter, and can optionally define a series of 0 or more Processors.

Contributing

Contributions are welcome! Please refer to our CONTRIBUTING document for general contribution guidelines, and individual plugin documentation for contributing to new and existing Conduit plugins.

Common Setups

The most common usage of Conduit is to get validated blocks from a local algod Algorand node, and adding them to a database (such as PostgreSQL). Users can separately (outside of Conduit) serve that data via an API to make available a variety of prepared queries--this is what the Algorand Indexer does.

Conduit works by fetching blocks one at a time via the configured Importer, sending the block data through the configured Processors, and terminating block handling via an Exporter (traditionally a database). For a step-by-step walkthrough of a basic Conduit setup, see Writing Blocks To Files.

Migrating from Indexer

Indexer was built in a way that strongly coupled it to Postgresql, and the defined REST API. We've built Conduit in a way which is backwards compatible with the preexisting Indexer API and Postgresql DB.

Going forward we will continue to maintain the Indexer application, however our main focus will be enabling and optimizing a multitude of use cases through the Conduit pipeline design rather the singular Indexer pipeline.

For a more detailed look at the differences between Conduit and Indexer, see our migration guide.

Known Issues

Restarting Follower Nodes Multiple Times in a Row

When a follower node is restarted, the sync round is advanced to the node's ledger round. This causes a chain reaction where the node's ledger round is then advanced by MaxAcctLookback rounds. When this happens, the node should temporarily have access to 2 * MaxAcctLookback ledger state delta responses because some had been previously persisted to disk. However, if the follower node is restarted a second time before conduit has consumed the temporary ledger state delta objects, the node will become desynchronized from Conduit.

When this happens the follower node must be manually re-synchronized with Conduit. This is done by launching a new follower node or running fast catchup to move to an earlier round.

About

Algorand's data pipeline framework.

Resources

License

Code of conduct

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Go 90.0%
  • Python 9.1%
  • Other 0.9%