LogPoller DSL Parser #959

EasterTheBunny · 2024-12-04T18:10:46Z

Description

This PR adds a DSL parser that converts basic query filters into postgresql queries for logpoller. Additionally solana address and transaction hashes are handled.

NONEVM-744

Requires Dependencies

NONEVM-745 LogPoller db models #921

pkg/solana/logpoller/query.go

reductionista · 2024-12-07T00:21:37Z

pkg/solana/logpoller/orm.go

@@ -156,3 +157,27 @@ func (o *DSORM) SelectLogs(ctx context.Context, start, end int64, address Public
 	}
 	return logs, nil
 }
+
+func (o *DSORM) FilteredLogs(ctx context.Context, filter []query.Expression, limitAndSort query.LimitAndSort, _ string) ([]Log, error) {


I was thinking the FilteredLogs method that calls this would also be in this PR, but I guess it makes more sense if I just include it in my Main LogPoller Loop PR... probably too difficult to add it here without the struct that's defined in RegisterFilters PR.

Oh, I do think we should add some tests of this function to orm_tests.go.

In evm, there are several tests that test it in different ways: TestORM_IndexedLogs, TestORM_SelectLogsCreatedAfter, TestORM_SelectIndexedLogsByTxHash, TestORM_SelectLogsWithSigsByBlockRangeFilter. Those also test the corresponding legacy functions which we don't need. But maybe we could just combine the parts that test FilteredLogs together into a single TestFilteredLogs here?

reductionista · 2024-12-07T00:33:49Z

pkg/solana/logpoller/parser_test.go

+
+		result, args, err := parser.buildQuery(chainID, expressions, limiter)
+		expected := logsQuery(
+			" WHERE chain_id = :solana_chain_id " +


I think this should just be :chain_id rather than :solana_chain_id, to match the db schema.

Also, hoping we can move this code to chainlink-framework at some point to re-use it for many chain families, abstracting it a bit to fill in the custom data types.

reductionista · 2024-12-07T01:02:40Z

pkg/solana/logpoller/parser.go

+	ErrInvalidCursorDir    = errors.New("invalid cursor direction")
+	ErrInvalidCursorFormat = errors.New("invalid cursor format")
+	ErrInvalidSortDir      = errors.New("invalid sort direction")
+	ErrInvalidSortType     = errors.New("invalid sort by type")


Good call on defining these all together! This will help later with abstraction.

reductionista · 2024-12-07T01:05:34Z

pkg/solana/logpoller/parser.go

+}
+
+func (v *pgDSLParser) whereClause(expressions []query.Expression, limiter query.LimitAndSort) (string, error) {
+	segment := fmt.Sprintf("WHERE %s = :solana_chain_id", chainIDFieldName)


Suggested change

segment := fmt.Sprintf("WHERE %s = :solana_chain_id", chainIDFieldName)

segment := fmt.Sprintf("WHERE %s = :chain_id", chainIDFieldName)

(see my comment in parser_test.go about this... probably should have left it here instead)

reductionista · 2024-12-07T01:31:49Z

pkg/solana/logpoller/parser.go

+	case *pgDSLParser:
+		v.VisitEventSigFilter(f)
+	}
+}


I think we're missing an essential feature here that CCIP will require, and that we won't be able to implement QueryKey without. You have eventSigFilter and eventByAddressFitler, but we also need an eventBySubKeyFilter so that we can filter by subkey.

We'll need that to implement support for IndexedSequencesKeyFilter and several other IndexedSequences... filters that can be passed to QueryKey. CCIP definitely uses some of those.

See: func (b *EventBinding) encodeComparator(comparator *primitives.Comparator) (query.Expression, error) in chainlink/core/services/relay/evm/read/event.go)

We'll have to call NewEventBySubKey there instead of NewEventByTopicFilter and NewEventByWordFilter.

I will be adding this when I do the remapping.

reductionista

Mostly missing eventBySubKeyFilter, but see comments suggesting adding some tests for FilteredLogs and one minor change (:solana_chain_id -> :chain_id)

dhaidashenko · 2024-12-09T19:07:31Z

pkg/solana/logpoller/parser.go

+	v.expression = fmt.Sprintf(
+		"%s = :%s",
+		txHashFieldName,
+		v.args.withIndexedField(txHashFieldName, txHash),


solana.PublicKey does not implement Value method, so not sure if the query won't error. Might need to wrap it into type defined in types file.

pkg/solana/logpoller/query.go

The base branch was changed.

cl-sonarqube-production · 2024-12-24T17:54:36Z

Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 75%)

See analysis details on SonarQube

EasterTheBunny requested review from a team as code owners December 4, 2024 18:10

EasterTheBunny changed the base branch from develop to feature/NONEVM-745-logpoller-db-models December 4, 2024 18:11

dhaidashenko previously approved these changes Dec 4, 2024

View reviewed changes

reductionista reviewed Dec 7, 2024

View reviewed changes

pkg/solana/logpoller/query.go Show resolved Hide resolved

reductionista reviewed Dec 7, 2024

View reviewed changes

reductionista requested changes Dec 7, 2024

View reviewed changes

dhaidashenko reviewed Dec 9, 2024

View reviewed changes

EasterTheBunny had a problem deploying to integration December 18, 2024 15:51 — with GitHub Actions Failure

EasterTheBunny temporarily deployed to integration December 18, 2024 15:51 — with GitHub Actions Inactive

dhaidashenko force-pushed the feature/NONEVM-745-logpoller-db-models branch from 8de5211 to df7ab9f Compare December 20, 2024 15:01

dhaidashenko requested review from a team as code owners December 20, 2024 15:01

dhaidashenko force-pushed the feature/NONEVM-745-logpoller-db-models branch from df7ab9f to 4178d4c Compare December 20, 2024 15:14

Base automatically changed from feature/NONEVM-745-logpoller-db-models to develop December 23, 2024 17:38

dhaidashenko and others added 5 commits December 24, 2024 09:52

Replace solana types with custom to support db read/write

0c1cbd5

remove redundant file

8478c35

gomodtidy

9599ab2

default primitive handling and address/hash filters

cfb3232

address feedback

30b67db

EasterTheBunny force-pushed the feature/NONEVM-744-dsl-parser branch from 23a782c to 30b67db Compare December 24, 2024 15:54

EasterTheBunny temporarily deployed to integration December 24, 2024 15:54 — with GitHub Actions Inactive

EasterTheBunny temporarily deployed to integration December 24, 2024 16:16 — with GitHub Actions Inactive

EasterTheBunny temporarily deployed to integration December 24, 2024 16:21 — with GitHub Actions Inactive

EasterTheBunny force-pushed the feature/NONEVM-744-dsl-parser branch from 5f187da to df0062c Compare December 24, 2024 16:52

EasterTheBunny temporarily deployed to integration December 24, 2024 16:52 — with GitHub Actions Inactive

EasterTheBunny temporarily deployed to integration December 24, 2024 16:53 — with GitHub Actions Inactive

EasterTheBunny temporarily deployed to integration December 24, 2024 16:57 — with GitHub Actions Inactive

EasterTheBunny force-pushed the feature/NONEVM-744-dsl-parser branch from df0062c to c46420d Compare December 24, 2024 17:23

EasterTheBunny temporarily deployed to integration December 24, 2024 17:23 — with GitHub Actions Inactive

EasterTheBunny had a problem deploying to integration December 24, 2024 17:23 — with GitHub Actions Error

EasterTheBunny force-pushed the feature/NONEVM-744-dsl-parser branch from c46420d to 039d625 Compare December 24, 2024 17:26

EasterTheBunny temporarily deployed to integration December 24, 2024 17:26 — with GitHub Actions Inactive

EasterTheBunny temporarily deployed to integration December 24, 2024 17:31 — with GitHub Actions Inactive

fix linter issues

995b46f

EasterTheBunny force-pushed the feature/NONEVM-744-dsl-parser branch from 039d625 to 995b46f Compare December 24, 2024 17:43

EasterTheBunny temporarily deployed to integration December 24, 2024 17:43 — with GitHub Actions Inactive

EasterTheBunny temporarily deployed to integration December 24, 2024 17:47 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LogPoller DSL Parser #959

LogPoller DSL Parser #959

EasterTheBunny commented Dec 4, 2024 •

edited by jira bot

Loading

reductionista Dec 7, 2024

reductionista Dec 7, 2024 •

edited

Loading

reductionista Dec 7, 2024

reductionista Dec 7, 2024

reductionista Dec 7, 2024

reductionista Dec 7, 2024 •

edited

Loading

reductionista Dec 7, 2024 •

edited

Loading

EasterTheBunny Dec 18, 2024

reductionista left a comment •

edited

Loading

dhaidashenko Dec 9, 2024

cl-sonarqube-production bot commented Dec 24, 2024

	segment := fmt.Sprintf("WHERE %s = :solana_chain_id", chainIDFieldName)
	segment := fmt.Sprintf("WHERE %s = :chain_id", chainIDFieldName)

LogPoller DSL Parser #959

Are you sure you want to change the base?

LogPoller DSL Parser #959

Conversation

EasterTheBunny commented Dec 4, 2024 • edited by jira bot Loading

Description

Requires Dependencies

reductionista Dec 7, 2024

Choose a reason for hiding this comment

reductionista Dec 7, 2024 • edited Loading

Choose a reason for hiding this comment

reductionista Dec 7, 2024

Choose a reason for hiding this comment

reductionista Dec 7, 2024

Choose a reason for hiding this comment

reductionista Dec 7, 2024

Choose a reason for hiding this comment

reductionista Dec 7, 2024 • edited Loading

Choose a reason for hiding this comment

reductionista Dec 7, 2024 • edited Loading

Choose a reason for hiding this comment

EasterTheBunny Dec 18, 2024

Choose a reason for hiding this comment

reductionista left a comment • edited Loading

Choose a reason for hiding this comment

dhaidashenko Dec 9, 2024

Choose a reason for hiding this comment

cl-sonarqube-production bot commented Dec 24, 2024

Quality Gate failed

EasterTheBunny commented Dec 4, 2024 •

edited by jira bot

Loading

reductionista Dec 7, 2024 •

edited

Loading

reductionista Dec 7, 2024 •

edited

Loading

reductionista Dec 7, 2024 •

edited

Loading

reductionista left a comment •

edited

Loading