Integration test for marconi indexer using local testnet #714

eyeinsky · 2022-09-20T09:29:44Z

Pre-submit checklist:

Branch
- Tests are provided (if possible)
- Commit sequence broadly makes sense
- Key commits have useful messages
- Formatting, PNG optimization, etc. are updated
PR
- Self-reviewed the diff
- Useful pull request description
- Reference the ADR in the PR and reference the PR in the ADR (if revelant)
- Reviewer requested

marconi/marconi.cabal

plutus-example/test/plutus-example-test.hs

not much actual achievement..

Did it via trial and error: - the SubmitResult gives the numbers it needs to balance - the process is iterative: e.g the expected fee changes when applying the numbers from the first run, but it quickly converges.

.. so the modules could be used for testing.

eyeinsky · 2022-09-27T15:55:10Z

@koslambrou @james-iohk ping for another review.

These were introduced by rebasing on new master.

catch-21 · 2022-09-28T15:23:02Z

marconi/test/Integration.hs

+
+    return indexer
+
+  utxoVKeyFile <- H.note $ tempAbsPath </> "shelley/utxo-keys/utxo1.vkey"


Once we scale up number of tests we'll want to reduce and reuse as much as possible. Are we also able to move stuff like this along with other testnet helpers, or does MonadTest prevent this?

marconi/test/Integration.hs

catch-21 · 2022-09-28T16:21:27Z

marconi/test/Integration.hs

+      -- wants. And although they change again after you correct them,
+      -- then the procedure converges quickly.
+      executionUnits = C.ExecutionUnits {C.executionSteps = 500_000, C.executionMemory = 10_000 }
+      tx2fee = 1000303 :: C.Lovelace


Is there not a way to have this calculated? it could vary with different protocol params or how it's calculated in other eras.

There is function constructBalancedTx from this PR IntersectMBO/cardano-node#4446, but it's very recent. The cardano-node we depend on doesn't have it.

I see. Until we use this we could use a higher fee (e.g. 2ada) to be certain.

This reverts commit ca2c082.

As otherwise the test fails with either plutus-tx-plugin required or not required.

eyeinsky · 2022-09-29T16:17:02Z

@koslambrou trying to satisfy CI I've done the following things (do these look ok?), and at the end there is a new issue again with CI

915d3c0 disable unused-packages warning for the test
441b165 copy configuration/defaults/byron-mainnet/configuration.yaml to marconi/test/ -- this I'm most unsure of. The test runner doesn't run in the full plutus-apps repo so sibling folders to marconi are not available, so I just copied the file to the test folder and referenced it from the cabal file as well. But maybe we have prior art on how to include non-code files.
23e58b6 add back the literal "always succeed" plutus script

The new issue: https://hydra.iohk.io/build/19716407/nixlog/4. It happens to me locally as well (using nix-build -A plutus-apps.haskell.packages.marconi.checks.marconi-test). Also searched if anyone else had something similar and found this: IntersectMBO/cardano-node#4194

koslambrou · 2022-09-29T17:43:30Z

Alright, fixed it locally. Let's see with CI.

The issue was that cardano-testnet expects to have some env variables available in order to fetch these files. I've set them in Nix. See nix/pkgs/haskell/haskell.nix.

eyeinsky · 2022-09-30T12:25:20Z

The latest error here was that "pokeSockAddr: path is too long" from Network.Socket.Types here. Locally I don't get this and currently the CI job is queued. I suspect it's the node's socket as there are no other sockets in use, but not entirely sure.

Made also a PR here to show more info haskell/network#537.

EDIT: found failing CI report too: https://hydra.iohk.io/build/19720430/nixlog/6

zeme-iohk · 2022-09-30T13:37:14Z

It's not too obvious to me how to pull in your custom network.
I've tried source-repository-package as well, and run into the autoconf problem, which led me here input-output-hk/haskell.nix#778 and then I tried the suggested fix by adding

        ({
          packages = {
            network.components.library.preConfigure = ''
              ${pkgs.autoconf}/bin/autoreconf -i
            '';
          };
        })

to modules in nix/pkgs/haskell/haskell.nix, but to no avail.
I'll keep trying though

This reverts commit c0835cf.

Co-authored-by: Konstantinos Lambrou-Latreille <konstantinos.lambrou@iohk.io>

koslambrou requested a review from catch-21 September 20, 2022 14:18

raduom self-requested a review September 21, 2022 06:52

eyeinsky force-pushed the ml/plt-647 branch from 031dd66 to bd282cd Compare September 21, 2022 16:42

eyeinsky marked this pull request as ready for review September 21, 2022 16:43

koslambrou reviewed Sep 21, 2022

View reviewed changes

marconi/marconi.cabal Outdated Show resolved Hide resolved

koslambrou reviewed Sep 21, 2022

View reviewed changes

plutus-example/test/plutus-example-test.hs Outdated Show resolved Hide resolved

eyeinsky and others added 24 commits September 27, 2022 10:39

wip

5625ed1

wip2

64a4373

WIP

0c39f0b

wip

54d6800

wip

70deab2

not much actual achievement..

notes, skip this commit

32a8f23

wip; Tue, Sep 6

3181ff4

WIP

c1b8a3a

wip: get first transaction balanced

0bc1279

Did it via trial and error: - the SubmitResult gives the numbers it needs to balance - the process is iterative: e.g the expected fee changes when applying the numbers from the first run, but it quickly converges.

wip: in process of preparing the second transaction

94ca4b7

Move temporary code into own module

0218766

wip Thursday

b06dd72

wip Friday

f3fa0a5

Move marconi workers to library such that they could be tested

959d1a8

Move Marconi.Logging and Orphans to library

6408c3d

.. so the modules could be used for testing.

tmp

43a4cc3

wip Monday

7e9d6a6

Remove old cli usages (test still passes)

b404fd1

wip

5f37b87

Add todos

7a81f28

Add better comments, refactor

f995149

Add waitUntilFileExists

4566178

Create always succeeding plutus script inline

fc1deac

Remove unused imports

971daca

eyeinsky added 3 commits September 27, 2022 11:02

Revert to loop with threadDelay

dee8ec6

Move plutus-streaming to local dependencies' section

2139885

Add test comments, clarify test names (they were the same before)

2bfbdbc

eyeinsky force-pushed the ml/plt-647 branch from 1f03a50 to 2bfbdbc Compare September 27, 2022 08:05

Move starting the testnet into a helper function

0a3719b

eyeinsky force-pushed the ml/plt-647 branch from ec973d5 to 0a3719b Compare September 27, 2022 15:54

eyeinsky requested a review from koslambrou September 27, 2022 16:08

koslambrou approved these changes Sep 27, 2022

View reviewed changes

eyeinsky added 2 commits September 28, 2022 18:10

Fix dependency issues

1341239

These were introduced by rebasing on new master.

Fix yet another dependency issue

0d908d4

catch-21 reviewed Sep 28, 2022

View reviewed changes

eyeinsky added 4 commits September 29, 2022 11:31

Use alwaysSucceedValidator from plutus-script-utils

ca2c082

Move configuration.yaml to within marconi

441b165

Revert "Use alwaysSucceedValidator from plutus-script-utils"

23e58b6

This reverts commit ca2c082.

Disable unused-packages test

915d3c0

As otherwise the test fails with either plutus-tx-plugin required or not required.

Fix failing marconi test in Nix

97da264

[tmp] What socket path and its length

c0835cf

eyeinsky force-pushed the ml/plt-647 branch from 0ee5dcd to c0835cf Compare September 30, 2022 10:03

eyeinsky mentioned this pull request Sep 30, 2022

Add marconi-mamba package #724

Merged

12 tasks

Add exception for getting temporary folder on darwin

dd14e32

Revert "[tmp] What socket path and its length"

6e7f362

This reverts commit c0835cf.

eyeinsky merged commit 41875be into main Sep 30, 2022

eyeinsky deleted the ml/plt-647 branch September 30, 2022 14:34

zeme-iohk mentioned this pull request Sep 30, 2022

Patch network package so that it shows better errors [DO NOT MERGE] #731

Closed

koslambrou added a commit that referenced this pull request Apr 6, 2023

Integration test for marconi indexer using local testnet (#714)

15c30b1

Co-authored-by: Konstantinos Lambrou-Latreille <konstantinos.lambrou@iohk.io>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integration test for marconi indexer using local testnet #714

Integration test for marconi indexer using local testnet #714

eyeinsky commented Sep 20, 2022

eyeinsky commented Sep 27, 2022

catch-21 Sep 28, 2022

catch-21 Sep 28, 2022

eyeinsky Sep 29, 2022

catch-21 Sep 29, 2022

eyeinsky commented Sep 29, 2022

koslambrou commented Sep 29, 2022

eyeinsky commented Sep 30, 2022 •

edited

Loading

zeme-iohk commented Sep 30, 2022 •

edited

Loading


		return indexer

		utxoVKeyFile <- H.note $ tempAbsPath </> "shelley/utxo-keys/utxo1.vkey"

Integration test for marconi indexer using local testnet #714

Integration test for marconi indexer using local testnet #714

Conversation

eyeinsky commented Sep 20, 2022

eyeinsky commented Sep 27, 2022

catch-21 Sep 28, 2022

Choose a reason for hiding this comment

catch-21 Sep 28, 2022

Choose a reason for hiding this comment

eyeinsky Sep 29, 2022

Choose a reason for hiding this comment

catch-21 Sep 29, 2022

Choose a reason for hiding this comment

eyeinsky commented Sep 29, 2022

koslambrou commented Sep 29, 2022

eyeinsky commented Sep 30, 2022 • edited Loading

zeme-iohk commented Sep 30, 2022 • edited Loading

eyeinsky commented Sep 30, 2022 •

edited

Loading

zeme-iohk commented Sep 30, 2022 •

edited

Loading