Improve integration tests

[paulo-ocean]

Fixes #250 .

Changes proposed in this PR:

• Refactor tests, change timeouts & eliminate most explicit delays
• Refactor waitToIndex, try emit/use events when possible (from indexer/processor). Catch/handle eventual test timeouts
• Clean tests logging (too much noise while reading config .env vars for instance) that are not even mandatory props
• Indexer crawling interval configurable (runs faster/often on tests as well). Default value remains otherwise
• Fix a few tests that depend on CI env (and they always fail locally). due to network issues like "EHOSTUNREACH" or "Invalid URL"
• Overall faster integration tests CI flow. Before was around 12 minutes, now around 7 minutes

[jamiehewitt15]

Tried running it a couple of times locally and both times it failed on the same test. Is this something to do with the timeout? Or have I done something wrong in the setup?

   1) should increase number of orders


  62 passing (2m)
  1 failing

  1) Indexer stores a new metadata events and orders.
       should increase number of orders:

      AssertionError: expected 1 to be above 1
      + expected - actual


      at Context.<anonymous> (file:///home/jamie/Desktop/ocean/ocean-node/dist/test/integration/indexer.test.js:276:53)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

[paulo-ocean]

Tried running it a couple of times locally and both times it failed on the same test. Is this something to do with the timeout? Or have I done something wrong in the setup?

   1) should increase number of orders


  62 passing (2m)
  1 failing

  1) Indexer stores a new metadata events and orders.
       should increase number of orders:

      AssertionError: expected 1 to be above 1
      + expected - actual


      at Context.<anonymous> (file:///home/jamie/Desktop/ocean/ocean-node/dist/test/integration/indexer.test.js:276:53)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Not sure, it does work for me.. but i also know that our tests don't usually work locally. In fact i changed a few that always fail locally and rely only on CI stuff/env. i can have a look later, maybe i can have a clue on the why it fails for you locally

[jamiehewitt15]

Could be the env values that I'm using, maybe something is missing. This is what I'm using at the moment:

export PRIVATE_KEY="0x1d751ded5a32226054cd2e71261039b65afb9ee1c746d055dd699b1150a5befc"
export IPFS_GATEWAY="https://ipfs.io/"
export ARWEAVE_GATEWAY="https://arweave.net/"
export RPCS='{ "1": {"rpc": "https://rpc.eth.gateway.fm", "chainId": 1, "network": "mainet", "chunkSize": 100}, "137": {"rpc": "https://polygon.meowrpc.com", "chainId": 137, "network": "polygon", "chunkSize": 100 }, "80001": {"rpc": "https://rpc-mumbai.maticvigil.com","chainId": 80001, "network": "polygon-mumbai", "chunkSize": 100 }, "8996": {"rpc": "http://127.0.0.1:8545", "chainId": 8996, "network": "development", "chunkSize": 100} }'
export FEE_TOKENS='{ "1": "0x967da4048cD07aB37855c090aAF366e4ce1b9F48", "137": "0x282d8efCe846A88B159800bd4130ad77443Fa1A1", "80001": "0xd8992Ed72C445c35Cb4A2be468568Ed1079357c8", "56": "0xDCe07662CA8EbC241316a15B611c89711414Dd1a" }'
export FEE_AMOUNT='{ "amount": 1, "unit": "MB" }'
export DB_URL="http://localhost:8108/?apiKey=xyz"
export OPERATOR_SERVICE_URL='["http://172.15.0.13:31000/"]'

[paulo-ocean]

gonna try with those... and let you know. i think maybe DB related, or something that is on CI and its not part of the .env.test file , but again, no branch at the moment has tests running locally (for a long time)
runs fine on CI though

[jamiehewitt15]

no branch at the moment has tests running locally (for a long time)

What about this branch, is it working for you every time? I ran it again but got a different error this time, I wonder if it might just be something to do with the timeout that's causing some random errors with the requests.

  5 passing (19s)
  1 failing

  1) Should run a complete node flow.
       should get file info with did:
     AssertionError: stream not present
      at Context.<anonymous> (file:///home/jamie/Desktop/ocean/ocean-node/dist/test/integration/completeFlow.test.js:201:9)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

[paulo-ocean]

no branch at the moment has tests running locally (for a long time)

What about this branch, is it working for you every time? I ran it again but got a different error this time, I wonder if it might just be something to do with the timeout that's causing some random errors with the requests.

  5 passing (19s)
  1 failing

  1) Should run a complete node flow.
       should get file info with did:
     AssertionError: stream not present
      at Context.<anonymous> (file:///home/jamie/Desktop/ocean/ocean-node/dist/test/integration/completeFlow.test.js:201:9)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Its not timeout related, i mean, you could increase the timeout on the test as much as you can, it will still timeout locally... the issue is env/db related for sure.. works fine on CI.
It doesn't work locally on develop branch either, for instance
I would say there is an issue with the DdoDatabase at least locally
i often have the errors:
DDO entry with ID did:op:XYZ not found in any schema.

[jamiehewitt15]

This is a big improvement in running the integration tests in the CI. Let's work on fixing them when running locally in a separate issue: #280