Feature download encryption

[paulo-ocean]

Fixes #38

Changes proposed in this PR:

• Implement downloadURL command with encryption following Implement encryption for downloadURL command #38

Some Notes:
On the directCommand request, the command/data itself is not all encrypted, only the AES secret key/init vector needed for the file encryption. If we we really want to encrypt also the entire command (all the params), that is also easy to change, but at least the node parameter (if present) must be visible, and we probably need to change a bit the payload (so the node knows if the command params need to be decrypted first)

Under src/helpers/scripts there is a clientExample.ts to test the flow
To run them:

• Open a terminal and setup node A (export port and private key) - Node A terminal
• Open a second terminal and setup node B (export port and private key) - Node B terminal
• Open a third terminal - this Client script
• 1. npm run start on terminal A
• 2. npm run start on terminal B
• 3. npm run client

Notes: regarding #79 there is a separate PR https://github.com/oceanprotocol/ocean-node/pull/80

It is set to run with node 20, most issue were related with module resolution. Seems the only way to get rid of all errors is to just use file extensions (they are mandatory on nodeJS) when importing local stuff from relative paths

[paulo-ocean]

btw, i forgot the private key thing (from config).. will address that in the meantime

[alexcos20]

still wip. got so far:

client:

alx@ubuntu:/ocean/ocean-node$ npm run client

> nodes-play@1.0.0 client
> mkdir -p ./dist/helpers/scripts/output && node dist/helpers/scripts/clientExample.js

secp256k1 unavailable, reverting to browser version
Got response from server... 200
Echo command status:  OK
url  http://127.0.0.1:8000/getP2pPeer
target id  16Uiu2HAkuYfgjXoGcSSLSpRPD6XtUgV71t5RqmTmcqdbmrWY9MJo
Got data from server: null
file:///ocean/ocean-node/dist/helpers/scripts/clientExample.js:153
        nodeId: data.id,
                     ^

TypeError: Cannot read properties of null (reading 'id')
    at getPeerDetails (file:///ocean/ocean-node/dist/helpers/scripts/clientExample.js:153:22)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async testEncryptionFlow (file:///ocean/ocean-node/dist/helpers/scripts/clientExample.js:171:25)
    at async file:///ocean/ocean-node/dist/helpers/scripts/clientExample.js:190:1

Node.js v20.9.0
alx@ubuntu:/ocean/ocean-node$ 

node1:

lx@ubuntu:/ocean/ocean-node$ npm run start

> nodes-play@1.0.0 start
> node --trace-warnings --experimental-specifier-resolution=node dist/index.js

secp256k1 unavailable, reverting to browser version





{
  level: 'info',
  message: '✅ [CONFIG] => Starting node with peerID:16Uiu2HAmUWwsSj39eAfi3GG9U2niNKi3FVxh3eTwyRxbs8cwCq72',
  component: 'CONFIG',
  timestamp: '2023-11-06 10:34:54'
}
{
  level: 'info',
  message: '[HTTP] => HTTP port: 8000',
  component: 'HTTP',
  timestamp: '2023-11-06 10:34:54'
}


{
  level: 'info',
  message: '[P2P] => Incoming direct command for peer self',
  component: 'P2P',
  timestamp: '2023-11-06 10:39:02'
}
{
  level: 'info',
  message: '[P2P] => Performing task: {"command":"echo","url":"http://example.com"}',
  component: 'P2P',
  timestamp: '2023-11-06 10:39:02'
}

i will retry

[paulo-ocean]

Hi @alexcos20 ,
the client example is using the following env keys/props
Node A:

export HTTP_API_PORT=8000
export PRIVATE_KEY=0xbee525d70c715bee6ca15ea5113e544d13cc1bb2817e07113d0af7755ddb6391

(node id will be : 16Uiu2HAkuYfgjXoGcSSLSpRPD6XtUgV71t5RqmTmcqdbmrWY9MJo)

Node B:

export HTTP_API_PORT=8001
export PRIVATE_KEY=0xcb345bd2b11264d523ddaf383094e2675c420a17511c3102a53817f13474a7ff

(will generate node Id: 16Uiu2HAmQU8YmsACkFjkaFqEECLN3Csu6JgoU3hw9EsPmk7i9TFL)

I had those values before on the clientExample.ts as well, for reference, but i removed them after your requests :-) ..
So now, basically you would need to set your own ids here:

const nodeA: P2PNode = {
  node_id: '16Uiu2HAkuYfgjXoGcSSLSpRPD6XtUgV71t5RqmTmcqdbmrWY9MJo',
  port: 8000
}

const nodeB: P2PNode = {
  node_id: '16Uiu2HAmQU8YmsACkFjkaFqEECLN3Csu6JgoU3hw9EsPmk7i9TFL',
  port: 8001
}

[alexcos20]

simple fix. in https://github.com/oceanprotocol/ocean-node/blob/feature-download-encryption/src/helpers/scripts/clientExample.ts#L347 from

 const url = `http://127.0.0.1:${nodeHttpPort}/getP2pPeer`

changed to

 const url = `http://127.0.0.1:${nodeHttpPort}/getP2PPeer`

now we have output.. still debugging.
managed to run the script, it took a lot !! (16 cpu, 16gb ram, gigabit connection)

ile download complete
Stream Ended! Saved file to path:  ./dist/helpers/scripts/output/received_out_

real	4m0.020s
user	0m1.650s
sys	2m10.446s

[paulo-ocean]

you're using the same example file? 'https://ia800909.us.archive.org/13/items/CC_1917_04_16_TheCure/CC_1917_04_16_TheCure_512kb.mp4'
that seems to be very slow download indeed

[alexcos20]

you're using the same example file? 'https://ia800909.us.archive.org/13/items/CC_1917_04_16_TheCure/CC_1917_04_16_TheCure_512kb.mp4' that seems to be very slow download indeed

yeah, most likely. cpu was 5% , download rate was 1Mb/s..

[alexcos20]

ok, analyzed the outputs, they are looking good, no diff between the 4 files. nicely done

[alexcos20]

lgtm, still some merge conflicts

[alexcos20]

should we move src/components/p2p/downloadHandler.ts to src/components/core/downloadHandler.ts ?

because downloadURL is a "core" feature

[paulo-ocean]

should we move src/components/p2p/downloadHandler.ts to src/components/core/downloadHandler.ts ?

because downloadURL is a "core" feature
I guess we could yes, the core actions taking place there are not exactly P2P related
I'm also taking care of the merge conflicts btw...
thanks