Skip to content
This repository has been archived by the owner on Feb 12, 2024. It is now read-only.

Fixed http-client dht/put #3516

Closed
wants to merge 1 commit into from
Closed

Conversation

rob-deutsch
Copy link

The http-client dht/put seems to refuse any possible key I can dream up to send it. It looks like its because its not operating on the same principle as dht/get.

@welcome
Copy link

welcome bot commented Jan 30, 2021

Thank you for submitting this PR!
A maintainer will be here shortly to review it.
We are super grateful, but we are also overloaded! Help us by making sure that:

  • The context for this PR is clear, with relevant discussion, decisions
    and stakeholders linked/mentioned.

  • Your contribution itself is clear (code comments, self-review for the
    rest) and in its best form. Follow the code contribution
    guidelines

    if they apply.

Getting other community members to do a review would be great help too on complex PRs (you can ask in the chats/forums). If you are unsure about something, just leave us a comment.
Next steps:

  • A maintainer will triage and assign priority to this PR, commenting on
    any missing things and potentially assigning a reviewer for high
    priority items.

  • The PR gets reviews, discussed and approvals as needed.

  • The PR is merged by maintainers when it has been approved and comments addressed.

We currently aim to provide initial feedback/triaging within two business days. Please keep an eye on any labelling actions, as these will indicate priorities and status of your contribution.
We are very grateful for your contribution!

@achingbrain
Copy link
Member

achingbrain commented Mar 1, 2021

Thanks for submitting this, sorry for the delay in reviewing.

The problem this addresses is that passing a Uint8Array as a value to a URLSearchParams results in the Uint8Array being stringified and not encoded as a string.

E.g.

Uint8Array.from([0, 1, 2]).toString()
// "0,1,2"

So:

new URLSearchParams({ data: Uint8Array.from([0, 1, 2]) }).toString()
// "data=0%2C1%2C2"

This is then interpreted as a byte array encoded as an ascii string by the HTTP API server:

decodeURIComponent('0%2C1%2C2')
// "0,1,2"
Uint8Array.from("0,1,2")
// Uint8Array(5) [0, 0, 1, 0, 2]

This PR will fix the issue, but only where byte values can be represented as a string by JavaScript - note that for byte values 128-254 this is not the case:

const bytes = Uint8Array.from([254])
// Uint8Array [254]
const str = new TextDecoder().decode(bytes)
// "�"
new TextEncoder().encode(str)
// Uint8Array(3) [239, 191, 189]

Ascii/windows-1252 encoding doesn't quite do what you think it'd do either:

const bytes = Uint8Array.from([254])
// Uint8Array [254]
const str = new TextDecoder('ascii').decode(bytes)
// "þ"
new TextEncoder().encode(str)
// Uint8Array(3) [195, 190]

encodeURIComponent doesn't fare much better, essentially just url encoding the output of new TextEncoder().encode(str):

encodeURIComponent("þ")
// "%C3%BE"

This is what would end up being sent and I don't think go-IPFS will interpret %C3%BE correctly - certainly if you send it as pubsub message data on the query string it's sent to subscribers as w74= which decoded is Uint8Array(2) [195, 190] and not Uint8Array [254].

For it to be interpreted as Uint8Array [254] we'd need to send %FE which we can't do using built-in JS functions. We'd have to write our own string encoder/decoder which then means everyone using the HTTP API from JS would need to do the same thing, or use this client library.

Really for this to be properly fixed in a way that doesn't require JS and go to agree on what 'URL encoded' means we need to be able to encode bytes sent in the query string.

This is simple to do from the JS side but will need some coordination from go-IPFS.

@achingbrain
Copy link
Member

Writeup for encoding bytes sent in the query string in go-IPFS - ipfs/kubo#7958

@lidel lidel added status/blocked Unable to be worked further until needs are met and removed need/analysis Needs further analysis before proceeding labels Mar 29, 2021
@lidel
Copy link
Member

lidel commented Mar 29, 2021

Needs ipfs/kubo#7958 first.

@achingbrain
Copy link
Member

Thank you so much for opening this PR and for your patience in getting it merged. js-ipfs is being deprecated in favor of Helia. You can #4336 and read the migration guide.

ipfs-http-client should be swapped out for kubo-rpc-client - it's a drop in replacement and where future bug fixes and features will land.

If this is still a problem, a new PR should be opened against kubo-rpc-client.

@achingbrain achingbrain closed this Jun 2, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
status/blocked Unable to be worked further until needs are met
Projects
No open projects
Status: Done
Development

Successfully merging this pull request may close these issues.

4 participants