ipfs files write offset command appears faulty

[RichardLitt]

I'm not sure I am doing this right, but I am having issues making the offset command work. It does not seem to work simultaneously with create, and any subsequent usage doesn't seem to follow the offset I specify.

What I've run, so far:

12:48 ~/src/http-api-spec (feature/files) * 🐕  echo 'hello world' | ipfs files write /test --offset=2 --create
12:48 ~/src/http-api-spec (feature/files) * 🐕  ipfs files read /test
hello world
12:48 ~/src/http-api-spec (feature/files) * 🐕  echo 'hello world' | ipfs files write /test --offset=4
12:49 ~/src/http-api-spec (feature/files) * 🐕  ipfs files read /test
hehello world

[RichardLitt]

I am also confused by truncate, which appears to only work with offset.

[daviddias]

Can you illustrate with your expectations? Seems from your example that create is working fine. Were you expecting for there to be an 'offset' on the first hello world? On the second example it looks it performed well (started writing 4 bytes from the left, note: offset is not number of chars, but number of bytes)

[daviddias]

re: truncate

imagine a file being an array: [a,b,c,d,e,f] and each position is a character encoded in just one byte (like ASCII)

• with offset 0 (default)
  • writing any number of 6 bytes or more would wipe out the existing data
  • writing any number of 5 bytes or less would still keep the bytes unchanged
• with offset 2
  • any write would preserve the first 2 bytes
• with offset 2 and truncate
  • any write would preserve the first 2 bytes and always wipe out the rest, independently of how many new bytes you write (it cuts (truncates) the array on the specified offset

[RichardLitt]

offset is not number of chars, but number of bytes

This may have been what threw me. I did not expect that. I expected the first 'hello world' (second command) to be offset, too.

[daviddias]

See it like a computer 💻👀

echo 'hello world' | ipfs files write /test --offset=2 --create

says that it has 1) create a file if it doesn't exist; 2) move the writing pointer to byte 2; 3) write the incoming bytes, in this case it is a string 'hello world'

1. check if the file exists
2. it doesn't, create
3. move the writing pointer to byte number 2, but here lies the trick, the file is empty, moving it to byte number two means that now I would be in fact writing two 0 bytes (0000 0000 0000 0000) to the file and there fore modifying it, which is something that the offset shouldn't do. Therefore, do nothing.
4. write the incoming hello bytes.

PS: on 3), someone might have thought that is a total valid operation and implementing a FS that does just that

[whyrusleeping]

the truncate flag truncates the file before writing. the offset flag, if specified past the bounds of a file, will sparsely expand the file to the given offset, and begin the write from there.

[whyrusleeping]

although, it does seem strange that you ended up with 'hehello world' instead of 'hellhello world' ill take a look at that

[RichardLitt]

@whyrusleeping That's because it uses bytes instead of characters, which to me seems wrong. Or, at least, we should document that.

@diasdavid Got it. Thanks for the explanation. I think this can be closed when we document that it is bytes and not chars.

[hackergrrl]

@diasdavid: are we not using UTF-8? Why wouldn't two english chars like "he" be two bytes?

[hackergrrl]

The only documentation in ipfs files write --help for truncate is "Truncate the file before writing", which is woefully vague -- I'm still not entirely clear on its function.

[RichardLitt]

Succinct: "Truncate the rest of the file from the offset, before appending input."

[daviddias]

@diasdavid: are we not using UTF-8? Why wouldn't two english chars like "he" be two bytes?

!!(!(===)!==) I got lost in which was the direction of your question 😁

In utf8, 'he' is in fact two bytes. utf8 uses varints

[achingbrain]

I think using bytes for the offset and not characters is sensible as characters would be dependent on interpreting the input character set and all bets will then be off if you're trying to write bytes into a binary file.

FWIW the --create and --offset commands do appear to work together but it's hard to see on the terminal as 0x00 is not a utf8 space character:

$ echo 'hello world' | ipfs files write /foo.txt --offset 10 --create
$ ipfs files read /foo.txt | xxd -p
0000000000000000000068656c6c6f20776f726c640a

[Stebalien]

Yes. This appears to work as intended. Closing as this is an old issue.