Add pseudo-file paths to load/save resources from/to PackedByteArrays instead of files

[darthLeviN]

Describe the project you are working on

A game that needs to append a compressed image to the save file that is created from a big dictionary.

Describe the problem or limitation you are having in your project

Resource and other file based saving method in godot exist. they are great, however i cannot use them to store the data into a buffer and later get the data in a base64 string format.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

A Pseudo file path that represents an existing Byte array (in godot 3 it's called PoolByteArray but i'm sure it's different in godot 4).
That path would be passed on to any function expecting a file path.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

A runtime only resource representing a byte array needs to be introduced to make the whole thing safe to do.
this resource would generate a file path. godot engine will realize that this file path belongs to the resource, and thus directs all the writes to the byte array.
this resource will have a function to access this data safely.
it's better to make sure it works well with multithreading, although i haven't done much of that in godot and don't know where multithreading in godot can be done.

If this enhancement will not be used often, can it be worked around with a few lines of script?

Not without creating a temporary file to act as a proxy.

Is there a reason why this should be core and not an add-on in the asset library?

file path protocol of godot needs to be extended to make this work.

[Calinou]

• Related to Add Memory-Mapped Files (VFS) to Godot Editor as memory:// #10978.

Are you expecting something similar to data URIs in CSS/JavaScript?

To me, it sounds like we should add additional methods that can load/save to PoolByteArrays (PackedByteArrays in master). I think this is more explicit over data URIs which are less efficient and are prone to typos.

See #3270 for an example where this was done for glTF.

[darthLeviN]

Are you expecting something similar to data URIs in CSS/JavaScript?

To me, it sounds like we should add additional methods that can load/save to PoolByteArrays (PackedByteArrays in master). I think this is more explicit over data URIs which are less efficient and are prone to typos.

See #3270 for an example where this was done for glTF.

i read up on it right now. no no, data URIs are similar to how i intend to use the data. not the feature i'm proposing.
yes, one should add additional methods, but i thought that would require actually doing the work. for the currently made classes without this functionality, and the future ones. the function names might end up with different names so that needs to be managed too.

and it won't be prone to typos since the address will be automatically generated.

It might be just easier to use both for godot developers and game developers using godot.

[Calinou]

It might be just easier to use both for godot developers and game developers using godot.

I feel the proposed solution goes against the "Prefer local solutions" guideline still. In a game engine, there aren't a whole lot of cases where you need internal "in-memory" file paths so to speak.

[Diddykonga]

I'm all for exposing the default Serialization/Deserialization methods of the ResourceSaver/Loader that works with bytes, as methods for the Object class, to support converting any Object to its byte representation.

Although as far as being able to reference Resources, which aren't actually Resources because they are never saved/loaded from disk, with a unique in-memory path, I disagree with. Instead store your data inside of a Node, which can already be referenced via NodePaths.

[Alzurana]

Hey there, stumbled across this as I am now also facing this problem. I love being able to just define resources for my data and not worry about any saving and loading but I need a way to do something else with them than just saving them in individual files. (For me it's voxel savegame data that I'd like to put into a larger container format, similar to the region file format of minecraft)

I do agree with @Diddykonga , just exposing serialization methods that already exist makes a lot of sense. I don't see loading and saving resources to byte streams as a very common problem. Exposing the functions that do this would therefor perfectly align with guideline '#6: Cater to common use cases, leave the door open for the rare ones'

I'm specifically mentioning this because @Calinou's concern would also be addressed with this. No new core solution would be written and no API would be broken.