Multiple Files in Hierarchical Folder Structure doesn't directly open attachments

[metal450]

When using a single-file database & opening an attachment, CherryTree warns you before quitting: "Temporary files were created and opened with external applications. Quit the external applications before quit CherryTree". That makes sense: since the attachments are embedded in the database itself, they have to be extracted somewhere to be edited, & then re-imported before CherryTree quits (or the edits will be lost).

One of the primary motivations in requesting "Multiple files in hierarchical folder structure" was that this would no longer be necessary: since all the attachments would then be stored directly on disk, double-clicking them can just open those files directly - thus you could quit CherryTree & continue editing them. It was the 3rd bullet in this issue: #1823:

"No risk of data loss when editing an attachment, as you're just editing directly on disk (i.e. even if you close CherryTree before saving the external edit, your editor just saves directly to disk)."

Also the last bullet in this follow-up: #1823 (comment)

"When opening an attachment, open the file in that directory. External file is edited directly, so saving the changes there is sufficient (no need to save the changes in the external editor, & then save changes in CT - could even quit CT while the external editor is still opened, & all will be good)."

However, I noticed that even with the new multi-file storage, CherryTree still shows this warning...and upon testing...it looks like it's actually not directly opening the attachments on disk. If I quit CherryTree and then save the attachment, no file in the "Hierarchical Folder" is modified, and the changes are lost.

Also, if I open an attachment, edit it, save, then save in CherryTree, it deletes the previous attachment & creates a new one. This also invalidates the 2nd bullet (motivation) in #1823:

"external backup software can see which attachment changed & needs to be versioned"

Because the attachment is deleted & created with a new name, rather than edited, external backup software can't identify it as an edit for the purpose of versioned backup.

All that is to say: when using multi-file storage, can we just have CherryTree directly launch the attachment in-place, so all the above benefits are realized?

[Kansai53]

It would also be nice if the separate files+folders were named in a way that they could be identified on disk. i.e. rather than just hashes, attachment filenames could be hash-attachmentname. And likewise, rather than the node folders being just numbers, they could be number-nodename. Then the tree could actually be interpreted on-disk if needed.

[metal450]

I realized that this also doesn't allow syncing software to benefit from block-level transfer. Since whenever you edit an attachment, it doesn't actually edit the attachment - rather it deletes the old file & recreates a new one with a different name. Thus the sync software thinks it's an entirely new file, and must transfer it in its entirety.

It would be much better if the attachments could be named as they are in the content, and when you edit them, it simply edits them directly on disk.

[metal450]

@giuspen any way you'd consider bumping this to the front of the queue if I were to send over a donation to motivate its development? :)

I totally understand CT is FOSS & there are tons of asks, so I'd be happy to contribute some sats if it could mean getting this done. Was super excited when #1823 was marked as resolved (it was my most eagerly awaited issue since 2021), but actually found that it didn't quite address the issues I was hoping.

Thanks again

[Ricent82]

I also wish attachments could be just edited directly. Is this difficult? It seems like it's almost completely done, since we have "separate file storage" mode already. Can the file embeds just reference the file directly, so the can be edited directly...?

[giuspen]

The multiple files data storage type has been a significant effort and I have tried for the first release to minimise the differences in behaviour compared to the other existing single file data types.
The idea was (and still is) to subsequently differentiate more exploiting the advantage of independent files.
I have been thinking about supporting embedded files with a constant name that can be editable externally also when cherrytree is closed, but in this case the undo/redo state machine wouldn't be able to track changes to this file.
The embedded file would in fact be almost like the link to file, just linking to somewhere internal to the folder structure and with embedded file appearance.
I guess I could support both and the user would choose. I will have a think.

[metal450]

So glad to see a response here! :)

The embedded file would in fact be almost like the link to file, just linking to somewhere internal to the folder structure and with embedded file appearance.

Yup, that's pretty much the exact implementation I had in mind from the original request ;)

but in this case the undo/redo state machine wouldn't be able to track changes to this file.

Sure, no issue there - that's to be expected.

This would honestly be such a huge usability boost: to not be concerned about having to leave CT opened & make sure changes are saved twice (once in the attachment's editor, again in CT), to be able to have unlimited-size attachments, to have recognizeable attachment names on disk, to have attachments be versioned by regular backup software (vs currently the filename aka hash changes with each change), etc. Would love love love to see this change 🙏 :)

[metal450]

Ahh SO excited to see motion on this! I check the commits literally every week with my fingers crossed & just saw the ones from a few days ago :)

Since I know there's often relatively long intervals between official builds, I have to ask: is this "ready to go" if I build from source myself? Or should still wait a bit before updating all my data?

[giuspen]

Hello @metal450 I just committed and pushed what was left to implement, it should be all done now and it would benefit from some testing from you.
If you are happy to build it yourself please go for it, if you need me to provide a binary I will help.
In order to enable this feature, you need to tick the checkbox in the preferences dialog, tab Miscellaneous, section Saving: "Multiple Files Storage, Use Embedded File Name on Disk".
I'm planning to release the new version v1.3.0 in about 2 to 3 weeks.

[metal450]

Oh if it's easy to post a binary, that would be great. I'm not setup for building atm but am so eager for this I was gonna get that setup (over the weekend) :) Note: I'm using both Windows & Linux

[giuspen]

Sure, I created and uploaded the Linux AppImage for now as quicker for me https://www.giuspen.net/software/CherryTree-1.2.0+32-x86_64.AppImage
Let me know if it's ok, I can create the windows bundle tomorrow.

[metal450]

Outstanding. Seems to work great.

The only comment/suggestion I'd make: attachment names are now easily identifiable on disk, but nodes aren't. Would it be possible to have the node folders named like id|name (as an example convention)? The name should probably be truncated to some max length to avoid long path issues on Windows, & normalized to remove disallowed chars, but implementation seems like it should be pretty easy - basically the software just ignores everything after the delimiter, still navigating by the id prefix. But then the folders are human-readable/navigable on disk, as the attachments now are :)

Other than that, no issues at all. Nice graceful migration too, where on the first "save" it updates the filenames.

[metal450]

Fyi building failed on Ubuntu (in VS Code, after installing all the packages in BUILDING.md):

/media/metal450/SSD-Data/Desktop/NoSync/cherrytree/src/ct/icons.gresource.cc up to date
Using gtest, gmock from: BUNDLED SRC
CMake Error at tests/CMakeLists.txt:10 (add_subdirectory):

Had to disable tests to get it to work (edited tasks.json & changed the command to ./build.sh Debug notests).

[metal450]

(was hoping it would spit out a Windows binary if I built it on Linux, so I didn't have to setup MSYS2 & possibly leave a bunch of clutter on the Windows install. But looks like it didn't. Would love that Windows binary too once you've got a sec, so I'm not using mismatched implementations on this dual-boot machine :) )

[giuspen]

I will build a windows binary as soon as I have a moment to sit at the pc. In which Ubuntu it failed to build the unit tests? Which language/locale?
I am aware of the issue of not easily recognise the folders on the file system, there is at least a ticket about that too, I will try to do something after the upcoming release.

[metal450]

I will build a windows binary as soon as I have a moment to sit at the pc.

Thanks!

In which Ubuntu it failed to build the unit tests? Which language/locale?

Technically it's KDE Neon, but it's based on 22.04 jammy. LANG=en_US.UTF-8.

[giuspen]

I noticed the error you pasted
CMake Error at tests/CMakeLists.txt:10 (add_subdirectory):

I reckon the issue you have building is simply that you forgot to do
git submodule update --init

[giuspen]

Please test also https://www.giuspen.net/software/cherrytree_1.2.0.0+32_win64_portable.7z

[metal450]

I reckon the issue you have building is simply that you forgot to do

...Yep, that would be the issue 🤦‍♂ 😊

Maybe a small suggestion for readability: I missed it because I went straight for the VS Code build instructions at the top, & just grabbed the ubuntu deps. Didn't even notice that there were "special" checkout instructions lower down in the ubuntu section, because I wasn't usuign the ubuntu-specific build command. Maybe the readme could start with a section/heading for "checkout the source" (which has those 3 lines, including the submodule command, as it applies to all platforms). Then each platform section like now? Just my 2c

[metal450]

Please test also https://www.giuspen.net/software/cherrytree_1.2.0.0+32_win64_portable.7z

Awesome, thanks! Will test after work today

[metal450]

Nevermind, couldn't bring myself to wait & just tested it out now haha.

Seems to work great on Windows too :) Just my one comment/wishlist about node names mentioned above, but aside from that, 100% perfecto!

[metal450]

Oh also one other thing that would be really handy: a context-menu item to reveal the file in the OS file browser (or if there's complex, copy its path to clipboard). To make it easier to find/navigate to the actual path on disk for a given attachment.

[metal450]

Ooh critical bug:

1. Create a new node
2. Type some text in it
3. Attach File -> Select a docx (that has some content in it)
4. Save
5. Double-click the docx in the node to open it. It's a blank docx, not the one you attached.

Reproduced it multiple times. On Windows.

Addendum:
6) If I keep deleting the attachment from the node & reattaching it, it ends up blank.
7) If I leave the node in CherryTree, then return to it, then attach the file, it gets attached correctly (attachment has the content of the file I selected)

[giuspen]

Thanks @metal450 I will produce a new build with fixes and will ask you to test that

[giuspen]

I uploaded a new pre 1.3.0 testing build called 1.2.0+51 with links at https://www.giuspen.net/cherrytree/#testing

[metal450]

Got it, will start using that version from now - thanks