DataLad as DVC for ML analysis

[adswa]

This is a section that takes a quite elaborate DVC tutorial and recreates it with DataLad.
I've split the original tutorial into three "parts" (Versioning data, Sharing data, Analysis), and in each part, I execute the DVC workflow and a corresponding DataLad workflow.
Instead of adjusting the workflow into a YODA-principle or similar DataLad-centric workflow, I'm recreating the workflow as closely as I can with DataLad as for a tool comparison in the "domain" of DVC.

Both DVC and DataLad pipelines are executed and included with output. Apart from a slightly different order and skipping the tutorial section in which only scripts are executed without pipelining ("Version Datasets and Models"), I've recreated the tutorial very closely, and didn't make any code changes.

I would appreciate feedback on the following:

• I don't want to throw shade at DVC, I just want to provide a comparison. I think it helps that it is a tutorial by them, but feedback on whether I am unfair or appear (overly) biased in the text around the code execution or in other ways would be appreciated.
• Are there more elegant ways to recreate the DVC workflow? Something that is a bit ugly is the comparison of metrics across analysis since we have no ML specific reporting. This tutorial is easy, it has only two models to compare, and I can make a diff between the two refs. Is there a way to compare the state of a file (metrics/accuracy.json) across more than two refs?

CC @mih @yarikoptic

[adswa]

Also, ideas on where to put such a comparison would be appreciated :) currently I just put it into the "clutter" chapter in "Beyond basics"

[adswa]

Preview link to the section: https://datalad-handbook--569.org.readthedocs.build/en/569/beyond_basics/101-168-dvc.html

[yarikoptic]

Some review on initial portion... yet to digest in full -- but looks great overall.

[adswa]

Took about a week longer than I wanted it to take, but this section is ready now ;-)

[yarikoptic]

Minor recommended suggestions in the opening... yet to read more. So far so good ;)

[yarikoptic]

I've reached the end

[adswa]

Thanks much @yarikoptic. I'll add the remainder of your comments locally, and then I'll squash the past 20 commits.

[adswa]

@yarikoptic, it doesn't seem as if DVC people are giving feedback on this. Do you think this is ready to be merged? My general plan would be to use some of the content in this tutorial also for a stand-alone, DataLad-centric usecase (for the "DataLad for machine-learning" workshop), but if this is ready from your POV, I would be happy to reduce the number of open PRs by one. :)

[yarikoptic]

yes, let's merge!!! Thank you for this section - I think it is great and will be a valuable addition to the handbook.