docs: Tutorial on experimentation #705
Conversation
Codecov ReportPatch coverage has no change and project coverage change:
Additional details and impacted files@@ Coverage Diff @@
## main #705 +/- ##
==========================================
- Coverage 65.57% 65.16% -0.42%
==========================================
Files 406 421 +15
Lines 28216 28414 +198
==========================================
+ Hits 18504 18517 +13
- Misses 9712 9897 +185
☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Should this tutorial show up in the left-hand sidebar on the docs site? I was only able to find it by searching for it.
(I was looking at the version of the docs rendered from this PR)
| :func:`~benchmarking.commons.hpo_main_simulator.main`, | ||
| :func:`~benchmarking.commons.hpo_main_local.main`, | ||
| :func:`~benchmarking.commons.hpo_main_sagemaker.main`, depending on the trial |
There was a problem hiding this comment.
This renders in such a way that you can't actually see which main is which:
|
Damn! The CI is flakey again:
|
|
For some reason the left sidebar still wasn't showing up for me on the previous build -- seems like something is still funky with the page structure |
|
For me, it works: "Rapid Experimentation with Syne Tune" |
| also retrieve extra results. This method returns a dictionary, whose | ||
| "extra_results" entry is what we need. | ||
|
|
||
| Advanced Experimenteering |
There was a problem hiding this comment.
Is the spelling of Experimenteering intentional?
| You do not have to re-run baselines, but can easily combine older results | ||
| with more recent ones. This is explained | ||
| `here <../visualization/comparisons.html#combining-sesults-from-multiple-studies>`__. | ||
| * When running many experiments, some will fail. Syne Tune supports you in not |
There was a problem hiding this comment.
some will fail
Not actionable for this PR, but I'd be interested to learn where the failures come from, and if there's anything we can do about them. (Syne Tune problem, or SageMaker problem, or user error, or other?)
There was a problem hiding this comment.
I'll change to "may fail".
There was a problem hiding this comment.
This is indeed quite rare. But I've even seen things where dataset files could not be downloaded.
Another thing happening to me is that experiments for a method are failing, because there is a bug, but they work for all others.
If you are very careful and always test everything locally etc., this is unlikely. But sometimes, a bug only appears quite later.
| @@ -22,7 +18,7 @@ | |||
| import syne_tune | |||
|
|
|||
|
|
|||
| sys.path.insert(0, os.path.abspath(".")) | |||
| sys.path.insert(0, os.path.abspath("../../")) | |||
There was a problem hiding this comment.
What changed?
The path was just the . since December
There was a problem hiding this comment.
No idea. But I recall there was something mentioned recently, that something with imports changed.
Anyway, I suppose what we had before was working due to some legacy reason, and somebody switched it off.
I have to admit, I have little clue about this conf.py, I just look at what Renate is doing.
Tutorial on experimentation in Syne Tune, as presented in the BER science review.
This is purely documentation along with an example, no code changes
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.