Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove timeline names from pageserver #1286

Merged
merged 11 commits into from
Mar 10, 2022
Merged

Conversation

SomeoneToIgnore
Copy link
Contributor

@SomeoneToIgnore SomeoneToIgnore commented Feb 16, 2022

Part of https://github.com/zenithdb/console/issues/693 and will require console adjustments.

Removes branch name, branch tag, related files and related code from pageserver's API, adjusting zenith and tests.
"Branch" concept is gone and now pageserver operates "timeline" only, yet the concept of "branching" is not removed itself: timelines still have ancestors and can branch off one from each other at given Lsn or from the end, as before.
Also, during API adjustment, "point in time" concept got removed, split into explicit "Lsn" and "ZTimelineId" parts.

One implicit advantage of the name removal is a more explicit separation between "postgres node name" and "branch name" that was used as the same entity in our tests when branching timelines.

This PR has a few tests failing, but most of them pass already, so it's time to consider, whether the approach is ok and if we want to go with it in general.

@@ -0,0 +1,320 @@
//!
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That used to be branches.rs, so not all of the code was added.

Copy link
Contributor

@hlinnaka hlinnaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm generally in favour of this, +1.

This does make the CLI user experience much worse though. Previously, you would do this:

./target/release/zenith init
./target/release/zenith start
./target/release/zenith pg start main

That doesn't work anymore, you have to specify a timeline ID to the 'zenith pg start' command:

$ ./target/debug/zenith pg start main
command failed: No timeline id, use --timeline to specify one

You have to copy-paste the random timeline ID from the output of the 'zenith init' command. And the advise above isn't actually correct: there is no '--timeline' option:

$ ./target/debug/zenith pg start main --timeline 2d88ad2ce4591b9fa55223b6d28153b8
error: Found argument '--timeline' which wasn't expected, or isn't valid in this context

	If you tried to supply `--timeline` as a value rather than a flag, use `-- --timeline`

USAGE:
    zenith pg start [OPTIONS] <node> [timeline]

For more information try --help

The correct syntax is:

./target/debug/zenith pg start main  2d88ad2ce4591b9fa55223b6d28153b8

Anyway, I'd like to keep that developer experience nicer. Could we still have branch names, just for the CLI? They don't need to be stored in the page server, just keep them e.g. a json file in the .zenith directory and map from the branch name to the timeline ID in the CLI.

@hlinnaka
Copy link
Contributor

Something's wrong with the 'timeline create' or 'list' command:

~/git-sandbox/zenith ((c89fd571...))$ ./target/debug/zenith timeline create --ancestor-timeline-id 627fbb511174568713344749430b7a17
Created timeline '78b01dcd8196eef6089212bbc39daf61' at 0/1698CC0 for tenant: 5cbf22b573093a87de3f207014ce750f
~/git-sandbox/zenith ((c89fd571...))$ ./target/debug/zenith timeline list
 627fbb511174568713344749430b7a17
 3ef01db24c8cbc1d58708235c736958d
 2d88ad2ce4591b9fa55223b6d28153b8

The newly created timeline isn't visible in the list.

@SomeoneToIgnore
Copy link
Contributor Author

Anyway, I'd like to keep that developer experience nicer. Could we still have branch names, just for the CLI?

Totally on board with the need to improve the UX, just not sure if branch names at its current form are good to use?
At least that "node_name" == "branch_name" == "timeline_spec" substitution mentioned in the comments above confuses me: if I'm having an alias, I'd like it to be the same alias in all 3 components semantically, not coincidentally.

Given that we already have to manipulate Lsn's in the CLI for cases when we have to start not on the timeline's end, I assumed that having a default timeline to ask for timeline_id input for non-default cases only could be enough for now (given that we implement timeline listing correctly)
More on that: #1286 (comment)

So, pg start main from your message on the new code should start a pg node on the default timeline of a default tenant.

@hlinnaka
Copy link
Contributor

Anyway, I'd like to keep that developer experience nicer. Could we still have branch names, just for the CLI?

Totally on board with the need to improve the UX, just not sure if branch names at its current form are good to use? At least that "node_name" == "branch_name" == "timeline_spec" substitution mentioned in the comments above confuses me: if I'm having an alias, I'd like it to be the same alias in all 3 components semantically, not coincidentally.

Fair enough. I agree it's a bit confusing.

Given that we already have to manipulate Lsn's in the CLI for cases when we have to start not on the timeline's end, I assumed that having a default timeline to ask for timeline_id input for non-default cases only could be enough for now (given that we implement timeline listing correctly) More on that: #1286 (comment)

So, pg start main from your message on the new code should start a pg node on the default timeline of a default tenant.

Ok, I guess that's enough. And we can add more decorations later when we start to do more stuff with branches, if we need to.

@SomeoneToIgnore SomeoneToIgnore changed the base branch from kb/zenith-test-harness to main February 17, 2022 00:08
@SomeoneToIgnore SomeoneToIgnore force-pushed the kb/remove-branch-files branch 3 times, most recently from eae0631 to e1d1202 Compare February 17, 2022 11:33
@SomeoneToIgnore
Copy link
Contributor Author

I should be fixing all (or close to that) tests now (including the list command), revamped the API, yet only now fully realized, that zenith is a CLI tool and does not save its memory between runs 🙂
So I think I'll fall back to the "names in json config on zenith level" approach and then this PR will be ready for review.

@SomeoneToIgnore SomeoneToIgnore force-pushed the kb/remove-branch-files branch 3 times, most recently from 9d0c711 to e672110 Compare February 17, 2022 11:55
Copy link
Contributor

@LizardWizzard LizardWizzard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work! I like these changes

As I mentioned in one of the comments we may even want to remove the create tenant flag from init and let the cli make needed api calls. It may be not so convenient for cli because then we have to start pageserver to make these calls as opposed to init which doesnt have to do so

+1 on a name->mapping for timelines, we can embed the table directly in cli's config file

@SomeoneToIgnore SomeoneToIgnore force-pushed the kb/remove-branch-files branch 7 times, most recently from aa420e6 to 222e935 Compare February 27, 2022 21:22
@SomeoneToIgnore SomeoneToIgnore marked this pull request as ready for review February 27, 2022 22:04
@SomeoneToIgnore
Copy link
Contributor Author

SomeoneToIgnore commented Feb 27, 2022

Names had returned to Zenith CLI, all the feedback had been addressed or answered. The tests seem to pass either.

Please have a look and tell me what you think, I'll prepare some digest on the broken APIs to discuss it in more detail.
One caveat that still exists is that all ZIds are serialized as bytes into toml, that is being addressed separately in #1330

@SomeoneToIgnore SomeoneToIgnore changed the title Remove timeline names Remove timeline names from pageserver Feb 28, 2022
@SomeoneToIgnore SomeoneToIgnore force-pushed the kb/remove-branch-files branch 3 times, most recently from 8a42a86 to e137591 Compare March 4, 2022 10:05
@SomeoneToIgnore SomeoneToIgnore force-pushed the kb/remove-branch-files branch 4 times, most recently from 9d505ac to 0990992 Compare March 6, 2022 19:55
@SomeoneToIgnore
Copy link
Contributor Author

CI tests pass now in the corresponding console PR: https://github.com/zenithdb/console/pull/782
I've reworked pageserver's http api further by

  • removing timeline creation from the tenant creation: we have a normal way to create an ancestor-less timeline via http now, so the client can decide when to do it)
  • returning 200 on tenant recreation: with the timeline logic removed, tenant creation is simple enough to return early on recreate attempts, assuming that the existing repo is fine (details)

Console code shrinked too, yet the currently do not have names in the logic at all, compared to our zenith cli implementation now.
Otherwise, I do not plan to touch those PRs, besides rebases and feedback addressing.

Currently, I plan to merge this PR due to mostly general feedback before, after its console counterpart will be ready for merging.
Please share your feedback in either of the PRs, if you have any.

@SomeoneToIgnore SomeoneToIgnore force-pushed the kb/remove-branch-files branch from dcb5c13 to 4da977a Compare March 7, 2022 22:02
Copy link
Contributor

@LizardWizzard LizardWizzard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks nice!

I've left some minor comments

@SomeoneToIgnore SomeoneToIgnore force-pushed the kb/remove-branch-files branch 2 times, most recently from 0015ca1 to b9c0415 Compare March 9, 2022 22:40
@SomeoneToIgnore SomeoneToIgnore force-pushed the kb/remove-branch-files branch from b9c0415 to 755fc52 Compare March 10, 2022 15:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants