Branch creation is possibly not persisted

[yeputons]

The initial discussion is here: #746 (comment)

I was able to reproduce the issue as well. Now I'm wondering what happens if the Compute Node goes down immediately after reporing successful branch creation back to the user: will Safekeepers preserve all necessary information about the branch or it will be lost?

Creating a branch does not create a new transaction, so it probably does not affect WAL, hence there is no immediate need to wait for WAL confirmation. We do not lose WAL, but losing branch information may be not great as well.

[kelvich]

cc @arssher @petuhovskiy

Branch creation happens in a following manner:

1. branch_create api call on the pageserver initializes new timeline
2. new compute node is started with that timeline_id
3. compute node generates some wal records which are sent to safekeepers

so from the safekeeper's point of view new branch == new node, so it is okay to don't know about a new timeline until first wal record arrives. Is anything wrong with this approach?

[arssher]

Yes, it ought to be okay as long as no one requests parent branch WAL from safekeepers, which I assume to be the case.

"branch information" appears on safekeeper only after first START_WAL_PUSH, and LSNs (flush, commit) are 0 until the first record (or real commit_lsn) arrives. If anything is confused by these 0, we can transfer them to timeline creation, but is it?

[yeputons]

It seems to me that information about branches is stored on pageserver only, isn't it?

If so, the following failure scenario looks possible:

1. A user creates a branch newbr. Pageserver creates all necessary files locally, but does not fsync them (it currently uses plain std::fs::write at least somewhere).
2. (optional) The user makes some small transactions.
3. The compute node persists data to safekeepers and reports the transaction as successful.
4. Page server goes down.
5. Information about newbr is completely lost, despite there are some WALs in Safekeepers.
6. Page server goes back up, but now the user cannot query against newbr as it's lost.

Severity depends on whether we recommend accessing branches by their names or timeline ids:

• In the former case it will be a simple failure like "branch not found" (and maybe some garbage stored on Safekeepers, possibly either eating up into user's quota or not eating up; not sure what's worse).
• In the latter case there may be some more sophisticated failure as Safekeepers try to push (or callmemaybe) WAL to a Pageserver and it does not know that such a branch even exists.

[arssher]

Indeed, pageserver should persist branch creation, likely even to s3.

[stepashka]

cc @kelvich

[stepashka]

@arssher, taking into account the new API for tenant creation, is this fixed by any chance?

[arssher]

Timeline creation API on safekeepers is not used yet.

Seems like the only thing to "fix" here is persisting branch creation on pageserver in fault tolerant way. It is minor and not urgent.

[LizardWizzard]

I think the primary source of truth regarding branches is console. Console creates branches on the pageserver, retries the corresponding api call in case of possible failures and so on.

Also after #1286 branch name mapping is removed from pageserver.
Then there are two cases for branch creation:

• Creation of the root branch. It involves calling to initdb and it generates a bunch of files. This operation is not transactional, so if it is interrupted in the middle the state is unpredictable. In that case we can only delete this branch and create a new one.
• Branching off the existing branch. Requires only metadata file creation, uses fsync + checksum. If metadata write is interrupted we can detect it and recreate timeline if needed.

[LizardWizzard]

Indeed, pageserver should persist branch creation, likely even to s3.

When new root branch is created pageserver performs a checkpoint which is uploaded to s3. Child branch will appear on s3 on first checkpoint.

[yeputons]

TODO for me: is this still actual? I suspect it still is: there is still an explicit timeline creation API in Pageserver.

[LizardWizzard]

FYI pageserver now uses more complicated init flow with uninit mark file for root timelines and for branches it is just a metadata file which is 512 bytes (one sector) and we run fsync on it and its parent directory. So should be fine as long as we dont return response earlier than the data is written

[shanyp]

@arssher is this still relevant? if not I could close https://github.com/neondatabase/cloud/issues/854

[arssher]

Most likely not (see last comment), let's close it.