Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

9.0: Correctly set Correlation and Causation IDs #3887

Open
skurfuerst opened this issue Sep 9, 2022 · 7 comments
Open

9.0: Correctly set Correlation and Causation IDs #3887

skurfuerst opened this issue Sep 9, 2022 · 7 comments
Labels

Comments

@skurfuerst
Copy link
Member

skurfuerst commented Sep 9, 2022

  • from 1 Command, multiple Events occur (Create Node with tethered child nodes)
    • Causation id (seems to be already impl.; not yet tested)
  • Publishing of nodes
    • Correlation ID for full publish process for ALL involved events.

Causation ID == "Command ID" (we don't store it, just conceptually)


PROBLEMS / concepts:

  • Publishing (applying events to other stream)
  • Rebase (applying COMMANDS to other stream) - maybe later "applying events to other stream, but they can fail"
  • Node with tethered children (multiple events from one command)

Intuition:

  • Causation ID should not change once set.
@bwaidelich
Copy link
Member

Our current idea, as discussed:

  • initiatingUserId, initiatingTimestamp and causationId are copied from initial events (Note: currently from the first event of a batch, which has the command attached to it in the metadata))
  • correlationId is always set/overridden on publishing/rebasing (= workspace related operations)

Example

Authenticated User "user1"

Create Node "home"

with tethered node "home/main"

  • NodeAggregateWasCreated

    • stream: "cs-user1-a"
    • id: "e1"
    • initiatingUserId: "user1"
    • initiatingTimestamp "t1"
    • causationId "causationId1" (generated) (TODAY: not set)
    • correlationId: ~
  • NodeAggregateWasCreated "home/main" (tethered)

    • stream: "cs-user1-a"
    • id "e2"
    • initiatingUserId "user1"
    • initiatingTimestamp "t1'"
    • causationId "causationId1" (generated) (TODAY: "e1")
    • correlationId: ~

Authenticated User "user2"

Create node "service"

  • NodeAggregateWasCreated "service"
    • stream: "cs-user2-a"
    • id "e3"
    • initiatingUserId "user2"
    • initiatingTimestamp "t2"
    • causationId "causationId2" (generated)
    • correlationId: ~

Authenticated User "user3"

Publishing of "user2" changes

just imagined, this is not really possible – but it's relevant for review workspaces

  • no rebase necessary (TODAY: we always rebase(?))

  • => copying of events from "cs-user2-a" to "cs-live"

  • NodeAggregateWasCreated "service"

    • stream: "cs-live"
    • id "e4"
    • initiatingUserId "user2" (copied from the original event "e3")
    • initiatingTimestamp "t2" (copied from the original event "e3")
    • causationId "causationId2" (copied from the original event "e3")
    • correlationId: "correlationId1"
  • WorkspaceWasPublished

    • stream: "workspace-live"
    • id: "e5"
    • initiatingUserId "user3"
    • initiatingTimestamp "t3"
    • causationId (?)
    • correlationId: "correlationId1" (generated for this publishing process)

Authenticated User "user1"

Rebasing

  • From "cs-user1-a" to "cs-user1-b" (because base content stream ("live") has new events)

  • TODAY: Re-handle commands, future hopefully: Re-apply events (can fail)

  • NodeAggregateWasCreated "home"

    • stream: "cs-user1-b"
    • id "e6"
    • initiatingUserId "user1" (copied from event "e1")
    • initiatingTimestamp "t1" (copied from event "e1")
    • causationId "causationId1" (copied from event "e1")
    • correlationId: "correlationId2"
  • NodeAggregateWasCreated "home/main" (tethered)

    • stream: "cs-user1-a"
    • id "e7"
    • initiatingUserId "user1" (copied from event "e1" because the command is attached to "e1", FUTURE: copied from "e2")
    • initiatingTimestamp "t1'" (copied from event "e1" because the command is attached to "e1", FUTURE: copied from "e2")
    • causationId "causationId1" (copied from event "e1" because the command is attached to "e1", FUTURE: copied from "e2")
    • correlationId: "correlationId2"
  • WorkspaceWasRebased

    • stream: "workspace-user1"
    • id "e8"
    • initiatingUserId "user1" (NOT copied!)
    • initiatingTimestamp "t4"
    • causationId (?)
    • correlationId: "correlationId2" (generated for this rebasing process)

Publishing of "user1" changes to "live"

  • NodeAggregateWasCreated "home"

    • stream: "cs-live"
    • id "e9"
    • initiatingUserId "user1" (copied from event "e6")
    • initiatingTimestamp "t1" (copied from event "e6")
    • causationId "causationId1" (copied from event "e6")
    • correlationId: "correlationId3"
  • NodeAggregateWasCreated "home/main" (tethered)

    • stream: "cs-live"
    • id "e10"
    • initiatingUserId "user1" (copied from event "e6" because the command is attached to "e1", FUTURE: copied from "e7")
    • initiatingTimestamp "t1'" (copied from event "e6" because the command is attached to "e1", FUTURE: copied from "e7")
    • causationId "causationId1" (copied from event "e6" because the command is attached to "e1", FUTURE: copied from "e7")
    • correlationId: "correlationId3"
  • WorkspaceWasPublished

    • stream: "workspace-live"
    • id: "e11"
    • initiatingUserId "user1"
    • initiatingTimestamp "t5"
    • causationId (?)
    • correlationId: "correlationId3" (generated for this publishing process)

bwaidelich added a commit that referenced this issue Sep 9, 2022
Makes the `UserIdProvider` a concept of the core package so that
the user id can be determined "lazily".

In practice this means that we can use the CR instance for _reading_
already before the authenticated user can be determined.

When handling commands, an exception will be thrown if no user can
be determined at this point.

This also adjusts the event decoration in `\Neos\ContentRepository\Core\ContentRepository::handle()`
such that `initiatingUserId` and `initiatingTimestamp` is not overridden
once it is set (see #3887)
@mhsdesign mhsdesign added the 9.0 label Jun 11, 2023
@bwaidelich
Copy link
Member

After an issue @ahaeslich had today with lots of ContentStreamWasForked events without corresponding Workspace events (see Slack discussion) this missing feature gets more critical and made me think:
The causationId does not have to be a UUID but it could contain some pre/suffix explaining the corresponding interaction, for example:
rebaseWorkspace-62a2c5d06b834252b686e00ce6aa2548.

If it turns out to be too much work to introduce the complete causation/correlation handling for now, we should at least add some property to the ForkContentStream command (and the corresponding event), e.g.:

ForkContentStream::create(
    $rebasedContentStream,
    $baseWorkspace->currentContentStreamId,
    sprintf('Rebase workspace "%s" (content stream "%s"), $command->workspaceName->value, $command->contentStreamId->value),
)

@mhsdesign
Copy link
Member

Ah this issue also made it to our list of big questions. But i dont really understand what is being discussed here, and it troubles me as this topic is marked critical ^^

@bwaidelich
Copy link
Member

it troubles me as this topic is marked critical

Don't worry. This is critical for debugging the event log, but at least the last suggestion should be very easy to add and without any breaking changes

@bwaidelich
Copy link
Member

If it turns out to be too much work to introduce the complete causation/correlation handling for now,
we should at least add some property to the ForkContentStream command

As discussed, I created a separate high-prio issue (#4758) for that part and remove this one from the priority list

@bwaidelich
Copy link
Member

bwaidelich commented Apr 28, 2024

Reminder: We should also set some kind of "originalEventId" metadata to events published to other content streams in order to correlate those.

This will allow us to detect "merges" rather than new events, i.e. instead of

%%{init: { 'logLevel': 'debug', 'theme': 'base', 'gitGraph': {'showBranches': true, 'showCommitLabel':true,'mainBranchName': 'Live', 'parallelCommits': false, 'rotateCommitLabel': true}} }%%
      gitGraph LR:
        branch "content stream A"
        commit id:"Node A created"
        commit id:"Node A properties set"
        commit id:"content stream A closed"
        branch "content stream B"
        commit id:"Content A stream forked'"
        commit id:"Node A created'"
        commit id:"Node A properties set'"
        commit id: "content stream B closed"
        branch "content stream C"
        commit id:"content stream B forked"
        commit id:"Node A created''"
        commit id:"Node A properties set''"
        checkout Live
        commit id:"Node A created'''"
        commit id:"Node A properties set'''"
        checkout "content stream C"
        commit id:"Content stream C removed"
        checkout "content stream B"
        commit id:"content stream B removed"
Loading

we could produce sth like

%%{init: { 'logLevel': 'debug', 'theme': 'base', 'gitGraph': {'showBranches': true, 'showCommitLabel':true,'mainBranchName': 'Live', 'parallelCommits': false, 'rotateCommitLabel': true}} }%%
      gitGraph LR:
        checkout Live
        commit id:"created"
        branch "content stream A"
        commit id:"Node A created"
        commit id:"Node A properties set"
        commit id:"content stream A closed"
        branch "content stream B"
        commit id:"content stream A forked'"
        merge "content stream A"
        checkout "content stream B"
        commit id: "content stream B closed"
        branch "content stream C"
        commit id:"content stream B forked"
        merge "content stream B"
        checkout Live
        merge "content stream C"
        checkout "content stream C"
        commit id:"content stream C removed"
        checkout "content stream B"
        commit id:"content stream B removed"
Loading

mhsdesign added a commit to mhsdesign/neos-development-collection that referenced this issue May 13, 2024
mhsdesign added a commit to mhsdesign/neos-development-collection that referenced this issue Jan 26, 2025
The refactored publishing V3 opens another now used advantage: For commands with multiple events like workspace publishing, we can now centrally add metadata like a simple correlation id.

Note that for 'simple' commands we dont need to do this as `RebaseableCommand::enrichWithCommand` will already group them with a causationId which is what we decided to use for tethered nodes for example. The id never changes for causation ids.
mhsdesign added a commit to mhsdesign/neos-development-collection that referenced this issue Jan 27, 2025
The refactored publishing V3 opens another now used advantage: For commands with multiple events like workspace publishing, we can now centrally add metadata like a simple correlation id.

Note that for 'simple' commands we dont need to do this as `RebaseableCommand::enrichWithCommand` will already group them with a causationId which is what we decided to use for tethered nodes for example. The id never changes for causation ids.
@mhsdesign
Copy link
Member

With #5451 this issues is partly solved as that change introduces debug metadata for the Fork events and sets correlation ids based on the parts of the command name and a random suffix which allows to easily spot a rebase, discard or publish in the event log.

mhsdesign added a commit to mhsdesign/neos-development-collection that referenced this issue Feb 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: No status
Development

No branches or pull requests

3 participants