Release of Canton 2.8.12
Canton 2.8.12 has been released on January 24, 2025. You can download the Daml Open Source edition from the Daml Connect Github Release Section. The Enterprise edition is available on Artifactory.
Please also consult the full documentation of this release.
Summary
This is a maintenance release that fixes one high and five medium severity issues. Please update during the next maintenance window.
What’s New
Memory check during node startup
A memory check has been introduced when starting the node. This check compares the memory allocated to the container
with the -Xmx JVM option.
The goal is to ensure that the container has sufficient memory to run the application.
To configure the memory check behavior, add one of the following to your configuration:
canton.parameters.startup-memory-check-config = warn // Default behavior: Logs a warning.
canton.parameters.startup-memory-check-config = crash // Terminates the node if the check fails.
canton.parameters.startup-memory-check-config = ignore // Skips the memory check entirely.
Minor Improvements
- Fixed one issue preventing a participant to connect to an old domain even if they support a common protocol version.
- Fixed a minor issue where the
validUntiltime of the topology transaction results was incorrectly set tovalidFrom
on the console client side. - Disabled the onboarding timeout for participants to support onboarding to domains with very large topology states
without annoying warnings and timeouts. - Removed warnings about failing periodic acknowledgements during initial domain onboarding of participants.
- Removed warnings about unhealthy sequencers during startup.
Bugfixes
(24-022, Medium): Participant replica does not clear package service cache
Issue Description
When a participant replica becomes active, it does not refresh the package dependency cache. If a vetting attempt is
made on the participant that fails because the package is not uploaded, the "missing package" response is cached.
If the package is then uploaded to another replica, and we switch to the original participant, this package service
cache will still record the package as nonexistent. When the package is used in a transaction, we will get a local
model conformance error as the transaction validator cannot find the package, whereas other parts of the participant
that don't use the package service can successfully locate it.
Affected Deployments
Participant
Affected Versions
2.8.0-2.8.11, 2.9.0-2.9.4
Impact
Replica crashes during transaction validation.
Symptom
Validating participant emits warning:
LOCAL_VERDICT_FAILED_MODEL_CONFORMANCE_CHECK(5,a2b60642): Rejected transaction due to a failed model conformance check: UnvettedPackages
And then emits an error:
An internal error has occurred.
java.lang.IllegalStateException: Mediator approved a request that we have locally rejected
Workaround
Restart recently active replica.
Likeliness
Likely to happen in any replicated participant setup with frequent vetting attempts and switches between active and
passive replicated participants between those vetting attempts.
Recommendation
Users are advised to upgrade to the next patch release during their maintenance window.
(24-028, Medium): ACS export and party replication is broken after hard domain migration
Issue Description
The macros for the various steps for migrating a party look up domain parameters in the topology store, but don't filter
out irrelevant domains. This results in the macro throwing an error because it finds multiple domain parameters after a
hard domain migration, even though one of them comes from an inactive domain.
Affected Deployments
Participant
Affected Versions
2.8.0-2.8.11, 2.9.0-2.9.5
Impact
You cannot migrate a party to or from a participant that went through a hard domain migration.
Symptom
Calling repair.party_migration.step1_store_acs fails with the error "Found more than one (2) domain parameters set for
the given domain and time!".
Workaround
The non-self-service work-around is to not call the party migration macros but replicate what these macros do.
Likeliness
The issue consistently occurs when calling the party migration macros after a hard domain migration.
Recommendation
Upgrade the involved participant nodes to the next patch release: 2.8.12 or 2.9.6.
(24-029, Medium): Domain topology manager gets stuck on too large batches
Issue Description
An off by one check fails in the topology dispatcher of the domain manager as
batches are not limited to N but to N+1, while we check for N.
Affected Deployments
Domain and Domain topology manager nodes.
Affected Versions
All versions before 2.8, 2.8.0-2.8.11, 2.9.0-2.9.5
Impact
Topology transactions stop being propagated through the system.
Symptom
Participants cannot onboard to domains, parties do not appear on the domain, uploaded dars cannot be used.
Workaround
Restart domain topology manager.
Likeliness
Can happen under high topology management load which is rather unusual (adding thousands of parties at full speed).
Recommendation
Update during the next maintenance window.
(25-001, Medium): Newly onboarded participants may compute a wrong topology state during bootstrapping
Issue Description
When a participant is onboarded to a domain, the domain manager will send the topology state to the participant. The
topology state is split into batches of 100. If the state contains an add and a subsequent remove of a topology transaction,
and these two topology transactions are in the same batch (so less than 100 transactions apart), but the namespace certificate
or identifier delegation is in a previous batch, then the participant will miss the removal of the topology transaction.
In the common cases, the namespace delegation is always followed by a subsequent add, but it can happen.
Affected Deployments
Participant
Affected Versions
All versions before 2.8, 2.8.0-2.8.11, 2.9.0-2.9.5
Impact
Depends on the type of topology transaction, but the result is a fork in the topology state, which in a rare but theoretically
possible case (observer nodes and participant using previously removed parties) might create a ledger fork, leading to
participants disconnecting from the domain.
Symptom
If the missed transaction was a mediator domain state, then the participant will fail to submit transactions whenever it
randomly selects the non-existent mediator.
Workaround
No workaround available. Manually repairing the topology state is likely possible, but not recommended.
Likeliness
Happens deterministically if the conditions are met, but the conditions are rare and require a specific sequence of
events with removal of topology state.
Recommendation
Upgrade before removing topology state (disabling parties, rolling keys) or onboarding a new participant to a domain
with a larger number of topology transactions that includes removals.
(25-002, Medium): Intermediate certificate renewal will delete topology state
Issue Description
A Canton node uses topology keys to sign topology transactions. The ultimate trust is tied to the root node key,
which by default is held by the node, but can be moved offline. In such a case, the node may use an intermediate
certificate to manage the topology state. In order to renew such intermediate certificates, the topology state needs
to be re-issued in 2.x, which can be done using the convenience function node.topology.all.renew(oldKey, newKey).
The convenience function contains an error that will instead of renewing the topology state, delete topology transactions
of the type party to participant, mediator domain state and participant domain state (the ones that contain the
replaceExisting flag).
Affected Deployments
Domain, Domain manager, Participant nodes.
Affected Versions
All versions before 2.8, 2.8.0-2.8.11, 2.9.0-2.9.5
Impact
Some of the topology state will be removed after running this operation.
Symptom
Parties, participants and mediators will be missing after running the operation.
Workaround
Manually re-add the missing parties, participants and mediators.
Likeliness
Deterministic if the convenience function is used.
Recommendation
Upgrade before renewing intermediate certificates.
(25-003, High): Identifier delegation cannot be renewed
Issue Description
A Canton node uses topology keys to sign topology transactions. The ultimate trust is tied to the root node key,
which by default is held by the node, but can be moved offline. In such a case, the node may use an intermediate
certificate to manage the topology state. If such an intermediate certificate is used to sign an identifier delegation
(used as an intermediate certificate for a specific uid), then the identifier delegation cannot be renewed,
as the renewal operation will remove the old and the new certificate from the in-memory state. Unfortunately,
after a restart, the certificate could be loaded again which can cause a ledger fork.
Affected Deployments
Domain, Domain manager, Participant nodes.
Affected Versions
All versions before 2.8, 2.8.0-2.8.11, 2.9.0-2.9.5
Impact
The topology state signed with a particular key authorized by an identifier delegation will be removed from the state,
and the key cannot be used to sign new transactions. After a restart of a node, the key would be loaded again, leading
to a possible ledger fork.
Symptom
Topology state missing after an intermediate certificate renewal, with a possible subsequent ledger fork after a restart.
Workaround
Theoretically issue a new identifier delegation for a new key and re-create the topology state. In practice, upgrade
all nodes before renewing intermediate certificates.
Likeliness
Deterministic if several intermediate certificates are used and one of them is rolled in the chain.
Recommendation
Update all nodes to a version with a fix before renewing intermediate certificates.
Compatibility
The following Canton protocol versions are supported:
| Dependency | Version |
|---|---|
| Canton protocol versions | 3, 4, 5 |
Canton has been tested against the following versions of its dependencies:
| Dependency | Version |
|---|---|
| Java Runtime | OpenJDK 64-Bit Server VM Zulu11.70+15-CA (build 11.0.22+7-LTS, mixed mode) |
| Postgres | Recommended: PostgreSQL 12.22 (Debian 12.22-1.pgdg120+1) – Also tested: PostgreSQL 11.16 (Debian 11.16-1.pgdg90+1), PostgreSQL 13.18 (Debian 13.18-1.pgdg120+1), PostgreSQL 14.15 (Debian 14.15-1.pgdg120+1), PostgreSQL 15.10 (Debian 15.10-1.pgdg120+1) |
| Oracle | 19.20.0 |