Skip to content

Commit

Permalink
add doc for stateful application failover
Browse files Browse the repository at this point in the history
Signed-off-by: changzhen <changzhen5@huawei.com>
  • Loading branch information
XiShanYongYe-Chang committed Nov 30, 2024
1 parent d38d08f commit 677b335
Show file tree
Hide file tree
Showing 2 changed files with 88 additions and 1 deletion.
39 changes: 39 additions & 0 deletions docs/userguide/failover/application-failover.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,45 @@ spec:
You can edit `suppressDeletion` to false in `gracefulEvictionTasks` to evict the application in the failed cluster after you confirm the failure.

## Stateful Application Failover Support

Starting from v1.12, the application-level failover feature adds support for stateful application failover, it provides a generalized way for users to define application state preservation in the context of cluster-to-cluster failovers.

In releases prior to v1.12, Karmada’s scheduling logic runs on the assumption that resources that are scheduled and rescheduled are stateless. In some cases, users may desire to conserve a certain state so that applications can resume from where they left off in the previous cluster. For CRDs dealing with data-processing (such as Flink or Spark), it can be particularly useful to restart applications from a previous checkpoint. That way applications can seamlessly resume processing data while avoiding double processing.

### Defining StatePreservation

`StatePreservation` is a field under `.spec.failover.application`, it defines the policy for preserving and restoring state data during failover events for stateful applications. When an application fails over from one cluster to another, this policy enables the extraction of critical data from the original resource configuration.

It contains a list of `StatePreservationRule` configurations. Each rule specifies a JSONPath expression targeting specific pieces of state data to be preserved during failover events. An `AliasLabelName` is associated with each rule, serving as a label key when the preserved data is passed to the new cluster. You can define the state preservation policy:

```yaml
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
name: example-propagation
spec:
#...
failover:
application:
decisionConditions:
tolerationSeconds: 60
purgeMode: Immediately
statePreservation:
rules:
- aliasLabelName: pre-updated-replicas
jsonPath: "{ .updatedReplicas }"
```

The above configuration will parse the `updatedReplicas` field from the application `.status` before migration. Upon successful migration, the extracted data is then re-injected into the new resource, ensuring that the application can resume operation with its previous state intact.

This capability requires enabling the `StatefulFailoverInjection` feature gate. `StatefulFailoverInjection` is currently in `Alpha` and is turned off by default.

> There are currently some restrictions on the use of this function, please pay attention when using it:
> 1. Only the scenario where an application is deployed in one cluster and migrated to another cluster is considered.
> 2. If consecutive failovers occur, for example, an application is migrated form clusterA to clusterB and then to clusterC, the PreservedLabelState before the last failover is used for injection. If the PreservedLabelState is empty, the injection is skipped.
> 3. The injection operation is performed only when PurgeMode is set to Immediately.

:::note

Application failover is still a work in progress. We are in the progress of gathering use cases. If you are interested in this feature, please feel free to start an enhancement issue to let us know.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -190,9 +190,57 @@ spec:
You can edit `suppressDeletion` to false in `gracefulEvictionTasks` to evict the application in the failed cluster after you confirm the failure.

## 无状态应用 Failover 支持

Starting from v1.12, the application-level failover feature adds support for stateful application failover, it provides a generalized way for users to define application state preservation in the context of cluster-to-cluster failovers.

In releases prior to v1.12, Karmada’s scheduling logic runs on the assumption that resources that are scheduled and rescheduled are stateless. In some cases, users may desire to conserve a certain state so that applications can resume from where they left off in the previous cluster.

For CRDs dealing with data-processing (such as Flink or Spark), it can be particularly useful to restart applications from a previous checkpoint. That way applications can seamlessly resume processing data while avoiding double processing.

从 v1.12 开始,应用故障转移特性增加了对有状态应用故障转移的支持,它为用户提供了一种通用的方式来定义在集群间故障转移情境下的应用状态保留。

在 v1.12 之前的版本中,Karmada 的调度逻辑基于这样的假设运行:被调度和重新调度的资源是无状态的。在某些情况下,用户可能希望保留某种状态,以便应用可以从之前集群中停止的地方恢复。对于处理数据加工的 CRD(例如 Flink 或 Spark),从之前的检查点重新启动应用可能特别有用。这样应用可以无缝地恢复数据处理,同时避免重复处理。

### 定义 StatePreservation

`StatePreservation` 是 `.spec.failover.application` 下的一个字段, 它定义了在有状态应用的故障转移事件期间保留和恢复状态数据的策略。当应用从一个集群故障转移到另一个集群时,此策略使得能够从原始资源配置中提取关键数据。

它包含了一系列 `StatePreservationRule` 配置。每条规则指定了一个 JSONPath 表达式,针对特定的状态数据片段,在故障转移事件中需要保留。每条规则都关联了一个 `AliasLabelName`,当保留的数据传递到新集群时,它作为标签键使用。你可以定义状态保留策略:

```yaml
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
name: example-propagation
spec:
#...
failover:
application:
decisionConditions:
tolerationSeconds: 60
purgeMode: Immediately
statePreservation:
rules:
- aliasLabelName: pre-updated-replicas
jsonPath: "{ .updatedReplicas }"
```

The above configuration will parse the `updatedReplicas` field from the application `.status` before migration. Upon successful migration, the extracted data is then re-injected into the new resource, ensuring that the application can resume operation with its previous state intact.

This capability requires enabling the `StatefulFailoverInjection` feature gate. `StatefulFailoverInjection` is currently in `Alpha` and is turned off by default.

上述配置将在迁移前解析应用 `.status` 中的 `updatedReplicas` 字段。成功迁移后,提取的数据将重新注入到新资源中,确保应用程序能够以其之前的状态恢复操作。

此功能需要启用 `StatefulFailoverInjection` 特性门控。`StatefulFailoverInjection` 目前处于 `Alpha` 阶段,默认情况下是关闭的。

> 目前该功能的使用尚有些限制,使用时请注意:
> 1. 仅考虑应用程序在一个集群中部署并迁移到另一个集群的场景。
> 2. 如果发生连续故障转移,例如,应用程序从 clusterA 迁移到 clusterB,然后再到 clusterC,最后一次故障转移前的 PreservedLabelState 用于注入。如果 PreservedLabelState 为空,则跳过注入。
> 3. 仅当 PurgeMode 设置为 Immediately 时,才执行注入操作。

:::note

Application failover is still a work in progress. We are in the progress of gathering use cases. If you are interested in this feature, please feel free to start an enhancement issue to let us know.

:::

0 comments on commit 677b335

Please sign in to comment.