Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(ava/advisor): init new advisor pipeline & built-in modules to plugin-based architecture #784

Open
wants to merge 6 commits into
base: refactor-advisor-pipeline
Choose a base branch
from
Prev Previous commit
Next Next commit
fix(ava/advisor): add advisor export types
chenluli committed Jun 17, 2024
commit 3d3244745b839df551252177c9a32d0649096625
Original file line number Diff line number Diff line change
@@ -2,15 +2,15 @@ import { getChartTypeRecommendations } from './get-chart-Type';

import type {
AdvisorPipelineContext,
ChartTypeRecommendInputParams,
ChartTypeRecommendInput,
ChartTypeRecommendOutput,
PluginType,
AdvisorPluginType,
} from '../../../../types';

export const chartTypeRecommendPlugin: PluginType<ChartTypeRecommendInputParams, ChartTypeRecommendOutput> = {
export const chartTypeRecommendPlugin: AdvisorPluginType<ChartTypeRecommendInput, ChartTypeRecommendOutput> = {
name: 'defaultChartTypeRecommend',
stage: ['chartTypeRecommend'],
execute(input: ChartTypeRecommendInputParams, context?: AdvisorPipelineContext): ChartTypeRecommendOutput {
execute(input: ChartTypeRecommendInput, context?: AdvisorPipelineContext): ChartTypeRecommendOutput {
const { dataProps } = input;
const { advisor, options } = context || {};
const chartTypeRecommendations = getChartTypeRecommendations({
Original file line number Diff line number Diff line change
@@ -3,9 +3,14 @@ import { cloneDeep } from 'lodash';
import { getDataProps } from './get-data-properties';
import { getSelectedData } from './get-selected-data';

import type { AdvisorPipelineContext, DataProcessorInput, DataProcessorOutput, PluginType } from '../../../../types';
import type {
AdvisorPipelineContext,
DataProcessorInput,
DataProcessorOutput,
AdvisorPluginType,
} from '../../../../types';

export const dataProcessorPlugin: PluginType<DataProcessorInput, DataProcessorOutput> = {
export const dataProcessorPlugin: AdvisorPluginType<DataProcessorInput, DataProcessorOutput> = {
name: 'defaultDataProcessor',
stage: ['dataAnalyze'],
execute: (input: DataProcessorInput, context: AdvisorPipelineContext): DataProcessorOutput => {
Original file line number Diff line number Diff line change
@@ -4,12 +4,17 @@ import { DEFAULT_COLOR } from '../../../constants';
import { applyDesignRules, applySmartColor, applyTheme } from './spec-processors';
import { getChartTypeSpec } from './get-chart-spec';

import type { AdvisorPipelineContext, SpecGeneratorInput, SpecGeneratorOutput, PluginType } from '../../../../types';
import type {
AdvisorPipelineContext,
SpecGeneratorInput,
SpecGeneratorOutput,
AdvisorPluginType,
} from '../../../../types';

export const specGeneratorPlugin: PluginType<SpecGeneratorInput, SpecGeneratorOutput> = {
// todo 内置的 visualEncode 和 spec generate 插件需要明确支持哪些图表类型
export const specGeneratorPlugin: AdvisorPluginType<SpecGeneratorInput, SpecGeneratorOutput> = {
name: 'defaultSpecGenerator',
stage: ['specGenerate'],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

如果注册多个 plugin 作用于同一个 stage 它的表现会如何,顺序是怎么管理的?
问这个问题是看到下面操作生成 spec 其实也是有一个生成 or 操作顺序的,那么其实可以拆多个 plugin 实现,实现业务自定义 theme 之类的需求

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

之前想的是,一个 component 里的 plugin 都是并行的(输入和输出都被约束为一致的),所以没有声明它的执行顺序。
如果需要所说的有一个顺序执行的,感觉可以后续扩展 componet 的定义,如上面的框图所示, component 有点是执行管理器的概念,可以是用扩展定义其类型和嵌套来实现?
Screenshot 2024-07-08 at 18 01 06

// todo 目前上一步输出是一个图表列表数组,这里原子能力实际上应该是只生成 spec
execute: (input: SpecGeneratorInput, context: AdvisorPipelineContext): SpecGeneratorOutput => {
const { chartTypeRecommendations, dataProps, data } = input;
const { options, advisor } = context || {};
@@ -18,12 +23,15 @@ export const specGeneratorPlugin: PluginType<SpecGeneratorInput, SpecGeneratorOu
const advices = chartTypeRecommendations
?.map((chartTypeAdvice) => {
const { chartType } = chartTypeAdvice;
const chartTypeSpec = getChartTypeSpec({
chartType,
data,
dataProps,
chartKnowledge: advisor.ckb[chartType],
});
const chartKnowledge = advisor.ckb[chartType];
const chartTypeSpec =
chartKnowledge?.toSpec(data, dataProps) ??
getChartTypeSpec({
chartType,
data,
dataProps,
chartKnowledge,
});

// step 3: apply spec processors such as design rules, theme, color, to improve spec
if (chartTypeSpec && refine) {
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import type { PluginType, VisualEncoderInput } from '../../../../types';
import type { AdvisorPluginType, VisualEncoderInput } from '../../../../types';

export const visualEncoderPlugin: PluginType<VisualEncoderInput, VisualEncoderInput> = {
export const visualEncoderPlugin: AdvisorPluginType<VisualEncoderInput, VisualEncoderInput> = {
name: 'defaultVisualEncoder',
stage: ['encode'],
execute: (input) => {
25 changes: 12 additions & 13 deletions packages/ava/src/advisor/advisor.ts
Original file line number Diff line number Diff line change
@@ -19,10 +19,10 @@ import type {
Lint,
AdvisorPipelineContext,
PipelineStageType,
PluginType,
AdvisorPluginType,
DataProcessorInput,
DataProcessorOutput,
ChartTypeRecommendInputParams,
ChartTypeRecommendInput,
ChartTypeRecommendOutput,
VisualEncoderInput,
VisualEncoderOutput,
@@ -43,32 +43,36 @@ export class Advisor {

dataAnalyzer: BaseComponent<DataProcessorInput, DataProcessorOutput>;

chartTypeRecommender: BaseComponent<ChartTypeRecommendInputParams, ChartTypeRecommendOutput>;
chartTypeRecommender: BaseComponent<ChartTypeRecommendInput, ChartTypeRecommendOutput>;

chartEncoder: BaseComponent<VisualEncoderInput, VisualEncoderOutput>;

specGenerator: BaseComponent<SpecGeneratorInput, SpecGeneratorOutput>;

context: AdvisorPipelineContext;

plugins: PluginType[];
plugins: AdvisorPluginType[];

pipeline: Pipeline;

constructor(
config: AdvisorConfig = {},
custom: {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

当前仅 constructor 输入 custom 不知道是否支持后续 实例动态注册?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

需要的,后续打算在 advisor 上增加 updateCkb, updateRuleBase, resigterComponents 等 api

plugins?: PluginType[];
plugins?: AdvisorPluginType[];
components?: BaseComponent[];
/** extra info to pass through the pipeline
* 额外透传到推荐 pipeline 中的业务信息
*/
extra?: Record<string, any>;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

同理,extra 直接放 ctx 上?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

是放在 context 上的,这里是 constructor 时进行初始化传参

} = {}
) {
// init
const { plugins, components, extra = {} } = custom;
this.ckb = ckb(config.ckbCfg);
this.ruleBase = processRuleCfg(config.ruleCfg);
this.context = { advisor: this };
this.context = { advisor: this, extra };
this.initDefaultComponents();
const defaultComponents = [this.dataAnalyzer, this.chartTypeRecommender, this.chartEncoder, this.specGenerator];
const { plugins, components } = custom;
this.plugins = plugins;
this.pipeline = new Pipeline({ components: components ?? defaultComponents });
}
@@ -83,11 +87,6 @@ export class Advisor {
this.specGenerator = new BaseComponent('specGenerate', { plugins: [specGeneratorPlugin], context: this.context });
}

// todo 定义多个链路串并联的方法
// private definePipeline(components: BaseComponent[]) {
// this.pipeline.components = components;
// }

// todo 暂时还在用旧链路,需要改造到新链路
advise(params: AdviseParams): Advice[] {
const adviseResult = dataToAdvices({ adviseParams: params, ckb: this.ckb, ruleBase: this.ruleBase });
@@ -119,7 +118,7 @@ export class Advisor {
return lintResult;
}

registerPlugins(plugins: PluginType[]) {
registerPlugins(plugins: AdvisorPluginType[]) {
const stage2Components: Record<PipelineStageType, BaseComponent> = {
dataAnalyze: this.dataAnalyzer,
chartTypeRecommend: this.chartTypeRecommender,
29 changes: 9 additions & 20 deletions packages/ava/src/advisor/pipeline/component.ts
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
import { AsyncParallelHook, SyncHook } from 'tapable';
import { last } from 'lodash';

import type { PluginType, AdvisorPipelineContext } from '../types';
import type { AdvisorPluginType, AdvisorPipelineContext } from '../types';

/** 收集多个 plugin 的输出结果 */
type PluginResultMap<Output = any> = Record<string, Output>;

export class BaseComponent<Input = any, Output = any> {
name: string;

plugins: PluginType<Input, Output>[] = [];
plugins: AdvisorPluginType<Input, Output>[] = [];

/** 是否存在异步插件 */
private hasAsyncPlugin: boolean;
@@ -18,15 +18,15 @@ export class BaseComponent<Input = any, Output = any> {

syncPluginManager: SyncHook<[Input, PluginResultMap<Output>], Output>;

afterPluginsExecute?: (params: PluginResultMap<Output>) => Output;
afterPluginsExecute?: (params: PluginResultMap<Output>, context?: AdvisorPipelineContext) => Output;

context?: AdvisorPipelineContext;

constructor(
name,
options?: {
plugins?: PluginType<Input, Output>[];
afterPluginsExecute?: (params: PluginResultMap<Output>) => Output;
plugins?: AdvisorPluginType<Input, Output>[];
afterPluginsExecute?: (params: PluginResultMap<Output>, context?: AdvisorPipelineContext) => Output;
context?: AdvisorPipelineContext;
}
) {
@@ -45,7 +45,7 @@ export class BaseComponent<Input = any, Output = any> {
return last(Object.values(params));
}

private isPluginAsync(plugin: PluginType) {
private isPluginAsync(plugin: AdvisorPluginType) {
// 检测插件是否为异步的,并设置hasAsyncPlugin标志位
if (plugin.execute.constructor.name === 'AsyncFunction') {
return true;
@@ -54,7 +54,7 @@ export class BaseComponent<Input = any, Output = any> {
}

// 处理 之前都是同步的插件,新追加注册一个异步的插件 的情况 -- 需要执行的地方就不能用
registerPlugin(plugin: PluginType) {
registerPlugin(plugin: AdvisorPluginType) {
plugin.onLoad?.(this.context);
this.plugins.push(plugin);
if (this.isPluginAsync(plugin)) {
@@ -94,22 +94,11 @@ export class BaseComponent<Input = any, Output = any> {
// console.warn('存在异步执行的插件,建议使用 executeAsync')
const pluginsOutput = {};
return this.pluginManager.promise(params, pluginsOutput).then(async () => {
return this.afterPluginsExecute?.(pluginsOutput);
return this.afterPluginsExecute?.(pluginsOutput, this.context);
});
}
const pluginsOutput = {};
this.syncPluginManager.call(params, pluginsOutput);
return this.afterPluginsExecute?.(pluginsOutput);
return this.afterPluginsExecute?.(pluginsOutput, this.context);
}

// todo 是否应该区分同步和异步接口
// executeAsync(params: Input): Promise<Output> {
// if(!this.hasAsyncPlugin) {
// console.warn('插件均同步执行,建议使用 execute')
// }
// const pluginsOutput = {};
// return this.pluginManager.promise(params, pluginsOutput).then(async () => {
// return this.afterPluginsExecute?.(pluginsOutput);
// });
// }
}
10 changes: 7 additions & 3 deletions packages/ava/src/advisor/types/component.ts
Original file line number Diff line number Diff line change
@@ -7,7 +7,7 @@ import type { MarkEncode } from './mark';
export type PipelineStageType = 'dataAnalyze' | 'chartTypeRecommend' | 'encode' | 'specGenerate';

/** 基础插件接口定义 */
export interface PluginType<Input = any, Output = any> {
export interface AdvisorPluginType<Input = any, Output = any> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

我有一个大胆的想法,如果 advice pipeline 是确定的,有几个 component 也是确定的。这条 pipeline 是不是都属于一个 ctx,那这条链路上的 input 和 output 其实都是从当前的 ctx 中获得,而不用区分?也就是说这里的 Input 和 Output 其实都是 ctx。况且看下面 stage 还有多个的情况。

Copy link
Member Author

@chenluli chenluli Jul 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

意思是说,pipeline context 里加上 input 和 output 属性么,这两个属性根据当前执行的环节进行更新?
每个环节的 input 和 output 可以挂到 ctx 上,不过前置环节的应该都需要挂上去,不能只挂当前环节的🤔️

每个 component 和 plugin 用类型约束,是想说用户在自定义自己的 plugin 时,能够明确需要的类型,都放 pipeline 感觉类型不好约束了?

/** 插件的唯一标识 */
name: string;
/** 插件运行的阶段,用于指定插件在 pipeline 的哪个环节运行 * */
@@ -30,7 +30,7 @@ export type DataProcessorOutput = {
dataProps: BasicDataPropertyForAdvice[];
};

export type ChartTypeRecommendInputParams = {
export type ChartTypeRecommendInput = {
dataProps: BasicDataPropertyForAdvice[];
};

@@ -43,7 +43,11 @@ export type SpecGeneratorInput = {
// 单独调用 SpecGenerator 时,还要额外计算 dataProps 么
dataProps: BasicDataPropertyForAdvice[];
};
export type SpecGeneratorOutput = { advices: Advice[] };
export type SpecGeneratorOutput = {
advices: (Omit<Advice, 'spec'> & {
spec: Record<string, any> | null;
})[];
};

export type VisualEncoderInput = {
chartType: ChartId;
6 changes: 3 additions & 3 deletions packages/ava/src/advisor/types/pipeline.ts
Original file line number Diff line number Diff line change
@@ -65,7 +65,9 @@ export type ChartAdviseParams = {
data: Data;
/** customized data props to advise */
dataProps?: Partial<BasicDataPropertyForAdvice>[];
/** @todo 确认下为啥这里和 options 里面都有 data fields to focus, apply in `data` and `dataProps` */
/** data fields to focus, apply in `data` and `dataProps`
* @todo 确认下为啥这里和 options 里面都有 是否应该废弃一处的
*/
fields?: string[];
/** advising options such as purpose, layout preferences */
options?: AdvisorOptions;
@@ -223,6 +225,4 @@ export type AdvisorPipelineContext = {
extra?: {
[key: string]: any;
};
/** 过程中的日志信息,开发调试用, todo 明确类型 */
logs?: { [key: string]: any }[];
};
8 changes: 8 additions & 0 deletions packages/ava/src/index.ts
Original file line number Diff line number Diff line change
@@ -17,6 +17,14 @@ export type {
LintParams,
RuleModule,
AdvisorConfig,
AdvisorPluginType,
AdvisorPipelineContext,
DataProcessorInput,
DataProcessorOutput,
ChartTypeRecommendInput,
ChartTypeRecommendOutput,
SpecGeneratorInput,
SpecGeneratorOutput,
} from './advisor';

/* CKB */