Skip to content

Commit

Permalink
feat: diff results to text
Browse files Browse the repository at this point in the history
  • Loading branch information
peko-thunder committed Jan 4, 2024
1 parent c5d333c commit 61a95cf
Show file tree
Hide file tree
Showing 14 changed files with 551 additions and 170 deletions.
5 changes: 3 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
coverage/
npm/
coverage/*
npm/*
!npm/README.md
153 changes: 5 additions & 148 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,155 +1,11 @@
# diff-unique-record

データレイクなど膨大なデータの中から差分を求められることが多々ある
これはオブジェクト型の配列データを新旧で比較して、その差分をプログラムで確認するためのツールです。
データレイクなどの膨大なデータの管理業務を行っていると、調査などでデータの新旧差分を出したい時がある
新旧のオブジェクト型の配列データを比較して、その差分をプログラムで確認するためのツールです。

注意点として階層の深いオブジェクトや配列には現時点で対応していません
階層の深いオブジェクトや配列には対応しておらず、単一階層のレコードを比較することを想定しています

## diffサンプル

実際に新旧のデータを投入して差分を抽出した結果です。

### 差分取得方法

新旧のデータで共通したユニークな`keys`を設定することで比較を行っています。サンプルでは`date`,`name`の組み合わせをユニークと定義しています。

```typescript
const result = diff({
old: oldDataList,
new: newDataList,
keys: ["date", "name"],
})
```

### oldDataList

```javascript
const oldDataList = [
// object/key unchanged
{
date: "2023-12-01",
name: "sales",
value: 100,
},
// object/key updated
{
date: "2023-12-01",
name: "rate",
value: 20,
},
// object removed
{
date: "2023-12-02",
name: "sales",
value: 80,
},
// key removed/added
{
date: "2023-12-04",
name: "sales",
value: 90,
oldtmp: "test",
},
]
```

### newDataList

```javascript
const newDataList = [
// object/key unchanged
{
date: "2023-12-01",
name: "sales",
value: 100,
},
// object/key updated
{
date: "2023-12-01",
name: "rate",
value: 30,
},
// object added
{
date: "2023-12-03",
name: "sales",
value: 90,
},
// key removed/added
{
date: "2023-12-04",
name: "sales",
value: 70,
newtmp: "test",
},
]
```

### results

```javascript
const results = [
{
type: "updated",
old: { date: "2023-12-01", name: "rate", value: 20 },
new: { date: "2023-12-01", name: "rate", value: 30 },
keys: [
{
key: "date",
type: "unchanged",
old: "2023-12-01",
new: "2023-12-01",
},
{ key: "name", type: "unchanged", old: "rate", new: "rate" },
{ key: "value", type: "updated", old: 20, new: 30 },
],
},
{
type: "unchanged",
old: { date: "2023-12-01", name: "sales", value: 100 },
new: { date: "2023-12-01", name: "sales", value: 100 },
keys: [
{
key: "date",
type: "unchanged",
old: "2023-12-01",
new: "2023-12-01",
},
{ key: "name", type: "unchanged", old: "sales", new: "sales" },
{ key: "value", type: "unchanged", old: 100, new: 100 },
],
},
{
type: "removed",
old: { date: "2023-12-02", name: "sales", value: 80 },
new: undefined,
keys: [],
},
{
type: "added",
old: undefined,
new: { date: "2023-12-03", name: "sales", value: 90 },
keys: [],
},
{
type: "updated",
old: { date: "2023-12-04", name: "sales", value: 90, oldtmp: "test" },
new: { date: "2023-12-04", name: "sales", value: 70, newtmp: "test" },
keys: [
{
key: "date",
type: "unchanged",
old: "2023-12-04",
new: "2023-12-04",
},
{ key: "name", type: "unchanged", old: "sales", new: "sales" },
{ key: "value", type: "updated", old: 90, new: 70 },
{ key: "oldtmp", type: "removed", old: "test", new: undefined },
{ key: "newtmp", type: "added", old: undefined, new: "test" },
],
},
]
```
使い方はこちらの[ドキュメント](./npm/README.md)を参照ください。

## Deno環境テンプレート

Expand Down Expand Up @@ -223,4 +79,5 @@ npm install -g @commitlint/cli @commitlint/config-conventional

```bash
deno task build-npm ${version}
npm publish ./npm
```
5 changes: 3 additions & 2 deletions mod/diff.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ export const diff = <O extends object, N extends object>(
const newData = newKeyMap.get(key)
const diffKey = getDiffKey(oldData, newData)

// Map.get() がundefinedの場合はデータ自体が存在しない
let diffType: DiffType = "unchanged"
if (oldData === undefined && newData) diffType = "added"
if (oldData && newData === undefined) diffType = "removed"
Expand All @@ -28,7 +29,7 @@ export const diff = <O extends object, N extends object>(
old: oldData,
new: newData,
keys: diffKey,
}
} as DiffResult<O, N>
})

return results
Expand Down Expand Up @@ -103,7 +104,7 @@ export const getDiffKey = <O extends object, N extends object>(
if (oldIndex.value === newIndex.value) diffType = "unchanged"

return {
key,
name: key,
type: diffType,
old: oldIndex.value,
new: newIndex.value,
Expand Down
1 change: 1 addition & 0 deletions mod/index.ts
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
export { diff } from "./diff.ts"
export { generateDiffText } from "./parser/index.ts"
export * from "./type.ts"
14 changes: 14 additions & 0 deletions mod/parser/AbstractDiffParser.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
import { DiffResultBase } from "../type.ts"

export default abstract class AbstractDiffParser {
abstract getDiff<O extends object, N extends object>(
result: DiffResultBase<O, N>,
): string

protected toJsonValue = (value: unknown): string | number | undefined => {
if (typeof value === "string") return `"${value}"`
if (typeof value === "number") return value

return undefined
}
}
21 changes: 21 additions & 0 deletions mod/parser/AddedObjectParser.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
import { AddedDiffResult } from "../type.ts"
import AbstractDiffParser from "./AbstractDiffParser.ts"

export default class AddedObjectParser extends AbstractDiffParser {
constructor(private indent: string) {
super()
}

public getDiff<O extends object, N extends object>(
result: AddedDiffResult<O, N>,
): string {
let text = ""
text += `+ {\n`
Object.entries(result.new).forEach(([key, value]) => {
text += `+ ${this.indent}${key}: ${this.toJsonValue(value)},\n`
})
text += `+ },\n`

return text
}
}
43 changes: 43 additions & 0 deletions mod/parser/KeyParser.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
import { DiffKey, ExistedDiffResult } from "../type.ts"
import AbstractDiffParser from "./AbstractDiffParser.ts"

export default class KeyParser extends AbstractDiffParser {
constructor(private indent: string) {
super()
}

public getDiff<O extends object, N extends object>(
result: ExistedDiffResult<O, N>,
) {
let text = ""
text += `${this.indent}{\n`
result.keys.forEach((key) => {
text += this.parseDetail(key)
})
text += `${this.indent}},\n`

return text
}

private parseDetail<O extends object, N extends object>(
key: DiffKey<O, N>,
): string {
const keyName = String(key.name)
const oldValue = this.toJsonValue(key.old)
const newValue = this.toJsonValue(key.new)

if (key.type === "added") {
return `+ ${this.indent}${keyName}: ${newValue},\n`
}
if (key.type === "removed") {
return `- ${this.indent}${keyName}: ${oldValue},\n`
}
if (key.type === "unchanged") {
return ` ${this.indent}${keyName}: ${oldValue},\n`
}
const removed = `- ${this.indent}${keyName}: ${oldValue},\n`
const added = `+ ${this.indent}${keyName}: ${newValue},\n`

return `${removed}${added}`
}
}
21 changes: 21 additions & 0 deletions mod/parser/RemovedObjectParser.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
import { RemovedDiffResult } from "../type.ts"
import AbstractDiffParser from "./AbstractDiffParser.ts"

export default class RemovedObjectParser extends AbstractDiffParser {
constructor(private indent: string) {
super()
}

public getDiff<O extends object, N extends object>(
result: RemovedDiffResult<O, N>,
): string {
let text = ""
text += `- {\n`
Object.entries(result.old).forEach(([key, value]) => {
text += `- ${this.indent}${key}: ${this.toJsonValue(value)},\n`
})
text += `- },\n`

return text
}
}
26 changes: 26 additions & 0 deletions mod/parser/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
import { DiffResult } from "../type.ts"
import AddedObjectParser from "./AddedObjectParser.ts"
import KeyParser from "./KeyParser.ts"
import RemovedObjectParser from "./RemovedObjectParser.ts"

export const generateDiffText = <O extends object, N extends object>(
results: DiffResult<O, N>[],
): string => {
const INDENT = " "
const parser = {
added: new AddedObjectParser(INDENT),
removed: new RemovedObjectParser(INDENT),
key: new KeyParser(INDENT),
}

let text = "[\n"
results.forEach((result) => {
if (result.type === "added") text += parser.added.getDiff(result)
if (result.type === "removed") text += parser.removed.getDiff(result)
if (result.type === "unchanged") text += parser.key.getDiff(result)
if (result.type === "updated") text += parser.key.getDiff(result)
})
text += "]"

return text
}
46 changes: 40 additions & 6 deletions mod/type.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,47 @@ export interface DiffParam<O extends object, N extends object> {
/**
* 新旧オブジェクトを比較した結果
*/
export interface DiffResult<O extends object, N extends object> {
export type DiffResult<O extends object, N extends object> =
| AddedDiffResult<O, N>
| RemovedDiffResult<O, N>
| ExistedDiffResult<O, N>

/**
* 比較結果のベースインターフェース
*/
export interface DiffResultBase<O extends object, N extends object> {
type: DiffType
keys: DiffKey<O, N>[]
// TODO: deep object やarrayの対応をするかどうか 一旦はシンプルな作りにする
// test: O[keyof O] extends object ? DiffObject<O, N> : DiffKey<O, N>[]
old?: O
new?: N
}

/**
* 追加判定の比較結果
*/
export interface AddedDiffResult<O extends object, N extends object>
extends DiffResultBase<O, N> {
type: "added"
old: undefined
new: N
}

/**
* 削除判定の比較結果
*/
export interface RemovedDiffResult<O extends object, N extends object>
extends DiffResultBase<O, N> {
type: "removed"
old: O
new: undefined
}

/**
* 既存判定の比較結果
*/
export interface ExistedDiffResult<O extends object, N extends object>
extends DiffResultBase<O, N> {
type: "updated" | "unchanged"
old: O
new: N
}

/**
Expand All @@ -28,7 +62,7 @@ export type DiffType = "added" | "removed" | "updated" | "unchanged"
* 新旧オブジェクト内のkeyを比較した結果
*/
export interface DiffKey<O extends object, N extends object> {
key: keyof O | keyof N
name: keyof O | keyof N
type: DiffType
old?: O[keyof O]
new?: N[keyof N]
Expand Down
Loading

0 comments on commit 61a95cf

Please sign in to comment.