feat: refactor entities #345

thantos · 2023-05-08T05:51:34Z

Entities now support a more classical entity pattern, more in line with Dynamo where the partition key and optional sort key must be provided at creation time.

The key is a composite key provided in the following ways:

Tuple - [partition, sort] - Delete/Get
Map - { [partitionField]: value, [sortField]?: value } - Delete/Get
In the value - { [partitionField]: value, [sortField]?: value } - Set

When a key is returned, we always use the Map format (or the in the value format for get).

Caveat 1: the schema is now required as I was struggling to get the types to work with just the type parameters.
Caveat 2: existing tables must be renamed and recreated

const myEntity = entity("myEntity", { 
   schema: { key: z.string(), value: z.number() },
   partitionKey: "key" });
const myEntity2 = entity("myEntity2", { 
   schema: { part: z.string(), key: z.string(), value: z.number() }, 
   partitionKey: "part", 
   sortKey: "key"});
// to use numbers or binary as keys, provide the type explictly
const myEntity3 = entity("myEntity3", { 
   schema: { key: z.number(), value: z.number() }, 
   partitionKey: { key: "key", type: "number" }});

const { version } = await myEntity.set({ key: "aValue", value: 1 });
await myEntity.set({ key: "aValue", value: 1 }, { expectedVersion: version  });
await myEntity.get({ key: "aValue" });
await myEntity.get(["aValue"]);
await myEntity.delete({ key: "aValue" });

await myEntity.query({ partition: "aValue" }); // without a sort key, only one value is returned.

sam-goodwin · 2023-05-08T05:52:20Z

Show some examples in the PR description?

apps/tests/aws-runtime/test/test-service.ts

sam-goodwin · 2023-05-08T19:30:04Z

Not sure what's going on with the way we configure partition/sort keys. Should be an array of field names

const myEntity3 = entity("myEntity3", { 
  attributes: { 
    key: z.number(),
    value: z.number(),
    createTime: z.date()
  }, 
  partition: ["key"],
  sort: ["value", "createTime"]
});

await myEntity3.query({
  key: "value",
  value: 1,
  createdTime: {
    startsWith: "2024-01-01T00:00Z",
    between: ["2024", "2025"]
  }
});

const myIndex = myEntity3.index({
  partition: ["key"],
  sort: ["value", "createTime"],
});

sam-goodwin · 2023-05-08T19:56:41Z

Some help with the types for capturing order of PK/SK

type Shape = {
  [attribute in string]: z.ZodType;
};

type Attributes = z.ZodObject<any> | Shape;

type AttributeNames<Attr extends Attributes> = Attr extends z.ZodObject<infer A>
  ? keyof A
  : keyof Attr;

export interface Entity2<
  Attr extends z.ZodObject<any> | Shape,
  Partition extends readonly AttributeNames<Attr>[],
  Sort extends readonly AttributeNames<Attr>[] | undefined = undefined
> {}

export function entity2<
  const Attr extends z.ZodObject<any> | Shape,
  const Partition extends readonly AttributeNames<Attr>[],
  const Sort extends readonly AttributeNames<Attr>[] | undefined = undefined
>(input: {
  attributes: Attr;
  partition: Partition;
  sort?: Sort;
}): Entity2<Attr, Partition, Sort> {
  throw new Error();
}

const a = entity2({
  attributes: z.object({
    key: z.string(),
    id: z.string(),
  }),
  partition: ["key", "id"],
  // sort: ["key", "id"],
});

packages/@eventual/core/src/entity.ts

sam-goodwin · 2023-05-11T16:39:12Z

packages/@eventual/core/src/entity.ts

-export interface Entity<E> extends Omit<EntitySpec, "schema" | "streams"> {
+export type AnyEntity = Entity<any, string, string | undefined>;
+
+export type EntityKeyType = string | number | EntityBinaryMember;


Don't suffix with Type

sam-goodwin · 2023-05-11T16:40:12Z

packages/@eventual/core/src/entity.ts

+export type EntityKeyField<E extends EntityValue> = {
+  [K in keyof E]: K extends string
+    ? E[K] extends EntityKeyType
+      ? K
+      : never
+    : never;
+}[keyof E];


Why not just keyof E or Extract<keyof E, string>?

sam-goodwin · 2023-05-11T16:45:25Z

packages/@eventual/core/src/entity.ts

+    : never;
+}[keyof E];
+
+export type EntityCompositeKeyFromEntity<E extends AnyEntity> =


Should AnyEntity be replaced with default type arguments?

sam-goodwin · 2023-05-11T16:57:37Z

packages/@eventual/core/src/entity.ts

+  partitionKey: EntityKeyReference<E, P>;
+  sortKey?: EntityKeyReference<E, S>;


Why are these not just partition: P? Types are a bit out of control.

Usually good practice to avoid overly using mapped types, especially when all you're doing is storing the data directly on the object.

sam-goodwin · 2023-05-11T16:57:56Z

packages/@eventual/core/src/entity.ts

-  streams: EntityStream<E>[];
+  partitionKey: EntityKeyReference<E, P>;
+  sortKey?: EntityKeyReference<E, S>;
+  schema?: ZodMappedSchema<E>;


What is a ZodMappedSchema?

Ok, so it constructs the schema from the value type. Shouldn't we be including the zod types in everything? Go backwards from schema?

sam-goodwin · 2023-05-11T16:58:33Z

packages/@eventual/core/src/entity.ts

+export type ZodMappedSchema<E extends EntityValue> = {
+  [k in keyof E]: z.Schema<E[k]>;
+};


Why is this type needed?

sam-goodwin · 2023-05-11T16:59:52Z

packages/@eventual/core/src/entity.ts

+export interface EntityOptions<
+  E extends EntityValue,
+  P extends EntityKeyField<E>,
+  S extends EntityKeyField<E> | undefined
+> {
+  partitionKey: EntityKeyReference<E, P>;
+  sortKey?: EntityKeyReference<E, S>;
+  schema: ZodMappedSchema<E>;
+}


This type is surprisingly complex.

export interface EntityOptions< A extends EntityAttributes | z.ZodObject<any>, P extends keyof E | (keyof E)[], S extends keyof E | (keyof E)[] | undefined = undefined > { partitionKey: P; sortKey?: S; attributes: A }

sam-goodwin · 2023-05-11T17:02:50Z

packages/@eventual/core/src/entity.ts

+> {
+  partitionKey: EntityKeyReference<E, P>;
+  sortKey?: EntityKeyReference<E, S>;
+  schema: ZodMappedSchema<E>;


Shouldn't this be called attributes?

sam-goodwin · 2023-05-11T17:12:27Z

packages/@eventual/core/src/entity.ts

   *
   * @default - retrieve values with no namespace.
   */
-  namespace?: string;
+  partition: E[Partition];


This seems like it only accepts a single value?

Have not pushed a commit since the last time we talked. Should be ready soon.

sam-goodwin · 2023-05-11T17:36:38Z

packages/@eventual/core/src/internal/entity-hook.ts

+export type EntityMethod = Exclude<
+  {
+    [k in keyof AnyEntity]: [AnyEntity[k]] extends [Function] ? k : never;
+  }[keyof AnyEntity],
+  "partitionKey" | "sortKey" | "stream" | "__entityBrand" | undefined
 >;

 /**
 * Registers and returns functioning {@link Entity}s.
 *
 * Does not handle the workflow case. That is handled by the {@link entity} function in core.
 */
-export interface EntityHook {
-  getEntity<Entity>(name: string): Promise<EntityMethods<Entity> | undefined>;
+export type EntityHook = {
+  [K in EntityMethod]: (
+    entityName: string,
+    ...args: Parameters<AnyEntity[K]>
+  ) => ReturnType<AnyEntity[K]>;
+} & {
  transactWrite(items: EntityTransactItem<any>[]): Promise<void>;
-}
+};


Why do we need these types? It's hard to follow

sam-goodwin · 2023-05-11T18:34:10Z

packages/@eventual/core/src/entity.ts

+export type EntityQueryKey<
+  Attr extends EntityAttributes = EntityAttributes,
+  Partition extends EntityCompositeKeyPart<Attr> = EntityCompositeKeyPart<Attr>,
+  Sort extends EntityCompositeKeyPart<Attr> | undefined =
+    | EntityCompositeKeyPart<Attr>
+    | undefined
+> = Partial<EntityCompositeKey<Attr, Partition, Sort>>;


export type EntityQueryKey< Attr extends EntityAttributes = EntityAttributes, Partition extends EntityCompositeKeyPart<Attr> = EntityCompositeKeyPart<Attr>, Sort extends EntityCompositeKeyPart<Attr> | undefined = | EntityCompositeKeyPart<Attr> | undefined > = { [pk in Partition[number]]: string; } & (Sort extends undefined ? {} : ProgressiveQueryKey<Attr, Exclude<Sort, undefined>>); type ProgressiveQueryKey< Attr extends EntityAttributes, Sort extends readonly (keyof Attr)[], Accum extends object = {} > = Sort extends [] ? Accum : Sort extends [ infer k extends keyof Attr, ...infer ks extends (keyof Attr)[] ] ? | (Accum & { [sk in k]: Attr[sk]; }) | ProgressiveQueryKey< Attr, ks, Accum & { [sk in Sort[0]]: Attr[sk]; } > : never;

packages/@eventual/core/src/entity/entity.ts

thantos added 3 commits May 6, 2023 01:45

building!

b50dbe9

support alternate key types

b7ef962

fix bugs and stuff

f0afada

thantos marked this pull request as draft May 8, 2023 05:52

sam-goodwin reviewed May 8, 2023

View reviewed changes

apps/tests/aws-runtime/test/test-service.ts Outdated Show resolved Hide resolved

apps/tests/aws-runtime/test/test-service.ts Outdated Show resolved Hide resolved

apps/tests/aws-runtime/test/test-service.ts Outdated Show resolved Hide resolved

thantos added 2 commits May 8, 2023 01:17

simplify schema input

f3a6040

tests are passing

b12b002

thantos marked this pull request as ready for review May 8, 2023 16:08

thantos requested a review from sam-goodwin May 8, 2023 16:08

clean up

34bbe57

thantos added 9 commits May 8, 2023 16:48

Merge branch 'main' into sussman/feat/entity_2

9dbda36

building with array keys

e86059a

tests pass

dad6a89

add tests for sort keys

5e8d5e0

reduce use of zod types

8bb2e88

improved type names

e1a879c

clean up key definition computation

cc010b0

Abstract entity store shared logic.

98fb53f

more types

d0ecc94

sam-goodwin requested changes May 11, 2023

View reviewed changes

thantos added 2 commits May 11, 2023 12:57

organization

ab02f33

more cleanup

54a08c7

sam-goodwin requested changes May 11, 2023

View reviewed changes

thantos added 3 commits May 11, 2023 13:43

tests passing

05c30cc

more things

93eb94b

fix bug and rename

4145566

change test table name

299965f

thantos requested a review from sam-goodwin May 11, 2023 21:59

sam-goodwin approved these changes May 11, 2023

View reviewed changes

packages/@eventual/core/src/entity/entity.ts Outdated Show resolved Hide resolved

packages/@eventual/core/src/entity/entity.ts Outdated Show resolved Hide resolved

feedback and refactor transact item

2e4b9c8

thantos merged commit b1c39b1 into main May 12, 2023

thantos deleted the sussman/feat/entity_2 branch May 12, 2023 02:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: refactor entities #345

feat: refactor entities #345

thantos commented May 8, 2023 •

edited

Loading

sam-goodwin commented May 8, 2023

sam-goodwin commented May 8, 2023 •

edited

Loading

sam-goodwin commented May 8, 2023

sam-goodwin May 11, 2023

sam-goodwin May 11, 2023

sam-goodwin May 11, 2023

sam-goodwin May 11, 2023

sam-goodwin May 11, 2023

sam-goodwin May 11, 2023

sam-goodwin May 11, 2023

sam-goodwin May 11, 2023

sam-goodwin May 11, 2023

sam-goodwin May 11, 2023

thantos May 11, 2023

sam-goodwin May 11, 2023

sam-goodwin May 11, 2023

		partitionKey: EntityKeyReference<E, P>;
		sortKey?: EntityKeyReference<E, S>;

feat: refactor entities #345

feat: refactor entities #345

Conversation

thantos commented May 8, 2023 • edited Loading

sam-goodwin commented May 8, 2023

sam-goodwin commented May 8, 2023 • edited Loading

sam-goodwin commented May 8, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thantos commented May 8, 2023 •

edited

Loading

sam-goodwin commented May 8, 2023 •

edited

Loading