Skip to content

Commit

Permalink
[skip actions]-release: v0.95.4
Browse files Browse the repository at this point in the history
  • Loading branch information
github-actions[bot] committed Sep 25, 2024
1 parent 4bcb323 commit 463f7a6
Show file tree
Hide file tree
Showing 8 changed files with 57 additions and 47 deletions.
4 changes: 2 additions & 2 deletions packages/backend/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@synthql/backend",
"type": "module",
"version": "0.94.4",
"version": "0.95.4",
"main": "build/src/index.cjs",
"module": "build/src/index.js",
"types": "build/types/src/index.d.ts",
Expand All @@ -28,7 +28,7 @@
"benchmarks": "yarn vitest run src/tests/benchmarks/bench.test.ts"
},
"dependencies": {
"@synthql/queries": "0.94.4",
"@synthql/queries": "0.95.4",
"kysely": "^0.27.3",
"pg": "^8.11.3",
"sql-formatter": "^15.0.2"
Expand Down
6 changes: 3 additions & 3 deletions packages/cli/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@synthql/cli",
"type": "module",
"version": "0.94.4",
"version": "0.95.4",
"main": "build/src/index.cjs",
"module": "build/src/index.js",
"types": "build/types/src/index.d.ts",
Expand All @@ -27,8 +27,8 @@
"format": "yarn prettier --config ../../prettier.config.js --write ./src/"
},
"dependencies": {
"@synthql/introspect": "0.94.4",
"@synthql/queries": "0.94.4",
"@synthql/introspect": "0.95.4",
"@synthql/queries": "0.95.4",
"ajv": "^8.17.1",
"extract-pg-schema": "5.1.1",
"pg": "^8.11.3",
Expand Down
68 changes: 39 additions & 29 deletions packages/docs/blog/2024-09-15-rfc-runtime-validation/index.md
Original file line number Diff line number Diff line change
@@ -1,26 +1,30 @@
---
slug: rfc-runtime-validation
title: "RFC: Runtime validation of QueryResult objects"
title: 'RFC: Runtime validation of QueryResult objects'
authors: [fhur]
tags: [rfc]
---

# Introduction

One of the big productivity boosts that SynthQL gives you is the ability to convert your database schema into TypeScript types.

To achieve this, we give you `synthql generate`, which on the surface works pretty well, but has a few big issues:

## 1. Schema drift

Not all PG types can be mapped to TS types manually. Some need to be narrowed manually, for example any JSONB type.

We support overriding types via `synthql.config.json`, but the mechanism is fundamentally unsafe: whatever type you write, correct or not, is compiled to TypeScript. We never actually check that data coming from the database conforms to this type, so over time this introduces the possibility of schema drift.

## 2. Schema override DX
Another annoying issue with SynthQL is the DX for overriding types.

Another annoying issue with SynthQL is the DX for overriding types.
The main issue I have is that the current `.json` based format doesn't let you re-use types, and so if you have a `MonetaryValue` type (for modelling numbers with currencies), you will end up duplicating that JSON Schema type over and over.

## 3. Views
Views turn out to be tremendously useful in SynthQL as they can surmount any limitation on the QueryBuilder's expressiveness.

Views turn out to be tremendously useful in SynthQL as they can surmount any limitation on the QueryBuilder's expressiveness.
If you have some query that you can't express with SynthQL, you can first express it as as SQL view, add that view to the `synthql.config.json`, and query it as if it were a table.
There is one problem with views: all the columns are nullable.

Expand All @@ -29,12 +33,14 @@ There is one problem with views: all the columns are nullable.
The idea is to validate a percentage of all response rows based on a JSON Schema derived from the Query instance.

## How does sampling work?

When a QueryEngine is constructed, we can pass a number indicating how many rows should be validated.

```tsx
// for backend validation
new QueryEngine({
// Validate 5% of all QueryResults
// A sample rate of 0 disables it,
// A sample rate of 0 disables it,
// A sample rate of 1 runs over all results
runtimeValidationSampleRate:0.05
})
Expand All @@ -46,56 +52,60 @@ If `runtimeValidationSampleRate > 0`, then we should iterate over the resulting

```tsx
for (const row of rows) {
if (Math.random() <= runtimeValidationSampleRate) {
validateRow(row)
}
if (Math.random() <= runtimeValidationSampleRate) {
validateRow(row);
}
}
```
```

## What do we validate against?

We start by creating a function that given a query, and the DB's schema, returns the JSON Schema for the result of that query.

```tsx
function getQueryResultSchema(query: AnyQuery, schema:Schema): JSONSchema
function getQueryResultSchema(query: AnyQuery, schema: Schema): JSONSchema;
```

This function should be quite straightforward to build.

## Where should we run validation?

There's two possibilities: we either run it in the backend, or in the frontend.

I think we should run it in the frontend. The reason being that the backend could in theory have a schema that is correct, but the frontend has a different schema, in which case you would get no validation errors.
I think we should run it in the frontend. The reason being that the backend could in theory have a schema that is correct, but the frontend has a different schema, in which case you would get no validation errors.

This is not a theoretical case, this can happen if the frontend is on an old version. We often have cases in Luminovo where the frontend is running a version from several weeks ago.

# Putting it all together

```tsx
function createValidator(
query: AnyQuery,
schema: Schema,
sampleRate:number
query: AnyQuery,
schema: Schema,
sampleRate: number,
): (queryResult) => boolean {
if (sampleRate === 0) {
return () => true;
}
if (sampleRate === 0) {
return () => true;
}
// this schema should be cached based on
// the query.hash
const rowSchema = getCachedJsonSchema(query, schema);

return (queryResult:unknown) => {
const rows = Array.isArray(queryResult) ? queryResult : [queryResult];
for (const row of rows) {
validateWithSampleRate(row,rowSchema,sampleRate)
}
}
return (queryResult: unknown) => {
const rows = Array.isArray(queryResult) ? queryResult : [queryResult];
for (const row of rows) {
validateWithSampleRate(row, rowSchema, sampleRate);
}
};
}

const validateQueryResult = createValidator({
query,
schema,
sampleRate
query,
schema,
sampleRate,
});

const queryEngine = new QueryEngine()
const queryResult = queryEngine.executeAndWait(query)
validateQueryResult(queryResult)
```
const queryEngine = new QueryEngine();
const queryResult = queryEngine.executeAndWait(query);
validateQueryResult(queryResult);
```
6 changes: 3 additions & 3 deletions packages/handler-express/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@synthql/handler-express",
"type": "module",
"version": "0.94.4",
"version": "0.95.4",
"main": "build/src/index.cjs",
"module": "build/src/index.js",
"types": "build/types/src/index.d.ts",
Expand All @@ -27,8 +27,8 @@
"format": "yarn prettier --config ../../prettier.config.js --write ./src/"
},
"dependencies": {
"@synthql/backend": "0.94.4",
"@synthql/queries": "0.94.4"
"@synthql/backend": "0.95.4",
"@synthql/queries": "0.95.4"
},
"devDependencies": {
"@types/express": "^4.17.21",
Expand Down
6 changes: 3 additions & 3 deletions packages/handler-next/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@synthql/handler-next",
"type": "module",
"version": "0.94.4",
"version": "0.95.4",
"main": "build/src/index.cjs",
"module": "build/src/index.js",
"types": "build/types/src/index.d.ts",
Expand All @@ -27,8 +27,8 @@
"format": "yarn prettier --config ../../prettier.config.js --write ./src/"
},
"dependencies": {
"@synthql/backend": "0.94.4",
"@synthql/queries": "0.94.4"
"@synthql/backend": "0.95.4",
"@synthql/queries": "0.95.4"
},
"devDependencies": {
"@vitest/coverage-v8": "^1.2.2",
Expand Down
4 changes: 2 additions & 2 deletions packages/introspect/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@synthql/introspect",
"type": "module",
"version": "0.94.4",
"version": "0.95.4",
"main": "build/src/index.cjs",
"module": "build/src/index.js",
"types": "build/types/src/index.d.ts",
Expand Down Expand Up @@ -31,7 +31,7 @@
},
"dependencies": {
"@apidevtools/json-schema-ref-parser": "^11.5.4",
"@synthql/queries": "0.94.4",
"@synthql/queries": "0.95.4",
"extract-pg-schema": "^5.1.1",
"json-schema-to-typescript": "^13.1.2"
},
Expand Down
2 changes: 1 addition & 1 deletion packages/queries/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@synthql/queries",
"type": "module",
"version": "0.94.4",
"version": "0.95.4",
"main": "build/src/index.cjs",
"module": "build/src/index.js",
"types": "build/types/src/index.d.ts",
Expand Down
8 changes: 4 additions & 4 deletions packages/react/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@synthql/react",
"type": "module",
"version": "0.94.4",
"version": "0.95.4",
"main": "build/src/index.cjs",
"module": "build/src/index.js",
"types": "build/types/src/index.d.ts",
Expand All @@ -27,9 +27,9 @@
"compile-executable-examples": "node ../../scripts/compile-executable-examples.cjs ./src/useSynthql.test.tsx"
},
"dependencies": {
"@synthql/backend": "0.94.4",
"@synthql/handler-express": "0.94.4",
"@synthql/queries": "0.94.4"
"@synthql/backend": "0.95.4",
"@synthql/handler-express": "0.95.4",
"@synthql/queries": "0.95.4"
},
"devDependencies": {
"@tanstack/react-query": "^4",
Expand Down

0 comments on commit 463f7a6

Please sign in to comment.