Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

codegen with swc on cross platform dev + docker not working #1496

Open
mrtom opened this issue Jun 20, 2023 · 35 comments
Open

codegen with swc on cross platform dev + docker not working #1496

mrtom opened this issue Jun 20, 2023 · 35 comments

Comments

@mrtom
Copy link
Contributor

mrtom commented Jun 20, 2023

I'm getting a load of errors of the form:

The 'this' context of type 'typeof Foo' is not assignable to method's 'this' of type 'new (viewer: Viewer<Ent<any> | null, ID | null>, data: Data) => FooBase'.
...
...
Property '__setRawDBData' is missing in type 'FooBase' but required in type 'Ent<Viewer<Ent<any> | null, ID | null>>'

I was using 0.1.0-alpha30, attempting to update to 0.1.0-alpha147 but that doesn't appear to have made a difference.

Unsure how best to diagnose. I can see the property exists in the latest version of the ent code (here).

Should I just remove all the generated files and re-gen? Is that safe?

Thanks :)

@mrtom
Copy link
Contributor Author

mrtom commented Jun 20, 2023

(I think I can just update the base classes myself easily enough to add in this one change, but that feels sketchy - I'd expect the framework to re-gen generated files for me)

@lolopinto
Copy link
Owner

I believe it's because the image needs to change too. can you try with the latest image 0.1.0-alpha.48-nodejs-18-dev

update your develop.Dockerfile:

FROM ghcr.io/lolopinto/ent:0.1.0-alpha.48-nodejs-18-dev

@lolopinto
Copy link
Owner

I just updated ent-starter images too

@mrtom
Copy link
Contributor Author

mrtom commented Jun 21, 2023

Thanks for the quick response.

Just so I'm clear on a couple of things:

  1. Has the __setRawDBData property been added or removed? Should I end up with it included if things go to plan?
  2. Should I expect the implementation of generated files to be overwritten as the ent framework evolves?

To your actual suggestion. I have tried upgrading as you suggested, but am running into problems. Here's what I've done:

  1. Upgraded by ent dependency: yarn add @snowtop/ent@0.1.0-alpha147
  2. Pulled the latest docker image: docker pull ghcr.io/lolopinto/ent:0.1.0-alpha.48-nodejs-18-dev
  3. Updated develop.Dockerfile: Replaced image to FROM ghcr.io/lolopinto/ent:0.1.0-alpha.48-nodejs-18-dev
  4. Updated the Dockerfile (although I'm not using that atm: FROM ghcr.io/lolopinto/ent:0.1.0-alpha.48-nodejs-18-slim
  5. Rebuilt the image: yarn run rebuild-image

Now, if I run eg yarn codegen I'm was getting an error about not being able to find swc/core. So I added that (yarn add -D @swc/core), and now I'm getting an error about missing bindings:

~/Code/backend (upgrade2*) » yarn codegen                                                                                                                                                                                                                        tom@Dunkertons
yarn run v1.22.19
$ docker-compose -f docker-compose.dev.yml run --rm app tsent codegen
Error: Bindings not found
    at Compiler.transformSync (/app/node_modules/@swc/core/index.js:250:15)
    at transformSync (/app/node_modules/@swc/core/index.js:348:21)
    at transformSync (/app/node_modules/@swc-node/core/index.ts:72:26)
    at compile (/app/node_modules/@swc-node/register/register.ts:102:40)
    at exts (/app/node_modules/@swc-node/register/register.ts:113:38)
    at Module._compile (/app/node_modules/pirates/lib/index.js:113:29)
    at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
    at Object.newLoader [as .ts] (/app/node_modules/pirates/lib/index.js:121:7)
    at Module.load (node:internal/modules/cjs/loader:1117:32)
    at Function.Module._load (node:internal/modules/cjs/loader:958:12)
Error:
  exit status 1
Usage:
  tsent [command]

Available Commands:

@mrtom
Copy link
Contributor Author

mrtom commented Jun 21, 2023

Oh, but that might be because I've not updated all the @gqlField bindings everywhere, which I think may have changed during the upgrade. BRB.

@mrtom
Copy link
Contributor Author

mrtom commented Jun 21, 2023

Are the docs here up to date?

i.e. should it be something like:

@gqlQuery({
    name: "viewer",
    type: ViewerType,
  })
export default class ViewerResolver {
  @gqlQuery({ name: "viewer", type: ViewerType })
  viewer(@gqlContextType() context: RequestContext): ViewerType {
    return new ViewerType(context.getViewer());
  }
}

(as per the docs), or

@gqlQuery({
    class: "ViewerType",
    name: "viewer",
    type: ViewerType,
    args: [
      gqlContextType(),
    ],
  })
export default class ViewerResolver {
  @gqlQuery({ name: "viewer", type: ViewerType })
  viewer(context: RequestContext): ViewerType {
    return new ViewerType(context.getViewer());
  }
}

Am assuming the latter, as per the simple example.

Will update my code assuming that's the case.

@lolopinto
Copy link
Owner

lolopinto commented Jun 21, 2023

Are the docs here up to date?

no :( there's #1406 to update them which I haven't done. will get on that soon.

it's the latter because I changed it to use new decorators in TS 5.0 since I kept running into issues with that and swc for faster codegen

#1405

@lolopinto
Copy link
Owner

Has the __setRawDBData property been added or removed? Should I end up with it included if things go to plan?

yes, you should end up with them

@lolopinto
Copy link
Owner

lolopinto commented Jun 21, 2023

Should I expect the implementation of generated files to be overwritten as the ent framework evolves?

yes
why files are broken up into generated/* and user editable files.

how can I make that clearer?

@lolopinto
Copy link
Owner

not sure what this missing bindings issue is? if you're still seeing them, can you add DISABLE_SWC=true in your docker-compose.dev.yml?

e.g. around https://github.com/lolopinto/ent/blob/main/examples/simple/docker-compose.dev.yml#L16

@lolopinto
Copy link
Owner

thanks for going through again. I'm about to release v0.1 and planned to do a documentation refresh around all the issues tagged with documentation

@mrtom
Copy link
Contributor Author

mrtom commented Jun 21, 2023

how can I make that clearer?

I'm not sure you need to. My assumption was that this would be the case, but having run into issues I was doubting it.

What I don't quite understand is that I started this project a couple of months ago using ghcr.io/lolopinto/ent:0.1.0-alpha.30-nodejs-17-dev, and this problem happened immediately.

So I don't think this is a bodged upgrade issue. Although it's possible I just happened to pick a version of the framework where this transition was in progress?

@mrtom
Copy link
Contributor Author

mrtom commented Jun 21, 2023

OK, adding the DISABLE_SWC flag has thrown up a new error when running yarn codegen. Looking into it now, but pasting here too for visibility:

~/Code/LimeAndSoda/roomtool-server (upgrade2*) » yarn codegen                                                                                                                                                                                                                    1 ↵ tom@Dunkertons
yarn run v1.22.19
$ docker-compose -f docker-compose.dev.yml run --rm app tsent codegen
checking for custom graphql definitions...
Error: Error: Cannot read properties of undefined (reading 'toString') loading /app/src/graphql/resolvers/organization.ts
    at /app/node_modules/@snowtop/ent/scripts/custom_graphql.js:286:15
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async requireFiles (/app/node_modules/@snowtop/ent/scripts/custom_graphql.js:273:5)
    at async captureCustom (/app/node_modules/@snowtop/ent/scripts/custom_graphql.js:238:9)
    at async Promise.all (index 1)
    at async main (/app/node_modules/@snowtop/ent/scripts/custom_graphql.js:371:42)
Error:
  error generating custom graphql: exit status 1
Usage:
  tsent [command]

FWIW I tried twice and had a different error the first time:

~/Code/LimeAndSoda/roomtool-server (upgrade2*) » yarn codegen                                                                                                                                                                                                                  130 ↵ tom@Dunkertons
yarn run v1.22.19
$ docker-compose -f docker-compose.dev.yml run --rm app tsent codegen
checking for custom graphql definitions...
Error: Error: Cannot read properties of undefined (reading 'kind') loading /app/src/graphql/resolvers/viewer.ts
    at /app/node_modules/@snowtop/ent/scripts/custom_graphql.js:286:15
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async requireFiles (/app/node_modules/@snowtop/ent/scripts/custom_graphql.js:273:5)
    at async captureCustom (/app/node_modules/@snowtop/ent/scripts/custom_graphql.js:238:9)
    at async Promise.all (index 1)
    at async main (/app/node_modules/@snowtop/ent/scripts/custom_graphql.js:371:42)
Error:
  error generating custom graphql: exit status 1
Usage:
  tsent [command]

@mrtom
Copy link
Contributor Author

mrtom commented Jun 21, 2023

Should also say that fixing up the gql field changes has fixed running the server with yarn start

@mrtom
Copy link
Contributor Author

mrtom commented Jun 21, 2023

I'm afraid I've got to bounce for the night - am on holiday and should probably spend some time with my wife :)

If useful, I've added you to the project on GH to have a look around - https://github.com/LimeAndSoda/roomtool-server/tree/upgrade2 is the branch that I'm running the upgrade in.

@lolopinto
Copy link
Owner

I'm afraid I've got to bounce for the night - am on holiday and should probably spend some time with my wife :)

If useful, I've added you to the project on GH to have a look around - https://github.com/LimeAndSoda/roomtool-server/tree/upgrade2 is the branch that I'm running the upgrade in.

cool thanks. will take a look later!

@lolopinto
Copy link
Owner

changes I made:

  • changed tsconfig.json to comment out these two:
    // "experimentalDecorators": true,
    // "emitDecoratorMetadata": true,

^ related to documentation changes with 5.0

in organization.ts:

changed from type: 'string' to

        type: "String",

I think those were the 2 big ones. everything else was small stuff.

had to create local db, had to change docker-compose.dev.yml to work for ola instead of tom and usual yarn codegen, yarn rebuild-image etc

@mrtom
Copy link
Contributor Author

mrtom commented Jun 22, 2023

Oh amazing, thank you!

Confirmed that's got both codegen and running the app working using the latest image.

Although I'm still having issues if I don't disable SWC. I'll keep that disabled for now, but let me know if you'd like any help/testing/whatever later with that. Thanks again!

@lolopinto
Copy link
Owner

can you try with this image

FROM ghcr.io/lolopinto/ent:0.1.0-alpha.49-test1-nodejs-18-dev

i made a small tweak and wanna see if it works? it works for me but i had done some debugging and not 100% sure if i somehow changed something independently

@mrtom
Copy link
Contributor Author

mrtom commented Jun 24, 2023

Steps:

  • Updated developer.Dockerfile, as above
  • Removed - DISABLE_SWC=true from docker-compose.dev.yaml
  • Ran docker pull ghcr.io/lolopinto/ent:0.1.0-alpha.49-test1-nodejs-18-dev from terminal
  • Ran yarn rebuild-image
  • Ran yarn codegen

Same issue as before I'm afraid:

~/Code/LimeAndSoda/roomtool-server (main*) » yarn codegen                                                                                                                                                                                                                            tom@Dunkertons
yarn run v1.22.19
$ docker-compose -f docker-compose.dev.yml run --rm app tsent codegen
Error: Bindings not found
    at Compiler.transformSync (/app/node_modules/@swc/core/index.js:250:15)
    at transformSync (/app/node_modules/@swc/core/index.js:348:21)
    at transformSync (/app/node_modules/@swc-node/core/index.ts:72:26)
    at compile (/app/node_modules/@swc-node/register/register.ts:102:40)
    at exts (/app/node_modules/@swc-node/register/register.ts:113:38)
    at Module._compile (/app/node_modules/pirates/lib/index.js:113:29)
    at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
    at Object.newLoader [as .ts] (/app/node_modules/pirates/lib/index.js:121:7)
    at Module.load (node:internal/modules/cjs/loader:1117:32)
    at Function.Module._load (node:internal/modules/cjs/loader:958:12)
Error:
  exit status 1
Usage:
  tsent [command]

Available Commands:
  alembic             alembic command
  codegen             runs the codegen (and db schema) migration
  completion          Generate the autocompletion script for the specified shell
  delete_schema       deletes the given schema
  detect_dangling     detects any dangling schema files
  downgrade           downgrade db
  fix-edges           fix edges db
  generate            generate a schema or other object
  help                Help about any command
  migrate_v0.1        migrate v0.1
  print_custom_schema prints the parsed custom graphql schema. only exists for debugging purposes and not guaranteed to exist forever
  print_schema        prints the parsed schema. only exists for debugging purposes and not guaranteed to exist forever
  run_script          run script
  squash              squash last N revs of the db into one
  upgrade             upgrade db

Flags:
      --debug         debug mode. add debug information to codegen e.g. how long commands take
      --debug_files   debug files mode. log files written and not written

Use "tsent [command] --help" for more information about a command.

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~/Code/LimeAndSoda/roomtool-server (main*) »

@mrtom
Copy link
Contributor Author

mrtom commented Jun 24, 2023

And just for completeness, if I re-add the flag to disable SWC in docker-compose.dev.yaml:

~/Code/LimeAndSoda/roomtool-server (main*) » yarn codegen                                                                                                                                                                                                                        1 ↵ tom@Dunkertons
yarn run v1.22.19
$ docker-compose -f docker-compose.dev.yml run --rm app tsent codegen
checking for custom graphql definitions...
generating ent code...
generating graphql code...
✨  Done in 13.58s.
~/Code/LimeAndSoda/roomtool-server (main*) »

@lolopinto
Copy link
Owner

ok, thanks! :(

@lolopinto
Copy link
Owner

I've been playing with it for a while and I can't 100% reliably solve it.

I think it's something to do with yarn

@mrtom
Copy link
Contributor Author

mrtom commented Jun 28, 2023

Which package manager are you using yourself?

If not yarn, would it help for me to repro the absence of an error with the other package manager?

@lolopinto
Copy link
Owner

I'm using npm. I'll figure it out with yarn. Was just giving an update so it wouldn't look like I forgot about it.

@mrtom
Copy link
Contributor Author

mrtom commented Jun 28, 2023

Thanks!

@lolopinto
Copy link
Owner

ok, i played with it even more and i believe i narrowed it down

when i had been using this for development, i had been building on an arm64 mac and building things locally without using docker

in production, doing it on an amd64 linux box and the docker image mirrors that

the issue that you were running into was the platform mismatch. there's ways around it but doesn't seem like it's worth doing that by default so i'm changing the behavior to not use swc and now to enable SWC need to use ENABLE_SWC=true env variable

@lolopinto
Copy link
Owner

i assume you're on a mac?

are you on arm64 or amd64?

if arm64, can you try this image https://github.com/lolopinto/ent/pkgs/container/ent/107819773?tag=0.1.0-alpha.50-test7-nodejs-18-dev

amd64 https://github.com/lolopinto/ent/pkgs/container/ent/107822985?tag=0.1.0-alpha.50-test8-nodejs-18-dev

different images because i was testing and faster to create single platform images

with @snowtop/ent@0.1.0-alpha160-test4

things work for me with these

@mrtom
Copy link
Contributor Author

mrtom commented Jul 10, 2023

OK, so this time:

  • Updated develop.Dockerfile to contain: FROM ghcr.io/lolopinto/ent:0.1.0-alpha.50-test7-nodejs-18-dev
  • Removed - DISABLE_SWC=true from docker-compose.dev.yaml
  • Added ENABLE_SWC=true to docker-compose.dev.yaml.
  • Ran docker pull ghcr.io/lolopinto/ent:0.1.0-alpha.50-test7-nodejs-18-dev in the terminal
  • Updated package.json to include "@snowtop/ent": "0.1.0-alpha160-test4", as a dependency
  • Ran yarn install
  • Ran yarn rebuild-image
  • Ran yarn codegen
➜  roomtool-server git:(main) ✗ yarn codegen
yarn run v1.22.19
$ docker-compose -f docker-compose.dev.yml run --rm app tsent codegen
node:internal/modules/cjs/loader:1080
  throw err;
  ^

Error: Cannot find module '@swc-node/register'
Require stack:
- internal/preload
    at Module._resolveFilename (node:internal/modules/cjs/loader:1077:15)
    at Module._load (node:internal/modules/cjs/loader:922:27)
    at internalRequire (node:internal/modules/cjs/loader:174:19)
    at Module._preloadModules (node:internal/modules/cjs/loader:1433:5)
    at loadPreloadModules (node:internal/process/pre_execution:606:5)
    at setupUserModules (node:internal/process/pre_execution:118:3)
    at prepareExecution (node:internal/process/pre_execution:109:5)
    at prepareMainThreadExecution (node:internal/process/pre_execution:37:3)
    at node:internal/main/run_main_module:10:1 {
  code: 'MODULE_NOT_FOUND',
  requireStack: [ 'internal/preload' ]
}

Node.js v18.16.1
Error:
  exit status 1
Usage:
  tsent [command]

Available Commands:
  alembic             alembic command
  codegen             runs the codegen (and db schema) migration
  completion          Generate the autocompletion script for the specified shell
  delete_schema       deletes the given schema
  detect_dangling     detects any dangling schema files
  downgrade           downgrade db
  fix-edges           fix edges db
  generate            generate a schema or other object
  help                Help about any command
  migrate_v0.1        migrate v0.1
  print_custom_schema prints the parsed custom graphql schema. only exists for debugging purposes and not guaranteed to exist forever
  print_schema        prints the parsed schema. only exists for debugging purposes and not guaranteed to exist forever
  run_script          run script
  squash              squash last N revs of the db into one
  upgrade             upgrade db

Flags:
      --debug         debug mode. add debug information to codegen e.g. how long commands take
      --debug_files   debug files mode. log files written and not written

Use "tsent [command] --help" for more information about a command.

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

@mrtom
Copy link
Contributor Author

mrtom commented Jul 10, 2023

Tried adding @swc-node/register as a dependency:

  • yarn add @swc-node/register
  • yarn codegen
➜  roomtool-server git:(main) ✗ yarn add @swc-node/register
yarn add v1.22.19
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
warning Pattern ["strip-ansi@^6.0.1"] is trying to unpack in the same destination "/Users/tom/Library/Caches/Yarn/v6/npm-strip-ansi-cjs-6.0.1-9e26c63d30f53443e9489495b2105d37b67a85d9-integrity/node_modules/strip-ansi-cjs" as pattern ["strip-ansi-cjs@npm:strip-ansi@^6.0.1"]. This could result in non-deterministic behavior, skipping.
warning Pattern ["string-width@^4.1.0"] is trying to unpack in the same destination "/Users/tom/Library/Caches/Yarn/v6/npm-string-width-cjs-4.2.3-269c7117d27b05ad2e536830a8ec895ef9c6d010-integrity/node_modules/string-width-cjs" as pattern ["string-width-cjs@npm:string-width@^4.2.0"]. This could result in non-deterministic behavior, skipping.
warning Pattern ["strip-ansi@^6.0.0"] is trying to unpack in the same destination "/Users/tom/Library/Caches/Yarn/v6/npm-strip-ansi-cjs-6.0.1-9e26c63d30f53443e9489495b2105d37b67a85d9-integrity/node_modules/strip-ansi-cjs" as pattern ["strip-ansi-cjs@npm:strip-ansi@^6.0.1"]. This could result in non-deterministic behavior, skipping.
[3/4] 🔗  Linking dependencies...
warning " > @snowtop/ent-phonenumber@0.1.0-alpha2" has incorrect peer dependency "@snowtop/ent@>= 0.1.0-alpha95".
warning " > next-auth@4.22.0" has unmet peer dependency "next@^12.2.5 || ^13".
warning " > next-auth@4.22.0" has unmet peer dependency "react@^17.0.2 || ^18".
warning " > next-auth@4.22.0" has unmet peer dependency "react-dom@^17.0.2 || ^18".
warning " > @swc-node/register@1.6.6" has unmet peer dependency "typescript@>= 4.3".
[4/4] 🔨  Building fresh packages...
success Saved lockfile.
success Saved 8 new dependencies.
info Direct dependencies
└─ @swc-node/register@1.6.6
info All dependencies
├─ @swc-node/core@1.10.4
├─ @swc-node/register@1.6.6
├─ @swc-node/sourcemap-support@0.3.0
├─ buffer-from@1.1.2
├─ colorette@2.0.20
├─ pirates@4.0.6
├─ source-map-support@0.5.21
└─ source-map@0.6.1
✨  Done in 2.81s.
➜  roomtool-server git:(main) ✗ yarn codegen
yarn run v1.22.19
$ docker-compose -f docker-compose.dev.yml run --rm app tsent codegen
Error: Bindings not found
    at Compiler.transformSync (/app/node_modules/@swc/core/index.js:250:15)
    at transformSync (/app/node_modules/@swc/core/index.js:348:21)
    at transformSync (/app/node_modules/@swc-node/core/index.ts:73:26)
    at compile (/app/node_modules/@swc-node/register/register.ts:102:40)
    at exts (/app/node_modules/@swc-node/register/register.ts:113:38)
    at Module._compile (/app/node_modules/pirates/lib/index.js:113:29)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Object.newLoader [as .ts] (/app/node_modules/pirates/lib/index.js:121:7)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Function.Module._load (node:internal/modules/cjs/loader:960:12)
Error:
  exit status 1
Usage:
  tsent [command]

Available Commands:
  alembic             alembic command
  codegen             runs the codegen (and db schema) migration
  completion          Generate the autocompletion script for the specified shell
  delete_schema       deletes the given schema
  detect_dangling     detects any dangling schema files
  downgrade           downgrade db
  fix-edges           fix edges db
  generate            generate a schema or other object
  help                Help about any command
  migrate_v0.1        migrate v0.1
  print_custom_schema prints the parsed custom graphql schema. only exists for debugging purposes and not guaranteed to exist forever
  print_schema        prints the parsed schema. only exists for debugging purposes and not guaranteed to exist forever
  run_script          run script
  squash              squash last N revs of the db into one
  upgrade             upgrade db

Flags:
      --debug         debug mode. add debug information to codegen e.g. how long commands take
      --debug_files   debug files mode. log files written and not written

Use "tsent [command] --help" for more information about a command.

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

@mrtom
Copy link
Contributor Author

mrtom commented Jul 10, 2023

And yes, this is an M2 so arm64.

Screenshot 2023-07-10 at 07 27 08

@lolopinto
Copy link
Owner

the first one should have failed so that's not surprising since no dependency and ENABLE_SWC was added

the default is no extra envs and it should work because it's not doing SWC

the latter not working is maddening lol. i tested it multiple times the other day but yup, not working for me anymore.

I think I'm going to punt on this for now and come back to it later with the node_modules hidden from docker approach. my guess is we need to run yarn install in develop.Dockerfile separately on docker.

@lolopinto
Copy link
Owner

thanks for debugging with me

@lolopinto lolopinto changed the title __setRawDBData type error codegen with swc on cross platform dev + docker not working Jul 10, 2023
@mrtom
Copy link
Contributor Author

mrtom commented Jul 10, 2023

You're welcome!

@mrtom
Copy link
Contributor Author

mrtom commented Jul 10, 2023

the default is no extra envs and it should work because it's not doing SWC

Yeah, I didn't test this because AIUI this is a problem with SWC. Let me know if you do want me to test this, but assume not for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants