refactor: merge turbopack crates into next.js #68127

ForsakenHarmony · 2024-07-24T20:54:37Z

What?

Moves all turbopack crates (and a few js packages) into the next.js repo.

Why?

https://vercel.com/blog/turbopack-moving-homes

How?

git-filter-repo https://github.com/newren/git-filter-repo (better than filter-branch)
and
git subtree

See: https://github.com/vercel/next-turbo-merge

Short version:

git clone git@github.com:vercel/next.js.git --filter=blob:none --no-tags --single-branch --branch canary "$NEXT_DIR"

git --git-dir="$NEXT_DIR/.git" remote add --fetch --track main --no-tags turbo git@github.com:vercel/turbo

git switch -C hrmny/turbopack-staging turbo/main --no-track

git filter-repo --force --refs hrmny/turbopack-staging \
  --path crates/ \
  --path xtask/ \
  --path packages/devlow-bench/ \
  --path packages/node-file-trace/ \
  --path packages/node-module-trace/ \
  --path packages/turbo-pack/ \
  --path packages/turbo-tasks/ \
  --path packages/turbo-tasks-examples/ \
  --path packages/turbo-tasks-fs/ \
  --path packages/turbo-tasks-macros/ \
  --path packages/turbo-tracing-next-plugin/ \
  --path packages/webpack-nmt/

git filter-repo --force --refs hrmny/turbopack-staging \
  --path-glob 'crates/turborepo*' --invert-paths

git filter-repo --refs hrmny/turbopack-staging --commit-callback '
  commit.message = re.sub(b"(^|\\(| )(#\\d\\d\\d\\d)(\\)| |\\r|$)", b"\\1vercel/turbo\\2\\3", commit.message, flags=re.MULTILINE)
'

git gc

git switch -C hrmny/merge-turbopack origin/canary --no-track

git subtree add -P turbopack hrmny/turbopack-staging

Closes PACK-3163

### Description Update missing fixes for the features. Closes PACK-2722

### Description  ### Testing Instructions  Closes PACK-2724

…borepo#7719) ### Description This can actually happen e. g. with next/font, which creates virtual css files that are not inside of the project directory ### Testing Instructions  Closes PACK-2732

### Description * Slow start for the concurrency of parallel booting node.js processes (1, 2, 4, ...) * Depending on historic statistics avoid booting up a new process, but wait for one to finish computing Loading the `/home` route on 16-core machine Before: ``` 16 x Node.js process startup 9.09s avg 568.07ms 16 x Node.js initialization 10.58s avg 660.96ms 276 x Node.js evaluation 30.17s avg 109.30ms ``` After: ``` 7 x Node.js process startup 1261.79ms avg 180.26ms 7 x Node.js initialization 3.15s avg 450.21ms 276 x Node.js evaluation 29.41s avg 106.55ms ``` ### Testing Instructions  Closes PACK-2680

…ercel/turborepo#7720) ### Description Async module handling was missing for side effects optimized modules ### Testing Instructions  Closes PACK-2733

…cel/turborepo#7718) ### Description `lightningcss` enforces the class name to end with `[local]`. See: https://lightningcss.dev/css-modules.html#css-grid ### Testing Instructions I'll add a test soon. Closes PACK-2731

…urbopack (vercel/turborepo#7711) ### Description Currently the development Turbopack runtime is always used as production builds are not fully implemented yet. This is the first step to having a production-specific runtime by adding a distinction between Development and Production for the runtime. This PR does not implement the production runtime, it only adds the plumbing for adding the production specific runtime. The runtime implementation will follow in a later PR.

…#7726) ### Description Trying to support sass-loader to not fail if it requires specific logger from loadercontext. Closes PACK-2737

) ### Description - Closes #63102 - Closes PACK-2717 ### Testing Instructions I added a test.

…ercel/turborepo#7715) This: - Implements a hook that enables callers like Next.js to perform their own logic (to be used in an upcoming linked PR) when HMR updates fail, such as anonymous function components - Logs an actionable message similar to Next.js+Webpack when updates fail Closes PACK-2726

Whether due to a runtime error or an apply error, it should be reloaded. Closes PACK-2751

### Description When accessing many exports of a barrel file with many `export *` it was a bit inefficient before since it was walking all star exports for every accessed export. After this change it only creates a map of the star exports once and accessing every export only looks up the result in the map. This makes it 4 times more efficient for a file with 140 `export *`. Probably even more for bigger barrel files. ### Testing Instructions  Closes PACK-2747

### Description For the `failsafe_analyze`, make it not failable by not bubbling up the error - the error will be propagated by chunk generation anyway. Closes PACK-2764

### Description  ### Testing Instructions  Closes PACK-2784

### Description The default for asset prefix is `"/"` and not `""` ### Testing Instructions #62134

### Description * Handle source maps returned webpack loaders * add workaround for Node.js bug to allow it using sectioned source maps * import source map for Source file ### Testing Instructions  Closes PACK-2783

### Description We already return the source map and don't want a source map comment on the generated code. (It's added by the chunk) ### Testing Instructions  Closes PACK-2794

…urbo-tasks (vercel/turborepo#7774) ### Description ### Testing Instructions  Closes PACK-2795

…po#7778) ### Description fixes #62037 ### Testing Instructions  Closes PACK-2800

…ack to raw (vercel/turborepo#7785) This causes Turbopack to emit an error-level issue when a module doesn’t have a type set on it. Previously, it fell back to assuming a raw type. Closes PACK-2802

### Description * implement all logger methods * pass logging to rust * show error when using error logging ### Testing Instructions  Closes PACK-2811 --------- Co-authored-by: Will Binns-Smith <wbinnssmith@gmail.com>

### Description * fix `alias` in `getResolve` * Include error message in resolving issue description * avoid emitting unsupported errors when request is aliased ### Testing Instructions  Closes PACK-2820

### Description Update SWC crates to swc-project/swc@ad932f0, to keep in sync. - Closes PACK-2807 - Closes PACK-2819 ### Testing Instructions See next.js counterpart: #63541

### Description * add import context `layer`, `supports`, `media` to single css chunks ### Testing Instructions  Closes PACK-2797

### Description The port number is too high and in the automatically chosen range. This could lead to port being already used. ### Testing Instructions  Closes PACK-2798

### Description * add support for recording attributes for spans after creation * In the trace viewer when searching only expand spans that contain the searched value * fix reseting the trace store to avoid accessing invalid indicies * remove old convert-trace tool ### Testing Instructions  Closes PACK-2786

…epo#7813) ### Description fixes SCSS HMR for imported files ### Testing Instructions  Closes PACK-2818 fixes PACK-2809

…urborepo#7764) ### Description * disable GC for tasks which use the streaming hack * invalidate dependent tasks when assigning to a recomputing cell ### Testing Instructions  Closes PACK-2787

### Description While debugging https://vercel.slack.com/archives/C03EWR7LGEN/p1711146825831219?thread_ts=1711143217.892349&cid=C03EWR7LGEN, found out sourcemap lookup fails on certain source and always returns synthetictoken only. Weirdly, the guards in this PR https://github.com/vercel/turbo/pull/7823/files#diff-2ce67e28c5b3144ec6f7a89167f3c96da9f9e268abf3fd685ce881d75a4cd8a5L319 is preventing those lookup - removing it makes correct sourcemap lookup occurs. I updated pkg to the version what swc uses and removed + ran next.js tests, seems like most of tests are running just fine (#63624) Still not 100% sure if this'll be ok or not, if not open to change for the correct fixes. Some related fixes getsentry/rust-sourcemap#60 might be the reason we don't see any regressions in the test.

### Description fixes a problem where an async module wrapper is generated for external modules in a "locals" module part when there is usage of the external module in the locals part fixes #63485 ### Testing Instructions  Closes PACK-2810 fixes PACK-2805

### Description Fix server action and rename the injected variable. The previous code fails if the name of the top-level variable is `v`. This was not a problem for almost all modules, but some packages like [`@firebase/webchannel-wrapper`](https://unpkg.com/@firebase/webchannel-wrapper@0.2.41/dist/index.esm.js) is published after mangling, so it can be a problem ### Testing Instructions See #66689

### Description Avoids recompiling too often ### Testing Instructions

### Description skips import to modules that are side effect free and have no exports referenced This allows to e. g. skip the `package` import completely in the following code: ``` js import { something } from "package"; // package is marked as side effect free if (falsy) { something(); } ``` This could happen e. g. with code like `if(process.env.NODE_ENV === "production")` ### Testing Instructions

### Description Remove `ModuleType::Mdx` and instead handle mdx files using ```rust ModuleRuleEffect::ModuleType(ModuleType::Typescript { transforms: ts_app_transforms, tsx: true, analyze_types: enable_types, options: ecmascript_options_vc, }), ModuleRuleEffect::SourceTransforms(Vc::cell(vec![Vc::upcast( MdxTransform::new(mdx_transform_options), )])), ``` ### Testing Instructions I ran these Next.js tests, which appear to be all mdx related tests there are ``` mdx with-mdx-rs app directory ✓ should work in initial html (1262 ms) ✓ should work using browser (1460 ms) ✓ should work in initial html with mdx import (145 ms) ✓ should work using browser with mdx import (1179 ms) ✓ should allow overriding components (1163 ms) ✓ should allow importing client components (26 ms) ✓ should work with next/image (424 ms) pages directory ✓ should work in initial html (1091 ms) ✓ should work using browser (1216 ms) ✓ should work in initial html with mdx import (147 ms) ✓ should work using browser with mdx import (1202 ms) ✓ should allow overriding components (1236 ms) ```

This hides them from Next.js for now I'll add a test case for Next.js as well Closes PACK-2877

Trying vercel/turborepo#8031 again ~~Depends on vercel/turborepo#8766 <img width="979" alt="Bildschirmfoto 2024-07-30 um 15 35 48" src="https://github.com/user-attachments/assets/32d88ffc-c816-41fb-928b-79d27e89b56d"> --------- Co-authored-by: 강동윤 (Donny) <kdy1997.dev@gmail.com>

…ercel/turborepo#8873) ### Description Allows to handle `if (unknown && falsy)` `if (unknown || truthy)` `if (unknown ?? nullish)` in static analysis. Hint: review each commit on it's own, since the first commit is only changing to a named struct. ### Testing Instructions

…ercel/turborepo#8890) ### Description * tracks early returns and their conditions * replaces unreachable code with `"TURBOPACK unreachable"` * finds hoisted declarations in unreachable code and keeps them ### Testing Instructions

git-subtree-dir: turbopack git-subtree-mainline: 344ddbf git-subtree-split: fb033c4

### What? Follow up for the previous PR in the stack updating references, lockfiles and snapshots. ### Why? https://vercel.com/blog/turbopack-moving-homes ### How? https://github.com/vercel/next-turbo-merge Closes PACK-3163

Moves all turbopack crates (and a few js packages) into the next.js repo. https://vercel.com/blog/turbopack-moving-homes `git-filter-repo` https://github.com/newren/git-filter-repo (better than filter-branch) and `git subtree` See: https://github.com/vercel/next-turbo-merge Short version: ```bash git clone git@github.com:vercel/next.js.git --filter=blob:none --no-tags --single-branch --branch canary "$NEXT_DIR" git --git-dir="$NEXT_DIR/.git" remote add --fetch --track main --no-tags turbo git@github.com:vercel/turbo git switch -C hrmny/turbopack-staging turbo/main --no-track git filter-repo --force --refs hrmny/turbopack-staging \ --path crates/ \ --path xtask/ \ --path packages/devlow-bench/ \ --path packages/node-file-trace/ \ --path packages/node-module-trace/ \ --path packages/turbo-pack/ \ --path packages/turbo-tasks/ \ --path packages/turbo-tasks-examples/ \ --path packages/turbo-tasks-fs/ \ --path packages/turbo-tasks-macros/ \ --path packages/turbo-tracing-next-plugin/ \ --path packages/webpack-nmt/ git filter-repo --force --refs hrmny/turbopack-staging \ --path-glob 'crates/turborepo*' --invert-paths git filter-repo --refs hrmny/turbopack-staging --commit-callback ' commit.message = re.sub(b"(^|\$| )(#\\d\\d\\d\\d)(\$| |\\r|$)", b"\\1vercel/turbo\\2\\3", commit.message, flags=re.MULTILINE) ' git gc git switch -C hrmny/merge-turbopack origin/canary --no-track git subtree add -P turbopack hrmny/turbopack-staging ``` Closes PACK-3163

# Warning: DO NOT squash merge ### What? Moves all turbopack crates (and a few js packages) into the next.js repo. ### Why? https://vercel.com/blog/turbopack-moving-homes ### How? `git-filter-repo` https://github.com/newren/git-filter-repo (better than filter-branch) and `git subtree` See: https://github.com/vercel/next-turbo-merge Short version: ```bash git clone git@github.com:vercel/next.js.git --filter=blob:none --no-tags --single-branch --branch canary "$NEXT_DIR" git --git-dir="$NEXT_DIR/.git" remote add --fetch --track main --no-tags turbo git@github.com:vercel/turbo git switch -C hrmny/turbopack-staging turbo/main --no-track git filter-repo --force --refs hrmny/turbopack-staging \ --path crates/ \ --path xtask/ \ --path packages/devlow-bench/ \ --path packages/node-file-trace/ \ --path packages/node-module-trace/ \ --path packages/turbo-pack/ \ --path packages/turbo-tasks/ \ --path packages/turbo-tasks-examples/ \ --path packages/turbo-tasks-fs/ \ --path packages/turbo-tasks-macros/ \ --path packages/turbo-tracing-next-plugin/ \ --path packages/webpack-nmt/ git filter-repo --force --refs hrmny/turbopack-staging \ --path-glob 'crates/turborepo*' --invert-paths git filter-repo --refs hrmny/turbopack-staging --commit-callback ' commit.message = re.sub(b"(^|\$| )(#\\d\\d\\d\\d)(\$| |\\r|$)", b"\\1vercel/turbo\\2\\3", commit.message, flags=re.MULTILINE) ' git gc git switch -C hrmny/merge-turbopack origin/canary --no-track git subtree add -P turbopack hrmny/turbopack-staging ``` Closes PACK-3163 # Conflicts: # .eslintignore # .eslintrc.cli.json # Cargo.lock # Cargo.toml # package.json # packages/next/package.json # pnpm-lock.yaml

kwonoj and others added 30 commits March 12, 2024 12:54

fix(image): fix build errors for the features (vercel/turborepo#7710)

af79e7b

### Description Update missing fixes for the features. Closes PACK-2722

allow to use full urls in browser runtime (vercel/turborepo#7712)

9267569

### Description  ### Testing Instructions  Closes PACK-2724

feat(node): augment webpack loader's logger context (vercel/turborepo…

722b879

…#7726) ### Description Trying to support sass-loader to not fail if it requires specific logger from loadercontext. Closes PACK-2737

fix(css): Do not resolve urls in supports query (vercel/turborepo#7730

bdd58b8

) ### Description - Closes #63102 - Closes PACK-2717 ### Testing Instructions I added a test.

Turbopack HMR: reload on any failed HMR update (vercel/turborepo#7744)

793f355

Whether due to a runtime error or an apply error, it should be reloaded. Closes PACK-2751

fix(mdx): allow failsafe analyze for mdxasset (vercel/turborepo#7757)

5e02af2

### Description For the `failsafe_analyze`, make it not failable by not bubbling up the error - the error will be propagated by chunk generation anyway. Closes PACK-2764

avoid panic (vercel/turborepo#7762)

ec4fec7

### Description  ### Testing Instructions  Closes PACK-2784

fix ASSET_PREFIX (vercel/turborepo#7750)

5e4156b

### Description The default for asset prefix is `"/"` and not `""` ### Testing Instructions #62134

add cut off depth to avoid stack overflow for infinite recursion in t…

d634976

…urbo-tasks (vercel/turborepo#7774) ### Description ### Testing Instructions  Closes PACK-2795

fallback to normal resolving if alias doesn't resolve (vercel/turbore…

43af5ee

…po#7778) ### Description fixes #62037 ### Testing Instructions  Closes PACK-2800

Turbopack: Emit an issue on unknown module types instead of falling b…

6105ced

…ack to raw (vercel/turborepo#7785) This causes Turbopack to emit an error-level issue when a module doesn’t have a type set on it. Previously, it fell back to assuming a raw type. Closes PACK-2802

Update swc_core to v0.90.24 (vercel/turborepo#7796)

6e49ae8

### Description Update SWC crates to swc-project/swc@ad932f0, to keep in sync. - Closes PACK-2807 - Closes PACK-2819 ### Testing Instructions See next.js counterpart: #63541

fix single css chunks with import context (vercel/turborepo#7775)

0108d33

### Description * add import context `layer`, `supports`, `media` to single css chunks ### Testing Instructions  Closes PACK-2797

change port of trace-server (vercel/turborepo#7776)

db67b05

### Description The port number is too high and in the automatically chosen range. This could lead to port being already used. ### Testing Instructions  Closes PACK-2798

fix webpack loader incorrectly calling custom_evaluate (vercel/turbor…

639e6c6

…epo#7813) ### Description fixes SCSS HMR for imported files ### Testing Instructions  Closes PACK-2818 fixes PACK-2809

kdy1 and others added 4 commits July 30, 2024 10:51

improve test glob to exclude output folder (vercel/turborepo#8875)

ac39709

### Description Avoids recompiling too often ### Testing Instructions

ForsakenHarmony force-pushed the hrmny/merge-turbopack branch from dd4ed78 to c4e331d Compare July 30, 2024 15:56

mischnic and others added 2 commits July 31, 2024 11:58

Make JS Lints have lower severity (vercel/turborepo#8837)

0879251

This hides them from Next.js for now I'll add a test case for Next.js as well Closes PACK-2877

ForsakenHarmony force-pushed the hrmny/merge-turbopack branch from c4e331d to 93986d9 Compare July 31, 2024 15:17

ForsakenHarmony mentioned this pull request Jul 31, 2024

chore: remove turbopack-binding and other unused crates #68362

Closed

ForsakenHarmony force-pushed the hrmny/merge-turbopack branch from 93986d9 to 8a2347b Compare July 31, 2024 22:22

ForsakenHarmony force-pushed the hrmny/merge-turbopack branch from 8a2347b to ee90487 Compare August 1, 2024 14:13

ForsakenHarmony marked this pull request as ready for review August 1, 2024 14:30

Add 'turbopack/' from commit 'fb033c4917bb1bb98b238f1b4c7a928b66a90887'

becc655

git-subtree-dir: turbopack git-subtree-mainline: 344ddbf git-subtree-split: fb033c4

ForsakenHarmony force-pushed the hrmny/merge-turbopack branch from ee90487 to becc655 Compare August 1, 2024 14:55

ForsakenHarmony requested review from sokra and bgw August 1, 2024 15:00

chore: fix references for the new turbopack crates (#68128)

7c3b279

### What? Follow up for the previous PR in the stack updating references, lockfiles and snapshots. ### Why? https://vercel.com/blog/turbopack-moving-homes ### How? https://github.com/vercel/next-turbo-merge Closes PACK-3163

ijjk added the type: next label Aug 1, 2024

ForsakenHarmony enabled auto-merge August 1, 2024 15:13

sokra disabled auto-merge August 1, 2024 15:35

sokra approved these changes Aug 1, 2024

View reviewed changes

ForsakenHarmony merged commit 238b882 into canary Aug 1, 2024
101 of 107 checks passed

ForsakenHarmony deleted the hrmny/merge-turbopack branch August 1, 2024 15:36

bgw mentioned this pull request Aug 1, 2024

chore: remove turbopack crates vercel/turborepo#8906

Merged

github-actions bot added the locked label Aug 16, 2024

github-actions bot locked as resolved and limited conversation to collaborators Aug 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: merge turbopack crates into next.js #68127

refactor: merge turbopack crates into next.js #68127

ForsakenHarmony commented Jul 24, 2024 •

edited by sokra

Loading

refactor: merge turbopack crates into next.js #68127

refactor: merge turbopack crates into next.js #68127

Conversation

ForsakenHarmony commented Jul 24, 2024 • edited by sokra Loading

What?

Why?

How?

ForsakenHarmony commented Jul 24, 2024 •

edited by sokra

Loading