Feature request: Watch mode

[dobesv]

Describe the feature you'd like to request

It would be nice if turbo could run in "watch mode

Describe the solution you'd like

Turbo could watch all the files it uses to calculate the build hash for each package/workspace, and re-run the build if one of those files changes.

Describe alternatives you've considered

I think I could probably setup nodemon to run turbo when a file changes in a package, and then turbo will just only build packages with a changed file.

[b12k]

This functionality potentially may have a collision with existing scripts (for example in nestjs and nextjs apps).

For example:
turbo run dev --watch where application dev npm script already has watch mode out of the box.

IMHO respective applications scripts is the right place for such things, not turbo.

[dobesv]

This functionality potentially may have a collision with existing scripts (for example in nestjs and nextjs apps).
turbo run dev --watch where application dev npm script already has watch mode out of the box.

I think it's simple not to use the watch feature in that case, isn't it ?

IMHO respective applications scripts is the right place for such things, not turbo.

Seems a fair point. However, I think turbo is specially positioned to know which files should trigger a build and watch those files. Providing it as part of turbo would be a convenience.

[franco-roura]

I second the motion, you usually don't want to "dev" your apps until you have "built" your packages, spawning dev in apps&packages concurrently will cause the app not to find the built package in some scenarios.

On the other hand, adding "dependsOn": ["^dev"] to the turbo.json is not an option, daemon/watcher scripts don't finish, so the dependency is never satisfied and apps/*/dev never starts.

If Turbo had an inner watcher to trigger build or codegen scripts on file changes it would save an interesting amount of time and performance by preventing developers from manually maintaining a script like the following for each package:

{
"watch:someapp": "nodemon --watch 'packages/somepackage' -e ts --exec 'turbo run build --scope=somepackage'"
}

[hrasekj]

It would be great feature. For example, when building packages with swc, and still need typechecks between packages.

Example

My solution on monorepo i'm working on, with 20+ packages.

turbo.json

{
  "pipeline": {
    "build": {
      "dependsOn": ["types", "^build"],
      "outputs": ["dist/**", ".next/**"]
    },
    "types": {
      "cache": false,
      "outputs": ["typings/**"]
    },
    "watch": {
      "cache": false
    }
  }
}

root package.json

{
  "name": "root",
  "private": true,
  "scripts": {
    "build": "turbo run build",
    "watch": "turbo run watch --parallel"
  }
}

packages/my-helpers/package.json

{
  "name": "my-helpers",
  "scripts": {
    "types": "tsc --emitDeclarationsOnly",
    "watch": "chokidar 'src/**/*.js' 'src/**/*.ts' -c 'pnpm -wc exec turbo run types  --filter=...^my-helpers'",
  }
}

This solution is nice, because i can see, that changes in one package will affect another package in scope.

There are some drawbacks in this:

• can't cache types pipeline
• must write package name inside watch script
• output logs are not nice

my-helpers:watch: • Packages in scope: @admin/web
my-helpers:watch: • Running types in 5 packages
my-helpers:watch: • Running types in 5 packages
my-helpers:watch: @admin/web:types: cache miss, executing 25984f71faae504a
...

Proposition

As i see it. Would be nice something like this in turbo.

turbo.json

{
  "pipeline": {
    "build": {
      "dependsOn": ["types", "^build"],
      "outputs": ["dist/**", ".next/**"]
    },
    "types": {
      "outputs": ["typings/**"]
    }
  }
}

root package.json

{
  "name": "root",
  "private": true,
  "scripts": {
    "watch": "turbo run types --watch"
  }
}

packages/my-lib/package.json

{
  "name": "my-lib",
  "scripts": {
    "types": "tsc --emitDeclarationsOnly"
  },
  "turboWatch": [
    "src/**/*.js",
    "src/**/*.ts",
  ]
}

Watch mode, should:

• run types script every time, when files described by turboWatch changes
• update types cache, on change, if necessary

Then running build pipeline will have types already run from cache.

[irontitan76]

I think this functionality is almost necessary at scale. Rather than running countless processes to watch each dependency for a target, one process should watch all the files that are in the dependency graph. If a change is detected then rerun the script for the affected dependencies. Rush.js does something similar with its experimental watchForChanges property.

Obviously, this functionality wouldn't be needed where watching becomes redundant and that is up for the developer to decide. So, if a developer had scripts like the following, he or she would not need to (and really should not) run this watch functionality with dev.

"scripts": {
  "build": "tsc --build",
  "dev": "tsc --watch",
}

I'm brainstorming a few ways to solve this with what @hrasekj stated being a valid solution. At the present moment, I think we could add a watch boolean property to the pipeline configuration objects. If true, it watches the files defined as inputs (if present) in each of the packages, but it does this as a whole based on the graph, rather than as individual processes.

[georeith]

Big +1 for this.

We're using Jest for tests and its watch mode doesn't detect changes in transitive monorepo dependencies like Turbo does for its caching.

On an extra note: a lot of watch modes overwrite STDOUT instead of appending, combined with something like this #219 and the option to overwrite STDOUT we could make watched output much easier to read.

[adam-coster]

This would be very helpful for places where native watchers fall short, which is pretty common for complex pipelines and inter-package dependencies.

For example, I have a monorepo that includes a package that must be run in a docker container. Getting that to work with the other packages is a pain -- the best way would be to have the container rebuild itself whenever the other dependencies change, so that it can re-copy those does into the container and relaunch it. If turbo had a watch mode that would be pretty straightforward. In the absence I that I have to do some shenanigans that don't auto-adapt to changes in the dependency graph, so both complexity and maintenance costs go way up.

I'd much rather run all of my scripts in non-watch mode and then have Turbo manage watching, since most tools that have built-in watch mode have extremely limited utility in the context of pipeline.

[ScottAwesome]

The litmus test for me on this is how wireit handles watching for changes. It works extremely well, and should be the model. Its pretty neat how they solved this on the tool level.

[awxalbert]

Same here, my scenario is when I develop Apollo GraphQL server, I may end up with a package.json like:

{
  "dev": "ts-node -r dotenv/config src/index.ts",
  "generate": "graphql-codegen"
}

Currently, I use nodemon and codegen watch mode to do this, which is really inconvenient and have problems. Really looking forward to having this feature.

[arimus]

Things can get tricky with build deps and triggering commands, which the workarounds provided above don't account for, so far as I can tell. Simple filesystem monitoring, without dependency graph context, are not solutions when the graph isn't simple / linear (e.g. A depends on B depends on C).

For example, a common scenario would be where an app depends on multiple libs that are all rebuilt. Even if you set up scripts to trigger once a dependency changes, the build order has to fully complete before a restart of the app should be triggered. Slightly more complex example with a single library at the base of those two libraries incorporated into a single app:

        ______ Lib A _____
       /                  \
Lib 0 -                    My App
       \______ Lib B _____/

• Lib 0 changed
• Lib 0 rebuilt
  • Lib A rebuild
    • changes detected, rebuild / restart My App
  • Lib B rebuild
    • changes detected, rebuild / restart My App

Of course you can put the de-bounce up really high to ensure the build of all the things happens before triggering the My App rebuild, but this is not a great solution for obvious reasons.

Turbo knows the entire build graph, so it would be fantastic if we could leverage that to only run some commands after turbo knows that all linked deps have built.

[dobesv]

Another question: if watch mode wasn't added in the core, does turbo export enough functionality as a go library to write a separate tool that reuses code from turbo to implement a watch mode? Maybe we could put up a bounty somewhere and someone can write another go app that wraps turbo with a watch mode.

[arimus]

Another question: if watch mode wasn't added in the core, does turbo export enough functionality as a go library to write a separate tool that reuses code from turbo to implement a watch mode? Maybe we could put up a bounty somewhere and someone can write another go app that wraps turbo with a watch mode.

Unless I'm mistaken, I don't think that turbo repo exports any functionality and there is no extensibility at the moment. That was one of the things that nx provided over turborepo. I agree that if there were some lifecycle plugin hooks for turbo, then that would require less to maintain in the core while still providing the necessary build context for separate tools. Even something as simple as hooks for the same events that are output to the console during build (cache replayed, cache miss / executing, build complete). Or barring that, executing registered commands for them as they occur.

File system watches alone won't work for many cases like my example above, unless you integrate the dependency graph as context. It still gets messy and complicated though. In the end, it'd be easier and more flexible for turbo (which already has the context) to trigger / notify at the right stages. Still stays simple and clean without a bunch of extra complexity.

[weyert]

I think they might be planning to use the (experimental) daemon for this; to allow script ability in Node.js by a socket connection using protobuffers

[scamden]

another +1. rush has this: https://rushjs.io/pages/advanced/watch_mode/ for inspo (oh just read #986 (comment) but link could still be helpful for reference). Works really well if you want to build everything in watch mode and have your dev processes just responding to the compiled files (rather than the overhead of numerous compilers running simultaneously)

also didn't yall say this was up next like a while ago? https://turborepo.com/posts/turbo-0-4-0

[dobesv]

I would be willing to try working on this, if the team would be willing to accept the pull request.

[nathanhammond]

This is something we are considering as a possible future. (There is a reason the issue remains open.)

We're not opposed to accepting it from an external contributor as a way to have it bumped up in priority but do know that it's not a trivial effort and if undertaken now will need to be written in both Go and Rust.

[weyert]

Why in both? I thought you would use Go code as a library in Rust? To me, requiring both, massively increase the hurdles for contributions

[nathanhammond]

@weyert because we can't ship it in Rust right now with the state of the migration so it'd have to be Go first, and then Rust eventually.

[gajus]

We needed something that works today with Turborepo, so I wrote Turbowatch

https://github.com/gajus/turbowatch#why-not-turborepo

[dobesv]

Turbowatch

For a second I thought it was actually something specific to turborepo. It would be nice if turbowatch could parse turbo.json and automatically watch based on the inputs for each task and then run turbo as appropriate to the changed files/targets/projects

[AlexAegis]

@hrasekj earlier had an example on how they solve live type updates. I went with a different route that does not involve turborepo or running anything at all to get instant type feedback, but if you with to publish it, it does require the help of a build tool (So I made a vite plugin).

The idea is that in the source packageJson (the one checked into your git repo), you define exports like this:

{
	"exports": {
		".": {
			"types": "./src/index.ts",
			"import": "./dist/index.js",
			"require": "./dist/index.cjs"
		}
	},
}

And once I build it, the packageJson file that will be distributed (that ends up in the dist directory) will have these paths adjusted to:

This packageJson is practically invisible when the package is used locally, this is only used
for distribution

{
	"exports": {
		".": {
			"types": "./index.d.ts",
			"import": "./index.js",
			"require": "./index.cjs"
		},
}

Being in a pnpm/npm workspace, my local packages are simply symlinked to eachothers node_modules folder so they only see the source packageJson file, telling typescript to watch the types directly from the source is an easy solution to solve live updates.

But this approach leaves a big problem behind: I still need to manually rebuild libraries that I use in applications. I can see the types as I write them, I just can't use them, and if I build an application in it's own dev watch mode, only that apps code is rebuilt. But as soon as I build the dependencies, the apps dev server sees that change and hot reloads my app. This is the last puzzle piece that I'm missing.

[marek-hanzal]

I've just want to add myself as a watcher as I've turborepo with quite a few packages and I've to manually switch witch ones should be watched as they're worked on. This is awfull, also starting tens of watches (2 per package: esbuild for JS and TSC for types) is incredibly heavy.

[gajus]

This is awfull, also starting tens of watches (2 per package: esbuild for JS and TSC for types) is incredibly heavy.

Please check https://github.com/gajus/turbowatch, and specifically the part about the motivation behind the project, as it addresses why I think it is unlikely that whatever the solution that Turborepo comes up with will be an optimal solution for everyone (due to the declarative nature of turbo.json).

The closest I've seen something that is reasonably close to a sane declarative API for this use case is https://github.com/google/wireit. However, even that would not have worked in our case due to a mixed bag of requirements (such as rebuilding and re-launching Docker containers as part of the dependency tree).

For what it is worth, we are very happy with the current setup (Turborepo + Turbowatch). However, I am also cognizant of the fact that our repo is probably hitting every imaginable edge case in terms of requirements, and that alternative solutions could be a lot more straightforward for those who manage to avoid our requirements.

[dobesv]

I've just want to add myself as a watcher

For future reference, these are the preferred way to follow / vote for a github item if you don't have a substantive comment:

• Subscribe button in the right sidebar
• Add a reaction to the issue description (e.g. a 👍 )

Hopefully you don't mind me giving this tip, just trying to be helpful!

[piyushchauhan2011]

I haven't tried the turbowatch and turbotree but for basic setup with tsup, having an extra gen-types task seems to resolve some issues but we build twice unfortunately

    "dev": {
      "cache": false,
      "persistent": true,
      "dependsOn": [
        "^gen-types"
      ]
    },
    "gen-types": {
      "cache": false,
      "dependsOn": [
        "^gen-types"
      ]
    },

    "build": "tsup --format esm,cjs",
    "dev": "tsup --format esm,cjs --watch",
    "gen-types": "tsc --declaration --emitDeclarationOnly",

import { defineConfig } from 'tsup';

export default defineConfig({
  entry: ['./src/index.ts'],
  splitting: false,
  sourcemap: true,
  clean: false,
  onSuccess: 'pnpm run gen-types',
});

But after trying rush with basic tsup setup, I would highly recommend it. Rush seems to work without any gen-types hack. No complicated dev thing, just straightforward build and build --watch mode with phase builds

[piyushchauhan2011]

@0x80 Thank you, i read about the issue tsup docs and i agree with you, but I think this friction of dev/watch and build with all good options like composite, incremental should be easy to configure. It seems very convoluted in turborepo at present moment

[0x80]

@piyushchauhan2011 Yes I agree, this seems to be the most significant hurdle in my current development setup.

[kaumac]

Just wanted to add my +1 here.
I'm trying to setup turborepo with my project and I've spent an entire day trying to find a good solution for this.
I think the current api is a bit too limited.

[RSS1102]

😟Is there any progress?

[QingShan-Xu]

+1

[gajus]

For what it is worth, https://github.com/gajus/turbowatch has been really good to us. Been running without an issue for more than 12 months now.

[accordionpeas]

Is there an example repo that shows how to setup turbowatch in a monorepo with turborepo? I find that the turbowatch docs don't really show how this should be done

[gajus]

It isn't monorepo specific.

All we do is that we have turbowatch.ts in every workspace, e.g. packages/sentry/turbowatch.ts will have:

import { defineConfig } from 'turbowatch';

export default defineConfig({
  project: __dirname,
  triggers: [
    {
      expression: [
        'anyof',
        [
          'allof',
          ['dirname', 'node_modules'],
          ['dirname', 'dist'],
          ['match', '*'],
        ],
        [
          'allof',
          ['not', ['dirname', 'node_modules']],
          ['dirname', 'src'],
          ['match', '*'],
        ],
      ],
      interruptible: false,
      name: 'build',
      onChange: async ({ spawn }) => {
        await spawn`pnpm run build`;
      },
    },
  ],
});

Then we just "watch" the entire monorepo with turbowatch ./packages/**/turbowatch.ts.

[accordionpeas]

Ok thanks very much, will take a look

[QingShan-Xu]

thank you very much

[QingShan-Xu]

It isn't monorepo specific.它不是特定于 monorepo 的。

All we do is that we have turbowatch.ts in every workspace, e.g. packages/sentry/turbowatch.ts will have:我们所做的就是在每个工作空间中都有 turbowatch.ts ，例如 packages/sentry/turbowatch.ts 将有：

import { defineConfig } from 'turbowatch';

export default defineConfig({
  project: __dirname,
  triggers: [
    {
      expression: [
        'anyof',
        [
          'allof',
          ['dirname', 'node_modules'],
          ['dirname', 'dist'],
          ['match', '*'],
        ],
        [
          'allof',
          ['not', ['dirname', 'node_modules']],
          ['dirname', 'src'],
          ['match', '*'],
        ],
      ],
      interruptible: false,
      name: 'build',
      onChange: async ({ spawn }) => {
        await spawn`pnpm run build`;
      },
    },
  ],
});

Then we just "watch" the entire monorepo with turbowatch ./packages/**/turbowatch.ts.然后我们只需用 turbowatch ./packages/**/turbowatch.ts “观看” 整个 monorepo 。

Cannot run on Windows system

[weyert]

Or you run turbo watch build —filter={packages\*} with the latest canary build :)

[gajus]

Or you run turbo watch build —filter={packages\*} with the latest canary build :)

Nice!

[weyert]

@gajus Yeah, looks pretty promising. I think it would be great as the developer of turbowatch to give feedback too :)

[neongreen]

I’ve been using Turbowatch until now, so I’d love to know how the built-in watch mode compares.

[gajus]

@gajus Yeah, looks pretty promising. I think it would be great as the developer of turbowatch to give feedback too :)

I have given a ton of feedback to the team already when they just started working on it. Eager to see what made it in!

[iamnafets]

Seems like this was launched with v2. https://turbo.build/blog/turbo-2-0

[anthonyshew]

Thank you everyone for the conversation around this feature. As @iamnafets mentioned, it is now available in Turborepo 2.0.