Angular 12.1.0 big rebuild time

[artaommahe]

🐞 Bug report

Command (mark with an x)

• new
• build
• serve
• test
• e2e
• generate
• add
• update
• lint
• extract-i18n
• run
• config
• help
• version
• doc

Is this a regression?

It was much faster for rebuilds with ng11.1.2 + NG_BUILD_IVY_LEGACY=1 (see the table below).

Description

Rebuild time for AppComponent changes (adding console.logs) increase drastically with the amount of code.

We have relatively big application. Here are some measures for starting server and rebuilding on changes in AppComponent for different cases.

	server	AppComponent
default	180s	65s
default, --aot=false	105s	3.5s
12 of 24 lazy routes disabled	125s	45s
18 of 24 lazy routes disabled	100s	8.5s
all lazy routes disabled	52s	2.2s
ng11 default	350s	14s

(ng11 - 11.1.2 version with NG_BUILD_IVY_LEGACY=1)
(all changes are adding console.log() statements to AppComponent, median of 3-4 passes excluding first on)

🔬 Minimal Reproduction

Don't have a public reproduction. Attaching rebuild performance profiling for 18 of 24 lazy routes disabled case.

CPU-20210707T141946.cpuprofile
https://drive.google.com/file/d/1LQzweIGTErNLmjcLhDjnGOSWiVbol-gR/view?usp=sharing

Cant profile the default case with all routes enabled - chrome crashes trying to dump all this stuff :(

🔥 Exception or Error

🌍 Your Environment

Angular CLI: 12.1.0
Node: 12.18.1
Package Manager: yarn 2.4.2
OS: win32 x64

Angular: 12.1.0
... animations, cdk, cli, common, compiler, compiler-cli, core
... elements, forms, language-service, localize, material
... material-moment-adapter, platform-browser
... platform-browser-dynamic, platform-server, router
... youtube-player

Package                         Version
---------------------------------------------------------
@angular-devkit/architect       0.1201.0
@angular-devkit/build-angular   12.1.0
@angular-devkit/core            12.1.0
@angular-devkit/schematics      12.1.0
@angular/flex-layout            12.0.0-beta.34
@ngtools/webpack                12.1.0
@nguniversal/builders           12.1.0
@nguniversal/express-engine     10.1.0
@schematics/angular             12.1.0
rxjs                            6.6.7
typescript                      4.3.4
webpack                         5.40.0

Anything else relevant?

This issue affects several of our projects relatively same size. Most of our code is inside libs/ subfolders and is imported directly by sources (nrwl/nx like monorepo).

Can debug something else, share screen or provide TeamViewer access to our codebase to check anything (discord drow#4335)

[alan-agius4]

I wonder if this the same as angular/angular-cli#21116 which was determined to be an issue in the TypeScript compiler.

Can you check if you get similar results running TSC directly.

tsc --watch --project <path-to-app-tsconfig>/tsconfig.app.json

[artaommahe]

> packages\student-cabinet> yarn tsc --watch --project tsconfig.app.json

[19:10:56] Starting compilation in watch mode...
[19:11:47] Found 0 errors. Watching for file changes.

[19:13:21] File change detected. Starting incremental compilation...
[19:13:22] Found 0 errors. Watching for file changes.

[19:13:34] File change detected. Starting incremental compilation...
[19:13:35] Found 0 errors. Watching for file changes.

Same changes in AppComponent, nothing commented out (default case from above), ~50s starting and ~1s rebuilding.

Also you can see low results for the default, --aot=false case in first post. So only aot mode is affected by this regression.

[JoostK]

Could you please also run 11 without NG_BUILD_IVY_LEGACY=1; that environment variable makes for an unfair comparison as NG_BUILD_IVY_LEGACY=1 moves type-checking off to a different process, not blocking JS emit.

The Ivy build pipeline is doing incremental type-checking, instead of offloading it to a different worker. From your profile however I notice that type-checking of templates have to be redone. This might be due to a path normalization issue on Windows, where the cached template diagnostics are missed and therefore recomputed.

Additionally, what is strange is that the profile is showing that program structure could not be reused. Was this perhaps the first rebuild? That would explain why no program reuse was available and also why template type-checking had to be redone (the initial rebuild is tainted because of how template type-checking is done, preventing program reuse and it invalidates prior type-check results)

[artaommahe]

Was this perhaps the first rebuild

There are 3-4 rebuilds starting with the second one for each case before getting average value. First rebuild in ng12 is roughly the same as the following ones, first one in ng11.1.2 with NG_BUILD_IVY_LEGACY=1 is 4-5 times slower (100s vs 14s following)

This might be due to a path normalization issue on Windows

same issue has my colleague with MacOS for the same project (asked him to recheck tomorrow and to record cpu profile for the 18 of 24 lazy routes disabled case)

run 11 without NG_BUILD_IVY_LEGACY=1;

ng 11.1.2, default case, NG_BUILD_IVY_LEGACY=0

• server start 400s
• first rebuild 337s
• second rebuild 325s

also noticed a huge memory consumption ~6.2gb for first rebuild and ~7.1gb for the second (looks like all free memory). Same for ng12 rebuilds - uses up to all available memory (started with node --max-old-space-size=8192)

[gianmd]

It seems very similar to angular/angular-cli#21228 , is possible that you have tailwindcss@2.2.4 in your project?

[artaommahe]

nope, regular scss styling, default cli serve (no webpack config customization), ~300-400 images required with !file-loader inline-loader (if this could matter)

[artaommahe]

cpu profile of 4th rebuild, 18 of 24 lazy routes disabled case, MacOS
https://drive.google.com/file/d/1PO2CBRNoeERlidPY3NMnOdK845R-4ceA/view?usp=sharing

[JoostK]

Could you set "angularCompilerOptions": { "tracePerformance" : "/path/to/log.json" } in a tsconfig.json file? That should write additional performance statistics to the specified file. That property is not documented/supported and the generated output format is unstable, but it might provide some insights into what is going on.

Note: the flag does not have an effect when used with the CLI, only using the ngc binary.

[artaommahe]

node --max-old-space-size=8000 ..\..\node_modules\@angular\compiler-cli\src\main.js -p tsconfig.app.json --watch

Server started

{
  "events": {
    "InputDtsFile": 1759,
    "InputTsFile": 10517,
    "AnalyzeComponent": 1682,
    "AnalyzeDirective": 77,
    "AnalyzeInjectable": 687,
    "AnalyzeNgModule": 559,
    "AnalyzePipe": 47,
    "TraitAnalyze": 3052,
    "GenerateTcb": 1682,
    "UpdateTypeCheckProgram": 1,
    "EmitSourceFile": 5374
  },
  "phases": {
    "Unaccounted": 2149638,
    "Setup": 1452,
    "TypeScriptProgramCreate": 14520731,
    "TypeScriptDiagnostics": 16945268,
    "Analysis": 6057360,
    "Resolve": 1093654,
    "CycleDetection": 135902,
    "TcbGeneration": 3850441,
    "TcbUpdateProgram": 3214251,
    "TypeScriptEmit": 15904762,
    "Compile": 4737047,
    "TtcDiagnostics": 8363886
  },
  "memory": {
    "Initial": 71360216,
    "TypeScriptProgramCreate": 577769256,
    "Analysis": 1347536192,
    "Resolve": 1363283944,
    "TtcUpdateProgram": 1831566080,
    "PreEmit": 1989555840,
    "Emit": 2415238888
  }
}

First rebuild

{
  "events": {
    "InputDtsFile": 1759,
    "InputTsFile": 10517,
    "AnalyzeComponent": 1682,
    "AnalyzeDirective": 77,
    "AnalyzeInjectable": 687,
    "AnalyzeNgModule": 559,
    "AnalyzePipe": 47,
    "TraitAnalyze": 3052,
    "GenerateTcb": 1682,
    "UpdateTypeCheckProgram": 1,
    "EmitSourceFile": 5374
  },
  "phases": {
    "Unaccounted": 73803,
    "Setup": 115024,
    "TypeScriptProgramCreate": 7789981,
    "Reconciliation": 17248,
    "TypeScriptDiagnostics": 16368127,
    "Analysis": 5063386,
    "Resolve": 1461536,
    "CycleDetection": 252679,
    "TcbGeneration": 4182421,
    "TcbUpdateProgram": 2983010,
    "TypeScriptEmit": 13762470,
    "Compile": 4761262,
    "TtcDiagnostics": 8074679
  },
  "memory": {
    "Initial": 1620269440,
    "TypeScriptProgramCreate": 1720257688,
    "Analysis": 2285657080,
    "Resolve": 2178945408,
    "TtcUpdateProgram": 2627735968,
    "PreEmit": 2780433640,
    "Emit": 2759891536
  }
}

Second rebuild

{
  "events": {
    "InputDtsFile": 1759,
    "InputTsFile": 10517,
    "AnalyzeComponent": 1682,
    "AnalyzeDirective": 77,
    "AnalyzeInjectable": 687,
    "AnalyzeNgModule": 559,
    "AnalyzePipe": 47,
    "TraitAnalyze": 3052,
    "GenerateTcb": 1682,
    "UpdateTypeCheckProgram": 1,
    "EmitSourceFile": 5374
  },
  "phases": {
    "Unaccounted": 87090,
    "Setup": 20138,
    "TypeScriptProgramCreate": 7773523,
    "Reconciliation": 24136,
    "TypeScriptDiagnostics": 15853157,
    "Analysis": 4532055,
    "Resolve": 975659,
    "CycleDetection": 141955,
    "TcbGeneration": 3536365,
    "TcbUpdateProgram": 2478772,
    "TypeScriptEmit": 12085801,
    "Compile": 3330321,
    "TtcDiagnostics": 7736459
  },
  "memory": {
    "Initial": 2762701880,
    "TypeScriptProgramCreate": 2861967504,
    "Analysis": 2445319304,
    "Resolve": 2449485656,
    "TtcUpdateProgram": 2886851304,
    "PreEmit": 3059858832,
    "Emit": 3411603528
  }
}

Third rebuild

{
  "events": {
    "InputDtsFile": 1759,
    "InputTsFile": 10517,
    "AnalyzeComponent": 1682,
    "AnalyzeDirective": 77,
    "AnalyzeInjectable": 687,
    "AnalyzeNgModule": 559,
    "AnalyzePipe": 47,
    "TraitAnalyze": 3052,
    "GenerateTcb": 1682,
    "UpdateTypeCheckProgram": 1,
    "EmitSourceFile": 5374
  },
  "phases": {
    "Unaccounted": 80811,
    "Setup": 17180,
    "TypeScriptProgramCreate": 7958840,
    "Reconciliation": 10814,
    "TypeScriptDiagnostics": 14910830,
    "Analysis": 4866759,
    "Resolve": 1037635,
    "CycleDetection": 127137,
    "TcbGeneration": 3612529,
    "TcbUpdateProgram": 2565787,
    "TypeScriptEmit": 12319332,
    "Compile": 3264253,
    "TtcDiagnostics": 7595661
  },
  "memory": {
    "Initial": 3400710728,
    "TypeScriptProgramCreate": 3495610032,
    "Analysis": 4120582984,
    "Resolve": 4135344448,
    "TtcUpdateProgram": 4566947992,
    "PreEmit": 4743617952,
    "Emit": 5093426736
  }
}

[artaommahe]

Thx to @JoostK, the main issue is fixed by this PR #42759. Recompile time droped to this (default case)

- first rebuild 41735ms
- second 15193ms
- third 18841ms
- fourth 16084ms

still investigating why it's 16-18s for two files recompiling (AppComponent + root module, or for main.ts changes)

[JoostK]

I think I have an idea for where the remaining time comes from. The template type checker may need to add type-checking code to user code, if the same effect cannot be achieved from within the type-check file (e.g. when a directive has an internal generic type, it cannot generate the type constructor external to the directive). If this adds imports to the file, the program structure cannot be reused and also incremental type-checking is not as effective, as imports affect the computed shape. This incurs significant costs on program creation and type-checking time.

[vugar005]

Maybe this blog by @brandonroberts can be helpful. https://dev.to/brandontroberts/speeding-up-the-development-serve-after-upgrading-to-angular-v12-5db5

[artaommahe]

Is not related to our issues. Already have these config after regular ng migration command

[JoostK]

I have been doing some experiments with the repo I typically use for perf issues, but this does not reproduce for me. I have forced the usage of inline type constructors but then #42759 is sufficient to mitigate the performance issues (that PR will be in this week's release).

I'm afraid that without a reproduction we can run locally this isn't actionable at the moment.

[pichuser]

@artaommahe did you find any solution? hugely slow

[artaommahe]

@pichuser nope, dunno what to debug. Still 16s to rebuild main.ts for a big project

[vugar005]

Hi @artaommahe can you check the fix in the last comment of #20792. CC: @JoostK, @alan-agius4

[artaommahe]

@vugar005

• already using regular angular builder for that project
• cant just remove yarn.lock and update all packages in our monorepo and i don't think i would help in any way
• re-checked all migrations adjustments to angular.json and it looks fine (under the spoiler)

"student-cabinet": {
  "projectType": "application",
  "schematics": {},
  "root": "packages/student-cabinet",
  "sourceRoot": "packages/student-cabinet/src",
  "prefix": "app",
  "architect": {
    "build": {
      "builder": "@angular-devkit/build-angular:browser",
      "options": {
        "outputPath": "packages/student-cabinet/dist",
        "index": "packages/student-cabinet/src/index.html",
        "main": "packages/student-cabinet/src/main.ts",
        "polyfills": "packages/student-cabinet/src/polyfills.ts",
        "tsConfig": "packages/student-cabinet/tsconfig.app.json",
        "assets": [
          "packages/student-cabinet/src/favicon.ico",
          "packages/student-cabinet/src/robots.txt",
          "packages/student-cabinet/src/assets",
          {
            "glob": "**/*",
            "input": "libs/shared/ui-kit-v2/assets/stratos-skyeng-font",
            "output": "./assets"
          },
          {
            "glob": "apple-developer-merchantid-domain-association.txt",
            "input": "packages/student-cabinet/src/",
            "output": "/.well-known"
          }
        ],
        "styles": ["packages/student-cabinet/src/assets/styles/studcab-styles.scss"],
        "scripts": [
          {
            "input": "packages/student-cabinet/src/assets/js/glider.min.js",
            "bundleName": "glider-bundle"
          }
        ],
        "stylePreprocessorOptions": {
          "includePaths": ["packages/student-cabinet/src/assets/styles/", "libs/"]
        },
        "allowedCommonJsDependencies": [
          "zone.js",
          "lodash",
          "js-cookie",
          "ua-parser-js",
          "ios-version",
          "util",
          "hammerjs",
          "vimbox-player",
          "react",
          "react-dom",
          "uuid",
          "date-fns-timezone"
        ]
      },
      "configurations": {
        "production": {
          "fileReplacements": [
            {
              "replace": "packages/student-cabinet/src/environments/environment.ts",
              "with": "packages/student-cabinet/src/environments/environment.prod.ts"
            }
          ],
          "outputHashing": "all",
          "vendorChunk": true,
          "budgets": [
            {
              "type": "initial",
              "maximumWarning": "2.6mb",
              "maximumError": "5mb"
            },
            {
              "type": "anyComponentStyle",
              "maximumWarning": "30kb",
              "maximumError": "45kb"
            }
          ]
        },
        "development": {
          "buildOptimizer": false,
          "optimization": false,
          "vendorChunk": true,
          "extractLicenses": false,
          "sourceMap": false,
          "namedChunks": true
        }
      },
      "defaultConfiguration": "production"
    },
    "serve": {
      "builder": "@angular-builders/custom-webpack:dev-server",
      "options": {
        "browserTarget": "student-cabinet:build",
        "port": 443,
        "ssl": true,
        "disableHostCheck": true
      },
      "configurations": {
        "production": {
          "browserTarget": "student-cabinet:build:production"
        },
        "development": {
          "browserTarget": "student-cabinet:build:development"
        }
      },
      "defaultConfiguration": "development"
    },
    "test": {
      "builder": "@nrwl/jest:jest",
      "options": {
        "jestConfig": "packages/student-cabinet/jest.config.js",
        "passWithNoTests": true
      },
      "outputs": ["coverage/packages/student-cabinet"]
    },
    "lint": {
      "builder": "@nrwl/linter:eslint",
      "options": {
        "lintFilePatterns": [
          "packages/student-cabinet/**/*.ts",
          "packages/student-cabinet/**/*.html",
          "packages/student-cabinet/**/translation/*.json"
        ]
      }
    }
  }
},

[alan-agius4]

@artaommahe, I noticed that you are using @angular-builders/custom-webpack:dev-server for your serve command. Can you check it the problem persists if you use the official builder @angular-devkit/build-angular:dev-server?

[artaommahe]

Removing "emitDecoratorMetadata": true from tsconfig.json reduced our build time significantly for production.

and

Deleted node_modules and package_lock.json
Ran npm install

haven't helped at all. Same ~16s for second+ rebuild of main.ts file

I noticed that you are using @angular-builders/custom-webpack:dev-server for your serve command.

🤦‍♂️ forgot to replace... Fixed it - nothing changed :(

ng 12.1.3

Just tried ng 12.2.1 - same 16s 😕

[alan-agius4]

@artaommahe, unfortunately as @JoostK mentioned without a reproduction, even shared privately it will not be possible to get to the bottom of this.

[artaommahe]

@alan-agius4 as i said in first post - i can debug anything or provide teamviewer access. Discord tag is also there.

[alxhub]

Closing as obsolete - please feel free to reopen if you're still experiencing pathological slowness with v13.2+

[angular-automatic-lock-bot]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}