Default to Composer-based site building instead of Drush Make

[geerlingguy]

Background:

• Use of Composer (instead of Drush Make) for building Drupal site #506 and PR Issue #506: Add support for composer based site builds #694
• http://docs.drupalvm.com/en/latest/deployment/composer/

Drupal VM currently supports building codebases using Composer or Drush make files. We were going to default to building with drupal-project and/or a composer.json file instead of the example drush.make.yml file, but some performance testing showed abysmal regressions in terms of the initial site build—and even fully cached site builds using Composer were 2x or more slower than Drush make file builds.

I'd like to move to using Composer as the default, as that's kind of where the puck is going... but before that can happen, we need to continue poking and prodding at Composer and all the default setup to make sure the Composer install is not a horrendously slow user experience, especially the first time (uncached).

@oxyc did some awesome investigative work and found that there are a few things we can do out of the gates to speed things up, like:

• Configure vagrant-cachier correctly to cache everything from Composer outside the VM so it persists across builds.
• Include prestissimo to speed up composer commands by parallelizing downloads
• Use --prefer-dist

But I'm hoping there are some other optimizations we can make to get the composer installation time down quite a bit. I know @thom8 suggested composer/composer#4961 might help by using shallow Git clones, but we definitely need hard data (benchmarks) to back up theories.

[oxyc]

Also worth checking how much slowdown "minimum-stability": "dev" causes. Ref. A bit old so might not be the case anymore.

[jmolivas]

did you turned off x-debug from the CLI?

[fubarhouse]

I'm curious about build times in comparison to the drupal-console and drush make methods, and the possibility of optionally switching xdebug on/off during this process...

[oxyc]

xdebug is by default disabled so the tests from earlier ran without it. I added a few tests in PR #700 to get some basic comparison readings with different configs.

[oxyc]

Test	802	803	804	805	806
Makefile	9:12	9:16	9:11	9:03	9:23
Composer			11:59	11:56	12:25
Composer + XDebug	13:19	13:11	14:17	13:11	13:10
Composer min stable		11:35	11:44	13:20	12:33
Composer shallow clones	12:36	12:00	12:09	11:47	12:32
Prestissimo dev-master					10:18 (!)

Before when we used working copy in the makefile the build time was 10-11min. Now it's pretty consistently ~9min.

Another note, I ran composer install -vvv with drupal/core:~8.0 as the only dependency. Downloading all those packagist json files took:

[217.6MB/400.82s] Analyzed 3319 packages to resolve dependencies
[217.6MB/400.82s] Analyzed 51116 rules to resolve dependencies

as soon as they were in my cache:

[215.2MB/24.05s] Analyzed 3275 packages to resolve dependencies
[215.2MB/24.05s] Analyzed 49891 rules to resolve dependencies

/cc @webflo is there any way we or packagist.drupal-composer.org can optimize that part? Or is it just how composer works?

Edit: Found this drupal-composer/drupal-packagist#48

---

The shallow clones PR won't help us here. All my packages are fetched as zipballs and they download in 3sec compared to the 400sec packagist json step.

[oxyc]

Perfect timing! hirak/prestissimo#101 (merged 22min ago haha)

With empty cache it's now down from 400s to:

[303.1MB/165.89s] Analyzed 5377 packages to resolve dependencies
[303.1MB/165.89s] Analyzed 94262 rules to resolve dependencies

[oxyc]

With parellel download of packages.json files I think we can actually go forward with defaulting to composer based installs. 10min 18sec on travis vs 9min 23sec for makefiles. We should be pretty much on par with when we were using working-copy makefile builds which we didn't remove until 3.0.0.

[oxyc]

Opened up a WIP PR #701

[geerlingguy]

Well... Maybe we helped push them over the edge there :)

Good thing I didn't push a tag yet!

[geerlingguy]

did you turned off x-debug from the CLI?

I don't enable it unless I explicitly need it—I'm an old-fashioned guy who normally prints/logs debug statements and I only resort to Xdebug and its ilk if needed :)

By default, it's not enabled in CLI in Drupal VM.

[geerlingguy]

From my home in STL, on my laptop, using time vagrant up each time:

Test type	Time
Makefile default config	00:06:01
Composer drupal-project (#701), 1st run	00:10:41
Composer drupal-project (#701), 2nd run	00:08:07
Composer drupal-project (#701), cachier 1st run	00:08:29
Composer drupal-project (#701), cachier 2nd run	00:06:34
Composer drupal-project (#701), cachier 2nd run (with :apt)	00:05:59

Cachier stored about 131M of files in my cache (eesh!)

[heddn]

@oxyc @geerlingguy you might also look at what is being done in https://github.com/platformsh/platformsh-example-drupal/pull/20 for some goodness.

Specifically: drush/policy.drush.inc

[geerlingguy]

@heddn - So far I don't want to go so far as to make people using a drush/make-based workflow feel like it's a painful experience, so a policy file discouraging/blocking the use of pm commands would be a bit extreme... but at some point, we'll be dropping Drupal 7 support entirely, and at that time I think it's fine to do this.

[geerlingguy]

Let's do this thing! I'm writing up a blog post with some of the learnings from here—thanks in HUGE part to @oxyc's experimentation. I think that's like 92 beers or so at this point. Sooner or later I'll just have to buy a keg. 🍻

[oxyc]

@heddn that looks to be based on drupal-composer/drupal-project which we actually default to now :) https://github.com/drupal-composer/drupal-project

[geerlingguy]

Blog post posted: http://www.jeffgeerling.com/blog/2016/speeding-composer-based-drupal-installation

Please let me know if there are errors or things missing there! Let's poke and prod the community to get things even better! :)

[joestewart]

Great post Jeff. One item related to drupal-packagist infrastructure. drupal.org is now publishing this information for contrib modules. Once the drupal core split is available drupal.org infrastructure can be used for the composer repository.

[geerlingguy]

@joestewart - Great point! Do you think it will be set up for caching via Fastly out of the gate, or will it take some time to get that part of the drupal.org-based packagist configured and running?

[webflo]

https://packages.drupal.org/8 is already backed by Fastly. I don't plan to change the infrastructure of https://packagist.drupal-composer.org/. I think its not worth the effort. I hope packages.drupal.org becomes stable in a few weeks.

[oxyc]

For reference here's my output of running:

rm -rf foo
rm composer.lock
composer clear-cache
composer create-project drupal-composer/drupal-project:8.x-dev foo --stability dev --no-progress --profile --prefer-dist -vvv

https://gist.github.com/oxyc/e9a89257fd05368992b0b722a71a0604

Some notes:

• 0-40s is essentially because we use create-project (auch). It seems Composer doesn't load plugins until after the project has been created and therefore the first repository packages.json files are downloaded synchronously. I'm guessing this is a Composer limitation but I opened a question over at Is it possible to hook in during create-project? hirak/prestissimo#103. If this is fixed upstream somehow/somewhere it would be a nice performance improvement for us.
• 42-233s is downloading all the dependency information. This was the step that used to be 400sec. This will improved further once https://packages.drupal.org is up and running.
• 233-236s dependency resolution
• 236-256s download packages. Last time I checked this was way faster, I think it's just temporary network issues on my side.
• 256-277s extracting zip files (I have a 1.1ghz dual-core CPU so this will be faster for others).
• 277-281s parsing suggestions etc, generating autoload
• 281-297s drupal scaffold which downloads index.php/.htaccess etc (this could technically be faster by being async Integrate with Prestissimo async downloaders drupal-composer/drupal-scaffold#42).

[geerlingguy]

Looks like getting packages.drupal.org up and running is going to be the biggest short-term win!

[Crell]

Since we were mentioned... 😄

The disabling of pm-* commands isn't a Platform.sh thing. It's a change that was made upstream to https://github.com/drupal-composer/drupal-project about 3 months ago. We're inheriting that project and following it as closely as we can, just tweaking the result a bit to make Drupal play nicer with our read-only file system.

I also just switched our installer to use packages.drupal.org today, at Mixologic's recommendation. Hopefully that will go live today or early tomorrow.

[geerlingguy]

I also just switched our installer to use packages.drupal.org today, at Mixologic's recommendation. Hopefully that will go live today or early tomorrow.

Nice - I'd love to hear how it goes! Any issue on drupal.org I could follow for progress/updates?

[oxyc]

I also just switched our installer to use packages.drupal.org today, at Mixologic's recommendation. Hopefully that will go live today or early tomorrow.

Cool I didn't understand it was up and running!

That particular step decreased by 62sec by switching over to https://packages.drupal.org/8 on my laptop.

---

Also tested how much making drupal-scaffold async would improve the performance. 5sec but unfortunately Prestissimos CurlRemoteFilesystem wasn't fully implemented so no easy-switch for when it's available.

[geerlingguy]

@oxyc - Can you submit a PR for it (the change to packages.drupal.org), and I'll merge once we get word upstream that it will be semi-officially or officially supported?

[Crell]

Note that you'll need to change the "conflict" line to "replace" when switching to the official repos.

[oxyc]

Prestissimo 0.3.1 now hooks into create-project and downloads all packages.json files in async. hirak/prestissimo#103 (comment)

[geerlingguy]

@oxyc - Nice! We're using

drupal-vm/default.config.yml

Line 238 in 7344434

- { name: hirak/prestissimo, release: '^0.3' }

, so we're already covered in terms of new installs and updates, right?

[oxyc]

Yup, not sure if the version is updated for existing boxes though.