Moving from bower to npm

[indicalabs]

Bower.io saying that "...psst! While Bower is maintained, we recommend yarn and webpack for new front-end projects! "

Should we migrate the exising bower assets configurations to npm?

What steps will reproduce the problem?

What is the expected result?

What do you get instead?

Additional info

Q	A
Yii version	2.0.?
PHP version
Operating system

[cebe]

something to consider when creating packages for 2.1. Thanks for noting that.

[schmunk42]

This is a very important issue!

Especially due to the fact that installation of bower and npm/yarn packages is totally different (flat vs. tree).

Since many repos are also dropping bower support, I started looking into using npm modules, but those currently lead to a bunch of dependency resolution issues, see also fxpio/composer-asset-plugin#307.
Many of those can not be solved with composer, because you can not have multiple versions of the same package with composer (and it's very unlikely that this is ever going to happen) - but with npm(!), which is to my understand possible, due to the fact that you can encapsulate code more easily in a JavaScript-based build process, than in many other languages. Please shed some more light on this if you can; see also Understanding the npm dependency model for some more information on that topic.

The good thing is, that we actually could use that to our advantage. While the current implementation of the fxp plugin parses dependencies from package.json for dependency resolution - shouldn't it be sufficient to use only peerDependencies for the SAT solver, along with PHP packages? They should work in very similar way. And it should be much less deps to check.

This would be much less error prone. But we would need a separate bundling process which reads package.json files for required assets. But then we'd have to best of both worlds: Dependency resolution for PHP & asset packages and freedom of bundling them for client side usage.

CC: @tonydspaniard @crisu83 @handcode @cebe @samdark

[johonunu]

@schmunk42 I would drop using composer-asset-plugin in 2.1, if you ask me. Composer should install php packages, not make it do other things. I would suggest others devs to use npm, yarn or webpack separately and make documentation on using them in yii2 projects. As for 2.0 version, I guess we would need to switch to npm from bower somehow.

[schmunk42]

I would drop using composer-asset-plugin in 2.1, if you ask me. Composer should install php packages, not make it do other things.

If you would completely drop dependency resolution for assets in 2.1 you could (need to) also remove assetManager completely, maybe even widgets - since they would make not much sense anymore. IMHO the current plugin version, which is mainly used for bower packages, actually works pretty well regarding dependency resolution - performance is another topic.
But it is way to strict when it comes to npm packages, actually it just works differently.

Rather than sacrificing dependency resolution between widgets/extension/modules and assets completely, I'd like to find/propose a solution which is much faster and less error-prone - and even without the need of AssetBundles at all.

How about changing the asset installation like so:

• you can still require a npm package in composer.json
• instead of crawling all dependencies, only peerDependencies are taken into account when doing dependency resolution
• no package is actually downloaded, but they are added to a Yii 2.1 project package.json which can then be installed with npm or yarn
  • because you can have multiple version of a package you can challenge this while bundling/parsing the JavaScript files, but you do not need to worry about it in step 1.
  • you can either include .js/.css from definitions in package.json or bundle it - no more AssetBundles which do different thing in YII_ENV_DEV and YII_ENV_PROD

[schmunk42]

@tonydspaniard I totally agree that there are tools which can handle asset management better than Yii + fxp currently does.

The use-case is not only about jQuery, take VueJs as an example - you start a nice module now, which requires VueJS 2.x, which might not be compatible with VueJs 3.x in the future. This is something you want to address. VueJs is the perfect example of a peerDependency.

I agree that we do not have to care about the sub-(non-peer)-dependencies like loadsh or whatever, this can and should handle webpack, gulp or whatever.

What I am proposing involves like only 2% of the - literally - constraints we have to obey now.
It boils down to "grab a list of npm modules and add them to a local project". The only "magic" involved would be that you would be notified early, if your npm installation would fail also. Even a bit more, you could react to these issues in advance.

[samdark]

IMHO I think that maybe is the time where Yii should consider the amazing tools that currently exist nowadays and make the jump if they need to make any changes to its asset management (yes, I said asset management, because all this trouble with npm or bower and including composer-asset-plugin is all about asset management).

As @schmunk42 noted, it will trigger removal of widgets (or at least reducing their scope a lot) cause most of widgets do not make sense w/o assets. Personally I'd do it to drop JS and assets burden but it could be too shocking change for Yii community.

Other frameworks even bet for certain js frameworks (VueJs) and helped the PHP framework users with pre-configured tools and js helpers to easily work with them. So I would consider creating those tools (webpack, npm, yarn, or whatever) and tell the users (officially) that they can choose whatever they wish here.

Never in the core again :) We've provided support for one of such great tools — jQuery which is now considered not-so-modern thing. We may consider providing template for REST apps (that's what's needed for vue, react etc.) But overall I'd like to focus on PHP and leave JS alone. Trends are moving too fast there.

[schmunk42]

Personally I'd do it to drop JS and assets burden but it could be too shocking change for Yii community.

A core completely without assets and an asset system as a Yiisoft extension would be great!

But overall I'd like to focus on PHP and leave JS alone. Trends are moving too fast there.

I agree, but we're also still building sites with jQuery in them 😨 😏
We need kind of an optional "minimal bridge" to the JS-ecosystem. We all lose, if everybody needs to reinvent asset-installation and bundling for their Yii 2.1 applications.

AFAIR the decision was once made, because Yii wanted to have a PHP only solution. This restriction should be dropped, but there should be a common workflow for developers when working with extensions and their assets.

[schmunk42]

@tonydspaniard It's not about locking deps in the end. It's about very minimal constraints between server and client packages and the fact that you might intervene in this process instead of completely separating it.

While this suitable for larger apps, it just makes no sense for tiny libs like a date-picker for example. You'd need another kind of "contact" between them, like a REST API or data-adapters, but this is not feasible for all situations.

[klimov-paul]

A core completely without assets and an asset system as a Yiisoft extension would be great!

This matter is already tracked by issue #8452.
There is a proposal for the future separation at #13839, however it is stuck at the present state.
Actually, I would appriciate, if instead of pointless debates here, people come to actual proposals for the possible implementation and concrete code reorganization.

Keep in mind that Yii core tools like yii2-debug and yii2-gii DEPENDS on JS code and jQuery in particular.

The core team needs a lot of helping hands to extract JS code into separated packages and update/rewrite JS parts of many crucial extensions like yii2-debug, yii2-gii and yii2-bootstrap. If you are willing to help - you may start now.

If you wish to continue disscussion about packages separation and structure, please use issue #8452 for that and do not mix 2 different things here.

[samdark]

and an asset system as a Yiisoft extension

Ideally without that or with it developed by interested parties. Possibly by the name of YiiSoft :)

there should be a common workflow for developers when working with extensions and their assets

Then there will be no difference. Add one of extensions and you have basically the same stuff you had in 2.0.

Yes, exactly my point. But what stops to support different types of asset management configuration outside of composer? Providing those tools I could use any of the js libraries I wish for SPA (API) or not.

Extensions and a huge mess if we'll not provide a standard way to manage dependencies and assets as we do now.

[samdark]

And yes, as @klimov-paul said, we need extra hands.

[samdark]

As for moving from bower itself, I don't think bower support will be dropped during Yii 2.0 lifetime so I'd consider it non-issue for 2.0.

[francoispluchino]

I agree with everyone about using Nodejs + NPM/Yarn to manage assets. I never liked to use the solver SAT of Composer to manage asset dependencies, but as your project, I have PHP libraries that must use asset dependencies for the front-end (as well as working with private repositories and lock file), it was for these reasons that I created this plugin.

In the meantime many things have changed on the side of Javascript, and Nodejs becomes almost mandatory to use and compile assets (typescript, babel, scss, less, etc...). NPM has evolved a lot in recent months (organizations, private repositories with git+ssh, a true lock file), and Bower is dead, and Yarn is a new great tool.

I personally removed my plugin from my main project for using Yarn + Webpack (recently). However, removing asset dependency management is really frustrating: when I install a new PHP library that using the assets, the front-end of the component does not work because there is no more management of the assets, and so the assets are missing if I do not add them manually to the package.json project file ... This is precisely the main purpose of the plugin.

All this to say that I started this morning to develop the version 2.0 using Nodejs (see the closed issue fxpio/composer-asset-plugin#93), but only compatible with NPM/Yarn, and installable in project mode. I know, there are Composer plugins that using Nodejs, but none allows to use the features to 100% for NPM or Yarn, and does not use the package.json file for PHP libraries, and therefore, don't use the standard workflow (not to mention the inability to return to previous versions of PHP dependencies in case of error). I respect their work very much, but it is not the approach I want for this tool. I am aware that this will create a break of compatibility (hence the major version change), but using the "best" of the both worlds is now possible (if I can put it that way). I found an elegant solution to manipulate and install NPM dependencies, and to go back to previous versions of PHP dependencies if an error occurs, while using the native package.json file with NPM or Yarn (you can choose the manager to use, and validate the version of this manager), but also using the Composer commands install/update/require/remove in a PHP project.

Of course, the entire plugin is configurable (in global config, in project config and with the environment variables) and it is possible to replace the installation or update commands to add options (ex. --dry-run to only check dependencies for NPM, or --flat for Yarn), but also to disable the installation of assets to keep only the modification of the file package.json, etc ... Perhaps you will be interested in this new version despite the break of compatibility.

Regarding migration: modify each PHP library by moving the asset dependencies of the composer.json file to a new package.json file and adding the fxp/composer-asset-plugin dependency to the composer.json file of each PHP library (in the section requires or requires-dev) should be very fast and simple. Moreover, placing the plugin dependency in each php library, allows you to make optional your widgets (and this plugin) if you wish.

For those interested, this new version of the plugin will be available tomorrow, I think at the end of the day, because I still have to write tests and the documentation.

PS. Composer Asset Plugin + Yarn: it's a rocket with the same advantages, and more, but without the disadvantages! :-)

[francoispluchino]

Given that there is a break compatibility, and it is impossible to have a version of the plugin installed globally, and another version installed in the project - because Composer will install the plugin in the project, but will only use the plugin installed globally - the plugin becomes a new project. Of course, the Composer Asset Plugin will continue to be maintained but it will no longer evolve without the contributions of the community.

To know: I have not yet finished, and it still lack the documentation for the use of the plugin, tests, and add the restoration of the project in the case if NPM or Yarn terminates with an error. All this will come in the next few days.

However, for the more reckless among you, you can start playing with Foxy (the new name of the plugin), and make returns and/or contribute. Do not forget, this is an alpha version, so a little leniency please :-).

[francoispluchino]

The Foxy's version v1.0.0-beta1 is released. All features are implemented and tested.