Deployer unable to find composer installed recipes

[jamieburchell]

Q	A
Issue Type	Question
Deployer Version	6.0.3
Local Machine OS	WSL
Remote Machine OS	CentOS 7

Description

Deployer fails to find and require recipes installed via the method included in the third party recipe instructions

Steps to reproduce

Install recipes:

composer require deployer/recipes --dev

Add a recipe as per docs:

require 'recipe/cachetool.php';

Run deployment:

dep deploy production
OR
php deployer.phar deploy production

Result:

PHP Warning:  require(recipe/cachetool.php): failed to open stream: No such file or directory in /mnt/d/web/deploy.php on line 3
PHP Fatal error:  require(): Failed opening required 'recipe/cachetool.php' (include_path='phar:///mnt/d/web/deployer.phar/bin/../:.:/usr/share/php') in /mnt/d/web/blog.ib3.uk/deploy.php on line 3

Content of deploy.php

<?php
namespace Deployer;

require 'recipe/composer.php';
require 'recipe/cachetool.php';

// Project name
set('application', 'Example');

// Project repository
set('repository', 'git@bitbucket.org:foo');

// Hosts
host('xxx')
	->stage('production')
	->user('deploy')
    ->set('deploy_path', '/var/www')
    ->set('cachetool', '127.0.0.1:9000');

// Shared files/dirs between deploys
add('shared_dirs', [
	'web/app/upgrade',
	'web/app/uploads',
	'web/app/wfcache',
	'web/app/wflogs'
]);

// Copy config
task('enable_prod_config', function() {
	run('cp "{{release_path}}/.env.prod" "{{release_path}}/.env"');
})->onStage('production');
after('deploy:update_code', 'enable_prod_config');

// Clear opcache
after('deploy:symlink', 'cachetool:clear:opcache');

// [Optional] Allocate tty for git clone. Default value is false.
set('git_tty', true);

// [Optional] if deploy fails automatically unlock.
after('deploy:failed', 'deploy:unlock');

Worked around the issue thus:

ini_set('include_path', ini_get('include_path') . PATH_SEPARATOR . 'vendor/deployer/recipes');

Should that be part of the documtation or is something broken?

[antonmedv]

I think it's a bug

[antonmedv]

Can you find and fix it?

[jamieburchell]

I can’t see how Deployer was designed to find the composer installed recipes. Presumably, it needs to know where the vendor directory is (read composer.json/assume default?) and to add the path to those recipes to the include path? Or the expectation on the user is to simply provide the full path to the third party recipes. Perhaps, if Deployer is run and installed using the local composer method, it would find the recipes?

[antonmedv]

It's using include path.

[jordanbrauer]

I am having this problem as well.

Perhaps recipes should be redesigned in how they are implemented? My initial thought would be to do something similar to how Symfony handles extension – simple abstract classes and interfaces.

Deployer already uses Symfony Console, so why not utilize the class autoloading of composer? It will make extension much easier I think.

[antonmedv]

@jordanbrauer what do u suggest?

[jamieburchell]

At a minimum update the documentation to show that the include path needs to include the deployer recipes directory. As it stands all the examples and documents suggest it "just works" without, when it doesn't.

[antonmedv]

Hmm, maybe better to fix include path of all repos? :) I'll do it.

[jordanbrauer]

@antonmedv I would handle recipes the same way Symfony handles their single components and autoloading.

• Have it so that a user only needs to require deployerphp/deployer for the core functionality (common recipe)
• Any additional recipes should be required as singletons (again, akin to Symfony single components), e.g., deployerphp/rsync
  • This allows users to select only the recipes they really need for their project
• To use a recipe, simply require/use the file/namespace in your deployer file (again, see usage of Symfony standalone components if you need an idea of what I mean)

Ideally, this would mean ditching the use of require "recipe/rsync.php"; in favour of use Deployer\Recipe\Rsync;. Since deployer seems to work off of global functions, using use will work fine for importing.

[jamieburchell]

Would this still work if like me, you've installed the deployer.phar in to the project root and not via composer? That is the first installation method shown on the install page.

[jordanbrauer]

Good question – I am sure that there would be a way to make it work yes. As an example, when you add entries to the "scripts" key of a composer.json file, Composer will update it's set of commands, adding yours to it (try it by running composer list, and you will see your commands).

My point is, is that if a PHP CLI application is installed globally via a .phar rather than as a local composer binary, the application can still be extended for a local instance only.

[antonmedv]

I think we should keep it simple and do not reinvent the wheel. Use what already in php and composer.

[jordanbrauer]

I am not sure I understand @antonmedv? None of what I mentioned requires reinventing the wheel and uses default composer autoloading with native PHP imports (use). 😮

[antonmedv]

It's impossible to autoload tasks. In php and composer we can autoload only classes (not even functions, but you can use function ).

I think something like this will be good:

require 'vendor/autoload.php'; // Or move it to deployer and automatically detect
require 'common.php'
require 'slack.php'

All recipes have to provide autoload.php file with set_include_path to composer.json.

[jordanbrauer]

Ahh, you are right! My bad.

I think that user should be responsible for handling the inclusion of their autoload file. This way they are able to bootstrap however they please, even if they are not using composer.

[kszymukowicz]

From my experience doing require 'vendor/autoload.php'; from deploy.php level can make a conflicts with deployer packages.

The reason is that composer uses spl_autoload_register(..., true, true); The important is the second 'true' param in spl_autoload_register function. Its "prepend" and means the classes searching for this spl_autoload_register will be done before other registered ealier (by composer autoload from deployer in that case). That means that if deployer will want to create symfony console object then it will use class from root project vendors and not from itself vendors (inside phar). That of course can lead to problems if there are different version of symfony console for deployer phar and for root project.

The solution I know is to make light loader and register it with spl_autoload_register but without the third "prepend" param set to true. Then if there are the same libraries used in deployer and root project - deployer will use itself library first.

The example of such loader which uses information generated by composer itself is in this little library: https://github.com/sourcebroker/deployer-loader

I'd be glad to hear for better solution.
So far its working for me well.

[antonmedv]

I think it will be cool to have possibility to include recipes like this without worried about autoload:

require 'common.php'
require 'slack.php'

Deployer already has autoload.php detection https://github.com/deployphp/deployer/blob/master/bin/dep#L52-L56
So it should work if proper include paths will be configured.
In case when deployer installed globally and locally, local version should be used (dep automatically will switch to local version). In this case recipes installed by composer will be used and this is correct.

[antonmedv]

So now, then using phar one think should be done is requiring projects autoload.php

[antonmedv]

As another solution: fork all used symfony components and rename namespace.

[kszymukowicz]

@antonmedv
How the "deployer already has autoload.php detection" should work then ?
I can not create object from my vendor class if I have no require 'vendor/autoload.php' (or my autoload) in deploy.php. So for my understanding the autoload from root project is not loaded.

[antonmedv]

So for my understanding the autoload from root project is not loaded.
Here is a problem: when using phar it will load only phar autoload, when composer installed – project root.

Yes, it's true.

One more solution: when building phar archive create a bootstrap file (with all needed classes) and require it as first line in bin/dep file. After load project root's autoload.php file.
This way symfony classes will be already preloaded and no need of deployer's autoload.
I like this approach better when others. But it may have a bug: if using something in deploy.php what is relaying on older symfony components when bootstraped with deployer.

[kszymukowicz]

@antonmedv
IMO there will be never problem with adding require 'vendor/autoload.php' to deploy.php when you install deployer with composer require deployer/deployer as at this level developer already needs to solve the potential conflicts.

Problem will only occur ONLY when we use phar based deployer. For that case there is simple solution however. Deployer needs to detect if there is vendors/autoload.php from root project and if yes they it should require that file BEFORE require autoload from phar. This way the deployer classes will have precedence over that one from root vendors.

[antonmedv]

Deployer needs to detect if there is vendors/autoload.php from root project and if yes they it should require that file BEFORE require autoload from phar.

Yes, this is a solution too. I think I'll try to implement this.

[antonmedv]

After lots of thinking, here is what I came up with:

Installing with composer

• Deployer will try to locate autoload.php file (local or global) and require it.
• Scan vendor dir for composer.json files and fetch stuff like this:

  "extra": {
      "deployer": {
          "include_path": "recipe/"
      }
  }

• Next add all found include_path to set_include_path — this allows to simple recipe requirement:

   require 'laravel.php';
   require 'slack.php';

Installing with archive

• Download phar to local deployer.phar or bin/dep.
• Commit phar file to repository (to save version number).
• Save scan on vendor step as on composer installation.
• No autoload.php file required, you need to manually add it to your deploy.php file if needed.
• All required files by deployer included in single bootstrap file inside phar.
• If called global installed dep it will check existing next files in project:
  • deployer.phar
  • bin/dep
  • vendor/bin/dep
• If on of this files found, call it instead of global on (this allows global dep (for example of version v7) to call project installed deployer).

[hooch]

Example remedy:

require 'vendor/deployer/recipes/recipe/rsync.php';

[jkobus]

Updating phar on our build server broke our deployment process due to:

dep6 deploy test -vvv
PHP Warning:  require_once(recipe/symfony4.php): failed to open stream: No such file or directory in /home/xxxxx/jenkins-slave/workspace/xxxxx.com_develop-

These recipes were present in that phar previously?

[antonmedv]

What version of dep?

[jkobus]

Well it seems that this problem is somewhere on my side, although I have no idea why this happens.
I downloaded deployer from this url - its 6.0.5:

https://deployer.org/releases/v6.0.5/deployer.phar (in bin as dep6)

I also have two other versions in bin:

6.3.0 (as dep)
4.3.1 (as dep4)

I only updated the 6.3.0 as two others are version-locked for compatibility.

The one used in deployment that went broken is 6.0.5 (dep6).

Im wondering if its possible that in bin I had an old phar with this receipt and after update to 6.3 it was removed. My include path during install:

PHP Fatal error:  require_once(): Failed opening required 'recipe/symfony4.php' (include_path='phar:///usr/local/bin/dep6/bin/../:.:/usr/share/php') in /home/xxx/jenkins-slave/workspace/xxx.com_develop-BE4ZFIKIUWRUVBXYXPR6E5MC2IFLQN4GNM2FRXXL35N7JLIBZUEA/deploy.php on line 9

[antonmedv]

include_path looks normal. Try running this script locally.

[davefarthing]

Just thought I'd share my experience with php 7.0.32:
I had the same problem require(): Failed opening required 'recipe/ .......

I finally traced it to the presence of the ioncube_loader extension. Disabled ioncube and everything works normally.

[ihipop]

include_path looks normal. Try running this script locally.

why not just include the autoloader？ #1371 (comment)

[alinalexandru]

I am having same problem caused by ioncube.
Everything works perfectly without ioncube

[alinalexandru]

This is how I "solved" the issue

require 'phar:///usr/local/bin/dep/recipe/common.php';

Even though the get_include_path() was
phar:///usr/local/bin/dep/bin/../:.:/usr/local/lib/php

[antonmedv]

Well. This one is over. Almost 3 years since opening. Now recipes gonna live in deployer repo: https://github.com/deployphp/deployer/tree/master/contrib

[jamieburchell]

What's the status of this please? I just noticed deployer/recipes is abandoned but including recipe/cachetool.php without them doesn't work for me and the composer and CodeIgniter recipes I use in older projects are missing it seems?

Was the recipe repo prematurely archived and abandoned before v7 of Deployer was released?

[memen45]

A little late, but for future reference: some recipes are in the contrib directory, some are in the recipe directory. Example for your case:

require 'contrib/cachetool.php';

Other example:

require 'recipe/laravel.php';

So check in which of the two directories the recipe resides, then include the correct one!

[jamieburchell]

A little late, but for future reference: some recipes are in the contrib directory, some are in the recipe directory.

And the documentation refers to something else that's abandoned.

https://deployer.org/recipes.html

[memen45]

Indeed, documentation is wildly incomplete and contradicting itself. I do not know what the idea is of the maintainer(s), as the website looks very nice, but the information is very limited.

[Schrank]

@jamieburchell @memen45 As always in live, playing with the kids, computer or enjoying lunch with your loved ones is more important than correcting documentation or code on a open source project.

That said: we are happy for every hint of wrong stuff. Happy for a pull request, but happy for a note as well. My free time highly differs, I don't know about Anton, but I guess the same. But I love to fix things on my free time :-)

[jamieburchell]

@jamieburchell @memen45 As always in live, playing with the kids, computer or enjoying lunch with your loved ones is more important than correcting documentation or code on a open source project.

Nobody said those things aren't more important.

I'm tired of reading "update the documentation yourself" on open source projects because the whole point usually is that someone has opened a bug report or is asking questions because they've spent a whole lot of time trying to fathom how something is supposed to work, hunting through code and issues and ironically, relying on the documentation.

In this case, we don't know what the intention is of the project - why deployer recipes were deprecated, what the expectation is to make it work or how it is supposed to work. So with no clue or documentation the only thing someone can do is create an issue, like this.

The time seems to be there to deprecate and change stuff and confuse everybody, but not to explain why.

I'd certainly prefer to play with my son than try and second guess, that's for sure.

[Schrank]

That is a very good point and I joined the team late, so same for me. Than let's fallback to the first point. You ask questions/tell us what your problems are and we try to improve the doc so the next one finds the answer in the doc, works for you?

[jamieburchell]

Hey, no blame here just frustration ;)

The original issue I opened was that following the documentation and including recipes didn't work without adding the location of said recipes to the include path. I don't know if that was ever addressed as all my deploy scripts just set the location of the recipes in the include path now.

Fast forward some time and the deployer/recipes package was archived but I couldn't figure out how to bring in the recipes I was using before such as cache tool because they were missing from the "recipes" directory. During this time the documentation was still (and is still) pointing at how to use the recipes via the archived packages. Today I learnt from a reply here that I should look for some of the recipes in the contrib folder but I have not tried that yet. It may be the missing part of the jigsaw. I still use the archived composer package and ignore the deprecation warning, because I assumed the stuff I used from there wasn't available without it.

[jamieburchell]

Yeah so there is no contrib folder because deployer v7 isn't outside of beta yet. So the deployer/recipes is still the only way to get this even though it's archived. And the original purpose of this bug report is tagged as v7 milestone too.

[memen45]

@jamieburchell @memen45 As always in live, playing with the kids, computer or enjoying lunch with your loved ones is more important than correcting documentation or code on a open source project.

Really appreciate the efforts made by the contributors of course! Just wanted to help out here. With regards to the documentation, I have seen many projects where this is lacking. And that is not a problem at all. I think the following could help a lot:

• Use versions in the documentation
• Mark incomplete sections in the documentation (e.g. 'This section is incomplete', 'This section requires examples' etc). This way users are aware that it isn't complete, and at the same time potential contributors know how to contribute.
• Provide some steps for contributors to contribute properly (i.e. how to build and test the project locally on your pc). I noticed a pull request Copy shared files from release #2267 where the contributor doesn't know how to make a test. And for me, I know some php and the code base seems to be very clearly written, but I have no idea how to modify and then build the source code again. Adding a small guide (if possible just provide links to existing guides and make remarks for the changes) could really help to make contributing easier.

Those are just some things that I noticed as I started diving in only weeks ago. Apart from this, I really like the Deployer software and the functionality it brings, so thanks a lot for that!

[memen45]

Yeah so there is no contrib folder because deployer v7 isn't outside of beta yet. So the deployer/recipes is still the only way to get this even though it's archived. And the original purpose of this bug report is tagged as v7 milestone too.

You are indeed correct, the latest stable version 6.8.0 doesn't have the contrib folder yet.

[jamieburchell]

I think that's where the confusion has come from - between the deployer/recipes composer package being archived/deprecated but not able to find the recipes in the current stable version and of course the confusing documentation when faced with the above. By the way, I love Deployer and use it on all my work projects.

[antonmedv]

Will work on documentation for sure. Just need to find time)

[trip-somers]

I just read every comment in this thread, and I'm still not sure what I'm supposed to do to fix this problem.

• Installed phar globally following installation instructions in documentation.
• require succeeds for present recipe folder.
• require fails for non-existent contrib folder.

I did not see a solution in this thread for getting contrib files to work. Did I miss it?

[memen45]

@trip-somers Install the beta 7.0.0 instead of 6.8.0 and you will find the contrib directory. There is no workaround as far as I know.

[vajda-media]

I have to use the new 8.x-dev version of deployer and sweating like dirty pig to get it run. Please I would really appriciate a v8 documentation as well. Many thanks!