optimalization of the file system enumerator, which now uses the glob function (in Symfony Finder)

[Rastusik]

• this helps with using aspects from vendor folder, which needs to be scanned, but with proper include paths, the scan is not so slow anymore

[icanhazstring]

Thanks for the pull request. Solid changes there.
Would love if you address the requested changes.

But don't know how @lisachenko goes with adding another package dependency.
For me its fine tho. Thanks again.

[icanhazstring]

I would recommend using Finder as a dependency for now. Should make up for a more solid test.

Also I just realized, that this operation here is a bit IO heavy.
If you call enumerate a number of times you will always hit IO to scan again for files.
I get that there could be new files added between two concurrent runs, but I guess this would not happen on a production environment.

Maybe @lisachenko could say more about this ;)

[Rastusik]

Well, the IO heavy operation is there anyway, whether with Finder or using the old way... I didn't want to mess with the original design, this was supposed to be just an optimization... hopefully there won't be a problem with the new dependency, since it isn't dependent of anything else...

[Rastusik]

@icanhazstring not sure what you meant with the requested changes - just the typo or also something regarding this conversation?

[icanhazstring]

Ye I know that this was like this before. Just realized it by now.
And after thinking this has nothing to do with this PR - might be subject for another PR - so for me everything is fine 👍

@lisachenko just has to accept/merge it ;)

[scrutinizer-notifier]

The inspection completed: 2 updated code elements

[Rastusik]

@icanhazstring sorry to bother, but is there a reason why this PR isn't merged yet? Do I have to do anything else to make things go faster? Just tell me and I'll do anything what is needed.

[icanhazstring]

@Rastusik unfortunately I don't have permission to merge. 😕

@lisachenko pls merge if you can :)

[lisachenko]

Hello, guys! Sorry for being unavailable, unfortunately I have not so much time now for working on my project, thus PRs accept time is increasing (

[lisachenko]

Regarding this PR, how it helps to apply aspects to vendor folders? Could you please elaborate? This feature was available before this PR.

[Rastusik]

@lisachenko yes, the mentioned feature (including aspects from vendor folder) was available before. This PR only optimizes the file scanning process, so the scan is much faster then before (the scan length is proportional to the amount of files in the vendor folder). The scans were slowing my development process, so I made them faster using glob patterns (under the hood in Symfony Finder)

[lisachenko]

@Rastusik ok, I understood. Of course, I don't like too much adding new dependency :) But reinventing the wheel is not good option too...

Ok, let's proceed with symfony/finder component dependency.

[lisachenko]

This issue brings some pain under Windows platform.
I can see a lot of PCRE warnings right now:

PHP Warning:  preg_match(): Compilation failed: a numbered reference must not be zero 
at offset 8 in ...\vendor\symfony\finder\Iterator\MultiplePcreFilterIterator.php on line 57

Looks like symfony/finder component don't like Windows slashes for path name.

[icanhazstring]

Ouch. Maybe we should setup a travis build for windows as well?
So we can see such errors before hand.

And ofc fix this one as well ;)

[lisachenko]

Ouch. Maybe we should setup a travis build for windows as well?

Yes, definitely. Previously AppVeyour was used, but it's buggy as hell, half of builds were broken. But now Travis-CI support Windows https://blog.travis-ci.com/2018-10-11-windows-early-release, so it's natural way to enable this.

[icanhazstring]

But now Travis-CI support Windows https://blog.travis-ci.com/2018-10-11-windows-early-release, so it's natural way to enable this.

Currently php support seems a bit buggy. But possible:
https://travis-ci.community/t/where-to-contribute-php-support-for-windows/304

[Rastusik]

If someone creates a windows build somewhere (at least appveyor), I can take o look and make a fix

[icanhazstring]

@Rastusik build on windows works now. You can start :D

[lisachenko]

I don't know why it is not reproduced on Windows build right at the moment, but look at https://3v4l.org/DITGg to see that it is real.

Stack trace is following:

PHP Warning:  preg_match(): Compilation failed: a numbered reference must not be zero at offset 8 in H:\Work\go\vendor\symfony\finder\Iterator\MultiplePcreFilterIterator.php on line 57
PHP Stack trace:
PHP   1. {main}() H:\Work\go\tests\Fixtures\project\bin\console:0
PHP   2. Symfony\Component\Console\Application->run($input = *uninitialized*, $output = *uninitialized*) H:\Work\go\tests\Fixtures\project\bin\console:22
PHP   3. Symfony\Component\Console\Application->doRun($input = class Symfony\Component\Console\Input\ArgvInput { private $tokens = array (0 => '--no-ansi', 1 => 'cache:warmup:aop', 2 => 'H:\\Work\\go\\tests\\Fixtures\\project\\web/../web/index.php'); private $parsed = array (); protected $definition = class Symfony\Component\Console\Input\InputDefinition { private $arguments = array (...); private $requiredCount = 2; private $hasAnArrayArgument = FALSE; private $hasOptional = FALSE; private $options = array (...); private $shortcuts = array (...) }; protected $stream = NULL; protected $options = array ('no-ansi' => TRUE); protected $arguments = array ('command' => 'cache:warmup:aop', 'loader' => 'H:\\Work\\go\\tests\\Fixtures\\project\\web/../web/index.php'); protected $interactive = TRUE }, $output = class Symfony\Component\Console\Output\ConsoleOutput { private $stderr = class Symfony\Component\Console\Output\StreamOutput { private $stream = resource(72) of type (stream); private ${Symfony\Component\Console\Output\Output}verbosity = 32; private ${Symfony\Component\Console\Output\Output}formatter = class Symfony\Component\Console\Formatter\OutputFormatter { ... } }; private ${Symfony\Component\Console\Output\StreamOutput}stream = resource(66) of type (stream); private ${Symfony\Component\Console\Output\Output}verbosity = 32; private ${Symfony\Component\Console\Output\Output}formatter = class Symfony\Component\Console\Formatter\OutputFormatter { private $decorated = FALSE; private $styles = array (...); private $styleStack = class Symfony\Component\Console\Formatter\OutputFormatterStyleStack { ... } } }) H:\Work\go\vendor\symfony\console\Application.php:148
PHP   4. Symfony\Component\Console\Application->doRunCommand($command = class Go\Console\Command\CacheWarmupCommand { protected $aspectKernel = class Go\Tests\TestProject\Kernel\DefaultAspectKernel { protected $options = array (...); protected $wasInitialized = TRUE; protected $container = class Go\Core\GoAspectContainer { ... } }; private ${Symfony\Component\Console\Command\Command}application = class Symfony\Component\Console\Application { private $commands = array (...); private $wantHelps = FALSE; private $runningCommand = ...; private $name = 'UNKNOWN'; private $version = 'UNKNOWN'; private $commandLoader = NULL; private $catchExceptions = TRUE; private $autoExit = TRUE; private $definition = class Symfony\Component\Console\Input\InputDefinition { ... }; private $helperSet = class Symfony\Component\Console\Helper\HelperSet { ... }; private $dispatcher = NULL; private $terminal = class Symfony\Component\Console\Terminal { ... }; private $defaultCommand = 'list'; private $singleCommand = FALSE; private $initialized = TRUE }; private ${Symfony\Component\Console\Command\Command}name = 'cache:warmup:aop'; private ${Symfony\Component\Console\Command\Command}processTitle = NULL; private ${Symfony\Component\Console\Command\Command}aliases = array (); private ${Symfony\Component\Console\Command\Command}definition = class Symfony\Component\Console\Input\InputDefinition { private $arguments = array (...); private $requiredCount = 2; private $hasAnArrayArgument = FALSE; private $hasOptional = FALSE; private $options = array (...); private $shortcuts = array (...) }; private ${Symfony\Component\Console\Command\Command}hidden = FALSE; private ${Symfony\Component\Console\Command\Command}help = 'Initializes the kernel and, if successful, warm up the cache for PHP\nfiles under the application directory.\n\nBy default, the cache directory is taken from configured AspectKernel class.'; private ${Symfony\Component\Console\Command\Command}description = 'Warm up the cache with woven aspects'; private ${Symfony\Component\Console\Command\Command}ignoreValidationErrors = FALSE; private ${Symfony\Component\Console\Command\Command}applicationDefinitionMerged = TRUE; private ${Symfony\Component\Console\Command\Command}applicationDefinitionMergedWithArgs = TRUE; private ${Symfony\Component\Console\Command\Command}code = NULL; private ${Symfony\Component\Console\Command\Command}synopsis = array ('short' => 'cache:warmup:aop <loader>', 'long' => 'cache:warmup:aop <loader>'); private ${Symfony\Component\Console\Command\Command}usages = array (); private ${Symfony\Component\Console\Command\Command}helperSet = class Symfony\Component\Console\Helper\HelperSet { private $helpers = array (...); private $command = NULL } }, $input = class Symfony\Component\Console\Input\ArgvInput { private $tokens = array (0 => '--no-ansi', 1 => 'cache:warmup:aop', 2 => 'H:\\Work\\go\\tests\\Fixtures\\project\\web/../web/index.php'); private $parsed = array (); protected $definition = class Symfony\Component\Console\Input\InputDefinition { private $arguments = array (...); private $requiredCount = 2; private $hasAnArrayArgument = FALSE; private $hasOptional = FALSE; private $options = array (...); private $shortcuts = array (...) }; protected $stream = NULL; protected $options = array ('no-ansi' => TRUE); protected $arguments = array ('command' => 'cache:warmup:aop', 'loader' => 'H:\\Work\\go\\tests\\Fixtures\\project\\web/../web/index.php'); protected $interactive = TRUE }, $output = class Symfony\Component\Console\Output\ConsoleOutput { private $stderr = class Symfony\Component\Console\Output\StreamOutput { private $stream = resource(72) of type (stream); private ${Symfony\Component\Console\Output\Output}verbosity = 32; private ${Symfony\Component\Console\Output\Output}formatter = class Symfony\Component\Console\Formatter\OutputFormatter { ... } }; private ${Symfony\Component\Console\Output\StreamOutput}stream = resource(66) of type (stream); private ${Symfony\Component\Console\Output\Output}verbosity = 32; private ${Symfony\Component\Console\Output\Output}formatter = class Symfony\Component\Console\Formatter\OutputFormatter { private $decorated = FALSE; private $styles = array (...); private $styleStack = class Symfony\Component\Console\Formatter\OutputFormatterStyleStack { ... } } }) H:\Work\go\vendor\symfony\console\Application.php:248
PHP   5. Go\Console\Command\CacheWarmupCommand->run($input = class Symfony\Component\Console\Input\ArgvInput { private $tokens = array (0 => '--no-ansi', 1 => 'cache:warmup:aop', 2 => 'H:\\Work\\go\\tests\\Fixtures\\project\\web/../web/index.php'); private $parsed = array (); protected $definition = class Symfony\Component\Console\Input\InputDefinition { private $arguments = array (...); private $requiredCount = 2; private $hasAnArrayArgument = FALSE; private $hasOptional = FALSE; private $options = array (...); private $shortcuts = array (...) }; protected $stream = NULL; protected $options = array ('no-ansi' => TRUE); protected $arguments = array ('command' => 'cache:warmup:aop', 'loader' => 'H:\\Work\\go\\tests\\Fixtures\\project\\web/../web/index.php'); protected $interactive = TRUE }, $output = class Symfony\Component\Console\Output\ConsoleOutput { private $stderr = class Symfony\Component\Console\Output\StreamOutput { private $stream = resource(72) of type (stream); private ${Symfony\Component\Console\Output\Output}verbosity = 32; private ${Symfony\Component\Console\Output\Output}formatter = class Symfony\Component\Console\Formatter\OutputFormatter { ... } }; private ${Symfony\Component\Console\Output\StreamOutput}stream = resource(66) of type (stream); private ${Symfony\Component\Console\Output\Output}verbosity = 32; private ${Symfony\Component\Console\Output\Output}formatter = class Symfony\Component\Console\Formatter\OutputFormatter { private $decorated = FALSE; private $styles = array (...); private $styleStack = class Symfony\Component\Console\Formatter\OutputFormatterStyleStack { ... } } }) H:\Work\go\vendor\symfony\console\Application.php:953
PHP   6. Go\Console\Command\CacheWarmupCommand->execute($input = class Symfony\Component\Console\Input\ArgvInput { private $tokens = array (0 => '--no-ansi', 1 => 'cache:warmup:aop', 2 => 'H:\\Work\\go\\tests\\Fixtures\\project\\web/../web/index.php'); private $parsed = array (); protected $definition = class Symfony\Component\Console\Input\InputDefinition { private $arguments = array (...); private $requiredCount = 2; private $hasAnArrayArgument = FALSE; private $hasOptional = FALSE; private $options = array (...); private $shortcuts = array (...) }; protected $stream = NULL; protected $options = array ('no-ansi' => TRUE); protected $arguments = array ('command' => 'cache:warmup:aop', 'loader' => 'H:\\Work\\go\\tests\\Fixtures\\project\\web/../web/index.php'); protected $interactive = TRUE }, $output = class Symfony\Component\Console\Output\ConsoleOutput { private $stderr = class Symfony\Component\Console\Output\StreamOutput { private $stream = resource(72) of type (stream); private ${Symfony\Component\Console\Output\Output}verbosity = 32; private ${Symfony\Component\Console\Output\Output}formatter = class Symfony\Component\Console\Formatter\OutputFormatter { ... } }; private ${Symfony\Component\Console\Output\StreamOutput}stream = resource(66) of type (stream); private ${Symfony\Component\Console\Output\Output}verbosity = 32; private ${Symfony\Component\Console\Output\Output}formatter = class Symfony\Component\Console\Formatter\OutputFormatter { private $decorated = FALSE; private $styles = array (...); private $styleStack = class Symfony\Component\Console\Formatter\OutputFormatterStyleStack { ... } } }) H:\Work\go\vendor\symfony\console\Command\Command.php:255
PHP   7. Go\Instrument\ClassLoading\CacheWarmer->warmUp() H:\Work\go\src\Console\Command\CacheWarmupCommand.php:52
PHP   8. Go\Instrument\FileSystem\Enumerator->enumerate() H:\Work\go\src\Instrument\ClassLoading\CacheWarmer.php:60
PHP   9. iterator_to_array(class Symfony\Component\Finder\Iterator\PathFilterIterator { protected $matchRegexps = array (); protected $noMatchRegexps = array (0 => '#H:\\Work\\go\\tests\\Fixtures\\project\\var\\cache\\aspect#', 1 => '#H:\\Work\\go\\src#', 2 => '#(^|/)\\..+(/|$)#') }) H:\Work\go\src\Instrument\FileSystem\Enumerator.php:80
PHP  10. Symfony\Component\Finder\Iterator\PathFilterIterator->rewind() H:\Work\go\src\Instrument\FileSystem\Enumerator.php:80
PHP  11. Symfony\Component\Finder\Iterator\PathFilterIterator->accept() H:\Work\go\src\Instrument\FileSystem\Enumerator.php:80
PHP  12. Symfony\Component\Finder\Iterator\PathFilterIterator->isAccepted($string = 'Annotation/Loggable.php') H:\Work\go\vendor\symfony\finder\Iterator\PathFilterIterator.php:35
PHP  13. preg_match('#H:\\Work\\go\\tests\\Fixtures\\project\\var\\cache\\aspect#', 'Annotation/Loggable.php') H:\Work\go\vendor\symfony\finder\Iterator\MultiplePcreFilterIterator.php:57

[Rastusik]

@lisachenko so I was thinking again and it seems that the fix could be quite easy, but I'm not sure whether I won't break anything with it. The problem seems to be the usage of backslashes in the windows paths. AFAIK the solution would be to use slashes everywhere, even on windows machines, but I'm not 100% sure with that.

The easy solution should be as follows. There is this method in the AspectKernel:

protected function normalizeOptions(array $options): array
    {
        $options = array_replace($this->getDefaultOptions(), $options);

        $options['cacheDir'] = PathResolver::realpath($options['cacheDir']);

        if (!$options['cacheDir']) {
            throw new \RuntimeException('You need to provide valid cache directory for Go! AOP framework.');
        }

        $options['excludePaths'][] = $options['cacheDir'];
        $options['excludePaths'][] = __DIR__ . '/../';
        $options['appDir']         = PathResolver::realpath($options['appDir']);
        $options['cacheFileMode']  = (int) $options['cacheFileMode'];
        $options['includePaths']   = PathResolver::realpath($options['includePaths']);
        $options['excludePaths']   = PathResolver::realpath($options['excludePaths']);

        return $options;

This method uses PathResolver to sanitize the paths in the options config.

In the realPath() method of the PathResolver, there is this code:

// resolve path parts (single dot, double dot and double delimiters)
        $path = str_replace(['/', '\\'], DIRECTORY_SEPARATOR, $path);
        if (strpos($path, '.') !== false) {
            $parts     = explode(DIRECTORY_SEPARATOR, $path);
            $absolutes = [];
            foreach ($parts as $part) {
                if ('.' === $part) {
                    continue;
                }
                if ('..' === $part) {
                    array_pop($absolutes);
                } else {
                    $absolutes[] = $part;
                }
            }
            $path = implode(DIRECTORY_SEPARATOR, $absolutes);
        }

I would suggest to not to use the DIRECTORY_SEPARATOR constant anywhere - I would replace the backslashes for slashes. What do you think?