BuildService finding non-existent files to copy in Debian #662

clifmo · 2024-11-10T14:19:54Z

I'm using Docker to build and I just sort of expected this to work. I've tried multiple version of PHP, different Debian releases. In short, BuildService tries to copy a file named _media.css which is the name of the folder, and not one of the file contents.

I started using Alpine but PHP glob is a known issue. So it's interesting that this also relates to glob(). MediaFile.php:109

I haven't had a chance to debug it but I'm curious why the directory is being considered as an asset, and what I can do to avoid this. Composer wants ^8.1 but I get a composer platform check error on 8.1. What am I missing?

    protected static function getMediaAssetFiles(): array
    {
        return glob(Hyde::path(static::getMediaGlobPattern()), GLOB_BRACE) ?: [];
    }

    protected static function getMediaGlobPattern(): string
    {
        return sprintf(Hyde::getMediaDirectory().'/{*,**/*,**/*/*}.{%s}', implode(',',
            Config::getArray('hyde.media_extensions', self::EXTENSIONS)
        ));
    }

FROM php:8.2-cli-buster AS build-php

WORKDIR  /app

COPY . .

RUN php hyde build

> [4/4] RUN php hyde build:
0.143
0.143
0.143                           Building your static site!
0.143
0.143
0.147 Removing all files from build directory...
0.147
0.147 Transferring Media Assets...
0.152  0/1 [░░░░░░░░░░░░░░░░░░░░░░░░░░░░]   0%
0.156    ErrorException
0.156
0.156   copy(/app/_media/_media.css): Failed to open stream: No such file or directory
0.156
0.156   at vendor/hyde/framework/src/Framework/Services/BuildService.php:60
0.157      56▕         $this->comment('Transferring Media Assets...');
0.157      57▕         $this->withProgressBar(MediaFile::files(), function (string $identifier): void {
0.157      58▕             $sitePath = Hyde::siteMediaPath($identifier);
0.157      59▕             $this->needsParentDirectory($sitePath);
0.157   ➜  60▕             copy(Hyde::mediaPath($identifier), $sitePath);
0.157      61▕         });
0.157      62▕
0.157      63▕         $this->newLine(2);
0.157      64▕     }
0.157
0.157       +17 vendor frames
0.157
0.157   18  hyde:35
0.157       LaravelZero\Framework\Kernel::handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
0.157
------
Dockerfile:8
--------------------
   6 |
   7 |     #RUN php hyde publish:homepage posts -n
   8 | >>> RUN php hyde build

The text was updated successfully, but these errors were encountered:

caendesilva · 2024-11-10T14:36:08Z

This feel very weird. Could you provide a full ls/tree of the source media directory so I can reproduce?

@clifmo Are you using _media.css as the filename of the directory? I could see that being the cause of the bug, since the glob doesn't differentiate. Should be a fairly easy fix, would just want the ls -r output of _media so I can verify a fix works for your problem.

clifmo · 2024-11-10T15:51:11Z

Sure I saw this on a fresh install and my active install only has one PNG file and app CSS. I'll provide reproduction steps once I'm at my PC

clifmo · 2024-11-10T17:44:29Z

I have CI/CD working in my homelab via Gitea Actions, a private Docker registry, and Watchtower to pull in updates. So I build via Dockerfile and publish in my action. Was hoping this could work but glob has been trouble.

Install
composer create-project hyde/hyde hyde-test && cd hyde-test

Check
Everthing looks good.
php hyde info
php hyde serve

ls -r _media
app.css

Docker Time

echo "FROM php:8.2-cli-buster AS build-php

WORKDIR  /app

COPY . .

RUN php hyde build" >> Dockerfile

docker build .

 > [4/4] RUN php hyde build:
0.188
0.188
0.188                           Building your static site!
0.188
0.188
0.190 Removing all files from build directory...
0.190
0.190 Transferring Media Assets...
0.193  0/1 [░░░░░░░░░░░░░░░░░░░░░░░░░░░░]   0%
0.195    ErrorException
0.195
0.195   copy(/app/_media/_media.css): Failed to open stream: No such file or directory
0.195
0.195   at vendor/hyde/framework/src/Framework/Services/BuildService.php:60
0.195      56▕         $this->comment('Transferring Media Assets...');
0.195      57▕         $this->withProgressBar(MediaFile::files(), function (string $identifier): void {
0.195      58▕             $sitePath = Hyde::siteMediaPath($identifier);
0.195      59▕             $this->needsParentDirectory($sitePath);
0.195   ➜  60▕             copy(Hyde::mediaPath($identifier), $sitePath);
0.195      61▕         });
0.195      62▕
0.195      63▕         $this->newLine(2);
0.195      64▕     }
0.195
0.195       +17 vendor frames
0.195
0.195   18  hyde:35
0.195       LaravelZero\Framework\Kernel::handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
0.195
------
Dockerfile:7
--------------------
   5 |     COPY . .
   6 |
   7 | >>> RUN php hyde build
   8 |
--------------------
ERROR: failed to solve: process "/bin/sh -c php hyde build" did not complete successfully: exit code: 1

Build from host: macOS Sequoia 15.0.1

❯ php hyde build


                          Building your static site!


Removing all files from build directory...

Transferring Media Assets...
 1/1 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%

Creating Blade Pages...
 2/2 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%


All done! Finished in 0.02 seconds (21.02ms) with 11.79MB peak memory usage
Congratulations! 🎉 Your static site has been built!
Your new homepage is stored here -> [..]/hyde-test/_site/index.html

❯ ls -r _media
app.css

❯ php -v
PHP 8.3.13 (cli) (built: Oct 22 2024 18:39:14) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.3.13, Copyright (c) Zend Technologies
    with Zend OPcache v8.3.13, Copyright (c), by Zend Technologies

FROM php:8.3 AS build-php is the same result.

caendesilva · 2024-11-10T18:03:26Z

Thank you so much for the detailed reproduction steps! I'll get right on debugging this.

See #662

caendesilva · 2024-11-10T18:16:43Z

It feels like glob() is the cause of many cross-platform issues (for example not existing on , so am considering switching to RecursiveIteratorIterator for improved stability in a future update.

I believe this issue is due to Debian having a different implementation for handling glob patterns. I tried a patch in 802ad4f, could you try to update your Docker environment to try out this branch in order to test if the fix works for you?

You can run this command to try out the development branch:

composer config repositories.framework vcs https://github.com/hydephp/framework && composer require "hyde/framework:dev-check-media-file-directories" --update-with-dependencies

clifmo · 2024-11-10T18:52:12Z

Well, I was working off my host, then copying into a docker environment during docker build. I did this entirely inside the container and didn't have issues.

I still had the same issue with my Dockerfile. I did a few more steps to prove it.

Dockerfile

Commented the RUN build command.

FROM php:8.2-cli-buster AS build-php

WORKDIR  /app

COPY . .

# RUN php hyde build

Build

❯ docker build . -f Dockerfile.test -t temp:latest
[+] Building 1.5s (8/8) FINISHED                                                                                                                                                   docker:desktop-linux
 => [internal] load build definition from Dockerfile.test                                                                                                                                          0.0s
 => => transferring dockerfile: 126B                                                                                                                                                               0.0s
 => [internal] load metadata for docker.io/library/php:8.2-cli-buster                                                                                                                              0.8s
 => [internal] load .dockerignore                                                                                                                                                                  0.0s
 => => transferring context: 2B                                                                                                                                                                    0.0s
 => [1/3] FROM docker.io/library/php:8.2-cli-buster@sha256:c1a06439f7b38487a79e0eebe5ccfea1f86978f68b6d51bbba3482c38622fa8d                                                                        0.0s
 => [internal] load build context                                                                                                                                                                  0.2s
 => => transferring context: 391.39kB                                                                                                                                                              0.2s
 => CACHED [2/3] WORKDIR  /app                                                                                                                                                                     0.0s
 => [3/3] COPY . .                                                                                                                                                                                 0.3s
 => exporting to image                                                                                                                                                                             0.1s
 => => exporting layers                                                                                                                                                                            0.1s
 => => writing image sha256:43eadbdad5488dc8265d1726b7e73522d2bcc38cd4dd48fd2dc198d571adec7a                                                                                                       0.0s
 => => naming to docker.io/library/temp:latest

Run into built container

docker run -it temp:latest sh

Check _media

# ls -r _media
app.css

Ensure dev framework

# cat vendor/hyde/framework/src/Support/Filesystem/MediaFile.php | grep "array_filter($files"
        return array_filter($files, 'is_file');

Build

# php hyde build


                          Building your static site!


Removing all files from build directory...

Transferring Media Assets...
 0/1 [░░░░░░░░░░░░░░░░░░░░░░░░░░░░]   0%
   ErrorException

  copy(/app/_media/_media.css): Failed to open stream: No such file or directory

  at vendor/hyde/framework/src/Framework/Services/BuildService.php:60
     56▕         $this->comment('Transferring Media Assets...');
     57▕         $this->withProgressBar(MediaFile::files(), function (string $identifier): void {
     58▕             $sitePath = Hyde::siteMediaPath($identifier);
     59▕             $this->needsParentDirectory($sitePath);
  ➜  60▕             copy(Hyde::mediaPath($identifier), $sitePath);
     61▕         });
     62▕
     63▕         $this->newLine(2);
     64▕     }

      +17 vendor frames

  18  hyde:35
      LaravelZero\Framework\Kernel::handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

Docker Run

Fresh container without copying from host.

Host:
docker run -it php:8.3-cli sh

Container:

php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
php -r "if (hash_file('sha384', 'composer-setup.php') === 'dac665fdc30fdd8ec78b38b9800061b4150413ff2e3b6f88543c636f7cd84f6db9189d43a81e5503cda447da73c7e5b6') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;"
php composer-setup.php
php -r "unlink('composer-setup.php');"# Installer verified

mv composer.phar /usr/local/bin/composer

apt update && apt install git
composer create-project hyde/hyde hyde-test && cd hyde-test

# php hyde build


                          Building your static site!


Removing all files from build directory...

Transferring Media Assets...
 1/1 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%

Creating Blade Pages...
 2/2 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%


All done! Finished in 0.01 seconds (8.80ms) with 12.32MB peak memory usage
Congratulations! 🎉 Your static site has been built!
Your new homepage is stored here -> file:///hyde-test/_site/index.html

caendesilva · 2024-11-10T20:29:13Z

@clifmo Does it seem like your issue was fixed by the patch/proper Docker setup?

clifmo · 2024-11-10T20:59:12Z

@caendesilva It does not, unfortunately. I really do appreciate the help. In short, this only happens when copying data from the host system into the container.

I stuck a dd() to see what it sees during the build. There's only one result before the array_filter.
It's in discoverMediaAssetFiles() ... tracking this into ProjectFile::construct() but taken it about as far as dd() will take me.

0.130 Transferring Media Assets...
0.132 array:1 [
0.132   0 => "/app/_media/app.css"
0.132 ]

    protected static function discoverMediaAssetFiles(): array
    {

        return collect(static::getMediaAssetFiles())->mapWithKeys(function (string $path): array {
            $file = static::make($path);
dd($file);
            return [$file->getIdentifier() => $file];
        })->all();
    }

 => ERROR [4/4] RUN php hyde build                                                                                                                                                                 0.1s
------
 > [4/4] RUN php hyde build:
0.117
0.117
0.117                           Building your static site!
0.117
0.117
0.119 Removing all files from build directory...
0.119
0.119 Transferring Media Assets...
0.121 Hyde\Support\Filesystem\MediaFile {#360
0.121   +path: "_media.css"
0.121 }
------
Dockerfile:8
--------------------

caendesilva · 2024-11-10T21:08:36Z

@clifmo Thanks for providing all these information points. Sadly, I am personally not knowledgeable enough in Docker to fully understand what quirks are happening here, so I asked on Twitter to see if anyone else could take a look so we can get this fixed.

In the meantime I will be working on switching to use the RecursiveDirectoryIterator as I think it will be more reliable, but I'm going on two out of state trips this week so it may take longer than I would wish for that to be released as I want to make sure it is done properly in order to not introduce any new issues.

clifmo · 2024-11-10T21:10:07Z

Have safe trips! I will possibly better learn Github Actions or hack something quick and dirty. Cheers

caendesilva · 2024-11-10T21:12:22Z

Have safe trips! I will possibly better learn Github Actions or hack something quick and dirty. Cheers

Thank you! By the way, we do have a GitHub Action if you're looking to build (and deploy) your Hyde sites from a CI. https://github.com/hydephp/action

Here is also a blog post if you want to make a more lower-level action using basic Action tooling. https://hydephp.com/posts/github-actions-deployment

Let me know if you need any help!

clifmo · 2024-11-10T21:33:07Z

Yea! I saw that. As mentioned, I'm deploying to a homelab so my go-to method is monitoring Docker image updates with Watchtower. I'll have to figure out something to grab the artifact and stick it into the bind mount path. I suppose I can just scp it since it's all within my network, but I'll have to dig deeper into the docker networking, since it's all on the same host.

caendesilva · 2024-11-11T12:52:27Z

@clifmo I got some answers on Twitter. Let me know if this could narrow it down for you.

either:
they haven't created the missing fill outside the container to be copied,
they did but at the wrong path when translated into the container,
they didn't build the file inside the container if it's a compiled file, or
something is slightly off between how their paths map
https://x.com/MtgDank/status/1855746481937936658

Could you also confirm the following:

You wrote "this only happens when copying data from the host system into the container", which data were you copying, and is this behaviour reproducible? I'm trying to figure out if there is somehow a file being created, or if PHP is fully imagining it.
Is there at any point an actual file named _media.css? And if so, do we have any idea where it is created, because that is not a file from Hyde.

clifmo · 2024-11-11T12:56:22Z

I provided reproduction steps based on a clean install so there are no additional files. I built the simplest docker image possible. It's in the reproduction steps. I figured out an actions-based approach which required a bit more than docs suggest, at least for my setup in Gitea. Happy to contribute that back. Sorry to be short, it's Monday.

caendesilva · 2024-11-11T13:00:14Z

I provided reproduction steps based on a clean install so there are no additional files. I built the simplest docker image possible. It's in the reproduction steps. I figured out an actions-based approach which required a bit more than docs suggest, at least for my setup in Gitea. Happy to contribute that back. Sorry to be short, it's Monday.

Thanks for the information. Just wanted to double check that everything in the reproduction steps were complete as I am not too familiar with Docker.

I do have this Dockerfile which may be helpful, but I'm not 100% sure it works:

FROM php:8.1-cli

# Install system dependencies
RUN apt update && apt install zip unzip

# Install Composer
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer

# Download base project files 
RUN curl -L https://github.com/hydephp/hyde/archive/refs/heads/master.zip -o hyde.zip \
	&& unzip hyde.zip \
	&& rm hyde.zip \
	&& mv hyde-master hyde

# Install Hyde
RUN cd /hyde && composer install

# Set the working directory
WORKDIR /hyde

# Make the HydeCLI executable
RUN chmod +x /hyde/hyde

# Set HydeCLI as the entrypoint
ENTRYPOINT ["/hyde/hyde"]

# Set the default command
CMD ["build"]

There is also an experimental official Docker image for the standalone CLI project that you could maybe use: https://github.com/hydephp/cli/

clifmo · 2024-11-11T13:06:38Z

Sounds good. I have moved on from this so there's no immediacy. However there still is something funky with the collection of media assets, and it's not glob. Although glob has its own issues. I will write up how I got actions working in a private network, Gitea, docker environment. Fair?

caendesilva · 2024-11-11T13:08:57Z

Sounds good. I have moved on from this so there's no immediacy. However there still is something funky with the collection of media assets, and it's not glob. Although glob has its own issues. I will write up how I got actions working in a private network, Gitea, docker environment. Fair?

That sounds great! Would love the read. I'll keep the issue open for a while, and work on moving away from glob, regardless of if it is relevant to this. Hope this doesn't discourage you from exploring HydePHP further.

clifmo · 2024-11-11T13:12:30Z

Absolutely not, you have been incredibly responsive. Next step is using the Laravel backend notifications to cross-post to socials. Has anyone you know worked on this? Staring with AT Protocol and ActivityPub

caendesilva · 2024-11-11T13:15:49Z

Absolutely not, you have been incredibly responsive. Next step is using the Laravel backend notifications to cross-post to socials. Has anyone you know worked on this? Staring with AT Protocol and ActivityPub

Glad to hear it, thank you so much! I have actually not tried using notifications from Hyde, I love the idea and is something I'd also love to read a writeup on. I have not tried AT Protocol or ActivityPub but any Laravel notification driver should work.

I think the build tasks feature would be great if you want to send notifications during the site build. https://hydephp.com/docs/1.x/build-tasks

clifmo · 2024-11-11T21:31:47Z

Definitely going to try. As promised, from idea to entirely self-hosted ci/CD in a weekend. https://blog.clifmo.com/posts/using-gitea-actions-to-deploy-hydephp-in-a-self-hosted-docker-environment.html

caendesilva · 2024-11-12T08:09:03Z

Definitely going to try. As promised, from idea to entirely self-hosted ci/CD in a weekend. https://blog.clifmo.com/posts/using-gitea-actions-to-deploy-hydephp-in-a-self-hosted-docker-environment.html

Amazing writeup! And thanks for the shoutout ❤️

caendesilva · 2024-12-21T13:10:28Z

@clifmo This fix is now released and can be used by updating to HydePHP Framework v1.7.4. Happy holidays!

caendesilva added a commit that referenced this issue Nov 10, 2024

Filter out directories from glob result

802ad4f

See #662

This was referenced Dec 11, 2024

Replace glob brace usages with more robust solutions hydephp/develop#2063

Closed

[1.x] Replace GLOB_BRACE with a more robust solution hydephp/develop#2064

Merged

caendesilva closed this as completed in hydephp/develop#2064 Dec 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BuildService finding non-existent files to copy in Debian #662

BuildService finding non-existent files to copy in Debian #662

clifmo commented Nov 10, 2024

caendesilva commented Nov 10, 2024 •

edited

Loading

clifmo commented Nov 10, 2024

clifmo commented Nov 10, 2024

caendesilva commented Nov 10, 2024

caendesilva commented Nov 10, 2024

clifmo commented Nov 10, 2024

caendesilva commented Nov 10, 2024

clifmo commented Nov 10, 2024

caendesilva commented Nov 10, 2024 •

edited

Loading

clifmo commented Nov 10, 2024

caendesilva commented Nov 10, 2024

clifmo commented Nov 10, 2024 •

edited

Loading

caendesilva commented Nov 11, 2024

clifmo commented Nov 11, 2024

caendesilva commented Nov 11, 2024

clifmo commented Nov 11, 2024

caendesilva commented Nov 11, 2024

clifmo commented Nov 11, 2024

caendesilva commented Nov 11, 2024

clifmo commented Nov 11, 2024

caendesilva commented Nov 12, 2024

caendesilva commented Dec 21, 2024

BuildService finding non-existent files to copy in Debian #662

BuildService finding non-existent files to copy in Debian #662

Comments

clifmo commented Nov 10, 2024

caendesilva commented Nov 10, 2024 • edited Loading

clifmo commented Nov 10, 2024

clifmo commented Nov 10, 2024

caendesilva commented Nov 10, 2024

caendesilva commented Nov 10, 2024

clifmo commented Nov 10, 2024

Docker Run

caendesilva commented Nov 10, 2024

clifmo commented Nov 10, 2024

caendesilva commented Nov 10, 2024 • edited Loading

clifmo commented Nov 10, 2024

caendesilva commented Nov 10, 2024

clifmo commented Nov 10, 2024 • edited Loading

caendesilva commented Nov 11, 2024

clifmo commented Nov 11, 2024

caendesilva commented Nov 11, 2024

clifmo commented Nov 11, 2024

caendesilva commented Nov 11, 2024

clifmo commented Nov 11, 2024

caendesilva commented Nov 11, 2024

clifmo commented Nov 11, 2024

caendesilva commented Nov 12, 2024

caendesilva commented Dec 21, 2024

caendesilva commented Nov 10, 2024 •

edited

Loading

caendesilva commented Nov 10, 2024 •

edited

Loading

clifmo commented Nov 10, 2024 •

edited

Loading