refactored SourceAdapters for better stream handling

CHANGELOG.md

@@ -9,10 +9,19 @@
 - MediaUploader will use the visibility defined on the filesystem disk config if the `makePublic()`/`makePrivate()` methods are not called.
 - MediaUploader now supports base64 and URL-encoded data URLs as an input source.
 - Improved MediaCollection annotions to support generic types
-- Removed `SourceAdapterInterface::getStreamResource()` method. The method has been replaced with the `getStream()`, which returns a PSR-7 `StreamInterface` instead.
 - Removed the `\Plank\Mediable\Stream` class in favor of the `guzzlehttp/psr7` implementation. This removes the direct dependency on the `psr/http-message` library.
-- The Media class now exposes a dynamic `url` attribute which will generate a URL for the file (equivalent to the getUrl() method).
+- The Media class now exposes a dynamic `url` attribute which will generate a URL for the file (equivalent to the `getUrl()` method).
 - Modernized the migration files to use more recent Laravel conventions.
+- All SourceAdapter classes have been significantly refactored.
+   - All sourceAdapters will now never load the entire file contents into memory to determine metadata about the file, in order to avoid memory exhaustion when dealing with large files.
+   - If reading the file is necessary, the adapter will attempt use a single streamed scan of the file to load all metadata at once, to speed up to the precess.
+  - Removed `getStreamResource()` method. The method has been replaced with the `getStream()`, which returns a PSR-7 `StreamInterface` instead.
+  - Added `hash()` method which is expected to return an md5 hash of the file contents, if available.
+  - The return types of the `path()`, `filename()`, `extension()`, methods are now nullable. If the adapter cannot determine the value from the information available, it should return null.
+  - If the extension is not available from the source adapter (e.g. source is a raw content string), the MediaUploader will attempt to infer the extension based on the MIME type.
+  - If the filename is not available from the source adapter (e.g. source is a raw content string), the MediaUploader will throw an exception if the filename is not provided as a configuration.
+  - Removed the `getContents()` method.
+  - Removed the `getSource()` method
 
 ## 5.5.0 - 2022-05-09
 - Filename and pathname sanitization will use the app locale when transliterating UTF-8 characters to ascii.

UPGRADING.md

@@ -6,11 +6,13 @@
 * Minimum Laravel version moved to 10
 * To add support for data URLs to the MediaUploader, the following entry should be added to the `source_adapters.pattern` field in `config/mediable.php`
   ```php
-  '^data:[^,]*,' => Plank\Mediable\SourceAdapters\DataUrlAdapter::class,
+  '^data:/?/?[^,]*,' => Plank\Mediable\SourceAdapters\DataUrlAdapter::class,
   ```
 * All properties now declare their types if able. If extending any class or implementing any interface from this package, property types may need to be updated.
-* The return type of `SourceAdapterInterface::getSource()` has been changed to `mixed`. If you have implemented a custom SourceAdapter, you will need to update the return type of this method.
-* The `SourceAdapterInterface::getStreamResource(): resource` method has been replaced with the `getStream(): StreamInterface` method. If you have implemented a custom SourceAdapter, you will need to implement the `getStream()` method instead.
+* If you have implemented a custom SourceAdapter, you will need to apply the following changes from the `SourceAdapterInterface` interface:
+  * The `getSource()`, `getContents()`, `getStreamResource()` methods have been removed
+  * Implement the `getStream(): StreamInterface` method.
+  * Implement the `getHash(): string` method.
 * The `Plank\Mediable\Stream` class has been removed in favor of the `guzzlehttp/psr7` implementation. If you were using this class directly, you will need use another PSR-7 compatible stream wrapper instead (such as Guzzle's).
 
 ## 4.x to 5.x

composer.json

@@ -26,7 +26,8 @@
         "guzzlehttp/guzzle": "^7.4.1",
         "symfony/http-foundation": "^6.0.3|^7.0",
         "guzzlehttp/psr7": "^2.6",
-        "symfony/mime": "^7.0"
+        "symfony/mime": "^7.0",
+        "ralouphie/mimey": "^1.0"
     },
     "require-dev": {
         "orchestra/testbench": "^8.0|^9.0",

config/mediable.php

@@ -61,6 +61,12 @@
      */
     'allow_unrecognized_types' => false,
 
+    /**
+     * Prefer the client-provided MIME type over the one inferred from the file contents, if provided
+     * May be slightly faster to compute, but is not guaranteed to be accurate if the source is untrusted
+     */
+    'prefer_client_mime_type' => false,
+
     /*
      * Only allow files with specific MIME type(s) to be uploaded
      */
@@ -214,7 +220,7 @@
             '^https?://' => Plank\Mediable\SourceAdapters\RemoteUrlAdapter::class,
             '^/' => Plank\Mediable\SourceAdapters\LocalPathAdapter::class,
             '^[a-zA-Z]:\\\\' => Plank\Mediable\SourceAdapters\LocalPathAdapter::class,
-            '^data:[^,]*,' => Plank\Mediable\SourceAdapters\DataUrlAdapter::class,
+            '^data:/?/?[^,]*,' => Plank\Mediable\SourceAdapters\DataUrlAdapter::class,
         ],
     ],
 

src/Exceptions/MediaUpload/ConfigurationException.php

@@ -37,4 +37,9 @@ public static function diskNotFound(string $disk): self
     {
         return new static("Cannot find disk named `{$disk}`.");
     }
+
+    public static function cannotInferFilename(): self
+    {
+        return new static('No filename is provided and cannot infer filename from the provided source.');
+    }
 }

src/MediaUploader.php

@@ -6,14 +6,14 @@
 use Illuminate\Contracts\Filesystem\Filesystem;
 use Illuminate\Filesystem\FilesystemManager;
 use League\Flysystem\UnableToRetrieveMetadata;
+use Mimey\MimeTypes;
 use Plank\Mediable\Exceptions\MediaUpload\ConfigurationException;
 use Plank\Mediable\Exceptions\MediaUpload\FileExistsException;
 use Plank\Mediable\Exceptions\MediaUpload\FileNotFoundException;
 use Plank\Mediable\Exceptions\MediaUpload\FileNotSupportedException;
 use Plank\Mediable\Exceptions\MediaUpload\FileSizeException;
 use Plank\Mediable\Exceptions\MediaUpload\ForbiddenException;
 use Plank\Mediable\Helpers\File;
-use Plank\Mediable\SourceAdapters\DataUrlAdapter;
 use Plank\Mediable\SourceAdapters\RawContentAdapter;
 use Plank\Mediable\SourceAdapters\SourceAdapterFactory;
 use Plank\Mediable\SourceAdapters\SourceAdapterInterface;
@@ -72,16 +72,23 @@ class MediaUploader
      */
     private array $options = [];
 
+    private MimeTypes $mimey;
+
     /**
      * Constructor.
      * @param FilesystemManager $filesystem
      * @param SourceAdapterFactory $factory
      * @param array|null $config
      */
-    public function __construct(FileSystemManager $filesystem, SourceAdapterFactory $factory, array $config = null)
-    {
+    public function __construct(
+        FileSystemManager $filesystem,
+        SourceAdapterFactory $factory,
+        MimeTypes $mimey,
+        array $config = null
+    ) {
         $this->filesystem = $filesystem;
         $this->factory = $factory;
+        $this->mimey = $mimey;
         $this->config = $config ?: config('mediable', []);
     }
 
@@ -93,7 +100,7 @@ public function __construct(FileSystemManager $filesystem, SourceAdapterFactory
      * @return $this
      * @throws ConfigurationException
      */
-    public function fromSource($source): self
+    public function fromSource(mixed $source): self
     {
         $this->source = $this->factory->create($source);
 
@@ -215,7 +222,7 @@ public function setModelClass(string $class): self
      */
     public function setMaximumSize(int $size): self
     {
-        $this->config['max_size'] = (int)$size;
+        $this->config['max_size'] = $size;
 
         return $this;
     }
@@ -227,7 +234,7 @@ public function setMaximumSize(int $size): self
      */
     public function setOnDuplicateBehavior(string $behavior): self
     {
-        $this->config['on_duplicate'] = (string)$behavior;
+        $this->config['on_duplicate'] = $behavior;
 
         return $this;
     }
@@ -360,7 +367,7 @@ public function setAllowedMimeTypes(array $allowedMimes): self
      */
     public function preferClientMimeType(): self
     {
-        $this->config['mime_type_source'] = 'client';
+        $this->config['prefer_client_mime_type'] = true;
 
         return $this;
     }
@@ -372,7 +379,7 @@ public function preferClientMimeType(): self
      */
     public function preferInferredMimeType(): self
     {
-        $this->config['mime_type_source'] = 'inferred';
+        $this->config['prefer_client_mime_type'] = false;
 
         return $this;
     }
@@ -614,12 +621,12 @@ public function replace(Media $media): Media
      */
     private function populateModel(Media $model): Media
     {
-        $model->size = $this->verifyFileSize($this->source->size());
-        $model->mime_type = $this->verifyMimeType($this->selectMimeType(
-            $this->source->mimeType(),
-            $this->source->clientMimeType()
-        ));
-        $model->extension = $this->verifyExtension($this->source->extension());
+        $model->size = $this->verifyFileSize($this->source->size() ?? 0);
+        $model->mime_type = $this->verifyMimeType($this->selectMimeType());
+        $model->extension = $this->verifyExtension(
+            $this->source->extension()
+                ?? $this->inferExtensionFromMime($model->mime_type)
+        );
         $model->aggregate_type = $this->inferAggregateType($model->mime_type, $model->extension);
 
         $model->disk = $this->disk ?: $this->config['default_disk'];
@@ -688,7 +695,7 @@ public function import(string $disk, string $directory, string $filename, string
         $model->filename = $filename;
         $model->extension = $this->verifyExtension($extension, false);
 
-        if (!$storage->has($model->getDiskPath())) {
+        if (!$storage->exists($model->getDiskPath())) {
             throw FileNotFoundException::fileNotFound($model->getDiskPath());
         }
 
@@ -747,12 +754,9 @@ public function update(Media $media):  bool
     public function verifyFile(): void
     {
         $this->verifySource();
-        $this->verifyFileSize($this->source->size());
+        $this->verifyFileSize($this->source->size() ?? 0);
         $this->verifyMimeType(
-            $this->selectMimeType(
-                $this->source->mimeType(),
-                $this->source->clientMimeType()
-            )
+            $this->selectMimeType()
         );
         $this->verifyExtension($this->source->extension());
     }
@@ -800,7 +804,7 @@ private function verifySource(): void
             throw ConfigurationException::noSourceProvided();
         }
         if (!$this->source->valid()) {
-            throw FileNotFoundException::fileNotFound($this->source->path());
+            throw FileNotFoundException::fileNotFound($this->source->path() ?? '');
         }
     }
 
@@ -816,13 +820,12 @@ private function inferMimeType(Filesystem $filesystem, string $path): string
         return $mime ?: 'application/octet-stream';
     }
 
-    private function selectMimeType(?string $inferredMimeType, ?string $clientMimeType): string
+    private function selectMimeType(): string
     {
-        $source = $this->config['mime_type_source'] ?? 'inferred';
-        if ($source === 'client') {
-            return $clientMimeType ?? $inferredMimeType ?? 'application/octet-stream';
+        if ($this->config['prefer_client_mime_type'] ?? false) {
+            return $this->source->clientMimeType() ?? $this->source->mimeType();
         }
-        return $inferredMimeType ?? $clientMimeType ?? 'application/octet-stream';
+        return $this->source->mimeType();
     }
 
     /**
@@ -845,6 +848,7 @@ private function verifyMimeType(string $mimeType): string
     /**
      * Ensure that the file's extension is allowed.
      * @param  string $extension
+     * @param  bool $toLower
      * @return string
      * @throws FileNotSupportedException If the file extension is not allowed
      */
@@ -886,7 +890,7 @@ private function verifyDestination(Media $model): void
     {
         $storage = $this->filesystem->disk($model->disk);
 
-        if ($storage->has($model->getDiskPath())) {
+        if ($storage->exists($model->getDiskPath())) {
             $this->handleDuplicate($model);
         }
     }
@@ -932,6 +936,7 @@ private function handleDuplicate(Media $model): Media
     /**
      * Delete the media that previously existed at a destination.
      * @param  Media $model
+     * @param  bool $withVariants
      * @return void
      */
     private function deleteExistingMedia(Media $model, bool $withVariants = false): void
@@ -979,7 +984,7 @@ private function generateUniqueFilename(Media $model): string
             }
             $path = "{$model->directory}/{$filename}.{$model->extension}";
             ++$counter;
-        } while ($storage->has($path));
+        } while ($storage->exists($path));
 
         return $filename;
     }
@@ -995,32 +1000,18 @@ private function generateFilename(): string
         }
 
         if ($this->hashFilename) {
-            return $this->generateHash();
+            return $this->source->hash();
         }
 
-        return File::sanitizeFileName($this->source->filename());
-    }
+        $filename = $this->source->filename();
 
-    /**
-     * Calculate hash of source contents.
-     * @return string
-     */
-    private function generateHash(): string
-    {
-        $ctx = hash_init('md5');
-
-        // We don't need to read the file contents if the source has a path
-        if ($this->source->path()) {
-            hash_update_file($ctx, $this->source->path());
-        } else {
-            hash_update($ctx, $this->source->contents());
+        if ($filename === null) {
+            ConfigurationException::cannotInferFilename();
         }
 
-        return hash_final($ctx);
+        return File::sanitizeFileName($filename);
     }
 
-
-
     private function writeToDisk(Media $model): void
     {
         $this->filesystem->disk($model->disk)
@@ -1039,4 +1030,9 @@ public function getOptions(): array
         }
         return $options;
     }
+
+    private function inferExtensionFromMime(?string $mime_type): string
+    {
+        return $this->mimey->getExtension($mime_type) ?: 'application/octet-stream';
+    }
 }

src/MediableServiceProvider.php

@@ -6,6 +6,7 @@
 use CreateMediableTables;
 use Illuminate\Contracts\Container\Container;
 use Illuminate\Support\ServiceProvider;
+use Mimey\MimeTypes;
 use Plank\Mediable\Commands\ImportMediaCommand;
 use Plank\Mediable\Commands\PruneMediaCommand;
 use Plank\Mediable\Commands\SyncMediaCommand;
@@ -108,6 +109,7 @@ public function registerUploader(): void
             return new MediaUploader(
                 $app['filesystem'],
                 $app['mediable.source.factory'],
+                $app[MimeTypes::class],
                 $app['config']->get('mediable')
             );
         });

src/SourceAdapters/DataUrlAdapter.php

@@ -4,50 +4,22 @@
 namespace Plank\Mediable\SourceAdapters;
 
 
+use GuzzleHttp\Psr7\Utils;
+
 /**
  * Raw content Adapter.
  *
  * Adapts a string representing raw contents.
  */
-class DataUrlAdapter extends RawContentAdapter
+class DataUrlAdapter extends StreamAdapter
 {
-    protected ?string $clientMimeType;
-
-    protected string $dataUrl;
-
     /**
      * Constructor.
      * @param string $source
      */
     public function __construct(string $source)
     {
-        $this->dataUrl = $source;
-        if (preg_match(
-                '/^data:(\w+\/[\.+\-\w]+(?:\w+=[^;]+;)*)?(;base64)?,/',
-                $source,
-                $matches
-            ) === 0
-        ) {
-            throw new \InvalidArgumentException('Invalid Data URL format');
-        }
-        $this->clientMimeType = $matches[1] ?? null;
-        $content = substr($source, strlen($matches[0]));
-        $decodedSource = ($matches[2] ?? '') === ';base64'
-            ? base64_decode($content)
-            : rawurldecode($content);
-        parent::__construct($decodedSource);
-    }
-
-    /**
-     * {@inheritdoc}
-     */
-    public function getSource(): mixed
-    {
-        return $this->dataUrl;
-    }
-
-    public function clientMimeType(): ?string
-    {
-        return $this->clientMimeType;
+        $source = preg_replace('/^data:\/?\/?/', 'data://', $source);
+        parent::__construct(Utils::streamFor(Utils::tryFopen($source, 'rb')));
     }
 }