Duplicating product copies product images couple of hundred times

[qbixx]

When duplicating a product to create a similar product all the required information is copied over. This also includes the product images. However when saving the new (duplicated) product, it takes quite a long time before the product get's saved. This is because the attached images are copied a few hundred times on the server.

The amount of copies is not always the same but ranges from about 200 to 2000 copies. The copies are all in the same folder that the original images resides and will have a format of e.g. [image]_100.jpg.

Preconditions

I've tested this both on Magento 2.1.4 and 2.1.6.
PHP 7.0.18 and MariaDB 10.1.22

Steps to reproduce

1. Create a new product and fill out the required fields. Also add an image
2. Save the product and click on the right side on 'Save & Duplicate'
3. Save the duplicated product
4. Open original product, 'Save & Duplicate' it.

Expected result

The image(s) on the duplicated product should be copied once.

Actual result

The image(s) on the duplicated product are copied multiple times.

[ZAV79]

I am also experiencing this - but to a somewhat different level.
When one of my users duplicates a product it seems that as above each image is duplicated multiple times, but it does each image in the entire catalog, not just the product being duplicated.

This means that duplicating just a couple of products can create 100000s images.
This issue was not realised until a 2TB disk had been eaten up.

We are using Magento 2.1.15, PHP 7.0.18, mysql 5.0.12 , Apache/2.4.6, varnish

Unfortunately because the products are not duplicated often, we don't know when this bug was introduced.

We suspect it is something to do with products that were imported using the migration tool from 1.9 as the error does not seem to happen with products added to the catalog after the data migration

[JasonScottM]

Magento ver. 2.1.5 BUG "multiple image generation on product copy".

this is down to a particularly bad piect of code in vendor\magento\module-catalog\Model\Product\Copier.php that loops infinately is the save fails due to an AlreadyExistsException. unfortunately data issues in the url rewrite table can cause this failure and as Magento devs failed to put in place a decent cleanup process to remove immages on save fail you get an ever increasing set untill php times out.

public function copy(\Magento\Catalog\Model\Product $product)
{
................
do {
$urlKey = $duplicate->getUrlKey();
$urlKey = preg_match('/(.*)-(\d+)$/', $urlKey, $matches)
? $matches[1] . '-' . ($matches[2] + 1)
: $urlKey . '-1';
$duplicate->setUrlKey($urlKey);
try {
$duplicate->save();
$isDuplicateSaved = true;
} catch (\Magento\Framework\Exception\AlreadyExistsException $e) {
}
} while (!$isDuplicateSaved);
...............
}

luckily there is code in vendor\magento\module-catalog-url-rewrite\observer\ProductProcessUrlRewriteSavingObserver.php that is a life saver

public function execute(\Magento\Framework\Event\Observer $observer)
{
.................................
if ($product->isVisibleInSiteVisibility()) {
$this->urlPersist->replace($this->productUrlRewriteGenerator->generate($product));
}
.................................
}

so by inserting

$duplicate->setVisibility(\Magento\Catalog\Model\Product\Visibility::VISIBILITY_NOT_VISIBLE);

into the object being saved the url rewrites will not be generated and the duplicate will save without the multiple image loop (though multiples will still occur if its been copied n times previously but only n copies)

magento devs really need to sort this out, relying on the save excepting rather than first looking up a free rewrite is an error, The code should lookup a free url against the url_key attribute values in catalog_product_entity_varchar first then attempt the save once only and throw the exception if it fails, PLEASE stop putting empty catch staements in loops !!!

my quick fix to correct this error is to insert $duplicate->setVisibility(\Magento\Catalog\Model\Product\Visibility::VISIBILITY_NOT_VISIBLE); in the core copy function which then reads:-

public function copy(\Magento\Catalog\Model\Product $product)
{
$product->getWebsiteIds();
$product->getCategoryIds();

    /** @var \Magento\Catalog\Model\Product $duplicate */
    $duplicate = $this->productFactory->create();
    $duplicate->setData($product->getData());
    $duplicate->setOptions([]);
    $duplicate->setIsDuplicate(true);
    $duplicate->setOriginalId($product->getEntityId());
    $duplicate->setStatus(\Magento\Catalog\Model\Product\Attribute\Source\Status::STATUS_DISABLED);
    $duplicate->setCreatedAt(null);
    $duplicate->setUpdatedAt(null);
    $duplicate->setId(null);
    $duplicate->setStoreId(\Magento\Store\Model\Store::DEFAULT_STORE_ID);
    $duplicate->setVisibility(\Magento\Catalog\Model\Product\Visibility::VISIBILITY_NOT_VISIBLE);    
    $this->copyConstructor->build($product, $duplicate);
    $isDuplicateSaved = false;
    do {
        $urlKey = $duplicate->getUrlKey();
        $urlKey = preg_match('/(.*)-(\d+)$/', $urlKey, $matches)
            ? $matches[1] . '-' . ($matches[2] + 1)
            : $urlKey . '-1';
        $duplicate->setUrlKey($urlKey);
        try {
            $duplicate->save();
            $isDuplicateSaved = true;
        } catch (\Magento\Framework\Exception\AlreadyExistsException $e) {
        }
    } while (!$isDuplicateSaved);
    $this->getOptionRepository()->duplicate($product, $duplicate);
    $metadata = $this->getMetadataPool()->getMetadata(ProductInterface::class);
    $product->getResource()->duplicate(
        $product->getData($metadata->getLinkField()),
        $duplicate->getData($metadata->getLinkField())
    );
    return $duplicate;
}

you will also note from the origional Magento code that if your URL keys / Sku convention is latter half numeric with a - in to ie AAAAA-00001 then prepare for a long wait on product copies or change the - in the logic to an _ if you dont use _ in your standard url keys.

[pascaladriaansen]

I think this is the same issue as #4096.

[veloraven]

Closed as a duplicate of #4096

[qbixx]

As far as I can see this isn't a duplicate of #4096.

When looking at the behaviour of #4096 there are some different things going on than in this issue.

When duplicating a product on this issue the page doesn't go in a loop to reload the product page. Also, with this issue the website doesn't become unavailable. The website itself is still working, only the disk space of the server is filling up with duplicate product images.

[jonahellison]

Confirmed this is still an issue on 2.1.8 for migrated products. The "setVisibility" fix from JasonScottM did work.

[vforvalerio87]

Is there any way to safely and reliably delete all duplicates for all products? Thanks

[JasonScottM]

If you are asking is there a way to delete all unused duplicate images? then yes, attached is a script I put together for that task, if your database has a prefix set you may need to change the table name. run the script and if you are happy the red results shown are the ones to delete, uncomment the 2 unlink lines are re-run. To be sure you can recover, I suggest backing up the pub/media/catalog Folder first. From: Valerio Versace [mailto:notifications@github.com] Sent: 14 September 2017 17:45 To: magento/magento2 <magento2@noreply.github.com> Cc: JasonScottM <jason.mummery@micommerce.com>; Comment <comment@noreply.github.com> Subject: Re: [magento/magento2] Duplicating product copies product images couple of hundred times (#9466) Is there any way to safely and reliably delete all duplicates for all products? Thanks — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#9466 (comment)> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AOY7R-jzSKtkE6I3vx29lKr-IVYmZcC3ks5siVgVgaJpZM4NNQ8J> . <https://github.com/notifications/beacon/AOY7R2kofUTbkZwpqO_eqlJn38NFrGXmks5siVgVgaJpZM4NNQ8J.gif>

[hostep]

God damned, we just ran into this today on a Magento 2.1.9 installation. 12 GB of diskspace eaten away in a 40 minutes by duplicating some products until our server crashed with no space left.

Would be great if this got fixed!

[hostep]

@magento-engcom-team: can this ticket get re-opened? It has the label G1 Passed while the similar looking issue #4096 has G1 Failed.
Thanks!

[hostep]

So the exact steps to reproduce are outlined below, because following the steps from @qbixx on a clean installation isn't going to trigger the problem:

Steps to reproduce

1. Set up a clean Magento 2.1.9 installation
2. Create a simple product, assign it an image 'pic.jpg' and save
3. Look at the image directory, it contains 1 image:

$ ls -1 pub/media/catalog/product/p/i/
pic.jpg

4. Save and duplicate the product
5. Look at the image directory, it contains 2 images (so far so good):

$ ls -1 pub/media/catalog/product/p/i/
pic.jpg
pic_1.jpg

6. Open the first product you created and choose to save and duplicate it again
7. Look at the image directory, it contains 4 images (uh oh, something's not right here):

$ ls -1 pub/media/catalog/product/p/i/
pic.jpg
pic_1.jpg
pic_2.jpg
pic_3.jpg

8. Open the first product you created and choose to save and duplicate it again
9. Look at the image directory, it contains 7 images (this is starting to get out of control):

$ ls -1 pub/media/catalog/product/p/i/
pic.jpg
pic_1.jpg
pic_2.jpg
pic_3.jpg
pic_4.jpg
pic_5.jpg
pic_6.jpg

...

Expected result

After every duplication, only one new image file is being created

Further discoveries

It no longer seems to be reproducible in Magento 2.2.0
This has part to do with the fact that if a duplicate url key is encountered, it now throws a different exception, instead of \Magento\Framework\Exception\AlreadyExistsException the exception Magento\UrlRewrite\Model\Exception\UrlAlreadyExistsException is being thrown, and this one isn't being catched in https://github.com/magento/magento2/blob/2.2.0/app/code/Magento/Catalog/Model/Product/Copier.php#L85
So it jumps out of that loop.
Interestingly, if I change the exception to be catched in 2.2.0, the problem still no longer occurs, so something else must have changed next to that different exception being thrown. If somebody knows what this could have been, that might be the real fix...

And I agree with @JasonScottM:

magento devs really need to sort this out, relying on the save excepting rather than first looking up a free rewrite is an error, The code should lookup a free url against the url_key attribute values in catalog_product_entity_varchar first then attempt the save once only and throw the exception if it fails, PLEASE stop putting empty catch staements in loops !!!

Trying to save a model and try to save it multiple times in a loop until an exception no longer is being thrown is a really bad idea.

[rcdlopez]

@JasonScottM - where is the script????

[JasonScottM]

Hi Robert I think that was stripped in my previous response so please use this link http://www.keepandshare.com/doc18/view.php?id=16316 <http://www.keepandshare.com/doc18/view.php?id=16316&da=y> &da=y Jason.

[rcdlopez]

@JasonScottM - the link you provided requires login.

[JasonScottM]

The link should work now.

[wget]

For other devs passing by, on the other related issue, a Magento CLI module has been developed in order to remove duplicated product media.
src.: #4096 (comment)
Direct link: https://github.com/ThomasNegeli/M2CliTools

[hostep]

Re-opening after a report from @VirtusB who mentions this bug possibly re-appeared in Magento 2.2.4

Feel free to close when I'm mistaken.

[mattyl]

we get this exception Magento\UrlRewrite\Model\Exception\UrlAlreadyExistsException
but it continues to create 10,000s of images in 2.2.4 multi store with migrated products (original comment in #4096

[anilbajracharya]

http://www.autotechbook.com/magento-2-2-4-duplicate-product-image-issue/
try this script basically what it does is remove unused duplicate image from media/catalogue/product

[hostep]

Hi @engcom-backlog-nazar

I'm re-opening this issue, since it hasn't been fixed in 2.3-develop branch yet (tested on commit 1a805d0)

Although the worst issue is probably fixed by 50592b4, I can still reproduce this issue with the steps from my earlier report: #9466 (comment)

This is still caused by that do while loop which tries to find a unique url key for a duplicated product by trying to save that product on and on until it finally finds a unique url key.

You can very easily see this happening by adding something like this and monitoring the system.log file:

diff --git a/app/code/Magento/Catalog/Model/Product/Copier.php b/app/code/Magento/Catalog/Model/Product/Copier.php
index ce6b4d98bbc..357b5e510fb 100644
--- a/app/code/Magento/Catalog/Model/Product/Copier.php
+++ b/app/code/Magento/Catalog/Model/Product/Copier.php
@@ -82,6 +82,9 @@ class Copier
             $urlKey = preg_match('/(.*)-(\d+)$/', $urlKey, $matches)
                 ? $matches[1] . '-' . ($matches[2] + 1)
                 : $urlKey . '-1';
+
+            \Magento\Framework\App\ObjectManager::getInstance()->get(\Psr\Log\LoggerInterface::class)->critical($urlKey);
+
             $duplicate->setUrlKey($urlKey);
             $duplicate->setData(Product::URL_PATH, null);
             try {

In my case, I had a product with the url key 'simple'. After duplicating that product 4 times, and then trying to duplicate it a few times more, this was the output:

[2018-11-21 20:39:55] main.CRITICAL: simple-1 [] []
[2018-11-21 20:39:55] main.CRITICAL: simple-2 [] []
[2018-11-21 20:39:55] main.CRITICAL: simple-3 [] []
[2018-11-21 20:39:56] main.CRITICAL: simple-4 [] []

[2018-11-21 20:40:29] main.CRITICAL: simple-1 [] []
[2018-11-21 20:40:29] main.CRITICAL: simple-2 [] []
[2018-11-21 20:40:29] main.CRITICAL: simple-3 [] []
[2018-11-21 20:40:29] main.CRITICAL: simple-4 [] []
[2018-11-21 20:40:29] main.CRITICAL: simple-5 [] []

[2018-11-21 20:46:28] main.CRITICAL: simple-1 [] []
[2018-11-21 20:46:28] main.CRITICAL: simple-2 [] []
[2018-11-21 20:46:29] main.CRITICAL: simple-3 [] []
[2018-11-21 20:46:29] main.CRITICAL: simple-4 [] []
[2018-11-21 20:46:29] main.CRITICAL: simple-5 [] []
[2018-11-21 20:46:29] main.CRITICAL: simple-6 [] []

And a whole bunch of extra images got created on the filesystem, even though I only needed 1 extra image per time I duplicated the product.

[ghost]

Hi @hostep thanks for clear description of the issue, this issue still present on 2.2 and 2.3 branches.

[tawfekov]

I was affected by this issue , the team depends on duplicate action , they filled up a disk of 440G completely with images

cleaning up the extra images was very hard to do , the only way I've found is : I had to remember the items they duplicated and delete their images from disk

you may need to delete them from catalog_product_entity_media_gallery table as well

I'm sharing my solution here to help the others , hopefully I could save someone's time

[wget]

@tawfekov cleaning up the extra images was very hard to do. However, several solutions (and tools) have been described above to solve this easily. Reading this whole thread could have you saved a lot of time :)

[tawfekov]

@wget these won't work out for me since I had custom module generating custom images that aren't saved in media tables

Thanks for willing to help 😀

P.s n98 has a similar addon that can clean it up too https://github.com/magento-hackathon/EAVCleaner#eav-cleaner-magerun-addon

[wget]

@tawfekov Didn't know this was also impacting custom modules; I only though magento- core modules were impacted. Thanks for the clarification. :)

[chickenland]

Just ran into this. Is 50592b4 the best solution, or should it actually regenerate the url_path alongside the url_key? I was thinking load in Magento\CatalogUrlRewrite\Model\ProductUrlPathGenerator and set the URLPath alongside the URLKey using getUrlPathWithSuffix($duplicate)

[hostep]

@chickenland: The url_path attribute for products is always empty in M2, the attribute is a left over from the M1 days. If somehow it is filled in (by migrating from M1 to M2 for example) then it should be fine to delete the values (which can only be done directly in the database), but test it first before doing this on a production environment.

[SolsWebdesign]

Ran into this on M2.2.6, @hostep is right: it happens with migrated M1 products that have url_path filled. I did a query for a test product:

SELECT * FROM catalog_product_entity_varchar WHERE attribute_id IN ( SELECT attribute_id FROM eav_attribute WHERE attribute_code IN ('url_path') AND entity_type_id = 4 ) AND entity_id = 8

where my product id is 8 and found that it was filled. Emptied the url_path for this product and it duplicates fine without the hundreds of copied images as before. So time for a simple plugin that empties the url_path before duplicate is called. Thank you @hostep for pointing me in the right direction.

[SolsWebdesign]

For those that need a solution for products migrated from M1 that cannot be duplicated in M2 due to this bug, make your own module with an etc\di.xml and add these lines:

<type name="Magento\Catalog\Model\Product\Copier">
        <plugin name="catalog_product_duplicate"
                type="Vendor\YourModuleName\Plugin\Copier"/>
 </type>

Create a Plugin folder and add the file with the following content:

namespace Vendor\YourModuleName\Plugin;

use Magento\Catalog\Model\Product;

class Copier
{
    public function beforeCopy($subject, \Magento\Catalog\Model\Product $product)
    {
        $product->setData('url_path', '');
        $product->save();
    }
}

Clear cache and di:compile and test.

[JMLucas96]

I followed the code when I save a product and it executes moveImageFromTmp($file) function in vendor\magento\module-catalog\Model\Product\Gallery\CreateHandler.php.

That function have a method called getUniqueFileName($file) and is that function that duplicates image and adds _X.jpg.

Simply replacing this line $destinationFile = $this->getUniqueFileName($file); for this $destinationFile = $file; we prevent duplicating images on Magento.

See completely function code here:
vendor\magento\module-catalog\Model\Product\Gallery\CreateHandler.php:

/**
     * Move image from temporary directory to normal
     *
     * @param string $file
     * @return string
     */
    protected function moveImageFromTmp($file)
    {
        $file = $this->getFilenameFromTmp($file);
        $destinationFile = $file;
        //$destinationFile = $this->getUniqueFileName($file); original code replaced by before line

        if ($this->fileStorageDb->checkDbUsage()) {
            $this->fileStorageDb->renameFile(
                $this->mediaConfig->getTmpMediaShortUrl($file),
                $this->mediaConfig->getMediaShortUrl($destinationFile)
            );

            $this->mediaDirectory->delete($this->mediaConfig->getTmpMediaPath($file));
            $this->mediaDirectory->delete($this->mediaConfig->getMediaPath($destinationFile));
        } else {
            $this->mediaDirectory->renameFile(
                $this->mediaConfig->getTmpMediaPath($file),
                $this->mediaConfig->getMediaPath($destinationFile)
            );
        }

        return str_replace('\\', '/', $destinationFile);
    }

Function that adds _XX.jpg to images and duplicating them:
vendor\magento\framework\File\Uploader.php

/**
     * Get new file name if the same is already exists
     *
     * @param string $destinationFile
     * @return string
     */
    public static function getNewFileName($destinationFile)
    {
        $fileInfo = pathinfo($destinationFile);
        if (file_exists($destinationFile)) {
            $index = 1;
            $baseName = $fileInfo['filename'] . '.' . $fileInfo['extension'];
            while (file_exists($fileInfo['dirname'] . '/' . $baseName)) {
                $baseName = $fileInfo['filename'] . '_' . $index . '.' . $fileInfo['extension'];
                $index++;
            }
            $destFileName = $baseName;
        } else {
            return $fileInfo['basename'];
        }

        return $destFileName;
    }

It works for me on Magento 2.1.8 💃 !

[michel334]

So after 2 years this is still an issue. A client of mine just got wrecked by this. What a joke.

[slavvka]

Hi @qbixx. Thank you for your report.
The issue has been fixed in #25875 by @JeroenVanLeusden in 2.4-develop branch
Related commit(s):

• 8e2959de1740f78ba9a57aa0e21e23ecca79dcc6
• a027e87a3d0393bc0639033e2e0ecf14d51f44a9
• 64cb774f748d375f72e2e0052261d907f6b23c87

The fix will be available with the upcoming 2.4.0 release.

[PsmIndia]

i think this issue still there, we are move magento 1 to magento 2.3.4, we i am try to duplicate, its take too much time, finally request crashed but when i login back duplicate product created , log error see below..

[2020-06-13 13:12:11] main.CRITICAL: We couldn't copy file catalog/product/b/a/bana-02_1.jpg. Please delete media with non-existing images and try again. {"exception":"[object] (Magento\Framework\Exception\LocalizedException(code: 0): We couldn't copy file catalog/product/b/a/bana-02_1.jpg. Please delete media with non-existing images and try again. at /data/web/magento2_staging/vendor/magento/module-catalog/Model/Product/Gallery/CreateHandler.php:473)"} []

[PsmIndia]

Need Urgent Help, magento setup version is 2.3.4[upgraded m1], we have "29542 records found" products, i am facing duplicate product issue, try to fix using available patch but its not fix.

Today i remove all products images in test environment, there no image for a single product, now i add product image for a single product, i save it, working cool, but when i try to do duplicate this product, its again went into infinite loop, within few mints it create huge[4.3GB] garbage of that image that recently add. now i am helpless.

[BB-000]

🙀🤯!

Can anyone confirm if https://github.com/ThomasNegeli/M2CliTools works on magneto 2.3x?