HOTFIX: #26607 Fix failing CI due to Functional Tests #26608

lbajsarowicz · 2020-01-31T10:33:38Z

Description (*)

During fix of #26607 with #26608 - I've noticed that the test is extended and actually - the Admin session is killed, completely ignoring the fact, that further actions that are performed in original tests still need the session.

Reorganized tests and it's extensions to work properly.
Removed the cache:flush bad practice loved by Core Team <3
Fixed issues caused by missing indexing (adding cron:run --group=index)
Invalid search returns returned (duplicated products)
Extended Credit Card expiration period to another 4 years

All above mentioned also for B2B and EE version.

Related Pull Requests

https://github.com/magento/partners-magento2ee/pull/135
https://github.com/magento/partners-magento2b2b/pull/45

Fixed Issues (if relevant)

MFTF: AdminReorderWithCatalogPriceTest is failing in CI process #26607

Manual testing scenarios (*)

Change README.md for any module
Submit PR to Github

Questions or comments

🤦‍♂️

Contribution checklist (*)

Pull request has a meaningful description of its purpose
All commits are accompanied by meaningful commit messages
All new or changed code is covered with unit/integration tests (if applicable)
All automated tests passed successfully (all builds are green)

m2-assistant · 2020-01-31T10:33:41Z

Hi @lbajsarowicz. Thank you for your contribution
Here is some useful tips how you can test your changes using Magento test environment.
Add the comment under your pull request to deploy test or vanilla Magento instance:

@magento give me test instance - deploy test instance based on PR changes
@magento give me 2.4-develop instance - deploy vanilla Magento instance

For more details, please, review the Magento Contributor Guide documentation.

… overlapping Cron executions

…`sku` as URL Key :-(

app/code/Magento/Bundle/Test/Mftf/Test/StorefrontCustomerSelectAndSetBundleOptionsTest.xml

lbajsarowicz · 2020-02-02T14:44:03Z

@magento run all tests

1 fixed = 1 new to be fixed 🤦‍♂️
And all the issues are similar and are result of invalid test isolation

lbajsarowicz · 2020-02-02T15:47:01Z

@magento run Functional Tests CE

Reason:

Nazar65 · 2020-02-02T16:05:54Z

@magento run Functional Tests CE

Reason:

Yes i had exact same issue when i was trying to fix failed b2b on my PR

lbajsarowicz · 2020-02-02T16:07:45Z

Working on it - the structure of the tests related to searching products are broken even if it fails

lbajsarowicz · 2020-02-02T19:05:07Z

@magento run all tests

I've removed sample products that are not used for testing, as it breaks all Elasticsuite tests 🤦‍♂️

lbajsarowicz · 2020-02-02T21:58:53Z

The issue of Tests Isolation is the issue, still.
The easiest way to avoid issues with searching by short description is to remove spaces from ShortDescription as it makes search catch other products.

There should be confirmation when accidentally click "Close" button 🤦‍♂️

magento-engcom-team · 2020-02-13T21:04:39Z

Hi @lenaorobei, thank you for the review.
ENGCOM-6825 has been created to process this Pull Request

lbajsarowicz · 2020-02-16T17:28:50Z

@magento run all tests

…i-failure # Conflicts: # app/code/Magento/Sales/Test/Mftf/Test/AdminReorderWithCatalogPriceRuleDiscountTest.xml

…nto hotfix/26607-ci-failure

lbajsarowicz · 2020-02-21T20:43:08Z

@magento run all tests
🙏 🙏 🙏

…6608

m2-assistant · 2020-02-22T04:26:58Z

Hi @lbajsarowicz, thank you for your contribution!
Please, complete Contribution Survey, it will take less than a minute.
Your feedback will help us to improve contribution process.

zakdma · 2020-02-24T11:46:27Z

@lbajsarowicz it is bad practice to hide issues adding reindexing everywhere even it is not necessary. You claim core team but do actually the same.
Also there is special AG for this purposes
https://github.com/magento/magento2/blob/2.4-develop/app/code/Magento/Indexer/Test/Mftf/ActionGroup/CliRunReindexUsingCronJobsActionGroup.xml

lbajsarowicz · 2020-02-24T11:49:54Z

@zakdma

I am not running reindexes
I am running Cron Jobs - which is exactly the same thing that happens on Production Environments

Cron Jobs from index group are responsible for updating single entities indexes based on _cl tables that are created once you configure reindex to run on schedule instead of realtime. Every single time when you create or modify products, you have to give the indexers the opportunity to update the indexes. Magento Core way is to run indexer:reindex and cache:flush which is a very expensive way to get the same work done.

I'd love to know your point of view.

lbajsarowicz · 2020-02-24T11:57:34Z

I don't understand @zakdma why - in your opinion - running Cron Jobs is a bad practice?

The Action Group you provided is not working as expected, because if you run cron:run twice, and it will be executed in the same minute - second one is going to be skipped.

zakdma · 2020-02-24T11:58:47Z

No, you are not right. First run schedule jobs and second one run scheduled jobs. Even in the same minute

zakdma · 2020-02-24T12:04:38Z

cron:run --group=index actually just do reindex of broken indices. If indices are not broken this is not necessary to run it. Indices are broken after, for instance, creating product attribute. That is expected behaviour and we need to reindex them after that. If previous test that create attribute ran before test that depends on indices it will fail but issue not failed test but test before it.

lbajsarowicz · 2020-02-24T12:07:04Z

There is a simple way to prove your ActionGroup does not work as expected:

Run:

watch -n5 "bin/magento cron:run --group=index -vvv"

And then watch the logs after several minutes:

[2020-02-24 12:02:38] report.INFO: Cron Job indexer_reindex_all_invalid is run [] []
[2020-02-24 12:02:39] report.INFO: Cron Job indexer_reindex_all_invalid is successfully finished. Statistics: {"sum":0.32009887695312,"count":1,"realmem":14680064,"emalloc":16307456,"realmem_start":48758784,"emalloc_start":45042280} [] []
[2020-02-24 12:02:39] report.INFO: Cron Job indexer_update_all_views is run [] []
[2020-02-24 12:02:39] report.INFO: Cron Job indexer_update_all_views is successfully finished. Statistics: {"sum":0.40363311767578,"count":1,"realmem":10485760,"emalloc":6242624,"realmem_start":63438848,"emalloc_start":61352328} [] []
[2020-02-24 12:03:06] report.INFO: Cron Job indexer_reindex_all_invalid is run [] []
[2020-02-24 12:03:06] report.INFO: Cron Job indexer_reindex_all_invalid is successfully finished. Statistics: {"sum":0.015038967132568,"count":1,"realmem":0,"emalloc":905128,"realmem_start":46661632,"emalloc_start":44110200} [] []
[2020-02-24 12:03:06] report.INFO: Cron Job indexer_update_all_views is run [] []
[2020-02-24 12:03:06] report.INFO: Cron Job indexer_update_all_views is successfully finished. Statistics: {"sum":0.1584370136261,"count":1,"realmem":6291456,"emalloc":6716520,"realmem_start":46661632,"emalloc_start":45017920} [] []
[2020-02-24 12:04:00] report.INFO: Cron Job indexer_reindex_all_invalid is run [] []
[2020-02-24 12:04:00] report.INFO: Cron Job indexer_reindex_all_invalid is successfully finished. Statistics: {"sum":0.014179229736328,"count":1,"realmem":0,"emalloc":905128,"realmem_start":46661632,"emalloc_start":44110200} [] []
[2020-02-24 12:04:00] report.INFO: Cron Job indexer_update_all_views is run [] []
[2020-02-24 12:04:00] report.INFO: Cron Job indexer_update_all_views is successfully finished. Statistics: {"sum":0.17619705200195,"count":1,"realmem":6291456,"emalloc":6716520,"realmem_start":46661632,"emalloc_start":45017920} [] []
[2020-02-24 12:05:05] report.INFO: Cron Job indexer_reindex_all_invalid is run [] []
[2020-02-24 12:05:05] report.INFO: Cron Job indexer_reindex_all_invalid is successfully finished. Statistics: {"sum":0.012212991714478,"count":1,"realmem":0,"emalloc":905128,"realmem_start":46661632,"emalloc_start":44110200} [] []
[2020-02-24 12:05:05] report.INFO: Cron Job indexer_update_all_views is run [] []
[2020-02-24 12:05:05] report.INFO: Cron Job indexer_update_all_views is successfully finished. Statistics: {"sum":0.14868688583374,"count":1,"realmem":6291456,"emalloc":6716520,"realmem_start":46661632,"emalloc_start":45017920} [] []
[2020-02-24 12:06:04] report.INFO: Cron Job indexer_reindex_all_invalid is run [] []
[2020-02-24 12:06:04] report.INFO: Cron Job indexer_reindex_all_invalid is successfully finished. Statistics: {"sum":0.016084909439087,"count":1,"realmem":0,"emalloc":905128,"realmem_start":46661632,"emalloc_start":44110200} [] []
[2020-02-24 12:06:04] report.INFO: Cron Job indexer_update_all_views is run [] []
[2020-02-24 12:06:04] report.INFO: Cron Job indexer_update_all_views is successfully finished. Statistics: {"sum":0.17505884170532,"count":1,"realmem":6291456,"emalloc":6716520,"realmem_start":46661632,"emalloc_start":45017920} [] []

I'd love to learn something new - so let me understand your point of view.

lbajsarowicz · 2020-02-24T12:12:45Z

@zakdma This is actually sad, that you are throwing stones in my direction without even checking the triggers that are declared in mview.xml.

As an example

magento2/app/code/Magento/Catalog/etc/mview.xml

Lines 54 to 60 in 9fe3b8a

    
           <view id="catalog_product_attribute" class="Magento\Catalog\Model\Indexer\Product\Eav" group="indexer"> 
        
               <subscriptions> 
        
                   <table name="catalog_product_entity_decimal" entity_column="entity_id" /> 
        
                   <table name="catalog_product_entity_int" entity_column="entity_id" /> 
        
                   <table name="catalog_product_entity_varchar" entity_column="entity_id" /> 
        
               </subscriptions> 
        
           </view>

Creates trigger on MySQL:

create or replace definer = magento@`%` trigger trg_catalog_product_entity_decimal_after_delete
	after delete
	on catalog_product_entity_decimal
	for each row
	BEGIN
SET @entity_id = (SELECT `entity_id` FROM `catalog_product_entity` WHERE `row_id` = OLD.`row_id`);
INSERT IGNORE INTO `catalogsearch_fulltext_cl` (`entity_id`) values(@entity_id);
INSERT IGNORE INTO `catalog_product_attribute_cl` (`entity_id`) values(@entity_id);
INSERT IGNORE INTO `catalogrule_product_cl` (`entity_id`) values(@entity_id);
INSERT IGNORE INTO `catalog_product_price_cl` (`entity_id`) values(@entity_id);
END;

create or replace definer = magento@`%` trigger trg_catalog_product_entity_decimal_after_insert
	after insert
	on catalog_product_entity_decimal
	for each row
	BEGIN
SET @entity_id = (SELECT `entity_id` FROM `catalog_product_entity` WHERE `row_id` = NEW.`row_id`);
INSERT IGNORE INTO `catalogsearch_fulltext_cl` (`entity_id`) values(@entity_id);
INSERT IGNORE INTO `catalog_product_attribute_cl` (`entity_id`) values(@entity_id);
INSERT IGNORE INTO `catalogrule_product_cl` (`entity_id`) values(@entity_id);
INSERT IGNORE INTO `catalog_product_price_cl` (`entity_id`) values(@entity_id);
END;

create or replace definer = magento@`%` trigger trg_catalog_product_entity_decimal_after_update
	after update
	on catalog_product_entity_decimal
	for each row
	BEGIN
SET @entity_id = (SELECT `entity_id` FROM `catalog_product_entity` WHERE `row_id` = NEW.`row_id`);
IF (NOT(NEW.`value_id` <=> OLD.`value_id`) OR NOT(NEW.`attribute_id` <=> OLD.`attribute_id`) OR NOT(NEW.`store_id` <=> OLD.`store_id`) OR NOT(NEW.`value` <=> OLD.`value`) OR NOT(NEW.`row_id` <=> OLD.`row_id`)) THEN INSERT IGNORE INTO `catalogsearch_fulltext_cl` (`entity_id`) values(@entity_id); END IF;
IF (NOT(NEW.`value_id` <=> OLD.`value_id`) OR NOT(NEW.`attribute_id` <=> OLD.`attribute_id`) OR NOT(NEW.`store_id` <=> OLD.`store_id`) OR NOT(NEW.`value` <=> OLD.`value`) OR NOT(NEW.`row_id` <=> OLD.`row_id`)) THEN INSERT IGNORE INTO `catalog_product_attribute_cl` (`entity_id`) values(@entity_id); END IF;
IF (NOT(NEW.`value_id` <=> OLD.`value_id`) OR NOT(NEW.`attribute_id` <=> OLD.`attribute_id`) OR NOT(NEW.`store_id` <=> OLD.`store_id`) OR NOT(NEW.`value` <=> OLD.`value`) OR NOT(NEW.`row_id` <=> OLD.`row_id`)) THEN INSERT IGNORE INTO `catalogrule_product_cl` (`entity_id`) values(@entity_id); END IF;
IF (NOT(NEW.`value_id` <=> OLD.`value_id`) OR NOT(NEW.`attribute_id` <=> OLD.`attribute_id`) OR NOT(NEW.`store_id` <=> OLD.`store_id`) OR NOT(NEW.`value` <=> OLD.`value`) OR NOT(NEW.`row_id` <=> OLD.`row_id`)) THEN INSERT IGNORE INTO `catalog_product_price_cl` (`entity_id`) values(@entity_id); END IF;
END;

As a result, the entity_id is being saved to catalogsearch_fulltext_cl (and 3 others).

The indexer is looking for changes in these tables to update indexes for single entities.

magento2/lib/internal/Magento/Framework/Mview/View/Changelog.php

Lines 145 to 166 in 31aebbf

    
           public function getList($fromVersionId, $toVersionId) 
        
           { 
        
               $changelogTableName = $this->resource->getTableName($this->getName()); 
        
               if (!$this->connection->isTableExists($changelogTableName)) { 
        
                   throw new ChangelogTableNotExistsException(new Phrase("Table %1 does not exist", [$changelogTableName])); 
        
               } 
        
               $select = $this->connection->select()->distinct( 
        
                   true 
        
               )->from( 
        
                   $changelogTableName, 
        
                   [$this->getColumnName()] 
        
               )->where( 
        
                   'version_id > ?', 
        
                   (int)$fromVersionId 
        
               )->where( 
        
                   'version_id <= ?', 
        
                   (int)$toVersionId 
        
               ); 
        
               return array_map('intval', $this->connection->fetchCol($select)); 
        
           }

zakdma · 2020-02-24T12:25:57Z

I think we don't understand each other.

MFTF Tests run on environment with cron disabled
First manual cron:run (on an clear database) will not run jobs but just schedule jobs
Second manual call cron:run will actually run jobs even if it run in the same minute with first call
--group=index will run job indexer_reindex_all_invalid that not just index changes (that created in tables) but totally reindex all broken indices (invalidated indices)
MFTF Tests environment using Update on Save index strategy, not Update by schedule

lbajsarowicz · 2020-02-24T12:36:10Z

That is why we need to run them manually
I've never had such issue. Previous <magentoCLI command="cron:run --group=index"...> and current <magentoCron groups="index"> works as expected. The only exception to <magentoCLI was the described case of Crons running on the same minute, especially for Functional tests running simultaneously.
It is going to run 3 jobs:

<group id="index">
        <job name="indexer_reindex_all_invalid" instance="Magento\Indexer\Cron\ReindexAllInvalid" method="execute">
            <schedule>* * * * *</schedule>
        </job>
        <job name="indexer_update_all_views" instance="Magento\Indexer\Cron\UpdateMview" method="execute">
            <schedule>* * * * *</schedule>
        </job>
        <job name="indexer_clean_all_changelogs" instance="Magento\Indexer\Cron\ClearChangelog" method="execute">
            <schedule>0 * * * *</schedule>
        </job>
    </group>

indexer_reindex_all_invalid is going to run for invalid indices
indexer_update_all_views is going to run for outdated indices

Still, I have no idea why are you throwing stones in my direction saying that I hide the issues. As you see above: It's necessary to run Cron for indexer_update_all_views every single time when entities are modified and indexers are configured to work on schedule.

zakdma · 2020-02-24T12:45:46Z

It is not issue but normal behaviour
But for MFTF tests make sense only indexer_reindex_all_invalid because MFTF tests env don't use triggers and tables with changes since it works in Update on Save mode

It's necessary to run Cron for indexer_update_all_views every single time when entities are modified and indexers are configured to work on schedule.

Again, MFTF test env works in Update on Save mode for now.

About hiding. It better to know what actually make index invalid. Maybe it is a bug that will be hidden if you reindex invalidated indices every time you change something. For Ex. we found bug with invalidated indices during fixing such "flaky" MFTF test. We could just add reindex and go ahead but we make deep investigation and finally found it. Now we have tons of tests "fixed" with reindex and hidden real cause of unexpected behaviour. That is sad.

lbajsarowicz · 2020-02-24T12:50:25Z

Actually, whether MFTF should run On schedule or On save should be defined at the very beginning of each suite. Let's consider that both way:
a) Reindex On save (realtime): cron:run --group=index does nothing
b) Reindex On schedule: cron:run --group=index runs indexer_update_all_views

That is why after createData, updateData and to be nice - even after deleteData you should run index crons. That is also recommended when you modify entity from Admin Panel.

It better to know what actually make index invalid. Maybe it is a bug that will be hidden if you reindex invalidated indices every time you change something. For Ex. we found bug with invalidated indices during fixing such "flaky" MFTF test. We could just add reindex and go ahead but we make deep investigation and finally found it. Now we have tons of tests "fixed" with reindex and hidden real cause of unexpected behaviour. That is sad.

That is why we have Integration Tests, actually.

zakdma · 2020-02-24T12:59:08Z

That is why we have Integration Tests, actually.

Not all code covered by IT.
It is always better to have more than one approach to test.

lbajsarowicz · 2020-02-24T13:03:32Z

There are no obstacles for you to contribute with your approach.
Ha! You are more than welcome to stabilize the Functional Tests.

zakdma · 2020-02-24T13:06:18Z

Actually I do it almost every single day.

magento#26607 Fix failing CI due to missing "createSimpleProductApi"

e6a3bee

magento-engcom-team added the Partner: Mediotype label Jan 31, 2020

magento-engcom-team added the partners-contribution Pull Request is created by Magento Partner label Jan 31, 2020

magento#26607 Additional changes that make Test more stable

e26a536

ihor-sviziev self-assigned this Jan 31, 2020

lbajsarowicz added 8 commits February 1, 2020 02:07

Trigger cron execution on index group to show products

075f9f4

Trigger cron execution on index group to show products

dab9463

Wait for Customer Section load

fe153fb

Extend Credit Card expiration date for another 4 years

c9d83d7

Fix invalid XML syntax

148011a

Reindex after creating Bundle product

2bdbf8b

Add <wait> before running cron, as simultaneously running workers are…

8450dc9

… overlapping Cron executions

Extend Credit Card expiration date for another 4 years

a2e16f3

lbajsarowicz changed the title ~~#26607 Fix failing CI due to missing "createSimpleProductApi"~~ #26607 Fix failing CI due to Functional Tests Feb 1, 2020

lbajsarowicz added 2 commits February 1, 2020 20:33

Add indexing after creating products + changes due to invalid use of …

e52bc2a

…`sku` as URL Key :-(

urlKey is not used

548da89

lbajsarowicz changed the title ~~#26607 Fix failing CI due to Functional Tests~~ HOTFIX: #26607 Fix failing CI due to Functional Tests Feb 1, 2020

ihor-sviziev requested changes Feb 2, 2020

View reviewed changes

app/code/Magento/Bundle/Test/Mftf/Test/StorefrontCustomerSelectAndSetBundleOptionsTest.xml Outdated Show resolved Hide resolved

ghost added the Progress: needs update label Feb 2, 2020

lbajsarowicz added 3 commits February 2, 2020 11:12

Trying to avoid waitForIndex

ea4a31e

Fix invalid URL for order_shipment page

b130bd0

Extend the time wait for success notification

fc6fd62

lbajsarowicz mentioned this pull request Feb 2, 2020

Visa expiration date fix functional test dataset #26665

Closed

lenaorobei added the Progress: accept label Feb 13, 2020

lenaorobei removed the Award: complex label Feb 13, 2020

ghost added Progress: pending approval Progress: accept and removed Progress: accept Progress: pending approval labels Feb 13, 2020

lbajsarowicz added 2 commits February 21, 2020 21:40

Merge remote-tracking branch 'origin/2.4-develop' into hotfix/26607-c…

d3eaa9e

…i-failure # Conflicts: # app/code/Magento/Sales/Test/Mftf/Test/AdminReorderWithCatalogPriceRuleDiscountTest.xml

Merge remote-tracking branch 'lbajsarowicz/hotfix/26607-ci-failure' i…

3588ab3

…nto hotfix/26607-ci-failure

slavvka mentioned this pull request Feb 22, 2020

MFTF: AdminReorderWithCatalogPriceTest is failing in CI process #26607

Closed

magento-engcom-team pushed a commit that referenced this pull request Feb 22, 2020

ENGCOM-6825: HOTFIX: #26607 Fix failing CI due to Functional Tests #2…

e4dd671

…6608

magento-engcom-team merged commit 451d490 into magento:2.4-develop Feb 22, 2020

sdzhepa mentioned this pull request May 9, 2022

tyutyu sdzhepa/magento2#5

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HOTFIX: #26607 Fix failing CI due to Functional Tests #26608

HOTFIX: #26607 Fix failing CI due to Functional Tests #26608

lbajsarowicz commented Jan 31, 2020 •

edited by VladimirZaets

Loading

m2-assistant bot commented Jan 31, 2020

lbajsarowicz commented Feb 2, 2020

lbajsarowicz commented Feb 2, 2020

Nazar65 commented Feb 2, 2020 •

edited

Loading

lbajsarowicz commented Feb 2, 2020 •

edited

Loading

lbajsarowicz commented Feb 2, 2020

lbajsarowicz commented Feb 2, 2020 •

edited

Loading

magento-engcom-team commented Feb 13, 2020

lbajsarowicz commented Feb 16, 2020

lbajsarowicz commented Feb 21, 2020

m2-assistant bot commented Feb 22, 2020

zakdma commented Feb 24, 2020 •

edited

Loading

lbajsarowicz commented Feb 24, 2020 •

edited

Loading

lbajsarowicz commented Feb 24, 2020

zakdma commented Feb 24, 2020 •

edited

Loading

zakdma commented Feb 24, 2020

lbajsarowicz commented Feb 24, 2020

lbajsarowicz commented Feb 24, 2020 •

edited

Loading

zakdma commented Feb 24, 2020 •

edited

Loading

lbajsarowicz commented Feb 24, 2020

zakdma commented Feb 24, 2020 •

edited

Loading

lbajsarowicz commented Feb 24, 2020 •

edited

Loading

zakdma commented Feb 24, 2020

lbajsarowicz commented Feb 24, 2020

zakdma commented Feb 24, 2020

HOTFIX: #26607 Fix failing CI due to Functional Tests #26608

HOTFIX: #26607 Fix failing CI due to Functional Tests #26608

Conversation

lbajsarowicz commented Jan 31, 2020 • edited by VladimirZaets Loading

Description (*)

Related Pull Requests

Fixed Issues (if relevant)

Manual testing scenarios (*)

Questions or comments

Contribution checklist (*)

m2-assistant bot commented Jan 31, 2020

lbajsarowicz commented Feb 2, 2020

lbajsarowicz commented Feb 2, 2020

Nazar65 commented Feb 2, 2020 • edited Loading

lbajsarowicz commented Feb 2, 2020 • edited Loading

lbajsarowicz commented Feb 2, 2020

lbajsarowicz commented Feb 2, 2020 • edited Loading

magento-engcom-team commented Feb 13, 2020

lbajsarowicz commented Feb 16, 2020

lbajsarowicz commented Feb 21, 2020

m2-assistant bot commented Feb 22, 2020

zakdma commented Feb 24, 2020 • edited Loading

lbajsarowicz commented Feb 24, 2020 • edited Loading

lbajsarowicz commented Feb 24, 2020

zakdma commented Feb 24, 2020 • edited Loading

zakdma commented Feb 24, 2020

lbajsarowicz commented Feb 24, 2020

lbajsarowicz commented Feb 24, 2020 • edited Loading

zakdma commented Feb 24, 2020 • edited Loading

lbajsarowicz commented Feb 24, 2020

zakdma commented Feb 24, 2020 • edited Loading

lbajsarowicz commented Feb 24, 2020 • edited Loading

zakdma commented Feb 24, 2020

lbajsarowicz commented Feb 24, 2020

zakdma commented Feb 24, 2020

lbajsarowicz commented Jan 31, 2020 •

edited by VladimirZaets

Loading

Nazar65 commented Feb 2, 2020 •

edited

Loading

lbajsarowicz commented Feb 2, 2020 •

edited

Loading

lbajsarowicz commented Feb 2, 2020 •

edited

Loading

zakdma commented Feb 24, 2020 •

edited

Loading

lbajsarowicz commented Feb 24, 2020 •

edited

Loading

zakdma commented Feb 24, 2020 •

edited

Loading

lbajsarowicz commented Feb 24, 2020 •

edited

Loading

zakdma commented Feb 24, 2020 •

edited

Loading

zakdma commented Feb 24, 2020 •

edited

Loading

lbajsarowicz commented Feb 24, 2020 •

edited

Loading