Fix: handle corner cases related to SCM with local sources optimization #4249

jgsogo · 2019-01-08T16:10:33Z

Changelog: Fix: Handle corner cases related to SCM with local sources optimization
Docs: Omit

closes [SVN] Optimization on SCM ('scm_folder.txt') breaks url "auto" on monorepo (some use-cases) #4222
closes? [SCM][SVN] Support working in a mono-repo checkout #3786
closes? Behaviour of scm plugin for specific commit with local build subdirectory #4139

It should fix some tests in PR #4193

Note to reviewers, these are the steps to follow:

Check new tests
Check modified tests
Review implementation
I think that some tests are duplicated (between the old and the new ones), I would remove the dupe ones in the old test suite because I think that the new ones are clearer and better structured. Which tests should we remove? (read this comment: Fix: handle corner cases related to SCM with local sources optimization #4249 (comment))

@tags: svn

… testing the same)

…rls won't be computed properly

… won't match the clause in the exclusion function. Also conditionally compute local_sources_path to scm_url

… scm

…repo-auto

memsharded

Please check broken CI

memsharded · 2019-01-17T11:47:27Z

conans/test/functional/command/source_test.py

@@ -20,8 +20,10 @@ class ScmtestConan(ConanFile):
 """
        client = TestClient()
        client.save({"conanfile.py": conanfile})
+        client.runner("git init .", cwd=client.current_folder)


Why? The test case was failing even if it wasn't a git repo. Maybe leave the test as it was, and if anything, adding a new one with a git repo too.

Without creating the Git repository this test fails (because it cannot get the URL to the remote). It is related to the comment about the difficult to understand block. If I execute conditionally that block for Git, then this test (and many others) can be kept as before.

Got it. Maybe an error test, if it is not a git repo, what is the error message. Who is launching it and at which step? Hint: should probably be on export time, not on "create" time.

What if:

URL is defined hardcoded with full url, not "auto"

revision=auto

conans/model/scm.py

conans/client/cmd/export.py

conans/test/functional/scm/scm_test.py

conans/test/unittests/client/cmd/source/test_run_scm.py

conans/client/source.py

conans/test/functional/scm/scm_test.py

memsharded · 2019-01-28T15:34:19Z

conans/client/cmd/export.py

-        # Generate the scm_folder.txt file pointing to the src_path
-        src_path = scm.get_repo_root()
-        save(scm_src_file, src_path.replace("\\", "/"))
+    captured = scm_data.capture_origin or scm_data.capture_revision

    if scm_data.url == "auto":


Check with a test of export+upload, without having a remote repo origin, and it should fail.

memsharded · 2019-01-28T15:36:56Z

conans/test/functional/command/source_test.py

@@ -20,8 +20,10 @@ class ScmtestConan(ConanFile):
 """
        client = TestClient()
        client.save({"conanfile.py": conanfile})
+        client.runner("git init .", cwd=client.current_folder)


Got it. Maybe an error test, if it is not a git repo, what is the error message. Who is launching it and at which step? Hint: should probably be on export time, not on "create" time.

memsharded · 2019-01-28T15:39:21Z

conans/test/functional/command/source_test.py

@@ -20,8 +20,10 @@ class ScmtestConan(ConanFile):
 """
        client = TestClient()
        client.save({"conanfile.py": conanfile})
+        client.runner("git init .", cwd=client.current_folder)


What if:

URL is defined hardcoded with full url, not "auto"

revision=auto

conans/test/functional/scm/scm_test.py

…ion)

jgsogo · 2019-01-28T17:16:40Z

I've added some tests and modified one behavior after talking with @memsharded; please @lasote have a look at them and let us know your opinion:

auto keywords, both for url and for revision, can only we used if the repository has a remote, otherwise it will return an ERROR with: ERROR: Repo origin cannot be deduced. This happens as early as possible, in the export command.

We think this is expected for url=auto, but we are not sure if this is expected for revision=auto too (having a hardcoded url). We could keep going, as the url is already known and use the local sha for the revision...

This change implies adding a remote to some existing tests with scm.
When there is no repo (the working copy is not a repo), the command export will fail if there is an auto for the url or for the revision.

…top folder

jgsogo added 12 commits January 4, 2019 14:53

reorder imports

6a5d3f4

tests for the bug

3b812b6

share as much code as possible between all tests (be sure that we are…

18a7a22

… testing the same)

conanfile is in root folder

7bf233b

reordered tests

843310c

add tests with conanfile in subfolder

3e4cfc5

fix tests when conanfile is in repo root

1ed8340

fix local source method

100eb2c

remove repeated tests

ca5b0d3

remove comments

7de4d29

remove test

4a29eb3

tag tests

1894a7a

ghost assigned jgsogo Jan 8, 2019

ghost added the stage: review label Jan 8, 2019

jgsogo added 16 commits January 8, 2019 17:42

normalize path before storing it in scm_folder

febe546

clean URL (remove peg-revision from SVN), otherwise relpath between u…

7eafda4

…rls won't be computed properly

normalice 'src', if it enters 'merge_directories' with a final dot it…

0dc41cf

… won't match the clause in the exclusion function. Also conditionally compute local_sources_path to scm_url

add function to clean url inside SCM class, it knows all the possible…

e45d606

… scm

fix tests that were failing

91b2b7b

add tests that should work for SVN using url='auto'

54e9a6d

Merge remote-tracking branch 'conan/develop' into issue/4222-scm-mono…

1c25bdd

…repo-auto

remove dupe test

8487db9

if we use 'scm.url=['auto']' we need a repository here

f763903

we need something that resembles an url/file/...

5148671

inherit from 'object', compatibility issues

b25fe45

if not local repo, then there will be no url

1db293e

order lists before comparing them

84e97b7

use forward slashes in command line (single back-slash will be escaped)

c2ec7c1

replace slashes (test failing in windows)

451253c

forward slashes (failing tests in Windwos)

c8f976a

ghost assigned jgsogo Jan 17, 2019

lasote unassigned jgsogo Jan 17, 2019

memsharded reviewed Jan 21, 2019

View reviewed changes

conans/client/cmd/export.py Outdated Show resolved Hide resolved

conans/test/functional/scm/scm_test.py Show resolved Hide resolved

conans/test/unittests/client/cmd/source/test_run_scm.py Outdated Show resolved Hide resolved

conans/client/source.py Outdated Show resolved Hide resolved

fix argument name changed

33d0c84

ghost assigned jgsogo Jan 21, 2019

classmethod to static

560d5d5

jgsogo commented Jan 21, 2019

View reviewed changes

conans/test/functional/scm/scm_test.py Outdated Show resolved Hide resolved

jgsogo added 3 commits January 21, 2019 15:41

move unittest to proper place

5e7b955

add function to SCM to handle the computation of the local sources path

c451b37

add docs

8855cc9

lasote unassigned jgsogo Jan 28, 2019

jgsogo added 2 commits January 28, 2019 12:28

it doesn't breaks, and the scm keeps its value

31121a2

remove FIXME

d500375

ghost assigned jgsogo Jan 28, 2019

memsharded approved these changes Jan 28, 2019

View reviewed changes

jgsogo added 4 commits January 28, 2019 17:17

add tests without repo and without origin

914d9a2

same for SVN

752ee6b

fix some unittests that now requires an origin to work

31dcb84

we need a remote to resolve the 'auto' (although it is only for revis…

f7cf4e7

…ion)

jgsogo added 5 commits January 29, 2019 10:20

do not check origin if url is defined

40834fc

if repo is Git, then local_source_folder should always equal to repo …

9c91fee

…top folder

avoid calling twice the function

7cfe96b

no need for valid URL if git

ec55353

typo

33734c4

memsharded approved these changes Jan 29, 2019

View reviewed changes

lasote merged commit a11b8a6 into conan-io:develop Jan 29, 2019

ghost removed the stage: review label Jan 29, 2019

jgsogo deleted the issue/4222-scm-monorepo-auto branch January 29, 2019 11:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix: handle corner cases related to SCM with local sources optimization #4249

Fix: handle corner cases related to SCM with local sources optimization #4249

jgsogo commented Jan 8, 2019 •

edited

Loading

memsharded left a comment

memsharded Jan 17, 2019

jgsogo Jan 21, 2019 •

edited

Loading

memsharded Jan 28, 2019

memsharded Jan 28, 2019

memsharded Jan 28, 2019

memsharded Jan 28, 2019

memsharded Jan 28, 2019

jgsogo commented Jan 28, 2019

Fix: handle corner cases related to SCM with local sources optimization #4249

Fix: handle corner cases related to SCM with local sources optimization #4249

Conversation

jgsogo commented Jan 8, 2019 • edited Loading

memsharded left a comment

Choose a reason for hiding this comment

memsharded Jan 17, 2019

Choose a reason for hiding this comment

jgsogo Jan 21, 2019 • edited Loading

Choose a reason for hiding this comment

memsharded Jan 28, 2019

Choose a reason for hiding this comment

memsharded Jan 28, 2019

Choose a reason for hiding this comment

memsharded Jan 28, 2019

Choose a reason for hiding this comment

memsharded Jan 28, 2019

Choose a reason for hiding this comment

memsharded Jan 28, 2019

Choose a reason for hiding this comment

jgsogo commented Jan 28, 2019

jgsogo commented Jan 8, 2019 •

edited

Loading

jgsogo Jan 21, 2019 •

edited

Loading