--skip-existing seems not to work with some Nexus version #693

cyrilou242 · 2020-09-15T14:28:44Z

Environment

Your operating system:

ProductName:	Mac OS X
ProductVersion:	10.13.6
BuildVersion:	17G13035

Version of python you are running:

Python 3.6.6

How did you install twine? Did you use your operating system's package manager or pip or something else?

pip

Version of twine:

twine version 3.2.0 (pkginfo: 1.5.0.1, requests: 2.24.0, setuptools: 49.6.0,
requests-toolbelt: 0.9.1, tqdm: 4.31.1)

Which package repository are you targeting?

Nexus Repository OSS (https://www.sonatype.com/nexus-repository-oss)
Version | 3.25.1-04
Edition | OSS
Build Revision | d5c8e51f950fa30ce691e52998479b09ec888a2e
Build Timestamp | 2020-07-30-1345-34270

Issue

The --skip-existing flag that is supposed to ignore upload errors when a package is already existing in the repository is not working with my configuration: Nexus Repository OSS: 3.25.1-04.

When trying to upload a package that is already existing, I had a look at what's happening in the skip_upload function:

twine/twine/commands/upload.py

Line 30 in b5fa957

def skip_upload(

status, reason and text values are obtained from the response

In my case, I have:
status: 400
reason: bad request
text:

<!doctype html>
<html lang="en">
<head>
  <title>400 - nexus repository manager</title>
  <meta http-equiv="content-type" content="text/html; charset=utf-8"/>
  <!--[if lt ie 9]>
  <script>(new image).src="https://[private.repo.io]/favicon.ico?3.25.1-04"</script>
  <![endif]-->
  <link rel="icon" type="image/png" href="https://[private.repo.io]/favicon-32x32.png?3.25.1-04" sizes="32x32">
  <link rel="mask-icon" href="https://[private.repo.io]/safari-pinned-tab.svg?3.25.1-04" color="#5bbad5">
  <link rel="icon" type="image/png" href="https://[private.repo.io]/favicon-16x16.png?3.25.1-04" sizes="16x16">
  <link rel="shortcut icon" href="https://[private.repo.io]/favicon.ico?3.25.1-04">
  <meta name="msapplication-tileimage" content="https://[private.repo.io]/mstile-144x144.png?3.25.1-04">
  <meta name="msapplication-tilecolor" content="#00a300">
  <link rel="stylesheet" type="text/css" href="https://[private.repo.io]/static/css/nexus-content.css?3.25.1-04"/>
</head>
<body>
<div class="nexus-header">
  <a href="https://[private.repo.io]">
    <div class="product-logo">
      <img src="https://[private.repo.io]/static/images/nexus.png?3.25.1-04" alt="product logo"/>
    </div>
    <div class="product-id">
      <div class="product-id__line-1">
        <span class="product-name">nexus repository manager</span>
      </div>
      <div class="product-id__line-2">
        <span class="product-spec">oss 3.25.1-04</span>
      </div>
    </div>
  </a>
</div>
<div class="nexus-body">
  <div class="content-header">
    <img src="https://[private.repo.io]/static/rapture/resources/icons/x32/exclamation.png?3.25.1-04" alt="exclamation point" aria-role="presentation"/>
    <span class="title">error 400</span>
    <span class="description">bad request</span>
  </div>
  <div class="content-body">
    <div class="content-section">
      repository does not allow updating assets: pypi-hosted
    </div>
      </div>
</div>
</body>
</html>

The skip_upload function is looking for a 400 and "updating assets" in the reason string, but in my case "updating assets" is not in reason but in text.

I would suggest to change

twine/twine/commands/upload.py

Line 49 in b5fa957

or (status == 400 and "updating asset" in reason)

with something like

        # Nexus Repository OSS (https://www.sonatype.com/nexus-repository-oss)
        or (status == 400 and ("updating asset" in reason  or "updating asset" in text))

Steps to Reproduce

I have not tried to replicate with a local nexus repo.

The text was updated successfully, but these errors were encountered:

sigmavirus24 · 2020-09-15T15:13:08Z

in text would be ridiculously slow. Would be so nice for these repositories to not just break things arbitrarily.

cyrilou242 · 2020-09-15T16:00:23Z

Thanks for the quick answer and I definitely agree with the point on repositories not breaking things.

Given that the boolean evaluation is lazy the above code would only impact uploads returning a 400 and not having "updating asset" in reason.
Do you think it's still too impactful on the run time ?

For my example, time seems to be ok:

import  timeit

text = """
<!doctype html>
<html lang="en">
<head>
  <title>400 - nexus repository manager</title>
  <meta http-equiv="content-type" content="text/html; charset=utf-8"/>
  <!--[if lt ie 9]>
  <script>(new image).src="https://[private.repo.io]/favicon.ico?3.25.1-04"</script>
  <![endif]-->
  <link rel="icon" type="image/png" href="https://[private.repo.io]/favicon-32x32.png?3.25.1-04" sizes="32x32">
  <link rel="mask-icon" href="https://[private.repo.io]/safari-pinned-tab.svg?3.25.1-04" color="#5bbad5">
  <link rel="icon" type="image/png" href="https://[private.repo.io]/favicon-16x16.png?3.25.1-04" sizes="16x16">
  <link rel="shortcut icon" href="https://[private.repo.io]/favicon.ico?3.25.1-04">
  <meta name="msapplication-tileimage" content="https://[private.repo.io]/mstile-144x144.png?3.25.1-04">
  <meta name="msapplication-tilecolor" content="#00a300">
  <link rel="stylesheet" type="text/css" href="https://[private.repo.io]/static/css/nexus-content.css?3.25.1-04"/>
</head>
<body>
<div class="nexus-header">
  <a href="https://[private.repo.io]">
    <div class="product-logo">
      <img src="https://[private.repo.io]/static/images/nexus.png?3.25.1-04" alt="product logo"/>
    </div>
    <div class="product-id">
      <div class="product-id__line-1">
        <span class="product-name">nexus repository manager</span>
      </div>
      <div class="product-id__line-2">
        <span class="product-spec">oss 3.25.1-04</span>
      </div>
    </div>
  </a>
</div>
<div class="nexus-body">
  <div class="content-header">
    <img src="https://[private.repo.io]/static/rapture/resources/icons/x32/exclamation.png?3.25.1-04" alt="exclamation point" aria-role="presentation"/>
    <span class="title">error 400</span>
    <span class="description">bad request</span>
  </div>
  <div class="content-body">
    <div class="content-section">
      repository does not allow updating assets: pypi-hosted
    </div>
      </div>
</div>
</body>
</html>
"""

setup = 'text = """{}""";'.format(text)
number=100000
r = timeit.timeit('"updating asset" in text', setup=setup, number=number)
print(r/number)

--> around 1e-06 seconds on a correct macbook (not very precise and more powerful that a ci machine indeed)

400 + "bad request" is obviously not enough.
Also in text is already in use for Artifactory:

twine/twine/commands/upload.py

Line 51 in b5fa957

or (status == 403 and "overwrite artifact" in text)

sigmavirus24 · 2020-09-15T21:01:10Z

I mean we might as ewll

cyrilou242 · 2020-09-17T14:32:18Z

This is fixed by #694.
I let a member close (depends if you want to close now or at next release)

cyrilou242 mentioned this issue Sep 16, 2020

Nexus repo fix #694

Merged

bhrutledge closed this as completed Sep 17, 2020

bhrutledge added this to the 3.3.0 milestone Oct 4, 2020

bhrutledge linked a pull request Oct 4, 2020 that will close this issue

Nexus repo fix #694

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

--skip-existing seems not to work with some Nexus version #693

--skip-existing seems not to work with some Nexus version #693

cyrilou242 commented Sep 15, 2020

sigmavirus24 commented Sep 15, 2020

cyrilou242 commented Sep 15, 2020 •

edited

Loading

sigmavirus24 commented Sep 15, 2020

cyrilou242 commented Sep 17, 2020 •

edited

Loading

--skip-existing seems not to work with some Nexus version #693

--skip-existing seems not to work with some Nexus version #693

Comments

cyrilou242 commented Sep 15, 2020

Environment

Issue

Steps to Reproduce

sigmavirus24 commented Sep 15, 2020

cyrilou242 commented Sep 15, 2020 • edited Loading

sigmavirus24 commented Sep 15, 2020

cyrilou242 commented Sep 17, 2020 • edited Loading

cyrilou242 commented Sep 15, 2020 •

edited

Loading

cyrilou242 commented Sep 17, 2020 •

edited

Loading