Upload to bintray in ci fails with AttributeError: 'NoneType' object has no attribute 'file_sums'

[madebr]

Uploading packages to bintray on travis-ci fails with an AttributeError.
https://travis-ci.org/madebr/conan-tcl/jobs/503306695#L2017

The package is a standard conan recipe: https://github.com/madebr/conan-tcl/tree/testing/8.6.9

Traceback (most recent call last):
  File "/opt/pyenv/versions/3.7.1/bin/run_create_in_docker", line 11, in <module>
    load_entry_point('conan-package-tools==0.24.0', 'console_scripts', 'run_create_in_docker')()
  File "/opt/pyenv/versions/3.7.1/lib/python3.7/site-packages/cpt/run_in_docker.py", line 45, in run
    runner.run()
  File "/opt/pyenv/versions/3.7.1/lib/python3.7/site-packages/cpt/runner.py", line 111, in run
    self._upload, package_id)
  File "/opt/pyenv/versions/3.7.1/lib/python3.7/site-packages/cpt/uploader.py", line 43, in upload_packages
    retry=int(self._upload_retry))
  File "/opt/pyenv/versions/3.7.1/lib/python3.7/site-packages/conans/client/conan_api.py", line 93, in wrapper
    return f(*args, **kwargs)
  File "/opt/pyenv/versions/3.7.1/lib/python3.7/site-packages/conans/client/conan_api.py", line 855, in upload
    retry_wait, integrity_check, policy, remote_name, query=query)
  File "/opt/pyenv/versions/3.7.1/lib/python3.7/site-packages/conans/client/cmd/uploader.py", line 87, in upload
    integrity_check, policy, remote, upload_recorder)
  File "/opt/pyenv/versions/3.7.1/lib/python3.7/site-packages/conans/client/cmd/uploader.py", line 189, in _upload_ref
    self._upload_recipe(ref, conanfile, retry, retry_wait, policy, recipe_remote)
  File "/opt/pyenv/versions/3.7.1/lib/python3.7/site-packages/conans/client/cmd/uploader.py", line 229, in _upload_recipe
    remote, remote_manifest)
  File "/opt/pyenv/versions/3.7.1/lib/python3.7/site-packages/conans/client/cmd/uploader.py", line 342, in _recipe_files_to_upload
    if remote_manifest == local_manifest:
  File "/opt/pyenv/versions/3.7.1/lib/python3.7/site-packages/conans/model/manifest.py", line 127, in __eq__
    return self.file_sums == other.file_sums
AttributeError: 'NoneType' object has no attribute 'file_sums'
Traceback (most recent call last):
  File "build.py", line 10, in <module>
    builder.run()
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/cpt/packager.py", line 449, in run
    self.run_builds(base_profile_name=base_profile_name)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/cpt/packager.py", line 554, in run_builds
    docker_entry_script=self.docker_entry_script)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/cpt/runner.py", line 237, in run
    raise Exception("Error building: %s" % command)

[memsharded]

Hi @madebr

I have tracked the issue to a broken package in your repository: https://bintray.com/madebr/maarten/tcl%3Amadebr/8.6.9%3Atesting#files/madebr%2Ftcl%2F8.6.9%2Ftesting%2Fexport

There is no conanmanifest.txt there. This might be an interrupted packaged upload (network issues, etc).

You need to remove the package first before uploading, or use --force argument.

@danimtb had already started a couple of days ago this PR #4662 to improve the robustness of Conan against this kind of transfer errors, hopefully it can be improved.

[madebr]

Thanks for tracking it down and fixing it in the future.
For now, I've removed all packages from bintray.

[monsdar]

Same issue over here, thanks for the quick solution

[monsdar]

Ok, update to my previous statement... After removing the remote packages from the server we still encounter the message. Here's what we know so far:

• 4 CI runners are building the same package with different configurations (different packageIds)
• All 4 CI runners are done at the same time
• All 4 CI runners are calling conan upload foo/1.0.0@user/release -r remote for the same package
• Some of the uploads fail with the above error message

We added a sleep of 0, 30, 60 and 90 seconds to the CI builds, so that they do not finish at the same time but slightly delayed. Then the uploads happen after each other instead of in parallel. With this hack we encounter no issues anymore.

We are still using Conan Server 1.3.3 though. So this could also be an issue that has been resolved. Our Conan client is at 1.13.0 right now. We will update the Conan server soon. We also have an Artifactory EE set up that we will start using with Conan in the near future.

I still need to reproduce this in an isolated setup. But right now this is our situation. I will take some time later this week to check how to repro this. When I have a repro setup I'll also test with different server versions.

[danimtb]

@monsdar yes, the scenario you are describing is totally possible.

This could happen in concurrent uploads as the packages might still be incomplete when another job tries to make the same package upload.

This error is something we have seen when packages are corrupted or have missing files in the server, so we have improved the upload workflow in #4662 to force the upload in the case the package is incomplete. This should mitigate your issue.

However, I strongly recommend you to set up an Artifactory for enterprise use as the conan_server is not prepared to scale and has limitations with concurrency.

[monsdar]

Yes, Artifactory will be the long-term solution. It's already set up, it's just not in productive use by now. We'll probably switch over sooner rather than later.

I was trying to reproduce this issue by running 4 docker containers that build the same component with different options (same package, different packageIds). When I trigger the upload on all 4 instances at the same time it works flawless everytime. Perhaps my repro environment is not set up quite right to cause the error.

[danimtb]

The collision happens when uploading the same package (same package ID) at the same time.

[monsdar]

Is it possible to have a lock in place for this?

[villytiger]

I have the same problem with Conan 1.14.1. In my configuration, I have 2 CI jobs building for 2 different architectures from the same recipe. I got the following error in one of them during upload.

Traceback (most recent call last):
  File "/home/user/.local/lib/python3.7/site-packages/conans/client/command.py", line 1579, in run
    method(args[0][1:])
  File "/home/user/.local/lib/python3.7/site-packages/conans/client/command.py", line 1182, in upload
    retry_wait=args.retry_wait, integrity_check=args.check)
  File "/home/user/.local/lib/python3.7/site-packages/conans/client/conan_api.py", line 93, in wrapper
    return f(*args, **kwargs)
  File "/home/user/.local/lib/python3.7/site-packages/conans/client/conan_api.py", line 868, in upload
    retry_wait, integrity_check, policy, remote_name, query=query)
  File "/home/user/.local/lib/python3.7/site-packages/conans/client/cmd/uploader.py", line 88, in upload
    integrity_check, policy, remote, upload_recorder)
  File "/home/user/.local/lib/python3.7/site-packages/conans/client/cmd/uploader.py", line 190, in _upload_ref
    self._upload_recipe(ref, conanfile, retry, retry_wait, policy, recipe_remote)
  File "/home/user/.local/lib/python3.7/site-packages/conans/client/cmd/uploader.py", line 230, in _upload_recipe
    remote, remote_manifest)
  File "/home/user/.local/lib/python3.7/site-packages/conans/client/cmd/uploader.py", line 344, in _recipe_files_to_upload
    if remote_manifest == local_manifest:
  File "/home/user/.local/lib/python3.7/site-packages/conans/model/manifest.py", line 132, in __eq__
    return self.file_sums == other.file_sums
AttributeError: 'NoneType' object has no attribute 'file_sums'

[lasote]

We can continue at #4953 the discussion. Thanks