Mismatched checksums on a couple gems

[tjgrathwell]

From rubygems/bundler#5332

Since Bundler 1.14 started validating checksums, a couple of people have had failed builds caused by seemingly invalid checksums in the RubyGems DB.

gem	rubygems says	shasum says
faraday_middleware	c56bb3661af1d5057e578d72d2f9e93ab52630b74ebe37ce1118c93a6ddff56b	ff2bdbb6bdd5b2293f8ce7cb93572318dd068506e12f987f04a65209a2bc69cb
azure-core	b04f7f994e3abc31003139a7ff8a682980c1d064a906a3290d5c369685bc478b	68ebac6994412a775584478961747a215d0844737a8b9a91405a0ac77fe6c514

@chrismo ran a check on the 700 most popular gems and didn't find any other instances, but that doesn't mean they aren't out there! It would be nice to know how these SHAs ended up this way.

[cpetschnig]

hash_kit 0.4.8 is also affected: The expected SHA256 checksum was
"f7ae2829a96761a0569eac8f20066038c6ace083b797465d19e2a6f03aa3af94", but the
checksum for the downloaded gem was
"e4c66f4c62f66946de11fbbd9839b8cce78f89661977ae03b551e446dc729c8e".

[sonalkr132]

I tried pushing faraday_middleware-0.11.0.gem and hash_kit-0.4.8.gem on my development instance, it generated the expected sha256 values, ie ff2bdbb6bdd5b2293f8ce7cb93572318dd068506e12f987f04a65209a2bc69cb and e4c66f4c62f66946de11fbbd9839b8cce78f89661977ae03b551e446dc729c8e.

I am afraid it is going to be difficult to solve unless we can reproduce the issue.

[sonalkr132]

Can someone please send me the original .gem file, the one which was generated using gem build ..?

cc: @iMacTia

[iMacTia]

Hi @sonalkr132 we have an automated process in Faraday and FaradayMiddleware that bundle and push the gem automatically from Travis. That's because I'm not the main maintainer of the gems and I don't have writing permissions on rubygems.

I think @mislav will be able to help in this case.

[iMacTia]

I'm not sure that will help, but the bugged checksum was generated from this tag https://github.com/lostisland/faraday_middleware/tree/v0.11.0

[sonalkr132]

I'm not sure that will help, but the bugged checksum was generated from this tag

Yeah, I already tried that. It generates a completely different checksum.

[iMacTia]

@navidemad I think that's not needed anymore as I released v0.11.0.1 yesterday
It's a carbon-copy of v0.11.0, but this time the correct checksum was generated

[sarangan12]

For the problem with the checksum of azure-core (and similar gems), should we release a new version? Is that the only solution?

Also, how could we be sure that re-releasing the gem will generate the correct checksum?

[007]

Unrelated to Faraday or azure-core, https://rubygems.org/gems/csvlint/versions/0.1.2 is failing validation.

RubyGems: 6af618357fdc57e8b87750742338032f9df8692d0f8f756436c6828383fd0f37
Actual: 24566781f6be914d26b8a31da50f8a6e1eb5d5026419923485e59351f6a84bc0

[sonalkr132]

I pushed some 15 versions to our staging site and all of them generated correct checksum.
Regretfully, pushing a new version seems to the only option here unless someone can up with a .gem file which generates an incorrect sha value.

[dwradcliffe]

I think most of these are older versions, probably from before we saved the checksums on gem push. I'm guessing there were a few that didn't get backfilled properly. Unless someone has an example of a recent (and repeatable) missmatch I don't think we have an ongoing problem here.

For these old ones, I think it's probably best to release new versions than go back and mess with the old versions.

[indirect]

I agree... we can correct the checksums in the database, but it's much safer to do them all at once. Then we also only have to expire the versions list and force everyone to download the whole thing again one time.

For the time being, please release new versions to force correct checksums, and we'll go back and fix the checksums on old versions all at once.

[arthurnn]

funny thing tho. I, THINK(not 100% sure), that when I ran the jobs the generate the checksums, I had something to double check them.

[segiddins]

Fixing the checksums wont expire the versions list, only the respective info files

[mwpastore]

mailgun-ruby-1.1.3 also seems to be affected.

[tiagoamaro]

Just bumped into this issue when downloading faraday 0.12.0 (using latest rubygems version: 2.6.11). I've tried the following approach:

For faraday 0.12.0:

• gem fetch faraday -v 0.12.0
• shasum -a256 faraday-0.12.0.gem
  • Resulted in the following hash: 974ec50064433c84a0015b9f985d676e82b5532fd82ca0e2e546fbd63034a1b8 faraday-0.12.0.gem
  • Rubygems hash: 467b18cc2683845e90996e2c7f3af0cf34493d78b687445b2b4999711f6f6dca

For Faraday 0.11.0:

• gem fetch faraday -v 0.11.0
• shasum -a256 faraday-0.11.0.gem
  • Resulted in the following hash: 5986f49009d638d971997082f1a6cf1ed835144a66d5c11dd495bf1d9b66e1a2 faraday-0.11.0.gem
  • Rubygems hash: 5986f49009d638d971997082f1a6cf1ed835144a66d5c11dd495bf1d9b66e1a2

[iMacTia]

Thanks for reporting this @tiagoamaro, I released version 0.12.0 this morning and it had the same issue we had with faraday_middleware.
Last time I solved releasing again, @sonalkr132 do you need time to investigate or can I proceed releasing again?

[sonalkr132]

do you need time to investigate or can I proceed releasing again?

I believe we know why this is happening. You are using Travis to push gems. Travis ends pushing the gem from all the jobs. One of them pass and rest fail. Because of the race condition in Api::V1::RubygemsController#create, we end up with incorrect checksums.
You are welcome to leave your thoughts about proposed fix on #1564.

For the time being, you can use condition: "$TRAVIS_RUBY_VERSION == x.x.x" in your .travis.yml file to ensure that you don't end up pushing same version multiple times.

[iMacTia]

Thanks for your input @sonalkr132.
The condition have been added and I'm re-publishing the gem, hopefully this time will be ok.