jmespath and JSON::Pure::Generator::State

[darkarnium]

Hey there,

We have had a situation in the last 24 hours, where a 'breaking' change was introduced into the jmespath dependency for aws-sdk-core. As a result, some internal deployment scripts which utilize aws-sdk-core were failing to properly serialize Hashes which contain FixNums into JSON.

As an example, the Hash in question contains an element where the value is seconds since epoch (which has been mocked below):

attributes = {}
attributes[:booted_at] = 1459531976
pp JSON.pretty_generate(attributes)

When this was attempted to be serialized using JSON.pretty_generate, it was failing with something along the lines of the following:

[2016-04-01T17:44:34+00:00] FATAL: wrong argument type JSON::Pure::Generator::State (expected Data)
/var/lib/gems/1.9.1/gems/json_pure-1.8.3/lib/json/pure/generator.rb:366:in `to_json': wrong argument type JSON::Pure::Generator::State (expected Data) (TypeError)
        from /var/lib/gems/1.9.1/gems/json_pure-1.8.3/lib/json/pure/generator.rb:366:in `block in json_transform'
        from /var/lib/gems/1.9.1/gems/json_pure-1.8.3/lib/json/pure/generator.rb:359:in `each'
        from /var/lib/gems/1.9.1/gems/json_pure-1.8.3/lib/json/pure/generator.rb:359:in `json_transform'
        from /var/lib/gems/1.9.1/gems/json_pure-1.8.3/lib/json/pure/generator.rb:341:in `to_json'
        from /var/lib/gems/1.9.1/gems/json_pure-1.8.3/lib/json/pure/generator.rb:293:in `generate'

Looking at the Gemspec for aws-sdk-core, it appears that this package has a 'twiddle-wakka' set to 1.0 for jmespath. As a result of this 'pessimistic' version pin, the new 'breaking' version of jmespath was pulled in to a pinned version of the aws-sdk-core and starting wrecking havoc.

The version of jmespath where we started seeing issues was 1.2.2 - which was released about 19 hours ago. It looks like as part of this release, json_pure was depended upon rather than json. This change seems to be where our problems were caused (due to some 'weirdness' with json_pure and FixNum).

Is there any plan to have a 'stronger' version pin on this dependency? This has caused us a number of pre-coffee hair-pulling moments this morning, and a late night for at least one of our guys yesterday. If not, hopefully this issue with help others seeing similar issues over the last 20 hours or so! :)

• https://github.com/jmespath/jmespath.rb/releases/tag/v1.2.2
• TypeError when using json_pure with Chef jmespath/jmespath.rb#20

Cheers!
Peter

[awood45]

Version 1.2.3 is out, this issue should provide more context: jmespath/jmespath.rb#20

Let me know if this helps.

[darkarnium]

Hey there,

Yep, that release does indeed resolve the issue - as does hacking in a to_json method on the FixNum class, albeit the latter being super dodgy :)

Cheers,
Peter

[awood45]

Sorry about the trouble, glad the new release solves the issue. Closing this for now, let us know if you have any other issues or questions with this.

[musiaht]

Will it be possible to consider stricter version pinning?

[trevorrowe]

@musiaht Do you mean stricter pinning against jmespath or in jmespath against the json gem?

[darkarnium]

@trevorrowe Not @musiaht but I'd guess that he may be referring to the pinning between aws-sdk-core and jmespath, as this is the particular pin that seemed to be the cause of this issue.

[musiaht]

@trevorrowe I meant pinning against jmespath; but after reading and jmespath/jmespath.rb#18 jmespath/jmespath.rb#20 more carefully, I'm not sure if that's going to help.

Thanks for your help