JSON parsing with "marshal.to" raises exception if float attribute value has no decimal delimiter

[darklynx]

JsonParsingError is raised if marshaling a number without decimal delimiter into a property of float type

Example

import marshal

type Payment* = object
    currency*: string
    amount*: float64
    desc*: string

let p = marshal.to[Payment]("{ \"currency\": \"EUR\", \"amount\": 5 }")
echo "Amount: ", p.amount

Current Output

$ ./parsejson
/Users/test/dev/projects/nim-playground/parsejson.nim(7) parsejson
/usr/local/Cellar/nim/1.4.2/nim/lib/pure/marshal.nim(343) to
/usr/local/Cellar/nim/1.4.2/nim/lib/pure/marshal.nim(276) loadAny
/usr/local/Cellar/nim/1.4.2/nim/lib/pure/marshal.nim(196) loadAny
/usr/local/Cellar/nim/1.4.2/nim/lib/pure/marshal.nim(269) loadAny
/usr/local/Cellar/nim/1.4.2/nim/lib/pure/parsejson.nim(522) raiseParseErr
Error: unhandled exception: unknown file(1, 32) Error: float expected expected [JsonParsingError]

Expected Output

$ ./parsejson
Amount: 5.0

Possible Solution

Replacing 5 with 5.0 in the JSON example provides expected result:

import marshal

type Payment* = object
    currency*: string
    amount*: float64
    desc*: string

let p = marshal.to[Payment]("{ \"currency\": \"EUR\", \"amount\": 5.0 }")
echo "Amount: ", p.amount

Additional Information

I'm very new to Nim, and I might be just wrong about my expectations towards the marshal.to method. I have a feeling that it was not originally designed to perform JSON parsing by the JSON rules, it just accidentally can do that because "marshal" and "unmarshal" are using JSON representation in Nim.

However, I have discovered this issue while trying to use OpenAPI generator for Nim language. It uses marshal.to method when parsing incoming JSON response into a structure of expected type: https://github.com/OpenAPITools/openapi-generator/blob/master/modules/openapi-generator/src/main/resources/nim-client/api.mustache#L24

It works quite nice, unlike json.to(parseJson("..."), T.typedesc) construction, which fails on optional properties: if JSON message does not contain some fields that are defined by type:

$ ./parsejson
/Users/test/dev/projects/nim-playground/parsejson.nim(7) parsejson
/usr/local/Cellar/nim/1.4.2/nim/lib/pure/json.nim(1288) to
/usr/local/Cellar/nim/1.4.2/nim/lib/pure/json.nim(1184) initFromJson
/usr/local/Cellar/nim/1.4.2/nim/lib/pure/json.nim(1000) initFromJson
Error: unhandled exception: key not found: .desc [KeyError]

Maybe I should just report a bug in the "OpenAPI generator" project to the author of Nim client generator. But I would like first to get confirmation that marchal.to will never be a fully functional JSON parser.

Nim version (I'm using home brew version):

$ nim -v
Nim Compiler Version 1.4.2 [MacOSX: amd64]
Compiled at 2020-12-01
Copyright (c) 2006-2020 by Andreas Rumpf

active boot switches: -d:release -d:nimUseLinenoise

[timotheecour]

• std/jsonutils to the rescue: this works:

when true:
  import std/[jsonutils,json]
  type Payment* = object
      currency*: string
      amount*: float64
      desc*: string
  var p: Payment
  p.fromJson("{ \"currency\": \"EUR\", \"amount\": 5 }".parseJson, opt = Joptions(allowMissingKeys: true))
  doAssert p.amount == 5.0

• P1: this doesn't work yet but should, PR welcome (should be an easy PR): adding opt optional param to jsonTo:

let p2 = "{ \"currency\": \"EUR\", \"amount\": 5 }".parseJson.jsonTo(Payment, opt = Joptions(allowMissingKeys: true))

jsonutils is the future IMO, it has more options, supports tuples (named, unnamed), supports object variants etc;
EDIT: => #16739

• P2: that said, IMO marshal.to[Payment] should allow taking in 5 in top example, for consistency with std/jsonutils and std/json, and the fact that var a: float; a = 5 works; PR welcome too

@darklynx would you consider your first PR to fix either of those? the 1st item is the easiest

[Araq]

But I would like first to get confirmation that marchal.to will never be a fully functional JSON parser.

Confirmed. You can only use "to" on strings created by "$$".

[planetis-m]

@darklynx you can also consider planetis-m/eminim but be aware it isn't a fully featured std/json replacement nim-lang/RFCs#247

[darklynx]

Thanks a lot for explanation to everyone!

@Araq just as I thought 👍
@timotheecour I just started to learn Nim language, not yet ready to provide patches for the language and its libraries 😅 I don't even have a feeling for the language itself
@planetis-m thanks for the hint, but I think std/jsonutils seem to be the right approach to recommend for OpenAPI generator for Nim client

I'm going to report the potential issues with current code generated by OpenAPI generator and suggest the usage of std/jsonutils