native API for creating dataverse reports parse error for valid json

[pameyer]

native API endpoint for creating a dataverse reports parse error for valid JSON.

curl -X POST -d @l0.json -H "X-Dataverse-key: $admin_api_key" $dataverse_host/api/dataverses/dv_alias

returns (w\ http 400):
{"status":"ERROR","message":"Error parsing Json: Invalid token=EOF at (line no=1, column no=0, offset=-1). Expected tokens are: [CURLYOPEN, SQUAREOPEN]"}

l01.json:
{
"name":"dv_name",
"alias":"dv_alias",
"dataverseContacts":[ {"contactEmail":"foo+dataverse@example.com"} ],
"affiliation":"aff",
"dataverseType":"LABORATORY"
}

jq and python 2.7.8 json library both think this is valid JSON. Could be either an endpoint glitch being reported to the user as parser error, or parser error - haven't investigated further yet.

[pameyer]

same behavior for using "--upload-file l0.json" instead of "@l0.json"

[pdurbin]

This works on my machine (I'm on 9024856):

curl -i -X POST --upload-file /tmp/foo.json -H "X-Dataverse-key: $API_TOKEN" http://localhost:8080/api/dataverses/root

HTTP/1.1 100 Continue

HTTP/1.1 201 Created
Server: GlassFish Server Open Source Edition  4.1 
X-Powered-By: Servlet/3.1 JSP/2.3 (GlassFish Server Open Source Edition  4.1  Java/Oracle Corporation/1.8)
Date: Thu, 15 Dec 2016 19:06:10 GMT
Location: http://localhost:8080/dataverses/dv_alias
Content-Type: application/json
Content-Length: 332

{"status":"OK","data":{"id":14,"alias":"dv_alias","name":"dv_name","affiliation":"aff","dataverseContacts":[],"permissionRoot":true,"dataverseType":"LABORATORY","ownerId":1,"creationDate":"2016-12-15T19:06:10Z","creator":{"identifier":"@dataverseAdmin","displayInfo":{"Title":"Dataverse Admin","email":"dataverse@mailinator.com"}}}}

However, I noticed that this doesn't (note dvThatDoesNotExist):

curl -i -X POST --upload-file /tmp/foo.json -H "X-Dataverse-key: $API_TOKEN" http://localhost:8080/api/dataverses/dvThatDoesNotExist

HTTP/1.1 100 Continue

HTTP/1.1 500 Internal Server Error
Server: GlassFish Server Open Source Edition  4.1 
X-Powered-By: Servlet/3.1 JSP/2.3 (GlassFish Server Open Source Edition  4.1  Java/Oracle Corporation/1.8)
Date: Thu, 15 Dec 2016 19:06:57 GMT
Content-Type: application/json
Connection: close
Content-Length: 159

{"status":"ERROR","code":500,"message":"Internal server error. More details available at the server logs.","incidentId":"0d791292-bd2e-4746-9ef7-8f9df217b716"}

Stacktrace:

API internal error 0d791292-bd2e-4746-9ef7-8f9df217b716: Null Pointer java.lang.NullPointerException at edu.harvard.iq.dataverse.api.Dataverses.addDataverse(Dataverses.java:138)

Line 138 is this: while (cause.getCause() != null) {

Same data @pameyer is using:

(dataverse) murphy:dataverse pdurbin$ cat /tmp/foo.json 
{
"name":"dv_name",
"alias":"dv_alias",
"dataverseContacts":[ {"contactEmail":"foo+dataverse@example.com"} ],
"affiliation":"aff",
"dataverseType":"LABORATORY"
}

@pameyer which version or commit are you on? Maybe you've found two bugs! 😄

[pameyer]

@pdurbin - I'm seeing this on 52e9ec3 .

[pdurbin]

@pameyer closer to your original bug report, this is what I get when I use curl -d on a file that doesn't exist on disk:

curl -i -X POST -d @/tmp/nosuchfile.json -H "X-Dataverse-key: $API_TOKEN" http://localhost:8080/api/dataverses/root

Warning: Couldn't read data from file "/tmp/nosuchfile.json", this makes an 
Warning: empty POST.
HTTP/1.1 400 Bad Request
Server: GlassFish Server Open Source Edition  4.1 
X-Powered-By: Servlet/3.1 JSP/2.3 (GlassFish Server Open Source Edition  4.1  Java/Oracle Corporation/1.8)
Content-Type: application/json
Date: Thu, 15 Dec 2016 19:32:56 GMT
Connection: close
Content-Length: 153

{"status":"ERROR","message":"Error parsing Json: Invalid token=EOF at (line no=1, column no=0, offset=-1). Expected tokens are: [CURLYOPEN, SQUAREOPEN]"}

$ cat /tmp/nosuchfile.json
cat: /tmp/nosuchfile.json: No such file or directory

[pdurbin]

@pameyer as we discussed I can't replicate the "Error parsing Json" error so I'm tempted to close this issue as "can't reproduce". It sounds like you want to try a few more things before we close it.

I did open a new issue at #3534 for the 500 error I stumbled upon.

[pameyer]

As @pdurbin mentioned; "unable to reproduce" means the problem is probably on my end (and the json file does exist). Doing some additional troubleshooting before closing in the event others run into the same problem.

For future reference; earlier comment about "-d @l0.json" and "--upload-file l0.json" was wrong (typo'd) - "-d @l0.json" fails and from curl docs probably inappropriate.

@pdurbin 's comment in #3534 indirectly revealed the initial problem: user error when attempting to create a dataverse under the root dataverse, and using the alias for the dataverse to be created as the $id parameter, or without an $id parameter (instead of using $id=$alias_of_root_dataverse).

Behavior with "-d @$filename" was identical to behavior of "--upload-file $non_existent_file" because I didn't do any form/url encoding or setting file param of the POST data in the "-d" usage (might or might not behave the same; not troubleshooting further).

[ghost]

@pameyer - Did you ever find a solution to the problem? We are getting the same parse error with our test dataverse installation (though no problem with our production installation)

[pdurbin]

@FASposito this issue was closed and I was having trouble reproducing it. I'm happy to open it back up. Can you please provide the following?

• curl command you're using (not with a real API token)
• JSON input file (you can attach this if you add ".txt"
• version of Dataverse you're testing.

Thanks!

p.s. I saw your post here too: https://groups.google.com/d/msg/dataverse-community/fKgtWBXNgBo/3-KW9qxmAwAJ

[ghost]

@pdurbin ,
Thanks very much for helping with this. The dataverse version we're running is 4.6.2. Here is the curl command:

curl -H "X-Dataverse-key: 9e76e321-b5db-4e51-a48a-0eec44d12718" -H "Content-type:application/json" -X POST -k -L -d @dataverse.create.json http://sandbox.vdc.ac/api/dataverses/play

This is the content of dataverse.create.json:

{"name":"Sposito Research","alias":"sposito","dataverseContacts":[{"contactEmail":"frank.sposito@univie.ac.at"}],"affiliation":"Austrian Social Science Data Archive","description":"Putting the social back into social sciences data","dataverseType":"UNCATEGORIZED"}

And here is the error:

{"status":"ERROR","message":"Error parsing Json: Invalid token=EOF at (line no=1, column no=0, offset=-1). Expected tokens are: [CURLYOPEN, SQUAREOPEN]"}

The json file works with both demo.dataverse.org and our production server, data.aussda.at.
Best, Frank

[pdurbin]

@FASposito I meant to mention I saw you briefly over here too: http://irclog.iq.harvard.edu/dataverse/2017-12-20 😄

It's actually pretty easy for me to reproduce the error above but I'm not sure if I'm doing the same thing you are. Check this out. If the JSON file is not present, you get the error. I'm demonstrating this by creating a new empty directory and then cd'ing into it, then running the curl command. This means that the JSON file (dataverse-complete.json for me) isn't there:

mkdir /tmp/new-empty-directory

/tmp/new-empty-directory

curl -H "X-Dataverse-key: 0cb4f4b4-4657-447d-8036-84b46b32fd1f" -H "Content-type:application/json" -X POST -d @dataverse-complete.json https://demo.dataverse.org/api/dataverses/harvard

Warning: Couldn't read data from file "dataverse-complete.json", this makes an
Warning: empty POST.

{"status":"ERROR","message":"Error parsing Json: Invalid token=EOF at (line no=1, column no=0, offset=-1). Expected tokens are: [CURLYOPEN, SQUAREOPEN]"}

Does this make sense? Is it possible that the file isn't present or that the combination of curl flags is causing the file not to be read? Rather that @dataverse.create.json it's possible to use the fully qualified path to the file like @/tmp/dataverse.create.json or whatever.

[ghost]

Hi Phil,
The json file is indeed present in its folder and its valid json. And I used the exact same command and file, with a different url and api key of course, to create "Sposito Research" dataverses on both our production installation and on demo.dataverse.org. Just to be sure I deleted and recreated the demo.dataverse.org version a couple of times, using both relative and absolute paths, just now. This was the most recent response:

{"status":"OK","data":{"id":26395,"alias":"sposito","name":"Sposito Research","a
ffiliation":"Austrian Social Science Data Archive","dataverseContacts":[],"permi
ssionRoot":true,"description":"Putting the social back into social sciences data
","dataverseType":"UNCATEGORIZED","ownerId":1,"creationDate":"2017-12-21T13:51:4
8Z","creator":{"id":988,"identifier":"@fsposito","displayName":"Frank Andreas Sp
osito","firstName":"Frank Andreas","lastName":"Sposito","email":"frank.sposito@u
nivie.ac.at","superuser":false,"affiliation":"University of Vienna","persistentU
serId":"fsposito","emailLastConfirmed":"2017-12-21T08:10:34Z","createdTime":"201
7-12-21T08:10:18Z","lastLoginTime":"2017-12-21T13:43:00Z","lastApiUseTime":"2017
-12-21T13:51:48Z","authenticationProviderId":"builtin"}}}

Also tried the fully qualified path on our development server just to be sure but it did not work.

We have a local configuration problem for sure, just don't know what it could be. Our sys admin is working on it. Our development installation is a snapshot of our production installation, minus ssl, so maybe something was lost in translation?

[pdurbin]

@FASposito I'm sorry I missing your comment above. Judging from the date, I was probably out the door to visit my family for Christmas. Are you still having this problem? Any update? Thanks!

[pdurbin]

Related:

• Docs: Sample create dataverse json file fails with parsing error. #5198

[pdurbin]

@pameyer @FASposito does pull request #5870 help?

[pdurbin]

There was a recent thread about the same problem: https://groups.google.com/d/msg/dataverse-dev/dZd-Pz-2sxY/NYtPNwTGAwAJ

[cmbz]

To focus on the most important features and bugs, we are closing issues created before 2020 (version 5.0) that are not new feature requests with the label 'Type: Feature'.

If you created this issue and you feel the team should revisit this decision, please reopen the issue and leave a comment.