Renovate does not support latest TOML v1.0.0 spec.

[BradenM]

How are you running Renovate?

Mend Renovate hosted app on github.com

If you're self-hosting Renovate, tell us what version of Renovate you run.

No response

If you're self-hosting Renovate, select which platform you are using.

No response

If you're self-hosting Renovate, tell us what version of the platform you run.

No response

Was this something which used to work for you, and then stopped?

I never saw this working

Describe the bug

Renovate does support the latest TOML v1.0.0 spec (which was released Jan 2021).

This (potentially) impacts any modules that require parsing of TOML files.

I ran into this issue when renovate failed to realize my project's pyproject.toml file due to a parsing exception thrown when it tried to read a heterogeneous array, such as:

[tool.poetry]
## ...
 include = [ 
   "README.md", 
   { path = "tests", format = "sdist" } 
 ] 

Support for this was added to toml in 1.0.0-rc.1 / 2020-04-01.

Poking around a bit, it appears renovate has a pinned dependency on @iarna/toml @ v2.2.5 which only supports TOML v0.5.0 (2018-07-26).

I'll also add that the latest published version of @iarna/toml only supports TOML v1.0.0-rc.1. It does not look as if the package owner has intentions to update the package based on the repository activity.

I have created a minimal reproduction that you can find at BradenM/renovate-toml-v1-repro.

It contains additional details as well as an actions workflow that runs renovate to reproduce the issue.

Relevant debug logs

Logs

DEBUG: Matched 1 file(s) for manager poetry: pyproject.toml (repository=bradenm/renovate-toml-v1-repro)
DEBUG: Error parsing pyproject.toml file (repository=bradenm/renovate-toml-v1-repro)
       "err": {
         "name": "TomlError",
         "fromTOML": true,
         "wrapped": null,
         "line": 9,
         "col": 38,
         "pos": 271,
         "message": "Inline lists must be a single type, not a mix of string and inline-table at row 10, col 39, pos 271:\n 9:   \"README.md\",\n10>   { path = \"tests\", format = \"sdist\" }\n                                          ^\n11: ]\n\n",
         "stack": "TomlError: Inline lists must be a single type, not a mix of string and inline-table at row 10, col 39, pos 271:\n 9:   \"README.md\",\n10>   { path = \"tests\", format = \"sdist\" }\n                                          ^\n11: ]\n\n\n    at TOMLParser.recordInlineListValue (/usr/src/app/node_modules/@iarna/toml/lib/toml-parser.js:1305:28)\n    at TOMLParser.runOne (/usr/src/app/node_modules/@iarna/toml/lib/parser.js:64:30)\n    at TOMLParser.returnNow (/usr/src/app/node_modules/@iarna/toml/lib/parser.js:107:17)\n    at TOMLParser.recordValue (/usr/src/app/node_modules/@iarna/toml/lib/toml-parser.js:523:19)\n    at TOMLParser.runOne (/usr/src/app/node_modules/@iarna/toml/lib/parser.js:64:30)\n    at TOMLParser.parse (/usr/src/app/node_modules/@iarna/toml/lib/parser.js:45:22)\n    at parseString (/usr/src/app/node_modules/@iarna/toml/parse-string.js:13:12)\n    at Object.extractPackageFile (/usr/src/app/node_modules/renovate/lib/modules/manager/poetry/extract.ts:130:26)\n    at extractPackageFile (/usr/src/app/node_modules/renovate/lib/modules/manager/index.ts:70:9)\n    at getManagerPackageFiles (/usr/src/app/node_modules/renovate/lib/workers/repository/extract/manager-files.ts:52:43)\n    at /usr/src/app/node_modules/renovate/lib/workers/repository/extract/index.ts:47:28\n    at async Promise.all (index 2)\n    at extractAllDependencies (/usr/src/app/node_modules/renovate/lib/workers/repository/extract/index.ts:45:26)\n    at extract (/usr/src/app/node_modules/renovate/lib/workers/repository/process/extract-update.ts:100:20)\n    at extractDependencies (/usr/src/app/node_modules/renovate/lib/workers/repository/process/index.ts:110:26)\n    at Object.renovateRepository (/usr/src/app/node_modules/renovate/lib/workers/repository/index.ts:48:52)\n    at attributes.repository (/usr/src/app/node_modules/renovate/lib/workers/global/index.ts:173:11)\n    at Object.start (/usr/src/app/node_modules/renovate/lib/workers/global/index.ts:158:7)\n    at /usr/src/app/node_modules/renovate/lib/renovate.ts:17:22"
       }

Have you created a minimal reproduction repository?

I have linked to a minimal reproduction repository in the bug description

[rarkins]

Did you find any alternative TOML parsing libraries for Node during your research?

[BradenM]

@rarkins
I am still looking around, but unfortunately nothing yet.

I am considering forking @iarna/toml and gauging what it would take to implement support for v1.

Alternatively, I may look deeper into your earlier suggestion #7556 (comment).

[Churro]

What about using https://www.npmjs.com/package/@ltd/j-toml instead? It's listed as v1.0.0-compliant here.

I just gave it a quick try with renovate: seems it would work as a drop-in replacement for pipenv, poetry, cargo, etc.
Only with Gradle version catalogs two lines would need to be modified (not the library's fault though).

[viceice]

• blocked by Please support toml spec 1.0 iarna/iarna-toml#48
• NPM does not seem up-to-date iarna/iarna-toml#31

[viceice]

What about using https://www.npmjs.com/package/@ltd/j-toml instead? It's listed as v1.0.0-compliant here.

I just gave it a quick try with renovate: seems it would work as a drop-in replacement for pipenv, poetry, cargo, etc. Only with Gradle version catalogs two lines would need to be modified (not the library's fault though).

that package seems not to be very popular and it seems to only have one maintainer. 😕

[rarkins]

Arguments against: it has 36k weekly downloads, which is not a lot especially compared to 3.9m of the package we currently have.

Arguments for: if it's a drop-in replacement, then we could always drop it out later if we need?

[rarkins]

You know what? Renovate only has 84k weekly downloads:

But then I looked and most of its downloads come from this: https://www.npmjs.com/package/feed-me-up-scotty

Which I don't think is exactly "robust" testing. If it's robustly tested then it's not via other open source projects.

[viceice]

renovate ist mostly used by our official docker images, which uses caching. so that's probably the cause for low weekly downloads. maybe many users now also use caching so that hides a lot more downloads

[Churro]

that package seems not to be very popular and it seems to only have one maintainer. 😕

Good point. It also has close to 0% test coverage 👎

[rarkins]

Strange that the JS ecosystem doesn't have a viable TOML v1 parser available! I wonder if there's much work to progress the @iarna/toml parser?

[viceice]

reopen, as it's not fixed

[BradenM]

reopen, as it's not fixed

@viceice Thank you. Should have just ref'ed it in the PR instead of allowing it to auto-close.

Strange that the JS ecosystem doesn't have a viable TOML v1 parser available! I wonder if there's much work to progress the @iarna/toml parser?

@rarkins Agreed. I still have intentions to explore what remains to be implemented in @iarna/toml to bring it up to full v1 spec compliance. Just looking through the commit history of the latest branch, the v3 release seems to be >= 90% of the way there already, so I suspect it would not require a significant amount of work. Just a matter of finding the time 😃 .

[epage]

A possible alternative is to instead get a wasm build of a compliant parser. I figure the ecosystems with likely the best toml parsers are (1) Go because the TOML compliance suite uses its parser, (2) Python because of pyproject.toml, and (3) Rust because cargo uses it for nearly everything. Of those, I'm assuming Rust is the best candidate for wrapping for JS. I maintain toml_edit / toml for Rust but have no experience with the JS ecosystem so I can't help much on that side.

[renovate-release]

🎉 This issue has been resolved in version 37.48.2 🎉

The release is available on:

• GitHub release
• 37.48.2

Your semantic-release bot 📦🚀