Override the newValue in a customManager?

[m-chebli]

How are you running Renovate?

Self-hosted Renovate

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

GitLab, with Renovate runner v39.86

Please tell us more about your question or problem

I'm trying to setup Renovate for my python project. The dependencies are stored in a pyproject.toml, and look like this:

[build-system]
requires = ["setuptools~=69.0", "wheel~=0.42"]
build-backend = "setuptools.build_meta"

[project]
dependencies = [
  "private-depA~=3.1",
  "private-depB[extra1, extra2]~=2.4",
  "pymongo~=4.6",
]

For all the dependencies, even the custom ones, the simple pep621 rule works fine. It has been generated automatically and tweaked to point toward my custom registry. But, for the dependency with extras, it gets the following dependency error:

{
   "packageName": "private-depB",
   "depName": "private-depB",
   "datasource": "pypi",
   "depType": "project.dependencies",
   "currentValue": "[extra1,extra2]~=2.4",
   "updates": [],
   "versioning": "pep440",
   "warnings": [],
   "skipReason": "invalid-value"
 },

I infer it's because the currentValue includes the extras (as mentioned here: #23343). I did not find any way to either accept extras or override the currentValue here, so I disabled the dependency in pep621, and added a customManager rule instead:

{
      "customType": "regex",
      "fileMatch": ["(^|/)python/pyproject\\.toml$"],
      "matchStrings": [
        "(?<depName>private-depB)\\[(?<extras>[a-zA-Z0-9_, ]+)\\]\\s*(?<currentValue>~=[0-9.]+)"
      ],
      "datasourceTemplate": "pypi",
      "versioningTemplate": "pep440",
      "depTypeTemplate": "build-system.requires",
      "registryUrlTemplate": "myPrivateRegistry"
    }

It kinda works, with the following dependency created:

{
             "deps": [
               {
                 "depName": "private-depB",
                 "currentValue": "~=2.4",
                 "datasource": "pypi",
                 "versioning": "poetry",
                 "registryUrls": [
                   "myCustomRegistry"
                 ],
                 "depType": "build-system.requires",
                 "replaceString": "private-depB[extra1,extra2]~=2.4",
                 "updates": [
                   {
                     "bucket": "non-major",
                     "newVersion": "2.5.0",
                     "newValue": "2.5.0",
                     "newMajor": 2,
                     "newMinor": 5,
                     "newPatch": 0,
                     "updateType": "minor",
                     "branchName": "chore/private-depB-2.x"
                   }
                 ],
                 "packageName": "private-depB",
                 "warnings": [],
                 "registryUrl": "myPrivateRegistry",
                 "currentVersion": "2.4.0",
                 "isSingleVersion": true
               }
             ],
             "matchStrings": [
               "(?<depName>private-depB)\\[(?<extras>[a-zA-Z0-9_, ]+)\\]\\s*(?<currentValue>~=[0-9.]+)"
             ],
             "datasourceTemplate": "pypi",
             "versioningTemplate": "poetry",
             "registryUrlTemplate": "myCustomRegistry",
             "depTypeTemplate": "build-system.requires",
             "packageFile": "python/pyproject.toml"
           }

The new version is detected, and a new merge request is created! But the version in my pyproject.toml is now ~=2.5.0 instead of ~=2.5, which does not mean the same thing..

I didn't find any way to override the newValue, or truncate it. I tried to sed my way into postUpgrades with "sed -i 's/~=\\([0-9]\\+\\.[0-9]\\+\\)\\.[0-9]/~=\\1/g' python/pyproject.toml", but got the following error:

INFO: Repository has invalid config (repository=my-repo)
       "error": {
         "validationMessage": "invalid escape sequence: \\1",
         "validationSource": "sed -i 's/~=\\([0-9]\\+\\.[0-9]\\+\\)\\.[0-9]/~=\\1/g' python/pyproject.toml",
         "validationError": "Invalid regular expression (re2): sed -i 's/~=\\([0-9]\\+\\.[0-9]\\+\\)\\.[0-9]/~=\\1/g' python/pyproject.toml",
         "message": "config-validation",
         "stack": "Error: config-validation\n    at regEx (/usr/local/renovate/lib/util/regex.ts:62:19)\n    at /usr/local/renovate/lib/workers/repository/update/branch/execute-post-upgrade-commands.ts:70:18\n    at Array.some (<anonymous>)\n    at postUpgradeCommandsExecutor (/usr/local/renovate/lib/workers/repository/update/branch/execute-post-upgrade-commands.ts:69:39)\n    at processBranch (/usr/local/renovate/lib/workers/repository/update/branch/index.ts:519:9)\n    at writeUpdates (/usr/local/renovate/lib/workers/repository/process/write.ts:166:17)\n    at update (/usr/local/renovate/lib/workers/repository/process/extract-update.ts:233:11)\n    at Object.renovateRepository (/usr/local/renovate/lib/workers/repository/index.ts:89:19)\n    at attributes.repository (/usr/local/renovate/lib/workers/global/index.ts:206:11)\n    at start (/usr/local/renovate/lib/workers/global/index.ts:191:7)\n    at /usr/local/renovate/lib/renovate.ts:19:22"
       }

I have tried playing with escape characters, but did not make it. I have considered running the sed command in a gitlab-ci step, but I would like to avoid this as much as possible.

What am I doing wrong?

Regards,
Manfred

Logs (if relevant)

Logs

Replace this text with your logs, between the starting and ending triple backticks

[secustor]

I have not looked into why your sed command is failing, but rather why it has not parsed the string correctly in the first place.

In fact it has been a bug and should be fixed with #33378