Allow upload of source code (and release/reviewer notes) using the API

[ElectronicManuel]

When uploading an addon using the API (https://addons-server.readthedocs.io/en/latest/topics/api/signing.html) there is no way to upload the source files automatically.

Steps:

1. push new version of the addon to repo
2. automated deployment deploys version to the mozilla store using the api
3. I navigate to the developer hub and upload the sources manually

It would be nice to have an api for uploading the source files in order to reduce manual intervention.

[stoically]

It might also be nice to allow attaching release- and reviewer-notes when uploading, so publishing could be completely automated.

This blocks mozilla/web-ext#804 & mozilla/web-ext#1624.

[IgnusG]

Any update on this issue? Just curios since it has been a while

[nhedger]

I'd like to help to implement this feature but I am unfamiliar with the codebase. Would anyone care to guide me through?

[AlexandraMoga]

@eviljeff I've verified the new API properties on -dev and these are my results. Please have a look at the source code section in particular because I feel the documentation is not very clear about it or I'm not understanding it fully.

Release Notes:

• release notes can be set at add-on submission time by adding a "version":{"release_notes":{"lang":"string"}} field in the JSON request data
• release notes can be added when a new version is submitted for an existing add-on and can also be edited by sending a version PATCH request

Source code Upload:

• source code can't be uploaded at add-on creation time (i.e. when we post the JSON request data for the first add-on version); if I want to add source code to the first version, I must send a PATCH request with the source file after the version has been created ❓
• source code can be uploaded when a new version is created; I can provide the source file and the upload uuid as form parameters in the version POST request form data; the documentation says that other fields can be set at this stage as well, such as compatibility or other fields from version creation, but so far I wasn't able to send them; for example, for "compatibility": ["firefox"] I'm getting a "compatibility": ["Invalid value"] error ❓
• version source code files can be updated through a version PATCH request ✔️
• only supported archives (.zip, .tar.gz, .tgz, .tar.bz2) can be uploaded ✔️
• I was able to upload larger files as well, but at >60Mb files I've run into a 502 error, so the same issue as 504 error when trying to upload the source code of a new version #1846

As a side note, there is a small error in the API docs:

I think this field was reversed and should be source: string|null.

[eviljeff]

@eviljeff I've verified the new API properties on -dev and these are my results. Please have a look at the source code section in particular because I feel the documentation is not very clear about it or I'm not understanding it fully.

Release Notes:

* release notes can be set at add-on submission time by adding a `"version":{"release_notes":{"lang":"string"}}` field in the JSON request data

* release notes can be added when a new version is submitted for an existing add-on and can also be edited by sending a version PATCH request

Source code Upload:

* source code can't be uploaded at add-on creation time (i.e. when we post the JSON request data for the first add-on version); if I want to add source code to the first version, I must send a [PATCH request](https://addons-server.readthedocs.io/en/latest/topics/api/addons.html#patch--api-v5-addons-addon-(int-addon_id%7Cstring-addon_slug%7Cstring-addon_guid)-versions-(int-id)-) with the source file after the version has been created ❓

Interesting. I can't see any particular reason why it wouldn't work, but I didn't add tests for it. Can you file a follow-up issue?

* source code can be uploaded when a new version is created; I can provide the `source` file and the `upload` uuid as form parameters in the [version POST request form data](https://addons-server.readthedocs.io/en/latest/topics/api/addons.html#post--api-v5-addons-addon-(int-addon_id%7Cstring-addon_slug%7Cstring-addon_guid)-versions-); the documentation says that other fields can be set at this stage as well, such as `compatibility` or other fields from version creation, but so far I wasn't able to send them; for example, for `"compatibility": ["firefox"]` I'm getting a  `"compatibility": ["Invalid value"]` error ❓

How are you specifying the values with a formdata formatted request? (I'm a little unclear myself on how complex data structures that are easy in JSON format would be specified as formdata)

As a side note, there is a small error in the API docs: I think this field was reversed and should be source: string|null.

whoops, yeah
-- edit: it's actually just an extra : that did it

[AlexandraMoga]

* source code can't be uploaded at add-on creation time (i.e. when we post the JSON request data for the first add-on version); if I want to add source code to the first version, I must send a [PATCH request](https://addons-server.readthedocs.io/en/latest/topics/api/addons.html#patch--api-v5-addons-addon-(int-addon_id%7Cstring-addon_slug%7Cstring-addon_guid)-versions-(int-id)-) with the source file after the version has been created ❓

Interesting. I can't see any particular reason why it wouldn't work, but I didn't add tests for it. Can you file a follow-up issue?

* source code can be uploaded when a new version is created; I can provide the `source` file and the `upload` uuid as form parameters in the [version POST request form data](https://addons-server.readthedocs.io/en/latest/topics/api/addons.html#post--api-v5-addons-addon-(int-addon_id%7Cstring-addon_slug%7Cstring-addon_guid)-versions-); the documentation says that other fields can be set at this stage as well, such as `compatibility` or other fields from version creation, but so far I wasn't able to send them; for example, for `"compatibility": ["firefox"]` I'm getting a  `"compatibility": ["Invalid value"]` error ❓

How are you specifying the values with a formdata formatted request? (I'm a little unclear myself on how complex data structures that are easy in JSON format would be specified as formdata)

@eviljeff I think this all boils down to your last question, because I haven't found the best way to send JSON with a multipart/form-data request yet. I'm using Postman, which has an integrated feature that allows you to specify the content type for each key:value sent, but I'm not sure how to use this for our case, since there is no key to encompass the whole JSON data we'd like to send when creating a new add-on.

[eviljeff]

@AlexandraMoga I've played around locally in tests and I'm not there is a way to submit complex (lists or objects with nested values) without using JSON.
(If there is, for version create then the key would be the object key, e.g. release_notes and I guess the value would be {"en-US": "release notes!"}, with the content type of application/json, but I'm not confident that'll work - I couldn't get it to work in test cases and my postman foo is weak)

Currently, I can't think of any way that source could be submitted as a nested value - i.e. during addon create, so I think that's an unsupported use-case. If there's no way to specify complex values at the same time as specifying a source file either (i.e. for version create/update) I'll update the api docs to make that clear.

[AlexandraMoga]

(If there is, for version create then the key would be the object key, e.g. release_notes and I guess the value would be {"en-US": "release notes!"}, with the content type of application/json, but I'm not confident that'll work - I couldn't get it to work in test cases and my postman foo is weak)

For the case of new version creation, I've also experimented with smaller bits of data and they weren't accepted either; In the case of release notes, the error message is the following:

{
    "release_notes": [
        "You must provide an object of {lang-code:value}."
    ]
}

So basically, if I want to attach source code, I'm limited to sending only the source file and the upload:uuid:

I'll update the api docs to make that clear.

Does this mean that you will update the Version sources docs to no longer include the compatibility bit from the required Form parameters section?

[eviljeff]

So basically, if I want to attach source code, I'm limited to sending only the source file and the upload:uuid:

I'll update the api docs to make that clear.

Does this mean that you will update the Version sources docs to no longer include the compatibility bit from the required Form parameters section?

yeah. Only simple values can be sent in a multipart form, so upload and license (because it's a slug). And sending the source during an Add-on create doesn't work.

[karlicoss]

Any chance we can get this reopened? I went through the docs, and doesn't look like source code or reviewer notes are supported?

[diox]

Both are supported in the API and documented (https://mozilla.github.io/addons-server/topics/api/addons.html#version-create). That being said you did well to file an issue to improve our docs around it.

[birtles]

@karlicoss There's a GitHub action for uploading to AMO with example code at: https://github.com/birchill/amo-upload/blob/ab657f9e64908ffda49bd3c4bfc1e8bc9fe6d243/index.js#L154-L162

[eviljeff]

@karlicoss I've not investigated but a common gotcha is declaring version level properties (metadata under version-create rather than addon-create) at the top add-on level instead. i.e. it should be this:

{
  "version": {
    "approval_notes": "foo"
  }
}

[karlicoss]

Thanks, putting it under version worked!
Unfortunately, I am having a different issue -- JSON.parse doesn't support multiline strings (e.g. like this)

{
  "version": {
    "approval_notes": "
  line 1
  line 2
    "
  }
}

, so I'm getting an error here https://github.com/mozilla/web-ext/blob/3e4faa26d4f8ad95cbdf5da75ca9b88dbcc20d9d/src/cmd/sign.js#L96

I could inline \n in the string directly, but then editing approval notes would be super painful. I suppose I could generate the json file from code to prevent this, but wondering if there is something I'm missing.

(I'd be very happy with simply putting a link to the instructions in the repository, but in the past reviewers demanded pasting detailed step by step addon building instructions in the reviewer notes)

[willdurand]

I think you'll have to inject a stringified representation of your approval notes.