RFC: BcNuGet package format

[freddydk]

BcNuGet

We want to create a standardized way of exchanging Business Central apps across different DevOps solutions. How can partners expose their apps or runtime packages as NuGet packages, which easily can be consumed by other partners.
Presentation from last weeks meeting: How do we solve the issue of having dependencies to AppSource apps.pdf

The description should also allow Microsoft to host a NuGet Server, in which we could host all AppSource Apps if we are allowed to do so by partners.

This discussion is an attempt to define the BcNuGet package format and also describe how the implementation of this should work in DevOps solutions.

Assumptions

Every BcNuGet will contain one app and a list of dependencies to other BcNuGet packages (apps) from that app.
A BcNuGet package can contain the app or runtime packages
If a BcNuGet package contains runtime packages, it must contain all necessary runtime packages (for all compatible versions) - see more under runtime packages
Test Apps can also be published as seperate BcNuGet packages if partners want you to run their test apps in your DevOps pipelines

File Format

<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2010/07/nuspec.xsd">
    <metadata>
        <id>$id</id>
        <version>$version</version>
        <title>$brief</title>
        <description>$description</description>
        <authors>$publisher</authors>
        <repository type="git" url="$repository" branch="$branch" commit="$sha" />
        <licenseUrl>$EULA</licenseUrl>
        <projectUrl>$projectUrl</projectUrl>
        <icon>$icon</icon>
        <readme>$readme</readme>
        <releaseNotes>$releaseNotes</releaseNotes>
        <businessCentral privacyStatement="$privacyStatement" help="$help" url="$url" runtime="runtime" platform="$platform" application="$application" ... >
            <idRanges>
                <idRange from="$from" to="$to" />
            </idRanges>
            <resourceExposurePolicy allowDebugging="$allowDebugging" allowDownloadingSource="$allowDownloadingSource" ...  />
        </businessCentral>
        <dependencies>
            <dependency id="$id" version="$version" />
        </dependencies>
    </metadata>
    <files>
        <file src="$publisher_$name_$version.app" target="$publisher_$name_$version.app" />
        <file src="$platform\$publisher_$name_$version.runtime.app" target="$platform\$publisher_$name_$version.runtime.app" />
        <file src="$icon" target="$icon" />
        <file src="$readme" target="$readme" />
    </files>
</package>

In general, fields that are optional in app.json are also optional in the BcNuGet spec (like icon). Description however is a requirement in nuspec and if the app.json file doesn't contain a description, we will use the name instead.

The BusinessCentral tag is the only "custom" nuspec field, which then contains all Business Central specific properties.
Attributes on the Business Central tag are the simple app.json properties (mandatory properties are mandatory)
resourceExposurePolicy tag contains a copy of the resourceExposurePolicy tag from app.json as xml
idRanges tag contains a copy of the idRanges from app.json as xml

Where
$id, $version, $brief, $description, $publisher and $EULA are the corresponding values from app.json
$rpid is the runtime package identifier for that specific runtime package
$icon is the path and name of the icon within the package (optional)
$readme is the path and name of the readme.md file within the package(optional)
$projectUrl is a URL to the project, which hosts this project
$repository, $branch, $sha is the git repository, branch and sha from where the package was created
$releaseNotes markdown content of release notes for this version (max. 35000 characters)
$from, $to are object range from..to fields from app.json

Platform

• Describes which platform the runtime package inside the folder was created for
• Microsoft will provide information/code, which can identify which different platforms we need to create runtime packages for
• Microsoft will provide information/code, to identify which runtime package to use for a given platform version

Dependency handling

Default behavior of NuGet dependency resolution can be found here: https://learn.microsoft.com/en-us/nuget/concepts/dependency-resolution
Business Central doesn't really have a dependency resolution mechanism, but works after the rather simplified assumption that if you set a dependency to version 2.3.0.0 - then you are compatible with 2.3.0.0 and all subsequent versions (also 3.0.0.0 and 4.0.0.0 etc)
So, what is the right approach?

Let's have a look at this scenario?

What version of App B and App C do you get if you install App A, 1.0.0.0?
My suggestion would be to install App B version 1.0.1.0 - which is the latest build of version 1.0, which you have a dependency on.
Even if you are running BC version 20, as App B version 1.0.1.0 also should be compatible with 20.x

What version of App B and App C do you get if you install App A, 2.0.0.0?
My suggestion would be to install App B and App C version 2.0.0.0. When trying to install on 19.0, it would fail before starting the installation. If you already have App C 1.0.0.0 installed in the environment, it is upgraded to 2.0.0.0

What version of App B and App C do you get if you install App A, 2.1.0.0?
My suggestion would be to install App B version 2.1.0.0 and App C version 2.0.0.0. When trying to install on 19.0, it would fail before starting the installation. If you already have App C 1.0.0.0 installed in the environment, it is upgraded to 2.0.0.0

[azwierzchowskitnp]

Introduction

In my opinion, we should make sure that NuGet works not only with DevOps but also covers other cases. I would prefer situation where NuGet server is the only source of apps inside a company. It means that people should also be able to go there, find and download packages. I understand that we are talking about DevOps here, but preparing for other scenarios will make this solution better and allow more people to use it.

We want to use DevOps to create NuGet packages and then allow developers/consultants to access to that server to search and download apps. We are also going to use our own version of "Extension Management" page to install and update apps directly from NuGet servers with automatic dependency resolution inside WebClient. It will make creation of demo/dev/test environments easier and help us with customers servers maintenance (if there is no DevOps "installation" task).

I was also showing an idea of NuGet version of "Download Symbols" command on my Directions session which could help developers to quickly switch between latest and lowest version of dependencies to check if their solution compiles. That idea could also be used in DevOps to quickly run that check before we create a docker (fail fast approach).

Let's make it compatible with the specification

I would prefer if we make sure that our packages are 100% compatible with NuGet format and all NuGet tools/servers work without any problems. Unfortunately, it is not the case with the idea of adding businessCentral node to the .nuspec file. When I tried to create package using NuGet CLI (nuget.exe), it failed with a message that businessCentral is an unexpected node in the file.

NuSpec specification can be found here:
https://learn.microsoft.com/en-us/nuget/reference/nuspec

Also, I don't see any benefit from keeping that information in .nuspec. Fortunately, it is ignored by AzureDevOps and GitHub NuGet repositories, so you can upload these incompatible packages there, but we should not assume that it will work with other hosting providers. NuGet servers have very nice set of APIs that allows to search and see packages properties (i.e., we can build list of dependencies to download without downloading any package), but because businessCentral node is not part of the specification, it is not included in these APIs. The only solution at this moment is to use Tags to add additional properties to the packages.

Proposed solution

Use NavxManifest.xml or app.json instead of businessCentral node

Instead of including businessCentral node in .nuspec file, we can include NavxManifest.xml from .app package or app.json in the NuGet package.

<file src="NavXManifest.xml" target="NavXManifest.xml" />

or

<file src="app.json" target="app.json" />

Use tags for important BC App properties

We can use <tags></tags> node that contains space separated list of tags for additional, BC-specific attributes, that we would like to check before we download a package. Obviously we will have to agree what tags we would like to use there.

Here is an example of valid .nuspec file that uses these tags:

• bcAppId:applicationId - id property value form app.json
• bcAppPackageType:packageType - SingleApp for one app in the package, but we can think about other scenarios like MultipleApps, ConfigPackages
• bcAppRuntime:runtime - runtime property value from app.json
• bcAppApplication:application - application property value from app.json
• bcAppPlatform:platform - platform property value from app.json

<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2013/05/nuspec.xsd" xmlns:bcns="http://schemas.microsoft.com/packaging/2022/11/bcspec.xsd">
  <metadata>
    <id>fad4be71-bf29-4ff6-e4e1-18d3968ff689</id>
    <version>1.10.0</version>
    <title>Customer extension</title>
    <authors>Partner</authors>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <description>Partner - Customer extension</description>
    <repository type="git" url="https://github.com/anzwdev/ALNuGetExperiments">
    </repository>
    <dependencies>
      <dependency id="bae47ec8-63aa-48f6-a4e1-90d3f68b66e1" version="1.0.0" />
      <dependency id="846de607-f93a-433d-940e-130a1830f422" version="1.0.0" />
    </dependencies>
    <tags>bcAppId:fad4be71-bf29-4ff6-e4e1-18d3968ff689 bcAppPackageType:SingleApp bcAppRuntime=8.0 bcAppPlatform:18.0.0.0 bcAppApplication:18.0.0.0</tags>
  </metadata>
  <files>
    <file src="Partner_Customer extension_1.10.0.0.app" target="Partner_Customer extension_1.10.0.0.app" />
    <file src="NavXManifest.xml" target="NavXManifest.xml" />
  </files>
</package>

And here is the result of a call to one of APIs on GitHub NuGet server. As you can see, we can find our tags there and we can use them to decide if we should download the package (i.e., by looking at the 'bcAppRuntime' or 'bcAppPlatform' tags)

{
    "@id": "https://nuget.pkg.github.com/AnZwDev/fad4be71-bf29-4ff6-e4e1-18d3968ff689/1.10.0.json",
    "packageContent": "https://nuget.pkg.github.com/AnZwDev/download/fad4be71-bf29-4ff6-e4e1-18d3968ff689/1.10.0/fad4be71-bf29-4ff6-e4e1-18d3968ff689.1.10.0.nupkg",
    "catalogEntry": {
        "@id": "https://nuget.pkg.github.com/AnZwDev/fad4be71-bf29-4ff6-e4e1-18d3968ff689/1.10.0.json",
        "authors": "Partner",
        "copyright": "",
        "dependencyGroups": [
            {
                "@id": "https://nuget.pkg.github.com/AnZwDev/fad4be71-bf29-4ff6-e4e1-18d3968ff689/1.10.0.json#dependencygroup",
                "@type": "PackageDependencyGroup",
                "dependencies": [
                    {
                        "@id": "https://nuget.pkg.github.com/AnZwDev/fad4be71-bf29-4ff6-e4e1-18d3968ff689/1.10.0.json#dependencygroup/bae47ec8-63aa-48f6-a4e1-90d3f68b66e1",
                        "@type": "PackageDependency",
                        "id": "bae47ec8-63aa-48f6-a4e1-90d3f68b66e1",
                        "range": "[1.0.0, )"
                    },
                    {
                        "@id": "https://nuget.pkg.github.com/AnZwDev/fad4be71-bf29-4ff6-e4e1-18d3968ff689/1.10.0.json#dependencygroup/846de607-f93a-433d-940e-130a1830f422",
                        "@type": "PackageDependency",
                        "id": "846de607-f93a-433d-940e-130a1830f422",
                        "range": "[1.0.0, )"
                    }
                ]
            }
        ],
        "description": "Partner - Customer extension",
        "iconUrl": "",
        "id": "fad4be71-bf29-4ff6-e4e1-18d3968ff689",
        "isPrerelease": false,
        "language": "",
        "licenseUrl": "",
        "packageContent": "https://nuget.pkg.github.com/AnZwDev/download/fad4be71-bf29-4ff6-e4e1-18d3968ff689/1.10.0/fad4be71-bf29-4ff6-e4e1-18d3968ff689.1.10.0.nupkg",
        "projectUrl": "",
        "requireLicenseAcceptance": false,
        "summary": "",
        "tags": "bcAppId:fad4be71-bf29-4ff6-e4e1-18d3968ff689 bcAppPackageType:SingleApp bcAppRuntime=8.0 bcAppPlatform:18.0.0.0 bcAppApplication:18.0.0.0",
        "version": "1.10.0"
    }
}

Readability

If you take a look at .nuspec and api call results and think how NuGet package files are named, you will probably see that we might have a readability problem there. It won't be an issue for automatic DevOps scripts, but it will be a problem for people doing any manual work (investigation, search, downloads, manual installation). Package files names are created from package id and version number. If we use just BC App Id for the NuGet package ID, we will have a lot of GUIDs without any explanation what they mean. If you download a few packages, you will see GUIDs and versions on the list of your files, if you open .nuspec files, you will see a list of GUIDs on the list of dependencies without really knowing what these dependencies are.

Maybe we should use a bit more for the package id? What if we use AppId + Publisher + Name? Maybe we could remove spaces from Publisher and replace spaces with dots in the Name to make it look like any other, non-al package. But be aware that NuGet server might have a limit on the Id length - NuGet.org has 128 characters limit, so I would truncate the id to 128 characters.

If we do it this way, the .nuspec from example above could look a bit more readable:

<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2013/05/nuspec.xsd" xmlns:bcns="http://schemas.microsoft.com/packaging/2022/11/bcspec.xsd">
  <metadata>
    <id>fad4be71-bf29-4ff6-e4e1-18d3968ff689.Patner.Customer.extension</id>
    <version>1.10.0</version>
    <title>Customer extension</title>
    <authors>Partner</authors>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <description>Partner - Customer extension</description>
    <repository type="git" url="https://github.com/anzwdev/ALNuGetExperiments">
    </repository>
    <dependencies>
      <dependency id="bae47ec8-63aa-48f6-a4e1-90d3f68b66e1.ISV1.Warehouse.Integration" version="1.0.0" />
      <dependency id="846de607-f93a-433d-940e-130a1830f422.ISV2.Web.Integration" version="1.0.0" />
    </dependencies>
    <tags>bcAppId:fad4be71-bf29-4ff6-e4e1-18d3968ff689 bcAppPackageType:SingleApp bcAppRuntime=8.0 bcAppPlatform:18.0.0.0 bcAppApplication:18.0.0.0</tags>
  </metadata>
  <files>
    <file src="Partner_Customer extension_1.10.0.0.app" target="Partner_Customer extension_1.10.0.0.app" />
    <file src="NavXManifest.xml" target="NavXManifest.xml" />
  </files>
</package>

I know that it breaks the current BC Apps dependency rules where we can change publisher and name at any time but makes maintenance much easier.

Pre-release versions

Maybe we coud use pre-release version suffix to mark some of our packages as pre-release? Nuget tools (i.e., nuget.exe) ignore these versions until you specify that you want to include them in search/installation. They are created by adding "-" and a text to the version number (i.e., 1.10.2.3-beta, 1.10.2.7-development, 2.6.3.5-customertest). This way we could reuse existing NuGet feeds to host these packages and define different "pre-release suffixes" in the pipelines. It will also help us with preventing a situation when a package is installed in an incorrect database.

[jonaswre]

@freddydk I agree with you adding new feeds should not be automated. this is the only manual work besides adding the dependency to the app.json. But I think for the spec its important how everything is structured.

My dream situation is not even that everything works magically. I think it's not to much effort to require somebody to specify where they want to get their apps from.
I think the most common use case is getting access to unreleased app versions before they hit the app source so that a partner can prepare for changes before they hit the app source.

Jonas came with a suggestion what we could check the signature on the app (certificate), but that would only help with appsource apps - and those will eventually get solved by the Microsoft hosted server anyway - so we still need to solve this for on-premises and other apps.

From a security standpoint we should verify every app we download it doesn't matter what app you download. The reason for this is that a partner could, even by accident publish an app of a another partner in their own feed, which might then be resolved beforehand.

I think to make it mandatory to sign every app you publish into a feed is not outrageous. A self signed certificate could also be used if we don't want the paywall of buying a certificate.

Every partner can keep a list of which publisher has which trusted certificate signature. It then wouldn't matter where you get your apps from because you could verify that each app was created by the intended publisher.

This is how Microsoft Updates must also work if you download them from untrusted system, for example your colleges computer.

I think using this mechanism the only assumption I make is I need to trust the content of the partners app which I need to do in every case. But I don't need to trust the source where I got it from.

[freddydk]

I think the most common use case is getting access to unreleased app versions before they hit the app source so that a partner can prepare for changes before they hit the app source.

Good and important scenario.
I do think that should eventually also be part of appsource (preview)

The problem with self-signed certificates is that they cannot be revoked. So, I trust a partner who loses his certificate - and now I potentially trust evil people,
And with trusted certificates, which are not self-signed needs to be refreshed every x years - meaning that all partners needs to update a key for the partner - right?

[jonaswre]

Self Signed Certificates can't be revoked that's true. But they could be excluded from the list of trusted certificates. I imagine the list as not trusted by default.

My current thinking is - a partner is maintaining their own list which would indeed mean that every partner needs to update that list every x years.

Solution for this is obviously a trusted registry for these certificate-publisher combinations. Maybe maintained by Microsoft as part of the AppSource?

Another solution could be partner maintained list were changes need to be aknowledged by the user.

[freddydk]

IMO Certificates could definitely be a 2nd line of defense - but I don't think it can be the primary line of defense.
Any solution where partners will actually download potentially evil stuff, just to discover that it isn't trusted won't pass our security review (pretty sure about that).
But to be sure, I will investigate how Microsoft does this with other things.
PowerShell modules, NuGet packages, etc.

[jonaswre]

That's basically the concept of File hash verification.
You download the exe and before execution you verify if what you have downloaded is actually what you wanted to download.

I'm interested to see what Microsoft does in these cases there must be an acceptable solution for this problem.