Support deploying multiple exes as a single self-contained set #53834
Comments
|
Tagging subscribers to this area: @vitek-karas, @agocke, @VSadov Issue DetailsThis will require both runtime and SDK work to select the appropriate set of shared assemblies.
|
dotnet-issue-labeler
bot
added
the
untriaged
|
@agocke Does this include publishing an executable projet as self-contained which itself references another executable project ?Both project should end up in the publish folder and use the same self-contained framework. |
|
This feature would be incredibly useful for projects that ship multiple small and focused command line utilities in a single release. Bonus points if trimming can be done based on the shared assembly set. |
|
Presupposes the UI a bit by assuming that "project references to exe projects" is the right way to design this, but yeah, that's the general idea. |
|
This would be just splendid to be supported. I've been thinking about designs for this for a while now. One idea i had would be updating the A.csproj <Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net5.0-windows</TargetFramework>
<OutputType>winexe</OutputType>
<UseWPF>true</UseWPF>
<DisableImplicitFrameworkReference>true</DisableImplicitFrameworkReference>
<OutputPath>$(SharedOutputPath)/client</OutputPath>
</PropertyGroup>
<ItemGroup>
<FrameworkReference Include="../runtime/runtime.msbuildproj" />
</ItemGroup>
</Project>B.csproj <Project Sdk="Microsoft.NET.Sdk.Web">
<PropertyGroup>
<TargetFramework>net5.0-windows</TargetFramework>
<DisableImplicitFrameworkReference>true</DisableImplicitFrameworkReference>
<OutputPath>$(SharedOutputPath)/server</OutputPath>
</PropertyGroup>
<ItemGroup>
<FrameworkReference Include="../runtime/runtime.msbuildproj" />
</ItemGroup>
</Project>runtime.msbuildproj <Project Sdk="Microsoft.NET.Sdk.RuntimeCache">
<PropertyGroup>
<TargetFramework>net5.0-windows</TargetFramework>
<OutputPath>$(SharedOutputPath)/runtime</OutputPath>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Newtonsoft.Json" Version="13.0.0" />
</ItemGroup>
</Project>The sdk could then use the FrameworkReference to traverse and query the outputpath of the runtimecache project and tell the respective targets to either generate .deps.json respecting those path (comparable to how it is run during debug when dlls of packagereferences etc. are resolved via path in deps) OR by declaring the outputpath of the frameworkreference project as a probingPath in the deps.json. I know this sounds like a stretch from the original requirement, but it really is in the spirit of the change needed, because theres almost no two individual apps that would generate the exact same sets of dependencies (especially with trimming). If you build everything into a single directory msbuild would have to A) concurrently build in the same directory which is problematic of it self, but theres also the problem of a second app publishing their dependencies in the same directory which might override hard dependencies onto different versions thus generating a whole other range of issues. |
|
@MeikTranel Thanks, this seems very interesting. Fair warning though, no timeline or commitment to this yet. .NET 6 is almost certainly impossible, we have too much work to do for existing scenarios. This could be on the table for .NET 7 though, depending on competing priorities. |
|
Haha sure no worries - I was expecting this 😊 One thing that would really help immensely would be just the ability to specify a probing path as a relative directory in the runtimeconconfig.json - one can already specify it in as an environment variable with an absolute path so with my utterly minimal knowledge of the host I assumed it should be possible. |
|
@MeikTranel I just want to make sure I understand what you mean by "probing path". Is it the path to the runtime install location (specified via |
|
The first one primarily - the latter should already be working with the runtimeconfig.json right? One use case we often use is something like a aspnetcore webapi bundled with a windows service host thats registered via install. Env variables are somewhat awkward in those situations. It almost always requires some sort of wrapping that makes it awkward. It would help tremendously if the runtimeconfig.json parsing or at least small parts of it would happen before the hostfxr is loaded. Basically any mechanism that can be stored permanently within the files built during dotnet build |
|
Thanks @MeikTranel. Personally I don't like parsing One of the ideas was something like:
This would have the downside that it's not possible to modify the path with simple text editor, basically one would have to rebuild the app to do that. The upside is that it's simple and doesn't require too much new code in the apphost. |
|
That seems fine with me. |
|
Very interested in this. We currently try to achieve this, by basically packing the application as NuGet package through a custom SDK: ** Sdk\Tool.props of Company.SDK.Bundle ** This allows us, to later deploy them together by creating a Dummy project using just ** Sdk\Sdk.props of Company.SDK.Bundle ** The bundle itself is then created using a simple csproj file: ** bundle.csproj ** This will also create the This has the benefit, that all common dependencies are published together and the dependency trees are calculated over all of them. We started doing this, because we have a dependency on gRPC, which will raise size pretty quickly due to it's native runtimes included per tool. In a regular deployment, we would have 3 copies of the same files for 90% of the DLLs... |
|
Noob here. I can understand that this issue asks about the But, what is stopping you from publishing your Since I'm fiddling with these systems myself at the moment, clarification would really help me, although it might not help this issue here overmuch, sorry. |
|
@bilbothebaggins One of the problems is how to guarantee dependency version consistency. Simple example:
The separate publish processes will produce C.dll to the output, but each will get a different version. If you copy over the directories you'll end up with one of those versions. Part of the feature should be that this gets resolved at build time, and the dependency is unified across the multiple apps (or you have a way to opt-out and have per-app dependencies). |
|
@vitek-karas Ah ok thanks. Where we hit a similar situation, we already unify all library versions across all projects via CPM and transitive pinning. So I took this for granted. Given unified transitive versions for all dependencies of all cooperating executables, would you expect any other problems when throwing them all in the same folder? |
It should probably work - but I didn't go through all the details and possible implications, so don't take this as an official statement please. One other thing to be mindful of - all the apps should be built with the same target framework, and same publish options (self-contained or not, and so on). Any differences in those could cause potential problems. |
Another would be trimming I guess? Also, are there any plans to work on this in .NET 9? |
The version is the least of the problems tho. We still have to guarantee that a shared dependency folder is being written to from different rid actors. |
|
Very interesting issue. We also have a scenario that would benefit from this, as we have this windows service host that relies on 2 command line tools that we also control. Initially, both tools were being copied to the same folder as the host itself, which created massive issues with dependencies like @vitek-karas described. Recently, we isolated each tool to its own subfolder inside of the host process but this results in quite a bit of duplication as now dependencies that both tools need have to be duplicated. I'd love to be able to natively have a "host" with "tools" where the dependencies are managed more efficiently, and it seems this here would allow at least some of that. |
|
We extended what I posted in #53834 (comment) to resolve some of those issues. We added some tasks to the But it would still be A LOT easier to just have a concept for this directly available by a concept of .NET (local) tool. Especially since this workaround needs to pre-compile and bundle files which also lead to requirements for the build chain (i.e. if you expect an EXE to be contained, the NuPkg has to be build on a Windows runner). ** AddToolDependencies.cs of Company.SDK.Bundle ** using JetBrains.Annotations;
using Microsoft.Build.Framework;
using Microsoft.Build.Utilities;
using System.Collections.Generic;
using System.IO;
using System.Linq;
namespace Company.SDK.Bundle
{
/// <summary>
/// Add dependency JSON files for all tools in the project.
/// </summary>
/// <inheritdoc />
[PublicAPI]
public class AddToolDependencies : Task
{
private const string RuntimeConfigFileName = ".runtimeconfig.json";
private const string DepsJsonFileName = ".deps.json";
private const string RelativePathMetadataName = "RelativePath";
/// <summary>
/// The path to the generated dependency file. This should be used as template
/// for all other tool dependency files.
/// </summary>
[Required]
public string ProjectDepsFilePath { get; set; } = null!;
/// <summary>
/// All files to be published. The task computes possible outputs from these
/// files, and returns them in the AdditionalOutputs list.
/// It detects tools by checking for an .runtimeconfig.json entry.
/// </summary>
[Required]
public ITaskItem[] ResolvedFileToPublish { get; set; } = null!;
/// <summary>
/// Output items per each tool output. They will have the ProjectDepsFilePath set
/// as Identity and the tool dependency file as RelativePath.
/// </summary>
[Output]
public ITaskItem[] AdditionalOutputs { get; private set; } = null!;
/// <inheritdoc />
public override bool Execute()
{
var mainDepsFileRelativeName = Path.GetFileName(ProjectDepsFilePath);
var inputs = ResolvedFileToPublish
.Select(input => input.GetMetadata(RelativePathMetadataName))
.Where(input => input.EndsWith(RuntimeConfigFileName));
var outputs = new List<ITaskItem>();
foreach (var input in inputs)
{
var relativeName = input.Replace(RuntimeConfigFileName, DepsJsonFileName);
if (relativeName.Equals(mainDepsFileRelativeName)) continue;
var ti = new TaskItem(ProjectDepsFilePath);
ti.SetMetadata(RelativePathMetadataName, relativeName);
outputs.Add(ti);
}
AdditionalOutputs = outputs.ToArray();
return true;
}
}
}** PatchRuntimeConfigs.cs of Company.SDK.Bundle ** using JetBrains.Annotations;
using Microsoft.Build.Framework;
using Microsoft.Build.Utilities;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using Newtonsoft.Json.Serialization;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
namespace Company.SDK.Bundle
{
/// <summary>
/// Path runtime configuration JSON files for all tools in the project.
/// </summary>
/// <inheritdoc />
[PublicAPI]
public class PatchRuntimeConfigs : Task
{
private static readonly JsonSerializer Serializer = new JsonSerializer
{
ContractResolver = new CamelCasePropertyNamesContractResolver(),
Formatting = Formatting.Indented,
DefaultValueHandling = DefaultValueHandling.Ignore
};
[PublicAPI]
private class RuntimeConfigFramework
{
private bool Equals(RuntimeConfigFramework other)
{
return Name == other.Name && Version == other.Version;
}
public override bool Equals(object? obj)
{
if (ReferenceEquals(null, obj)) return false;
if (ReferenceEquals(this, obj)) return true;
if (obj.GetType() != GetType()) return false;
return Equals((RuntimeConfigFramework)obj);
}
public override int GetHashCode()
{
// It's not possible to make those readonly since they are initialized by property through JSON deserialization.
// The correct way would be to have the properties marked as "init" instead of "set", but this requires C# 8 which
// is not possible with netstandard2.1...
// ReSharper disable NonReadonlyMemberInGetHashCode
return HashCode.Combine(Name, Version);
// ReSharper restore NonReadonlyMemberInGetHashCode
}
public string? Name { get; set; }
public string? Version { get; set; }
}
private const string RuntimeConfigFileName = ".runtimeconfig.json";
private const string CopyToPublishDirectoryMetadataName = "CopyToPublishDirectory";
/// <summary>
/// The file name of the generated runtime configuration file.
/// </summary>
[Required]
public string ProjectRuntimeConfigFileName { get; set; } = null!;
/// <summary>
/// The path to the generated runtime configuration file. This should be used to patch
/// all other tool runtime configuration files.
/// </summary>
[Required]
public string ProjectRuntimeConfigFilePath { get; set; } = null!;
/// <summary>
/// All files that were published. The task computes possible outputs from these
/// files, and returns them in the PatchedOutputs list.
/// It detects tools by checking for an .runtimeconfig.json entry.
/// </summary>
[Required]
public ITaskItem[] FileWrites { get; set; } = null!;
/// <summary>
/// Output items per each patched tool. They are references to the original (patched) items.
/// </summary>
[Output]
public ITaskItem[] PatchedOutputs { get; private set; } = null!;
/// <inheritdoc />
public override bool Execute()
{
var inputs = FileWrites
.Where(input => input.GetMetadata(CopyToPublishDirectoryMetadataName) != string.Empty &&
input.ItemSpec.EndsWith(RuntimeConfigFileName));
var mainRuntimeConfig = ReadFromJsonFile(ProjectRuntimeConfigFilePath);
var tfm = GetTfm(mainRuntimeConfig);
if (tfm == null)
{
Log.LogError("Not able to fetch tfm from ProjectRuntimeConfigFilePath. Patching not possible.");
return false;
}
var frameworks = GetFrameworks(mainRuntimeConfig);
if (frameworks == null)
{
Log.LogError("Not able to fetch frameworks from ProjectRuntimeConfigFilePath. Patching not possible.");
return false;
}
var outputs = new List<ITaskItem>();
foreach (var input in inputs)
{
var absoluteName = input.ItemSpec;
var relativeName = Path.GetFileName(absoluteName);
if (relativeName.Equals(ProjectRuntimeConfigFileName)) continue;
if (!PatchJson(absoluteName, tfm, frameworks)) continue;
outputs.Add(input);
}
PatchedOutputs = outputs.ToArray();
return true;
}
private string? GetTfm(JObject config)
{
if (!config.TryGetValue("runtimeOptions", out var optionsToken) || !(optionsToken is JObject options))
{
Log.LogError("The config file is not containing the required runtimeOptions property.");
return null;
}
if (!options.TryGetValue("tfm", out var tfmToken))
{
Log.LogError("The config file is not containing the required tfm property.");
return null;
}
return tfmToken.Value<string>();
}
private RuntimeConfigFramework[]? GetFrameworks(JObject config)
{
if (!config.TryGetValue("runtimeOptions", out var optionsToken) || !(optionsToken is JObject options))
{
Log.LogError("The config file is not containing the required runtimeOptions property.");
return null;
}
if (options.TryGetValue("framework", out var frameworkToken) && frameworkToken is JObject framework)
{
return new[] { framework.ToObject<RuntimeConfigFramework>()! };
}
if (!options.TryGetValue("frameworks", out var frameworksToken) || !(frameworksToken is JArray frameworks))
{
Log.LogError("The config file is not containing the required framework OR frameworks property.");
return null;
}
return frameworks.Select(f => f.ToObject<RuntimeConfigFramework>()!).ToArray();
}
private bool PatchJson(string toolRuntimeConfigPath, string tfm, RuntimeConfigFramework[] frameworks)
{
var toolRuntimeConfig = ReadFromJsonFile(toolRuntimeConfigPath);
var patchedAnything = PatchTfm(toolRuntimeConfigPath, toolRuntimeConfig, tfm);
patchedAnything |= PatchFrameworks(toolRuntimeConfigPath, toolRuntimeConfig, frameworks);
if (!patchedAnything)
{
return false;
}
WriteToJsonFile(toolRuntimeConfigPath, toolRuntimeConfig);
return true;
}
private bool PatchTfm(string toolRuntimeConfigPath, JObject toolRuntimeConfig, string tfm)
{
var toolTfm = GetTfm(toolRuntimeConfig);
if (toolTfm == null)
{
Log.LogError("Not able to fetch tfm from {0}. File will remain unpatched.", toolRuntimeConfigPath);
return false;
}
// Get highest tfm:
var highestTfm = string.Compare(tfm, toolTfm, StringComparison.Ordinal) > 0
? tfm
: toolTfm;
// Nothing to patch:
if (toolTfm == tfm) return false;
// Replace tfm with highest:
var runtimeOptions = toolRuntimeConfig["runtimeOptions"]!;
runtimeOptions["tfm"] = highestTfm;
return true;
}
private bool PatchFrameworks(string toolRuntimeConfigPath, JObject toolRuntimeConfig, RuntimeConfigFramework[] frameworks)
{
var toolFrameworks = GetFrameworks(toolRuntimeConfig);
if (toolFrameworks == null)
{
Log.LogError("Not able to fetch frameworks from {0}. File will remain unpatched.", toolRuntimeConfigPath);
return false;
}
// Get runtime framework version:
var runtimeVersion = frameworks.FirstOrDefault()?.Version;
// Patch all tool frameworks to the runtime version:
var atLeastOnePatched = false;
foreach (var framework in toolFrameworks)
{
if (framework.Version == runtimeVersion) continue;
atLeastOnePatched = true;
framework.Version = runtimeVersion;
}
// Get combination of frameworks:
var combinedFrameworks = frameworks
.Union(toolFrameworks)
.ToArray();
// Nothing to patch:
if (!atLeastOnePatched && combinedFrameworks.Length == toolFrameworks.Length) return false;
// Make sure we always use "frameworks" and not "framework":
var runtimeOptions = toolRuntimeConfig["runtimeOptions"]!;
runtimeOptions["framework"]?.Parent?.Replace(new JProperty("frameworks"));
// Replace frameworks with combined list:
runtimeOptions["frameworks"] = new JArray(combinedFrameworks.Select(framework =>
JObject.FromObject(framework, Serializer)));
return true;
}
private static JObject ReadFromJsonFile(string path)
{
using var reader = new JsonTextReader(File.OpenText(path));
return JObject.Load(reader);
}
private static void WriteToJsonFile(string fileName, JObject value)
{
using var writer = new JsonTextWriter(new StreamWriter(File.Create(fileName)));
Serializer.Serialize(writer, value);
}
}
}** Sdk\Sdk.targets of Company.SDK.Bundle ** <Project ToolsVersion="14.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<Import Sdk="Microsoft.NET.Sdk" Project="Sdk.targets" />
<PropertyGroup>
<MSBuildAllProjects>$(MSBuildAllProjects);$(MSBuildThisFileFullPath)</MSBuildAllProjects>
<_SDK_MsBuildAssembly>netstandard2.1\SDK.MSBuild.dll</_SDK_MsBuildAssembly>
</PropertyGroup>
<UsingTask AssemblyFile="$(_SDK_MsBuildAssembly)" TaskName="Company.SDK.Bundle.AddToolDependencies" />
<UsingTask AssemblyFile="$(_SDK_MsBuildAssembly)" TaskName="Company.SDK.Bundle.PatchRuntimeConfigs" />
<Target Name="_AddToolDependencies" AfterTargets="ComputeResolvedFilesToPublishList">
<AddToolDependencies ProjectDepsFilePath="$(ProjectDepsFilePath)"
ResolvedFileToPublish="@(ResolvedFileToPublish)">
<Output TaskParameter="AdditionalOutputs" ItemName="_AdditionalOutputs" />
</AddToolDependencies>
<ItemGroup>
<!-- Copy the .deps.json file from the Bootstrapper to all tools. -->
<ResolvedFileToPublish Include="@(_AdditionalOutputs)">
<CopyToPublishDirectory>PreserveNewest</CopyToPublishDirectory>
</ResolvedFileToPublish>
</ItemGroup>
<Message Importance="High" Text="Added additional tool dependencies:" Condition="'@(_AdditionalOutputs)' != ''" />
<Message Importance="High" Text="- %(_AdditionalOutputs.RelativePath)" Condition="'@(_AdditionalOutputs)' != ''" />
</Target>
<Target Name="_PatchRuntimeConfigs" AfterTargets="CopyFilesToPublishDirectory">
<PatchRuntimeConfigs ProjectRuntimeConfigFileName="$(ProjectRuntimeConfigFileName)"
ProjectRuntimeConfigFilePath="$(ProjectRuntimeConfigFilePath)"
FileWrites="@(FileWrites)">
<Output TaskParameter="PatchedOutputs" ItemName="_PatchedOutputs" />
</PatchRuntimeConfigs>
<Message Importance="High" Text="Patched runtime configs:" Condition="'@(_PatchedOutputs)' != ''" />
<Message Importance="High" Text="- %(_PatchedOutputs.RelativePath)" Condition="'@(_PatchedOutputs)' != ''" />
</Target>
</Project> |
|
This just goes to show how a (naively spelled out of course) simple change would prove tremendous in value. We need in this order:
In that Order. Each of these steps would unlock a whole slew of opportunities and therefore make these next steps easier to build. |
This is not currently supported. The closest supported deployment is publishing multiple framework-dependent apps, then setting
DOTNET_ROOTbefore running them to use an unzipped copy of the shared framework.Supporting this will require both runtime and SDK work to select the appropriate set of shared assemblies.
The text was updated successfully, but these errors were encountered: