dotnet publish: AfterTargets="AfterPublish" triggers before publish is complete

[catmanjan]

Hello we have a project like many others publishing docker images to kubernetes.

To try to make developers life easier I added the following to my pubxml:

<?xml version="1.0" encoding="utf-8"?>
<!--
https://go.microsoft.com/fwlink/?LinkID=208121.
-->
<Project>
  <PropertyGroup>
    ...snip...
  <Target Name="CustomActionsBeforePublish" BeforeTargets="BeforePublish">
    <Exec Command="az acr login -n icognitionhydra" />
  </Target>
  <Target Name="RestartPod" AfterTargets="AfterPublish">
	<Exec Command="az aks get-credentials -g hydraKubernetes -n hydra" />
    <Exec Command="kubectl rollout restart deployment hydra" />
  </Target>
</Project>

However, I notice the RestartPods target occurs before the image is fully published to my ACR, resulting in a race condition that sometimes Kubernetes beats, meaning the old image is deployed instead of the new...

What should my AfterTargets be set to if I want it to only run the rollout commands after the container is fully published to ACR?

[baronfel]

I'd need to take a look at a binlog to double-check, but my first recommendation would be the PublishContainer target - that's the target that actually creates + pushes container images to the desired destination, whether that's local daemon, tarball, or remote registry.

[catmanjan]

@baronfel Thanks I'll try that now - out of interest where can someone read about all the different "AfterTargets" options?

I promise I did try searching myself!

[catmanjan]

@baronfel I just tried this, but the commands never executed

  <Target Name="RestartPod" AfterTargets="PublishContainer">
	<Exec Command="az aks get-credentials -g hydraKubernetes -n hydra" />
    <Exec Command="kubectl rollout restart deployment hydra" />
  </Target>

[baronfel]

Is there any way you can get an https://aka.ms/binlog and use the viewer to take a look? I added a target just like yours to a sample project I had, and then published that project with dotnet publish -t:PublishContainer and see the target firing as I expect:

[catmanjan]

@baronfel thanks are you able to interpret this for me? Could the issue be that this publish is occuring within a docker container and so my pubxml commands aren't being triggered?

[baronfel]

Where is this DockerBuild target coming from? I'm not familiar with it at all.

[catmanjan]

@baronfel I am just using the right-click Publish to Azure Container Registry feature that Visual Studio suggests

[catmanjan]

Here is what I see in the output window, notice how the kubectl command is triggered, the publish "completes" but the actual push to ACR does not occur until after the publish is complete?

Step 30/30 : LABEL com.microsoft.visual-studio.project-name=Hydra
 ---> Using cache
 ---> daefef911fe7
[Warning] One or more build-args [BUILD_CONFIGURATION] were not consumed
Successfully built daefef911fe7
Successfully tagged hydra:latest
az aks get-credentials -g hydraKubernetes -n hydra
EXEC(0,0): Warning : Merged "hydra" as current context in C:\Users\sysadmin\.kube\config
kubectl rollout restart deployment hydra
deployment.apps/hydra restarted
========== Build: 9 succeeded, 0 failed, 3 up-to-date, 0 skipped ==========
========== Build completed at 11:23 PM and took 05:30.684 minutes ==========
========== Publish: 1 succeeded, 0 failed, 0 skipped ==========
========== Publish completed at 11:23 PM and took 05:30.684 minutes ==========
The push refers to repository [icognitionhydra.azurecr.io/hydra]
7f2261dc2e3e: Preparing
e96d0e66376d: Preparing
442f1ae5f49d: Preparing
6ee03fa940d9: Preparing
e47c0a849651: Preparing
ec54d5dc8e32: Preparing
de7434dd2381: Preparing
3284c7044632: Preparing
8b2d1b359bba: Preparing
39663a95f47b: Preparing
15eb04cadfb8: Preparing
1686300b7367: Preparing
8d22affe413b: Preparing
10c31dce9201: Preparing
4b92f58a784a: Preparing
1fcedfbb9275: Preparing
bfb1551e29cb: Preparing
09eb169299e4: Preparing
6002749272ba: Preparing
22fdd7b27df5: Preparing
97c8018c7277: Preparing
ec54d5dc8e32: Waiting
de7434dd2381: Waiting
3284c7044632: Waiting
8b2d1b359bba: Waiting
39663a95f47b: Waiting
15eb04cadfb8: Waiting
1686300b7367: Waiting
8d22affe413b: Waiting
10c31dce9201: Waiting
4b92f58a784a: Waiting
1fcedfbb9275: Waiting
bfb1551e29cb: Waiting
09eb169299e4: Waiting
6002749272ba: Waiting
22fdd7b27df5: Waiting
97c8018c7277: Waiting
7f2261dc2e3e: Layer already exists
e47c0a849651: Layer already exists
e96d0e66376d: Layer already exists
6ee03fa940d9: Layer already exists
ec54d5dc8e32: Layer already exists
de7434dd2381: Layer already exists
8b2d1b359bba: Layer already exists
3284c7044632: Layer already exists
39663a95f47b: Layer already exists
15eb04cadfb8: Layer already exists
442f1ae5f49d: Layer already exists
1686300b7367: Layer already exists
8d22affe413b: Layer already exists
10c31dce9201: Layer already exists
4b92f58a784a: Layer already exists
bfb1551e29cb: Layer already exists
1fcedfbb9275: Layer already exists
09eb169299e4: Layer already exists
6002749272ba: Layer already exists
97c8018c7277: Layer already exists
22fdd7b27df5: Layer already exists
latest: digest: sha256:b14508506e1448ce6945f9cc4b6b09318f20dab21fae11f36552ba723e06c4b3 size: 4725
Successfully pushed docker image with tag 'latest'

[baronfel]

It's going to be very hard to help you because VS right click publish behaves significantly differently than CLI publish. I would strongly recommend using the CLI to publish directly as much as possible, because that's where we have the best ability to inspect and reproduce your scenarios.

In the editor, VS sequences the Publish and the PublishContainer operations separately, and I think the screenshot you showed a bit ago is from the former - I don't see the SDK Container publish call at all.

[catmanjan]

@baronfel developers would prefer to publish from within the IDE - do I need to log this somewhere else? Do you know where?

[baronfel]

@anvillan could you help me with some insight into how the VS container publish works/is triggered? Is is possible to get binlogs for the actual PublishContainer step?

[catmanjan]

Not sure if its useful but here are all the binlogs VS generated for me doing various publishes via right-click publish... https://filebin.net/1qqoq7nxbm7b4exw

[anvillan]

@baronfel

I just ran a .NET SDK Container publish process to afterTargetACR ACR.

AfterTargetsPublishBinLogs.zip

[catmanjan]

I don't really know how to interpret this, but to me it seems like the publish is doing what its supposed to, so perhaps it is rather that the container push (which presumeably is being performed by docker desktop?) is occuring in another process and dotnet isnt waiting for the result of that process before continuing

[baronfel]

@catmanjan thanks for those binlogs - they were very helpful! Ok, so it's as I expected - your pubxml is using the VS Docker tools to do publishing, not the .NET SDK - your pubxml is using the ContainerRegistry PublishProvider and the Custom WebPublishMethod. The expected way to trigger the .NET SDK container publishing is via the Container WebPublishMethod. I feel like a lot of the confusion in this issue is because of the mixing of these two related-but-separate deployment methods. You are not using the SDK container publishing here, you're using the Microsoft.Docker VS tools - those tools likely have some customization that only push the container after the related Publish call occurs.

[catmanjan]

Ok cool, should I leave this issue here or log one elsewhere?

…

On Thu, 27 Feb 2025, 2:13 am Chet Husk, ***@***.***> wrote: @catmanjan <https://github.com/catmanjan> thanks for those binlogs - they were very helpful! Ok, so it's as I expected - your pubxml is using the VS Docker tools to do publishing, not the .NET SDK - your pubxml is using the ContainerRegistry PublishProvider and the Custom WebPublishMethod. The expected way to trigger the .NET SDK container publishing is via the Container WebPublishMethod. I feel like a lot of the confusion in this issue is because of the mixing of these two related-but-separate deployment methods. You are not using the SDK container publishing here, you're using the Microsoft.Docker VS tools - those tools likely have some customization that only push the container *after* the related Publish call occurs. — Reply to this email directly, view it on GitHub <#46999 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABBRBUI6A45A7BGORXGM7L2RXK3LAVCNFSM6AAAAABXSC6GGCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBVGM3DKNRYGE> . You are receiving this because you were mentioned.Message ID: ***@***.***> [image: baronfel]*baronfel* left a comment (dotnet/sdk#46999) <#46999 (comment)> @catmanjan <https://github.com/catmanjan> thanks for those binlogs - they were very helpful! Ok, so it's as I expected - your pubxml is using the VS Docker tools to do publishing, not the .NET SDK - your pubxml is using the ContainerRegistry PublishProvider and the Custom WebPublishMethod. The expected way to trigger the .NET SDK container publishing is via the Container WebPublishMethod. I feel like a lot of the confusion in this issue is because of the mixing of these two related-but-separate deployment methods. You are not using the SDK container publishing here, you're using the Microsoft.Docker VS tools - those tools likely have some customization that only push the container *after* the related Publish call occurs. — Reply to this email directly, view it on GitHub <#46999 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABBRBUI6A45A7BGORXGM7L2RXK3LAVCNFSM6AAAAABXSC6GGCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBVGM3DKNRYGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[baronfel]

I believe that you should log one in VS Developer Community for the Docker Tools team to investigate - that's how that team responses to user feedback. Sorry for the back-and-forth!