Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to a version of NuGet that understand .NET Standard and .NET Core 2.0 #935

Merged
merged 6 commits into from
Mar 6, 2017

Conversation

dsplaisted
Copy link
Member

@@ -6,7 +6,8 @@
<MsBuildPackagesVersion>0.1.0-preview-00028-160627</MsBuildPackagesVersion>
<DependencyModelVersion>1.0.3</DependencyModelVersion>
<PlatformAbstractionsVersion>1.0.3</PlatformAbstractionsVersion>
<NuGetVersion>4.0.0-rtm-2323</NuGetVersion>
<!-- non-official NuGet build taken from https://github.com/nuget/nuget.client/tree/release-4.1.0-netstandard2.0 to contain "2.0" TFMs -->
Copy link
Contributor

@nguerrera nguerrera Mar 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this comment? It is bound to get out of date as this version is bumped. What is non-official about it?

Copy link
Member

@eerhardt eerhardt Mar 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This build is from a branch that isn't an official NuGet dev branch. It is branched off of the official branch and then @ericstj made a change do do the netcoreapp2.0 => netstandard2.0 mapping.

The point of the comment is to tell people that we are using basically a "private" NuGet build and to not take a new NuGet build that doesn't have the netstandard2.0 mapping.

Maybe we could add a test that restoring a netcoreapp2.0 app that references netstandard2.0 works? That would ensure we don't take a "bad" NuGet build.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, please help me keep an eye on future updates so that when this version becomes official, we remove the comment.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When are we getting off a "private" branch and on to a proper vNext branch for nuget?

Copy link
Contributor

@nguerrera nguerrera Mar 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And, yes, please add a test.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've asked the NuGet team multiple times.

/cc @rrelyea @emgarten @rohit21agrawal

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can remove the comment. @emgarten is getting the nuget changes merged into the 15.3 VS branch today

@nguerrera
Copy link
Contributor

There are assembly loading failures on core. I think you may need to update the CLI we use to build and test as well.

Copy link
Contributor

@nguerrera nguerrera left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Waiting for tests to pass

Copy link
Member

@emgarten emgarten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is an official build now with netstandard2.0 support. Use 4.3.0-beta1-2342

@nguerrera
Copy link
Contributor

There is an official build now with netstandard2.0 support. Use 4.3.0-beta1-2342

:) You probably have to put that in CLI first, then integrate CLI version bump here because the issue with tasks loading different nuget than CLI is still not fixed in CLI.

@emgarten
Copy link
Member

emgarten commented Mar 3, 2017

@nguerrera alright, I'll work on integrating it there also. Can we update this PR to target the new version? I'm trying to make sure this all goes in today.

@nguerrera
Copy link
Contributor

@emgarten
Copy link
Member

emgarten commented Mar 3, 2017

CLI PR is here dotnet/cli#5912 for reference

@dsplaisted
Copy link
Member Author

@livarcocc @piotrpMSFT It looks like this is failing because the master branch of the CLI doesn't have the 1.1.1 shared framework. Can you update it to include 1.1.1?

2.0.0-alpha-5212
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@livarcocc @piotrpMSFT - looks like we got a bad change in the CLI. We are no longer padding our commit counts. We should fix that.

@dsplaisted
Copy link
Member Author

It looks like the Ubuntu builds are failing due to NuGet/Home#4424. @eerhardt, is there any workaround we can apply?

Copy link
Member

@emgarten emgarten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NuGet version looks good

string selfContainedExecutableFullPath = Path.Combine(outputDirectory.FullName, selfContainedExecutable);

// Workaround for https://github.com/NuGet/Home/issues/4424
if (!RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we put this in a common spot in the test framework?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, thanks

@@ -131,6 +131,9 @@ Copyright (c) .NET Foundation. All rights reserved.

<Message Text="CrossgenCommandline: $(CrossgenCommandline)"/>

<!-- Workaround for https://github.com/NuGet/Home/issues/4424 -->
<Exec Command="chmod 755 $(CrossgenExe)" Condition="'$(OS)' != 'Windows_NT'"/>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we log a tracking bug to take this out when the underlying issue is fixed? That way we remember to remove it.

In general, this is bad in our product because we don't know if we have access to do this or not.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -0,0 +1,26 @@
using FluentAssertions;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit) copyright.

@omajid
Copy link
Member

omajid commented Mar 8, 2017

FYI, this breaks the build on RHEL:

dotnet-install: Calling: machine_has curl
dotnet-install: Calling: calculate_vars 
dotnet-install: Calling: get_azure_channel_from_channel rel-1.0.0
dotnet-install: azure_channel=rel-1.0.0
dotnet-install: Calling: get_normalized_architecture_from_architecture <auto>
dotnet-install: Calling: get_machine_architecture 
dotnet-install: Calling: get_normalized_architecture_from_architecture x64
dotnet-install: normalized_architecture=x64
dotnet-install: Calling: get_specific_version_from_version https://dotnetcli.azureedge.net/dotnet rel-1.0.0 x64 2.0.0-alpha-5212
dotnet-install: specific_version=2.0.0-alpha-5212
dotnet-install: Calling: construct_download_link https://dotnetcli.azureedge.net/dotnet rel-1.0.0 x64 2.0.0-alpha-5212
dotnet-install: Calling: get_current_os_name 
dotnet-install: download_link=https://dotnetcli.azureedge.net/dotnet/Sdk/2.0.0-alpha-5212/dotnet-dev-rhel-x64.2.0.0-alpha-5212.tar.gz
dotnet-install: Calling: resolve_installation_path <auto>
dotnet-install: Calling: get_user_share_path 
dotnet-install: resolve_installation_path: share_path=/home/dotnet-ci/.jenkins/workspace/dotnet-sdk-master/.dotnet_cli
dotnet-install: install_root=/home/dotnet-ci/.jenkins/workspace/dotnet-sdk-master/.dotnet_cli
dotnet-install: Calling: check_pre_reqs 
ldconfig is not in PATH, trying /sbin/ldconfig.
dotnet-install: Calling: install_dotnet 
dotnet-install: Calling: is_dotnet_package_installed /home/dotnet-ci/.jenkins/workspace/dotnet-sdk-master/.dotnet_cli sdk 2.0.0-alpha-5212
dotnet-install: Calling: combine_paths /home/dotnet-ci/.jenkins/workspace/dotnet-sdk-master/.dotnet_cli sdk
dotnet-install: combine_paths: root_path=/home/dotnet-ci/.jenkins/workspace/dotnet-sdk-master/.dotnet_cli
dotnet-install: combine_paths: child_path=sdk
dotnet-install: Calling: combine_paths /home/dotnet-ci/.jenkins/workspace/dotnet-sdk-master/.dotnet_cli/sdk 2.0.0-alpha-5212
dotnet-install: combine_paths: root_path=/home/dotnet-ci/.jenkins/workspace/dotnet-sdk-master/.dotnet_cli/sdk
dotnet-install: combine_paths: child_path=2.0.0-alpha-5212
dotnet-install: is_dotnet_package_installed: dotnet_package_path=/home/dotnet-ci/.jenkins/workspace/dotnet-sdk-master/.dotnet_cli/sdk/2.0.0-alpha-5212
dotnet-install: Zip path: /tmp/dotnet.Yp6C2ATs0
dotnet-install: Downloading https://dotnetcli.azureedge.net/dotnet/Sdk/2.0.0-alpha-5212/dotnet-dev-rhel-x64.2.0.0-alpha-5212.tar.gz
dotnet-install: Calling: download https://dotnetcli.azureedge.net/dotnet/Sdk/2.0.0-alpha-5212/dotnet-dev-rhel-x64.2.0.0-alpha-5212.tar.gz /tmp/dotnet.Yp6C2ATs0
dotnet_install: Error: Download failed

@eerhardt
Copy link
Member

eerhardt commented Mar 8, 2017

@omajid and @dsplaisted,

The CLI's RHEL has been broken for a while now. @piotrpMSFT has a PR out to fix it dotnet/cli#5945.

The issue is there is no RHEL CLI build for version 2.0.0-alpha-5212.

When the CLI fixes the RHEL problem and has a new build out, we should update the version number here to a fully completed CLI build.

BTW - @omajid, maybe you know why this is happening. On our RHEL docker container (and only in the RHEL container), we are seeing git commands like git rev-list --count HEAD output warning information to stdout. For example, the output looks like this:

setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
5378

The docker container we are using is: microsoft/dotnet-buildtools-prereqs:rhel7_prereqs. I'm not sure if we didn't set something up correctly in that container. I don't know where the dockerfile is for it, maybe @MichaelSimons knows?

@MichaelSimons
Copy link
Member

The Dockerfile is in an private repo. FWIW we have plans to move this to the open.

@eerhardt
Copy link
Member

eerhardt commented Mar 8, 2017

Here's the dockerfile:

FROM rhel7
MAINTAINER Matt Ellis <matell@microsoft.com>

# Install the base toolchain we need to build anything (clang, cmake, make and the like)
# this does not include libraries that we need to compile different projects, we'd like
# them in a different layer.

# Our toolchain is CMake 3.3.2 from GhettoForge; LLVM 3.6.2 built from source ourselves; wget,
# which, and make (from RHEL).
RUN yum install -y https://matell.blob.core.windows.net/rpms/jsoncpp-0.10.5-1.el7.x86_64.rpm \
        https://matell.blob.core.windows.net/rpms/cmake-3.3.2-1.gf.el7.x86_64.rpm \
        https://matell.blob.core.windows.net/rpms/clang-3.6.2-1.el7.centos.x86_64.rpm \
        https://matell.blob.core.windows.net/rpms/clang-libs-3.6.2-1.el7.centos.x86_64.rpm \
        https://matell.blob.core.windows.net/rpms/lldb-3.6.2-1.el7.centos.x86_64.rpm \
        https://matell.blob.core.windows.net/rpms/lldb-devel-3.6.2-1.el7.centos.x86_64.rpm \
        https://matell.blob.core.windows.net/rpms/llvm-3.6.2-1.el7.centos.x86_64.rpm \
        https://matell.blob.core.windows.net/rpms/llvm-libs-3.6.2-1.el7.centos.x86_64.rpm \
        wget \
        which \
        make && \
    yum clean all

# Install tools used by the VSO build automation.  We use NodeJS from nodesource.com.
RUN yum install -y https://rpm.nodesource.com/pub_0.10/el/7/x86_64/nodesource-release-el7-1.noarch.rpm && \
    yum updateinfo && \
    yum install -y git \
        zip \
        tar \
        nodejs \
    yum clean all && \
    npm install -g azure-cli

# Dependencies of CoreCLR and CoreFX.  Everything except lttng is present in RHEL, for lttng we
# get the development packages from EfficiOS.
RUN wget -P /etc/yum.repos.d/ https://packages.efficios.com/repo.files/EfficiOS-RHEL7-x86-64.repo && \
    rpmkeys --import https://packages.efficios.com/rhel/repo.key && \
    yum updateinfo && \
    yum install --enablerepo=rhel-7-server-optional-rpms -y libicu-devel \
        libuuid-devel \
        libcurl-devel \
        openssl-devel \
        libunwind-devel \
        lttng-ust-devel && \
    yum clean all

# Define the en_US.UTF-8 locale (without this, processes MSBuild trys to launch complain about
# being unable to use the en_US.UTF-8 locale which can cause downstream failures if you capture
# the output of a run)
RUN localedef -c -i en_US -f UTF-8 en_US.UTF-8

@eerhardt
Copy link
Member

eerhardt commented Mar 8, 2017

@ellismg - I see you added the last line in that dockerfile:

# Define the en_US.UTF-8 locale (without this, processes MSBuild trys to launch complain about
# being unable to use the en_US.UTF-8 locale which can cause downstream failures if you capture
# the output of a run)
RUN localedef -c -i en_US -f UTF-8 en_US.UTF-8

Do you know why executing git commands in our build is outputting the following?

setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
5378

@ellismg
Copy link
Contributor

ellismg commented Mar 8, 2017

@eerhardt I understand the context, I'll take a look. I would have expected this would have fixed it...

@omajid
Copy link
Member

omajid commented Mar 8, 2017

I just built/tested the dockerfile and I can not reproduce this with plain git:

FROM rhel7
MAINTAINER Matt Ellis <matell@microsoft.com>

# Install the base toolchain we need to build anything (clang, cmake, make and the like)
# this does not include libraries that we need to compile different projects, we'd like
# them in a different layer.

# Our toolchain is CMake 3.3.2 from GhettoForge; LLVM 3.6.2 built from source ourselves; wget,
# which, and make (from RHEL).
RUN yum install -y https://matell.blob.core.windows.net/rpms/jsoncpp-0.10.5-1.el7.x86_64.rpm \
        https://matell.blob.core.windows.net/rpms/cmake-3.3.2-1.gf.el7.x86_64.rpm \
        https://matell.blob.core.windows.net/rpms/clang-3.6.2-1.el7.centos.x86_64.rpm \
        https://matell.blob.core.windows.net/rpms/clang-libs-3.6.2-1.el7.centos.x86_64.rpm \
        https://matell.blob.core.windows.net/rpms/lldb-3.6.2-1.el7.centos.x86_64.rpm \
        https://matell.blob.core.windows.net/rpms/lldb-devel-3.6.2-1.el7.centos.x86_64.rpm \
        https://matell.blob.core.windows.net/rpms/llvm-3.6.2-1.el7.centos.x86_64.rpm \
        https://matell.blob.core.windows.net/rpms/llvm-libs-3.6.2-1.el7.centos.x86_64.rpm \
        wget \
        which \
        make && \
    yum clean all

# Install tools used by the VSO build automation.  We use NodeJS from nodesource.com.
RUN yum install -y https://rpm.nodesource.com/pub_0.10/el/7/x86_64/nodesource-release-el7-1.noarch.rpm && \
    yum updateinfo && \
    yum install -y git \
        zip \
        tar \
        nodejs \
    yum clean all && \
    npm install -g azure-cli

# Dependencies of CoreCLR and CoreFX.  Everything except lttng is present in RHEL, for lttng we
# get the development packages from EfficiOS.
RUN wget -P /etc/yum.repos.d/ https://packages.efficios.com/repo.files/EfficiOS-RHEL7-x86-64.repo && \
    rpmkeys --import https://packages.efficios.com/rhel/repo.key && \
    yum updateinfo && \
    yum install --enablerepo=rhel-7-server-optional-rpms -y libicu-devel \
        libuuid-devel \
        libcurl-devel \
        openssl-devel \
        libunwind-devel \
        lttng-ust-devel && \
    yum clean all

# Define the en_US.UTF-8 locale (without this, processes MSBuild trys to launch complain about
# being unable to use the en_US.UTF-8 locale which can cause downstream failures if you capture
# the output of a run)
RUN localedef -c -i en_US -f UTF-8 en_US.UTF-8

RUN export LC_ALL="en_US.UTF-8"
CMD mkdir foo && cd foo && git config --global "user.name" "anonymous" && git config --global "user.email" "anonymous" && git init && echo "Hello World" > ignore && git add ignore && git commit -m "first" && git rev-list --count HEAD

When I run the above, I just get "1" as the output and no error messages.

$ sudo docker run --rm -ti dea9c7679a57
Initialized empty Git repository in /foo/.git/
[master (root-commit) 4b84508] first
 1 file changed, 1 insertion(+)
 create mode 100644 ignore
1

I also tried with LANG instead of LC_ALL and that didn't make a difference.

@ellismg
Copy link
Contributor

ellismg commented Mar 8, 2017

The problem is something specific to how MSBuild Exec stuff worked inside the container (which sort of meshes with what I remember from back when I added that line in the to the dockerfile). Like @omajid I don't repo the issue inside the container when running straight git commands, but I do see it when I invoke MSBuild. I'm continuing to investigate and hope to have a resolution later today.

@ellismg
Copy link
Contributor

ellismg commented Mar 9, 2017

MSBuild essentially generates a file that looks like this, which it then invokes with #!/bin/sh (the logic is a little more convoluted):

#!/bin/sh

export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8

git rev-list --count HEAD

If you run that inside the container, you see the output you might expect:

[root@76250420ec8d cli]# ./repro.sh
./repro.sh: line 4: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
./repro.sh: line 4: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
5387

Interestingly enough, in the container I see this:

[root@76250420ec8d cli]# locale -a
C
POSIX

Once I run localedef -c -i en_US -f UTF-8 en_US.UTF-8 then the locale shows up in locale -a and running the shell script doesn't produce any warning output.

Maybe the image we have on docker hub doesn't match that Dockerfile for some reason? @MichaelSimons

@ellismg
Copy link
Contributor

ellismg commented Mar 9, 2017

Maybe the image we have on docker hub doesn't match that Dockerfile for some reason? @MichaelSimons

Note that previously we we were managing this stuff without @MichaelSimons's oversite we didn't have great operational hygine on ensuring the dockerfiles were always checked in somewhere sane and published from that location. So I could totally imagine that the image that made it's way to dockerhub did not match what we have in the dockerfile today.

@rainersigwald
Copy link
Member

@ellismg Should we (MSBuild) do something else in this situation? We were trying to approximate our Windows behavior (to allow commands with high Unicode characters) but I'd totally believe we're doing it wrong . . .

@MichaelSimons
Copy link
Member

@ellismg, @eerhardt - I think I see what is happening. @eerhardt stated here that microsoft/dotnet-buildtools-prereqs:rhel7_prereqs is being used. This is not the most recent version of the rhel7 image that we have - microsoft/dotnet-buildtools-prereqs:rhel7_prereqs_2 is. The checked in Dockerfile is for the latest version.

FYI the docker history command can be of great help in situations like this. If a Docker images was built with a Dockerfile, it will tell you exactly how it was created. The formatting isn't always the easiest to read because of indentation included in the Dockerfile but you can get what you need especially if you specify the --no-trunc option.

docker history microsoft/dotnet-buildtools-prereqs:rhel7_prereqs
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
ffb8aa5a02ca        14 months ago       /bin/sh -c wget -P /etc/yum.repos.d/ https...   141 MB
<missing>           14 months ago       /bin/sh -c yum install -y https://rpm.node...   518 MB
<missing>           14 months ago       /bin/sh -c yum install -y https://dl.fedor...   290 MB
<missing>           14 months ago       /bin/sh -c #(nop) MAINTAINER Matt Ellis <m...   0 B
<missing>           15 months ago                                                       202 MB              Imported from -
docker history microsoft/dotnet-buildtools-prereqs:rhel7_prereqs_2
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
6030370d4d83        11 months ago       /bin/sh -c localedef -c -i en_US -f UTF-8 ...   1.61 MB
<missing>           14 months ago       /bin/sh -c wget -P /etc/yum.repos.d/ https...   141 MB
<missing>           14 months ago       /bin/sh -c yum install -y https://rpm.node...   518 MB
<missing>           14 months ago       /bin/sh -c yum install -y https://dl.fedor...   290 MB
<missing>           14 months ago       /bin/sh -c #(nop) MAINTAINER Matt Ellis <m...   0 B
<missing>           15 months ago                                                       202 MB              Imported from -

You can clearly see the V2 version of the image added the localedef -c -i en_US -f UTF-8 en_US.UTF-8 logic. Update the image being used to microsoft/dotnet-buildtools-prereqs:rhel7_prereqs_2

@eerhardt
Copy link
Member

eerhardt commented Mar 9, 2017

Thanks so much for tracking this issue down, @ellismg @MichaelSimons.

Update the image being used to microsoft/dotnet-buildtools-prereqs:rhel7_prereqs_2

I can update our VSTS build def.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.