NuGet 4.4.0 Error Code 1 Packages Already Installed

[gep13]

@thnk2wn commented on Tue Oct 24 2017

What You Are Seeing?

Having nuget.exe 4.4.0 in Cake/Tools with packages already installed results in error code 1 "all packages are already installed". This causes build.ps1 to fail early restoring tools/addins and stops the build process. Generally this isn't a problem on the build server with fresh downloads and builds but it happens regularly for local builds. Perhaps LASTEXITCODE changes with NuGet 4.4.0. This version of NuGet was needed for some .NET Standard 2.0 issues.

What is Expected?

Expecting Cake's build.ps1 to detect such "error" (warning at best?) and to proceed regardless of whether packages are already installed.

What version of Cake are you using?

0.23.0

Are you running on a 32 or 64 bit system?

64

What environment are you running on? Windows? Linux? Mac?

Windows 10 Fall Creators Update.

Are you running on a CI Server? If so, which one?

Both CI and local. CI is TeamCity 9.x.

How Did You Get This To Happen? (Steps to Reproduce)

Download NuGet 4.4.0 from https://www.nuget.org/downloads (it appears to be preview currently).
Place this download in Cake/tools.
Invoke the build once via build.ps1, all is fine.
Invoke the build a 2nd time with the packages already installed.
Receive "error" 1 that packages are already installed.

Some more context in this post:
https://geoffhudik.com/tech/2017/10/24/build-blues-with-net-standard-2-0/

Output Log

C:\source\projectname\cake [feature/Win10FallCreatorsUpdate ≡]> .\build.ps1 -Verbosity diagnostic
Preparing to run build script...
Feeds used:
  C:\Users\geoffh\.nuget\packages\
  https://api.nuget.org/v3/index.json
  https://www.myget.org/F/msbuildsdkextras/api/v3/index.json
  C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\

An error occured while restoring NuGet tools.
At C:\source\projectname\cake\build.ps1:173 char:9
+         Throw "An error occured while restoring NuGet tools."
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (An error occure...ng NuGet tools.:String) [], RuntimeException
    + FullyQualifiedErrorId : An error occured while restoring NuGet tools.

Possible Issues Here

1. Cake should perhaps treat exit code 1 with "packages already installed" as not an error that stops the script. Perhaps this behavior changed with Nuget 4.4.0.

2. The error message indicates neither the exit code nor the nuget output so it says nothing useful to help the situation.

3. The word "occured" is misspelled - should be "occurred".

4. When an error happens here, Pop-Location isn't executed so it leaves things at a different location than where things were when the script started, leading to possible issues later.

5. Ideally there'd be a way to specify the NuGet version (like when referencing Nuget package tools/addins) of NuGet.exe itself instead of having to place a version here and commit to source control. I don't think the Cake build script could even download NuGet as it would have to happen before #tool or #addin and I think those have to be first.

Workaround Changes Made (build.ps1)

if(-Not $SkipToolPackageRestore.IsPresent) {
    Push-Location
    Set-Location $TOOLS_DIR

    # Check for changes in packages.config and remove installed tools if true.
    [string] $md5Hash = MD5HashFile($PACKAGES_CONFIG)
    if((!(Test-Path $PACKAGES_CONFIG_MD5)) -Or
      ($md5Hash -ne (Get-Content $PACKAGES_CONFIG_MD5 ))) {
        Write-Verbose -Message "Missing or changed package.config hash..."
        Remove-Item * -Recurse -Exclude packages.config,nuget.exe
    }

    Write-Verbose -Message "Restoring tools from NuGet..."
    $NuGetOutput = Invoke-Expression "&`"$NUGET_EXE`" install -ExcludeVersion -OutputDirectory `"$TOOLS_DIR`""

    if ($LASTEXITCODE -ne 0) {
        Write-Warning ($NuGetOutput | out-string)

        if ($LASTEXITCODE -eq 1 -and $NuGetOutput -match ' are already installed') {
            Write-Verbose -Message 'Packages already installed, continuing.'
        }
        else {
            Pop-Location
            throw ("NuGet returned error {0} restoring NuGet tools." -f $LASTEXITCODE)
        }
    }
    else
    {
        $md5Hash | Out-File $PACKAGES_CONFIG_MD5 -Encoding "ASCII"
        Write-Verbose -Message ($NuGetOutput | out-string)
    }
    
    Pop-Location
}

---

@ahoefling commented on Sat Nov 04 2017

I did some investigation on this and I came to the same conclusion, it appears that Cake is using NuGet version 4.3 and there have been changes to NuGet 4.4 that directly apply to the latest windows fall creators update and netstandard builds. If I run the command nuget restore project.sln with NuGet 4.3 I get errors but if I run the same command with NuGet 4.4 the restore runs without issue.

Environment

Name	Version
Windows 10 Fall Creators Update	16299
Xamarin.Forms	2.4.018342
UWP	10.0.16299
Cake	0.23.0

Errors
This error is displayed in the build log and as @thnk2wn mentioned in the shared link it is a problem with NuGet 4.3 which I also reproduced by running the following command

• nuget restore project.sln

Errors in C:\source\TabStrip\CarouselView\CarouselView.FormsPlugin.UWP\CarouselView.FormsPlugin.UWP.csproj
    NU1201: Project CarouselView.FormsPlugin.Abstractions is not compatible with uap10.0.16299 (UAP,Version=v10.0.16299). Project CarouselView.FormsPlugin.Abstractions supports: netstandard2.0 (.NETStandard,Version=v2.0)
    NU1201: Project CarouselView.FormsPlugin.Abstractions is not compatible with uap10.0.16299 (UAP,Version=v10.0.16299) / win10-arm. Project CarouselView.FormsPlugin.Abstractions supports: netstandard2.0 (.NETStandard,Version=v2.0)
    NU1201: Project CarouselView.FormsPlugin.Abstractions is not compatible with uap10.0.16299 (UAP,Version=v10.0.16299) / win10-arm-aot. Project CarouselView.FormsPlugin.Abstractions supports: netstandard2.0 (.NETStandard,Version=v2.0)
    NU1201: Project CarouselView.FormsPlugin.Abstractions is not compatible with uap10.0.16299 (UAP,Version=v10.0.16299) / win10-x64. Project CarouselView.FormsPlugin.Abstractions supports: netstandard2.0 (.NETStandard,Version=v2.0)
    NU1201: Project CarouselView.FormsPlugin.Abstractions is not compatible with uap10.0.16299 (UAP,Version=v10.0.16299) / win10-x64-aot. Project CarouselView.FormsPlugin.Abstractions supports: netstandard2.0 (.NETStandard,Version=v2.0)
    NU1201: Project CarouselView.FormsPlugin.Abstractions is not compatible with uap10.0.16299 (UAP,Version=v10.0.16299) / win10-x86. Project CarouselView.FormsPlugin.Abstractions supports: netstandard2.0 (.NETStandard,Version=v2.0)
    NU1201: Project CarouselView.FormsPlugin.Abstractions is not compatible with uap10.0.16299 (UAP,Version=v10.0.16299) / win10-x86-aot. Project CarouselView.FormsPlugin.Abstractions supports: netstandard2.0 (.NETStandard,Version=v2.0)
Errors in C:\source\TabStrip\src\TabStrip.FormsPlugin.UWP\TabStrip.FormsPlugin.UWP.csproj
    NU1201: Project CarouselView.FormsPlugin.Abstractions is not compatible with uap10.0.16299 (UAP,Version=v10.0.16299). Project CarouselView.FormsPlugin.Abstractions supports: netstandard2.0 (.NETStandard,Version=v2.0)
    NU1201: Project CarouselView.FormsPlugin.Abstractions is not compatible with uap10.0.16299 (UAP,Version=v10.0.16299) / win10-arm. Project CarouselView.FormsPlugin.Abstractions supports: netstandard2.0 (.NETStandard,Version=v2.0)
    NU1201: Project CarouselView.FormsPlugin.Abstractions is not compatible with uap10.0.16299 (UAP,Version=v10.0.16299) / win10-arm-aot. Project CarouselView.FormsPlugin.Abstractions supports: netstandard2.0 (.NETStandard,Version=v2.0)
    NU1201: Project CarouselView.FormsPlugin.Abstractions is not compatible with uap10.0.16299 (UAP,Version=v10.0.16299) / win10-x64. Project CarouselView.FormsPlugin.Abstractions supports: netstandard2.0 (.NETStandard,Version=v2.0)
    NU1201: Project CarouselView.FormsPlugin.Abstractions is not compatible with uap10.0.16299 (UAP,Version=v10.0.16299) / win10-x64-aot. Project CarouselView.FormsPlugin.Abstractions supports: netstandard2.0 (.NETStandard,Version=v2.0)
    NU1201: Project CarouselView.FormsPlugin.Abstractions is not compatible with uap10.0.16299 (UAP,Version=v10.0.16299) / win10-x86. Project CarouselView.FormsPlugin.Abstractions supports: netstandard2.0 (.NETStandard,Version=v2.0)
    NU1201: Project CarouselView.FormsPlugin.Abstractions is not compatible with uap10.0.16299 (UAP,Version=v10.0.16299) / win10-x86-aot. Project CarouselView.FormsPlugin.Abstractions supports: netstandard2.0 (.NETStandard,Version=v2.0)

Attempted Solution
I first attempted cloning the repository to see how hard it would be to update the cake to use netstandard 2.0 and quickly learned there are a lot of dependencies and it will require more than an hour of time.

Work Around
I tried updating the powershell script with the provided script above and that did not work because the nuget restore referenced from cake is still using NuGet 4.3. I ended up making modifications to the powershell script to use the following logic:

1. Check if nuget is a version < 4.4
2. if true download version 4.4
3. prior to running the cake build execute nuget restore project.sln
4. execute cake build

I understand this is a hard coded hack but it solves my problem for now and hopefully this is a suitable workaround for others that run into this problem while we wait for an updated version.

Powershell Updates
Variables:

$TOOLS_DIR = Join-Path $PSScriptRoot "tools"
$ADDINS_DIR = Join-Path $TOOLS_DIR "Addins"
$MODULES_DIR = Join-Path $TOOLS_DIR "Modules"
$NUGET_EXE = Join-Path $TOOLS_DIR "nuget.exe"
$CAKE_EXE = Join-Path $TOOLS_DIR "Cake/Cake.exe"
$NUGET_URL = "https://dist.nuget.org/win-x86-commandline/v4.4.0/nuget.exe"
$PACKAGES_CONFIG = Join-Path $TOOLS_DIR "packages.config"
$PACKAGES_CONFIG_MD5 = Join-Path $TOOLS_DIR "packages.config.md5sum"
$ADDINS_PACKAGES_CONFIG = Join-Path $ADDINS_DIR "packages.config"
$MODULES_PACKAGES_CONFIG = Join-Path $MODULES_DIR "packages.config"

Note the download path for nuget is changed

Nuget:

# Try download NuGet.exe if not exists
if(!(Test-Path $NUGET_EXE))
{
	Write-Verbose -Message "Downloading NuGet.exe..."
    try {
        $wc = GetProxyEnabledWebClient
        $wc.DownloadFile($NUGET_URL, $NUGET_EXE)
    } catch {
        Throw "Could not download NuGet.exe."
    }
}
else
{
	# Check if the current version of nuget is 4.4 if not download it
	$version = (nuget.exe | Select-Object -First 1).Split(':')[1].Trim()
	if(!($version.StartsWith("4.4")))
	{
		Write-Verbose -Message "Downloading NuGet.exe..."
		try {
			$wc = GetProxyEnabledWebClient
			$wc.DownloadFile($NUGET_URL, $NUGET_EXE)
		} catch {
			Throw "Could not download NuGet.exe."
		}
	}
}

then at the bottom of the build.ps1 script should look like this to force a nuget restore

Execute NuGet restore and Cake Build

# run nuget command
nuget restore Project.sln

# Start Cake
Write-Host "Running build script..."
&$CAKE_EXE $cakeArguments
exit $LASTEXITCODE

Just to reiterate - my solution is intended to get my projects by until cake supports NuGet 4.4, I understand it is not the most elegant way to solve this problem. If you choose to use the updates provided there may be side effects with your project.

---

@gep13 commented on Sat Nov 04 2017

@thnk2wn @ahoefling since these issues seems to be related to the bootstrapper that is provided with Cake, I am going to close this issue out, and move it to the resources repo, which is where the bootstrapper lives.

[gep13]

@thnk2wn @ahoefling there are quite a few issues raised between the comments that you have made, so I am going to try to address them individually...

[gep13]

@ahoefling said...
it appears that Cake is using NuGet version 4.3

That is not strictly true. While today, the Cake Bootstrapper is downloading NuGet.exe version 4.3, it is not "tied" to it. The download URL that we are using is this:

https://dist.nuget.org/win-x86-commandline/latest/nuget.exe

which pulls down the latest, released, and stable version of the NuGet client which is 4.3.0.4406. The 4.4.0 version of NuGet is currently the pre-release version, and by default, the Bootstrapper will not download that.

Now, that is not to say that you can't modify the bootstrapper as you have, you are perfectly entitled to do that. The bootstapper that we ship is simple an example, which tries to adhere to some best practices. We fully expect that people are going to need to modify the default functionality in order to suit their needs.

[gep13]

@ahoefling said...
I tried updating the powershell script with the provided script above and that did not work because the nuget restore referenced from cake is still using NuGet 4.3

I just want to clarify this a little bit...

The default bootstrapper attempts to be a little bit smart, and it will only download NuGet.exe if it needs to. For example, if when you run the bootstrapper NuGet.exe is already on your path, then the bootstrapper won't download it into the local tools folder.

Similarly, if you have already ran the bootstrapper once, and it has placed the NuGet.exe into the tools folder, if you modify the bootstrapper with an alternative download URL, it won't download that version, as there is already a version of NuGet.exe in the tools folder. To remedy that, you could simply delete the NuGet.exe in the tools folder, and re-run the bootstrapper, and it will download the specified version.

[gep13]

@ahoefling said...
Just to reiterate - my solution is intended to get my projects by until cake supports NuGet 4.4, I understand it is not the most elegant way to solve this problem.

So hopefully based on the comments above you will see that this is already the case today, as Cake does support that version.

[gep13]

@thnk2wn said...
The word "occured" is misspelled - should be "occurred".

Agreed. I have corrected this in the develop branch of this repo, and I will push it to master later.

[gep13]

@thnk2wn...
Cake should perhaps treat exit code 1 with "packages already installed" as not an error that stops the script. Perhaps this behavior changed with Nuget 4.4.0.

Cake treats any non zero exit code as an error. This isn't a Cake standard, this is a rule across almost all of the tools/systems that I am familiar with. I don't think that it would be a good idea for Cake, or any other system, to add special handling for this. If anything, this sounds like a serious regression in the preview version of the NuGet client, and as such an issue should be raised there:

https://github.com/nuget/home/issues

[gep13]

@thnk2wn said...
The error message indicates neither the exit code nor the nuget output so it says nothing useful to help the situation.

This is something that could likely be handled better, I agree. Would you mind raising a separate issue about this, as this issue is getting quite busy.

[gep13]

@thnk2wn said...
When an error happens here, Pop-Location isn't executed so it leaves things at a different location than where things were when the script started, leading to possible issues later.

Agreed, this should be handled in the case of an error. Again, would you mind raising a separate issue about this?

[gep13]

@thnk2wn said....
Ideally there'd be a way to specify the NuGet version (like when referencing Nuget package tools/addins) of NuGet.exe itself instead of having to place a version here and commit to source control. I don't think the Cake build script could even download NuGet as it would have to happen before #tool or #addin and I think those have to be first.

This is possible today.

With the 0.22.0 release of Cake (which was announced here https://cakebuild.net/blog/2017/09/cake-v0.22.0-released) we introduced an opt-in in-proc nuget client for resolving dependencies for your build scripts. As a result, if you choose to go down this route, you don't need to have the bootstrapper do the work of resolving the NuGet.exe. Instead, you can have this inline in your build.cake file like any other tool. For example, you could do this:

#tool "nuget:https://www.nuget.org/api/v2?package=NuGet.CommandLine&version=4.3.0"

NOTE: It doesn't look like the preview version of the NuGet.CommandLine is pushed to NuGet.org, but I suspect that it is pushed to MyGet somewhere.

NOTE: This process will only work using the opt-in in-proc version of the NuGet Client, and will require changes to your bootstrapper to make sure everything works as you want it.

[gep13]

@thnk2wn @ahoefling hopefully I have addressed all your points, and concerns. If not, then please feel free to post back here.

[SkyeHoefling]

@gep13 said...
The default bootstrapper attempts to be a little bit smart, and it will only download NuGet.exe if it needs to. For example, if when you run the bootstrapper NuGet.exe is already on your path, then the bootstrapper won't download it into the local tools folder.

The big change that I made to the bootsrapper:

• Check the version of NuGet and if the version is < 4.4 then download version 4.4

@gep13 said...
That is not strictly true. While today, the Cake Bootstrapper is downloading NuGet.exe version 4.3, it is not "tied" to it. The download URL that we are using is this ...

I might have missed something in the bootstrapper but I did not see anywhere in the script did it check version number. That means just by updating the nuget url as suggested in that comment would not work if nuget is already installed. I added a simple check into my bootsrapper to handle that and then update nuget to 4.4

If I missed something in the bootstrapper that checks version numbers can you please point it out to me. I prefer to keep my bootstrapper as close to mainline as I can.

[SkyeHoefling]

@gep13 thanks for the fast response on all of this!!

[devlead]

This relates to NuGet/Home#6144