From 31306780b5afed3b93cf04c0c73c39030ae67d29 Mon Sep 17 00:00:00 2001 From: Dean Ellis Date: Wed, 29 Jan 2020 12:46:24 +0000 Subject: [PATCH 1/5] [Xamarin.Android.Build.Tasks] Run `aapt2 compile` incrementally. The `aapt2 compile` command runs in two modes. The one we currently use is the `archive` mode. We calling `aapt2 compile` with the `--dir` argument you end up generating one `flata` archive for all the files. The side effect of this is that even if you only change one file, it will need to regenerate the entire `flata` archive. But it has a second mode. Rather than using `--dir` you just send in a single file. This then writes a single `.flat` file to the output directory. While this does mean you have to call `aapt2 compile` for EVERY file, it does mean we can leverage MSbuilds support for partial targets. This means MSbuild will detect ONLY the files which changed and allow us to call `aapt2 compile` with just THOSE files. One exception to this new system are references which use the `AndroidSkipResourceProcessing` metadata. In those cases the chance of those libraries being updated on a regular basis is quite low. So in that case using a `flata` archive will be better since the files won't be changing much. While this may impact on initial build times, the goal is to make incremental builds quicker. This is especially true for users to use ALLOT of `AndroidResource` items. A note regarding the `aapt2 daemon` mode. In order to write accented characters we need to set the `StandardInput` encoding to UTF8. This is not possible directly in netstandard 2.0. So we have to use `Console.InputEncoding` instead. Also not that we MUST not include a BOM when writting the commands. This is because `aapt2` will try to parse the BOM as command characters. Also the `aapt2 link` command sometimes reports it is "Done" before it has even written the archive for the file. So we can get into a position where we think we are done but the file is not on disk. So we have had to include a nasty wait which will poll for the existence of the expected output file and only return when it exists. The good news is we know at this point if the command failed or not, so we can bypass the check on failure. --- .vscode/settings.json | 3 +- .../Android/Xamarin.Android.Aapt.targets | 13 + .../Android/Xamarin.Android.Aapt2.targets | 171 ++++----- .../Tasks/Aapt2.cs | 113 +++--- .../Tasks/Aapt2Compile.cs | 100 ++++-- .../Tasks/Aapt2Link.cs | 189 ++++++---- .../Tasks/AndroidComputeResPaths.cs | 4 + .../Tasks/CollectNonEmptyDirectories.cs | 67 +++- .../Tasks/ConvertCustomView.cs | 15 +- .../Tasks/CopyIfChanged.cs | 9 +- .../Tasks/PrepareWearApplicationFiles.cs | 5 + .../AndroidUpdateResourcesTest.cs | 29 +- .../Xamarin.Android.Build.Tests/BuildTest.cs | 19 +- .../IncrementalBuildTest.cs | 34 +- .../Tasks/Aapt2Tests.cs | 330 +++++++++++++++++- .../Tasks/CopyIfChangedTests.cs | 40 +++ .../Tasks/ManagedResourceParserTests.cs | 6 +- .../Utilities/MockBuildEngine.cs | 17 +- .../Android/KnownPackages.cs | 22 ++ .../Utilities/Aapt2Daemon.cs | 259 ++++++++++++++ .../Utilities/AndroidResource.cs | 17 +- .../Utilities/Files.cs | 2 + .../Utilities/OutputLine.cs | 8 +- .../Xamarin.Android.Common.targets | 53 ++- src/aapt2/aapt2.targets | 2 +- .../Tests/PerformanceTest.cs | 13 + .../MSBuildDeviceIntegration.csv | 1 + 27 files changed, 1238 insertions(+), 303 deletions(-) create mode 100644 src/Xamarin.Android.Build.Tasks/Utilities/Aapt2Daemon.cs diff --git a/.vscode/settings.json b/.vscode/settings.json index 157e2faf4cf..9b018dda9b5 100644 --- a/.vscode/settings.json +++ b/.vscode/settings.json @@ -4,5 +4,6 @@ "bin/TestDebug/MSBuildDeviceIntegration/MSBuildDeviceIntegration.dll", "bin/TestDebug/Xamarin.Android.Build.Tests.dll", "bin/TestDebug/Xamarin.Android.Build.Tests.Commercial.dll", - ] + ], + "cmake.configureOnOpen": false } \ No newline at end of file diff --git a/src/Xamarin.Android.Build.Tasks/MSBuild/Xamarin/Android/Xamarin.Android.Aapt.targets b/src/Xamarin.Android.Build.Tasks/MSBuild/Xamarin/Android/Xamarin.Android.Aapt.targets index d9ce1e0beff..561a41f7f67 100644 --- a/src/Xamarin.Android.Build.Tasks/MSBuild/Xamarin/Android/Xamarin.Android.Aapt.targets +++ b/src/Xamarin.Android.Build.Tasks/MSBuild/Xamarin/Android/Xamarin.Android.Aapt.targets @@ -16,6 +16,19 @@ Copyright (C) 2019 Microsoft Corporation. All rights reserved. + + + <_UpdateAndroidResgenInputs> + $(_UpdateAndroidResgenInputs); + @(_LibraryResourceDirectoryStamps); + + <_CreateBaseApkInputs> + $(_CreateBaseApkInputs); + @(_LibraryResourceDirectoryStamps); + + + + diff --git a/src/Xamarin.Android.Build.Tasks/MSBuild/Xamarin/Android/Xamarin.Android.Aapt2.targets b/src/Xamarin.Android.Build.Tasks/MSBuild/Xamarin/Android/Xamarin.Android.Aapt2.targets index ed7f367058e..6b0d6e4ec79 100644 --- a/src/Xamarin.Android.Build.Tasks/MSBuild/Xamarin/Android/Xamarin.Android.Aapt2.targets +++ b/src/Xamarin.Android.Build.Tasks/MSBuild/Xamarin/Android/Xamarin.Android.Aapt2.targets @@ -17,6 +17,30 @@ Copyright (C) 2011-2012 Xamarin. All rights reserved. + + 0 + <_Aapt2DaemonKeepInDomain Condition=" '$(_Aapt2DaemonKeepInDomain)' == '' ">false + + + + + + <_SetLatestTargetFrameworkVersionDependsOnTargets> + $(_SetLatestTargetFrameworkVersionDependsOnTargets); + _CreateAapt2VersionCache; + + <_PrepareUpdateAndroidResgenDependsOnTargets> + _CompileResources; + _Aapt2UpdateAndroidResgenInputs; + $(_PrepareUpdateAndroidResgenDependsOnTargets); + + <_AfterConvertCustomView> + $(_AfterConvertCustomView); + _FixupCustomViewsForAapt2; + + + + @@ -37,7 +61,9 @@ Copyright (C) 2011-2012 Xamarin. All rights reserved. /> <_CompiledFlataArchive Include="$(_AndroidLibrayProjectIntermediatePath)**\*.flata" /> - <_CompiledFlataArchive Include="$(IntermediateOutputPath)\*.flata" /> + <_CompiledFlataArchive Include="$(_AndroidLibrayProjectIntermediatePath)**\*.flat" /> + <_CompiledFlataArchive Include="$(_AndroidLibraryFlatFilesDirectory)*.flat" /> + <_CompiledFlataArchive Include="$(_AndroidLibraryFlatArchivesDirectory)\*.flata" /> <_CompiledFlataStamp Include="$(_AndroidLibrayProjectIntermediatePath)**\compiled.stamp" /> + + - - - - - - - - - - - - - - - <_MissingStampFiles Include="@(_LibraryResourceHashDirectories->'%(StampFile)')" Condition="!Exists('%(StampFile)')" /> - <_HashStampFiles Include="@(_LibraryResourceHashDirectories->'$(_AndroidLibraryFlatArchivesDirectory)%(Hash).stamp')" /> - <_HashFlataFiles Include="@(_LibraryResourceHashDirectories->'$(_AndroidLibraryFlatArchivesDirectory)%(Hash).flata')" /> - - - - - - - + + + + <_CompileResourcesInputs Include="@(_AndroidResourceDest)"> + %(Identity) + + <_CompiledFlatFiles Include="@(_CompileResourcesInputs->'%(_ArchiveDirectory)%(_FlatFile)')" /> + + + - - - - + + + + + <_UpdateAndroidResgenInputs> + $(_UpdateAndroidResgenInputs); + @(_CompiledFlatFiles); + @(_LibraryResourceDirectoryStamps); + + <_CreateBaseApkInputs> + $(_CreateBaseApkInputs); + @(_CompiledFlatFiles); + @(_LibraryResourceDirectoryStamps); + + + Condition=" '$(_AndroidUseAapt2)' == 'True' " + > --no-version-vectors $(AndroidAapt2LinkExtraArgs) <_Aapt2ProguardRules Condition=" '$(AndroidLinkTool)' != '' ">$(IntermediateOutputPath)aapt_rules.txt @@ -158,6 +161,8 @@ Copyright (C) 2011-2012 Xamarin. All rights reserved. - + + <_ItemsToFixup Include="@(_CompileResourcesInputs)" Condition=" '@(_ProcessedCustomViews->'%(Identity)')' == '%(Identity)' "/> + - + - + + Condition=" '$(_AndroidUseAapt2)' == 'True' " + > <_ProtobufFormat Condition=" '$(AndroidPackageFormat)' == 'aab' ">True <_ProtobufFormat Condition=" '$(_ProtobufFormat)' == '' ">False resource_name_case_map; + public int DaemonMaxInstanceCount { get; set; } + + public bool DaemonKeepInDomain { get; set; } public ITaskItem [] ResourceDirectories { get; set; } @@ -39,81 +44,61 @@ protected string ResourceDirectoryFullPath (string resourceDirectory) return (Path.IsPathRooted (resourceDirectory) ? resourceDirectory : Path.Combine (WorkingDirectory, resourceDirectory)).TrimEnd ('\\'); } + protected string GetFullPath (string dir) + { + return (Path.IsPathRooted (dir) ? dir : Path.GetFullPath (Path.Combine (WorkingDirectory, dir))); + } + protected string GenerateFullPathToTool () { return Path.Combine (ToolPath, string.IsNullOrEmpty (ToolExe) ? ToolName : ToolExe); } - protected bool RunAapt (string commandLine, IList output) + protected virtual int GetRequiredDaemonInstances () { - var stdout_completed = new ManualResetEvent (false); - var stderr_completed = new ManualResetEvent (false); - var psi = new ProcessStartInfo () { - FileName = GenerateFullPathToTool (), - Arguments = commandLine, - UseShellExecute = false, - RedirectStandardOutput = true, - RedirectStandardError = true, - StandardOutputEncoding = Encoding.UTF8, - CreateNoWindow = true, - WindowStyle = ProcessWindowStyle.Hidden, - WorkingDirectory = WorkingDirectory, - }; - object lockObject = new object (); - using (var proc = new Process ()) { - proc.OutputDataReceived += (sender, e) => { - if (e.Data != null) - lock (lockObject) - output.Add (new OutputLine (e.Data, stdError: false)); - else - stdout_completed.Set (); - }; - proc.ErrorDataReceived += (sender, e) => { - if (e.Data != null) - lock (lockObject) - output.Add (new OutputLine (e.Data, stdError: !IsAapt2Warning (e.Data))); - else - stderr_completed.Set (); - }; - LogDebugMessage ("Executing {0}", commandLine); - proc.StartInfo = psi; - proc.Start (); - proc.BeginOutputReadLine (); - proc.BeginErrorReadLine (); - CancellationToken.Register (() => { - try { - proc.Kill (); - } catch (Exception) { - } - }); - proc.WaitForExit (); - if (psi.RedirectStandardError) - stderr_completed.WaitOne (TimeSpan.FromSeconds (30)); - if (psi.RedirectStandardOutput) - stdout_completed.WaitOne (TimeSpan.FromSeconds (30)); - return proc.ExitCode == 0 && !output.Any (x => x.StdError); - } + return 1; + } + + Aapt2Daemon daemon; + + internal Aapt2Daemon Daemon => daemon; + public override bool Execute () + { + // Must register on the UI thread! + // We don't want to use up ALL the available cores especially when + // running in the IDE. So lets cap it at DefaultMaxAapt2Daemons (6). + int maxInstances = Math.Min (Environment.ProcessorCount-1, DefaultMaxAapt2Daemons); + if (DaemonMaxInstanceCount == 0) + DaemonMaxInstanceCount = maxInstances; + else + DaemonMaxInstanceCount = Math.Min (DaemonMaxInstanceCount, maxInstances); + daemon = Aapt2Daemon.GetInstance (BuildEngine4, GenerateFullPathToTool (), + DaemonMaxInstanceCount, GetRequiredDaemonInstances (), registerInDomain: DaemonKeepInDomain); + return base.Execute (); } - bool IsAapt2Warning (string singleLine) + ConcurrentBag jobs = new ConcurrentBag (); + + protected long RunAapt (string [] args, string outputFile) { - var match = AndroidRunToolTask.AndroidErrorRegex.Match (singleLine.Trim ()); - if (match.Success) { - var file = match.Groups ["file"].Value; - var level = match.Groups ["level"].Value.ToLowerInvariant (); - var message = match.Groups ["message"].Value; - if (singleLine.StartsWith ($"{ToolName} W", StringComparison.OrdinalIgnoreCase)) - return true; - if (file.StartsWith ("W/", StringComparison.OrdinalIgnoreCase)) - return true; - if (message.Contains ("warn:")) - return true; - if (level.Contains ("warning")) - return true; - } - return false; + LogDebugMessage ($"Executing {string.Join (" ", args)}"); + long jobid = daemon.QueueCommand (args, outputFile); + jobs.Add (jobid); + return jobid; } + protected void ProcessOutput () + { + Aapt2Daemon.Job[] completedJobs = Daemon.WaitForJobsToComplete (jobs); + foreach (var job in completedJobs) { + foreach (var line in job.Output) { + if (!LogAapt2EventsFromOutput (line.Line, MessageImportance.Normal, job.Succeeded)) { + break; + } + } + } + } + protected bool LogAapt2EventsFromOutput (string singleLine, MessageImportance messageImportance, bool apptResult) { if (string.IsNullOrEmpty (singleLine)) diff --git a/src/Xamarin.Android.Build.Tasks/Tasks/Aapt2Compile.cs b/src/Xamarin.Android.Build.Tasks/Tasks/Aapt2Compile.cs index 786277acd30..b981e689146 100644 --- a/src/Xamarin.Android.Build.Tasks/Tasks/Aapt2Compile.cs +++ b/src/Xamarin.Android.Build.Tasks/Tasks/Aapt2Compile.cs @@ -19,54 +19,104 @@ public class Aapt2Compile : Aapt2 { public override string TaskPrefix => "A2C"; List archives = new List (); + List files = new List (); public string ExtraArgs { get; set; } public string FlatArchivesDirectory { get; set; } + public string FlatFilesDirectory { get; set; } + public ITaskItem [] ResourcesToCompile { get; set; } + [Output] public ITaskItem [] CompiledResourceFlatArchives => archives.ToArray (); - public override System.Threading.Tasks.Task RunTaskAsync () + [Output] + public ITaskItem [] CompiledResourceFlatFiles => files.ToArray (); + + protected override int GetRequiredDaemonInstances () + { + return Math.Min ((ResourcesToCompile ?? ResourceDirectories).Length, DaemonMaxInstanceCount); + } + + public async override System.Threading.Tasks.Task RunTaskAsync () { LoadResourceCaseMap (); - return this.WhenAllWithLock (ResourceDirectories, ProcessDirectory); + await this.WhenAllWithLock (ResourcesToCompile ?? ResourceDirectories, ProcessDirectory); + + ProcessOutput (); + + for (int i = archives.Count -1; i > 0; i-- ) { + if (!File.Exists (archives[i].ItemSpec)) { + archives.RemoveAt (i); + } + } } - void ProcessDirectory (ITaskItem resourceDirectory, object lockObject) + void ProcessDirectory (ITaskItem item, object lockObject) { - if (!Directory.EnumerateDirectories (resourceDirectory.ItemSpec).Any ()) + var flatFile = item.GetMetadata ("_FlatFile"); + bool isDirectory = flatFile.EndsWith (".flata", StringComparison.OrdinalIgnoreCase); + if (string.IsNullOrEmpty (flatFile)) { + FileAttributes fa = File.GetAttributes (item.ItemSpec); + isDirectory = (fa & FileAttributes.Directory) == FileAttributes.Directory; + } + + string fileOrDirectory = item.GetMetadata ("ResourceDirectory"); + if (string.IsNullOrEmpty (fileOrDirectory) || !isDirectory) + fileOrDirectory = item.ItemSpec; + if (isDirectory && !Directory.EnumerateDirectories (fileOrDirectory).Any ()) return; - var output = new List (); - var hash = resourceDirectory.GetMetadata ("Hash"); - var filename = !string.IsNullOrEmpty (hash) ? hash : "compiled"; - var outputArchive = Path.Combine (FlatArchivesDirectory, $"{filename}.flata"); - var success = RunAapt (GenerateCommandLineCommands (resourceDirectory, outputArchive), output); - if (success && File.Exists (Path.Combine (WorkingDirectory, outputArchive))) { - lock (lockObject) - archives.Add (new TaskItem (outputArchive)); + string outputArchive = isDirectory ? GetFullPath (FlatArchivesDirectory) : GetFullPath (FlatFilesDirectory); + string targetDir = item.GetMetadata ("_ArchiveDirectory"); + if (!string.IsNullOrEmpty (targetDir)) { + outputArchive = GetFullPath (targetDir); } - foreach (var line in output) { - if (!LogAapt2EventsFromOutput (line.Line, MessageImportance.Normal, success)) - break; + Directory.CreateDirectory (outputArchive); + string expectedOutputFile; + if (isDirectory) { + if (string.IsNullOrEmpty (flatFile)) + flatFile = item.GetMetadata ("Hash"); + var filename = !string.IsNullOrEmpty (flatFile) ? flatFile : "compiled"; + if (!filename.EndsWith (".flata", StringComparison.OrdinalIgnoreCase)) + filename = $"{filename}.flata"; + outputArchive = Path.Combine (outputArchive, filename); + expectedOutputFile = outputArchive; + } else { + expectedOutputFile = Path.Combine (outputArchive, flatFile); + } + RunAapt (GenerateCommandLineCommands (fileOrDirectory, isDirectory, outputArchive), expectedOutputFile); + if (isDirectory) { + lock (lockObject) + archives.Add (new TaskItem (expectedOutputFile)); + } else { + lock (lockObject) + files.Add (new TaskItem (expectedOutputFile)); } } - protected string GenerateCommandLineCommands (ITaskItem dir, string outputArchive) + protected string[] GenerateCommandLineCommands (string fileOrDirectory, bool isDirectory, string outputArchive) { - var cmd = new CommandLineBuilder (); - cmd.AppendSwitch ("compile"); - cmd.AppendSwitchIfNotNull ("-o ", outputArchive); - if (!string.IsNullOrEmpty (ResourceSymbolsTextFile)) - cmd.AppendSwitchIfNotNull ("--output-text-symbols ", ResourceSymbolsTextFile); - cmd.AppendSwitchIfNotNull ("--dir ", dir.ItemSpec.TrimEnd ('\\')); + List cmd = new List (); + cmd.Add ("compile"); + cmd.Add ($"-o"); + cmd.Add (GetFullPath (outputArchive)); + if (!string.IsNullOrEmpty (ResourceSymbolsTextFile)) { + cmd.Add ($"--output-text-symbols"); + cmd.Add (GetFullPath (ResourceSymbolsTextFile)); + } + if (isDirectory) { + cmd.Add ("--dir"); + cmd.Add (GetFullPath (fileOrDirectory).TrimEnd ('\\')); + } else + cmd.Add (GetFullPath (fileOrDirectory)); if (!string.IsNullOrEmpty (ExtraArgs)) - cmd.AppendSwitch (ExtraArgs); + cmd.Add (ExtraArgs); if (MonoAndroidHelper.LogInternalExceptions) - cmd.AppendSwitch ("-v"); - return cmd.ToString (); + cmd.Add ("-v"); + return cmd.ToArray (); } } diff --git a/src/Xamarin.Android.Build.Tasks/Tasks/Aapt2Link.cs b/src/Xamarin.Android.Build.Tasks/Tasks/Aapt2Link.cs index bc50cfa711e..24e9c591a58 100644 --- a/src/Xamarin.Android.Build.Tasks/Tasks/Aapt2Link.cs +++ b/src/Xamarin.Android.Build.Tasks/Tasks/Aapt2Link.cs @@ -17,6 +17,7 @@ namespace Xamarin.Android.Tasks { //aapt2 link -o resources.apk.bk --manifest Foo.xml --java . --custom-package com.infinitespace_studios.blankforms -R foo2 -v --auto-add-overlay public class Aapt2Link : Aapt2 { + static Regex exraArgSplitRegEx = new Regex (@"[\""].+?[\""]|[\''].+?[\'']|[^ ]+", RegexOptions.Compiled | RegexOptions.IgnoreCase | RegexOptions.Multiline); public override string TaskPrefix => "A2L"; [Required] @@ -38,6 +39,8 @@ public class Aapt2Link : Aapt2 { public ITaskItem CompiledResourceFlatArchive { get; set; } + public ITaskItem [] CompiledResourceFlatFiles { get; set; } + public string AndroidComponentResgenFlagFile { get; set; } public string AssetsDirectory { get; set; } @@ -78,6 +81,13 @@ public class Aapt2Link : Aapt2 { AssemblyIdentityMap assemblyMap = new AssemblyIdentityMap (); List tempFiles = new List (); + Dictionary apks = new Dictionary (); + string proguardRuleOutputTemp; + + protected override int GetRequiredDaemonInstances () + { + return Math.Min (CreatePackagePerAbi ? (SupportedAbis?.Length ?? 1) : 1, DaemonMaxInstanceCount); + } public async override System.Threading.Tasks.Task RunTaskAsync () { @@ -86,7 +96,36 @@ public async override System.Threading.Tasks.Task RunTaskAsync () assemblyMap.Load (Path.Combine (WorkingDirectory, AssemblyIdentityMapFile)); + proguardRuleOutputTemp = GetTempFile (); + await this.WhenAll (ManifestFiles, ProcessManifest); + + ProcessOutput (); + // now check for + foreach (var kvp in apks) { + string currentResourceOutputFile = kvp.Key; + bool aaptResult = Daemon.JobSucceded (kvp.Value); + LogDebugMessage ($"Processing {currentResourceOutputFile} JobId: {kvp.Value} Exists: {File.Exists (currentResourceOutputFile)} JobWorked: {aaptResult}"); + if (!string.IsNullOrEmpty (currentResourceOutputFile)) { + var tmpfile = currentResourceOutputFile + ".bk"; + // aapt2 might not produce an archive and we must provide + // and -o foo even if we don't want one. + if (File.Exists (tmpfile)) { + if (aaptResult) { + LogDebugMessage ($"Copying {tmpfile} to {currentResourceOutputFile}"); + MonoAndroidHelper.CopyIfZipChanged (tmpfile, currentResourceOutputFile); + } + File.Delete (tmpfile); + } + // Delete the archive on failure + if (!aaptResult && File.Exists (currentResourceOutputFile)) { + LogDebugMessage ($"Link did not succeed. Deleting {currentResourceOutputFile}"); + File.Delete (currentResourceOutputFile); + } + } + } + if (!string.IsNullOrEmpty (ProguardRuleOutput)) + MonoAndroidHelper.CopyIfChanged (proguardRuleOutputTemp, ProguardRuleOutput); } finally { lock (tempFiles) { foreach (var temp in tempFiles) { @@ -97,13 +136,9 @@ public async override System.Threading.Tasks.Task RunTaskAsync () } } - string GenerateCommandLineCommands (string ManifestFile, string currentAbi, string currentResourceOutputFile) + string [] GenerateCommandLineCommands (string ManifestFile, string currentAbi, string currentResourceOutputFile) { - var cmd = new CommandLineBuilder (); - cmd.AppendSwitch ("link"); - if (MonoAndroidHelper.LogInternalExceptions) - cmd.AppendSwitch ("-v"); - + List cmd = new List (); string manifestDir = Path.Combine (Path.GetDirectoryName (ManifestFile), currentAbi != null ? currentAbi : "manifest"); Directory.CreateDirectory (manifestDir); string manifestFile = Path.Combine (manifestDir, Path.GetFileName (ManifestFile)); @@ -114,7 +149,7 @@ string GenerateCommandLineCommands (string ManifestFile, string currentAbi, stri manifest.CalculateVersionCode (currentAbi, VersionCodePattern, VersionCodeProperties); } catch (ArgumentOutOfRangeException ex) { LogCodedError ("XA0003", ManifestFile, 0, ex.Message); - return string.Empty; + return cmd.ToArray (); } } if (currentAbi != null && string.IsNullOrEmpty (VersionCodePattern)) { @@ -122,25 +157,38 @@ string GenerateCommandLineCommands (string ManifestFile, string currentAbi, stri } if (!manifest.ValidateVersionCode (out string error, out string errorCode)) { LogCodedError (errorCode, ManifestFile, 0, error); - return string.Empty; + return cmd.ToArray (); } manifest.ApplicationName = ApplicationName; manifest.Save (LogCodedWarning, manifestFile); - cmd.AppendSwitchIfNotNull ("--manifest ", manifestFile); + cmd.Add ("link"); + if (MonoAndroidHelper.LogInternalExceptions) + cmd.Add ("-v"); + cmd.Add ($"--manifest"); + cmd.Add (GetFullPath (manifestFile)); if (!string.IsNullOrEmpty (JavaDesignerOutputDirectory)) { var designerDirectory = Path.IsPathRooted (JavaDesignerOutputDirectory) ? JavaDesignerOutputDirectory : Path.Combine (WorkingDirectory, JavaDesignerOutputDirectory); Directory.CreateDirectory (designerDirectory); - cmd.AppendSwitchIfNotNull ("--java ", JavaDesignerOutputDirectory); + cmd.Add ("--java"); + cmd.Add (GetFullPath (JavaDesignerOutputDirectory)); + } + if (PackageName != null) { + cmd.Add ("--custom-package"); + cmd.Add (PackageName.ToLowerInvariant ()); } - if (PackageName != null) - cmd.AppendSwitchIfNotNull ("--custom-package ", PackageName.ToLowerInvariant ()); if (AdditionalResourceArchives != null) { for (int i = AdditionalResourceArchives.Length - 1; i >= 0; i--) { var flata = Path.Combine (WorkingDirectory, AdditionalResourceArchives [i].ItemSpec); - if (File.Exists (flata)) { - cmd.AppendSwitchIfNotNull ("-R ", flata); + if (Directory.Exists (flata)) { + foreach (var line in Directory.EnumerateFiles (flata, "*.flat", SearchOption.TopDirectoryOnly)) { + cmd.Add ("-R"); + cmd.Add (GetFullPath (line)); + } + } else if (File.Exists (flata)) { + cmd.Add ("-R"); + cmd.Add (GetFullPath (flata)); } else { LogDebugMessage ("Archive does not exist: " + flata); } @@ -149,49 +197,91 @@ string GenerateCommandLineCommands (string ManifestFile, string currentAbi, stri if (CompiledResourceFlatArchive != null) { var flata = Path.Combine (WorkingDirectory, CompiledResourceFlatArchive.ItemSpec); - if (File.Exists (flata)) { - cmd.AppendSwitchIfNotNull ("-R ", flata); + if (Directory.Exists (flata)) { + foreach (var line in Directory.EnumerateFiles (flata, "*.flat", SearchOption.TopDirectoryOnly)) { + cmd.Add ("-R"); + cmd.Add (GetFullPath (line)); + } + } else if (File.Exists (flata)) { + cmd.Add ("-R"); + cmd.Add (GetFullPath (flata)); } else { LogDebugMessage ("Archive does not exist: " + flata); } } + + if (CompiledResourceFlatFiles != null) { + List appFiles = new List (); + for (int i = CompiledResourceFlatFiles.Length - 1; i >= 0; i--) { + var file = CompiledResourceFlatFiles [i]; + if (!string.IsNullOrEmpty (file.GetMetadata ("ResourceDirectory")) && File.Exists (file.ItemSpec)) { + cmd.Add ("-R"); + cmd.Add (GetFullPath (file.ItemSpec)); + } else { + appFiles.Add(file); + } + } + foreach (var file in appFiles) { + if (File.Exists (file.ItemSpec)) { + cmd.Add ("-R"); + cmd.Add (GetFullPath (file.ItemSpec)); + } + } + } - cmd.AppendSwitch ("--auto-add-overlay"); + cmd.Add ("--auto-add-overlay"); if (!string.IsNullOrWhiteSpace (UncompressedFileExtensions)) - foreach (var ext in UncompressedFileExtensions.Split (new char [] { ';', ',' }, StringSplitOptions.RemoveEmptyEntries)) - cmd.AppendSwitchIfNotNull ("-0 ", ext.StartsWith (".", StringComparison.OrdinalIgnoreCase) ? ext : $".{ext}"); + foreach (var ext in UncompressedFileExtensions.Split (new char [] { ';', ',' }, StringSplitOptions.RemoveEmptyEntries)) { + cmd.Add ("-0"); + cmd.Add (ext.StartsWith (".", StringComparison.OrdinalIgnoreCase) ? ext : $".{ext}"); + } - if (!string.IsNullOrEmpty (ExtraPackages)) - cmd.AppendSwitchIfNotNull ("--extra-packages ", ExtraPackages); + if (!string.IsNullOrEmpty (ExtraPackages)) { + cmd.Add ("--extra-packages"); + cmd.Add (ExtraPackages); + } - cmd.AppendSwitchIfNotNull ("-I ", JavaPlatformJarPath); + cmd.Add ("-I"); + cmd.Add (GetFullPath (JavaPlatformJarPath)); - if (!string.IsNullOrEmpty (ResourceSymbolsTextFile)) - cmd.AppendSwitchIfNotNull ("--output-text-symbols ", ResourceSymbolsTextFile); + if (!string.IsNullOrEmpty (ResourceSymbolsTextFile)) { + cmd.Add ("--output-text-symbols"); + cmd.Add (GetFullPath (ResourceSymbolsTextFile)); + } if (ProtobufFormat) - cmd.AppendSwitch ("--proto-format"); + cmd.Add ("--proto-format"); var extraArgsExpanded = ExpandString (ExtraArgs); if (extraArgsExpanded != ExtraArgs) LogDebugMessage (" ExtraArgs expanded: {0}", extraArgsExpanded); - if (!string.IsNullOrWhiteSpace (extraArgsExpanded)) - cmd.AppendSwitch (extraArgsExpanded); + if (!string.IsNullOrWhiteSpace (extraArgsExpanded)) { + foreach (Match match in exraArgSplitRegEx.Matches (extraArgsExpanded)) { + string value = match.Value.Trim (' ', '"', '\''); + if (!string.IsNullOrEmpty (value)) + cmd.Add (value); + } + } if (!string.IsNullOrWhiteSpace (AssetsDirectory)) { var assetDir = AssetsDirectory.TrimEnd ('\\'); if (!Path.IsPathRooted (assetDir)) assetDir = Path.Combine (WorkingDirectory, assetDir); - if (!string.IsNullOrWhiteSpace (assetDir) && Directory.Exists (assetDir)) - cmd.AppendSwitchIfNotNull ("-A ", assetDir); + if (!string.IsNullOrWhiteSpace (assetDir) && Directory.Exists (assetDir)) { + cmd.Add ("-A"); + cmd.Add (GetFullPath (assetDir)); + } } if (!string.IsNullOrEmpty (ProguardRuleOutput)) { - cmd.AppendSwitchIfNotNull ("--proguard ", ProguardRuleOutput); + cmd.Add ("--proguard"); + cmd.Add (GetFullPath (proguardRuleOutputTemp)); } - cmd.AppendSwitchIfNotNull ("-o ", currentResourceOutputFile); - return cmd.ToString (); + cmd.Add ("-o"); + cmd.Add (GetFullPath (currentResourceOutputFile)); + + return cmd.ToArray (); } string ExpandString (string s) @@ -212,33 +302,10 @@ string ExpandString (string s) return s; } - bool ExecuteForAbi (string cmd, string currentResourceOutputFile) + bool ExecuteForAbi (string [] cmd, string currentResourceOutputFile) { - var output = new List (); - var aaptResult = RunAapt (cmd, output); - var success = !string.IsNullOrEmpty (currentResourceOutputFile) - ? File.Exists (Path.Combine (currentResourceOutputFile + ".bk")) - : aaptResult; - foreach (var line in output) { - if (!LogAapt2EventsFromOutput (line.Line, MessageImportance.Normal, success)) - break; - } - if (!string.IsNullOrEmpty (currentResourceOutputFile)) { - var tmpfile = currentResourceOutputFile + ".bk"; - // aapt2 might not produce an archive and we must provide - // and -o foo even if we don't want one. - if (File.Exists (tmpfile)) { - if (aaptResult) { - MonoAndroidHelper.CopyIfZipChanged (tmpfile, currentResourceOutputFile); - } - File.Delete (tmpfile); - } - // Delete the archive on failure - if (!aaptResult && File.Exists (currentResourceOutputFile)) { - File.Delete (currentResourceOutputFile); - } - } - return aaptResult; + apks.Add (currentResourceOutputFile, RunAapt (cmd, currentResourceOutputFile)); + return true; } bool ManifestIsUpToDate (string manifestFile) @@ -250,7 +317,7 @@ bool ManifestIsUpToDate (string manifestFile) void ProcessManifest (ITaskItem manifestFile) { - var manifest = Path.IsPathRooted (manifestFile.ItemSpec) ? manifestFile.ItemSpec : Path.Combine (WorkingDirectory, manifestFile.ItemSpec); + var manifest = GetFullPath (manifestFile.ItemSpec); if (!File.Exists (manifest)) { LogDebugMessage ("{0} does not exists. Skipping", manifest); return; @@ -275,8 +342,8 @@ void ProcessManifest (ITaskItem manifestFile) var currentResourceOutputFile = abi != null ? string.Format ("{0}-{1}", outputFile, abi) : outputFile; if (!string.IsNullOrEmpty (currentResourceOutputFile) && !Path.IsPathRooted (currentResourceOutputFile)) currentResourceOutputFile = Path.Combine (WorkingDirectory, currentResourceOutputFile); - string cmd = GenerateCommandLineCommands (manifest, abi, currentResourceOutputFile); - if (string.IsNullOrWhiteSpace (cmd) || !ExecuteForAbi (cmd, currentResourceOutputFile)) { + string[] cmd = GenerateCommandLineCommands (manifest, abi, currentResourceOutputFile); + if (!cmd.Any () || !ExecuteForAbi (cmd, currentResourceOutputFile)) { Cancel (); } } diff --git a/src/Xamarin.Android.Build.Tasks/Tasks/AndroidComputeResPaths.cs b/src/Xamarin.Android.Build.Tasks/Tasks/AndroidComputeResPaths.cs index fa08507d2b3..1060d59b6eb 100644 --- a/src/Xamarin.Android.Build.Tasks/Tasks/AndroidComputeResPaths.cs +++ b/src/Xamarin.Android.Build.Tasks/Tasks/AndroidComputeResPaths.cs @@ -49,6 +49,8 @@ public class AndroidComputeResPaths : AndroidTask public bool LowercaseFilenames { get; set; } public string ProjectDir { get; set; } + + public string AndroidLibraryFlatFilesDirectory { get; set; } [Output] public ITaskItem[] IntermediateFiles { get; set; } @@ -131,6 +133,8 @@ public override bool RunTask () } var newItem = new TaskItem (dest); newItem.SetMetadata ("LogicalName", rel); + newItem.SetMetadata ("_FlatFile", Monodroid.AndroidResource.CalculateAapt2FlatArchiveFileName (dest)); + newItem.SetMetadata ("_ArchiveDirectory", AndroidLibraryFlatFilesDirectory); item.CopyMetadataTo (newItem); intermediateFiles.Add (newItem); resolvedFiles.Add (item); diff --git a/src/Xamarin.Android.Build.Tasks/Tasks/CollectNonEmptyDirectories.cs b/src/Xamarin.Android.Build.Tasks/Tasks/CollectNonEmptyDirectories.cs index 902790c794b..018f9dc4db9 100644 --- a/src/Xamarin.Android.Build.Tasks/Tasks/CollectNonEmptyDirectories.cs +++ b/src/Xamarin.Android.Build.Tasks/Tasks/CollectNonEmptyDirectories.cs @@ -1,4 +1,6 @@ +using System; using System.Collections.Generic; +using System.Text; using System.IO; using System.Linq; using Microsoft.Build.Utilities; @@ -11,6 +13,7 @@ public class CollectNonEmptyDirectories : AndroidTask { public override string TaskPrefix => "CNE"; List output = new List (); + List libraryResourceFiles = new List (); [Required] public ITaskItem[] Directories { get; set; } @@ -24,6 +27,9 @@ public class CollectNonEmptyDirectories : AndroidTask { [Output] public ITaskItem[] Output => output.ToArray (); + [Output] + public ITaskItem[] LibraryResourceFiles => libraryResourceFiles.ToArray (); + public override bool RunTask () { var libraryProjectDir = Path.GetFullPath (LibraryProjectIntermediatePath); @@ -32,27 +38,64 @@ public override bool RunTask () Log.LogDebugMessage ($"Directory does not exist, skipping: {directory.ItemSpec}"); continue; } - var firstFile = Directory.EnumerateFiles(directory.ItemSpec, "*.*", SearchOption.AllDirectories).FirstOrDefault (); - if (firstFile != null) { + string stampFile = directory.GetMetadata ("StampFile"); + if (string.IsNullOrEmpty (stampFile)) { + if (Path.GetFullPath (directory.ItemSpec).StartsWith (libraryProjectDir)) { + // If inside the `lp` directory + stampFile = Path.GetFullPath (Path.Combine (directory.ItemSpec, "..", "..")) + ".stamp"; + } else { + // Otherwise use a hashed stamp file + stampFile = Path.Combine (StampDirectory, Files.HashString (directory.ItemSpec) + ".stamp"); + } + } + + bool generateArchive = false; + bool.TryParse (directory.GetMetadata (ResolveLibraryProjectImports.AndroidSkipResourceProcessing), out generateArchive); + + IEnumerable files; + string fileCache = Path.Combine (directory.ItemSpec, "..", "files.cache"); + DateTime lastwriteTime = File.Exists (stampFile) ? File.GetLastWriteTimeUtc (stampFile) : DateTime.MinValue; + DateTime cacheLastWriteTime = File.Exists (fileCache) ? File.GetLastWriteTimeUtc (fileCache) : DateTime.MinValue; + + if (File.Exists (fileCache) && cacheLastWriteTime >= lastwriteTime) { + Log.LogDebugMessage ($"Reading cached Library resources list from {fileCache}"); + files = File.ReadAllLines (fileCache); + } else { + if (!File.Exists (fileCache)) + Log.LogDebugMessage ($"Cached Library resources list {fileCache} does not exist."); + else + Log.LogDebugMessage ($"Cached Library resources list {fileCache} is out of date."); + if (generateArchive) { + files = new string[1] { stampFile }; + } else { + files = Directory.EnumerateFiles(directory.ItemSpec, "*.*", SearchOption.AllDirectories); + } + } + + if (files.Any ()) { + if (!File.Exists (fileCache) || cacheLastWriteTime < lastwriteTime) + File.WriteAllLines (fileCache, files, Encoding.UTF8); var taskItem = new TaskItem (directory.ItemSpec, new Dictionary () { - {"FileFound", firstFile }, + {"FileFound", files.First () }, }); directory.CopyMetadataTo (taskItem); - string stampFile = directory.GetMetadata ("StampFile"); - if (string.IsNullOrEmpty (stampFile)) { - if (Path.GetFullPath (directory.ItemSpec).StartsWith (libraryProjectDir)) { - // If inside the `lp` directory - stampFile = Path.GetFullPath (Path.Combine (directory.ItemSpec, "..", "..")) + ".stamp"; - } else { - // Otherwise use a hashed stamp file - stampFile = Path.Combine (StampDirectory, Files.HashString (directory.ItemSpec) + ".stamp"); - } + if (string.IsNullOrEmpty (directory.GetMetadata ("StampFile"))) { taskItem.SetMetadata ("StampFile", stampFile); } else { Log.LogDebugMessage ($"%(StampFile) already set: {stampFile}"); } output.Add (taskItem); + foreach (var file in files) { + var fileTaskItem = new TaskItem (file, new Dictionary () { + { "ResourceDirectory", directory.ItemSpec }, + { "StampFile", generateArchive ? stampFile : file }, + { "Hash", stampFile }, + { "_ArchiveDirectory", Path.Combine (directory.ItemSpec, "..", "flat" + Path.DirectorySeparatorChar) }, + { "_FlatFile", generateArchive ? $"{Path.GetFileNameWithoutExtension (stampFile)}.flata" : Monodroid.AndroidResource.CalculateAapt2FlatArchiveFileName (file) }, + }); + libraryResourceFiles.Add (fileTaskItem); + } } } return !Log.HasLoggedErrors; diff --git a/src/Xamarin.Android.Build.Tasks/Tasks/ConvertCustomView.cs b/src/Xamarin.Android.Build.Tasks/Tasks/ConvertCustomView.cs index e5dafc22e4f..889150fef5b 100644 --- a/src/Xamarin.Android.Build.Tasks/Tasks/ConvertCustomView.cs +++ b/src/Xamarin.Android.Build.Tasks/Tasks/ConvertCustomView.cs @@ -79,24 +79,23 @@ public override bool RunTask () } } } - var output = new List (); + var output = new Dictionary (); foreach (var file in processed) { ITaskItem resdir = ResourceDirectories?.FirstOrDefault (x => file.StartsWith (x.ItemSpec)) ?? null; - if (resdir == null) { + if (output.ContainsKey (file)) continue; - } - var hash = resdir.GetMetadata ("Hash"); - var stamp = resdir.GetMetadata ("StampFile"); + var hash = resdir?.GetMetadata ("Hash") ?? null; + var stamp = resdir?.GetMetadata ("StampFile") ?? null; var filename = !string.IsNullOrEmpty (hash) ? hash : "compiled"; var stampFile = !string.IsNullOrEmpty (stamp) ? stamp : $"{filename}.stamp"; Log.LogDebugMessage ($"{filename} {stampFile}"); - output.Add (new TaskItem (file, new Dictionary { + output.Add (file, new TaskItem (Path.GetFullPath (file), new Dictionary { { "StampFile" , stampFile }, { "Hash" , filename }, - { "ResourceDirectory", resdir.ItemSpec } + { "Resource", resdir?.ItemSpec ?? file }, })); } - Processed = output.ToArray (); + Processed = output.Values.ToArray (); return !Log.HasLoggedErrors; } diff --git a/src/Xamarin.Android.Build.Tasks/Tasks/CopyIfChanged.cs b/src/Xamarin.Android.Build.Tasks/Tasks/CopyIfChanged.cs index 0a47c816315..10f81fd3e34 100644 --- a/src/Xamarin.Android.Build.Tasks/Tasks/CopyIfChanged.cs +++ b/src/Xamarin.Android.Build.Tasks/Tasks/CopyIfChanged.cs @@ -24,11 +24,18 @@ public class CopyIfChanged : AndroidTask [Required] public ITaskItem[] DestinationFiles { get; set; } + public bool CompareFileLengths { get; set; } + [Output] public ITaskItem[] ModifiedFiles { get; set; } private List modifiedFiles = new List(); + public CopyIfChanged () + { + CompareFileLengths = true; + } + public override bool RunTask () { if (SourceFiles.Length != DestinationFiles.Length) @@ -41,7 +48,7 @@ public override bool RunTask () continue; } var dest = new FileInfo (DestinationFiles [i].ItemSpec); - if (dest.Exists && dest.LastWriteTimeUtc > src.LastWriteTimeUtc && dest.Length == src.Length) { + if (dest.Exists && dest.LastWriteTimeUtc > src.LastWriteTimeUtc && (CompareFileLengths ? dest.Length == src.Length : true)) { Log.LogDebugMessage ($" Skipping {src} it is up to date"); continue; } diff --git a/src/Xamarin.Android.Build.Tasks/Tasks/PrepareWearApplicationFiles.cs b/src/Xamarin.Android.Build.Tasks/Tasks/PrepareWearApplicationFiles.cs index 0b65c6ea200..ab86ce3834b 100644 --- a/src/Xamarin.Android.Build.Tasks/Tasks/PrepareWearApplicationFiles.cs +++ b/src/Xamarin.Android.Build.Tasks/Tasks/PrepareWearApplicationFiles.cs @@ -17,6 +17,7 @@ public class PrepareWearApplicationFiles : AndroidTask public string PackageName { get; set; } public string WearAndroidManifestFile { get; set; } public string IntermediateOutputPath { get; set; } + public string AndroidLibraryFlatFilesDirectory { get; set; } public string WearApplicationApkPath { get; set; } [Output] public ITaskItem WearableApplicationDescriptionFile { get; set; } @@ -62,8 +63,12 @@ public override bool RunTask () modified.Add (intermediateXmlFile); } WearableApplicationDescriptionFile = new TaskItem (intermediateXmlFile); + WearableApplicationDescriptionFile.SetMetadata ("_FlatFile", Monodroid.AndroidResource.CalculateAapt2FlatArchiveFileName (intermediateXmlFile)); + WearableApplicationDescriptionFile.SetMetadata ("_ArchiveDirectory", AndroidLibraryFlatFilesDirectory); WearableApplicationDescriptionFile.SetMetadata ("IsWearApplicationResource", "True"); BundledWearApplicationApkResourceFile = new TaskItem (intermediateApkPath); + BundledWearApplicationApkResourceFile.SetMetadata ("_FlatFile", Monodroid.AndroidResource.CalculateAapt2FlatArchiveFileName (intermediateApkPath)); + BundledWearApplicationApkResourceFile.SetMetadata ("_ArchiveDirectory", AndroidLibraryFlatFilesDirectory); BundledWearApplicationApkResourceFile.SetMetadata ("IsWearApplicationResource", "True"); ModifiedFiles = modified.ToArray (); diff --git a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/AndroidUpdateResourcesTest.cs b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/AndroidUpdateResourcesTest.cs index 980f6615cdd..d0b7d77ba97 100644 --- a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/AndroidUpdateResourcesTest.cs +++ b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/AndroidUpdateResourcesTest.cs @@ -252,6 +252,11 @@ public void RepetiviteBuildUpdateSingleResource ([Values (false, true)] bool use Assert.IsTrue ( b.Output.IsTargetSkipped ("_LinkAssembliesNoShrink"), "The Target _LinkAssembliesNoShrink should have been skipped"); + if (useAapt2) { + Assert.IsTrue ( + b.Output.IsTargetSkipped ("_CompileResources"), + "The target _CompileResources should have been skipped"); + } image1.Timestamp = DateTimeOffset.UtcNow; var layout = proj.AndroidResources.First (x => x.Include() == "Resources\\layout\\Main.axml"); layout.Timestamp = DateTimeOffset.UtcNow; @@ -271,6 +276,11 @@ public void RepetiviteBuildUpdateSingleResource ([Values (false, true)] bool use Assert.IsFalse ( b.Output.IsTargetSkipped ("_CreateBaseApk"), "The Target _CreateBaseApk should not have been skipped"); + if (useAapt2) { + Assert.IsTrue ( + b.Output.IsTargetPartiallyBuilt ("_CompileResources"), + "The target _CompileResources should have been partially built"); + } } } @@ -502,9 +512,9 @@ void CheckCustomView (Xamarin.Tools.Zip.ZipArchive zip, params string [] paths) var doc = XDocument.Load (customViewPath); Assert.IsNotNull (doc.Element ("LinearLayout"), "PreferenceScreen should be present in preferences.xml"); Assert.IsNull (doc.Element ("LinearLayout").Element ("Classlibrary1.CustomTextView"), - "Classlibrary1.CustomTextView should have been replaced with an $(Hash).CustomTextView"); + $"Classlibrary1.CustomTextView should have been replaced with an $(Hash).CustomTextView in {customViewPath}"); Assert.IsNull (doc.Element ("LinearLayout").Element ("classlibrary1.CustomTextView"), - "classlibrary1.CustomTextView should have been replaced with an $(Hash).CustomTextView"); + $"classlibrary1.CustomTextView should have been replaced with an $(Hash).CustomTextView in {customViewPath}"); //Now check the zip var customViewInZip = "res/layout/" + Path.GetFileName (customViewPath); @@ -520,9 +530,9 @@ void CheckCustomView (Xamarin.Tools.Zip.ZipArchive zip, params string [] paths) // Don't use `StringAssert` because `contents` make the failure message unreadable. var contents = reader.ReadToEnd (); Assert.IsFalse (contents.Contains ("Classlibrary1.CustomTextView"), - "Classlibrary1.CustomTextView should have been replaced with an $(Hash).CustomTextView"); + $"Classlibrary1.CustomTextView should have been replaced with an $(Hash).CustomTextView in {customViewInZip} in package"); Assert.IsFalse (contents.Contains ("classlibrary1.CustomTextView"), - "classlibrary1.CustomTextView should have been replaced with an $(Hash).CustomTextView"); + $"classlibrary1.CustomTextView should have been replaced with an $(Hash).CustomTextView in {customViewInZip} in package"); } } } @@ -1030,7 +1040,7 @@ public void BuildAppWithManagedResourceParserAndLibraries () Assert.LessOrEqual (libBuilder.LastBuildTime.TotalMilliseconds, maxBuildTimeMs, $"DesignTime build should be less than {maxBuildTimeMs} milliseconds."); Assert.IsFalse (libProj.CreateBuildOutput (libBuilder).IsTargetSkipped ("_ManagedUpdateAndroidResgen"), "Target '_ManagedUpdateAndroidResgen' should have run."); - Assert.AreEqual (!appBuilder.RunningMSBuild, appBuilder.DesignTimeBuild (appProj), "Application project should have built"); + Assert.IsFalse (appBuilder.DesignTimeBuild (appProj), "Application project should have built"); Assert.LessOrEqual (appBuilder.LastBuildTime.TotalMilliseconds, maxBuildTimeMs, $"DesignTime build should be less than {maxBuildTimeMs} milliseconds."); Assert.IsFalse (appProj.CreateBuildOutput (appBuilder).IsTargetSkipped ("_ManagedUpdateAndroidResgen"), "Target '_ManagedUpdateAndroidResgen' should have run."); @@ -1259,7 +1269,14 @@ public void CustomViewAddResourceId ([Values (false, true)] bool useAapt2) proj.Touch (@"Resources\layout\Main.axml"); Assert.IsTrue (b.Build (proj, doNotCleanupOnUpdate: true), "second build should have succeeded"); - + if (useAapt2) { + Assert.IsTrue ( + b.Output.IsTargetPartiallyBuilt ("_CompileResources"), + "The target _CompileResources should have been partially built"); + Assert.IsTrue ( + b.Output.IsTargetSkipped ("_FixupCustomViewsForAapt2"), + "The target _FixupCustomViewsForAapt2 should have been skipped"); + } var r_java = Path.Combine (Root, b.ProjectDirectory, proj.IntermediateOutputPath, "android", "src", "unnamedproject", "unnamedproject", "R.java"); FileAssert.Exists (r_java); var r_java_contents = File.ReadAllLines (r_java); diff --git a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/BuildTest.cs b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/BuildTest.cs index f268e9884f7..80ce82063aa 100644 --- a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/BuildTest.cs +++ b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/BuildTest.cs @@ -333,14 +333,27 @@ public void BuildXamarinFormsMapsApplication () var proj = new XamarinFormsMapsApplicationProject (); using (var b = CreateApkBuilder (Path.Combine ("temp", TestName))) { Assert.IsTrue (b.Build (proj), "first should have succeeded."); + b.BuildLogFile = "build2.log"; Assert.IsTrue (b.Build (proj, doNotCleanupOnUpdate: true, saveProject: false), "second should have succeeded."); var targets = new [] { - "_CompileAndroidLibraryResources", + "_CompileResources", "_UpdateAndroidResgen", }; foreach (var target in targets) { Assert.IsTrue (b.Output.IsTargetSkipped (target), $"`{target}` should be skipped."); } + proj.Touch ("MainPage.xaml"); + b.BuildLogFile = "build3.log"; + Assert.IsTrue (b.Build (proj, doNotCleanupOnUpdate: true, saveProject: false), "third should have succeeded."); + foreach (var target in targets) { + Assert.IsTrue (b.Output.IsTargetSkipped (target), $"`{target}` should be skipped."); + } + Assert.IsFalse (b.Output.IsTargetSkipped ("CoreCompile"), $"`CoreCompile` should not be skipped."); + b.BuildLogFile = "build4.log"; + Assert.IsTrue (b.Build (proj, doNotCleanupOnUpdate: true, saveProject: false), "forth should have succeeded."); + foreach (var target in targets) { + Assert.IsTrue (b.Output.IsTargetSkipped (target), $"`{target}` should be skipped."); + } } } @@ -2780,6 +2793,7 @@ public void CompileBeforeUpgradingNuGet () proj.PackageReferences.Add (KnownPackages.SupportV7MediaRouter_27_0_2_1); using (var b = CreateApkBuilder (Path.Combine ("temp", TestName))) { + b.ThrowOnBuildFailure = false; var projectDir = Path.Combine (Root, b.ProjectDirectory); if (Directory.Exists (projectDir)) Directory.Delete (projectDir, true); @@ -2787,7 +2801,8 @@ public void CompileBeforeUpgradingNuGet () proj.PackageReferences.Clear (); //NOTE: we can get all the other dependencies transitively, yay! - proj.PackageReferences.Add (KnownPackages.XamarinForms_4_0_0_425677); + proj.PackageReferences.Add (KnownPackages.XamarinForms_4_4_0_991265); + Assert.IsTrue (b.Restore (proj, doNotCleanupOnUpdate: true), "Restore should have worked."); Assert.IsTrue (b.Build (proj, saveProject: true, doNotCleanupOnUpdate: true), "second build should have succeeded."); Assert.IsTrue (StringAssertEx.ContainsText (b.LastBuildOutput, "Refreshing Xamarin.Android.Support.v7.AppCompat.dll"), "`ResolveLibraryProjectImports` should not skip `Xamarin.Android.Support.v7.AppCompat.dll`!"); Assert.IsTrue (StringAssertEx.ContainsText (b.LastBuildOutput, "Deleting unknown jar: support-annotations.jar"), "`support-annotations.jar` should be deleted!"); diff --git a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/IncrementalBuildTest.cs b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/IncrementalBuildTest.cs index 619f37cfbec..40f49d9fb27 100644 --- a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/IncrementalBuildTest.cs +++ b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/IncrementalBuildTest.cs @@ -599,12 +599,26 @@ public CustomTextView(Context context, IAttributeSet attributes) : base(context, using (var libBuilder = CreateDllBuilder (Path.Combine (path, lib.ProjectName), false)) using (var appBuilder = CreateApkBuilder (Path.Combine (path, app.ProjectName))) { + libBuilder.BuildLogFile = "build.log"; Assert.IsTrue (libBuilder.Build (lib), "first library build should have succeeded."); + appBuilder.BuildLogFile = "build.log"; Assert.IsTrue (appBuilder.Build (app), "first app build should have succeeded."); + if (useAapt2) { + var aapt2TargetsShouldRun = new [] { + "_FixupCustomViewsForAapt2", + "_CompileResources" + }; + foreach (var target in aapt2TargetsShouldRun) { + Assert.IsFalse (appBuilder.Output.IsTargetSkipped (target), $"{target} should run!"); + } + } + lib.Touch ("Bar.cs"); + libBuilder.BuildLogFile = "build2.log"; Assert.IsTrue (libBuilder.Build (lib, doNotCleanupOnUpdate: true, saveProject: false), "second library build should have succeeded."); + appBuilder.BuildLogFile = "build2.log"; Assert.IsTrue (appBuilder.Build (app, doNotCleanupOnUpdate: true, saveProject: false), "second app build should have succeeded."); var targetsShouldSkip = new [] { @@ -627,6 +641,16 @@ public CustomTextView(Context context, IAttributeSet attributes) : base(context, foreach (var target in targetsShouldRun) { Assert.IsFalse (appBuilder.Output.IsTargetSkipped (target), $"`{target}` should *not* be skipped!"); } + + if (useAapt2) { + var aapt2TargetsShouldBeSkipped = new [] { + "_FixupCustomViewsForAapt2", + "_CompileResources" + }; + foreach (var target in aapt2TargetsShouldBeSkipped) { + Assert.IsTrue (appBuilder.Output.IsTargetSkipped (target), $"{target} should be skipped!"); + } + } } } @@ -687,7 +711,7 @@ public void ResolveLibraryProjectImports ([Values (true, false)] bool useAapt2) "_CreateBaseApk", }; if (useAapt2) { - targets.Add ("_ConvertLibraryResourcesCases"); + targets.Add ("_ConvertResourcesCases"); } foreach (var targetName in targets) { Assert.IsTrue (b.Output.IsTargetSkipped (targetName), $"`{targetName}` should be skipped!"); @@ -1102,6 +1126,14 @@ public void AndroidXMigrationBug () source = source.Replace ("Foo", "Bar"); proj.Touch ("Foo.cs"); Assert.IsTrue (b.Build (proj, doNotCleanupOnUpdate: true), "second build should have succeeded."); + var targets = new [] { + "_CompileResources", + "_UpdateAndroidResgen", + "_GenerateAndroidResourceDir", + }; + foreach (var target in targets) { + Assert.IsTrue (b.Output.IsTargetSkipped (target), $"`{target}` should be skipped."); + } } } } diff --git a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Tasks/Aapt2Tests.cs b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Tasks/Aapt2Tests.cs index 772f6073b43..91ce92ef4a5 100644 --- a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Tasks/Aapt2Tests.cs +++ b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Tasks/Aapt2Tests.cs @@ -1,3 +1,4 @@ +using System; using System.Collections.Generic; using System.IO; using System.Linq; @@ -12,32 +13,114 @@ namespace Xamarin.Android.Build.Tests { public class Aapt2Tests : BaseTest { - void CallAapt2Compile (IBuildEngine engine, string dir, string outputPath, List output = null) + void CallAapt2Compile (IBuildEngine engine, string dir, string outputPath, string flatFilePath, List output = null, int maxInstances = 0, bool keepInDomain = false) { var errors = new List (); + ITaskItem item; + if (File.Exists (dir)) { + item = CreateTaskItemForResourceFile (dir); + } else { + item = new TaskItem(dir, new Dictionary { + { "ResourceDirectory", dir }, + { "Hash", output != null ? Files.HashString (dir) : "compiled" }, + { "_FlatFile", output != null ? Files.HashString (dir) + ".flata" : "compiled.flata" }, + { "_ArchiveDirectory", outputPath } + }); + } var task = new Aapt2Compile { BuildEngine = engine, ToolPath = GetPathToAapt2 (), + ResourcesToCompile = new ITaskItem [] { + item, + }, ResourceDirectories = new ITaskItem [] { - new TaskItem(dir, new Dictionary { - { "Hash", output != null ? Files.HashString (dir) : "compiled" } - }), + new TaskItem (dir), }, FlatArchivesDirectory = outputPath, + FlatFilesDirectory = flatFilePath, + DaemonMaxInstanceCount = maxInstances, + DaemonKeepInDomain = keepInDomain, }; - Assert.True (task.Execute (), "task should have succeeded."); + MockBuildEngine mockEngine = (MockBuildEngine)engine; + Assert.True (task.Execute (), $"task should have succeeded. {string.Join (" ", mockEngine.Errors.Select (x => x.Message))}"); output?.AddRange (task.CompiledResourceFlatArchives); } + ITaskItem CreateTaskItemForResourceFile (string root, string dir, string file) + { + string ext = Path.GetExtension (file); + if (dir.StartsWith ("values")) + ext = ".arsc"; + return new TaskItem (Path.Combine (root, dir, file), new Dictionary { { "_FlatFile", $"{dir}_{Path.GetFileNameWithoutExtension (file)}{ext}.flat" } } ); + } + + ITaskItem CreateTaskItemForResourceFile (string file) + { + string ext = Path.GetExtension (file); + string dir = Path.GetFileName (Path.GetDirectoryName (file)); + if (dir.StartsWith ("values")) + ext = ".arsc"; + return new TaskItem (file, new Dictionary { { "_FlatFile", $"{dir}_{Path.GetFileNameWithoutExtension (file)}{ext}.flat" } } ); + } + [Test] - public void Aapt2Link () + [TestCase (6, 6, 3, 2)] + [TestCase (6, 6, 2, 1)] + [TestCase (6, 6, 6, 50)] + [TestCase (1, 1, 1, 10)] + public void Aapt2DaemonInstances (int maxInstances, int expectedMax, int expectedInstances, int numLayouts) { - var path = Path.Combine (Root, "temp", "Aapt2Link"); + var path = Path.Combine (Root, "temp", TestName); Directory.CreateDirectory (path); var resPath = Path.Combine (path, "res"); var archivePath = Path.Combine (path, "flata"); + var flatFilePath = Path.Combine(path, "flat"); Directory.CreateDirectory (resPath); Directory.CreateDirectory (archivePath); + Directory.CreateDirectory (flatFilePath); + Directory.CreateDirectory (Path.Combine (resPath, "values")); + Directory.CreateDirectory (Path.Combine (resPath, "layout")); + File.WriteAllText (Path.Combine (resPath, "values", "strings.xml"), @"foo"); + for (int i = 0; i < numLayouts; i++) + File.WriteAllText (Path.Combine (resPath, "layout", $"main{i}.xml"), @""); + File.WriteAllText (Path.Combine (path, "AndroidManifest.xml"), @""); + File.WriteAllText (Path.Combine (path, "foo.map"), @"a\nb"); + var errors = new List (); + var warnings = new List (); + List files = new List (); + IBuildEngine4 engine = new MockBuildEngine (System.Console.Out, errors, warnings); + files.Add (CreateTaskItemForResourceFile (resPath, "values", "strings.xml")); + for (int i = 0; i < numLayouts; i++) + files.Add (CreateTaskItemForResourceFile (resPath, "layout", $"main{i}.xml")); + var task = new Aapt2Compile { + BuildEngine = engine, + ToolPath = GetPathToAapt2 (), + ResourcesToCompile = files.ToArray (), + ResourceDirectories = new ITaskItem [] { new TaskItem (resPath) }, + FlatArchivesDirectory = archivePath, + FlatFilesDirectory = flatFilePath, + DaemonMaxInstanceCount = maxInstances, + DaemonKeepInDomain = false, + }; + Assert.True (task.Execute (), $"task should have succeeded. {string.Join (";", errors.Select (x => x.Message))}"); + Aapt2Daemon daemon = (Aapt2Daemon)engine.GetRegisteredTaskObject (typeof(Aapt2Daemon).FullName, RegisteredTaskObjectLifetime.Build); + Assert.IsNotNull (daemon, "Should have got a Daemon"); + Assert.AreEqual (expectedMax, daemon.MaxInstances, $"Expected {expectedMax} but was {daemon.MaxInstances}"); + Assert.AreEqual (expectedInstances, daemon.CurrentInstances, $"Expected {expectedInstances} but was {daemon.CurrentInstances}"); + Directory.Delete (Path.Combine (Root, path), recursive: true); + } + + [Test] + public void Aapt2Link ([Values (true, false)] bool compilePerFile) + { + var path = Path.Combine (Root, "temp", TestName); + Directory.CreateDirectory (path); + var resPath = Path.Combine (path, "res"); + var archivePath = Path.Combine (path, "flata"); + var flatFilePath = Path.Combine(path, "flat"); + Directory.CreateDirectory (resPath); + Directory.CreateDirectory (archivePath); + Directory.CreateDirectory (flatFilePath); Directory.CreateDirectory (Path.Combine (resPath, "values")); Directory.CreateDirectory (Path.Combine (resPath, "layout")); File.WriteAllText (Path.Combine (resPath, "values", "strings.xml"), @"foo"); @@ -54,9 +137,22 @@ public void Aapt2Link () var warnings = new List (); IBuildEngine engine = new MockBuildEngine (TestContext.Out, errors, warnings); var archives = new List(); - CallAapt2Compile (engine, resPath, archivePath); - CallAapt2Compile (engine, Path.Combine (libPath, "0", "res"), archivePath, archives); - CallAapt2Compile (engine, Path.Combine (libPath, "1", "res"), archivePath, archives); + if (compilePerFile) { + foreach (var file in Directory.EnumerateFiles (resPath, "*.*", SearchOption.AllDirectories)) { + CallAapt2Compile (engine, file, archivePath, flatFilePath); + } + } else { + CallAapt2Compile (engine, resPath, archivePath, flatFilePath); + } + CallAapt2Compile (engine, Path.Combine (libPath, "0", "res"), archivePath, flatFilePath, archives); + CallAapt2Compile (engine, Path.Combine (libPath, "1", "res"), archivePath, flatFilePath, archives); + List items = new List (); + if (compilePerFile) { + // collect all the flat archives + foreach (var file in Directory.EnumerateFiles (flatFilePath, "*.flat", SearchOption.AllDirectories)) { + items.Add (new TaskItem (file)); + } + } int platform = 0; using (var b = new Builder ()) { platform = b.GetMaxInstalledPlatform (); @@ -67,13 +163,14 @@ public void Aapt2Link () ToolPath = GetPathToAapt2 (), ResourceDirectories = new ITaskItem [] { new TaskItem (resPath) }, ManifestFiles = new ITaskItem [] { new TaskItem (Path.Combine (path, "AndroidManifest.xml")) }, - AdditionalResourceArchives = archives.ToArray (), - CompiledResourceFlatArchive = new TaskItem (Path.Combine (archivePath, "compiled.flata")), + AdditionalResourceArchives = !compilePerFile ? archives.ToArray () : null, + CompiledResourceFlatArchive = !compilePerFile ? new TaskItem (Path.Combine (archivePath, "compiled.flata")) : null, + CompiledResourceFlatFiles = compilePerFile ? items.ToArray () : null, OutputFile = outputFile, AssemblyIdentityMapFile = Path.Combine (path, "foo.map"), JavaPlatformJarPath = Path.Combine (AndroidSdkPath, "platforms", $"android-{platform}", "android.jar"), }; - Assert.True (task.Execute (), "task should have succeeded."); + Assert.True (task.Execute (), $"task should have succeeded. {string.Join (";", errors.Select (x => x.Message))}"); Assert.AreEqual (0, errors.Count, "There should be no errors."); Assert.LessOrEqual (0, warnings.Count, "There should be 0 warnings."); Assert.True (File.Exists (outputFile), $"{outputFile} should have been created."); @@ -90,6 +187,7 @@ public void Aapt2Compile () Directory.CreateDirectory (path); var resPath = Path.Combine (path, "res"); var archivePath = Path.Combine(path, "flata"); + var flatFilePath = Path.Combine(path, "flat"); Directory.CreateDirectory(resPath); Directory.CreateDirectory(archivePath); Directory.CreateDirectory (Path.Combine (resPath, "values")); @@ -101,8 +199,15 @@ public void Aapt2Compile () var task = new Aapt2Compile { BuildEngine = engine, ToolPath = GetPathToAapt2 (), + ResourcesToCompile = new ITaskItem [] { + new TaskItem (resPath, new Dictionary () { + { "ResourceDirectory", resPath }, + } + ) + }, ResourceDirectories = new ITaskItem [] { new TaskItem (resPath) }, FlatArchivesDirectory = archivePath, + FlatFilesDirectory = flatFilePath, }; Assert.True (task.Execute (), "task should have succeeded."); var flatArchive = Path.Combine (archivePath, "compiled.flata"); @@ -113,6 +218,130 @@ public void Aapt2Compile () Directory.Delete (Path.Combine (Root, path), recursive: true); } + [Test] + public void Aapt2CompileUmlautsAndSpaces () + { + var path = Path.Combine (Root, "temp", "Aapt2CompileÜmläüt Files"); + Directory.CreateDirectory (path); + var resPath = Path.Combine (path, "res"); + var archivePath = Path.Combine(path, "flata"); + var flatFilePath = Path.Combine(path, "flat"); + Directory.CreateDirectory(resPath); + Directory.CreateDirectory(archivePath); + Directory.CreateDirectory(flatFilePath); + Directory.CreateDirectory (Path.Combine (resPath, "values")); + Directory.CreateDirectory (Path.Combine (resPath, "layout")); + File.WriteAllText (Path.Combine (resPath, "values", "strings.xml"), @"foo"); + File.WriteAllText (Path.Combine (resPath, "layout", "main.xml"), @""); + List files = new List (); + files.Add (CreateTaskItemForResourceFile (resPath, "values", "strings.xml")); + files.Add (CreateTaskItemForResourceFile (resPath, "layout", "main.xml")); + var errors = new List (); + IBuildEngine engine = new MockBuildEngine (TestContext.Out, errors); + var task = new Aapt2Compile { + BuildEngine = engine, + ToolPath = GetPathToAapt2 (), + ResourcesToCompile = files.ToArray (), + ResourceDirectories = new ITaskItem [] { new TaskItem (resPath) }, + FlatArchivesDirectory = archivePath, + FlatFilesDirectory = flatFilePath, + }; + Assert.True (task.Execute (), $"task should have succeeded. {string.Join (";", errors.Select (x => x.Message))}"); + var flatArchive = Path.Combine (archivePath, "compiled.flata"); + Assert.False (File.Exists (flatArchive), $"{flatArchive} should not have been created."); + foreach (var file in files) { + string f = Path.Combine (flatFilePath, file.GetMetadata ("_FlatFile")); + Assert.True (File.Exists (f), $"{f} should have been created."); + } + Directory.Delete (Path.Combine (Root, path), recursive: true); + } + + [Test] + public void CollectNonEmptyDirectoriesTest () + { + var path = Path.Combine (Root, "temp", TestName); + Directory.CreateDirectory (path); + var resPath = Path.Combine (path, "res"); + var archivePath = Path.Combine (path, "flata"); + var flatFilePath = Path.Combine(path, "flat"); + Directory.CreateDirectory (resPath); + Directory.CreateDirectory (Path.Combine (path, "stamps")); + Directory.CreateDirectory (archivePath); + Directory.CreateDirectory (flatFilePath); + Directory.CreateDirectory (Path.Combine (resPath, "values")); + Directory.CreateDirectory (Path.Combine (resPath, "layout")); + File.WriteAllText (Path.Combine (resPath, "values", "strings.xml"), @"foo"); + File.WriteAllText (Path.Combine (resPath, "layout", "main.xml"), @""); + var libPath = Path.Combine (path, "lp"); + Directory.CreateDirectory (libPath); + Directory.CreateDirectory (Path.Combine (libPath, "0", "res", "values")); + Directory.CreateDirectory (Path.Combine (libPath, "1", "res", "values")); + File.WriteAllText (Path.Combine (libPath, "0", "res", "values", "strings.xml"), @"foo1"); + File.WriteAllText (Path.Combine (libPath, "1", "res", "values", "strings.xml"), @"foo2"); + File.WriteAllText (Path.Combine (path, "AndroidManifest.xml"), @""); + File.WriteAllText (Path.Combine (path, "foo.map"), @"a\nb"); + var errors = new List (); + IBuildEngine engine = new MockBuildEngine (TestContext.Out, errors); + var task = new CollectNonEmptyDirectories { + BuildEngine = engine, + Directories = new ITaskItem[] { + new TaskItem (resPath), + new TaskItem (Path.Combine (libPath, "0", "res"), new Dictionary { + { "AndroidSkipResourceProcessing", "True" }, + { "StampFile", "0.stamp" }, + }), + new TaskItem (Path.Combine (libPath, "1", "res")), + }, + LibraryProjectIntermediatePath = libPath, + StampDirectory = Path.Combine(path, "stamps"), + }; + Assert.True (task.Execute (), $"task should have succeeded. {string.Join (";", errors.Select (x => x.Message))}"); + Assert.AreEqual (3, task.Output.Length, "Output should have 3 items in it."); + Assert.AreEqual (4, task.LibraryResourceFiles.Length, "Output should have 3 items in it."); + Assert.AreEqual ("layout_main.xml.flat", task.LibraryResourceFiles[0].GetMetadata ("_FlatFile")); + Assert.AreEqual ("values_strings.arsc.flat", task.LibraryResourceFiles[1].GetMetadata ("_FlatFile")); + Assert.AreEqual ("0.flata", task.LibraryResourceFiles[2].GetMetadata ("_FlatFile")); + Assert.AreEqual ("values_strings.arsc.flat", task.LibraryResourceFiles[3].GetMetadata ("_FlatFile")); + } + + [Test] + public void Aapt2CompileFiles () + { + var path = Path.Combine (Root, "temp", "Aapt2CompileFiles"); + Directory.CreateDirectory (path); + var resPath = Path.Combine (path, "res"); + var archivePath = Path.Combine(path, "flata"); + var flatFilePath = Path.Combine(path, "flat"); + Directory.CreateDirectory(resPath); + Directory.CreateDirectory(archivePath); + Directory.CreateDirectory(flatFilePath); + Directory.CreateDirectory (Path.Combine (resPath, "values")); + Directory.CreateDirectory (Path.Combine (resPath, "layout")); + File.WriteAllText (Path.Combine (resPath, "values", "strings.xml"), @"foo"); + File.WriteAllText (Path.Combine (resPath, "layout", "main.xml"), @""); + List files = new List (); + files.Add (CreateTaskItemForResourceFile (resPath, "values", "strings.xml")); + files.Add (CreateTaskItemForResourceFile (resPath, "layout", "main.xml")); + var errors = new List (); + IBuildEngine engine = new MockBuildEngine (TestContext.Out, errors); + var task = new Aapt2Compile { + BuildEngine = engine, + ToolPath = GetPathToAapt2 (), + ResourcesToCompile = files.ToArray (), + ResourceDirectories = new ITaskItem [] { new TaskItem (resPath) }, + FlatArchivesDirectory = archivePath, + FlatFilesDirectory = flatFilePath, + }; + Assert.True (task.Execute (), $"task should have succeeded. {string.Join (";", errors.Select (x => x.Message))}"); + var flatArchive = Path.Combine (archivePath, "compiled.flata"); + Assert.False (File.Exists (flatArchive), $"{flatArchive} should not have been created."); + foreach (var file in files) { + string f = Path.Combine (flatFilePath, file.GetMetadata ("_FlatFile")); + Assert.True (File.Exists (f), $"{f} should have been created."); + } + Directory.Delete (Path.Combine (Root, path), recursive: true); + } + [Test] public void Aapt2CompileFixesUpErrors () { @@ -120,6 +349,7 @@ public void Aapt2CompileFixesUpErrors () Directory.CreateDirectory (path); var resPath = Path.Combine (path, "res"); var archivePath = Path.Combine(path, "flata"); + var flatFilePath = Path.Combine(path, "flat"); Directory.CreateDirectory(resPath); Directory.CreateDirectory(archivePath); Directory.CreateDirectory (Path.Combine (resPath, "values")); @@ -148,8 +378,14 @@ public void Aapt2CompileFixesUpErrors () var task = new Aapt2Compile { BuildEngine = engine, ToolPath = GetPathToAapt2 (), - ResourceDirectories = new ITaskItem [] { new TaskItem (resPath) }, - FlatArchivesDirectory = archivePath, + ResourceDirectories = new ITaskItem [] { + new TaskItem (resPath, new Dictionary () { + { "ResourceDirectory", resPath }, + } + ) + }, + FlatArchivesDirectory = archivePath, + FlatFilesDirectory = flatFilePath, ResourceNameCaseMap = $"Layout{directorySeperator}Main.xml|layout{directorySeperator}main.axml;Values{directorySeperator}Strings.xml|values{directorySeperator}strings.xml", }; Assert.False (task.Execute (), "task should not have succeeded."); @@ -171,7 +407,7 @@ public void Aapt2Disabled () Assert.IsTrue (b.Build (proj), "Build should have succeeded."); Assert.IsFalse (StringAssertEx.ContainsText (b.LastBuildOutput, "Aapt2Link"), "Aapt2Link task should not run!"); Assert.IsFalse (StringAssertEx.ContainsText (b.LastBuildOutput, "Aapt2Compile"), "Aapt2Compile task should not run!"); - Assert.IsTrue (b.Output.IsTargetSkipped ("_CreateAapt2VersionCache"), "_CreateAapt2VersionCache target should be skipped!"); + Assert.IsFalse (StringAssertEx.ContainsText (b.LastBuildOutput, "_CreateAapt2VersionCache"), "_CreateAapt2VersionCache target should not run!"); } } @@ -182,6 +418,7 @@ public void Aapt2AndroidResgenExtraArgsAreInvalid () Directory.CreateDirectory (path); var resPath = Path.Combine (path, "res"); var archivePath = Path.Combine(path, "flata"); + var flatFilePath = Path.Combine(path, "flat"); Directory.CreateDirectory(resPath); Directory.CreateDirectory(archivePath); Directory.CreateDirectory (Path.Combine (resPath, "values")); @@ -191,10 +428,16 @@ public void Aapt2AndroidResgenExtraArgsAreInvalid () File.WriteAllText (Path.Combine (path, "AndroidManifest.xml"), @""); File.WriteAllText (Path.Combine (path, "foo.map"), @"a\nb"); var errors = new List (); - IBuildEngine engine = new MockBuildEngine (TestContext.Out, errors); + var warnings = new List (); + var messages = new List (); + IBuildEngine engine = new MockBuildEngine (TestContext.Out, errors, warnings, messages); var archives = new List(); - CallAapt2Compile (engine, resPath, archivePath); + CallAapt2Compile (engine, resPath, archivePath, flatFilePath); var outputFile = Path.Combine (path, "resources.apk"); + int platform = 0; + using (var b = new Builder ()) { + platform = b.GetMaxInstalledPlatform (); + } var task = new Aapt2Link { BuildEngine = engine, ToolPath = GetPathToAapt2 (), @@ -203,6 +446,7 @@ public void Aapt2AndroidResgenExtraArgsAreInvalid () CompiledResourceFlatArchive = new TaskItem (Path.Combine (path, "compiled.flata")), OutputFile = outputFile, AssemblyIdentityMapFile = Path.Combine (path, "foo.map"), + JavaPlatformJarPath = Path.Combine (AndroidSdkPath, "platforms", $"android-{platform}", "android.jar"), ExtraArgs = "--no-crunch " }; Assert.False (task.Execute (), "task should have failed."); @@ -210,6 +454,53 @@ public void Aapt2AndroidResgenExtraArgsAreInvalid () Directory.Delete (Path.Combine (Root, path), recursive: true); } + [Test] + public void Aapt2AndroidResgenExtraArgsAreSplit () + { + var path = Path.Combine (Root, "temp", TestName); + Directory.CreateDirectory (path); + var resPath = Path.Combine (path, "res"); + var archivePath = Path.Combine(path, "flata"); + var flatFilePath = Path.Combine(path, "flat"); + Directory.CreateDirectory(resPath); + Directory.CreateDirectory(archivePath); + Directory.CreateDirectory (Path.Combine (resPath, "values")); + Directory.CreateDirectory (Path.Combine (resPath, "layout")); + File.WriteAllText (Path.Combine (resPath, "values", "strings.xml"), @"foo"); + File.WriteAllText (Path.Combine (resPath, "layout", "main.xml"), @""); + File.WriteAllText (Path.Combine (path, "AndroidManifest.xml"), @""); + File.WriteAllText (Path.Combine (path, "foo.map"), @"a\nb"); + var errors = new List (); + var warnings = new List (); + var messages = new List (); + IBuildEngine engine = new MockBuildEngine (TestContext.Out, errors, warnings, messages); + var archives = new List(); + CallAapt2Compile (engine, resPath, archivePath, flatFilePath); + var outputFile = Path.Combine (path, "resources.apk"); + int platform = 0; + string emitids = Path.Combine (path, "emitids.txt"); + string Rtxt = Path.Combine (path, "R.txt"); + using (var b = new Builder ()) { + platform = b.GetMaxInstalledPlatform (); + } + var task = new Aapt2Link { + BuildEngine = engine, + ToolPath = GetPathToAapt2 (), + ResourceDirectories = new ITaskItem [] { new TaskItem (resPath) }, + ManifestFiles = new ITaskItem [] { new TaskItem (Path.Combine (path, "AndroidManifest.xml")) }, + CompiledResourceFlatArchive = new TaskItem (Path.Combine (archivePath, "compiled.flata")), + OutputFile = outputFile, + AssemblyIdentityMapFile = Path.Combine (path, "foo.map"), + JavaPlatformJarPath = Path.Combine (AndroidSdkPath, "platforms", $"android-{platform}", "android.jar"), + ExtraArgs = $@"--no-version-vectors -v --emit-ids ""{emitids}"" --output-text-symbols '{Rtxt}'" + }; + Assert.True (task.Execute (), $"task should have succeeded. {string.Join (" ", errors.Select (e => e.Message))}"); + Assert.AreEqual (0, errors.Count, $"No errors should have been raised. {string.Join (" ", errors.Select (e => e.Message))}"); + Assert.True (File.Exists (emitids), $"{emitids} should have been created."); + Assert.True (File.Exists (Rtxt), $"{Rtxt} should have been created."); + Directory.Delete (Path.Combine (Root, path), recursive: true); + } + [Test] [TestCase ("1.0", "", "XA0003")] [TestCase ("-1", "", "XA0004")] @@ -230,6 +521,7 @@ public void CheckForInvalidVersionCode (string versionCode, string versionCodePa }); var resPath = Path.Combine (path, "res"); var archivePath = Path.Combine (path, "flata"); + var flatFilePath = Path.Combine(path, "flat"); Directory.CreateDirectory (resPath); Directory.CreateDirectory (archivePath); Directory.CreateDirectory (Path.Combine (resPath, "values")); @@ -241,7 +533,7 @@ public void CheckForInvalidVersionCode (string versionCode, string versionCodePa var errors = new List (); IBuildEngine engine = new MockBuildEngine (TestContext.Out, errors); var archives = new List (); - CallAapt2Compile (engine, resPath, archivePath); + CallAapt2Compile (engine, resPath, archivePath, flatFilePath); var outputFile = Path.Combine (path, "resources.apk"); var manifestFile = Path.Combine (path, "AndroidManifest.xml"); var task = new Aapt2Link { diff --git a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Tasks/CopyIfChangedTests.cs b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Tasks/CopyIfChangedTests.cs index 6e2a96d9708..b6db9b48ebd 100644 --- a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Tasks/CopyIfChangedTests.cs +++ b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Tasks/CopyIfChangedTests.cs @@ -95,6 +95,26 @@ public void DestinationOlder () FileAssert.AreEqual (src, dest); } + [Test] + public void DestinationOlderNoLengthCheck () + { + var src = NewFile ("foo"); + var dest = NewFile ("bar"); + var now = DateTime.UtcNow; + File.SetLastWriteTimeUtc (src, now); + File.SetLastWriteTimeUtc (dest, now.AddSeconds (-1)); // destination is older + + var task = new CopyIfChanged { + BuildEngine = new MockBuildEngine (TestContext.Out), + SourceFiles = ToArray (src), + DestinationFiles = ToArray (dest), + CompareFileLengths = false, + }; + Assert.IsTrue (task.Execute (), "task.Execute() should have succeeded."); + Assert.AreEqual (1, task.ModifiedFiles.Length, "Changes should have been made."); + FileAssert.AreEqual (src, dest); + } + [Test] public void SourceOlder () { @@ -132,6 +152,26 @@ public void SourceOlderDifferentLength () FileAssert.AreEqual (src, dest); } + [Test] + public void SourceOlderDifferentLengthButNoLengthCheck () + { + var src = NewFile ("foo"); + var dest = NewFile ("foofoo"); + var now = DateTime.UtcNow; + File.SetLastWriteTimeUtc (src, now.AddSeconds (-1)); // source is older + File.SetLastWriteTimeUtc (dest, now); + + var task = new CopyIfChanged { + BuildEngine = new MockBuildEngine (TestContext.Out), + SourceFiles = ToArray (src), + DestinationFiles = ToArray (dest), + CompareFileLengths = false, + }; + Assert.IsTrue (task.Execute (), "task.Execute() should have succeeded."); + Assert.AreEqual (0, task.ModifiedFiles.Length, "Changes should not have been made."); + FileAssert.AreNotEqual (src, dest); + } + [Test] public void CaseChanges () { diff --git a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Tasks/ManagedResourceParserTests.cs b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Tasks/ManagedResourceParserTests.cs index 6c1016c9c92..92ef13f7a04 100644 --- a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Tasks/ManagedResourceParserTests.cs +++ b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Tasks/ManagedResourceParserTests.cs @@ -467,7 +467,8 @@ public void CompareAapt2AndManagedParserOutput () CreateResourceDirectory (path); File.WriteAllText (Path.Combine (Root, path, "foo.map"), @"a\nb"); Directory.CreateDirectory (Path.Combine (Root, path, "java")); - IBuildEngine engine = new MockBuildEngine (TestContext.Out); + List errors = new List (); + IBuildEngine engine = new MockBuildEngine (TestContext.Out, errors: errors); var aapt2Compile = new Aapt2Compile { BuildEngine = engine, ToolPath = GetPathToAapt2 (), @@ -480,9 +481,10 @@ public void CompareAapt2AndManagedParserOutput () }), }, FlatArchivesDirectory = Path.Combine (Root, path), + FlatFilesDirectory = Path.Combine (Root, path), }; - Assert.IsTrue (aapt2Compile.Execute (), "Aapt2 Compile should have succeeded."); + Assert.IsTrue (aapt2Compile.Execute (), $"Aapt2 Compile should have succeeded. {string.Join (" ", errors.Select (x => x.Message))}"); int platform = 0; using (var b = new Builder ()) { platform = b.GetMaxInstalledPlatform (); diff --git a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Utilities/MockBuildEngine.cs b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Utilities/MockBuildEngine.cs index cab12d58a59..6fc3761582d 100644 --- a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Utilities/MockBuildEngine.cs +++ b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests/Utilities/MockBuildEngine.cs @@ -1,5 +1,6 @@ using System; using System.Collections; +using System.Collections.Concurrent; using System.Collections.Generic; using System.IO; using Microsoft.Build.Framework; @@ -18,11 +19,11 @@ public MockBuildEngine (TextWriter output, IList errors = n private TextWriter Output { get; } - private IList Errors { get; } + public IList Errors { get; } - private IList Warnings { get; } + public IList Warnings { get; } - private IList Messages { get; } + public IList Messages { get; } int IBuildEngine.ColumnNumberOfTaskNode => -1; @@ -62,11 +63,15 @@ void IBuildEngine.LogWarningEvent (BuildWarningEventArgs e) Warnings.Add (e); } - private Dictionary Tasks = new Dictionary (); + private ConcurrentDictionary Tasks = new ConcurrentDictionary (); void IBuildEngine4.RegisterTaskObject (object key, object obj, RegisteredTaskObjectLifetime lifetime, bool allowEarlyCollection) { - Tasks [key] = obj; + if (key is null) + throw new ArgumentNullException ("key"); + if (obj is null) + throw new ArgumentNullException ("obj"); + Tasks.TryAdd (key, obj); } object IBuildEngine4.GetRegisteredTaskObject (object key, RegisteredTaskObjectLifetime lifetime) @@ -78,7 +83,7 @@ object IBuildEngine4.GetRegisteredTaskObject (object key, RegisteredTaskObjectLi object IBuildEngine4.UnregisterTaskObject (object key, RegisteredTaskObjectLifetime lifetime) { if (Tasks.TryGetValue (key, out object value)) { - Tasks.Remove (key); + Tasks.TryRemove (key, out value); } return value; } diff --git a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Android/KnownPackages.cs b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Android/KnownPackages.cs index 7165b65c589..9aee6e07ac8 100644 --- a/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Android/KnownPackages.cs +++ b/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Android/KnownPackages.cs @@ -221,6 +221,28 @@ public static class KnownPackages }, } }; + public static Package XamarinForms_4_4_0_991265 = new Package { + Id = "Xamarin.Forms", + Version = "4.4.0.991265", + TargetFramework = "MonoAndroid90", + References = { + new BuildItem.Reference ("Xamarin.Forms.Platform.Android") { + MetadataValues = "HintPath=..\\packages\\Xamarin.Forms.4.4.0.991265\\lib\\MonoAndroid90\\Xamarin.Forms.Platform.Android.dll" + }, + new BuildItem.Reference ("FormsViewGroup") { + MetadataValues = "HintPath=..\\packages\\Xamarin.Forms.4.4.0.991265\\lib\\MonoAndroid90\\FormsViewGroup.dll" + }, + new BuildItem.Reference ("Xamarin.Forms.Core") { + MetadataValues = "HintPath=..\\packages\\Xamarin.Forms.4.4.0.991265\\lib\\MonoAndroid90\\Xamarin.Forms.Core.dll" + }, + new BuildItem.Reference ("Xamarin.Forms.Xaml") { + MetadataValues = "HintPath=..\\packages\\Xamarin.Forms.4.4.0.991265\\lib\\MonoAndroid90\\Xamarin.Forms.Xaml.dll" + }, + new BuildItem.Reference ("Xamarin.Forms.Platform") { + MetadataValues = "HintPath=..\\packages\\Xamarin.Forms.4.4.0.991265\\lib\\MonoAndroid90\\Xamarin.Forms.Platform.dll" + }, + } + }; public static Package AndroidXMigration = new Package { Id = "Xamarin.AndroidX.Migration", Version = "1.0.0-rc1", diff --git a/src/Xamarin.Android.Build.Tasks/Utilities/Aapt2Daemon.cs b/src/Xamarin.Android.Build.Tasks/Utilities/Aapt2Daemon.cs new file mode 100644 index 00000000000..3e672418963 --- /dev/null +++ b/src/Xamarin.Android.Build.Tasks/Utilities/Aapt2Daemon.cs @@ -0,0 +1,259 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Diagnostics; +using System.IO; +using System.Text; +using System.Threading; +using Microsoft.Build.Framework; +using TPL = System.Threading.Tasks; + +namespace Xamarin.Android.Tasks +{ + internal class Aapt2Daemon : IDisposable + { + public static Aapt2Daemon GetInstance (IBuildEngine4 engine, string aapt2, int numberOfInstances, int initalNumberOfDaemons, bool registerInDomain = false) + { + var area = registerInDomain ? RegisteredTaskObjectLifetime.AppDomain : RegisteredTaskObjectLifetime.Build; + Aapt2Daemon daemon = (Aapt2Daemon)engine.GetRegisteredTaskObject (typeof (Aapt2Daemon).FullName, area); + if (daemon == null) + { + daemon = new Aapt2Daemon (aapt2, numberOfInstances, initalNumberOfDaemons); + engine.RegisterTaskObject (typeof (Aapt2Daemon).FullName, daemon, area, allowEarlyCollection: false); + } + return daemon; + } + + public class Job + { + TPL.TaskCompletionSource tcs = new TPL.TaskCompletionSource (); + List output = new List (); + public string[] Commands { get; private set; } + public long JobId { get; private set; } + public string OutputFile { get; private set; } + public bool Succeeded { get; set; } + public TPL.Task Task => tcs.Task; + public IList Output => output; + public Job (string[] commands, long jobId, string outputFile) + { + Commands = commands; + JobId = jobId; + OutputFile = outputFile; + } + + public void Complete (bool result) + { + Succeeded = !result; + tcs.TrySetResult (!result); + } + } + + readonly object lockObject = new object (); + readonly BlockingCollection pendingJobs = new BlockingCollection (); + readonly ConcurrentDictionary jobs = new ConcurrentDictionary (); + readonly CancellationTokenSource tcs = new CancellationTokenSource (); + readonly ConcurrentBag daemons = new ConcurrentBag (); + + long jobsRunning = 0; + long jobId = 0; + int maxInstances = 0; + + public CancellationToken Token => tcs.Token; + + public bool JobsInQueue => pendingJobs.Count > 0; + + public bool JobsRunning + { + get + { + return Interlocked.Read (ref jobsRunning) > 0; + } + } + public string Aapt2 { get; private set; } + + public string ToolName { get { return Path.GetFileName (Aapt2); } } + + public int MaxInstances => maxInstances; + + public int CurrentInstances => daemons.Count; + + public Aapt2Daemon (string aapt2, int maxNumberOfInstances, int initalNumberOfDaemons) + { + Aapt2 = aapt2; + maxInstances = maxNumberOfInstances; + for (int i = 0; i < initalNumberOfDaemons; i++) { + SpawnAapt2Daemon (); + } + } + + void SpawnAapt2Daemon () + { + // Don't spawn too many + if (daemons.Count >= maxInstances) + return; + var thread = new Thread (Aapt2DaemonStart) + { + IsBackground = true + }; + thread.Start (); + daemons.Add(thread); + } + + public void Dispose () + { + Stop (); + tcs.Cancel (); + } + + public long QueueCommand (string[] job, string outputFile) + { + if (!pendingJobs.IsAddingCompleted) + { + long id = Interlocked.Add (ref jobId, 1); + var j = new Job (job, id, outputFile); + jobs [j.JobId] = j; + pendingJobs.Add (j); + // if we have allot of pending jobs, spawn more daemons + if (pendingJobs.Count > (daemons.Count * 2)) { + SpawnAapt2Daemon (); + } + return j.JobId; + } + return -1; + } + + public bool JobSucceded (long jobid) { + return jobs [jobid].Succeeded; + } + + public Job [] WaitForJobsToComplete (IEnumerable jobIds) + { + List completedJobsTasks = new List (); + List results = new List (); + foreach (var job in jobIds) { + completedJobsTasks.Add (jobs [job].Task); + results.Add (jobs [job]); + } + TPL.Task.WaitAll (completedJobsTasks.ToArray ()); + return results.ToArray (); + } + + public void Stop () + { + //This will cause '_jobs.GetConsumingEnumerable' to stop blocking and exit when it's empty + pendingJobs.CompleteAdding (); + } + + private void Aapt2DaemonStart () + { + ProcessStartInfo info = new ProcessStartInfo (Aapt2) + { + Arguments = "daemon", + CreateNoWindow = true, + WindowStyle = ProcessWindowStyle.Hidden, + RedirectStandardInput = true, + RedirectStandardError = true, + RedirectStandardOutput = true, + UseShellExecute = false, + WorkingDirectory = Path.GetTempPath (), + StandardErrorEncoding = Encoding.UTF8, + StandardOutputEncoding = Encoding.UTF8, + // Cant use this cos its netstandard 2.1 only + // and we are using netstandard 2.0 + //StandardInputEncoding = Encoding.UTF8, + }; + // We need to FORCE the StandardInput to be UTF8 so we can use + // accented characters. Also DONT INCLUDE A BOM!! + // otherwise aapt2 will try to interpret the BOM as an argument. + Process aapt2; + lock (lockObject) { + Encoding current = Console.InputEncoding; + try { + Console.InputEncoding = new UTF8Encoding (false); + aapt2 = new Process (); + aapt2.StartInfo = info; + aapt2.Start (); + } finally { + Console.InputEncoding = current; + } + } + try { + foreach (var job in pendingJobs.GetConsumingEnumerable (tcs.Token)) { + Interlocked.Add (ref jobsRunning, 1); + bool errored = false; + try { + // try to write Unicode UTF8 to aapt2 + StreamWriter writer = aapt2.StandardInput; + foreach (var arg in job.Commands) + { + writer.WriteLine (arg); + } + writer.WriteLine (); + writer.Flush (); + string line; + + Queue stdError = new Queue (); + while ((line = aapt2.StandardError.ReadLine ()) != null) { + if (string.Compare (line, "Done", StringComparison.OrdinalIgnoreCase) == 0) { + break; + } + if (string.Compare (line, "Error", StringComparison.OrdinalIgnoreCase) == 0) { + errored = true; + continue; + } + // we have to queue the output because the "Done"/"Error" lines are + //written after all the messages. So to process the warnings/errors + // correctly we need to do this after we know if worked or failed. + stdError.Enqueue (line); + } + //now processed the output we queued up + while (stdError.Count > 0) { + line = stdError.Dequeue (); + job.Output.Add (new OutputLine (line, stdError: !IsAapt2Warning (line), errored: errored, jobId: job.JobId)); + } + // wait for the file we expect to be created. There can be a delay between + // the daemon saying "Done" and the file finally being written to disk. + if (!string.IsNullOrEmpty (job.OutputFile) && !errored) { + while (!File.Exists (job.OutputFile)) { + Thread.Sleep (10); + } + } + } catch (Exception ex) { + errored = true; + job.Output.Add (new OutputLine (ex.Message, stdError: true, errored: errored, job.JobId)); + } finally { + Interlocked.Decrement (ref jobsRunning); + jobs [job.JobId].Complete (errored); + } + } + } + catch (OperationCanceledException) + { + // Ignore this error. It occurs when the Task is cancelled. + } + aapt2.StandardInput.WriteLine ("quit"); + aapt2.StandardInput.WriteLine (); + aapt2.WaitForExit ((int)TimeSpan.FromSeconds (5).TotalMilliseconds); + } + + bool IsAapt2Warning (string singleLine) + { + var match = AndroidRunToolTask.AndroidErrorRegex.Match (singleLine.Trim ()); + if (match.Success) + { + var file = match.Groups ["file"].Value; + var level = match.Groups ["level"].Value.ToLowerInvariant(); + var message = match.Groups ["message"].Value; + if (singleLine.StartsWith ($"{ToolName} W", StringComparison.OrdinalIgnoreCase)) + return true; + if (file.StartsWith ("W/", StringComparison.OrdinalIgnoreCase)) + return true; + if (message.Contains ("warn:")) + return true; + if (level.Contains ("warning")) + return true; + } + return false; + } + } +} \ No newline at end of file diff --git a/src/Xamarin.Android.Build.Tasks/Utilities/AndroidResource.cs b/src/Xamarin.Android.Build.Tasks/Utilities/AndroidResource.cs index 5d53db191ee..64b53f3a5e5 100644 --- a/src/Xamarin.Android.Build.Tasks/Utilities/AndroidResource.cs +++ b/src/Xamarin.Android.Build.Tasks/Utilities/AndroidResource.cs @@ -228,5 +228,20 @@ private static string TryLowercaseValue (string value, string resourceBasePath, } return value; } + + + /// Compute the output filename that aapt2 will produce for a resource + /// for example + /// layout\main.xml => layout_main.xml.flat + /// values\values.xml -> values_values.arsc.flat + /// values\strings.xml -> values_strings.arsc.flat + public static string CalculateAapt2FlatArchiveFileName (string file) + { + var dir = Path.GetFileName (Path.GetDirectoryName (file)).TrimEnd ('\\').TrimEnd ('/'); + var ext = Path.GetExtension (file); + if (dir.StartsWith ("values", StringComparison.OrdinalIgnoreCase)) + ext = ".arsc"; + return $"{dir}_{Path.GetFileNameWithoutExtension (file)}{ext}.flat"; + } } -} +} \ No newline at end of file diff --git a/src/Xamarin.Android.Build.Tasks/Utilities/Files.cs b/src/Xamarin.Android.Build.Tasks/Utilities/Files.cs index d2bd1d00eca..e7dca14929b 100644 --- a/src/Xamarin.Android.Build.Tasks/Utilities/Files.cs +++ b/src/Xamarin.Android.Build.Tasks/Utilities/Files.cs @@ -351,6 +351,8 @@ public static bool ExtractAll (ZipArchive zip, string destination, Action$(IntermediateOutputPath)aapt2.version <_AndroidIntermediateDesignTimeBuildDirectory>$(IntermediateOutputPath)designtime\ <_AndroidLibraryFlatArchivesDirectory>$(IntermediateOutputPath)flata\ + <_AndroidLibraryFlatFilesDirectory>$(IntermediateOutputPath)flat\ <_AndroidStampDirectory>$(IntermediateOutputPath)stamp\ <_AndroidBuildIdFile>$(IntermediateOutputPath)buildid.txt <_AndroidApplicationSharedLibraryPath>$(IntermediateOutputPath)app_shared_libraries\ @@ -445,6 +446,7 @@ Copyright (C) 2011-2012 Xamarin. All rights reserved. @@ -773,7 +775,13 @@ Copyright (C) 2011-2012 Xamarin. All rights reserved. DependsOnTargets="_SetLatestTargetFrameworkVersion"> - + + <_SetLatestTargetFrameworkVersionDependsOnTargets> + _ResolveSdks; + + + + @@ -1286,7 +1294,13 @@ because xbuild doesn't support framework reference assemblies. - + @@ -1305,6 +1319,7 @@ because xbuild doesn't support framework reference assemblies. /> @@ -1472,7 +1487,27 @@ because xbuild doesn't support framework reference assemblies. + + + <_UpdateAndroidResgenInputs> + $(_UpdateAndroidResgenInputs); + @(_ModifiedResources); + + <_CreateBaseApkInputs> + $(_CreateBaseApkInputs); + @(_ModifiedResources); + + + + + + <_PrepareUpdateAndroidResgenDependsOnTargets> + _IncludeModifiedFilesInUpdateAndroidResgenInputs; + + + @@ -1507,18 +1542,18 @@ because xbuild doesn't support framework reference assemblies. <_UpdateAndroidResgenInputs> $(MSBuildAllProjects); @(_AndroidResourceDest); - @(_LibraryResourceDirectoryStamps); $(_AndroidBuildPropertiesCache); $(ProjectAssetsFile); $(_AndroidLibraryProjectImportsCache); $(_AndroidLibraryImportsCache); + @(_ModifiedResources); + DependsOnTargets="$(_UpdateAndroidResgenDependsOnTargets);$(_AfterGenerateAndroidResourceDir);_PrepareUpdateAndroidResgen"> @@ -2141,7 +2176,7 @@ because xbuild doesn't support framework reference assemblies. _GenerateJavaStubs; _ManifestMerger; _ConvertCustomView; - _FixupCustomViewsForAapt2; + $(_AfterConvertCustomView); _GenerateEnvironmentFiles; _AddStaticResources; $(_AfterAddStaticResources); @@ -2221,7 +2256,7 @@ because xbuild doesn't support framework reference assemblies. _GenerateJavaStubs; _ManifestMerger; _ConvertCustomView; - _FixupCustomViewsForAapt2; + $(_AfterConvertCustomView); _GenerateEnvironmentFiles; _GetLibraryImports; _CheckDuplicateJavaLibraries; @@ -2234,7 +2269,6 @@ because xbuild doesn't support framework reference assemblies. ;@(_AndroidResourceDest) ;@(_AndroidAssetsDest) ;$(_AcwMapFile) - ;@(_LibraryResourceDirectoryStamps) ;$(_AndroidBuildPropertiesCache) @@ -2268,7 +2302,7 @@ because xbuild doesn't support framework reference assemblies. - + <_JavaStubFiles Include="$(_AndroidIntermediateJavaSourceDirectory)**\*.java" /> @@ -2598,7 +2632,7 @@ because xbuild doesn't support framework reference assemblies. _GenerateJavaStubs; _ManifestMerger; _ConvertCustomView; - _FixupCustomViewsForAapt2; + $(_AfterConvertCustomView); $(AfterGenerateAndroidManifest); _GenerateEnvironmentFiles; _CompileJava; @@ -3113,6 +3147,7 @@ because xbuild doesn't support framework reference assemblies. + diff --git a/src/aapt2/aapt2.targets b/src/aapt2/aapt2.targets index 2894432332b..f2406d80440 100644 --- a/src/aapt2/aapt2.targets +++ b/src/aapt2/aapt2.targets @@ -1,7 +1,7 @@ - 3.5.0-5435860 + 3.5.3-5435860 ResolveReferences; _DownloadAapt2; diff --git a/tests/MSBuildDeviceIntegration/Tests/PerformanceTest.cs b/tests/MSBuildDeviceIntegration/Tests/PerformanceTest.cs index 8eb49ff3d3a..216ee1f14d5 100644 --- a/tests/MSBuildDeviceIntegration/Tests/PerformanceTest.cs +++ b/tests/MSBuildDeviceIntegration/Tests/PerformanceTest.cs @@ -62,11 +62,24 @@ void Profile (ProjectBuilder builder, Action action, [CallerMemb action (builder); var actual = builder.LastBuildTime.TotalMilliseconds; + TestContext.Out.WriteLine($"expected: {expected}ms, actual: {actual}ms"); if (actual > expected) { Assert.Fail ($"Exceeded expected time of {expected}ms, actual {actual}ms"); } } + [Test] + public void Build_From_Clean_DontIncludeRestore () + { + var proj = new XamarinAndroidApplicationProject (); + proj.MainActivity = proj.DefaultMainActivity; + using (var builder = CreateApkBuilder ()) { + builder.Target = "Build"; + builder.Restore (proj); + Profile (builder, b => b.Build (proj)); + } + } + [Test] public void Build_No_Changes () { diff --git a/tests/msbuild-times-reference/MSBuildDeviceIntegration.csv b/tests/msbuild-times-reference/MSBuildDeviceIntegration.csv index 2c34cb9015f..73704537f5c 100644 --- a/tests/msbuild-times-reference/MSBuildDeviceIntegration.csv +++ b/tests/msbuild-times-reference/MSBuildDeviceIntegration.csv @@ -2,6 +2,7 @@ # First non-comment row is human description of columns Test Name,Time in ms (int) # Data +Build_From_Clean_DontIncludeRestore,10000 Build_No_Changes,3250 Build_CSharp_Change,4450 Build_AndroidResource_Change,4150 From ac32ffd6faa3d00da87eed7fa851d119cb99fecd Mon Sep 17 00:00:00 2001 From: Dean Ellis Date: Mon, 23 Mar 2020 11:34:35 +0000 Subject: [PATCH 2/5] Code Review Changes. --- src/Xamarin.Android.Build.Tasks/Tasks/Aapt2Link.cs | 3 ++- src/Xamarin.Android.Build.Tasks/Tasks/ConvertCustomView.cs | 2 +- src/Xamarin.Android.Build.Tasks/Tasks/CopyIfChanged.cs | 3 +-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/src/Xamarin.Android.Build.Tasks/Tasks/Aapt2Link.cs b/src/Xamarin.Android.Build.Tasks/Tasks/Aapt2Link.cs index 24e9c591a58..5d171592de1 100644 --- a/src/Xamarin.Android.Build.Tasks/Tasks/Aapt2Link.cs +++ b/src/Xamarin.Android.Build.Tasks/Tasks/Aapt2Link.cs @@ -304,7 +304,8 @@ string ExpandString (string s) bool ExecuteForAbi (string [] cmd, string currentResourceOutputFile) { - apks.Add (currentResourceOutputFile, RunAapt (cmd, currentResourceOutputFile)); + lock (apks) + apks.Add (currentResourceOutputFile, RunAapt (cmd, currentResourceOutputFile)); return true; } diff --git a/src/Xamarin.Android.Build.Tasks/Tasks/ConvertCustomView.cs b/src/Xamarin.Android.Build.Tasks/Tasks/ConvertCustomView.cs index 889150fef5b..3de704d6597 100644 --- a/src/Xamarin.Android.Build.Tasks/Tasks/ConvertCustomView.cs +++ b/src/Xamarin.Android.Build.Tasks/Tasks/ConvertCustomView.cs @@ -79,7 +79,7 @@ public override bool RunTask () } } } - var output = new Dictionary (); + var output = new Dictionary (processed.Count); foreach (var file in processed) { ITaskItem resdir = ResourceDirectories?.FirstOrDefault (x => file.StartsWith (x.ItemSpec)) ?? null; if (output.ContainsKey (file)) diff --git a/src/Xamarin.Android.Build.Tasks/Tasks/CopyIfChanged.cs b/src/Xamarin.Android.Build.Tasks/Tasks/CopyIfChanged.cs index 10f81fd3e34..5d6224127fe 100644 --- a/src/Xamarin.Android.Build.Tasks/Tasks/CopyIfChanged.cs +++ b/src/Xamarin.Android.Build.Tasks/Tasks/CopyIfChanged.cs @@ -24,7 +24,7 @@ public class CopyIfChanged : AndroidTask [Required] public ITaskItem[] DestinationFiles { get; set; } - public bool CompareFileLengths { get; set; } + public bool CompareFileLengths { get; set; } = true; [Output] public ITaskItem[] ModifiedFiles { get; set; } @@ -33,7 +33,6 @@ public class CopyIfChanged : AndroidTask public CopyIfChanged () { - CompareFileLengths = true; } public override bool RunTask () From 14b7d58d765d2796f9f50ff02a8107befcfeca70 Mon Sep 17 00:00:00 2001 From: Dean Ellis Date: Mon, 23 Mar 2020 13:59:52 +0000 Subject: [PATCH 3/5] rework from review --- src/Xamarin.Android.Build.Tasks/Tasks/ConvertCustomView.cs | 2 -- 1 file changed, 2 deletions(-) diff --git a/src/Xamarin.Android.Build.Tasks/Tasks/ConvertCustomView.cs b/src/Xamarin.Android.Build.Tasks/Tasks/ConvertCustomView.cs index 3de704d6597..6a5cb9ef77a 100644 --- a/src/Xamarin.Android.Build.Tasks/Tasks/ConvertCustomView.cs +++ b/src/Xamarin.Android.Build.Tasks/Tasks/ConvertCustomView.cs @@ -82,8 +82,6 @@ public override bool RunTask () var output = new Dictionary (processed.Count); foreach (var file in processed) { ITaskItem resdir = ResourceDirectories?.FirstOrDefault (x => file.StartsWith (x.ItemSpec)) ?? null; - if (output.ContainsKey (file)) - continue; var hash = resdir?.GetMetadata ("Hash") ?? null; var stamp = resdir?.GetMetadata ("StampFile") ?? null; var filename = !string.IsNullOrEmpty (hash) ? hash : "compiled"; From 92a6d707bd10cfb2c21b6f9c5a425d5354dc4aa8 Mon Sep 17 00:00:00 2001 From: Jonathan Pryor Date: Tue, 24 Mar 2020 12:00:06 -0400 Subject: [PATCH 4/5] Fix indentation --- .../Xamarin/Android/Xamarin.Android.Aapt.targets | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/Xamarin.Android.Build.Tasks/MSBuild/Xamarin/Android/Xamarin.Android.Aapt.targets b/src/Xamarin.Android.Build.Tasks/MSBuild/Xamarin/Android/Xamarin.Android.Aapt.targets index 561a41f7f67..fb8feef9f85 100644 --- a/src/Xamarin.Android.Build.Tasks/MSBuild/Xamarin/Android/Xamarin.Android.Aapt.targets +++ b/src/Xamarin.Android.Build.Tasks/MSBuild/Xamarin/Android/Xamarin.Android.Aapt.targets @@ -22,10 +22,10 @@ Copyright (C) 2019 Microsoft Corporation. All rights reserved. $(_UpdateAndroidResgenInputs); @(_LibraryResourceDirectoryStamps); - <_CreateBaseApkInputs> - $(_CreateBaseApkInputs); - @(_LibraryResourceDirectoryStamps); - + <_CreateBaseApkInputs> + $(_CreateBaseApkInputs); + @(_LibraryResourceDirectoryStamps); + @@ -104,4 +104,4 @@ Copyright (C) 2019 Microsoft Corporation. All rights reserved. AndroidSdkPlatform="$(_AndroidApiLevel)" /> - \ No newline at end of file + From ae511416a7f8bf84b8839830cb6ff48053987893 Mon Sep 17 00:00:00 2001 From: Jonathan Pryor Date: Tue, 24 Mar 2020 12:04:56 -0400 Subject: [PATCH 5/5] Fix indentation --- .../Xamarin.Android.Common.targets | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets b/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets index 1194993484f..12b865f4171 100644 --- a/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets +++ b/src/Xamarin.Android.Build.Tasks/Xamarin.Android.Common.targets @@ -1493,10 +1493,10 @@ because xbuild doesn't support framework reference assemblies. $(_UpdateAndroidResgenInputs); @(_ModifiedResources); - <_CreateBaseApkInputs> - $(_CreateBaseApkInputs); - @(_ModifiedResources); - + <_CreateBaseApkInputs> + $(_CreateBaseApkInputs); + @(_ModifiedResources); +