[mtouch/mmp] Fix tracking of whether the static registrar should run …

…again or not.

The recent changes to support optimizing away the dynamic registrar caused the
Xamarin.MTouch.RebuildTest_WithExtensions test to regress.

The problem
-----------

* The linker now collects and stores information the static registrar needs.
* This information is not restored from disk when the linker realizes that it
  can reload previously linked assemblies instead of executing again.
* The static registrar runs again (for another reason).
* The information the static registrar needs isn't available, and incorrect
  output follows.

So fix 1: show an error if the static registrar runs when the linker loaded
cached results.

The exact scenario the test ran into is this:

* 1st build: everything is new and everything is built.
* 2nd build: contents of .exe changes, the linker runs again, the static
  registrar runs again, but sees that the generated output didn't change, so
  it doesn't write the new content to disk (this is an optimization to avoid
  compiling the registrar.m file again unless needed).
* 3rd build: only the .exe timestamp changes, the linker sees nothing changes
  in the contents of the .exe and loads the previously linked assemblies from
  disk, the static registrar sees that the .exe's timestamp is newer than
  registrar.m's timestamp and run again, but doesn't produce the right result
  because it doesn't have the information it needs.

Considered solutions
--------------------

1. Only track timestamps, not file contents. This is not ideal, since it will
   result in more work done: in particular for the case above, it would add a
   registrar.m compilation in build #2, and linker rerun + static registrar
   rerun + registrar.m compilation + final native link in build #3.
2. Always write the output of the static registrar, even if it hasn't changed.
   This is not ideal either, since it will also result in more work done: for
   the case above, it would add a registrar.m compilation + final native link
   in build #3.
3. Always write the output of the static registrar, but track if it changed or
   not, and if it didn't, just touch registrar.o instead of recompiling it.
   This only means the final native link in build #3 is added (see #5 for why
   this is worse than it sounds).
4. Always write the output of the static registrar, but track it it changed or
   not, and if it didn't, just touch registrar.o instead of recompiling it,
   and track that too, so that the final native link in build #3 isn't needed
   anymore. Unfortunately this may result in incorrect behavior, because now
   the msbuild tasks will detect that the executable has changed, and may run
   dsymutil + strip again. The executable didn't actually change, which means
   it would be the previously stripped executable, and thus we'd end up with
   an empty .dSYM because we ran dsymtil on an already stripped executable.
5. Idea #4, but write the output of the final link into a temporary directory
   instead of the .app, so that we could track whether we should update the
   executable in the .app or not. This is not optimal either, because
   executables can be *big* (I've seen multi-GB tvOS bitcode executables), and
   extra copies of such files should not be taken lightly.
6. Idea #4, but tell the MSBuild tasks that dsymutil/strip doesn't need to be
   rerun even if the timestamp of the executable changed. This might actually
   work, but now the solution's become quite complex.

Implemented solution
--------------------

Use stamp files to detect whether a file is up-to-date or not.

In particular:

* When we don't write to a file because the new contents are identical to the
  old contents, we now touch a .stamp file. This stamp file means "the
  accompanying file was determined to be up-to-date when the stamp was
  touched."
* When checking whether a file is up-to-date, also check for the presence of a
  .stamp file, and if it exists, use the highest timestamp between the stamp
  file and the actual file.

Now the test scenario becomes:

* 1st build: everything is new and everything is built.
* 2nd build: contents of .exe changes, the linker runs again, the static
  registrar runs again, but sees that the generated output didn't change, so
  it doesn't write the new content to disk, but it creates a registrar.m.stamp
  file to indicate the point in time when registrar.m was considered up-to-
  date.
* 3rd build: only the .exe timestamp changes, the linker sees nothing changes
  in the contents of the .exe and loads the previously linked assemblies from
  disk, the static registrar sees that the .exe's timestamp is *older* than
  registrar.m.stamp's timestamp and doesn't run again.

We only use the stamp file for source code (registrar.[m|h], main.[m|h],
pinvokes.[m|h]), since using it every time has too much potential for running
into other problems (for instance we should never create .stamp files inside
the .app).

Fixes these test failures:

    1) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("single","",False,System.String[])
      single
      Expected: <empty>
      But was:  < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory371/testApp.app/testApp is modified, timestamp: 2/15/2018 3:04:11 PM > 2/15/2018 3:04:09 PM" >

    2) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("dual","armv7,arm64",False,System.String[])
      dual
      Expected: <empty>
      But was:  < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory375/testApp.app/testApp is modified, timestamp: 2/15/2018 3:06:03 PM > 2/15/2018 3:06:00 PM" >

    3) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("llvm","armv7+llvm",False,System.String[])
      llvm
      Expected: <empty>
      But was:  < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory379/testApp.app/testApp is modified, timestamp: 2/15/2018 3:07:14 PM > 2/15/2018 3:07:12 PM" >

    4) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("debug","",True,System.String[])
      debug
      Expected: <empty>
      But was:  < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory383/testApp.app/testApp is modified, timestamp: 2/15/2018 3:08:16 PM > 2/15/2018 3:08:13 PM" >

    5) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("single-framework","",False,System.String[])
      single-framework
      Expected: <empty>
      But was:  < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory387/testApp.app/testApp is modified, timestamp: 2/15/2018 3:09:18 PM > 2/15/2018 3:09:16 PM" >

Fixes xamarin/maccore#641

tools/common/Application.cs

@@ -126,7 +126,13 @@ public string PlatformName {
 			}
 		}
 
-		public static bool IsUptodate (string source, string target, bool check_contents = false)
+		// Checks if the source file has a time stamp later than the target file.
+		//
+		// Optionally check if the contents of the files are different after checking the timestamp.
+		//
+		// If check_stamp is true, the function will use the timestamp of a "target".stamp file
+		// if it's later than the timestamp of the "target" file itself.
+		public static bool IsUptodate (string source, string target, bool check_contents = false, bool check_stamp = true)
 		{
 			if (Driver.Force)
 				return false;
@@ -138,6 +144,14 @@ public static bool IsUptodate (string source, string target, bool check_contents
 				return false;
 			}
 
+			if (check_stamp) {
+				var tfi_stamp = new FileInfo (target + ".stamp");
+				if (tfi_stamp.Exists && tfi_stamp.LastWriteTimeUtc > tfi.LastWriteTimeUtc) {
+					Driver.Log (3, "Target '{0}' has a stamp file with newer timestamp ({1} > {2}), using the stamp file's timestamp", target, tfi_stamp.LastWriteTimeUtc, tfi.LastWriteTimeUtc);
+					tfi = tfi_stamp;
+				}
+			}
+
 			var sfi = new FileInfo (source);
 
 			if (sfi.LastWriteTimeUtc <= tfi.LastWriteTimeUtc) {
@@ -323,7 +337,10 @@ public static void UpdateFile (string source, string target, bool check_contents
 		}
 
 		// Checks if any of the source files have a time stamp later than any of the target files.
-		public static bool IsUptodate (IEnumerable<string> sources, IEnumerable<string> targets)
+		//
+		// If check_stamp is true, the function will use the timestamp of a "target".stamp file
+		// if it's later than the timestamp of the "target" file itself.
+		public static bool IsUptodate (IEnumerable<string> sources, IEnumerable<string> targets, bool check_stamp = true)
 		{
 			if (Driver.Force)
 				return false;
@@ -356,6 +373,14 @@ public static bool IsUptodate (IEnumerable<string> sources, IEnumerable<string>
 					return false;
 				}
 
+				if (check_stamp) {
+					var tfi_stamp = new FileInfo (t + ".stamp");
+					if (tfi_stamp.Exists && tfi_stamp.LastWriteTimeUtc > tfi.LastWriteTimeUtc) {
+						Driver.Log (3, "Target '{0}' has a stamp file with newer timestamp ({1} > {2}), using the stamp file's timestamp", t, tfi_stamp.LastWriteTimeUtc, tfi.LastWriteTimeUtc);
+						tfi = tfi_stamp;
+					}
+				}
+
 				var lwt = tfi.LastWriteTimeUtc;
 				if (max_source > lwt) {
 					Driver.Log (3, "Prerequisite '{0}' is newer than target '{1}' ({2} vs {3}).", max_s, t, max_source, lwt);

tools/common/Driver.cs

@@ -305,7 +305,7 @@ static void FileMove (string source, string target)
 			File.Move (source, target);
 		}
 
-		static void MoveIfDifferent (string path, string tmp)
+		static void MoveIfDifferent (string path, string tmp, bool use_stamp = false)
 		{
 			// Don't read the entire file into memory, it can be quite big in certain cases.
 
@@ -326,10 +326,12 @@ static void MoveIfDifferent (string path, string tmp)
 				FileMove (tmp, path);
 			} else {
 				Log (3, "Target {0} is up-to-date.", path);
+				if (use_stamp)
+					Driver.Touch (path + ".stamp");
 			}
 		}
 
-		public static void WriteIfDifferent (string path, string contents)
+		public static void WriteIfDifferent (string path, string contents, bool use_stamp = false)
 		{
 			var tmp = path + ".tmp";
 
@@ -342,7 +344,7 @@ public static void WriteIfDifferent (string path, string contents)
 				}
 
 				File.WriteAllText (tmp, contents);
-				MoveIfDifferent (path, tmp);
+				MoveIfDifferent (path, tmp, use_stamp);
 			} catch (Exception e) {
 				File.WriteAllText (path, contents);
 				ErrorHelper.Warning (1014, e, "Failed to re-use cached version of '{0}': {1}.", path, e.Message);
@@ -351,7 +353,7 @@ public static void WriteIfDifferent (string path, string contents)
 			}
 		}
 
-		public static void WriteIfDifferent (string path, byte[] contents)
+		public static void WriteIfDifferent (string path, byte[] contents, bool use_stamp = false)
 		{
 			var tmp = path + ".tmp";
 
@@ -363,7 +365,7 @@ public static void WriteIfDifferent (string path, byte[] contents)
 				}
 
 				File.WriteAllBytes (tmp, contents);
-				MoveIfDifferent (path, tmp);
+				MoveIfDifferent (path, tmp, use_stamp);
 			} catch (Exception e) {
 				File.WriteAllBytes (path, contents);
 				ErrorHelper.Warning (1014, e, "Failed to re-use cached version of '{0}': {1}.", path, e.Message);

tools/common/PInvokeWrapperGenerator.cs

@@ -63,8 +63,8 @@ public void End ()
 
 			Registrar.GeneratePInvokeWrappersEnd ();
 
-			Driver.WriteIfDifferent (HeaderPath, hdr.ToString () + "\n" + decls.ToString () + "\n" + ifaces.ToString () + "\n") ;
-			Driver.WriteIfDifferent (SourcePath, mthds.ToString () + "\n" + sb.ToString () + "\n");
+			Driver.WriteIfDifferent (HeaderPath, hdr.ToString () + "\n" + decls.ToString () + "\n" + ifaces.ToString () + "\n", true);
+			Driver.WriteIfDifferent (SourcePath, mthds.ToString () + "\n" + sb.ToString () + "\n", true);
 		}
 
 		public void ProcessMethod (MethodDefinition method)

tools/common/StaticRegistrar.cs

@@ -4564,12 +4564,12 @@ void Generate (string header_path, string source_path)
 
 			FlushTrace ();
 
-			Driver.WriteIfDifferent (source_path, methods.ToString ());
+			Driver.WriteIfDifferent (source_path, methods.ToString (), true);
 
 			header.AppendLine ();
 			header.AppendLine (declarations);
 			header.AppendLine (interfaces);
-			Driver.WriteIfDifferent (header_path, header.ToString ());
+			Driver.WriteIfDifferent (header_path, header.ToString (), true);
 
 			header.Dispose ();
 			header = null;

tools/common/Target.cs

@@ -402,7 +402,7 @@ internal string GenerateReferencingSource (string reference_m, IEnumerable<Symbo
 			sb.AppendLine ("}");
 			sb.AppendLine ();
 
-			Driver.WriteIfDifferent (reference_m, sb.ToString ());
+			Driver.WriteIfDifferent (reference_m, sb.ToString (), true);
 
 #if MTOUCH
 			foreach (var abi in GetArchitectures (AssemblyBuildTarget.StaticObject)) {

tools/mmp/.gitignore

@@ -2,3 +2,5 @@ mmp
 temp-dir-mmp
 Xamarin.Mac.registrar.*
 SdkVersions.cs
+*.stamp
+

tools/mmp/driver.cs

@@ -68,6 +68,7 @@ enum Action {
 	}
 
 	public static partial class Driver {
+		internal const string NAME = "mmp";
 		internal static Application App = new Application (Environment.GetCommandLineArgs ());
 		static Target BuildTarget = new Target (App);
 		static List<string> references = new List<string> ();

tools/mtouch/.gitignore

@@ -7,3 +7,5 @@ SdkVersions.cs
 simlauncher-sgen
 simlauncher32-sgen
 simlauncher64-sgen
+*.stamp
+

tools/mtouch/mtouch.cs

@@ -79,6 +79,8 @@ public enum OutputFormat {
 namespace Xamarin.Bundler
 {
 	partial class Driver {
+		internal const string NAME = "mtouch";
+
 		public static void ShowHelp (OptionSet os)
 		{
 			Console.WriteLine ("mtouch - Mono Compiler for iOS");
@@ -698,7 +700,7 @@ public static string GenerateMain (Application app, IEnumerable<Assembly> assemb
 					sw.WriteLine ("}");
 
 				}
-				WriteIfDifferent (main_source, sb.ToString ());
+				WriteIfDifferent (main_source, sb.ToString (), true);
 			} catch (MonoTouchException) {
 				throw;
 			} catch (Exception e) {