Merge pull request SCons#4324 from jcbrill/jbrill-gh4312-fixplus

Fix 4312 and add Windows on ARM64 support for msvc.

CHANGES.txt

@@ -16,6 +16,33 @@ RELEASE  VERSION/DATE TO BE FILLED IN LATER
     - Obsoleted YACCVCGFILESUFFIX, being replaced by YACC_GRAPH_FILE_SUFFIX.
       If YACC_GRAPH_FILE_SUFFIX is not set, it will respect YACCVCGFILESUFFIX.
 
+  From Joseph Brill:
+    - Fix issue #4312: the cached installed msvc list had an indirect dependency
+      on the target architecture in the environment dictionary.  The first call
+      to construct the installed msvc list now forces the target architecture to be
+      undefined, constructs the installed msvc list, and then restores the original
+      target architecture.
+      Note: an indirect dependency on the VSWHERE construction variable in the
+      environment remains.
+    - Fix issue #4312: explicitly guard against an empty regular expression list
+      when msvc is not installed.
+    - When trying to find a valid msvc batch file, check that the compiler executable
+      (cl.exe) exists for VS6 to VS2015 to avoid executing the msvc batch file.  Always
+      check that the compiler executable is found on the msvc script environment path
+      after running the msvc batch file.  Only use the sdk batch files when all of the
+      msvc script host/target combinations have been exhausted and a valid script was
+      not found.
+    - Add ARM64 host configurations for windows and msvc.
+      Note: VS2013 and earlier has not been tested on ARM64.
+    - If necessary, automatically define VSCMD_SKIP_SENDTELEMETRY for VS2019 and later
+      on ARM64 hosts when using an arm32 build of python to prevent a powershell dll
+      not found error pop-up window.
+    - Fix an issue where test SConfTests.py would fail when mscommon debugging
+      was enabled.  The mscommon debug filter class registered with the logging
+      module was refactored.
+    - Add arm64 to the MSVS supported architectures list for VS2017 and later to be
+      consistent with the current documentation of MSVS_ARCH.
+
   From Mats Wichmann
     - C scanner's dictifyCPPDEFINES routine did not understand the possible
       combinations of CPPDEFINES - not aware of a "name=value" string either

RELEASE.txt

@@ -19,6 +19,10 @@ NEW FUNCTIONALITY
 - D compilers : added support for generation of .di interface files.
   New variables DI_FILE_DIR, DI_FILE_DIR_PREFIX, DI_FILE_DIR_SUFFIX,
   DI_FILE_SUFFIX.
+- MSVC: If available, native arm64 tools will be used on arm64 hosts for VS2022.
+- MSVC: If necessary, automatically define VSCMD_SKIP_SENDTELEMETRY for VS2019 and later
+  on arm64 hosts when using an arm (32-bit) build of python to prevent a powershell
+  error pop-up window (powershell dll not found).
 
 DEPRECATED FUNCTIONALITY
 ------------------------
@@ -47,6 +51,14 @@ CHANGED/ENHANCED EXISTING FUNCTIONALITY
   Default value for $YACC_GRAPH_FILE_SUFFIX changed to '.gv' to match
   current bison default (since bison 3.8). Set this variable to '.dot'
   if using byacc. Fixes #4326 and #4327.
+- MSVC: When trying to find a valid msvc batch file, the existence of the msvc compiler
+  executable is verified for VS6 to VS2015 to avoid executing the msvc batch file when
+  the host/target tool is known not to be present. Always check that the msvc compiler
+  executable is found on the msvc script environment path after running the msvc batch
+  file.  This is predominately needed for recent versions of Visual Studio where the msvc
+  batch file exists but an individual msvc toolset may not support the host/target
+  architecture combination.  For example, when using VS2022 on arm64, the arm64 native
+  tools are only installed for the 14.3x toolsets.
 
 FIXES
 -----
@@ -59,6 +71,22 @@ FIXES
   embedded in a sequence, or by itself.  The conditional C scanner thus
   did not always properly apply the defines. The regular C scanner does
   not use these, so was not affected.  [fixes #4193]
+- MSVC: The installed msvc list is calculated once and cached.  There was an indirect
+  dependency on the target architecture when determining if each version of msvc
+  was installed based on the initial invocation.  It was possible that an msvc instance
+  would not be considered installed due to the tools for the requested target
+  architecture not being installed.  The initial call to construct the installed msvc
+  list now uses an undefined target architecture to evaluate all potential host/target
+  combinations when evaluating if the msvc tools are installed for a given Visual Studio
+  installation.
+- MSVC: Erroneous construction of the installed msvc list (as described above) caused an
+  index error in the msvc support code.  An explicit check was added to prevent indexing
+  into an empty list.  Fixes #4312.
+- MSCommon: Test SConfTests.py would fail when mscommon debugging was enabled via the
+  MSVC_MSCOMMON_DEBUG environment variable.  The mscommon logging filter class registered
+  with the python logging module was refactored to prevent test failure.
+- MSVS: Add arm64 to the MSVS supported architectures list for VS2017 and later to be
+  consistent with the current documentation of MSVS_ARCH.
 
 IMPROVEMENTS
 ------------

SCons/Platform/Platform.xml

@@ -173,8 +173,8 @@ else:
       Valid host arch values are
       <literal>x86</literal> and <literal>arm</literal>
       for 32-bit hosts and
-      <literal>amd64</literal> and <literal>x86_64</literal>
-      for 64-bit hosts.
+      <literal>amd64</literal>, <literal>arm64</literal>,
+      and <literal>x86_64</literal> for 64-bit hosts.
     </para>
     <para>
       Should be considered immutable. 

SCons/Platform/win32.py

@@ -298,6 +298,11 @@ def __init__(self, arch, synonyms=[]) -> None:
         ['AMD64', 'amd64', 'em64t', 'EM64T', 'x86_64'],
     ),
 
+    ArchDefinition(
+        'arm64',
+        ['ARM64', 'aarch64', 'AARCH64', 'AArch64'],
+    ),
+
     ArchDefinition(
         'ia64',
         ['IA64'],
@@ -315,9 +320,20 @@ def get_architecture(arch=None):
     """Returns the definition for the specified architecture string.
 
     If no string is specified, the system default is returned (as defined
-    by the PROCESSOR_ARCHITEW6432 or PROCESSOR_ARCHITECTURE environment
-    variables).
+    by the registry PROCESSOR_ARCHITECTURE value, PROCESSOR_ARCHITEW6432
+    environment variable, PROCESSOR_ARCHITECTURE environment variable, or
+    the platform machine).
     """
+    if arch is None:
+        if SCons.Util.can_read_reg:
+            try:
+                k=SCons.Util.RegOpenKeyEx(SCons.Util.hkey_mod.HKEY_LOCAL_MACHINE,
+                                          'SYSTEM\\CurrentControlSet\\Control\\Session Manager\\Environment')
+                val, tok = SCons.Util.RegQueryValueEx(k, 'PROCESSOR_ARCHITECTURE')
+            except SCons.Util.RegError:
+                val = ''
+            if val and val in SupportedArchitectureMap:
+                arch = val
     if arch is None:
         arch = os.environ.get('PROCESSOR_ARCHITEW6432')
         if not arch:

SCons/Tool/MSCommon/MSVC/SetupEnvDefault.py

@@ -188,19 +188,22 @@ def register_iserror(env, tool, msvc_exists_func):
         debug('msvc default:check tools:nchar=%d, tools=%s', tools_nchar, tools)
 
         # iteratively remove default tool sequences (longest to shortest)
-        re_nchar_min, re_tools_min = _Data.default_tools_re_list[-1]
-        if tools_nchar >= re_nchar_min and re_tools_min.search(tools):
-            # minimum characters satisfied and minimum pattern exists
-            for re_nchar, re_default_tool in _Data.default_tools_re_list:
-                if tools_nchar < re_nchar:
-                    # not enough characters for pattern
-                    continue
-                tools = re_default_tool.sub('', tools).strip(_Data.separator)
-                tools_nchar = len(tools)
-                debug('msvc default:check tools:nchar=%d, tools=%s', tools_nchar, tools)
-                if tools_nchar < re_nchar_min or not re_tools_min.search(tools):
-                    # less than minimum characters or minimum pattern does not exist
-                    break
+        if not _Data.default_tools_re_list:
+            debug('default_tools_re_list=%s', _Data.default_tools_re_list)
+        else:
+            re_nchar_min, re_tools_min = _Data.default_tools_re_list[-1]
+            if tools_nchar >= re_nchar_min and re_tools_min.search(tools):
+                # minimum characters satisfied and minimum pattern exists
+                for re_nchar, re_default_tool in _Data.default_tools_re_list:
+                    if tools_nchar < re_nchar:
+                        # not enough characters for pattern
+                        continue
+                    tools = re_default_tool.sub('', tools).strip(_Data.separator)
+                    tools_nchar = len(tools)
+                    debug('msvc default:check tools:nchar=%d, tools=%s', tools_nchar, tools)
+                    if tools_nchar < re_nchar_min or not re_tools_min.search(tools):
+                        # less than minimum characters or minimum pattern does not exist
+                        break
 
         # construct non-default list(s) tools set
         tools_set = {msvc_tool for msvc_tool in tools.split(_Data.separator) if msvc_tool}

SCons/Tool/MSCommon/common.py

@@ -46,30 +46,31 @@ class MSVCCacheInvalidWarning(SCons.Warnings.WarningOnByDefault):
 if LOGFILE:
     import logging
 
-    modulelist = (
-        # root module and parent/root module
-        'MSCommon', 'Tool',
-        # python library and below: correct iff scons does not have a lib folder
-        'lib',
-        # scons modules
-        'SCons', 'test', 'scons'
-    )
+    class _Debug_Filter(logging.Filter):
+        # custom filter for module relative filename
 
-    def get_relative_filename(filename, module_list):
-        if not filename:
+        modulelist = (
+            # root module and parent/root module
+            'MSCommon', 'Tool',
+            # python library and below: correct iff scons does not have a lib folder
+            'lib',
+            # scons modules
+            'SCons', 'test', 'scons'
+        )
+
+        def get_relative_filename(self, filename, module_list):
+            if not filename:
+                return filename
+            for module in module_list:
+                try:
+                    ind = filename.rindex(module)
+                    return filename[ind:]
+                except ValueError:
+                    pass
             return filename
-        for module in module_list:
-            try:
-                ind = filename.rindex(module)
-                return filename[ind:]
-            except ValueError:
-                pass
-        return filename
 
-    class _Debug_Filter(logging.Filter):
-        # custom filter for module relative filename
         def filter(self, record) -> bool:
-            relfilename = get_relative_filename(record.pathname, modulelist)
+            relfilename = self.get_relative_filename(record.pathname, self.modulelist)
             relfilename = relfilename.replace('\\', '/')
             record.relfilename = relfilename
             return True
@@ -208,6 +209,17 @@ def has_reg(value) -> bool:
 # Functions for fetching environment variable settings from batch files.
 
 
+def _force_vscmd_skip_sendtelemetry(env):
+
+    if 'VSCMD_SKIP_SENDTELEMETRY' in env['ENV']:
+        return False
+
+    env['ENV']['VSCMD_SKIP_SENDTELEMETRY'] = '1'
+    debug("force env['ENV']['VSCMD_SKIP_SENDTELEMETRY']=%s", env['ENV']['VSCMD_SKIP_SENDTELEMETRY'])
+
+    return True
+
+
 def normalize_env(env, keys, force: bool=False):
     """Given a dictionary representing a shell environment, add the variables
     from os.environ needed for the processing of .bat files; the keys are
@@ -256,7 +268,7 @@ def normalize_env(env, keys, force: bool=False):
     return normenv
 
 
-def get_output(vcbat, args=None, env=None):
+def get_output(vcbat, args=None, env=None, skip_sendtelemetry=False):
     """Parse the output of given bat file, with given args."""
 
     if env is None:
@@ -295,20 +307,23 @@ def get_output(vcbat, args=None, env=None):
     ]
     env['ENV'] = normalize_env(env['ENV'], vs_vc_vars, force=False)
 
+    if skip_sendtelemetry:
+        _force_vscmd_skip_sendtelemetry(env)
+
     if args:
         debug("Calling '%s %s'", vcbat, args)
-        popen = SCons.Action._subproc(env,
-                                      '"%s" %s & set' % (vcbat, args),
-                                      stdin='devnull',
-                                      stdout=subprocess.PIPE,
-                                      stderr=subprocess.PIPE)
+        cmd_str = '"%s" %s & set' % (vcbat, args)
     else:
         debug("Calling '%s'", vcbat)
-        popen = SCons.Action._subproc(env,
-                                      '"%s" & set' % vcbat,
-                                      stdin='devnull',
-                                      stdout=subprocess.PIPE,
-                                      stderr=subprocess.PIPE)
+        cmd_str = '"%s" & set' % vcbat
+
+    popen = SCons.Action._subproc(
+        env,
+        cmd_str,
+        stdin='devnull',
+        stdout=subprocess.PIPE,
+        stderr=subprocess.PIPE
+    )
 
     # Use the .stdout and .stderr attributes directly because the
     # .communicate() method uses the threading module on Windows