Merge commit '5a78423193c48cae4800eb1ec44b62d14ae03920' into master

This is a merge of upstream master up until this dep file parsing change, which will need more verification given our depfiles: ninja-build#1039 The biggest change in this merge is probably the use of 64-bit timestamps. * commit '5a78423193c48cae4800eb1ec44b62d14ae03920': (80 commits) Add OSX build on travis (ninja-build#1502) Fit --help output into 80 columns and move verbose up, fix ninja-build#1500 Improve error message when Python wasn't found for the browse tool Explain why there's no browse tool, fix ninja-build#1478 HACKING: use `Python` for the project HACKING: fix some whitespace nits Fix rendering of code block, fix ninja-build#955 Make -j 0 run unlimited parallel builds, fix ninja-build#1309 Enable ANSI escape sequences on Windows 10 terminals Add --verbose as an alternative spelling for -v, fix ninja-build#1310 Add unit test for ninja-build#1491 Allow disabling of escape code stripping, fix ninja-build#1475 subprocess-win32: add hint on ERROR_INVALID_PARAMETER Win32Fatal: support a "hint" for the error manual: mention the "invalid parameter" case util: don't add ellipses width when deciding if they're necessary Verify GetFullPathName return value Make output_test.py independent of the environment Fix older VS compatibility issues and PDB files generation issue. (ninja-build#1435) Fix parsing some special chars in depfiles ... Change-Id: I315385cecc748ff351614792a9921ec87e5f4cf3
mydongistiny · May 2, 2019 · ee41acc · ee41acc
2 parents 4f8e35a + 5a78423
commit ee41acc
Show file tree

Hide file tree

Showing 52 changed files with 824 additions and 347 deletions.
diff --git a/.gitignore b/.gitignore
@@ -32,3 +32,6 @@ TAGS
 # Ninja output
 .ninja_deps
 .ninja_log
+
+# Visual Studio Code project files
+/.vscode/
diff --git a/.travis.yml b/.travis.yml
@@ -1,6 +1,15 @@
+matrix:
+  include:
+    - os: linux
+      compiler: gcc
+    - os: linux
+      compiler: clang
+    - os: osx
 sudo: false
 language: cpp
-compiler:
-  - gcc
-  - clang
-script: ./configure.py --bootstrap && ./ninja all && ./ninja_test --gtest_filter=-SubprocessTest.SetWithLots && ./misc/ninja_syntax_test.py
+script:
+  - ./configure.py --bootstrap
+  - ./ninja all
+  - ./ninja_test --gtest_filter=-SubprocessTest.SetWithLots
+  - ./misc/ninja_syntax_test.py
+  - ./misc/output_test.py
diff --git a/HACKING.md b/HACKING.md
@@ -13,14 +13,50 @@ run `ninja_test` when developing.
 Ninja is built using itself.  To bootstrap the first binary, run the
 configure script as `./configure.py --bootstrap`.  This first compiles
 all non-test source files together, then re-builds Ninja using itself.
-You should end up with a `ninja` binary (or `ninja.exe`) in the source root.
+You should end up with a `ninja` binary (or `ninja.exe`) in the project root.
 
 #### Windows
 
 On Windows, you'll need to install Python to run `configure.py`, and
 run everything under a Visual Studio Tools Command Prompt (or after
-running `vcvarsall` in a normal command prompt).  See below if you
-want to use mingw or some other compiler instead of Visual Studio.
+running `vcvarsall` in a normal command prompt).
+
+For other combinations such as gcc/clang you will need the compiler
+(gcc/cl) in your PATH and you will have to set the appropriate
+platform configuration script.
+
+See below if you want to use mingw or some other compiler instead of
+Visual Studio.
+
+##### Using Visual Studio
+Assuming that you now have Python installed, then the steps for building under
+Windows using Visual Studio are:
+
+Clone and checkout the latest release (or whatever branch you want). You
+can do this in either a command prompt or by opening a git bash prompt:
+
+```
+    $ git clone git://github.com/ninja-build/ninja.git && cd ninja
+    $ git checkout release
+```
+
+Then:
+
+1. Open a Windows command prompt in the folder where you checked out ninja.
+2. Select the Microsoft build environment by running
+`vcvarsall.bat` with the appropriate environment.
+3. Build ninja and test it.
+
+The steps for a Visual Studio 2015 64-bit build are outlined here:
+
+```
+    > "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" x64
+    > python configure.py --bootstrap
+    > ninja --help
+```
+Copy the ninja executable to another location, if desired, e.g. C:\local\Ninja.
+
+Finally add the path where ninja.exe is to the PATH variable.
 
 ### Adjusting build flags
 
@@ -95,7 +131,7 @@ and run that directly on some representative input files.
 Generally it's the [Google C++ coding style][], but in brief:
 
 * Function name are camelcase.
-* Member methods are camelcase, expect for trivial getters which are
+* Member methods are camelcase, except for trivial getters which are
   underscore separated.
 * Local variables are underscore separated.
 * Member variables are underscore separated and suffixed by an extra

diff --git a/RELEASING b/RELEASING
@@ -1,19 +1,20 @@
 Notes to myself on all the steps to make for a Ninja release.
 
 Push new release branch:
-1. Consider sending a heads-up to the ninja-build mailing list first
-2. Make sure branches 'master' and 'release' are synced up locally
-3. update src/version.cc with new version (with ".git"), then
+1. Run afl-fuzz for a day or so (see HACKING.md) and run ninja_test
+2. Consider sending a heads-up to the ninja-build mailing list first
+3. Make sure branches 'master' and 'release' are synced up locally
+4. Update src/version.cc with new version (with ".git"), then
        git commit -am 'mark this 1.5.0.git'
-4. git checkout release; git merge master
-5. fix version number in src/version.cc (it will likely conflict in the above)
-6. fix version in doc/manual.asciidoc (exists only on release branch)
-7. commit, tag, push (don't forget to push --tags)
+5. git checkout release; git merge master
+6. Fix version number in src/version.cc (it will likely conflict in the above)
+7. Fix version in doc/manual.asciidoc (exists only on release branch)
+8. commit, tag, push (don't forget to push --tags)
        git commit -am v1.5.0; git push origin release
        git tag v1.5.0; git push --tags
        # Push the 1.5.0.git change on master too:
        git checkout master; git push origin master
-8. construct release notes from prior notes
+9. Construct release notes from prior notes
    credits: git shortlog -s --no-merges REV..
 
 Release on github:

diff --git a/appveyor.yml b/appveyor.yml
@@ -0,0 +1,40 @@
+version: 1.0.{build}
+image: Visual Studio 2017
+
+environment:
+  CLICOLOR_FORCE: 1
+  CHERE_INVOKING: 1 # Tell Bash to inherit the current working directory
+  matrix:
+    - MSYSTEM: MINGW64
+    - MSYSTEM: MSVC
+
+for:
+  -
+    matrix:
+      only:
+        - MSYSTEM: MINGW64
+    build_script:
+      ps: "C:\\msys64\\usr\\bin\\bash -lc @\"\n
+      pacman -S --quiet --noconfirm --needed re2c 2>&1\n
+      sed -i 's|cmd /c $ar cqs $out.tmp $in && move /Y $out.tmp $out|$ar crs $out $in|g' configure.py\n
+      ./configure.py --bootstrap --platform mingw 2>&1\n
+      ./ninja all\n
+      ./ninja_test 2>&1\n
+      ./misc/ninja_syntax_test.py 2>&1\n\"@"
+  -
+    matrix:
+      only:
+        - MSYSTEM: MSVC
+    build_script:
+    - cmd: >-
+        call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars64.bat"
+
+        python configure.py --bootstrap
+
+        ninja.bootstrap.exe all
+
+        ninja_test
+
+        python misc/ninja_syntax_test.py
+
+test: off
diff --git a/configure.py b/configure.py
@@ -98,7 +98,7 @@ def is_aix(self):
         return self._platform == 'aix'
 
     def uses_usr_local(self):
-        return self._platform in ('freebsd', 'openbsd', 'bitrig', 'dragonfly')
+        return self._platform in ('freebsd', 'openbsd', 'bitrig', 'dragonfly', 'netbsd')
 
     def supports_ppoll(self):
         return self._platform in ('freebsd', 'linux', 'openbsd', 'bitrig',
@@ -256,7 +256,7 @@ def _run_command(self, cmdline):
 if '--bootstrap' in configure_args:
     configure_args.remove('--bootstrap')
 n.variable('configure_args', ' '.join(configure_args))
-env_keys = set(['CXX', 'AR', 'CFLAGS', 'LDFLAGS'])
+env_keys = set(['CXX', 'AR', 'CFLAGS', 'CXXFLAGS', 'LDFLAGS'])
 configure_env = dict((k, os.environ[k]) for k in os.environ if k in env_keys)
 if configure_env:
     config_str = ' '.join([k + '=' + pipes.quote(configure_env[k])
@@ -359,6 +359,11 @@ def binary(name):
     if platform.uses_usr_local():
         cflags.append('-I/usr/local/include')
         ldflags.append('-L/usr/local/lib')
+    if platform.is_aix():
+        # printf formats for int64_t, uint64_t; large file support
+        cflags.append('-D__STDC_FORMAT_MACROS')
+        cflags.append('-D_LARGE_FILES')
+
 
 libs = []
 
@@ -403,6 +408,10 @@ def shell_escape(str):
 
 if 'CFLAGS' in configure_env:
     cflags.append(configure_env['CFLAGS'])
+    ldflags.append(configure_env['CFLAGS'])
+if 'CXXFLAGS' in configure_env:
+    cflags.append(configure_env['CXXFLAGS'])
+    ldflags.append(configure_env['CXXFLAGS'])
 n.variable('cflags', ' '.join(shell_escape(flag) for flag in cflags))
 if 'LDFLAGS' in configure_env:
     ldflags.append(configure_env['LDFLAGS'])
@@ -411,7 +420,7 @@ def shell_escape(str):
 
 if platform.is_msvc():
     n.rule('cxx',
-        command='$cxx $cflags -c $in /Fo$out',
+        command='$cxx $cflags -c $in /Fo$out /Fd' + built('$pdb'),
         description='CXX $out',
         deps='msvc'  # /showIncludes is included in $cflags.
     )
@@ -519,6 +528,9 @@ def can_generate_proto_header():
 n.newline()
 
 n.comment('Core source files all build into ninja library.')
+cxxvariables = []
+if platform.is_msvc():
+    cxxvariables = [('pdb', 'ninja.pdb')]
 for name in ['build',
              'build_log',
              'clean',
@@ -543,15 +555,15 @@ def can_generate_proto_header():
              'thread_pool',
              'util',
              'version']:
-    objs += cxx(name, order_only=src('frontend.pb.h'))
+    objs += cxx(name, order_only=src('frontend.pb.h'), variables=cxxvariables)
 if platform.is_windows():
     for name in ['subprocess-win32',
                  'includes_normalize-win32',
                  'msvc_helper-win32',
                  'msvc_helper_main-win32']:
-        objs += cxx(name)
+        objs += cxx(name, variables=cxxvariables)
     if platform.is_msvc():
-        objs += cxx('minidump-win32')
+        objs += cxx('minidump-win32', variables=cxxvariables)
     objs += cc('getopt')
 else:
     objs += cxx('subprocess-posix')
@@ -574,7 +586,7 @@ def can_generate_proto_header():
 all_targets = []
 
 n.comment('Main executable is library plus main() function.')
-objs = cxx('ninja')
+objs = cxx('ninja', variables=cxxvariables)
 ninja = n.build(binary('ninja'), 'link', objs, implicit=ninja_lib,
                 variables=[('libs', libs)])
 n.newline()
@@ -589,6 +601,8 @@ def can_generate_proto_header():
 n.comment('Tests all build into ninja_test executable.')
 
 objs = []
+if platform.is_msvc():
+    cxxvariables = [('pdb', 'ninja_test.pdb')]
 
 for name in ['build_log_test',
              'build_test',
@@ -608,10 +622,10 @@ def can_generate_proto_header():
              'subprocess_test',
              'test',
              'util_test']:
-    objs += cxx(name)
+    objs += cxx(name, variables=cxxvariables)
 if platform.is_windows():
     for name in ['includes_normalize_test', 'msvc_helper_test']:
-        objs += cxx(name)
+        objs += cxx(name, variables=cxxvariables)
 
 ninja_test = n.build(binary('ninja_test'), 'link', objs, implicit=ninja_lib,
                      variables=[('libs', libs)])
@@ -627,7 +641,9 @@ def can_generate_proto_header():
              'hash_collision_bench',
              'manifest_parser_perftest',
              'clparser_perftest']:
-  objs = cxx(name)
+  if platform.is_msvc():
+    cxxvariables = [('pdb', name + '.pdb')]
+  objs = cxx(name, variables=cxxvariables)
   all_targets += n.build(binary(name), 'link', objs,
                          implicit=ninja_lib, variables=[('libs', libs)])
 

diff --git a/doc/manual.asciidoc b/doc/manual.asciidoc
@@ -594,7 +594,7 @@ Ninja supports this processing in two forms.
    to its stdout.  Ninja then filters these lines from the displayed
    output.  No `depfile` attribute is necessary, but the localized string
    in front of the the header file path. For instance
-   `msvc_deps_prefix = Note: including file: `
+   `msvc_deps_prefix = Note: including file:`
    for a English Visual Studio (the default). Should be globally defined.
 +
 ----
@@ -881,7 +881,8 @@ quoting rules are deterimined by the called program, which on Windows
 are usually provided by the C library.  If you need shell
 interpretation of the command (such as the use of `&&` to chain
 multiple commands), make the command execute the Windows shell by
-prefixing the command with `cmd /c`.
+prefixing the command with `cmd /c`. Ninja may error with "invalid parameter"
+which usually indicates that the command line length has been exceeded.
 
 [[ref_outputs]]
 Build outputs

diff --git a/misc/ninja-mode.el b/misc/ninja-mode.el
@@ -56,7 +56,7 @@
                (save-excursion
                  (goto-char (line-end-position 0))
                  (or
-                  ;; If we're continuting the previous line, it's not a
+                  ;; If we're continuing the previous line, it's not a
                   ;; comment.
                   (not (eq ?$ (char-before)))
                   ;; Except if the previous line is a comment as well, as the

diff --git a/misc/ninja.vim b/misc/ninja.vim
@@ -1,8 +1,8 @@
 " ninja build file syntax.
 " Language: ninja build file as described at
 "           http://ninja-build.org/manual.html
-" Version: 1.4
-" Last Change: 2014/05/13
+" Version: 1.5
+" Last Change: 2018/04/05
 " Maintainer: Nicolas Weber <nicolasweber@gmx.de>
 " Version 1.4 of this script is in the upstream vim repository and will be
 " included in the next vim release. If you change this, please send your change
@@ -21,7 +21,10 @@ set cpo&vim
 
 syn case match
 
-syn match ninjaComment /#.*/  contains=@Spell
+" Comments are only matched when the # is at the beginning of the line (with
+" optional whitespace), as long as the prior line didn't end with a $
+" continuation.
+syn match ninjaComment /\(\$\n\)\@<!\_^\s*#.*$/  contains=@Spell
 
 " Toplevel statements are the ones listed here and
 " toplevel variable assignments (ident '=' value).
@@ -38,12 +41,13 @@ syn match ninjaKeyword "^subninja\>"
 " limited set of magic variables, 'build' allows general
 " let assignments.
 " manifest_parser.cc, ParseRule()
-syn region ninjaRule start="^rule" end="^\ze\S" contains=ALL transparent
-syn keyword ninjaRuleCommand contained command deps depfile description generator
+syn region ninjaRule start="^rule" end="^\ze\S" contains=TOP transparent
+syn keyword ninjaRuleCommand contained containedin=ninjaRule command
+                                     \ deps depfile description generator
                                      \ pool restat rspfile rspfile_content
 
-syn region ninjaPool start="^pool" end="^\ze\S" contains=ALL transparent
-syn keyword ninjaPoolCommand contained depth
+syn region ninjaPool start="^pool" end="^\ze\S" contains=TOP transparent
+syn keyword ninjaPoolCommand contained containedin=ninjaPool  depth
 
 " Strings are parsed as follows:
 " lexer.in.cc, ReadEvalString()

diff --git a/misc/ninja_syntax.py b/misc/ninja_syntax.py
@@ -60,7 +60,7 @@ def rule(self, name, command, description=None, depfile=None,
             self.variable('deps', deps, indent=1)
 
     def build(self, outputs, rule, inputs=None, implicit=None, order_only=None,
-              variables=None, implicit_outputs=None):
+              variables=None, implicit_outputs=None, pool=None):
         outputs = as_list(outputs)
         out_outputs = [escape_path(x) for x in outputs]
         all_inputs = [escape_path(x) for x in as_list(inputs)]
@@ -81,6 +81,8 @@ def build(self, outputs, rule, inputs=None, implicit=None, order_only=None,
 
         self._line('build %s: %s' % (' '.join(out_outputs),
                                      ' '.join([rule] + all_inputs)))
+        if pool is not None:
+            self._line('  pool = %s' % pool)
 
         if variables:
             if isinstance(variables, dict):

diff --git a/misc/ninja_syntax_test.py b/misc/ninja_syntax_test.py
@@ -46,13 +46,13 @@ def test_few_long_words(self):
                          self.out.getvalue())
 
     def test_comment_wrap(self):
-        # Filenames shoud not be wrapped
+        # Filenames should not be wrapped
         self.n.comment('Hello /usr/local/build-tools/bin')
         self.assertEqual('# Hello\n# /usr/local/build-tools/bin\n',
                          self.out.getvalue())
 
     def test_short_words_indented(self):
-        # Test that indent is taking into acount when breaking subsequent lines.
+        # Test that indent is taking into account when breaking subsequent lines.
         # The second line should not be '    to tree', as that's longer than the
         # test layout width of 8.
         self.n._line('line_one to tree')