Add the experimental git survey command to analyze (large) local repo…

…sitories (#667)

This command is inspired by [`git
sizer`](https://github.com/github/git-sizer), having the advantage of
being much closer to the internals of Git.

The intention is to provide a built-in command that can be used to
analyze large repositories for performance and scaling problems, for
growth over time, and to correlate with other measurements (in
particular with Trace2 data collected e.g. via
https://github.com/git-ecosystem/trace2receiver/).

.gitignore

@@ -164,6 +164,7 @@
 /git-submodule
 /git-submodule--helper
 /git-subtree
+/git-survey
 /git-svn
 /git-switch
 /git-symbolic-ref

Documentation/config.txt

@@ -536,6 +536,8 @@ include::config/status.txt[]
 
 include::config/submodule.txt[]
 
+include::config/survey.txt[]
+
 include::config/tag.txt[]
 
 include::config/tar.txt[]

Documentation/config/survey.txt

@@ -0,0 +1,41 @@
+survey.namerev::
+	Boolean to show/hide `git name-rev` information for
+	each reported commit and the containing commit of each
+	reported tree and blob.
+
+survey.progress::
+	Boolean to show/hide progress information.  Defaults to
+	true when interactive (stderr is bound to a TTY).
+
+survey.showBlobSizes::
+	A non-negative integer value.  Requests details on the <n>
+	largest file blobs by size in bytes.  Provides a default
+	value for `--blob-sizes=<n>` in linkgit:git-survey[1].
+
+survey.showCommitParents::
+	A non-negative integer value.  Requests details on the <n>
+	commits with the most number of parents.  Provides a default
+	value for `--commit-parents=<n>` in linkgit:git-survey[1].
+
+survey.showCommitSizes::
+	A non-negative integer value.  Requests details on the <n>
+	largest commits by size in bytes.  Generally, these are the
+	commits with the largest commit messages.  Provides a default
+	value for `--commit-sizes=<n>` in linkgit:git-survey[1].
+
+survey.showTreeEntries::
+	A non-negative integer value.  Requests details on the <n>
+	trees (directories) with the most number of entries (files
+	and subdirectories).  Provides a default value for
+	`--tree-entries=<n>` in linkgit:git-survey[1].
+
+survey.showTreeSizes::
+	A non-negative integer value.  Requests details on the <n>
+	largest trees (directories) by size in bytes.  This will
+	set will usually be equal to the `survey.showTreeEntries`
+	set, but may be skewed by very long file or subdirectory
+	entry names.  Provides a default value for
+	`--tree-sizes=<n>` in linkgit:git-survey[1].
+
+survey.verbose::
+	Boolean to show/hide verbose output.  Default to false.

Documentation/git-survey.txt

@@ -0,0 +1,108 @@
+git-survey(1)
+=============
+
+NAME
+----
+git-survey - EXPERIMENTAL: Measure various repository dimensions of scale
+
+SYNOPSIS
+--------
+[verse]
+(EXPERIMENTAL!) `git survey` <options>
+
+DESCRIPTION
+-----------
+
+Survey the repository and measure various dimensions of scale.
+
+As repositories grow to "monorepo" size, certain data shapes can cause
+performance problems.  `git-survey` attempts to measure and report on
+known problem areas.
+
+Ref Selection and Reachable Objects
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In this first analysis phase, `git survey` will iterate over the set of
+requested branches, tags, and other refs and treewalk over all of the
+reachable commits, trees, and blobs and generate various statistics.
+
+OPTIONS
+-------
+
+--progress::
+	Show progress.  This is automatically enabled when interactive.
+
+--json::
+	Print results in JSON rather than in a human-friendly format.
+
+--[no-]name-rev::
+	Print `git name-rev` output for each commit, tree, and blob.
+	Defaults to true.
+
+Ref Selection
+~~~~~~~~~~~~~
+
+The following options control the set of refs that `git survey` will examine.
+By default, `git survey` will look at tags, local branches, and remote refs.
+If any of the following options are given, the default set is cleared and
+only refs for the given options are added.
+
+--all-refs::
+	Use all refs.  This includes local branches, tags, remote refs,
+	notes, and stashes.  This option overrides all of the following.
+
+--branches::
+	Add local branches (`refs/heads/`) to the set.
+
+--tags::
+	Add tags (`refs/tags/`) to the set.
+
+--remotes::
+	Add remote branches (`refs/remote/`) to the set.
+
+--detached::
+	Add HEAD to the set.
+
+--other::
+	Add notes (`refs/notes/`) and stashes (`refs/stash/`) to the set.
+
+Large Item Selection
+~~~~~~~~~~~~~~~~~~~~
+
+The following options control the optional display of large items under
+various dimensions of scale.  The OID of the largest `n` objects will be
+displayed in reverse sorted order.  For each, `n` defaults to 10.
+
+--commit-parents::
+	Shows the OIDs of the commits with the most parent commits.
+
+--commit-sizes::
+	Shows the OIDs of the largest commits by size in bytes.  This is
+	usually the ones with the largest commit messages.
+
+--tree-entries::
+	Shows the OIDs of the trees with the most number of entries.  These
+	are the directories with the most number of files or subdirectories.
+
+--tree-sizes::
+	Shows the OIDs of the largest trees by size in bytes.  This set
+	will usually be the same as the vector of number of entries unless
+	skewed by very long entry names.
+
+--blob-sizes::
+	Shows the OIDs of the largest blobs by size in bytes.
+
+OUTPUT
+------
+
+By default, `git survey` will print information about the repository in a
+human-readable format that includes overviews and tables.
+
+CONFIGURATION
+-------------
+
+include::config/survey.txt[]
+
+GIT
+---
+Part of the linkgit:git[1] suite

Makefile

@@ -1309,6 +1309,7 @@ BUILTIN_OBJS += builtin/sparse-checkout.o
 BUILTIN_OBJS += builtin/stash.o
 BUILTIN_OBJS += builtin/stripspace.o
 BUILTIN_OBJS += builtin/submodule--helper.o
+BUILTIN_OBJS += builtin/survey.o
 BUILTIN_OBJS += builtin/symbolic-ref.o
 BUILTIN_OBJS += builtin/tag.o
 BUILTIN_OBJS += builtin/unpack-file.o

builtin.h

@@ -238,6 +238,7 @@ int cmd_status(int argc, const char **argv, const char *prefix);
 int cmd_stash(int argc, const char **argv, const char *prefix);
 int cmd_stripspace(int argc, const char **argv, const char *prefix);
 int cmd_submodule__helper(int argc, const char **argv, const char *prefix);
+int cmd_survey(int argc, const char **argv, const char *prefix);
 int cmd_switch(int argc, const char **argv, const char *prefix);
 int cmd_symbolic_ref(int argc, const char **argv, const char *prefix);
 int cmd_tag(int argc, const char **argv, const char *prefix);