Factor common prefix text out of Regex alternations

[stephentoub]

Given a regex like "this|that|there", we will now factor out the common prefix into a new node concatenated with the alternation, e.g. "th(?:is|at|ere)". This has a few benefits, including exposing more text to FindFirstChar if this is at the beginning of the sequence, reducing backtracking, and enabling further reduction/optimization opportunities in the alternation.

For example, in this trivial microbenchmark, this|that will be refactored into th(?:is|at), which will cause FindFirstChar to generate a Boyer-Moore search for "th" instead of just looking for 't', and it'll cause Go to generate a check for "th" before the alternation, such that it won't re-check "th" when it has to backtrack and try the second branch of the alternation.

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using System.Text.RegularExpressions;

[MemoryDiagnoser]
public class Program
{
    static void Main(string[] args) => BenchmarkSwitcher.FromAssemblies(new[] { typeof(Program).Assembly }).Run(args);

    private Regex _regex = new Regex(@"this|that", RegexOptions.Compiled);

    [Benchmark] public bool Test() => _regex.IsMatch("now is the time");
}

Method	Toolchain	Mean	Ratio
Test	\master\corerun.exe	60.27 ns	1.00
Test	\pr\corerun.exe	46.22 ns	0.78

Without whitespace diff: https://github.com/dotnet/runtime/pull/2171/files?w=1

Contributes to #1349
cc: @danmosemsft, @eerhardt, @pgovind, @ViktorHofer

[eerhardt]