CRuby test failures

[kddnewton]

When fully switching everything over to Prism, there were some additional test failures because the Ruby process was manually invoked in the test suite and it wasn't respecting --parser=prism. We need to fix these test failures to get the parser merged. They are categorized below.

• Missing some method call argument optimizations

We need to implement all of the same recent optimizations to setup_args that were added in 3.4. This largely involves avoiding array and hash allocations. This is done through checking for specific patterns and using a combination of flags, instructions like pushtoarray and concattoarray, and generally avoiding any unnecessary arrays.

TestAllocation::MethodCall#test_anonymous_splat_and_anonymous_keyword_splat_parameters
TestAllocation::MethodCall#test_argument_forwarding
TestAllocation::MethodCall#test_keyword_and_keyword_splat_parameter
TestAllocation::MethodCall#test_keyword_parameter
TestAllocation::MethodCall#test_keyword_splat_parameter
TestAllocation::MethodCall#test_nested_anonymous_splat_and_anonymous_keyword_splat_parameters
TestAllocation::MethodCall#test_nested_argument_forwarding
TestAllocation::MethodCall#test_no_array_allocation_with_splat_and_nonstatic_keywords
TestAllocation::MethodCall#test_no_parameters
TestAllocation::MethodCall#test_optional_parameter
TestAllocation::MethodCall#test_positional_splat_and_keyword_parameter
TestAllocation::MethodCall#test_positional_splat_and_keyword_splat_parameter
TestAllocation::MethodCall#test_required_and_keyword_splat_parameter
TestAllocation::MethodCall#test_required_parameter
TestAllocation::MethodCall#test_required_positional_and_keyword_parameter
TestAllocation::MethodCall#test_ruby2_keywords

• Syntax error message format does not match

In some cases, the error message generated by Prism does not match the error message generated by the current parser. We need to match this so that the test assertion does not have to change.

TestException#test_syntax_error_detailed_message
TestParse#test_unclosed_unicode_escape_at_eol_bug_19750
TestRegexp#test_extended_comment_invalid_escape_bug_18294
TestRegexp#test_nonextended_section_of_extended_regexp_bug_19379

• Unused block warning was not implemented for Prism

We need to implement the unused block warning for Prism, since it was only implemented for compile.c. This will involve implementing the same changes to prism_compile.c as in this commit: ruby/ruby@e9d7478dedb.

TestMethod#test_warn_unused_block

• Not processing switches from shebang

Some files have shebangs with additional switches after them, like #!ruby -s -, which could potentially set additional global variables from the command line. We now use a callback to process those, but are missing something to do with encoding to make this test pass.

TestRubyOptions#test_shebang

• Not handling FIFO objects

We likely need to match the behavior from this commit: ruby/ruby@bc8687acd62. Essentially since we're going through a different codepath than CRuby for reading source files, we need to make sure we have the same behavior. This has something to do with file systems that is outside my knowledge area, so extensive testing will be required.

TestRequire#test_loading_fifo_fd_leak
TestRequire#test_loading_fifo_threading_raise
TestRequire#test_loading_fifo_threading_success

• Unknown

I'm not sure why these tests are failing, and they need additional investigation.

TestCoverage#test_coverage_in_main_script
TestIO#test_set_lineno
TestTRICK2015#test_ksk_1

Note that to get these failures, you need to have Prism as the default parser, which involves using the following diff:

diff --git a/ruby.c b/ruby.c
index 3c78ada936..d9b7824a98 100644
--- a/ruby.c
+++ b/ruby.c
@@ -1428,10 +1428,10 @@ proc_long_options(ruby_cmdline_options_t *opt, const char *s, long argc, char **
     }
     else if (is_option_with_arg("parser", Qfalse, Qtrue)) {
         if (strcmp("prism", s) == 0) {
-            (*rb_ruby_prism_ptr()) = true;
+            // default behavior
         }
         else if (strcmp("parse.y", s) == 0) {
-            // default behavior
+            (*rb_ruby_prism_ptr()) = false;
         }
         else {
             rb_raise(rb_eRuntimeError, "unknown parser %s", s);
@@ -3077,8 +3077,6 @@ ruby_process_options(int argc, char **argv)
     VALUE iseq;
     const char *script_name = (argc > 0 && argv[0]) ? argv[0] : ruby_engine;
 
-    (*rb_ruby_prism_ptr()) = false;
-
     if (!origarg.argv || origarg.argc <= 0) {
         origarg.argc = argc;
         origarg.argv = argv;
diff --git a/vm.c b/vm.c
index c7ccbc9550..8a66428eeb 100644
--- a/vm.c
+++ b/vm.c
@@ -4461,7 +4461,7 @@ rb_ruby_verbose_ptr(void)
     return &cr->verbose;
 }
 
-static bool prism;
+static bool prism = true;
 
 bool *
 rb_ruby_prism_ptr(void)

[kddnewton]

Closing this in favor of the individual tickets that have been opened.