Kernel VM support post merge cleanup

[mraleph]

Bootstrapping:

• Bring in changes that support bootstrapping from Kernel compiled core libraries
• LookupStaticMethodByKernelProcedure resolve factory redirections manually, this can probably be removed when we bootstrap
• KernelReader::ReadProgram temporary skips dart:vmservice_io to work around some class finalization issues, fix this when we start bootstrapping
• kernel::FlowGraphBuilder::VisitStaticInvocation has some workarounds for hybrid Kernel-app - Source-core-libraries mode, figure out if they can be removed if we bootstrap from kernel.
• Bootstrapping reads kernel file twice. See CreateIsolateAndSetupHelper.

Frontend:

• Kernel compiler seems to generate dynamic bounds in generics sometimes, which is incorrect and is being worked around in KernelReader::ReadPreliminaryClass (tracking bug: kernel/42)

Types:

• kernel::FlowGraphBuilder::TranslateFunctionNode should pass type parameters to generic closures
• kernel::FlowGraphBuilder::LoadInstantiatorTypeArguments can be cleaned up to use Class::IsGeneric.

Low (code sharing):

• We special case identical(x, y) in both parser and kernel graph builder. Figure out if this can be moved to a special shared place.
• kernel::FlowGraphBuilder::MethodKind should be merged and shared with RecognizeArithmeticOp.

Low (async):

• kernel::ScopeBuilder::VisitYieldStatement promotes all visible local variables into the context. It only should promote those that are used across yields.
• :await_jump_var and :await_ctx_var are bad names - they are used for both await and yield.
• kernel::FlowGraphBuilder::VisitYieldStatement distinguishes between await (that can throw) and yield (that can't throw) by number of parameters to the function. The should be a better way.

Low:

• Figure out why binary size of the precompiled runtime grows after Kernel landing even though nothing from the Kernel files should be included.
• Investigate storage growth policy in List<T>::EnsureInitialized
• kImplicitGetter and kImplicitSetter could contain RawField inside RawFunction::data_.

Other:

• Merge source position information changes by @jensjoha
• Change constant caching to use normal TokenPositions
• KernelReader::LookupClass creates a dummy script per class.
• KernelReader::ReadLibrary creates a fake script for each library, is there a way around it?
• KernelReader::ReadPreliminaryClass manually marks interfaces as implemented
• kernel::FlowGraphBuilder::BuildGraphOfFunction does not setup correct scope for the constructor arguments
• Implement checked mode

Fixed:

• Background JIT compiler does work with Kernel
• We reverted part of CaptureLocalVariables that prevented capture of some artifical variables. Instead use set_is_forced_stack.
• Support field guards (see kernel::FlowGraphBuilder::BuildGraphOfFieldAccessor and kernel::FlowGraphBuilder::TranslateInitializers)
• kernel::ScopeBuilder::AddParameter/AddVariable, does not handle final
• kernel::FlowGraphBuilder::StoreIndexed/StoreInstanceField could omit write barrier when storing constants
• kernel::FlowGraphBuilder::VisitInvalidExpression should generate better error messages (not really needed)
• kernel::FlowGraphBuilder::VisitVariableSet has workaround for the rasta issue - IR might contain stores to final/const variables.
• kernel::FlowGraphBuilder::VisitStaticGet contains similarly unreachable code path.
• kernel::FlowGraphBuilder::VisitStaticInvocation has some workarounds for rasta issues, figure out if dartk needs these workarounds
• kernel::ConstantEvaluator::VisitMethodInvocation assumes that method invocation can never result in noSuchMethod, front-end must guarantee that;
• kernel::DartTypeTranslator::VisitFunctionType does not handle function types composed of malformed types correctly
• Ensure that field, variables and parameters are annotated with correct types

[mraleph]

/cc @kmillikin @mkustermann

[jensjoha]

The following (non-existing) test fails now:

import 'dart:io';
import 'package:expect/expect.dart';

testStdIn() {
  stdin;
  return 42;
}

main() {
  Expect.isTrue(testStdIn() == 42);
}

seemingly because of dart:nativewrappers previously (in kernel-sdk) being part of the patched sdk as patched_sdk/lib/nativewrappers with content

library dart.nativewrappers;

class NativeFieldWrapperClass1 extends Object {}
class NativeFieldWrapperClass2 extends Object {}
class NativeFieldWrapperClass3 extends Object {}
class NativeFieldWrapperClass4 extends Object {}

and now (in dart-sdk) as patched_sdk/lib/html/dartium/ with content

library nativewrappers;

class NativeFieldWrapperClass1 {
  NativeFieldWrapperClass1() {
    throw new UnsupportedError(
        "Generative constructors not supported on native types.");
  }
}

class NativeFieldWrapperClass2 {
  NativeFieldWrapperClass2() {
    throw new UnsupportedError(
        "Generative constructors not supported on native types.");
  }
}

class NativeFieldWrapperClass3 {
  NativeFieldWrapperClass3() {
    throw new UnsupportedError(
        "Generative constructors not supported on native types.");
  }
}

class NativeFieldWrapperClass4 {
  NativeFieldWrapperClass4() {
    throw new UnsupportedError(
        "Generative constructors not supported on native types.");
  }
}

meaning that one gets an exception: Unsupported operation: Generative constructors not supported on native types.

[jensjoha]

Just as a note I have the put the "Merge source position information change" in review.

Another note, and whether this is useful or not I'm not sure --- if not feel free to ignore it. When running the kernel tests in debug mode (I know I was told not to) I more or less get these 5 types of errors:

runtime/vm/class_finalizer.cc: 2482: error: expected: cls.IsTopLevel() || cls.IsTypedefClass() || (Array::Handle(cls.functions()).Length() > 0)

Happens < 10 times. E.g. via

python tools/test.py -cdartk -t120 language/issue23244_test

---

runtime/vm/intermediate_language.cc: 51: error: expected: !Compiler::IsBackgroundCompilation() || !field.IsOriginal()

Happens lots of times. E.g. via

python tools/test.py -cdartk -t120 language/regress_26453_test

---

runtime/vm/kernel_to_il.cc: 2453: error: expected: stack_ == NULL

Happens twice. Via

python tools/test.py -cdartk -t120 language/async_star_regression_fisk_test

and

python tools/test.py -cdartk -t120 language/async_star_test

---

runtime/vm/raw_object.cc: 40: error: Invalid class id encountered 0

Happens only on

python tools/test.py -cdartk -t120 co19/Language/Expressions/Shift/syntax_t15

---

runtime/vm/kernel_reader.cc: 130: error: expected: !main_obj.IsNull()

Happens only on

python tools/test.py -cdartk -t120 co19/Language/Libraries_and_Scripts/Scripts/top_level_main_t05

[mraleph]

I have started working on fixing this issues. To track the progress.

• Unsupported operation: Generative constructors not supported on native types.
• ../../runtime/vm/object.cc: 8859: error: expected: Dart::snapshot_kind() == Snapshot::kAppNoJIT (CL with a fix in review)
• ../../runtime/vm/intermediate_language.cc: 51: error: expected: !Compiler::IsBackgroundCompilation() || !field.IsOriginal() (CL with a fix in review)
• ../../runtime/vm/class_finalizer.cc: 2482: error: expected: cls.IsTopLevel() || cls.IsTypedefClass() || (Array::Handle(cls.functions()).Length() > 0)
• ../../runtime/vm/kernel_to_il.cc: 2453: error: expected: stack_ == NULL
• ../../runtime/vm/kernel_reader.cc: 130: error: expected: !main_obj.IsNull()

[mkustermann]

Status file update for debug mode after bootstrapping (see 8cb83ec):

• Invalid class id during isolate shutdown Heap::VerifyGC
• Some out of memory flakiness

[mraleph]

I am attaching diff between dartk and non-dartk expectation for the JIT mode for co19 language suites. Right now it's around 600 tests. We should drive that to 0.
diff.txt

[kasperl]

Filed separate issue for the test expectations; see issue #28263.

[mraleph]

Filed a separate issue for enabling CC tests through kernel based front-end #28264

[a-siva]

@mraleph can you look at your original list here and mark the ones which are done, so we know what is still left.