Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

debugging-sos-lldb-via-core fails with .NET 8 #259

Closed
tmds opened this issue Jun 16, 2023 · 3 comments · Fixed by #280
Closed

debugging-sos-lldb-via-core fails with .NET 8 #259

tmds opened this issue Jun 16, 2023 · 3 comments · Fixed by #280

Comments

@tmds
Copy link
Member

tmds commented Jun 16, 2023

Failure on RHEL9 x64 .NET8 vmr preview build:

+ lldb --batch -c coredump.193103 --one-line 'gcroot 7fb2f8839f28'
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.	Program arguments: lldb --batch -c coredump.193103 --one-line "gcroot 7fb2f8839f28"
1.	HandleCommand(command = "gcroot 7fb2f8839f28")�
 #0 0x00007f01da3a8e45 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/lib64/libLLVM-15.so+0xba8e45)
 #1 0x00007f01da3a6ee4 llvm::sys::RunSignalHandlers() (/lib64/libLLVM-15.so+0xba6ee4)
 #2 0x00007f01da3a933d (/lib64/libLLVM-15.so+0xba933d)
 #3 0x00007f01c76989ae IsRunningOnAlternateStack(void*) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/pal/src/exception/signal.cpp:318:9
 #4 0x00007f01c76989ae invoke_previous_action(sigaction*, int, siginfo_t*, void*, bool) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/pal/src/exception/signal.cpp:414:31
 #5 0x00007f01c7697e97 sigsegv_handler(int, siginfo_t*, void*) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/pal/src/exception/signal.cpp:0:5
 #6 0x00007f01d9054df0 __restore_rt (/lib64/libc.so.6+0x54df0)
 #7 0x00007ec067f37c05 ReadyToRun_EnclosingTypeMap::GetEnclosingTypeNoThrow(unsigned int, unsigned int*, IMDInternalImport*) const /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/readytoruninfo.cpp:1823:16
 #8 0x00007ec067f37c05 ReadyToRunInfo::GetEnclosingToken(IMDInternalImport*, ModuleBase*, unsigned int, unsigned int*) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/readytoruninfo.cpp:255:16
 #9 0x00007ec067f37c05 ReadyToRunInfo::CompareTypeNameOfTokens(unsigned int, IMDInternalImport*, ModuleBase*, unsigned int, IMDInternalImport*, ModuleBase*) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/readytoruninfo.cpp:299:91
#10 0x00007ec067f372c6 ReadyToRunInfo::TryLookupTypeTokenFromName(NameHandle const*, unsigned int*) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/readytoruninfo.cpp:0:0
#11 0x00007ec067f00fc1 ClassLoader::GetClassValue(NameHandleTable, NameHandle const*, __VoidPtr*, EEClassHashTable**, Module*, HashedTypeEntry*, Loader::LoadFlag, int&) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/clsload.cpp:699:17
#12 0x00007ec067f0171f ClassLoader::FindClassModuleThrowing(NameHandle const*, TypeHandle*, unsigned int*, Module**, unsigned int*, HashedTypeEntry*, Module*, Loader::LoadFlag) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/clsload.cpp:1233:5
#13 0x00007ec067f00545 ClassLoader::LoadTypeHandleThrowing(NameHandle*, ClassLoadLevel, Module*) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/clsload.cpp:1441:14
#14 0x00007ec067f0043f ClassLoader::LoadTypeHandleThrowIfFailed(NameHandle*, ClassLoadLevel, Module*) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/clsload.cpp:364:26
#15 0x00007ec067f0241a ClassLoader::LoadTypeDefOrRefThrowing(ModuleBase*, unsigned int, ClassLoader::NotFoundAction, ClassLoader::PermitUninstantiatedFlag, unsigned int, ClassLoadLevel) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/clsload.cpp:0:29
#16 0x00007ec067f20bf7 SigPointer::GetTypeHandleThrowing(ModuleBase*, SigTypeContext const*, ClassLoader::LoadTypesFlag, ClassLoadLevel, int, Substitution const*, ZapSig::Context const*, MethodTable*, SigPointer::HandleRecursiveGenericsForFieldLayoutLoad*) const /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/siginfo.cpp:1340:43
#17 0x00007ec067f0f8bd MetaSig::GetLastTypeHandleThrowing(ClassLoader::LoadTypesFlag, ClassLoadLevel, int) const /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/siginfo.hpp:0:33
#18 0x00007ec067f0f8bd FieldDesc::LookupFieldTypeHandle(ClassLoadLevel, int) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/field.cpp:154:16
#19 0x00007ec067e85f88 ClrDataAccess::GetFieldDescData(unsigned long, DacpFieldDescData*) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/debug/daccess/request.cpp:1823:37
#20 0x00007f014a2759de 
/tmp/tmp.lB8mXDReBr/dotnet-regular-tests/debugging-sos-lldb-via-core/test.sh: line 18: 193250 Segmentation fault      (core dumped) lldb --batch -c "${coredump}" "${commands[@]}"
Process Exit Code: 139

Failure on RHEL9 arm64 .NET8 vmr preview build:

+ lldb --batch -c coredump.230782 --one-line eestack
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.	Program arguments: lldb --batch -c coredump.230782 --one-line eestack
1.	HandleCommand(command = "eestack")�
 #0 0x0000ffff8b7920e8 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/lib64/libLLVM-15.so+0xb920e8)
 #1 0x0000ffff8b790230 llvm::sys::RunSignalHandlers() (/lib64/libLLVM-15.so+0xb90230)
 #2 0x0000ffff8b792674 (/lib64/libLLVM-15.so+0xb92674)
 #3 0x0000ffff78a662f4 IsRunningOnAlternateStack(void*) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/pal/src/exception/signal.cpp:318:9
 #4 0x0000ffff78a662f4 invoke_previous_action(sigaction*, int, siginfo_t*, void*, bool) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/pal/src/exception/signal.cpp:414:31
 #5 0x0000ffff78a658b8 sigsegv_handler(int, siginfo_t*, void*) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/pal/src/exception/signal.cpp:0:5
 #6 0x0000ffff90faf7d0 (linux-vdso.so.1+0x7d0)
 #7 0x0000ffbe1431b44c MethodTable::GetClass_NoLogging() /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/methodtable.inl:25:18
 #8 0x0000ffbe1431b44c MethodTable::GetClassWithPossibleAV() /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/methodtable.inl:67:12
 #9 0x0000ffbe1431b44c MethodTable::ValidateWithPossibleAV() /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/methodtable.cpp:493:65
#10 0x0000ffbe1427e8d8 DacValidateMD(__DPtr<MethodDesc>) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/debug/daccess/request.cpp:207:20
#11 0x0000ffbe14280f24 ClrDataAccess::GetMethodDescData(unsigned long, unsigned long, DacpMethodDescData*, unsigned int, DacpReJitData*, unsigned int*) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/debug/daccess/request.cpp:852:9
#12 0x0000ffbe2c476e20 (/home/tester/.dotnet/sos/libsos.so+0x76e20)
#13 0x0000ffbe2c441ddc (/home/tester/.dotnet/sos/libsos.so+0x41ddc)
#14 0x0000ffbe2c442174 (/home/tester/.dotnet/sos/libsos.so+0x42174)
#15 0x0000ffbe2c450284 (/home/tester/.dotnet/sos/libsos.so+0x50284)
#16 0x0000ffbe2c450968 EEStack (/home/tester/.dotnet/sos/libsos.so+0x50968)
#17 0x0000ffff78ecc564 sosCommand::DoExecute(lldb::SBDebugger, char**, lldb::SBCommandReturnObject&) (/home/tester/.dotnet/sos/libsosplugin.so+0x23564)
#18 0x0000ffff90206ca8 (/lib64/liblldb.so.15+0x206ca8)
#19 0x0000ffff90503b78 (/lib64/liblldb.so.15+0x503b78)
#20 0x0000ffff904f8980 (/lib64/liblldb.so.15+0x4f8980)
#21 0x0000ffff904fc810 (/lib64/liblldb.so.15+0x4fc810)
#22 0x0000ffff9044572c (/lib64/liblldb.so.15+0x44572c)
#23 0x0000ffff90425924 (/lib64/liblldb.so.15+0x425924)
#24 0x0000ffff904fdf60 (/lib64/liblldb.so.15+0x4fdf60)
#25 0x0000ffff90227728 lldb::SBDebugger::RunCommandInterpreter(lldb::SBCommandInterpreterRunOptions const&) (/lib64/liblldb.so.15+0x227728)
#26 0x0000000000407050 (/usr/bin/lldb+0x407050)
#27 0x0000000000407f04 (/usr/bin/lldb+0x407f04)
#28 0x0000ffff8aa7e79c __libc_start_call_main (/lib64/libc.so.6+0x2c79c)
#29 0x0000ffff8aa7e86c __libc_start_main@GLIBC_2.17 (/lib64/libc.so.6+0x2c86c)
#30 0x00000000004045f0 (/usr/bin/lldb+0x4045f0)
/tmp/tmp.CR2oHVonRO/dotnet-regular-tests/debugging-sos-lldb-via-core/test.sh: line 18: 231219 Segmentation fault      (core dumped) lldb --batch -c "${coredump}" "${commands[@]}"
Process Exit Code: 139

I've not observed failures on RHEL8 so far.

This test was re-enabled mid March (2023).
@omajid have you seen it fail for .NET 6 or .NET 7 since then?
@omajid
Copy link
Member

omajid commented Jun 16, 2023

@omajid have you seen it fail for .NET 6 or .NET 7 since then?

No.

@omajid
Copy link
Member

omajid commented Jun 16, 2023

Which is a bit strange, because gcroot -all is known to crash: dotnet/diagnostics#3726

@tmds
Copy link
Member Author

tmds commented Jun 19, 2023

Also failing on Fedora 38 x64:

+ lldb --batch -c coredump.127169 --one-line 'gcroot 7fa2ea035ec8'
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.	Program arguments: lldb --batch -c coredump.127169 --one-line "gcroot 7fa2ea035ec8"
1.	HandleCommand(command = "gcroot 7fa2ea035ec8")�
 #0 0x00007feafecc350a llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/lib64/libLLVM-16.so+0xcc350a)
 #1 0x00007feafecc0e94 llvm::sys::RunSignalHandlers() (/lib64/libLLVM-16.so+0xcc0e94)
 #2 0x00007feafecc40bb (/lib64/libLLVM-16.so+0xcc40bb)
 #3 0x00007feaeb2d0cce IsRunningOnAlternateStack(void*) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/pal/src/exception/signal.cpp:318:9
 #4 0x00007feaeb2d0cce invoke_previous_action(sigaction*, int, siginfo_t*, void*, bool) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/pal/src/exception/signal.cpp:414:31
 #5 0x00007feaeb2d0345 sigsegv_handler(int, siginfo_t*, void*) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/pal/src/exception/signal.cpp:624:1
 #6 0x00007feafda5fb70 __restore_rt (/lib64/libc.so.6+0x3db70)
 #7 0x00007fa9a10a5b85 ReadyToRun_EnclosingTypeMap::GetEnclosingTypeNoThrow(unsigned int, unsigned int*, IMDInternalImport*) const /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/readytoruninfo.cpp:1823:16
 #8 0x00007fa9a10a5b85 ReadyToRunInfo::GetEnclosingToken(IMDInternalImport*, ModuleBase*, unsigned int, unsigned int*) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/readytoruninfo.cpp:255:16
 #9 0x00007fa9a10a5b85 ReadyToRunInfo::CompareTypeNameOfTokens(unsigned int, IMDInternalImport*, ModuleBase*, unsigned int, IMDInternalImport*, ModuleBase*) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/readytoruninfo.cpp:299:91
#10 0x00007fa9a10a5225 ReadyToRunInfo::TryLookupTypeTokenFromName(NameHandle const*, unsigned int*) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/readytoruninfo.cpp:204:22
#11 0x00007fa9a106eb0e ClassLoader::GetClassValue(NameHandleTable, NameHandle const*, __VoidPtr*, EEClassHashTable**, Module*, HashedTypeEntry*, Loader::LoadFlag, int&) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/clsload.cpp:699:17
#12 0x00007fa9a106f221 ClassLoader::FindClassModuleThrowing(NameHandle const*, TypeHandle*, unsigned int*, Module**, unsigned int*, HashedTypeEntry*, Module*, Loader::LoadFlag) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/clsload.cpp:1233:5
#13 0x00007fa9a106df45 ClassLoader::LoadTypeHandleThrowing(NameHandle*, ClassLoadLevel, Module*) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/clsload.cpp:1441:14
#14 0x00007fa9a106de7f ClassLoader::LoadTypeHandleThrowIfFailed(NameHandle*, ClassLoadLevel, Module*) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/clsload.cpp:364:26
#15 0x00007fa9a106fed2 ClassLoader::LoadTypeDefOrRefThrowing(ModuleBase*, unsigned int, ClassLoader::NotFoundAction, ClassLoader::PermitUninstantiatedFlag, unsigned int, ClassLoadLevel) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/clsload.cpp:0:29
#16 0x00007fa9a108e7c2 SigPointer::GetTypeHandleThrowing(ModuleBase*, SigTypeContext const*, ClassLoader::LoadTypesFlag, ClassLoadLevel, int, Substitution const*, ZapSig::Context const*, MethodTable*, SigPointer::HandleRecursiveGenericsForFieldLayoutLoad*) const /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/siginfo.cpp:1340:43
#17 0x00007fa9a107cd3c FieldDesc::LookupFieldTypeHandle(ClassLoadLevel, int) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/field.cpp:155:1
#18 0x00007fa9a0fef62f ClrDataAccess::GetFieldDescData(unsigned long, DacpFieldDescData*) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/debug/daccess/request.cpp:1823:37
#19 0x00007fea6dec5dde 
LLDB diagnostics will be written to /tmp/diagnostics-64255f
Please include the directory content when filing a bug report
/tmp/tmp.H0Tx3QaQPi/dotnet-regular-tests/debugging-sos-lldb-via-core/test.sh: line 18: 127554 Segmentation fault      (core dumped) lldb --batch -c "${coredump}" "${commands[@]}"
Process Exit Code: 139

and rawhide x64:

+ lldb --batch -c coredump.127149 --one-line 'gcroot 7ec67c83a448'
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.	Program arguments: lldb --batch -c coredump.127149 --one-line "gcroot 7ec67c83a448"
1.	HandleCommand(command = "gcroot 7ec67c83a448")�
 #0 0x00007efd3eec350a llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/lib64/libLLVM-16.so+0xcc350a)
 #1 0x00007efd3eec0e94 llvm::sys::RunSignalHandlers() (/lib64/libLLVM-16.so+0xcc0e94)
 #2 0x00007efd3eec40bb (/lib64/libLLVM-16.so+0xcc40bb)
 #3 0x00007efd2b2d0cae IsRunningOnAlternateStack(void*) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/pal/src/exception/signal.cpp:318:9
 #4 0x00007efd2b2d0cae invoke_previous_action(sigaction*, int, siginfo_t*, void*, bool) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/pal/src/exception/signal.cpp:414:31
 #5 0x00007efd2b2d0325 sigsegv_handler(int, siginfo_t*, void*) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/pal/src/exception/signal.cpp:624:1
 #6 0x00007efd3da3e7d0 __restore_rt (/lib64/libc.so.6+0x3e7d0)
 #7 0x00007ebbe50a5b85 ReadyToRun_EnclosingTypeMap::GetEnclosingTypeNoThrow(unsigned int, unsigned int*, IMDInternalImport*) const /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/readytoruninfo.cpp:1823:16
 #8 0x00007ebbe50a5b85 ReadyToRunInfo::GetEnclosingToken(IMDInternalImport*, ModuleBase*, unsigned int, unsigned int*) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/readytoruninfo.cpp:255:16
 #9 0x00007ebbe50a5b85 ReadyToRunInfo::CompareTypeNameOfTokens(unsigned int, IMDInternalImport*, ModuleBase*, unsigned int, IMDInternalImport*, ModuleBase*) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/readytoruninfo.cpp:299:91
#10 0x00007ebbe50a5225 ReadyToRunInfo::TryLookupTypeTokenFromName(NameHandle const*, unsigned int*) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/readytoruninfo.cpp:204:22
#11 0x00007ebbe506eb0e ClassLoader::GetClassValue(NameHandleTable, NameHandle const*, __VoidPtr*, EEClassHashTable**, Module*, HashedTypeEntry*, Loader::LoadFlag, int&) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/clsload.cpp:699:17
#12 0x00007ebbe506f221 ClassLoader::FindClassModuleThrowing(NameHandle const*, TypeHandle*, unsigned int*, Module**, unsigned int*, HashedTypeEntry*, Module*, Loader::LoadFlag) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/clsload.cpp:1233:5
#13 0x00007ebbe506df45 ClassLoader::LoadTypeHandleThrowing(NameHandle*, ClassLoadLevel, Module*) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/clsload.cpp:1441:14
#14 0x00007ebbe506de7f ClassLoader::LoadTypeHandleThrowIfFailed(NameHandle*, ClassLoadLevel, Module*) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/clsload.cpp:364:26
#15 0x00007ebbe506fed2 ClassLoader::LoadTypeDefOrRefThrowing(ModuleBase*, unsigned int, ClassLoader::NotFoundAction, ClassLoader::PermitUninstantiatedFlag, unsigned int, ClassLoadLevel) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/clsload.cpp:0:29
#16 0x00007ebbe508e7c2 SigPointer::GetTypeHandleThrowing(ModuleBase*, SigTypeContext const*, ClassLoader::LoadTypesFlag, ClassLoadLevel, int, Substitution const*, ZapSig::Context const*, MethodTable*, SigPointer::HandleRecursiveGenericsForFieldLayoutLoad*) const /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/siginfo.cpp:1340:43
#17 0x00007ebbe507cd3c FieldDesc::LookupFieldTypeHandle(ClassLoadLevel, int) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/vm/field.cpp:155:1
#18 0x00007ebbe4fef62f ClrDataAccess::GetFieldDescData(unsigned long, DacpFieldDescData*) /home/tester/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/debug/daccess/request.cpp:1823:37
#19 0x00007efcade95dde 
LLDB diagnostics will be written to /tmp/diagnostics-5153ad
Please include the directory content when filing a bug report
/tmp/tmp.Tr2whynNjE/dotnet-regular-tests/debugging-sos-lldb-via-core/test.sh: line 18: 127439 Segmentation fault      (core dumped) lldb --batch -c "${coredump}" "${commands[@]}"
Process Exit Code: 139

@tmds tmds changed the title debugging-sos-lldb-via-core fails on RHEL9 with .NET 8 debugging-sos-lldb-via-core fails with .NET 8 Jun 19, 2023
@tmds tmds mentioned this issue Aug 29, 2023
Merged
@tmds tmds closed this as completed in #280 Aug 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

debugging-sos-lldb-via-core: re-enable test for .NET 8. tmds/dotnet-regular-tests
2 participants
@omajid @tmds