Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Generating crash reproducer segfaults #11593

Closed
stellaraccident opened this issue Dec 18, 2022 · 8 comments
Closed

Generating crash reproducer segfaults #11593

stellaraccident opened this issue Dec 18, 2022 · 8 comments
Assignees
@jpienaar

Comments

@stellaraccident
Copy link
Collaborator

Repro:

D:\sd\SHARK\shark.venv\lib\site-packages\iree\compiler\tools\..\_mlir_libs\iree-compile.exe --iree-input-type=none --iree-vm-bytecode-module-output-format=flatbuffer-binary --iree-hal-target-backends=vulkan --iree-llvm-embedded-linker-path=D:\sd\SHARK\shark.venv\lib\site-packages\iree\compiler\tools\..\_mlir_libs\iree-lld.exe --mlir-print-debuginfo --mlir-print-op-on-diagnostic=false --mlir-pass-pipeline-crash-reproducer=D:\ireetemps\core-reproducer.mlir --iree-llvm-target-cpu-features=host --iree-stream-resource-index-bits=64 --iree-vm-target-index-bits=64 --iree-util-zero-fill-elided-attrs --iree-flow-enable-padding-linalg-ops --iree-flow-linalg-ops-padding-size=32 --iree-flow-enable-conv-img2col-transform 20221218_vae_spirv_bug.mlir

Should fail with a stack trace of some kind (this is from Windows but I expect it is not architecture specific):

PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Exception Code: 0xC0000005
 #0 0x00007fff197fbfd2 mlirTypeIDHashValue (D:\sd\SHARK\shark.venv\lib\site-packages\iree\compiler\_mlir_libs\IREECompiler.dll+0x37fbfd2)
 #1 0x00007fff197fbca7 mlirTypeIDHashValue (D:\sd\SHARK\shark.venv\lib\site-packages\iree\compiler\_mlir_libs\IREECompiler.dll+0x37fbca7)
 #2 0x00007fff197f73f4 mlirTypeIDHashValue (D:\sd\SHARK\shark.venv\lib\site-packages\iree\compiler\_mlir_libs\IREECompiler.dll+0x37f73f4)
 #3 0x00007fff197f8a11 mlirTypeIDHashValue (D:\sd\SHARK\shark.venv\lib\site-packages\iree\compiler\_mlir_libs\IREECompiler.dll+0x37f8a11)
 #4 0x00007fff197f31c0 mlirTypeIDHashValue (D:\sd\SHARK\shark.venv\lib\site-packages\iree\compiler\_mlir_libs\IREECompiler.dll+0x37f31c0)
 #5 0x00007fff197f8445 mlirTypeIDHashValue (D:\sd\SHARK\shark.venv\lib\site-packages\iree\compiler\_mlir_libs\IREECompiler.dll+0x37f8445)
 #6 0x00007fff197f7300 mlirTypeIDHashValue (D:\sd\SHARK\shark.venv\lib\site-packages\iree\compiler\_mlir_libs\IREECompiler.dll+0x37f7300)
 #7 0x00007fff197f8a11 mlirTypeIDHashValue (D:\sd\SHARK\shark.venv\lib\site-packages\iree\compiler\_mlir_libs\IREECompiler.dll+0x37f8a11)
 #8 0x00007fff197f2d94 mlirTypeIDHashValue (D:\sd\SHARK\shark.venv\lib\site-packages\iree\compiler\_mlir_libs\IREECompiler.dll+0x37f2d94)
 #9 0x00007fff17979116 mlirTypeIDHashValue (D:\sd\SHARK\shark.venv\lib\site-packages\iree\compiler\_mlir_libs\IREECompiler.dll+0x1979116)
#10 0x00007fff197f7312 mlirTypeIDHashValue (D:\sd\SHARK\shark.venv\lib\site-packages\iree\compiler\_mlir_libs\IREECompiler.dll+0x37f7312)
#11 0x00007fff197f8a11 mlirTypeIDHashValue (D:\sd\SHARK\shark.venv\lib\site-packages\iree\compiler\_mlir_libs\IREECompiler.dll+0x37f8a11)
#12 0x00007fff197f4868 mlirTypeIDHashValue (D:\sd\SHARK\shark.venv\lib\site-packages\iree\compiler\_mlir_libs\IREECompiler.dll+0x37f4868)
#13 0x00007fff16306cbc mlirDialectGetContext (D:\sd\SHARK\shark.venv\lib\site-packages\iree\compiler\_mlir_libs\IREECompiler.dll+0x306cbc)
#14 0x00007fff199ca212 mlirTypeIDHashValue (D:\sd\SHARK\shark.venv\lib\site-packages\iree\compiler\_mlir_libs\IREECompiler.dll+0x39ca212)
#15 0x00007fff199c9825 mlirTypeIDHashValue (D:\sd\SHARK\shark.venv\lib\site-packages\iree\compiler\_mlir_libs\IREECompiler.dll+0x39c9825)
#16 0x00007fffdb156c0c (C:\WINDOWS\System32\ucrtbase.dll+0x26c0c)
#17 0x00007fffdc7f5550 (C:\WINDOWS\System32\KERNEL32.DLL+0x15550)
#18 0x00007fffdd84485b (C:\WINDOWS\SYSTEM32\ntdll.dll+0x485b)

(side note: we should figure out how to remove that hardcoded reference to the LLVM github issues)
@jpienaar
Copy link
Member

Re side note: we should run llvm::setBugReportMsg, I can do that along with this.

@jpienaar
Copy link
Member

#11602 should handle side-note. Looking at repro'ing segfault

@jpienaar
Copy link
Member

Stack on Linux

Please report issues to https://github.com/iree-org/iree/issues and include the crash backtrace.
 #0 0x000055f4429981cd llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /proc/self/cwd/external/llvm-project/llvm/lib/Support/Unix/Signals.inc:567:11
 #1 0x000055f44299865b PrintStackTraceSignalHandler(void*) /proc/self/cwd/external/llvm-project/llvm/lib/Support/Unix/Signals.inc:641:1
 #2 0x000055f4429960d6 llvm::sys::RunSignalHandlers() /proc/self/cwd/external/llvm-project/llvm/lib/Support/Signals.cpp:104:5
 #3 0x000055f442998da5 SignalHandler(int) /proc/self/cwd/external/llvm-project/llvm/lib/Support/Unix/Signals.inc:412:1
 #4 0x00007f105203daa0 (/lib/x86_64-linux-gnu/libc.so.6+0x3daa0)
 #5 0x000055f442559d95 formatPassOpReproducerMessage(mlir::Diagnostic&, std::pair<mlir::Pass*, mlir::Operation*>) /proc/self/cwd/external/llvm-project/mlir/lib/Pass/PassCrashRecovery.cpp:225:34
 #6 0x000055f44255b877 mlir::detail::PassCrashReproducerGenerator::finalize(mlir::Operation*, mlir::LogicalResult)::$_1::operator()(std::pair<mlir::Pass*, mlir::Operation*> const&) const /proc/self/cwd/external/llvm-project/mlir/lib/Pass/PassCrashRecovery.cpp:259:27
 #7 0x000055f44255b821 void llvm::interleave<__gnu_cxx::__normal_iterator<std::pair<mlir::Pass*, mlir::Operation*> const*, std::vector<std::pair<mlir::Pass*, mlir::Operation*>, std::allocator<std::pair<mlir::Pass*, mlir::Operation*> > > >, mlir::detail::PassCrashReproducerGenerator::finalize(mlir::Operation*, mlir::LogicalResult)::$_1, void llvm::interleave<llvm::SetVector<std::pair<mlir::Pass*, mlir::Operation*>, std::vector<std::pair<mlir::Pass*, mlir::Operation*>, std::allocator<std::pair<mlir::Pass*, mlir::Operation*> > >, llvm::DenseSet<std::pair<mlir::Pass*, mlir::Operation*>, llvm::DenseMapInfo<std::pair<mlir::Pass*, mlir::Operation*>, void> > >, mlir::detail::PassCrashReproducerGenerator::finalize(mlir::Operation*, mlir::LogicalResult)::$_1, mlir::Diagnostic, std::pair<mlir::Pass*, mlir::Operation*> const>(llvm::SetVector<std::pair<mlir::Pass*, mlir::Operation*>, std::vector<std::pair<mlir::Pass*, mlir::Operation*>, std::allocator<std::pair<mlir::Pass*, mlir::Operation*> > >, llvm::DenseSet<std::pair<mlir::Pass*, mlir::Operation*>, llvm::DenseMapInfo<std::pair<mlir::Pass*, mlir::Operation*>, void> > > const&, mlir::Diagnostic&, mlir::detail::PassCrashReproducerGenerator::finalize(mlir::Operation*, mlir::LogicalResult)::$_1, llvm::StringRef const&)::'lambda'(), void>(llvm::SetVector<std::pair<mlir::Pass*, mlir::Operation*>, std::vector<std::pair<mlir::Pass*, mlir::Operation*>, std::allocator<std::pair<mlir::Pass*, mlir::Operation*> > >, llvm::DenseSet<std::pair<mlir::Pass*, mlir::Operation*>, llvm::DenseMapInfo<std::pair<mlir::Pass*, mlir::Operation*>, void> > >, llvm::SetVector<std::pair<mlir::Pass*, mlir::Operation*>, std::vector<std::pair<mlir::Pass*, mlir::Operation*>, std::allocator<std::pair<mlir::Pass*, mlir::Operation*> > >, llvm::DenseSet<std::pair<mlir::Pass*, mlir::Operation*>, llvm::DenseMapInfo<std::pair<mlir::Pass*, mlir::Operation*>, void> > >, mlir::detail::PassCrashReproducerGenerator::finalize(mlir::Operation*, mlir::LogicalResult)::$_1, mlir::Diagnostic) /proc/self/cwd/external/llvm-project/llvm/include/llvm/ADT/STLExtras.h:2064:24
 #8 0x000055f44255b783 void llvm::interleave<llvm::SetVector<std::pair<mlir::Pass*, mlir::Operation*>, std::vector<std::pair<mlir::Pass*, mlir::Operation*>, std::allocator<std::pair<mlir::Pass*, mlir::Operation*> > >, llvm::DenseSet<std::pair<mlir::Pass*, mlir::Operation*>, llvm::DenseMapInfo<std::pair<mlir::Pass*, mlir::Operation*>, void> > >, mlir::detail::PassCrashReproducerGenerator::finalize(mlir::Operation*, mlir::LogicalResult)::$_1, mlir::Diagnostic, std::pair<mlir::Pass*, mlir::Operation*> const>(llvm::SetVector<std::pair<mlir::Pass*, mlir::Operation*>, std::vector<std::pair<mlir::Pass*, mlir::Operation*>, std::allocator<std::pair<mlir::Pass*, mlir::Operation*> > >, llvm::DenseSet<std::pair<mlir::Pass*, mlir::Operation*>, llvm::DenseMapInfo<std::pair<mlir::Pass*, mlir::Operation*>, void> > > const&, mlir::Diagnostic&, mlir::detail::PassCrashReproducerGenerator::finalize(mlir::Operation*, mlir::LogicalResult)::$_1, llvm::StringRef const&) /proc/self/cwd/external/llvm-project/llvm/include/llvm/ADT/STLExtras.h:2085:1
 #9 0x000055f442559d51 void llvm::interleaveComma<llvm::SetVector<std::pair<mlir::Pass*, mlir::Operation*>, std::vector<std::pair<mlir::Pass*, mlir::Operation*>, std::allocator<std::pair<mlir::Pass*, mlir::Operation*> > >, llvm::DenseSet<std::pair<mlir::Pass*, mlir::Operation*>, llvm::DenseMapInfo<std::pair<mlir::Pass*, mlir::Operation*>, void> > >, mlir::detail::PassCrashReproducerGenerator::finalize(mlir::Operation*, mlir::LogicalResult)::$_1, mlir::Diagnostic, std::pair<mlir::Pass*, mlir::Operation*> const>(llvm::SetVector<std::pair<mlir::Pass*, mlir::Operation*>, std::vector<std::pair<mlir::Pass*, mlir::Operation*>, std::allocator<std::pair<mlir::Pass*, mlir::Operation*> > >, llvm::DenseSet<std::pair<mlir::Pass*, mlir::Operation*>, llvm::DenseMapInfo<std::pair<mlir::Pass*, mlir::Operation*>, void> > > const&, mlir::Diagnostic&, mlir::detail::PassCrashReproducerGenerator::finalize(mlir::Operation*, mlir::LogicalResult)::$_1) /proc/self/cwd/external/llvm-project/llvm/include/llvm/ADT/STLExtras.h:2099:1
#10 0x000055f4425599b2 mlir::detail::PassCrashReproducerGenerator::finalize(mlir::Operation*, mlir::LogicalResult) /proc/self/cwd/external/llvm-project/mlir/lib/Pass/PassCrashRecovery.cpp:256:5
#11 0x000055f44255cb56 (anonymous namespace)::CrashReproducerInstrumentation::runAfterPassFailed(mlir::Pass*, mlir::Operation*) /proc/self/cwd/external/llvm-project/mlir/lib/Pass/PassCrashRecovery.cpp:363:3
#12 0x000055f44254279b mlir::PassInstrumentor::runAfterPassFailed(mlir::Pass*, mlir::Operation*) /proc/self/cwd/external/llvm-project/mlir/lib/Pass/Pass.cpp:975:12
#13 0x000055f442542477 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) /proc/self/cwd/external/llvm-project/mlir/lib/Pass/Pass.cpp:498:7
#14 0x000055f442542b6f mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) /proc/self/cwd/external/llvm-project/mlir/lib/Pass/Pass.cpp:529:16
#15 0x000055f442548e78 mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::$_14::operator()(mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo&) const /proc/self/cwd/external/llvm-project/mlir/lib/Pass/Pass.cpp:749:36
#16 0x000055f442548a59 mlir::LogicalResult mlir::failableParallelForEach<__gnu_cxx::__normal_iterator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo*, std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo> > >, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::$_14&>(mlir::MLIRContext*, __gnu_cxx::__normal_iterator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo*, std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo> > >, __gnu_cxx::__normal_iterator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo*, std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo> > >, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::$_14&) /proc/self/cwd/external/llvm-project/mlir/include/mlir/IR/Threading.h:46:18

mlir::detail::PassCrashReproducerGenerator::finalize call inside mlir::detail::PassCrashReproducerGenerator::finalize call suspect.

@jpienaar
Copy link
Member

Address 0x613000c8ff80 is a wild pointer inside of access range of size 0x000000000008

AddressSanitizer: heap-buffer-overflow on address

Unfortunately TSAN & MSAN is failing to build via cmake (well tsan is failing to configure while msan fails before I get to interesting parts)

@jpienaar
Copy link
Member

I don't have the smoking gun yet, but crash is in llvm-project/mlir/lib/Pass/PassCrashRecovery.cpp when creating additional note about which passes are running (llvm::interleaveComma(impl->runningPasses, note ...), if I disable that, I get fine run in one case and what seems to be like continuous reproducer creation.

@jpienaar
Copy link
Member

Finished dumping all well without that. Except that the reproducer got clobbered and seems like pipeline keeps running post failure.

@jpienaar
Copy link
Member

Should be fixed by llvm/llvm-project@383329b

@jpienaar
Copy link
Member

Fix integrated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jpienaar jpienaar

Labels
None yet
Projects
No open projects
IREE
Archived in project
Milestone
No milestone
Development

No branches or pull requests
2 participants
@stellaraccident @jpienaar