Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TiFlash crashes when multiple regions of the same table received concurrently #8217

Closed
JaySon-Huang opened this issue Oct 18, 2023 · 0 comments · Fixed by #8229
Closed

TiFlash crashes when multiple regions of the same table received concurrently #8217

JaySon-Huang opened this issue Oct 18, 2023 · 0 comments · Fixed by #8229
Labels
affects-7.5 component/storage severity/major type/bug The issue is confirmed as a bug.

Comments

@JaySon-Huang
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

https://ci.pingcap.net/blue/organizations/jenkins/tiflash-ghpr-integration-tests/detail/tiflash-ghpr-integration-tests/14745/pipeline/89

tiflash crashes with error logging as below

[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.568 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=439]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.570 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=278]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.580 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=279]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.581 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=280]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.583 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=546]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.585 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=547]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.587 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=318]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.588 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=328]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.589 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=621]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.591 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=672]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.592 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=673]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.594 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=356]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.595 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=374]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.597 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=429]
[2023-10-18T07:22:07.331Z] [2023/10/18 15:21:50.598 +08:00] [WARN] [TiDBSchemaSyncer.cpp:197] ["Can't find related database_id and logical_table_id from table_id_map, try to syncSchemas. physical_table_id=420"] [source="keyspace=4294967295"] [thread_id=384]
[2023-10-18T07:22:07.332Z] [2023/10/18 15:21:50.798 +08:00] [WARN] [InterpreterCreateQuery.cpp:584] ["createTable failed with error code is 337, error info is DB::Exception: Table db_2.t_420 is creating or attaching right now, stack_info is 
       0x41aa8de    StackTrace::StackTrace() [tiflash+68856030]
       0x4199182    DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+68784514]
       0xc72e3c6    DB::DDLGuard::DDLGuard(std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >&, std::__1::mutex&, std::__1::unique_lock<std::__1::mutex>&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+208856006]
       0xc73af23    std::__1::__unique_if<DB::DDLGuard>::__unique_single std::__1::make_unique<DB::DDLGuard, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >&, std::__1::mutex&, std::__1::unique_lock<std::__1::mutex>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >&, std::__1::mutex&, std::__1::unique_lock<std::__1::mutex>&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+208908067]
       0xc72e542    DB::Context::getDDLGuard(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const [tiflash+208856386]
       0xc72e6e4    DB::Context::getDDLGuardIfTableDoesntExist(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const [tiflash+208856804]
       0xc9c6580    DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) [tiflash+211576192]
       0xc9c7d37    DB::InterpreterCreateQuery::execute() [tiflash+211582263]
       0xdcf52d0    DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyCreatePhysicalTable(std::__1::shared_ptr<TiDB::DBInfo> const&, std::__1::shared_ptr<TiDB::TableInfo> const&) [tiflash+231690960]
       0xdcf31e5    DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyTable(long, long, long) [tiflash+231682533]
       0xdd717d5    DB::TiDBSchemaSyncer<false, false>::syncTableSchema(DB::Context&, long) [tiflash+232200149]
       0xc8551b8    DB::TiDBSchemaSyncerManager::syncTableSchema(DB::Context&, unsigned int, long) [tiflash+210063800]
       0xe7c706b    DB::AtomicGetStorageSchema(std::__1::shared_ptr<DB::Region> const&, DB::TMTContext&) [tiflash+243036267]
       0xe6dd0c6    DB::KVStore::preHandleSSTsToDTFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&) [tiflash+242077894]
       0xe6dc9d2    DB::KVStore::preHandleSnapshotToFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, std::__1::optional<unsigned long>, DB::TMTContext&) [tiflash+242076114]
       0xe6b17f1    PreHandleSnapshot [tiflash+241899505]
  0x7efea91d2446    proxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::h518132ab56815620 [libtiflash_proxy.so+144516166]
  0x7efea917105d    engine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl::hf81459364c0001a1 [libtiflash_proxy.so+144117853]
  0x7efea44b7580    engine_store_ffi::core::forward_raft::snapshot::_$LT$impl$u20$engine_store_ffi..core..forwarder..ProxyForwarder$LT$T$C$ER$GT$$GT$::pre_apply_snapshot::_$u7b$$u7b$closure$u7d$$u7d$::hd24fbd9514a2a419 [libtiflash_proxy.so+63665536]
  0x7efea41bb3e2    _$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h41642df54e347a84 [libtiflash_proxy.so+60535778]
  0x7efea4776d29    yatp::task::future::RawTask$LT$F$GT$::poll::h02d660f374e802bb [libtiflash_proxy.so+66546985]
  0x7efeaa7219ee    yatp::task::future::TaskCell::poll::h456b23dfe6dff129 [libtiflash_proxy.so+166861294]
  0x7efeaa7227a4    _$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::he3a0e65ec3d37bb6 [libtiflash_proxy.so+166864804]
  0x7efeaa71ce73    yatp::pool::worker::WorkerThread$LT$T$C$R$GT$::run::hbe089727646bb21a [libtiflash_proxy.so+166841971]
  0x7efeaa6e3f0d    yatp::pool::builder::LazyBuilder$LT$T$GT$::build::_$u7b$$u7b$closure$u7d$$u7d$::hfea975d1d8662764 [libtiflash_proxy.so+166608653]
  0x7efeaa6d872c    std::sys_common::backtrace::__rust_begin_short_backtrace::h4a1b918dd92bb4bb [libtiflash_proxy.so+166561580]
  0x7efeaa70fd6d    std::thread::Builder::spawn_unchecked_::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h584a80ae5b79b471 [libtiflash_proxy.so+166788461]
  0x7efeaa6e7061    _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h11083f11ca183545 [libtiflash_proxy.so+166621281]
  0x7efeaa70ff96    std::panicking::try::do_call::h8105d4df961e925b [libtiflash_proxy.so+166789014]
  0x7efeaa72323b    __rust_try [libtiflash_proxy.so+166867515]
  0x7efeaa70febf    std::panicking::try::h99fd52b9928d9a80 [libtiflash_proxy.so+166788799]
  0x7efeaa70e371    std::panic::catch_unwind::h1258304a0f4464c1 [libtiflash_proxy.so+166781809]"] [source=InterpreterCreateQuery] [thread_id=439]
[2023-10-18T07:22:07.332Z] [2023/10/18 15:21:50.799 +08:00] [WARN] [InterpreterCreateQuery.cpp:584] ["createTable failed with error code is 337, error info is DB::Exception: Table db_2.t_420 is creating or attaching right now, stack_info is 
       0x41aa8de    StackTrace::StackTrace() [tiflash+68856030]
       0x4199182    DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+68784514]
       0xc72e3c6    DB::DDLGuard::DDLGuard(std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >&, std::__1::mutex&, std::__1::unique_lock<std::__1::mutex>&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+208856006]
       0xc73af23    std::__1::__unique_if<DB::DDLGuard>::__unique_single std::__1::make_unique<DB::DDLGuard, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >&, std::__1::mutex&, std::__1::unique_lock<std::__1::mutex>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >&, std::__1::mutex&, std::__1::unique_lock<std::__1::mutex>&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+208908067]
       0xc72e542    DB::Context::getDDLGuard(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const [tiflash+208856386]
       0xc72e6e4    DB::Context::getDDLGuardIfTableDoesntExist(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const [tiflash+208856804]
       0xc9c6580    DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) [tiflash+211576192]
       0xc9c7d37    DB::InterpreterCreateQuery::execute() [tiflash+211582263]
       0xdcf52d0    DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyCreatePhysicalTable(std::__1::shared_ptr<TiDB::DBInfo> const&, std::__1::shared_ptr<TiDB::TableInfo> const&) [tiflash+231690960]
       0xdcf31e5    DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyTable(long, long, long) [tiflash+231682533]
       0xdd717d5    DB::TiDBSchemaSyncer<false, false>::syncTableSchema(DB::Context&, long) [tiflash+232200149]
       0xc8551b8    DB::TiDBSchemaSyncerManager::syncTableSchema(DB::Context&, unsigned int, long) [tiflash+210063800]
       0xe7c706b    DB::AtomicGetStorageSchema(std::__1::shared_ptr<DB::Region> const&, DB::TMTContext&) [tiflash+243036267]
       0xe6dd0c6    DB::KVStore::preHandleSSTsToDTFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&) [tiflash+242077894]
       0xe6dc9d2    DB::KVStore::preHandleSnapshotToFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, std::__1::optional<unsigned long>, DB::TMTContext&) [tiflash+242076114]
       0xe6b17f1    PreHandleSnapshot [tiflash+241899505]
  0x7efea91d2446    proxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::h518132ab56815620 [libtiflash_proxy.so+144516166]
  0x7efea917105d    engine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl::hf81459364c0001a1 [libtiflash_proxy.so+144117853]
  0x7efea44b7580    engine_store_ffi::core::forward_raft::snapshot::_$LT$impl$u20$engine_store_ffi..core..forwarder..ProxyForwarder$LT$T$C$ER$GT$$GT$::pre_apply_snapshot::_$u7b$$u7b$closure$u7d$$u7d$::hd24fbd9514a2a419 [libtiflash_proxy.so+63665536]
  0x7efea41bb3e2    _$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h41642df54e347a84 [libtiflash_proxy.so+60535778]
  0x7efea4776d29    yatp::task::future::RawTask$LT$F$GT$::poll::h02d660f374e802bb [libtiflash_proxy.so+66546985]
  0x7efeaa7219ee    yatp::task::future::TaskCell::poll::h456b23dfe6dff129 [libtiflash_proxy.so+166861294]
  0x7efeaa7227a4    _$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::he3a0e65ec3d37bb6 [libtiflash_proxy.so+166864804]
  0x7efeaa71ce73    yatp::pool::worker::WorkerThread$LT$T$C$R$GT$::run::hbe089727646bb21a [libtiflash_proxy.so+166841971]
  0x7efeaa6e3f0d    yatp::pool::builder::LazyBuilder$LT$T$GT$::build::_$u7b$$u7b$closure$u7d$$u7d$::hfea975d1d8662764 [libtiflash_proxy.so+166608653]
  0x7efeaa6d872c    std::sys_common::backtrace::__rust_begin_short_backtrace::h4a1b918dd92bb4bb [libtiflash_proxy.so+166561580]
  0x7efeaa70fd6d    std::thread::Builder::spawn_unchecked_::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h584a80ae5b79b471 [libtiflash_proxy.so+166788461]
  0x7efeaa6e7061    _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h11083f11ca183545 [libtiflash_proxy.so+166621281]
  0x7efeaa70ff96    std::panicking::try::do_call::h8105d4df961e925b [libtiflash_proxy.so+166789014]
  0x7efeaa72323b    __rust_try [libtiflash_proxy.so+166867515]
  0x7efeaa70febf    std::panicking::try::h99fd52b9928d9a80 [libtiflash_proxy.so+166788799]
  0x7efeaa70e371    std::panic::catch_unwind::h1258304a0f4464c1 [libtiflash_proxy.so+166781809]"] [source=InterpreterCreateQuery] [thread_id=278]
[2023-10-18T07:22:07.333Z] [2023/10/18 15:21:50.799 +08:00] [WARN] [InterpreterCreateQuery.cpp:584] ["createTable failed with error code is 337, error info is DB::Exception: Table db_2.t_420 is creating or attaching right now, stack_info is 
       0x41aa8de    StackTrace::StackTrace() [tiflash+68856030]
       0x4199182    DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+68784514]
       0xc72e3c6    DB::DDLGuard::DDLGuard(std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >&, std::__1::mutex&, std::__1::unique_lock<std::__1::mutex>&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+208856006]
       0xc73af23    std::__1::__unique_if<DB::DDLGuard>::__unique_single std::__1::make_unique<DB::DDLGuard, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >&, std::__1::mutex&, std::__1::unique_lock<std::__1::mutex>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >&, std::__1::mutex&, std::__1::unique_lock<std::__1::mutex>&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+208908067]
       0xc72e542    DB::Context::getDDLGuard(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const [tiflash+208856386]
       0xc72e6e4    DB::Context::getDDLGuardIfTableDoesntExist(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const [tiflash+208856804]
       0xc9c6580    DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) [tiflash+211576192]
       0xc9c7d37    DB::InterpreterCreateQuery::execute() [tiflash+211582263]
       0xdcf52d0    DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyCreatePhysicalTable(std::__1::shared_ptr<TiDB::DBInfo> const&, std::__1::shared_ptr<TiDB::TableInfo> const&) [tiflash+231690960]
       0xdcf31e5    DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyTable(long, long, long) [tiflash+231682533]
       0xdd717d5    DB::TiDBSchemaSyncer<false, false>::syncTableSchema(DB::Context&, long) [tiflash+232200149]
       0xc8551b8    DB::TiDBSchemaSyncerManager::syncTableSchema(DB::Context&, unsigned int, long) [tiflash+210063800]
       0xe7c706b    DB::AtomicGetStorageSchema(std::__1::shared_ptr<DB::Region> const&, DB::TMTContext&) [tiflash+243036267]
       0xe6dd0c6    DB::KVStore::preHandleSSTsToDTFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&) [tiflash+242077894]
       0xe6dc9d2    DB::KVStore::preHandleSnapshotToFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, std::__1::optional<unsigned long>, DB::TMTContext&) [tiflash+242076114]
       0xe6b17f1    PreHandleSnapshot [tiflash+241899505]
  0x7efea91d2446    proxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::h518132ab56815620 [libtiflash_proxy.so+144516166]
  0x7efea917105d    engine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl::hf81459364c0001a1 [libtiflash_proxy.so+144117853]
  0x7efea44b7580    engine_store_ffi::core::forward_raft::snapshot::_$LT$impl$u20$engine_store_ffi..core..forwarder..ProxyForwarder$LT$T$C$ER$GT$$GT$::pre_apply_snapshot::_$u7b$$u7b$closure$u7d$$u7d$::hd24fbd9514a2a419 [libtiflash_proxy.so+63665536]
  0x7efea41bb3e2    _$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h41642df54e347a84 [libtiflash_proxy.so+60535778]
  0x7efea4776d29    yatp::task::future::RawTask$LT$F$GT$::poll::h02d660f374e802bb [libtiflash_proxy.so+66546985]
  0x7efeaa7219ee    yatp::task::future::TaskCell::poll::h456b23dfe6dff129 [libtiflash_proxy.so+166861294]
  0x7efeaa7227a4    _$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::he3a0e65ec3d37bb6 [libtiflash_proxy.so+166864804]
  0x7efeaa71ce73    yatp::pool::worker::WorkerThread$LT$T$C$R$GT$::run::hbe089727646bb21a [libtiflash_proxy.so+166841971]
  0x7efeaa6e3f0d    yatp::pool::builder::LazyBuilder$LT$T$GT$::build::_$u7b$$u7b$closure$u7d$$u7d$::hfea975d1d8662764 [libtiflash_proxy.so+166608653]
  0x7efeaa6d872c    std::sys_common::backtrace::__rust_begin_short_backtrace::h4a1b918dd92bb4bb [libtiflash_proxy.so+166561580]
  0x7efeaa70fd6d    std::thread::Builder::spawn_unchecked_::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h584a80ae5b79b471 [libtiflash_proxy.so+166788461]
  0x7efeaa6e7061    _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h11083f11ca183545 [libtiflash_proxy.so+166621281]
  0x7efeaa70ff96    std::panicking::try::do_call::h8105d4df961e925b [libtiflash_proxy.so+166789014]
  0x7efeaa72323b    __rust_try [libtiflash_proxy.so+166867515]
  0x7efeaa70febf    std::panicking::try::h99fd52b9928d9a80 [libtiflash_proxy.so+166788799]
  0x7efeaa70e371    std::panic::catch_unwind::h1258304a0f4464c1 [libtiflash_proxy.so+166781809]"] [source=InterpreterCreateQuery] [thread_id=279]
[2023-10-18T07:22:07.334Z] [2023/10/18 15:21:50.800 +08:00] [ERROR] [Exception.cpp:91] ["Code: 107, e.displayText() = DB::Exception: Cannot open file /tmp/tiflash/data/db/metadata/db_2/t_420.sql, errno: 2, strerror: No such file or directory: keyspace=4294967295 physical_table_id=-1: (while preHandleSnapshot region_id=604, index=7, term=6), e.what() = DB::Exception, Stack trace:


       0x41aa8de    StackTrace::StackTrace() [tiflash+68856030]
       0x4199182    DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+68784514]
       0x41cd3ba    DB::ErrnoException::ErrnoException(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int) [tiflash+68998074]
       0x41c921e    DB::throwFromErrno(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int) [tiflash+68981278]
       0xc6e72dc    DB::PosixRandomAccessFile::PosixRandomAccessFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, std::__1::shared_ptr<DB::ReadLimiter> const&, std::__1::shared_ptr<DB::FileSegment> const&) [tiflash+208564956]
       0xc6dbb8b    DB::PosixRandomAccessFile* std::__1::construct_at<DB::PosixRandomAccessFile, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, DB::PosixRandomAccessFile*>(DB::PosixRandomAccessFile*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208518027]
       0xc6db91b    void std::__1::allocator_traits<std::__1::allocator<DB::PosixRandomAccessFile> >::construct<DB::PosixRandomAccessFile, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, void, void>(std::__1::allocator<DB::PosixRandomAccessFile>&, DB::PosixRandomAccessFile*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208517403]
       0xc6db61e    std::__1::__shared_ptr_emplace<DB::PosixRandomAccessFile, std::__1::allocator<DB::PosixRandomAccessFile> >::__shared_ptr_emplace<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&>(std::__1::allocator<DB::PosixRandomAccessFile>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208516638]
       0xc6db444    std::__1::shared_ptr<DB::PosixRandomAccessFile> std::__1::allocate_shared<DB::PosixRandomAccessFile, std::__1::allocator<DB::PosixRandomAccessFile>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, void>(std::__1::allocator<DB::PosixRandomAccessFile> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208516164]
       0xc6da9e7    std::__1::shared_ptr<DB::PosixRandomAccessFile> std::__1::make_shared<DB::PosixRandomAccessFile, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, void>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208513511]
       0xc6d688f    DB::FileProvider::newRandomAccessFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::EncryptionPath const&, std::__1::shared_ptr<DB::ReadLimiter> const&, int) const [tiflash+208496783]
       0xc70342c    DB::ReadBufferFromFileProvider::ReadBufferFromFileProvider(std::__1::shared_ptr<DB::FileProvider> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::EncryptionPath const&, unsigned long, std::__1::shared_ptr<DB::ReadLimiter> const&, int, char*, unsigned long) [tiflash+208679980]
       0xc6b2be9    DB::DatabaseTiFlash::alterTable(DB::Context const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::ColumnsDescription const&, std::__1::function<void (DB::IAST&)> const&) [tiflash+208350185]
       0xd4eca4f    DB::updateDeltaMergeTableCreateStatement(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<DB::SortColumnDescription, std::__1::allocator<DB::SortColumnDescription> > const&, DB::ColumnsDescription const&, DB::OrderedNameSet const&, std::__1::optional<std::__1::reference_wrapper<TiDB::TableInfo const> >, unsigned long, DB::Context const&) [tiflash+223267407]
       0xd4ed475    DB::StorageDeltaMerge::alterSchemaChange(std::__1::shared_ptr<DB::RWLock::LockHolderImpl> const&, TiDB::TableInfo&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context const&) [tiflash+223270005]
       0xdcf3799    DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyTable(long, long, long) [tiflash+231683993]
       0xdd717d5    DB::TiDBSchemaSyncer<false, false>::syncTableSchema(DB::Context&, long) [tiflash+232200149]
       0xc8551b8    DB::TiDBSchemaSyncerManager::syncTableSchema(DB::Context&, unsigned int, long) [tiflash+210063800]
       0xe7c706b    DB::AtomicGetStorageSchema(std::__1::shared_ptr<DB::Region> const&, DB::TMTContext&) [tiflash+243036267]
       0xe6dd0c6    DB::KVStore::preHandleSSTsToDTFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&) [tiflash+242077894]
       0xe6dc9d2    DB::KVStore::preHandleSnapshotToFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, std::__1::optional<unsigned long>, DB::TMTContext&) [tiflash+242076114]
       0xe6b17f1    PreHandleSnapshot [tiflash+241899505]
  0x7efea91d2446    proxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::h518132ab56815620 [libtiflash_proxy.so+144516166]
  0x7efea917105d    engine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl::hf81459364c0001a1 [libtiflash_proxy.so+144117853]
  0x7efea44b7580    engine_store_ffi::core::forward_raft::snapshot::_$LT$impl$u20$engine_store_ffi..core..forwarder..ProxyForwarder$LT$T$C$ER$GT$$GT$::pre_apply_snapshot::_$u7b$$u7b$closure$u7d$$u7d$::hd24fbd9514a2a419 [libtiflash_proxy.so+63665536]
  0x7efea41bb3e2    _$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h41642df54e347a84 [libtiflash_proxy.so+60535778]
  0x7efea4776d29    yatp::task::future::RawTask$LT$F$GT$::poll::h02d660f374e802bb [libtiflash_proxy.so+66546985]
  0x7efeaa7219ee    yatp::task::future::TaskCell::poll::h456b23dfe6dff129 [libtiflash_proxy.so+166861294]
  0x7efeaa7227a4    _$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::he3a0e65ec3d37bb6 [libtiflash_proxy.so+166864804]
  0x7efeaa71ce73    yatp::pool::worker::WorkerThread$LT$T$C$R$GT$::run::hbe089727646bb21a [libtiflash_proxy.so+166841971]
  0x7efeaa6e3f0d    yatp::pool::builder::LazyBuilder$LT$T$GT$::build::_$u7b$$u7b$closure$u7d$$u7d$::hfea975d1d8662764 [libtiflash_proxy.so+166608653]
  0x7efeaa6d872c    std::sys_common::backtrace::__rust_begin_short_backtrace::h4a1b918dd92bb4bb [libtiflash_proxy.so+166561580]"] [source="DB::RawCppPtr DB::PreHandleSnapshot(DB::EngineStoreServerWrap *, DB::BaseBuffView, uint64_t, DB::SSTViewVec, uint64_t, uint64_t)"] [thread_id=547]
[2023-10-18T07:22:07.334Z] [2023/10/18 15:21:50.800 +08:00] [ERROR] [Exception.cpp:91] ["Code: 107, e.displayText() = DB::Exception: Cannot open file /tmp/tiflash/data/db/metadata/db_2/t_420.sql, errno: 2, strerror: No such file or directory: keyspace=4294967295 physical_table_id=-1: (while preHandleSnapshot region_id=582, index=7, term=6), e.what() = DB::Exception, Stack trace:


       0x41aa8de    StackTrace::StackTrace() [tiflash+68856030]
       0x4199182    DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+68784514]
       0x41cd3ba    DB::ErrnoException::ErrnoException(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int) [tiflash+68998074]
       0x41c921e    DB::throwFromErrno(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int) [tiflash+68981278]
       0xc6e72dc    DB::PosixRandomAccessFile::PosixRandomAccessFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, std::__1::shared_ptr<DB::ReadLimiter> const&, std::__1::shared_ptr<DB::FileSegment> const&) [tiflash+208564956]
       0xc6dbb8b    DB::PosixRandomAccessFile* std::__1::construct_at<DB::PosixRandomAccessFile, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, DB::PosixRandomAccessFile*>(DB::PosixRandomAccessFile*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208518027]
       0xc6db91b    void std::__1::allocator_traits<std::__1::allocator<DB::PosixRandomAccessFile> >::construct<DB::PosixRandomAccessFile, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, void, void>(std::__1::allocator<DB::PosixRandomAccessFile>&, DB::PosixRandomAccessFile*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208517403]
       0xc6db61e    std::__1::__shared_ptr_emplace<DB::PosixRandomAccessFile, std::__1::allocator<DB::PosixRandomAccessFile> >::__shared_ptr_emplace<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&>(std::__1::allocator<DB::PosixRandomAccessFile>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208516638]
       0xc6db444    std::__1::shared_ptr<DB::PosixRandomAccessFile> std::__1::allocate_shared<DB::PosixRandomAccessFile, std::__1::allocator<DB::PosixRandomAccessFile>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, void>(std::__1::allocator<DB::PosixRandomAccessFile> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208516164]
       0xc6da9e7    std::__1::shared_ptr<DB::PosixRandomAccessFile> std::__1::make_shared<DB::PosixRandomAccessFile, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, void>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208513511]
       0xc6d688f    DB::FileProvider::newRandomAccessFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::EncryptionPath const&, std::__1::shared_ptr<DB::ReadLimiter> const&, int) const [tiflash+208496783]
       0xc70342c    DB::ReadBufferFromFileProvider::ReadBufferFromFileProvider(std::__1::shared_ptr<DB::FileProvider> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::EncryptionPath const&, unsigned long, std::__1::shared_ptr<DB::ReadLimiter> const&, int, char*, unsigned long) [tiflash+208679980]
       0xc6b2be9    DB::DatabaseTiFlash::alterTable(DB::Context const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::ColumnsDescription const&, std::__1::function<void (DB::IAST&)> const&) [tiflash+208350185]
       0xd4eca4f    DB::updateDeltaMergeTableCreateStatement(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<DB::SortColumnDescription, std::__1::allocator<DB::SortColumnDescription> > const&, DB::ColumnsDescription const&, DB::OrderedNameSet const&, std::__1::optional<std::__1::reference_wrapper<TiDB::TableInfo const> >, unsigned long, DB::Context const&) [tiflash+223267407]
       0xd4ed475    DB::StorageDeltaMerge::alterSchemaChange(std::__1::shared_ptr<DB::RWLock::LockHolderImpl> const&, TiDB::TableInfo&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context const&) [tiflash+223270005]
       0xdcf3799    DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyTable(long, long, long) [tiflash+231683993]
       0xdd717d5    DB::TiDBSchemaSyncer<false, false>::syncTableSchema(DB::Context&, long) [tiflash+232200149]
       0xc8551b8    DB::TiDBSchemaSyncerManager::syncTableSchema(DB::Context&, unsigned int, long) [tiflash+210063800]
       0xe7c706b    DB::AtomicGetStorageSchema(std::__1::shared_ptr<DB::Region> const&, DB::TMTContext&) [tiflash+243036267]
       0xe6dd0c6    DB::KVStore::preHandleSSTsToDTFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&) [tiflash+242077894]
       0xe6dc9d2    DB::KVStore::preHandleSnapshotToFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, std::__1::optional<unsigned long>, DB::TMTContext&) [tiflash+242076114]
       0xe6b17f1    PreHandleSnapshot [tiflash+241899505]
  0x7efea91d2446    proxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::h518132ab56815620 [libtiflash_proxy.so+144516166]
  0x7efea917105d    engine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl::hf81459364c0001a1 [libtiflash_proxy.so+144117853]
  0x7efea44b7580    engine_store_ffi::core::forward_raft::snapshot::_$LT$impl$u20$engine_store_ffi..core..forwarder..ProxyForwarder$LT$T$C$ER$GT$$GT$::pre_apply_snapshot::_$u7b$$u7b$closure$u7d$$u7d$::hd24fbd9514a2a419 [libtiflash_proxy.so+63665536]
  0x7efea41bb3e2    _$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h41642df54e347a84 [libtiflash_proxy.so+60535778]
  0x7efea4776d29    yatp::task::future::RawTask$LT$F$GT$::poll::h02d660f374e802bb [libtiflash_proxy.so+66546985]
  0x7efeaa7219ee    yatp::task::future::TaskCell::poll::h456b23dfe6dff129 [libtiflash_proxy.so+166861294]
  0x7efeaa7227a4    _$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::he3a0e65ec3d37bb6 [libtiflash_proxy.so+166864804]
  0x7efeaa71ce73    yatp::pool::worker::WorkerThread$LT$T$C$R$GT$::run::hbe089727646bb21a [libtiflash_proxy.so+166841971]
  0x7efeaa6e3f0d    yatp::pool::builder::LazyBuilder$LT$T$GT$::build::_$u7b$$u7b$closure$u7d$$u7d$::hfea975d1d8662764 [libtiflash_proxy.so+166608653]
  0x7efeaa6d872c    std::sys_common::backtrace::__rust_begin_short_backtrace::h4a1b918dd92bb4bb [libtiflash_proxy.so+166561580]"] [source="DB::RawCppPtr DB::PreHandleSnapshot(DB::EngineStoreServerWrap *, DB::BaseBuffView, uint64_t, DB::SSTViewVec, uint64_t, uint64_t)"] [thread_id=328]
[2023-10-18T07:22:07.335Z] [2023/10/18 15:21:50.801 +08:00] [ERROR] [Exception.cpp:91] ["Code: 107, e.displayText() = DB::Exception: Cannot open file /tmp/tiflash/data/db/metadata/db_2/t_420.sql, errno: 2, strerror: No such file or directory: keyspace=4294967295 physical_table_id=-1: (while preHandleSnapshot region_id=602, index=7, term=6), e.what() = DB::Exception, Stack trace:


       0x41aa8de    StackTrace::StackTrace() [tiflash+68856030]
       0x4199182    DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+68784514]
       0x41cd3ba    DB::ErrnoException::ErrnoException(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int) [tiflash+68998074]
       0x41c921e    DB::throwFromErrno(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int) [tiflash+68981278]
       0xc6e72dc    DB::PosixRandomAccessFile::PosixRandomAccessFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, std::__1::shared_ptr<DB::ReadLimiter> const&, std::__1::shared_ptr<DB::FileSegment> const&) [tiflash+208564956]
       0xc6dbb8b    DB::PosixRandomAccessFile* std::__1::construct_at<DB::PosixRandomAccessFile, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, DB::PosixRandomAccessFile*>(DB::PosixRandomAccessFile*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208518027]
       0xc6db91b    void std::__1::allocator_traits<std::__1::allocator<DB::PosixRandomAccessFile> >::construct<DB::PosixRandomAccessFile, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, void, void>(std::__1::allocator<DB::PosixRandomAccessFile>&, DB::PosixRandomAccessFile*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208517403]
       0xc6db61e    std::__1::__shared_ptr_emplace<DB::PosixRandomAccessFile, std::__1::allocator<DB::PosixRandomAccessFile> >::__shared_ptr_emplace<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&>(std::__1::allocator<DB::PosixRandomAccessFile>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208516638]
       0xc6db444    std::__1::shared_ptr<DB::PosixRandomAccessFile> std::__1::allocate_shared<DB::PosixRandomAccessFile, std::__1::allocator<DB::PosixRandomAccessFile>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, void>(std::__1::allocator<DB::PosixRandomAccessFile> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208516164]
       0xc6da9e7    std::__1::shared_ptr<DB::PosixRandomAccessFile> std::__1::make_shared<DB::PosixRandomAccessFile, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&, void>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int&, std::__1::shared_ptr<DB::ReadLimiter> const&) [tiflash+208513511]
       0xc6d688f    DB::FileProvider::newRandomAccessFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::EncryptionPath const&, std::__1::shared_ptr<DB::ReadLimiter> const&, int) const [tiflash+208496783]
       0xc70342c    DB::ReadBufferFromFileProvider::ReadBufferFromFileProvider(std::__1::shared_ptr<DB::FileProvider> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::EncryptionPath const&, unsigned long, std::__1::shared_ptr<DB::ReadLimiter> const&, int, char*, unsigned long) [tiflash+208679980]
       0xc6b2be9    DB::DatabaseTiFlash::alterTable(DB::Context const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::ColumnsDescription const&, std::__1::function<void (DB::IAST&)> const&) [tiflash+208350185]
       0xd4eca4f    DB::updateDeltaMergeTableCreateStatement(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<DB::SortColumnDescription, std::__1::allocator<DB::SortColumnDescription> > const&, DB::ColumnsDescription const&, DB::OrderedNameSet const&, std::__1::optional<std::__1::reference_wrapper<TiDB::TableInfo const> >, unsigned long, DB::Context const&) [tiflash+223267407]
       0xd4ed475    DB::StorageDeltaMerge::alterSchemaChange(std::__1::shared_ptr<DB::RWLock::LockHolderImpl> const&, TiDB::TableInfo&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context const&) [tiflash+223270005]
       0xdcf3799    DB::SchemaBuilder<DB::SchemaGetter, DB::SchemaNameMapper>::applyTable(long, long, long) [tiflash+231683993]
       0xdd717d5    DB::TiDBSchemaSyncer<false, false>::syncTableSchema(DB::Context&, long) [tiflash+232200149]
       0xc8551b8    DB::TiDBSchemaSyncerManager::syncTableSchema(DB::Context&, unsigned int, long) [tiflash+210063800]
       0xe7c706b    DB::AtomicGetStorageSchema(std::__1::shared_ptr<DB::Region> const&, DB::TMTContext&) [tiflash+243036267]
       0xe6dd0c6    DB::KVStore::preHandleSSTsToDTFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&) [tiflash+242077894]
       0xe6dc9d2    DB::KVStore::preHandleSnapshotToFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, std::__1::optional<unsigned long>, DB::TMTContext&) [tiflash+242076114]
       0xe6b17f1    PreHandleSnapshot [tiflash+241899505]
  0x7efea91d2446    proxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::h518132ab56815620 [libtiflash_proxy.so+144516166]
  0x7efea917105d    engine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl::hf81459364c0001a1 [libtiflash_proxy.so+144117853]
  0x7efea44b7580    engine_store_ffi::core::forward_raft::snapshot::_$LT$impl$u20$engine_store_ffi..core..forwarder..ProxyForwarder$LT$T$C$ER$GT$$GT$::pre_apply_snapshot::_$u7b$$u7b$closure$u7d$$u7d$::hd24fbd9514a2a419 [libtiflash_proxy.so+63665536]
  0x7efea41bb3e2    _$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h41642df54e347a84 [libtiflash_proxy.so+60535778]
  0x7efea4776d29    yatp::task::future::RawTask$LT$F$GT$::poll::h02d660f374e802bb [libtiflash_proxy.so+66546985]
  0x7efeaa7219ee    yatp::task::future::TaskCell::poll::h456b23dfe6dff129 [libtiflash_proxy.so+166861294]
  0x7efeaa7227a4    _$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::he3a0e65ec3d37bb6 [libtiflash_proxy.so+166864804]
  0x7efeaa71ce73    yatp::pool::worker::WorkerThread$LT$T$C$R$GT$::run::hbe089727646bb21a [libtiflash_proxy.so+166841971]
  0x7efeaa6e3f0d    yatp::pool::builder::LazyBuilder$LT$T$GT$::build::_$u7b$$u7b$closure$u7d$$u7d$::hfea975d1d8662764 [libtiflash_proxy.so+166608653]
  0x7efeaa6d872c    std::sys_common::backtrace::__rust_begin_short_backtrace::h4a1b918dd92bb4bb [libtiflash_proxy.so+166561580]"] [source="DB::RawCppPtr DB::PreHandleSnapshot(DB::EngineStoreServerWrap *, DB::BaseBuffView, uint64_t, DB::SSTViewVec, uint64_t, uint64_t)"] [thread_id=318]
[2023-10-18T07:22:07.336Z] [2023/10/18 15:21:50.808 +08:00] [ERROR] [BaseDaemon.cpp:370] [########################################] [source=BaseDaemon] [thread_id=762]

4. What is your TiFlash version? (Required)

d8e8eff
@hongyunyan hongyunyan mentioned this issue Oct 23, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Oct 25, 2023
@hongyunyan
ddl: Fix corner case when multiple region creates one table simultane…
d4edc74 
…ously (#8229)

close #8217
@ti-chi-bot ti-chi-bot mentioned this issue Oct 25, 2023
Merged
12 tasks
ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this issue Oct 25, 2023
@hongyunyan @ti-chi-bot
ddl: Fix corner case when multiple region creates one table simultane…
aa903d4 
…ously (pingcap#8229)

close pingcap#8217
ti-chi-bot bot pushed a commit that referenced this issue Oct 27, 2023
@ti-chi-bot
ddl: Fix corner case when multiple region creates one table simultane…
b6b2d1e 
…ously (#8229) (#8238)

close #8217
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-7.5 component/storage severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

ddl: Fix corner case when multiple region creates one table simultaneously hongyunyan/tiflash
1 participant
@JaySon-Huang