[Bug] Sector removed from both Seal and Store repo during FinalizeSector

[xingjl6280]

Describe the bug
Here's the procedure:

1. A miner and a remote worker, attaching a same Store nfs as permanent storage
2. During FinalizeSector, miner fetched the sealed sector from remote worker
3. Miner moved the sector into permanent Store disk,
4. Miner requested the remote worker to remove the sector from Seal disk, by identifying the sector by url like below
   http://111.62.109.69:3456/remote/sealed/s-t0723538-2
   http://111.62.109.69:3456/remote/cache/s-t0723538-2
5. The remote worker received the remove request, but delete the sector from both Seal and Store disk, as the sector url didn't tell much!!
6. Then the sector stuck in FinalizeSector, without any data file on disk.
7. Punishment !!

Version (run lotus version):
lotus version 1.8.0+mainnet+git.5c56017db

To Reproduce
I believe my description of the bug could tell how to reproduce.

Expected behavior
Sector in store disk should not be delete like this.

Logs
I mixed the Miner and Worker to show a clear timeline.

[worker] 2021-04-27T23:30:30.227+0800	DEBUG	advmgr	sector-storage/worker_local.go:129	acquired sector {{723538 2} 9} (e:7; a:0): {{0 0} /lotus/storage/seal/unsealed/s-t0723538-2 /lotus/storage/seal/sealed/s-t0723538-2 /lotus/storage/seal/cache/s-t0723538-2}

[worker] 2021-04-27T23:30:30.239+0800	DEBUG	advmgr	sector-storage/worker_local.go:129	acquired sector {{723538 2} 9} (e:4; a:0): {{0 0}   /lotus/storage/seal/cache/s-t0723538-2}

[worker] 2021-04-27T23:30:30.951+0800	INFO	stores	stores/local.go:608	remove /lotus/storage/seal/unsealed/s-t0723538-2

[Miner] 2021-04-27T23:30:30.958+0800	INFO	stores	stores/remote.go:224	Fetch http://111.62.109.69:3456/remote/sealed/s-t0723538-2 -> /lotus/storage/store/sealed/fetching/s-t0723538-2

[worker] 2021-04-27T23:30:30.982+0800	INFO	stores	stores/http_handler.go:58	SERVE GET /remote/sealed/s-t0723538-2

[Miner] 2021-04-27T23:40:19.425+0800	DEBUG	stores	stores/util_unix.go:28	move sector data	{"from": "/lotus/storage/store/sealed/fetching/s-t0723538-2", "to": "/lotus/storage/store/sealed/s-t0723538-2"}

[Miner] 2021-04-27T23:40:26.299+0800	INFO	stores	stores/remote.go:334	Delete http://111.62.109.69:3456/remote/sealed/s-t0723538-2

[worker] 2021-04-27T23:40:26.316+0800	INFO	stores	stores/http_handler.go:128	SERVE DELETE /remote/sealed/s-t0723538-2
[worker] 2021-04-27T23:40:26.323+0800	INFO	stores	stores/local.go:608	remove /lotus/storage/seal/sealed/s-t0723538-2
[worker] 2021-04-27T23:40:26.325+0800	INFO	stores	stores/local.go:608	remove /lotus/storage/store/sealed/s-t0723538-2

[Miner] 2021-04-27T23:40:31.997+0800	INFO	stores	stores/remote.go:224	Fetch http://111.62.109.69:3456/remote/cache/s-t0723538-2 -> /lotus/storage/store/cache/fetching/s-t0723538-2
[worker] 2021-04-27T23:40:32.014+0800	INFO	stores	stores/http_handler.go:58	SERVE GET /remote/cache/s-t0723538-2
[worker] 

[Miner] 2021-04-27T23:40:33.360+0800	DEBUG	stores	stores/util_unix.go:28	move sector data	{"from": "/lotus/storage/store/cache/fetching/s-t0723538-2", "to": "/lotus/storage/store/cache/s-t0723538-2"}

[Miner] 2021-04-27T23:40:33.368+0800	INFO	stores	stores/remote.go:334	Delete http://111.62.109.69:3456/remote/cache/s-t0723538-2

[worker] 2021-04-27T23:40:33.384+0800	INFO	stores	stores/http_handler.go:128	SERVE DELETE /remote/cache/s-t0723538-2
[worker] 2021-04-27T23:40:33.385+0800	INFO	stores	stores/local.go:608	remove /lotus/storage/seal/cache/s-t0723538-2
[worker] 2021-04-27T23:40:33.396+0800	INFO	stores	stores/local.go:608	remove /lotus/storage/store/cache/s-t0723538-2

[Miner] 2021-04-27T23:40:35.160+0800	DEBUG	advmgr	sector-storage/worker_local.go:129	acquired sector {{723538 2} 9} (e:6; a:0): {{0 0}  /lotus/storage/store/sealed/s-t0723538-2 /lotus/storage/store/cache/s-t0723538-2}

Screenshots

Additional context
How to recover the NO-DATA sector, and How to retrieve my pledged money?

and for now, most urgently, how could I prevent this happening again?
I set the fetchLimit of the miner to 0, could this be a temporary workaround?

[xingjl6280]

Still no reply?

[rjan90]

It seems like this issue has been fixed with 7494, and can be closed now. #rengjøring