[Deal Making Issue]

[jamerduhgamer]

Basic Information
Client and Miner side for online retrieval deal.

Describe the problem
The client is trying to retrieval an online storage deal and is running into "ERROR: retrieval failed: Retrieve: Retrieval Error: error generated by data transfer: deal data transfer failed: incomplete response".

Version
lotus version 1.5.3+mainnet+git.358773e2b

Setup
GeForce RTX 2080Ti
256 GB RAM
2 TB NVMe M.2
AMD EPYC 7F72 24-Core Processor

To Reproduce
Steps to reproduce the behavior:

1. lotus client retrieve --miner f066596 bafyaa6qsgafcmalqudsaeiekkmiypfxq6vgdwepdafg6jyacye46gurjichuapbzxwqhoywq7yjaagelucbyabasgafcmalqudsaeif3e6tsyc7c3anlmobix3744duq7cyasjsmrsmbkdn4besizu5ffyjaagf6zcc36aykcqeaeghh3sbl6bzaqcaibaaeedt5zav7am output-test.car
2. See error

Deal status
Creation Verified ProposalCid DealId State Client
Size Price Duration TransferChannelID Message
Mar 31 18:26:52 false bafyreia2kdy6sad3be2hnb5ywvno7jrxcvr7sa5q7qytuv2lxtqxvhlzcq 1653845 StorageDealActive
f3rxhdrohdvzunxweebfvnch6izl4rron7qo7gb5x5czu47xdg25g3ys57tovcwoz2wgsr2ystugfyimcmhpbq 2GiB 0.0104405 FIL 522025 12D3KooWAWe8axhW2GM6MmuDSzE1Ln5BZdgvX9swhCsUySUncTHp-12D3KooWH3mBSF2RFQJFZdqzkVBqSdspoAAunHkViTf5AQhubv1c-0

Lotus daemon and miner logs
Client Logs
lotus client retrieve --miner f066596 bafyaa6qsgafcmalqudsaeiekkmiypfxq6vgdwepdafg6jyacye46gurjichuapbzxwqhoywq7yjaagelucbyabasgafcmalqudsaeif3e6tsyc7c3anlmobix3744duq7cyasjsmrsmbkdn4besizu5ffyjaagf6zcc36aykcqeaeghh3sbl6bzaqcaibaaeedt5zav7am output-test.car

Recv: 0 B, Paid 0 FIL, ClientEventOpen (DealStatusNew)
Recv: 0 B, Paid 0 FIL, ClientEventDealProposed (DealStatusWaitForAcceptance)
Recv: 122 B, Paid 0 FIL, ClientEventBlocksReceived (DealStatusWaitForAcceptance)
Recv: 122 B, Paid 0 FIL, ClientEventDealAccepted (DealStatusAccepted)
Recv: 122 B, Paid 0 FIL, ClientEventPaymentChannelCreateInitiated (DealStatusPaymentChannelCreating)
Recv: 122 B, Paid 0 FIL, ClientEventDataTransferError (DealStatusErrored)
Recv: 0 B, Paid 0 FIL, ClientEventOpen (DealStatusNew)
ERROR: retrieval failed: Retrieve: Retrieval Error: error generated by data transfer: deal data transfer failed: incomplete response

Miner Logs
2021-04-05T23:31:51.091-0700 INFO dt-impl impl/events.go:255 received new channel request from 12D3KooWAWe8axhW2GM6MmuDSzE1Ln5BZdgvX9swhCsUySUncTHp
2021-04-05T23:31:51.092-0700 INFO markets loggers/loggers.go:30 retrieval provider event {"name": "ProviderEventOpen", "deal ID": "1", "receiver": "12D3KooWAWe8axhW2GM6MmuDSzE1Ln5BZdgvX9swhCsUySUncTHp", "state": "DealStatusNew", "message": ""}
2021-04-05T23:31:51.093-0700 INFO markets loggers/loggers.go:30 retrieval provider event {"name": "ProviderEventDealAccepted", "deal ID": "1", "receiver": "12D3KooWAWe8axhW2GM6MmuDSzE1Ln5BZdgvX9swhCsUySUncTHp", "state": "DealStatusUnsealing", "message": ""}
2021-04-05T23:32:02.362-0700 INFO markets loggers/loggers.go:30 retrieval provider event {"name": "ProviderEventUnsealComplete", "deal ID": "1", "receiver": "12D3KooWAWe8axhW2GM6MmuDSzE1Ln5BZdgvX9swhCsUySUncTHp", "state": "DealStatusUnsealed", "message": ""}
2021-04-05T23:32:02.362-0700 INFO dt-impl impl/impl.go:386 resume channel 12D3KooWAWe8axhW2GM6MmuDSzE1Ln5BZdgvX9swhCsUySUncTHp-12D3KooWH3mBSF2RFQJFZdqzkVBqSdspoAAunHkViTf5AQhubv1c-2
2021-04-05T23:32:02.363-0700 INFO markets loggers/loggers.go:30 retrieval provider event {"name": "ProviderEventBlockSent", "deal ID": "1", "receiver": "12D3KooWAWe8axhW2GM6MmuDSzE1Ln5BZdgvX9swhCsUySUncTHp", "state": "DealStatusOngoing", "message": ""}
2021-04-05T23:32:02.364-0700 WARN dt-impl impl/events.go:235 data transfer channel 12D3KooWAWe8axhW2GM6MmuDSzE1Ln5BZdgvX9swhCsUySUncTHp-12D3KooWH3mBSF2RFQJFZdqzkVBqSdspoAAunHkViTf5AQhubv1c-2 failed to transfer data: graphsync response to peer 12D3KooWAWe8axhW2GM6MmuDSzE1Ln5BZdgvX9swhCsUySUncTHp did not complete: response status code RequestCompletedPartial
2021-04-05T23:32:02.364-0700 INFO markets loggers/loggers.go:30 retrieval provider event {"name": "ProviderEventDataTransferError", "deal ID": "1", "receiver": "12D3KooWAWe8axhW2GM6MmuDSzE1Ln5BZdgvX9swhCsUySUncTHp", "state": "DealStatusErrored", "message": "deal data transfer failed: data transfer channel 12D3KooWAWe8axhW2GM6MmuDSzE1Ln5BZdgvX9swhCsUySUncTHp-12D3KooWH3mBSF2RFQJFZdqzkVBqSdspoAAunHkViTf5AQhubv1c-2 failed to transfer data: graphsync response to peer 12D3KooWAWe8axhW2GM6MmuDSzE1Ln5BZdgvX9swhCsUySUncTHp did not complete: response status code RequestCompletedPartial"}
2021-04-05T23:32:44.771-0700 INFO graphsync impl/graphsync.go:309 Graphsync ReceiveError from 12D3KooWAWe8axhW2GM6MmuDSzE1Ln5BZdgvX9swhCsUySUncTHp: stream reset
2021-04-06T01:59:33.973-0700 INFO dt-impl impl/events.go:255 received new channel request from 12D3KooWAWe8axhW2GM6MmuDSzE1Ln5BZdgvX9swhCsUySUncTHp
2021-04-06T01:59:33.975-0700 INFO markets loggers/loggers.go:30 retrieval provider event {"name": "ProviderEventOpen", "deal ID": "2", "receiver": "12D3KooWAWe8axhW2GM6MmuDSzE1Ln5BZdgvX9swhCsUySUncTHp", "state": "DealStatusNew", "message": ""}
2021-04-06T01:59:33.976-0700 INFO markets loggers/loggers.go:30 retrieval provider event {"name": "ProviderEventDealAccepted", "deal ID": "2", "receiver": "12D3KooWAWe8axhW2GM6MmuDSzE1Ln5BZdgvX9swhCsUySUncTHp", "state": "DealStatusUnsealing", "message": ""}
2021-04-06T01:59:43.948-0700 INFO markets loggers/loggers.go:30 retrieval provider event {"name": "ProviderEventUnsealComplete", "deal ID": "2", "receiver": "12D3KooWAWe8axhW2GM6MmuDSzE1Ln5BZdgvX9swhCsUySUncTHp", "state": "DealStatusUnsealed", "message": ""}
2021-04-06T01:59:43.948-0700 INFO dt-impl impl/impl.go:386 resume channel 12D3KooWAWe8axhW2GM6MmuDSzE1Ln5BZdgvX9swhCsUySUncTHp-12D3KooWH3mBSF2RFQJFZdqzkVBqSdspoAAunHkViTf5AQhubv1c-3
2021-04-06T01:59:43.948-0700 INFO markets loggers/loggers.go:30 retrieval provider event {"name": "ProviderEventBlockSent", "deal ID": "2", "receiver": "12D3KooWAWe8axhW2GM6MmuDSzE1Ln5BZdgvX9swhCsUySUncTHp", "state": "DealStatusOngoing", "message": ""}
2021-04-06T01:59:43.949-0700 WARN dt-impl impl/events.go:235 data transfer channel 12D3KooWAWe8axhW2GM6MmuDSzE1Ln5BZdgvX9swhCsUySUncTHp-12D3KooWH3mBSF2RFQJFZdqzkVBqSdspoAAunHkViTf5AQhubv1c-3 failed to transfer data: graphsync response to peer 12D3KooWAWe8axhW2GM6MmuDSzE1Ln5BZdgvX9swhCsUySUncTHp did not complete: response status code RequestCompletedPartial
2021-04-06T01:59:43.949-0700 INFO markets loggers/loggers.go:30 retrieval provider event {"name": "ProviderEventDataTransferError", "deal ID": "2", "receiver": "12D3KooWAWe8axhW2GM6MmuDSzE1Ln5BZdgvX9swhCsUySUncTHp", "state": "DealStatusErrored", "message": "deal data transfer failed: data transfer channel 12D3KooWAWe8axhW2GM6MmuDSzE1Ln5BZdgvX9swhCsUySUncTHp-12D3KooWH3mBSF2RFQJFZdqzkVBqSdspoAAunHkViTf5AQhubv1c-3 failed to transfer data: graphsync response to peer 12D3KooWAWe8axhW2GM6MmuDSzE1Ln5BZdgvX9swhCsUySUncTHp did not complete: response status code RequestCompletedPartial"}
2021-04-06T02:14:14.277-0700 INFO graphsync impl/graphsync.go:309 Graphsync ReceiveError from 12D3KooWAWe8axhW2GM6MmuDSzE1Ln5BZdgvX9swhCsUySUncTHp: stream reset

** Code modifications **
No code modifications

[pooja]

@dirkmc Could we look into this? We had the folks who reported #5829 try to make a storage deal with the same data but with online data transfer (5829 was with offline data transfer) and then attempt a retrieval deal. This is the bug that's been reported since the retrievals aren't working.

Other details include:

• The retrieval deal was stuck in ClientEventPaymentChannelCreateInitiated (DealStatusPaymentChannelCreating) all weekend before the lotus daemon crashed (possibly due to being out of space)
• It was suggested that the "invalid transition in queue" state (in the screenshot below) is a bug
• @ribasushi do you have anything else to add?

[pooja]

Also @ethanwu155 in case you want to drop any other details

[ethanwu155]

@pooja nothing else to add. you guys summed it up well. thanks!

[jamerduhgamer]

Both parties will try updating to v1.6.0 and then test the retrieval deal process again

[jennijuju]

related #5829

[bq1024]

Just tried to retrieve an offline deal with v1.6.0, but got the same error:

> Recv: 0 B, Paid 0 FIL, ClientEventOpen (DealStatusNew)
> Recv: 0 B, Paid 0 FIL, ClientEventDealProposed (DealStatusWaitForAcceptance)
> Recv: 0 B, Paid 0 FIL, ClientEventDealAccepted (DealStatusAccepted)
> Recv: 0 B, Paid 0 FIL, ClientEventPaymentChannelSkip (DealStatusOngoing)
> Recv: 0 B, Paid 0 FIL, ClientEventDataTransferError (DealStatusErrored)
> Recv: 0 B, Paid 0 FIL, ClientEventOpen (DealStatusNew)
ERROR: retrieval failed: Retrieve: Retrieval Error: error generated by data transfer: deal data transfer failed: incomplete response

[github-actions]

Oops, seems like we needed more information for this issue, please comment with more details or this issue will be closed in 24 hours.

[github-actions]

This issue was closed because it is missing author input.