Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sp cli: make sectors list much faster #10202

Merged
merged 3 commits into from
Feb 15, 2023
Merged

sp cli: make sectors list much faster #10202

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
81e056b
sp cli: make sectors list much faster
magik6k Feb 7, 2023
17adcb7
sp cli: correct order in parallel sectors list
magik6k Feb 9, 2023
6028d7b
sectors list: Explain http mode
magik6k Feb 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 35 additions & 5 deletions cmd/lotus-miner/sectors.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import (
"sort"
"strconv"
"strings"
"sync"
"time"

"github.com/docker/go-units"
Expand All @@ -33,11 +34,14 @@ import (
"github.com/filecoin-project/lotus/chain/types"
lcli "github.com/filecoin-project/lotus/cli"
cliutil "github.com/filecoin-project/lotus/cli/util"
"github.com/filecoin-project/lotus/lib/result"
"github.com/filecoin-project/lotus/lib/strle"
"github.com/filecoin-project/lotus/lib/tablewriter"
sealing "github.com/filecoin-project/lotus/storage/pipeline"
)

const parallelSectorChecks = 300

var sectorsCmd = &cli.Command{
Name: "sectors",
Usage: "interact with sector store",
Expand Down Expand Up @@ -306,9 +310,14 @@ var sectorsListCmd = &cli.Command{
Usage: "only show sectors which aren't in the 'Proving' state",
Aliases: []string{"u"},
},
&cli.Int64Flag{
Name: "check-parallelism",
Usage: "number of parallel requests to make for checking sector states",
Value: parallelSectorChecks,
},
},
Action: func(cctx *cli.Context) error {
minerApi, closer, err := lcli.GetStorageMinerAPI(cctx)
minerApi, closer, err := lcli.GetStorageMinerAPI(cctx, cliutil.StorageMinerUseHttp)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was getting bottlenecked by a single-threaded json encoder in websocket mode. In http mode all those requests are running on entirely separate goroutines, and don't get blocked on json encoding/decoding

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, thanks. Say that in a comment, please, then merge?

if err != nil {
return err
}
Expand Down Expand Up @@ -407,16 +416,37 @@ var sectorsListCmd = &cli.Command{

fast := cctx.Bool("fast")

for _, s := range list {
st, err := minerApi.SectorsStatus(ctx, s, !fast)
if err != nil {
throttle := make(chan struct{}, cctx.Int64("check-parallelism"))

slist := make([]result.Result[api.SectorInfo], len(list))
var wg sync.WaitGroup
for i, s := range list {
throttle <- struct{}{}
wg.Add(1)
go func(i int, s abi.SectorNumber) {
defer wg.Done()
defer func() { <-throttle }()
r := result.Wrap(minerApi.SectorsStatus(ctx, s, !fast))
if r.Error != nil {
r.Value.SectorID = s
}
slist[i] = r
}(i, s)
}
wg.Wait()

for _, rsn := range slist {
if rsn.Error != nil {
tw.Write(map[string]interface{}{
"ID": s,
"ID": rsn.Value.SectorID,
"Error": err,
})
continue
}

st := rsn.Value
s := st.SectorID

if !showRemoved && st.State == api.SectorState(sealing.Removed) {
continue
}
Expand Down
15 changes: 8 additions & 7 deletions documentation/en/cli-lotus-miner.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1717,13 +1717,14 @@ USAGE:
lotus-miner sectors list [command options] [arguments...]

OPTIONS:
--events, -e display number of events the sector has received (default: false)
--fast, -f don't show on-chain info for better performance (default: false)
--initial-pledge, -p display initial pledge (default: false)
--seal-time, -t display how long it took for the sector to be sealed (default: false)
--show-removed, -r show removed sectors (default: false)
--states value filter sectors by a comma-separated list of states
--unproven, -u only show sectors which aren't in the 'Proving' state (default: false)
--check-parallelism value number of parallel requests to make for checking sector states (default: 300)
--events, -e display number of events the sector has received (default: false)
--fast, -f don't show on-chain info for better performance (default: false)
--initial-pledge, -p display initial pledge (default: false)
--seal-time, -t display how long it took for the sector to be sealed (default: false)
--show-removed, -r show removed sectors (default: false)
--states value filter sectors by a comma-separated list of states
--unproven, -u only show sectors which aren't in the 'Proving' state (default: false)

```

Expand Down