Skip to content

Commit

Permalink
perf(storage): use a lightweight method to get the data size of the s…
Browse files Browse the repository at this point in the history
…torage

The previous method to get the data size of storage belonging to a log stream was costly. Concretely
it used too many heap objects to call `filepath.Walk` API. To avoid this problem, this patch changes
the implementation of `internal/storage.(*Storage).DiskUsage` to use the built-in method of pebble
DB - `DiskSpaceUsage`.

Resolves #210
  • Loading branch information
ijsong committed Nov 2, 2022
1 parent 2651a9b commit 0965fd9
Show file tree
Hide file tree
Showing 3 changed files with 55 additions and 58 deletions.
5 changes: 2 additions & 3 deletions internal/storage/storage.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ import (
"go.uber.org/multierr"

"github.com/kakao/varlog/pkg/types"
"github.com/kakao/varlog/pkg/util/fputil"
"github.com/kakao/varlog/proto/varlogpb"
)

Expand Down Expand Up @@ -285,8 +284,8 @@ func (s *Storage) Path() string {
return s.path
}

func (s *Storage) DiskUsage() int64 {
return fputil.DirectorySize(s.path)
func (s *Storage) DiskUsage() uint64 {
return s.db.Metrics().DiskSpaceUsage()
}

// Close closes the storage.
Expand Down
100 changes: 50 additions & 50 deletions proto/snpb/metadata.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 3 additions & 5 deletions proto/snpb/metadata.proto
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,7 @@ message StorageNodeMetadataDescriptor {

// ClusterID is the identifier of the cluster.
uint32 cluster_id = 1 [
(gogoproto.casttype) =
"github.com/kakao/varlog/pkg/types.ClusterID",
(gogoproto.casttype) = "github.com/kakao/varlog/pkg/types.ClusterID",
(gogoproto.customname) = "ClusterID",
(gogoproto.jsontag) = "clusterId"
];
Expand Down Expand Up @@ -74,8 +73,7 @@ message LogStreamReplicaMetadataDescriptor {
// Version is the latest version of the commit received from the metadata
// repository.
uint64 version = 3
[(gogoproto.casttype) =
"github.com/kakao/varlog/pkg/types.Version"];
[(gogoproto.casttype) = "github.com/kakao/varlog/pkg/types.Version"];

// GlobalHighWatermark is the latest high watermark received from the metadata
// repository.
Expand All @@ -100,7 +98,7 @@ message LogStreamReplicaMetadataDescriptor {

// Path is the directory where the data for the log stream replica is stored.
string path = 7;
int64 storage_size_bytes = 8;
uint64 storage_size_bytes = 8;

// CreatedTime
//
Expand Down

0 comments on commit 0965fd9

Please sign in to comment.