feat(inspect): add influx_inspect verify-tombstone tool #15952

e-dard · 2019-11-17T14:53:12Z

This PR adds a new tool for verifying tombstone files. To use the tool simply point it at an arbitrary path; the default is ~/.influxdb.

Use the tool like this:

$ influx_inspect verify-tombstone

The tool will then find all of the .tombstone files recursively under that path and verify each one. Currently it verifies files serially; it would be trivial to parallelise this work in future. The path provided can be arbitrary—it doesn't need to be an InfluxDB data directory for example.

The tool iterates through each tombstone entry in each file. You can set verbosity levels to decide how much information you want:

no verbosity set: tool tells you when it has completed verifying a tombstone file;
-v (verbose): tool tells you when it's verifying a file and will tell you it's progress every 5 million tombstone entries;
-vv (very verbose): tool emits every series key and time range in the tombstone files. Times are nano seconds since the epoch;
-vvv (very very verbose): tool emits every series key and time range the tombstone files. Times are RFC3339 Nano formatted strings.

rawkode · 2019-11-17T17:15:15Z

cmd/influx_inspect/main.go

+	case "verify-tombstone":
+		name := tombstone.NewCommand()
+		if err := name.Run(args...); err != nil {
+			return fmt.Errorf("verify-seriesfile: %s", err)


Should say verify-tombstone

rawkode · 2019-11-17T17:17:26Z

cmd/influx_inspect/verify/tombstone/verify.go

+		return err
+	}
+
+	if *v {


Could the complexity be simplified by checking v first, then vv, and finally vvv; using the most verbose option?

Would allow removal of complicated if statements

Or let higher verbosity levels override lower levels.

That's what I said, but in a terrible fashion :)

dgnorton

minor nits but LGTM

dgnorton · 2019-11-18T17:31:12Z

cmd/influx_inspect/verify/tombstone/verify.go

@@ -0,0 +1,145 @@
+// Package tsm verifies integrity of TSM files.


nit: fix comment

dgnorton · 2019-11-18T17:51:25Z

cmd/influx_inspect/verify/tombstone/verify.go

+		return err
+	}
+
+	if *v {


Or let higher verbosity levels override lower levels.

dgnorton · 2019-11-18T18:00:03Z

cmd/influx_inspect/verify/tombstone/verify.go

+		var totalEntries int64
+		err := tombstoner.Walk(func(t tsm1.Tombstone) error {
+			totalEntries++
+			if v.verbosity > quiet && totalEntries%(10*1e6) == 0 {


10e6 or 1e7?

e-dard · 2020-03-12T13:58:02Z

@dgnorton thanks for getting to this!

v1.8.4 [2021-01-27] ------------------- ### Bugfixes - [#19696](influxdata/influxdb#19697): fix(flux): add durations to Flux logging v1.8.3 [2020-09-30] ------------------- ### Features - [#19187](influxdata/influxdb#19187): feat: Collect values written stats. - [#19611](influxdata/influxdb#19611): feat: Add -lponly flag to export sub-command. ### Bugfixes - [#19409](influxdata/influxdb#19409): chore: update uuid library from satori to gofrs. - [#19439](influxdata/influxdb#19439): fix(storage): ArrayFilterCursor truncation for multi-block data. - [#19460](influxdata/influxdb#19460): chore: Use latest version of influxql package. - [#19512](influxdata/influxdb#19512): chore: Quiet static analysis tools. - [#19592](influxdata/influxdb#19592): fix(services/storage): multi measurement queries return all applicable series. - [#19612](influxdata/influxdb#19612): fix: lock map before writes. v1.8.2 [2020-08-13] ------------------- ### Bugfixes - [#19253](influxdata/influxdb#19253): fix(tsdb): Revert disable series id set cache size by default. v1.8.1 [2020-07-08] ------------------- ### Features - [#18457](influxdata/influxdb#18457): feat(query): Parallelize field iterator planning. - [#18886](influxdata/influxdb#18886): feat(http): Allow user supplied HTTP headers. ### Bugfixes - [#17319](influxdata/influxdb#17319): fix(flux): buckets call no longer panics. - [#18212](influxdata/influxdb#18212): fix(tsdb): Defer closing of underlying SeriesIDSetIterators. - [#18286](influxdata/influxdb#18286): fix(tsdb): Disable series id set cache size by default. - [#18299](influxdata/influxdb#18299): refactor(http): Simplify Authorizer. - [#18694](influxdata/influxdb#18694): fix(tsi1): wait deleting epoch before dropping shard. - [#18687](influxdata/influxdb#18687): perf(tsi1): batch write tombstone entries when dropping/deleting. - [#18826](influxdata/influxdb#18826): fix: gracefully handle errors when creating snapshots. - [#18849](influxdata/influxdb#18849): chore(build): remove all of the go1.12 references from build. v1.8.0 [2020-04-11] ------------------- ### Features - [#15952](influxdata/influxdb#15952): Add influx_inspect verify-tombstone tool. - [#16542](influxdata/influxdb#16542): Add offline series compaction to influx_inspect buildtsi. - [#16599](influxdata/influxdb#16599): Make influx CLI support custom HTTP endpoint. - [#16908](influxdata/influxdb#16908): Add support for InfluxDB 2.0 write API. - [#17621](influxdata/influxdb#17621): Update Flux to v0.65.0. - [#17188](influxdata/influxdb#17188): Enhance support for bound parameters. ### Bugfixes - [#10503](influxdata/influxdb#10503): Delete rebuilds series index when series to be deleted are only found in cache. - [#10504](https://github.com/influxdata/influxdb/issue/10504): Delete rebuilds series index when series to be deleted are outside timerange. - [#14485](influxdata/influxdb#14485): Parse Accept header correctly. - [#16524](influxdata/influxdb#16524): Upgrade compaction error log from `Info` to `Warn`. - [#16525](influxdata/influxdb#16525): Remove double increment of meta index. - [#16595](influxdata/influxdb#16595): Improve series cardinality limit for inmem index. - [#16606](influxdata/influxdb#16606): Ensure all block data returned. - [#16627](influxdata/influxdb#16627): Skip WriteSnapshot during backup if snapshotter is busy. - [#16709](influxdata/influxdb#16709): Reduce influxd and influx startup time if Flux isn't used. - [#16762](influxdata/influxdb#16762): Fix bugs in -compact-series-file. - [#16944](influxdata/influxdb#16944): Update to Go 1.13.8 and Go modules. - [#17032](influxdata/influxdb#17032): Fix a SIGSEGV when accessing tsi active log. - [#17656](influxdata/influxdb#17656): Verify precision in write requests. - [#17698](influxdata/influxdb#17698): Enable configuration of TLS 1.3.

e-dard added 2 commits November 17, 2019 14:43

refactor(storage): add tombstone extension

f7f19c5

feat(inspect): add verify-tombstone

92b9bde

e-dard force-pushed the er-verify-tombstone branch from 68d4acd to 92b9bde Compare November 17, 2019 14:53

e-dard requested a review from dgnorton November 17, 2019 14:55

rawkode reviewed Nov 17, 2019

View reviewed changes

dgnorton approved these changes Nov 19, 2019

View reviewed changes

timhallinflux added the 1.x label Mar 4, 2020

dgnorton added 3 commits March 11, 2020 15:28

fix: typo in error message

b93bac1

fix: use highest verbosity level specified

0fcb240

fix: update package comment

fca2ee0

dgnorton merged commit 25381f9 into 1.8 Mar 11, 2020

sanderson mentioned this pull request Mar 11, 2020

Document 'influx inspect verify-tombstone' command influxdata/docs.influxdata.com-ARCHIVE#2712

Closed

jacobmarble deleted the er-verify-tombstone branch January 2, 2024 22:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(inspect): add influx_inspect verify-tombstone tool #15952

feat(inspect): add influx_inspect verify-tombstone tool #15952

e-dard commented Nov 17, 2019 •

edited

Loading

rawkode Nov 17, 2019

rawkode Nov 17, 2019

dgnorton Nov 18, 2019

rawkode Nov 19, 2019

dgnorton left a comment

dgnorton Nov 18, 2019

dgnorton Nov 18, 2019

dgnorton Nov 18, 2019

e-dard commented Mar 12, 2020

		@@ -0,0 +1,145 @@
		// Package tsm verifies integrity of TSM files.

feat(inspect): add influx_inspect verify-tombstone tool #15952

feat(inspect): add influx_inspect verify-tombstone tool #15952

Conversation

e-dard commented Nov 17, 2019 • edited Loading

rawkode Nov 17, 2019

Choose a reason for hiding this comment

rawkode Nov 17, 2019

Choose a reason for hiding this comment

dgnorton Nov 18, 2019

Choose a reason for hiding this comment

rawkode Nov 19, 2019

Choose a reason for hiding this comment

dgnorton left a comment

Choose a reason for hiding this comment

dgnorton Nov 18, 2019

Choose a reason for hiding this comment

dgnorton Nov 18, 2019

Choose a reason for hiding this comment

dgnorton Nov 18, 2019

Choose a reason for hiding this comment

e-dard commented Mar 12, 2020

e-dard commented Nov 17, 2019 •

edited

Loading