Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

telemetry: Adding telemetry for alter table reorganize partition #40258

Merged
merged 4 commits into from
Jan 18, 2023
Merged

telemetry: Adding telemetry for alter table reorganize partition #40258

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
bf341b3
Adding telemetry for alter table reorganize partition
mjonss Dec 30, 2022
6b93d42
Merge branch 'feature/reorganize-partition' into reorg-part-telemetry
mjonss Jan 3, 2023
c7b23fc
Merge branch 'feature/reorganize-partition' into reorg-part-telemetry
mjonss Jan 18, 2023
84bca8b
Enabled tests.
mjonss Jan 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion executor/adapter.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -197,7 +197,7 @@ type TelemetryInfo struct {
UseNonRecursive bool
UseRecursive bool
UseMultiSchemaChange bool
UesExchangePartition bool
UseExchangePartition bool
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch!

UseFlashbackToCluster bool
PartitionTelemetry *PartitionTelemetryInfo
AccountLockTelemetry *AccountLockTelemetryInfo
Expand All @@ -220,6 +220,7 @@ type PartitionTelemetryInfo struct {
UseAddIntervalPartition bool
UseDropIntervalPartition bool
UseCompactTablePartition bool
UseReorganizePartition bool
}

// AccountLockTelemetryInfo records account lock/unlock information during execution
Expand Down
7 changes: 6 additions & 1 deletion executor/builder.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1087,7 +1087,12 @@ func (b *executorBuilder) setTelemetryInfo(v *plannercore.DDL) {
}
b.Ti.PartitionTelemetry.UseAddIntervalPartition = true
case ast.AlterTableExchangePartition:
b.Ti.UesExchangePartition = true
b.Ti.UseExchangePartition = true
case ast.AlterTableReorganizePartition:
if b.Ti.PartitionTelemetry == nil {
b.Ti.PartitionTelemetry = &PartitionTelemetryInfo{}
}
b.Ti.PartitionTelemetry.UseReorganizePartition = true
}
}
case *ast.CreateTableStmt:
Expand Down
31 changes: 21 additions & 10 deletions metrics/telemetry.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,13 @@ var (
Name: "compact_partition_usage",
Help: "Counter of compact table partition",
})
TelemetryReorganizePartitionCnt = prometheus.NewCounter(
prometheus.CounterOpts{
Namespace: "tidb",
Subsystem: "telemetry",
Name: "reorganize_partition_usage",
Help: "Counter of alter table reorganize partition",
})
)

// readCounter reads the value of a prometheus.Counter.
Expand Down Expand Up @@ -269,6 +276,7 @@ type TablePartitionUsageCounter struct {
TablePartitionAddIntervalPartitionsCnt int64 `json:"table_partition_add_interval_partitions_cnt"`
TablePartitionDropIntervalPartitionsCnt int64 `json:"table_partition_drop_interval_partitions_cnt"`
TablePartitionComactCnt int64 `json:"table_TablePartitionComactCnt"`
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Not really part of this PR) Can this be changed to TablePartitionCompactCnt / table_TablePartitionComactCnt (adding a 'p' Comact -> Compact)? Since the json is already defined.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess It is possible. But you need to contact the telemetry(observe) team to change the column name, and change how their HTTP api handles telemetry requests.

TablePartitionReorganizePartitionCnt int64 `json:"table_reorganize_partition_cnt"`
}

// ExchangePartitionUsageCounter records the usages of exchange partition.
Expand Down Expand Up @@ -307,22 +315,24 @@ func (c TablePartitionUsageCounter) Cal(rhs TablePartitionUsageCounter) TablePar
TablePartitionAddIntervalPartitionsCnt: c.TablePartitionAddIntervalPartitionsCnt - rhs.TablePartitionAddIntervalPartitionsCnt,
TablePartitionDropIntervalPartitionsCnt: c.TablePartitionDropIntervalPartitionsCnt - rhs.TablePartitionDropIntervalPartitionsCnt,
TablePartitionComactCnt: c.TablePartitionComactCnt - rhs.TablePartitionComactCnt,
TablePartitionReorganizePartitionCnt: c.TablePartitionReorganizePartitionCnt - rhs.TablePartitionReorganizePartitionCnt,
}
}

// ResetTablePartitionCounter gets the TxnCommitCounter.
func ResetTablePartitionCounter(pre TablePartitionUsageCounter) TablePartitionUsageCounter {
return TablePartitionUsageCounter{
TablePartitionCnt: readCounter(TelemetryTablePartitionCnt),
TablePartitionListCnt: readCounter(TelemetryTablePartitionListCnt),
TablePartitionRangeCnt: readCounter(TelemetryTablePartitionRangeCnt),
TablePartitionHashCnt: readCounter(TelemetryTablePartitionHashCnt),
TablePartitionRangeColumnsCnt: readCounter(TelemetryTablePartitionRangeColumnsCnt),
TablePartitionRangeColumnsGt1Cnt: readCounter(TelemetryTablePartitionRangeColumnsGt1Cnt),
TablePartitionRangeColumnsGt2Cnt: readCounter(TelemetryTablePartitionRangeColumnsGt2Cnt),
TablePartitionRangeColumnsGt3Cnt: readCounter(TelemetryTablePartitionRangeColumnsGt3Cnt),
TablePartitionListColumnsCnt: readCounter(TelemetryTablePartitionListColumnsCnt),
TablePartitionMaxPartitionsCnt: mathutil.Max(readCounter(TelemetryTablePartitionMaxPartitionsCnt)-pre.TablePartitionMaxPartitionsCnt, pre.TablePartitionMaxPartitionsCnt),
TablePartitionCnt: readCounter(TelemetryTablePartitionCnt),
TablePartitionListCnt: readCounter(TelemetryTablePartitionListCnt),
TablePartitionRangeCnt: readCounter(TelemetryTablePartitionRangeCnt),
TablePartitionHashCnt: readCounter(TelemetryTablePartitionHashCnt),
TablePartitionRangeColumnsCnt: readCounter(TelemetryTablePartitionRangeColumnsCnt),
TablePartitionRangeColumnsGt1Cnt: readCounter(TelemetryTablePartitionRangeColumnsGt1Cnt),
TablePartitionRangeColumnsGt2Cnt: readCounter(TelemetryTablePartitionRangeColumnsGt2Cnt),
TablePartitionRangeColumnsGt3Cnt: readCounter(TelemetryTablePartitionRangeColumnsGt3Cnt),
TablePartitionListColumnsCnt: readCounter(TelemetryTablePartitionListColumnsCnt),
TablePartitionMaxPartitionsCnt: mathutil.Max(readCounter(TelemetryTablePartitionMaxPartitionsCnt)-pre.TablePartitionMaxPartitionsCnt, pre.TablePartitionMaxPartitionsCnt),
TablePartitionReorganizePartitionCnt: readCounter(TelemetryReorganizePartitionCnt),
}
}

Expand All @@ -343,6 +353,7 @@ func GetTablePartitionCounter() TablePartitionUsageCounter {
TablePartitionAddIntervalPartitionsCnt: readCounter(TelemetryTablePartitionAddIntervalPartitionsCnt),
TablePartitionDropIntervalPartitionsCnt: readCounter(TelemetryTablePartitionDropIntervalPartitionsCnt),
TablePartitionComactCnt: readCounter(TelemetryCompactPartitionCnt),
TablePartitionReorganizePartitionCnt: readCounter(TelemetryReorganizePartitionCnt),
}
}

Expand Down
6 changes: 5 additions & 1 deletion session/session.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,7 @@ var (
telemetryTablePartitionDropIntervalUsage = metrics.TelemetryTablePartitionDropIntervalPartitionsCnt
telemetryExchangePartitionUsage = metrics.TelemetryExchangePartitionCnt
telemetryTableCompactPartitionUsage = metrics.TelemetryCompactPartitionCnt
telemetryReorganizePartitionUsage = metrics.TelemetryReorganizePartitionCnt

telemetryLockUserUsage = metrics.TelemetryAccountLockCnt.WithLabelValues("lockUser")
telemetryUnlockUserUsage = metrics.TelemetryAccountLockCnt.WithLabelValues("unlockUser")
Expand Down Expand Up @@ -4006,7 +4007,7 @@ func (s *session) updateTelemetryMetric(es *executor.ExecStmt) {
telemetryFlashbackClusterUsage.Inc()
}

if ti.UesExchangePartition {
if ti.UseExchangePartition {
telemetryExchangePartitionUsage.Inc()
}

Expand Down Expand Up @@ -4051,6 +4052,9 @@ func (s *session) updateTelemetryMetric(es *executor.ExecStmt) {
if ti.PartitionTelemetry.UseCompactTablePartition {
telemetryTableCompactPartitionUsage.Inc()
}
if ti.PartitionTelemetry.UseReorganizePartition {
telemetryReorganizePartitionUsage.Inc()
}
}

if ti.AccountLockTelemetry != nil {
Expand Down
3 changes: 3 additions & 0 deletions telemetry/data_feature_usage_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -240,6 +240,7 @@ func TestTablePartition(t *testing.T) {
require.Equal(t, int64(0), usage.TablePartition.TablePartitionCreateIntervalPartitionsCnt)
require.Equal(t, int64(0), usage.TablePartition.TablePartitionAddIntervalPartitionsCnt)
require.Equal(t, int64(0), usage.TablePartition.TablePartitionDropIntervalPartitionsCnt)
require.Equal(t, int64(0), usage.TablePartition.TablePartitionReorganizePartitionCnt)

telemetry.PostReportTelemetryDataForTest()
tk.MustExec("drop table if exists pt1")
Expand All @@ -251,6 +252,7 @@ func TestTablePartition(t *testing.T) {
"partition p4 values less than (15))")
tk.MustExec("alter table pt1 first partition less than (9)")
tk.MustExec("alter table pt1 last partition less than (21)")
tk.MustExec("alter table pt1 reorganize partition p4 into (partition p4 values less than (13), partition p5 values less than (15))")
tk.MustExec("drop table if exists pt1")
tk.MustExec("create table pt1 (d datetime primary key, v varchar(255)) partition by range columns(d)" +
" interval (1 day) first partition less than ('2022-01-01') last partition less than ('2022-02-22')")
Expand All @@ -275,6 +277,7 @@ func TestTablePartition(t *testing.T) {
require.Equal(t, int64(1), usage.TablePartition.TablePartitionCreateIntervalPartitionsCnt)
require.Equal(t, int64(1), usage.TablePartition.TablePartitionAddIntervalPartitionsCnt)
require.Equal(t, int64(1), usage.TablePartition.TablePartitionDropIntervalPartitionsCnt)
require.Equal(t, int64(1), usage.TablePartition.TablePartitionReorganizePartitionCnt)

tk.MustExec("drop table if exists pt2")
tk.MustExec("create table pt2 (a int,b int) partition by range(a) (" +
Expand Down