Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

util: Remove deprecated grpc metrics code #4225

Merged
merged 2 commits into from
Nov 3, 2023
Merged

util: Remove deprecated grpc metrics code #4225

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ae4050f
util: remove deprecated grpc metrics
Madhu-1 Nov 2, 2023
1a4d4c5
util: rearrange struct to fix lint
Madhu-1 Nov 2, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 3 additions & 11 deletions cmd/cephcsi.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,8 +106,8 @@ func init() {
"",
"Comma separated string of mount options accepted by ceph-fuse mounter")

// liveness/grpc metrics related flags
flag.IntVar(&conf.MetricsPort, "metricsport", 8080, "TCP port for liveness/grpc metrics requests")
// liveness/profile metrics related flags
flag.IntVar(&conf.MetricsPort, "metricsport", 8080, "TCP port for liveness/profile metrics requests")
flag.StringVar(
&conf.MetricsPath,
"metricspath",
Expand All @@ -116,14 +116,6 @@ func init() {
flag.DurationVar(&conf.PollTime, "polltime", time.Second*pollTime, "time interval in seconds between each poll")
flag.DurationVar(&conf.PoolTimeout, "timeout", time.Second*probeTimeout, "probe timeout in seconds")

flag.BoolVar(&conf.EnableGRPCMetrics, "enablegrpcmetrics", false, "[DEPRECATED] enable grpc metrics")
flag.StringVar(
&conf.HistogramOption,
"histogramoption",
"0.5,2,6",
"[DEPRECATED] Histogram option for grpc metrics, should be comma separated value, "+
"ex:= 0.5,2,6 where start=0.5 factor=2, count=6")

flag.UintVar(
&conf.RbdHardMaxCloneDepth,
"rbdhardmaxclonedepth",
Expand Down Expand Up @@ -210,7 +202,7 @@ func main() {

setPIDLimit(&conf)

if conf.EnableGRPCMetrics || conf.Vtype == livenessType {
if conf.EnableProfiling || conf.Vtype == livenessType {
// validate metrics endpoint
conf.MetricsIP = os.Getenv("POD_IP")

Expand Down
2 changes: 0 additions & 2 deletions docs/deploy-cephfs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,9 @@ make image-cephcsi
| `--pidlimit` | _0_ | Configure the PID limit in cgroups. The container runtime can restrict the number of processes/tasks which can cause problems while provisioning (or deleting) a large number of volumes. A value of `-1` configures the limit to the maximum, `0` does not configure limits at all. |
| `--metricsport` | `8080` | TCP port for liveness metrics requests |
| `--metricspath` | `/metrics` | Path of prometheus endpoint where metrics will be available |
| `--enablegrpcmetrics` | `false` | [Deprecated] Enable grpc metrics collection and start prometheus server |
| `--polltime` | `60s` | Time interval in between each poll |
| `--timeout` | `3s` | Probe timeout in seconds |
| `--clustername` | _empty_ | Cluster name to set on subvolume |
| `--histogramoption` | `0.5,2,6` | [Deprecated] Histogram option for grpc metrics, should be comma separated value (ex:= "0.5,2,6" where start=0.5 factor=2, count=6) |
| `--forcecephkernelclient` | `false` | Force enabling Ceph Kernel clients for mounting on kernels < 4.17 |
| `--kernelmountoptions` | _empty_ | Comma separated string of mount options accepted by cephfs kernel mounter |
| `--fusemountoptions` | _empty_ | Comma separated string of mount options accepted by ceph-fuse mounter |
Expand Down
2 changes: 0 additions & 2 deletions docs/deploy-rbd.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,11 +37,9 @@ make image-cephcsi
| `--pidlimit` | _0_ | Configure the PID limit in cgroups. The container runtime can restrict the number of processes/tasks which can cause problems while provisioning (or deleting) a large number of volumes. A value of `-1` configures the limit to the maximum, `0` does not configure limits at all. |
| `--metricsport` | `8080` | TCP port for liveness metrics requests |
| `--metricspath` | `"/metrics"` | Path of prometheus endpoint where metrics will be available |
| `--enablegrpcmetrics` | `false` | [Deprecated] Enable grpc metrics collection and start prometheus server |
| `--polltime` | `"60s"` | Time interval in between each poll |
| `--timeout` | `"3s"` | Probe timeout in seconds |
| `--clustername` | _empty_ | Cluster name to set on RBD image |
| `--histogramoption` | `0.5,2,6` | [Deprecated] Histogram option for grpc metrics, should be comma separated value (ex:= "0.5,2,6" where start=0.5 factor=2, count=6) |
| `--domainlabels` | _empty_ | Kubernetes node labels to use as CSI domain labels for topology aware provisioning, should be a comma separated value (ex:= "failure-domain/region,failure-domain/zone") |
| `--rbdhardmaxclonedepth` | `8` | Hard limit for maximum number of nested volume clones that are taken before a flatten occurs |
| `--rbdsoftmaxclonedepth` | `4` | Soft limit for maximum number of nested volume clones that are taken before a flatten occurs |
Expand Down
11 changes: 3 additions & 8 deletions internal/cephfs/driver.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,15 +167,10 @@ func (fs *Driver) Run(conf *util.Config) {
// passing nil for replication server as cephFS does not support mirroring.
RS: nil,
}
server.Start(conf.Endpoint, conf.HistogramOption, srv, conf.EnableGRPCMetrics)
if conf.EnableGRPCMetrics {
log.WarningLogMsg("EnableGRPCMetrics is deprecated")
go util.StartMetricsServer(conf)
}
server.Start(conf.Endpoint, srv)

if conf.EnableProfiling {
if !conf.EnableGRPCMetrics {
go util.StartMetricsServer(conf)
}
go util.StartMetricsServer(conf)
log.DebugLogMsg("Registering profiling handler")
go util.EnableProfiling()
}
Expand Down
41 changes: 5 additions & 36 deletions internal/csi-common/server.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,24 +19,20 @@ package csicommon
import (
"net"
"os"
"strconv"
"strings"
"sync"

"github.com/ceph/ceph-csi/internal/util/log"

"github.com/container-storage-interface/spec/lib/go/csi"
"github.com/csi-addons/spec/lib/go/replication"
grpc_prometheus "github.com/grpc-ecosystem/go-grpc-prometheus"
"github.com/prometheus/client_golang/prometheus"
"google.golang.org/grpc"
"k8s.io/klog/v2"
)

// NonBlockingGRPCServer defines Non blocking GRPC server interfaces.
type NonBlockingGRPCServer interface {
// Start services at the endpoint
Start(endpoint, hstOptions string, srv Servers, metrics bool)
Start(endpoint string, srv Servers)
// Waits for the service to stop
Wait()
// Stops the service gracefully
Expand Down Expand Up @@ -65,9 +61,9 @@ type nonBlockingGRPCServer struct {
}

// Start start service on endpoint.
func (s *nonBlockingGRPCServer) Start(endpoint, hstOptions string, srv Servers, metrics bool) {
func (s *nonBlockingGRPCServer) Start(endpoint string, srv Servers) {
s.wg.Add(1)
go s.serve(endpoint, hstOptions, srv, metrics)
go s.serve(endpoint, srv)
}

// Wait blocks until the WaitGroup counter.
Expand All @@ -85,7 +81,7 @@ func (s *nonBlockingGRPCServer) ForceStop() {
s.server.Stop()
}

func (s *nonBlockingGRPCServer) serve(endpoint, hstOptions string, srv Servers, metrics bool) {
func (s *nonBlockingGRPCServer) serve(endpoint string, srv Servers) {
proto, addr, err := parseEndpoint(endpoint)
if err != nil {
klog.Fatal(err.Error())
Expand All @@ -103,11 +99,7 @@ func (s *nonBlockingGRPCServer) serve(endpoint, hstOptions string, srv Servers,
klog.Fatalf("Failed to listen: %v", err)
}

opts := []grpc.ServerOption{
NewMiddlewareServerOption(metrics),
}

server := grpc.NewServer(opts...)
server := grpc.NewServer()
s.server = server

if srv.IS != nil {
Expand All @@ -124,29 +116,6 @@ func (s *nonBlockingGRPCServer) serve(endpoint, hstOptions string, srv Servers,
}

log.DefaultLog("Listening for connections on address: %#v", listener.Addr())
if metrics {
ho := strings.Split(hstOptions, ",")
const expectedHo = 3
if len(ho) != expectedHo {
klog.Fatalf("invalid histogram options provided: %v", hstOptions)
}
start, e := strconv.ParseFloat(ho[0], 32)
if e != nil {
klog.Fatalf("failed to parse histogram start value: %v", e)
}
factor, e := strconv.ParseFloat(ho[1], 32)
if err != nil {
klog.Fatalf("failed to parse histogram factor value: %v", e)
}
count, e := strconv.Atoi(ho[2])
if err != nil {
klog.Fatalf("failed to parse histogram count value: %v", e)
}
buckets := prometheus.ExponentialBuckets(start, factor, count)
bktOptions := grpc_prometheus.WithHistogramBuckets(buckets)
grpc_prometheus.EnableHandlingTimeHistogram(bktOptions)
grpc_prometheus.Register(server)
}
err = server.Serve(listener)
if err != nil {
klog.Fatalf("Failed to server: %v", err)
Expand Down
10 changes: 2 additions & 8 deletions internal/nfs/driver/driver.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,15 +77,9 @@ func (fs *Driver) Run(conf *util.Config) {
srv.CS = controller.NewControllerServer(cd)
}

server.Start(conf.Endpoint, conf.HistogramOption, srv, conf.EnableGRPCMetrics)
if conf.EnableGRPCMetrics {
log.WarningLogMsg("EnableGRPCMetrics is deprecated")
go util.StartMetricsServer(conf)
}
server.Start(conf.Endpoint, srv)
if conf.EnableProfiling {
if !conf.EnableGRPCMetrics {
go util.StartMetricsServer(conf)
}
go util.StartMetricsServer(conf)
log.DebugLogMsg("Registering profiling handler")
go util.EnableProfiling()
}
Expand Down
10 changes: 2 additions & 8 deletions internal/rbd/driver/driver.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,11 +177,7 @@ func (r *Driver) Run(conf *util.Config) {
CS: r.cs,
NS: r.ns,
}
s.Start(conf.Endpoint, conf.HistogramOption, srv, conf.EnableGRPCMetrics)
if conf.EnableGRPCMetrics {
log.WarningLogMsg("EnableGRPCMetrics is deprecated")
go util.StartMetricsServer(conf)
}
s.Start(conf.Endpoint, srv)

r.startProfiling(conf)

Expand Down Expand Up @@ -241,9 +237,7 @@ func (r *Driver) setupCSIAddonsServer(conf *util.Config) error {
// starts the required profiling services.
func (r *Driver) startProfiling(conf *util.Config) {
if conf.EnableProfiling {
if !conf.EnableGRPCMetrics {
go util.StartMetricsServer(conf)
}
go util.StartMetricsServer(conf)
log.DebugLogMsg("Registering profiling handler")
go util.EnableProfiling()
}
Expand Down
54 changes: 25 additions & 29 deletions internal/util/util.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,37 +93,20 @@ type Config struct {
PluginPath string // location of cephcsi plugin
StagingPath string // location of cephcsi staging path
DomainLabels string // list of domain labels to read from the node

// metrics related flags
MetricsPath string // path of prometheus endpoint where metrics will be available
HistogramOption string // Histogram option for grpc metrics, should be comma separated value,
// ex:= "0.5,2,6" where start=0.5 factor=2, count=6
MetricsIP string // TCP port for liveness/ metrics requests
MetricsPath string // path of prometheus endpoint where metrics will be available
MetricsIP string // TCP port for liveness/ metrics requests

// CSI-Addons endpoint
CSIAddonsEndpoint string

// Cluster name
ClusterName string

// mount option related flags
KernelMountOptions string // Comma separated string of mount options accepted by cephfs kernel mounter
FuseMountOptions string // Comma separated string of mount options accepted by ceph-fuse mounter

PidLimit int // PID limit to configure through cgroups")
MetricsPort int // TCP port for liveness/grpc metrics requests
PollTime time.Duration // time interval in seconds between each poll
PoolTimeout time.Duration // probe timeout in seconds
EnableGRPCMetrics bool // option to enable grpc metrics

EnableProfiling bool // flag to enable profiling
IsControllerServer bool // if set to true start provisioner server
IsNodeServer bool // if set to true start node server
Version bool // cephcsi version

// SkipForceFlatten is set to false if the kernel supports mounting of
// rbd image or the image chain has the deep-flatten feature.
SkipForceFlatten bool

// cephfs related flags
ForceKernelCephFS bool // force to use the ceph kernel client even if the kernel is < 4.17

SetMetadata bool // set metadata on the volume

// RbdHardMaxCloneDepth is the hard limit for maximum number of nested volume clones that are taken before a flatten
// occurs
RbdHardMaxCloneDepth uint
Expand All @@ -142,11 +125,24 @@ type Config struct {
// reached cephcsi will start flattening the older rbd images.
MinSnapshotsOnImage uint

// CSI-Addons endpoint
CSIAddonsEndpoint string
PidLimit int // PID limit to configure through cgroups")
MetricsPort int // TCP port for liveness/grpc metrics requests
PollTime time.Duration // time interval in seconds between each poll
PoolTimeout time.Duration // probe timeout in seconds

// Cluster name
ClusterName string
EnableProfiling bool // flag to enable profiling
IsControllerServer bool // if set to true start provisioner server
IsNodeServer bool // if set to true start node server
Version bool // cephcsi version

// SkipForceFlatten is set to false if the kernel supports mounting of
// rbd image or the image chain has the deep-flatten feature.
SkipForceFlatten bool

// cephfs related flags
ForceKernelCephFS bool // force to use the ceph kernel client even if the kernel is < 4.17

SetMetadata bool // set metadata on the volume

// Read affinity related options
EnableReadAffinity bool // enable OSD read affinity.
Expand Down