Collector offline #163
Collector offline #163
Conversation
There was a problem hiding this comment.
I think you need to rework the package structure and add the glue code that calls these new classes so we can understand how the pieces fit together as no 100% clear at the moment
This is definitely not a collector client as we've described it so should be separated very clearly
pkg/telemetry/tag/tags.go
Outdated
| CacheKeyTag = "cache_key" | ||
| EdgeTypeTag = "edge_type" | ||
| CollectorTag = "collector" | ||
| CollectorS3Bucket = "s3_bucket" |
There was a problem hiding this comment.
these should have the corrersponding function (like the other tags)
There was a problem hiding this comment.
I am not using it, those are for span tags not metrics. Do you still want the function to be created ?
pkg/telemetry/span/spans.go
Outdated
| CollectorReadFile = "kubehound.collector.readFile" | ||
| CollectorS3Push = "kubehound.collector.s3_push" | ||
| CollectorS3Download = "kubehound.collector.s3_download" | ||
| CollectorFileWriterWrite = "kubehound.collector.file_writer.write" |
There was a problem hiding this comment.
maybe CollectorWriterXXX with tags for S3/tar etc would be more extensible
There was a problem hiding this comment.
I did not go in this direction, because I wanted to capture the network latency with the s3 push and download which does not exist with the file writer (only local).
pkg/kubehound/core/core.go
Outdated
| @@ -203,3 +203,37 @@ func Launch(ctx context.Context, opts ...LaunchOption) error { | |||
|
|
|||
| return nil | |||
| } | |||
|
|
|||
| func Bootstrap(ctx context.Context, cfg *config.KubehoundConfig) (string, context.Context, error) { | |||
There was a problem hiding this comment.
not called FYI ( I know its a draft)
| @@ -127,6 +137,19 @@ func NewConfig(configPath string) (*KubehoundConfig, error) { | |||
| return &kc, nil | |||
| } | |||
|
|
|||
| // NewConfig creates a new config instance from the provided file using viper. | |||
| func NewInlineConfig() (*KubehoundConfig, error) { | |||
There was a problem hiding this comment.
why inline? not sure I understand the naming
There was a problem hiding this comment.
Inline as for commands provided by an inline command and not a config file
pkg/collector/writer/writer.go
Outdated
| Write(context.Context, []byte, string) error | ||
| Flush(context.Context) error | ||
| Close(context.Context) error | ||
| GetOutputPath() string |
There was a problem hiding this comment.
OutputPath() is more go :)
pkg/collector/offline_dump.go
Outdated
| @@ -0,0 +1,247 @@ | |||
| package collector | |||
There was a problem hiding this comment.
I think this needs to go in a different package (and you need to consider naming) as currently quite confusing!
There was a problem hiding this comment.
maybe pkg/offline ? with the file writer pieces
I think this should be a separate binary. If you consider the KHaaS use case we just want a small binary to collect data and upload to s3 so no need to ship all the kubehound stuff like mongo/janus clients etc to do that |
pkg/collector/k8s_api.go
Outdated
|
|
||
| raw, err := kubeConfig.RawConfig() | ||
| // Generate metrics for k8sAPI collector | ||
| func (c *k8sAPICollector) computeMetrics(ctx context.Context) error { |
There was a problem hiding this comment.
this feels like something that should be done in DD based on the CollectorWait metric vs the code here to me
There was a problem hiding this comment.
could not get to have relevant metric from Datadog directly to work
…collector-dump-local-s3
This PR add a function to add an offline collector. It allows to dump locally or directly on a s3 bucket.
./kubehound dump local --compress --output-dir /tmp/kubehound --debug./kubehound dump local --output-dir /tmp/kubehound --debug./kubehound collector s3 --bucket kubehound --region us-east-1