Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Telegraf plugin for mongodb try to gather_col_stats on collection type VIEW #11270

Closed
mishutka-mmx opened this issue Jun 8, 2022 · 6 comments · Fixed by #11280
Closed

Telegraf plugin for mongodb try to gather_col_stats on collection type VIEW #11270

mishutka-mmx opened this issue Jun 8, 2022 · 6 comments · Fixed by #11280
Labels
area/mongodb bug unexpected problem or unintended behavior help wanted Request for community participation, code, contribution

Comments

@mishutka-mmx
Copy link

Relevant telegraf.conf

[[outputs.file]]
   ## Files to write to, "stdout" is a specially handled file.
   files = ["stdout", "/root/metrics.out"]

     data_format = "influx"
[[inputs.mongodb]]

servers = ["mongodb://telegraf:manager@localhost:27017/admin"]
gather_cluster_status = true
gather_perdb_stats = true
gather_col_stats = true
gather_top_stat = true
col_stats_dbs = []
interval = "30s"

Logs from Telegraf

[root@gmv01ndbmgr01 ~]# /usr/bin/telegraf --config ./my.conf --debug  
2022-06-08T09:13:19Z I! Starting Telegraf 1.22.3
2022-06-08T09:13:19Z I! Loaded inputs: mongodb
2022-06-08T09:13:19Z I! Loaded aggregators: 
2022-06-08T09:13:19Z I! Loaded processors: 
2022-06-08T09:13:19Z I! Loaded outputs: file
2022-06-08T09:13:19Z I! Tags enabled: LifeCycle=Text host=gmv01ndbmgr01 ims_system_id=666
2022-06-08T09:13:19Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"gmv01ndbmgr01", Flush Interval:10s
2022-06-08T09:13:19Z D! [agent] Initializing plugins
2022-06-08T09:13:19Z D! [agent] Connecting outputs
2022-06-08T09:13:19Z D! [agent] Attempting connection to [outputs.file]
2022-06-08T09:13:19Z D! [agent] Successfully connected to outputs.file
2022-06-08T09:13:19Z D! [agent] Starting service inputs
2022-06-08T09:13:29Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2022-06-08T09:13:30Z E! [inputs.mongodb] error getting col stats from "system.tenantMigration.oplogView": (CommandNotSupportedOnView) Namespace local.system.tenantMigration.oplogView is a view, not a collection

System info

Telegraf 1.22.3

Docker

No response

Steps to reproduce

1.mongodb 5.0.9
2. telegraf user has root permissions in mangodb
3. Simple config for telegraf to run

Expected behavior

Telegraf mongodb plugin should gather_col_stats on collection "type" : "collection"

Actual behavior

Telegraf mongodb plugin gather gather_col_stats on all types of collections

Additional info

Hello.

MongoDb does not support collStats commad on collectin type view.

Example:

MYRS:PRIMARY> use local
switched to db local
MYRS:PRIMARY> show collections
oplog.rs
replset.election
replset.initialSyncId
replset.minvalid
replset.oplogTruncateAfterPoint
startup_log
system.replset
system.rollback.id
system.tenantMigration.oplogView
system.views
MYRS:PRIMARY> db.runCommand( { collStats : "system.tenantMigration.oplogView" } )
{
"ok" : 0,
"errmsg" : "Namespace local.system.tenantMigration.oplogView is a view, not a collection",
"code" : 166,
"codeName" : "CommandNotSupportedOnView",
"$clusterTime" : {
"clusterTime" : Timestamp(1654680040, 1),
"signature" : {
"hash" : BinData(0,"NqmNCz7AR6ZvHq05VXuEobarqyw="),
"keyId" : NumberLong("7106730213999902725")
}
},
"operationTime" : Timestamp(1654680040, 1)
}

Telegraf plugin try to gather_col_stats on collection type "view" , and also gets error :

2022-06-08T09:13:30Z E! [inputs.mongodb] error getting col stats from "system.tenantMigration.oplogView": (CommandNotSupportedOnView) Namespace local.system.tenantMigration.oplogView is a view, not a collection
@mishutka-mmx mishutka-mmx added the bug unexpected problem or unintended behavior label Jun 8, 2022
@powersj
Copy link
Contributor

powersj commented Jun 9, 2022

Hi,

The list of collections is made here. The listCollections docs does say these return all collections and views.

The listCollections takes a filter where the type field could specify collections and timeseries to omit views?

Thanks

@powersj powersj added the help wanted Request for community participation, code, contribution label Jun 9, 2022
@mishutka-mmx
Copy link
Author

Hello, thank you for response.

The listCollections takes a [filter] where the type field could specify collections and timeseries to omit views?

Yes, It would be grate to collect gather_col_stats only by collections and timeseries .

powersj added a commit to powersj/telegraf that referenced this issue Jun 10, 2022
@powersj
fix: filter out views in mongodb lookup
bcd9b44 
fixes: influxdata#11270
@powersj powersj mentioned this issue Jun 10, 2022
Merged
@powersj
Copy link
Contributor

powersj commented Jun 10, 2022

@mishutka-mmx,

I have PR #11280 up with a potential filter for this. Can you try downloading the artifact from that PR and let me know if it resolves this issue?

Thanks!

@mishutka-mmx
Copy link
Author

@powersj

Hello,
I've downloaded linux amd64.tar.gz

/data/telegraf-1.23.0/usr/bin/telegraf -version
Telegraf 1.23.0-bcd9b44c (git: pull/11280 bcd9b44)

No more Error: (CommandNotSupportedOnView) Namespace local.system.tenantMigration.oplogView is a view, not a collection. Now telegraf exlude views.

But now telegraf show another errors:
Error getting collection names: (BadValue) $not needs a regex or a document

/data/telegraf-1.23.0/usr/bin/telegraf --config ./my.conf --debug
2022-06-12T16:44:01Z I! Starting Telegraf 1.23.0-bcd9b44c
2022-06-12T16:44:01Z I! Loaded inputs: mongodb
2022-06-12T16:44:01Z I! Loaded aggregators:
2022-06-12T16:44:01Z I! Loaded processors:
2022-06-12T16:44:01Z I! Loaded outputs: file
2022-06-12T16:44:01Z I! Tags enabled: LifeCycle=Text host=gmv01ndbmgr01 ims_system_id=666
2022-06-12T16:44:01Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"gmv01ndbmgr01", Flush Interval:10s
2022-06-12T16:44:01Z D! [agent] Initializing plugins
2022-06-12T16:44:01Z D! [agent] Connecting outputs
2022-06-12T16:44:01Z D! [agent] Attempting connection to [outputs.file]
2022-06-12T16:44:01Z D! [agent] Successfully connected to outputs.file
2022-06-12T16:44:01Z D! [agent] Starting service inputs
2022-06-12T16:44:11Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2022-06-12T16:44:21Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2022-06-12T16:44:30Z E! [inputs.mongodb] Error getting collection names: (BadValue) $not needs a regex or a document
2022-06-12T16:44:30Z E! [inputs.mongodb] Error getting collection names: (BadValue) $not needs a regex or a document
2022-06-12T16:44:30Z E! [inputs.mongodb] Error getting collection names: (BadValue) $not needs a regex or a document
2022-06-12T16:44:30Z E! [inputs.mongodb] Error getting collection names: (BadValue) $not needs a regex or a document
2022-06-12T16:44:30Z E! [inputs.mongodb] Error getting collection names: (BadValue) $not needs a regex or a document
2022-06-12T16:44:31Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
^C2022-06-12T16:44:35Z D! [agent] Stopping service inputs
2022-06-12T16:44:35Z D! [agent] Input channel closed
2022-06-12T16:44:35Z I! [agent] Hang on, flushing any cached metrics before shutdown
2022-06-12T16:44:35Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2022-06-12T16:44:35Z I! [agent] Stopping running outputs
2022-06-12T16:44:35Z D! [agent] Stopped Successfully

powersj added a commit to powersj/telegraf that referenced this issue Jun 13, 2022
@powersj
fix: filter out views in mongodb lookup
a8dee38 
Views do not support the collection stats. The list collection names
returns collections, timeseries, and views. This removes views.

fixes: influxdata#11270
@powersj
Copy link
Contributor

powersj commented Jun 13, 2022 •
edited
Loading

But now telegraf show another errors:
Error getting collection names: (BadValue) $not needs a regex or a document

Thanks for trying it. I've updated the PR, can you give it another attempt once new artifacts are posted?

Thanks again!

edit:

for reference I ran the following commands on a mongo server to setup a couple collections and views:

> db.createCollection("collectionOne")
> db.createCollection("collectionTwo")
> db.createView('testView','collectionOne', [{
    $lookup: {
        from: "collectionOne",
        pipeline: [
            {
                $match: {
                    $expr: { $eq: ["$id", "42"] }
                }
            },
        ],
        as: "testing"
    }
},
{
    $project: {
        _id: 0,
        __v: 0
    }
}])
> db.getCollectionNames()

@mishutka-mmx
Copy link
Author

Hello.
Thanks you alot.
Now there is no error about VIEWS and debug output is free from errors.

[root@gmv01ndbmgr01 ~]# /data/telegraf-1.24.0/usr/bin/telegraf --version
Telegraf 1.24.0-ac2f5809 (git: pull/11280 ac2f5809)

[root@gmv01ndbmgr01 ~]# /data/telegraf-1.24.0/usr/bin/telegraf --config ./my.conf --debug
2022-06-16T05:43:04Z I! Starting Telegraf 1.24.0-ac2f5809
2022-06-16T05:43:04Z I! Loaded inputs: mongodb
2022-06-16T05:43:04Z I! Loaded aggregators: 
2022-06-16T05:43:04Z I! Loaded processors: 
2022-06-16T05:43:04Z I! Loaded outputs: file
2022-06-16T05:43:04Z I! Tags enabled: LifeCycle=Text host=gmv01ndbmgr01 ims_system_id=666
2022-06-16T05:43:04Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"gmv01ndbmgr01", Flush Interval:10s
2022-06-16T05:43:04Z D! [agent] Initializing plugins
2022-06-16T05:43:04Z D! [agent] Connecting outputs
2022-06-16T05:43:04Z D! [agent] Attempting connection to [outputs.file]
2022-06-16T05:43:04Z D! [agent] Successfully connected to outputs.file
2022-06-16T05:43:04Z D! [agent] Starting service inputs
2022-06-16T05:43:14Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2022-06-16T05:43:24Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2022-06-16T05:43:34Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2022-06-16T05:43:44Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2022-06-16T05:43:54Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
mongodb,LifeCycle=Text,host=gmv01ndbmgr01,hostname=localhost:27017,ims_system_id=666,node_type=PRI

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/mongodb bug unexpected problem or unintended behavior help wanted Request for community participation, code, contribution
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: filter out views in mongodb lookup powersj/telegraf
2 participants
@powersj @mishutka-mmx