Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add system socket MetricSet #3246

Merged
merged 3 commits into from
Jan 2, 2017

Conversation

andrewkroh
Copy link
Member

@andrewkroh andrewkroh commented Dec 23, 2016

The system.socket metricset reports an event for each new TCP socket that it
sees. It does this by polling the kernel to get a dump of all sockets.
So using a short polling interval with this metricset is important to
not miss short lived connections.

The metricset reports the process that has the socket open. It does this
by associating the socket's inode to the process that has a file
descriptor open pointing to the socket's inode. It reads /proc and
/proc//fd just prior to polling the kernel to get all sockets.

A reverse lookup can be performed by the metricset on the remote IP and the
returned hostname will be added to the event and cached. The is disabled by
default and can be enabled through the configuration. If a hostname is found
then the eTLD+1 (effective top-level domain plus one level) value will also be
added to the event.

For the IP address fields the index template for Elasticsearch 5.x uses
the ip field type. But for Elasticsearch 2.x it uses string because the
ip field type in 2.x does not support IPv6 addresses.

@ruflin
Copy link
Member

ruflin commented Dec 23, 2016

Don't forget the CHANGELOG ;-)

@andrewkroh andrewkroh force-pushed the feature/system-socket-metricset branch from 78119ca to 6abd7c9 Compare December 23, 2016 23:37
@andrewkroh
Copy link
Member Author

I added a changelog entry and modified the mapping for ES 5.x to use ip instead of keyword for the IP address fields.

@andrewkroh
Copy link
Member Author

Fixes #3257

Copy link
Contributor

@tsg tsg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Nice work!

return nil, err
}
if os.Geteuid() != 0 {
debugf("socket process info will only be available for metricbeat " +
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would make sense as a warning, maybe?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about INFO?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deal :)


// IsEnabled returns true if enabled is not defined or is set to true.
func (c ReverseLookupConfig) IsEnabled() bool {
return c.Enabled == nil || *c.Enabled == true
Copy link
Contributor

@tsg tsg Jan 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This mean that by default reverse lookup is enabled, right? I'm not sure about enabling that by default since it can cause performance issues, especially since there's no configurable timeout on the lookup, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, there is no configurable lookup timeout and the requests are executed serially. It's not an optimal implementation and it can slow down fetching. I'll change it to disabled by default.

The system.socket metricset reports an event for each new TCP socket that it
sees. It does this by polling the kernel to get a dump of all sockets.
So using a short polling interval with this metricset is important to
not miss short lived connections.

The metricset reports the process that has the socket open. It does this
by associating the socket's inode to the process that has a file
descriptor open pointing to the socket's inode. It reads /proc and
/proc/<pid>/fd just prior to polling the kernel to get all sockets.

A reverse lookup can be performed by the metricset on the remote IP and the
returned hostname will be added to the event and cached. The is disabled by
default and can be enabled through the configuration.  If a hostname is found
then the eTLD+1 (effective top-level domain plus one level) value will also be
added to the event.

For the IP address fields the index template for Elasticsearch 5.x uses
the ip field type. But for Elasticsearch 2.x it uses string because the
ip field type in 2.x does not support IPv6 addresses.
@andrewkroh andrewkroh force-pushed the feature/system-socket-metricset branch from 0a3fc90 to c5a6c97 Compare January 2, 2017 17:04
@andrewkroh
Copy link
Member Author

Rebased. I recommend using the "Rebase and merge" option which will add my 3 separate commits, but will not create a merge commit.

@tsg tsg merged commit 3ab08d1 into elastic:master Jan 2, 2017
@monicasarbu monicasarbu deleted the feature/system-socket-metricset branch January 6, 2017 08:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants