-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add UDP input back #2751
Add UDP input back #2751
Conversation
@renan- Thanks for the work. The reason we removed it was we are not going to support json on this endpoint. We are only going to support the new line protocol here likely. I'll get @pauldix to confirm though to be sure. Regardless, I imagine at the very least we would do something similar to the http endpoint and detect json vs. line protocol to support both. |
Yeah, I'd prefer to have UDP support only the line protocol. Which would mean it would need configuration options like the Graphite and CollectD input plugins. Set which database and retention policy you're writing into. See #2696 for discussion on the merits of line protocol over JSON. |
Thanks for the quick answers guys ! I desperately need UDP support for my use case (to avoid any latency when writing data) so would you be interested in another PR supporting line protocol instead of JSON ? |
@renan- that would be great, we'd gladly take it. Going to cut RC32 tomorrow so if you can get it in time, it'll be included! Thanks again for helping out! |
@renan- oh, another thing to look at is to have it use the batcher that graphite and collectd now use. |
Thanks for the tip, starting to work on it. |
0f27ecf
to
df60fa5
Compare
Updated the code, now waiting for your feedback. |
df60fa5
to
043613d
Compare
@@ -77,6 +78,7 @@ type Config struct { | |||
Graphites []graphite.Config `toml:"graphite"` | |||
Collectd collectd.Config `toml:"collectd"` | |||
OpenTSDB opentsdb.Config `toml:"opentsdb"` | |||
UDP udp.Config `toml:"udp"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please add a test in config_test.go
for this addition of the config.
@renan- Thanks for the hard work! Certainly headed in the correct direction. Please let me know if you have any questions. I know not all of our services are in perfect shape yet either, but feel free to look at collectd to get an idea of some of the patterns we are currently using (I reference some as well in my comments). |
b203245
to
4bf9784
Compare
f0dd608
to
1f102f1
Compare
@corylanou You're welcome! I updated the PR once again based on your comments. I still wonder if the retention policy should be in config or always set to the default one. One more thing: what about ConsistencyLevel value when creating a WritePointsRequest ? |
1f102f1
to
a757c01
Compare
@@ -1010,6 +1010,9 @@ func NewConfig() *run.Config { | |||
c.HTTPD.Enabled = true | |||
c.HTTPD.BindAddress = "127.0.0.1:0" | |||
c.HTTPD.LogEnabled = testing.Verbose() | |||
|
|||
c.UDP.BindAddress = "127.0.0.1:0" | |||
c.UDP.Database = "db0" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We actually don't want to add these. This would enable the UDP endpoint for every test. We use this function to create a base
config for the tests to use, and then turn on specific endpoints in the tests themselves. Simply revert the change to this file and we should be fine.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tests on CircleCI won't pass without these (panic because of the error returned by ListenAndServe when BindAddress or Database are empty)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the real problem here is that the server isn't observing if UDP is enabled, and actually adding it to the services. I'm guessing a couple of our endpoints have that problem. I'll merge this, and then do another PR that fixes services starting that aren't enabled.
@renan- Awesome! Thanks for all the work. Very much appreciated! Had one last minor comment. Looks like something changed that is preventing a merge so you'll have to rebase quick to make this mergeable. For now let the default RP be used. It can be a feature we add in the future to specify an RP. If we do, all endpoints should likely support it as well. Same with consistency level. feels like that should be a config option as well, but I'm fine with this PR not addressing it. Can be an easy feature to add. Thanks again for the quick turn around. Hopefully we'll get this merged for tomorrow's RC. |
a757c01
to
3203183
Compare
3203183
to
3b09a59
Compare
Hi there,
Seeing that my previous pull request is not likely to get merged because of the big refactoring going on, here's a PR to add UDP support back (not sure why it was removed) in alpha1. I hope this will be helpful, and I'm looking forward to the next release ! (got rc31 running in production and still seeing some issues as well as feeling the lack of CQs).
Keep up the good work,
Renan.