first stab at a graphite listener

[Dieterbe]

this is a RFC. it has some comments, questions and TODO's in the code.

most notably https://github.com/Dieterbe/influxdb/blob/5a35aec4deb0d83a89a24112737ac2f8430d9e27/src/api/graphite/api.go#L148-L160 needs some work

[pauldix]

You don't have to set timestamp if you call to WriteSeriesData as the points come in, it'll be assigned automatically. Sequence number will always get assigned automatically.

You should be using the objects from protocol.pb.go. Basically, create a series object, columns, and then points. Create field values for the points. Timestamp and Sequence number should not be their own fields.

This might give you a better idea:
https://github.com/influxdb/influxdb/blob/master/src/common/serialize_series.go#L56-L132

[Dieterbe]

pushed a bunch of fixes, but now my getAuth() doesn't work because (at startup) len(names) == 0.
how should i fix this? shouldn't always at least 1 admin user exist?

// getAuth assures that the user property is a user with access to the graphite database
func (self *Server) getAuth() {
    // just use any (the first) of the list of admins.
    // can we assume there's always at least 1 ?
    names := self.clusterConfig.GetClusterAdmins()
    self.user = self.clusterConfig.GetClusterAdmin(names[0])
}

[pauldix]

It could be empty if the Raft server hasn't started up yet. I would pull the setting of the user credentials outside of the New method for the listener and instead set them when you call ListenAndServe. And make that happen after everything else has been initialized.

[Dieterbe]

I think this is ready for review. I tested it by feeding it data using the diamond agent and that seems to work fine.
2 notes though:

• still need to implement check if db exists and create if needed (paul you said you guys would do this IIRC)
• I did see some errors in the output wrt raft/shard logs, which is seemingly unrelated to the graphite listener. see https://gist.github.com/Dieterbe/9413295 should i open a new ticket for this?

[pauldix]

Agreed. #310 will add that functionality to requests, which will make it possible to update the interface on the coordinator. Will definitely make things more efficient.

[pauldix]

This looks good to me. We can help with the database validation. I'd like @jvshahid to take a look too.

For the Raft log thing, log an issue with as much info as you can and we'll see if we can repro. Thanks!

[zorkian]

I think the shutdown here doesn't do much; or at least, not what you hope.

It works in the HTTP API because the underlying net/http module's Serve() method has some intelligent handling of Accept() failures: http://golang.org/src/pkg/net/http/server.go?s=45477:45526#L1618

In short it determines if it's a temporary or permanent failure, and it aborts the accept loop on permanent failures. That allows the Serve() method to exit, which ends up triggering the defer to send the true on the shutdown channel.

I haven't fully thought through the implications of shutdown in InfluxDB (just started looking at the code today) so I don't know what the right solution in API modules like this is.

[zorkian]

(Thank you for doing this; I've used it as a base for my own OpenTSDB ingester: zorkian@b57b611 ... not done yet, but it's a start and you should recognize the structure!)

[schmurfy]

it is nice to see some other listeners but wouldn't be better if there was a nicer way to have each plugins have its own config without having it merged like that in the rest of the config file ?

[[input_plugins]]
  name = "http_json"
  port = 8086

[[input_plugins]]
  name = "udp_whatever"
  port = 6666

[[input_plugins]]
  name = "http_graphite"
  port = 67888

instead of the current model which seems to be:

[api]
  port = 8086

[graphite]
  port = 67888

This is just an idea but it would help cleaning up the config file and show clearly what is an input plugin and allow users to quickly turn them on and off by commenting the whole block.

another idea:

  [input_plugins.http_json]
    port = 8086

  [input_plugins.udp_whatever]
    port = 6666
    other_param = 42

  [input_plugins.http_graphite]
    port = 67888

The specific syntax does not matter that much to me, what matters is that the plugins are clearly identified in the config and not only by comments.

[pauldix]

I agree, the last example you posted looks best to me.

[schmurfy]

I am not familiar enough withe codebase but could there be a way to have plugins code cleanly split from the core (and to an extend maybe not compiled at all when not needed) ?

Here is an example of what I meant (again that's not the way things are laid out but more the idea behind it):

plugins
  http_graphite
    config.go
    plugin.go

  http_json
    config.go
    plugin.go

I am sure that problem will arise sooner than later as more plugins are added especially if they need specific libraries not needed by the influxdb core, another factor is that once some plugins are out there the "architecture" won't be as easy to change as it could be now.

One of the plugins I want to give a try at is json over udp, when the server writing to influxdb is on the same machine I am sure it could provide a nice boost over http and completely remove its overhead.

[Frusty]

Any plans to make this ingest the Graphite pickle protocol as well?

Here would be my use case:

I currently have a relay in every data center that forwards data from every server to a data center local graphite instance and a centrally hosted instance as well. So I have one point, where I have all data center specific metrics, but I also have a global view. Problem is that the global instance is getting under water more and more.
What I plan to do is relay data to a global InfluxDB cluster as well and evaluate if it is something I would bet on for full production use (it does sound very promising, but I need some test data).

So what do you think? Maybe there are there better options for me?

[Dieterbe]

I'm not a big fan of the pickle protocol. seems like it got added to some of the python daemons because somebody thought it would be cool but IMHO in reality it doesn't help with anything, is hard to debug, and breaks compatibility between different services.. tools that only support pickle protocol should support the plaintext protocol and in my view everything should just use the plaintext protocol. (or if you want to optimize, something truly cross-language such as msgpack, but that's a whole other story)

[Frusty]

I am so with you!

But I think my current setup is actually a common one and it might be something to consider to help people ease into InfluxDB.

I am not sure if there is something similar like the carbon relay for Influx. For me, this is pretty vital, because I do not want every server to be able to directly talk with InfluxDB and do not want to open up a port to the outside world for it. A central relay that could handle this would be cool. But I guess that does not fit into this PR :)

[jvshahid]

Thanks @Dieterbe. The pull request is merged in. I made some changes and added a test.

[Dieterbe]

cool. note that there's one thing still left todo

~/w/e/influxdb ❯❯❯ ack TODO src/api/graphite/api.go
// TODO: check that database exists and create it if not

(paul said this would be easy for you guys to add and i didn't have to worry about it :))

[Dieterbe]

btw @Frusty here's a relay that talks the plaintext protocol. I'll be using it to send metrics to both carbon and influx. https://github.com/rcrowley/carbon-relay-ng

[pcn]

@Frusty the pickle protocol is too complex to implement reasonably in another language because the pickle itself is tightly tied to the implementation of python.

You're much better off transporting with the line protocol for now.

[nivertech]

@Dieterbe does graphite plugin works in v5.6?
https://groups.google.com/forum/#!topic/influxdb/eYgnZF0kGhw