add support for unix socket binding service #7130

oiooj · 2016-08-09T07:17:08Z

Rebased/mergable
Tests pass
CHANGELOG.md updated
Sign [CLA]

Feature Request in #7075 and #4877. I tested the code, just like the UDP service.

start

[root@LT influxd]# ./influxd run

 8888888           .d888 888                   8888888b.  888888b.
   888            d88P"  888                   888  "Y88b 888  "88b
   888            888    888                   888    888 888  .88P
   888   88888b.  888888 888 888  888 888  888 888    888 8888888K.
   888   888 "88b 888    888 888  888  Y8bd8P' 888    888 888  "Y88b
   888   888  888 888    888 888  888   X88K   888    888 888    888
   888   888  888 888    888 Y88b 888 .d8""8b. 888  .d88P 888   d88P
 8888888 888  888 888    888  "Y88888 888  888 8888888P"  8888888P"

[run] 2016/08/09 14:30:03 InfluxDB starting, version unknown, branch unknown, commit unknown
[run] 2016/08/09 14:30:03 Go version go1.6.2, GOMAXPROCS set to 2
[run] 2016/08/09 14:30:03 Using configuration at: /etc/influxdb/influxdb.conf
[store] 2016/08/09 14:30:03 Using data dir: /var/lib/influxdb/data
[subscriber] 2016/08/09 14:30:03 opened service
[monitor] 2016/08/09 14:30:03 Starting monitor system
[monitor] 2016/08/09 14:30:03 'build' registered for diagnostics monitoring
[monitor] 2016/08/09 14:30:03 'runtime' registered for diagnostics monitoring
[monitor] 2016/08/09 14:30:03 'network' registered for diagnostics monitoring
[monitor] 2016/08/09 14:30:03 'system' registered for diagnostics monitoring
[shard-precreation] 2016/08/09 14:30:03 Starting precreation service with check interval of 10m0s, advance period of 30m0s
[snapshot] 2016/08/09 14:30:03 Starting snapshot service
[admin] 2016/08/09 14:30:03 Starting admin service
[admin] 2016/08/09 14:30:03 Listening on HTTP: [::]:8083
[continuous_querier] 2016/08/09 14:30:03 Starting continuous query service
[httpd] 2016/08/09 14:30:03 Starting HTTP service
[httpd] 2016/08/09 14:30:03 Authentication enabled: false
[httpd] 2016/08/09 14:30:03 Listening on HTTP: [::]:8086
[retention] 2016/08/09 14:30:03 Starting retention policy enforcement service with check interval of 30m0s
[monitor] 2016/08/09 14:30:03 Storing statistics in database '_internal' retention policy 'monitor', at interval 10s
[unixsocket] 2016/08/09 14:30:03 Started listening on unix socket: /tmp/i.sock
[run] 2016/08/09 14:30:03 Listening for signals

write points example code:

package main

import "net"
import "time"
import "fmt"

func main() {
        addr, err := net.ResolveUnixAddr("unix", "/tmp/i.sock")
        if err != nil {
                panic(err)
        }
        c, err := net.DialUnix("unixgram", nil, addr)
        if err != nil {
                panic(err)
        }
        for {
                _, err := c.Write([]byte("cpu,host=serverA,region=us_west value=0.64\n"))
                if err != nil {
                        fmt.Println(err)
                }
                time.Sleep(1e9)
        }
}

query via influx

[root@LT influx]# ./influx 
Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.
Connected to http://localhost:8086 version unknown
InfluxDB shell version: unknown
> show databases
name: databases
---------------
name
unixsocket
_internal

> use unixsocket
Using database unixsocket
> select * from cpu limit 10
name: cpu
---------
time                    host    region  value
1470725065606349352     serverA us_west 0.64
1470725066606549012     serverA us_west 0.64
1470725067606678699     serverA us_west 0.64
1470725068606809610     serverA us_west 0.64
1470725069606963202     serverA us_west 0.64
1470725070607052270     serverA us_west 0.64
1470725071607210170     serverA us_west 0.64
1470725072607312210     serverA us_west 0.64
1470725073607432441     serverA us_west 0.64
1470725074607556441     serverA us_west 0.64

shutdown

[run] 2016/08/09 15:15:53 Signal received, initializing clean shutdown...
[run] 2016/08/09 15:15:53 Waiting for clean shutdown...
[monitor] 2016/08/09 15:15:53 shutting down monitor system
[monitor] 2016/08/09 15:15:53 terminating storage of statistics
[shard-precreation] 2016/08/09 15:15:53 Precreation service terminating
[snapshot] 2016/08/09 15:15:53 snapshot listener closed
[continuous_querier] 2016/08/09 15:15:53 continuous query service terminating
[retention] 2016/08/09 15:15:53 retention policy enforcement terminating
[unixsocket] 2016/08/09 15:15:53 unix socket listener closed
[unixsocket] 2016/08/09 15:15:53 Service closed
[subscriber] 2016/08/09 15:15:53 closed service
[run] 2016/08/09 15:15:53 server shutdown completed

mention-bot · 2016-08-09T07:17:09Z

@oiooj, thanks for your PR! By analyzing the annotation information on this pull request, we identified @gunnaraasen and @e-dard to be potential reviewers

e-dard · 2016-08-09T12:25:10Z

services/unixsocket/service.go

+// SetLogOutput sets the writer to which all logs are written. It must not be
+// called after Open is called.
+func (s *Service) SetLogOutput(w io.Writer) {
+	s.Logger = log.New(w, "[unixsocket] ", log.LstdFlags)


nit: while I know other services have used this pattern, it should really be s.Logger.SetOutput(w) 😄

Also as it stands we would need to protect access to s.Logger, but we can avoid that with SetOutput

e-dard · 2016-08-09T12:29:25Z

@oiooj thanks for this PR, it looks great. It will need some input from some other core team members.

I'm not sure why the TestContinuousQueryService_Run test is failing. I don't think it is to do with your changes. Maybe @jsternberg or @dgnorton could look into that.

oiooj · 2016-08-09T13:26:24Z

@e-dard updated, and all checks have passed. It's strange.

jsternberg · 2016-08-09T14:52:56Z

I don't think this resolves the issue from #7075. As I understood that, they wanted to be able to use the entire HTTP service using a unix socket so it would basically expose that service.

I like this method of configuration better, but I think it would be good to mimic the http service rather than only the write protocol. That way, someone can communicate using HTTP over the unix socket similar to how Docker works. I think it would be possible by just opening a unix socket listener and then passing it to the same handler we use for httpd. See the httpd service for an example.

oiooj · 2016-08-09T15:23:16Z

@jsternberg I don't think so, TCP UDP unix socket, this three protocols are independent. And HTTP over the TCP, unix socket is similar to UDP service , just as a client to write points, high performance is very important., and the line protocol is simple enough. It's no need to expose HTTP service over unix socket.

phemmer · 2016-08-09T15:25:46Z

TCP UDP unix socket, this three protocols are independent. And HTTP over the TCP, unix socket is similar to UDP service

Not true. A unix domain socket can be either SOCK_STREAM, or SOCK_DGRAM. SOCK_STREAM is what TCP uses. SOCK_DGRAM is what UDP uses.

jsternberg · 2016-08-09T15:34:49Z

Unix sockets act more similar to TCP than UDP. I'm sure @phemmer is right though and unix sockets can also be opened as a datagram socket, but that's not the only way of using it.

The reason why I'm saying using the HTTP handler would be more efficient is because it gives more freedom to be able to specify which database and which retention policy you want to write to while also making the query endpoint available. So you would just have to configure it once rather than configure it once per database/retention policy (with having no ability to query points).

I personally think this functionality should just be implemented within the HTTP service itself. In this section of the code, you would just have to open a "unix" listener similar to how docker does it (although I don't know which parts of this function we need and which parts we don't).

jsternberg · 2016-08-09T15:45:00Z

@oiooj I'm a bit confused. What do you mean by unix sockets being similar to udp service? That would be a design decision rather than a technology decision. Unix sockets can be opened as a stream socket and can support HTTP.

UDP cannot support HTTP. HTTP requires a bidirectional stream which the UDP protocol can't support. It's OK for UDP to have its current limitations because UDP doesn't have the capability of doing something more complicated. It's possible to do more with unix sockets and #7075 doesn't seem to be focused only on writes.

oiooj · 2016-08-09T15:52:31Z

@jsternberg sorry, I'm confused. lol

jsternberg · 2016-08-09T15:53:51Z

No problem. That's why we do these code reviews. 😄

oiooj · 2016-08-09T15:56:39Z

Don't forget to tell me the final decision，I will fix this if I'm free

jsternberg · 2016-08-09T22:35:06Z

I think we're going to go with building the unix socket binding into the httpd service. If you can update your PR to do that instead, that would be very helpful.

oiooj · 2016-08-10T06:46:26Z

check #7135 and close this PR

oiooj force-pushed the new-feature branch from a1ab76d to fb8f95c Compare August 9, 2016 07:50

e-dard reviewed Aug 9, 2016
View reviewed changes

add support for unix socket binding service

10b9c70

oiooj force-pushed the new-feature branch from fb8f95c to 10b9c70 Compare August 9, 2016 13:15

oiooj closed this Aug 10, 2016

oiooj deleted the new-feature branch May 29, 2017 14:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add support for unix socket binding service #7130

add support for unix socket binding service #7130

oiooj commented Aug 9, 2016

mention-bot commented Aug 9, 2016

e-dard Aug 9, 2016 •

edited

Loading

e-dard commented Aug 9, 2016

oiooj commented Aug 9, 2016

jsternberg commented Aug 9, 2016

oiooj commented Aug 9, 2016 •

edited

Loading

phemmer commented Aug 9, 2016

jsternberg commented Aug 9, 2016

jsternberg commented Aug 9, 2016

oiooj commented Aug 9, 2016

jsternberg commented Aug 9, 2016

oiooj commented Aug 9, 2016

jsternberg commented Aug 9, 2016

oiooj commented Aug 10, 2016

add support for unix socket binding service #7130

add support for unix socket binding service #7130

Conversation

oiooj commented Aug 9, 2016

start

write points example code:

query via influx

shutdown

mention-bot commented Aug 9, 2016

e-dard Aug 9, 2016 • edited Loading

Choose a reason for hiding this comment

e-dard commented Aug 9, 2016

oiooj commented Aug 9, 2016

jsternberg commented Aug 9, 2016

oiooj commented Aug 9, 2016 • edited Loading

phemmer commented Aug 9, 2016

jsternberg commented Aug 9, 2016

jsternberg commented Aug 9, 2016

oiooj commented Aug 9, 2016

jsternberg commented Aug 9, 2016

oiooj commented Aug 9, 2016

jsternberg commented Aug 9, 2016

oiooj commented Aug 10, 2016

e-dard Aug 9, 2016 •

edited

Loading

oiooj commented Aug 9, 2016 •

edited

Loading