create http dispatcher #91

patrickdemers6 · 2023-12-20T23:51:33Z

Description

Creating an HTTP dispatcher intended to be used for small-scale personal projects. Data received from a vehicle is delivered through a POST request to the specified http server. The body of this request is JSON to make for easy usage.

The existing dispatchers are great for high reliability, high throughput systems, but for an owner wanting to stream data from their own vehicles, setting up and maintaining this infrastructure is a bit much. This allows enthusiasts to create a simple web server which receives the data.

As called out in the updated Readme, this should not be used in production grade systems. Failed messages are discarded and sending http requests does not scale well.

Type of change

Please select all options that apply to this change:

New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Bug fix (non-breaking change which fixes an issue)
Documentation update

Checklist:

Confirm you have completed the following steps:

My code follows the style of this project.
I have performed a self-review of my code.
I have made corresponding updates to the documentation.
I have added/updated unit tests to cover my changes.
I have added/updated integration tests to cover my changes.

Additional Notes

I am naming the integration tests container so it can be referenced through Docker DNS.

JSON payload format:

{
  "createdAt": "2023-12-21T02:46:51.386927610Z",
  "vin": "device-1"
  "data": [
    {
      "key": "VehicleName",
      "value": {
        "stringValue": "My Test Vehicle"
      }
    },
    {
      "key": "Location",
      "value": {
        "locationValue": {
          "latitude": -37.412374,
          "longitude": 122.145867
        }
      }
    }
  ]
}

agbpatro · 2023-12-20T23:55:05Z

datastore/http/http.go

+		namespace:        namespace,
+		metricsCollector: metricsCollector,
+		records:          requests,
+		address:          config.Address,


can address be localhost? We should prevent that in validateConfig

Why do you say we should prevent localhost?

will that not mean http producer can call fleet telemetry server itself?

The producer could call fleet telemetry server but I believe that would be rejected due to invalid certificates. And it would be useless.

The use case I am picturing is a server running with these processes:

fleet-telemetry: exposed on port 4443

custom-web-server: internally on port 3000. This is what receives vehicle data over http.

datastore/http/http.go

test/integration/test.sh

datastore/http/http.go

datastore/simple/logger.go

agbpatro · 2024-01-03T01:20:39Z

datastore/simple/logger_test.go

@@ -14,51 +15,28 @@ import (
 var _ = Describe("Proto Logger", func() {
 	var (
 		protoLogger *simple.ProtoLogger
+		logger      *logrus.Logger


do we need logger here?

Revised. This section is a bit complicated due to BeforeEach needing to define an outer and local scope variable from a function that returns multiple values.

datastore/simple/logger_test.go

test/integration/integration_test.go

patrickdemers6 · 2024-01-08T19:27:54Z

@agbpatro / @aaronpkahn could I get eyes on this again?

agbpatro · 2024-01-08T19:35:55Z

@agbpatro / @aaronpkahn could I get eyes on this again?

I am still evaluating the value proposition for it. We already have support ZMQ for which small scale developers can use for their application without infra overhead and can do same things which http producer implemented here does

hauserkristof · 2024-01-17T21:28:57Z

a overhead and can do same things which http producer implemented here does

Hey @agbpatro !

I would really like to see this feature, it is a great idea, and provides a simple and easy solution for integration and PoC codes.

agbpatro

@patrickdemers6 can you rebase the PR so that it picks up latest changes and resolve merge conflicts?

agbpatro · 2024-01-22T23:38:57Z

datastore/http/http.go

+
+func (p *Producer) sendHTTP(record *telemetry.Record) {
+	url := fmt.Sprintf("%s?namespace=%s&type=%s", p.address, p.namespace, record.TxType)
+	payloadJSON, err := record.GetJSONPayload()


change this to record.Payload() to make it consistent with other producers? We have a top level support to send data in json https://github.com/teslamotors/fleet-telemetry/blob/main/README.md#backendsdispatchers

datastore/http/http.go

test/integration/integration_test.go

datastore/http/http.go

test/integration/integration_test.go

agbpatro · 2024-01-23T18:22:03Z

datastore/http/http.go

+}
+
+func (p *Producer) getRecordChannel(vin string) chan *telemetry.Record {
+	return p.workerChannels[vinToHash(vin)%uint32(len(p.workerChannels))]


hmm. I agree with #91 (comment) but at the same time, it could be possible that not all channels will be used (all vins getting partitioned into a single channel or only one vin to stream and increasing worker channel will not make a difference). I guess this is fine since its not going to be recommended producer but maybe we should add a comment in readme about relation between worker channels and vins associated with this account

I agree it's good to call this out. I have added comentary about this to WorkerCount in Config. We could also add a label to http_produce_buffer_size metric calling out size of each worker's buffer. Want me to add this?

patrickdemers6 · 2024-01-24T00:11:15Z

datastore/http/http.go

+	contentType := protobufContentType
+	if p.produceDecodedRecords {
+		contentType = jsonContentType
+	}
+	req.Header.Set("Content-Type", contentType)


Updated to set Content-Type based on json vs protobuf. I'm using application/x-protobuf when it's the proto, but there are not clear standards on this available. Open to suggestion on this.

patrickdemers6 requested review from aaronpkahn and agbpatro December 20, 2023 23:57

agbpatro reviewed Dec 21, 2023

View reviewed changes

patrickdemers6 force-pushed the http-producer branch 2 times, most recently from bfeb4d8 to 04e67b7 Compare December 21, 2023 03:03

patrickdemers6 requested a review from agbpatro December 21, 2023 03:15

agbpatro reviewed Jan 2, 2024

View reviewed changes

datastore/http/http.go Outdated Show resolved Hide resolved

agbpatro reviewed Jan 2, 2024

View reviewed changes

datastore/simple/logger.go Outdated Show resolved Hide resolved

patrickdemers6 force-pushed the http-producer branch from 04e67b7 to 9d8cc25 Compare January 3, 2024 00:56

patrickdemers6 requested a review from agbpatro January 3, 2024 00:57

agbpatro reviewed Jan 3, 2024

View reviewed changes

test/integration/integration_test.go Outdated Show resolved Hide resolved

patrickdemers6 force-pushed the http-producer branch from 9d8cc25 to bbb9b89 Compare January 3, 2024 03:38

patrickdemers6 requested a review from agbpatro January 3, 2024 03:44

patrickdemers6 force-pushed the http-producer branch from bbb9b89 to a887fa7 Compare January 3, 2024 16:44

agbpatro mentioned this pull request Jan 10, 2024

Allow transmitting data in json bytes #95

Merged

9 tasks

agbpatro reviewed Jan 22, 2024

View reviewed changes

patrickdemers6 force-pushed the http-producer branch from a887fa7 to a327de1 Compare January 22, 2024 23:26

agbpatro reviewed Jan 22, 2024

View reviewed changes

datastore/http/http.go Outdated Show resolved Hide resolved

agbpatro reviewed Jan 22, 2024

View reviewed changes

test/integration/integration_test.go Show resolved Hide resolved

patrickdemers6 force-pushed the http-producer branch from a327de1 to 313008f Compare January 22, 2024 23:56

aaronpkahn reviewed Jan 23, 2024

View reviewed changes

datastore/http/http.go Outdated Show resolved Hide resolved

aaronpkahn reviewed Jan 23, 2024

View reviewed changes

datastore/http/http.go Outdated Show resolved Hide resolved

agbpatro reviewed Jan 23, 2024

View reviewed changes

test/integration/integration_test.go Outdated Show resolved Hide resolved

patrickdemers6 force-pushed the http-producer branch from 313008f to 3a8d02c Compare January 23, 2024 03:40

agbpatro reviewed Jan 23, 2024

View reviewed changes

agbpatro approved these changes Jan 23, 2024

View reviewed changes

patrickdemers6 force-pushed the http-producer branch from 3a8d02c to 622a612 Compare January 24, 2024 00:09

patrickdemers6 commented Jan 24, 2024

View reviewed changes

patrickdemers6 force-pushed the http-producer branch from 622a612 to 4630a2a Compare January 24, 2024 00:18

create http dispatcher

84e8562

patrickdemers6 force-pushed the http-producer branch from 4630a2a to 84e8562 Compare January 24, 2024 01:03

netdata-be mentioned this pull request Jul 8, 2024

Pub/Sub requirement seems to be overkill MyTeslaMate/websocket#2

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

create http dispatcher #91

create http dispatcher #91

patrickdemers6 commented Dec 20, 2023 •

edited

Loading

agbpatro Dec 20, 2023

patrickdemers6 Dec 21, 2023

agbpatro Dec 21, 2023

patrickdemers6 Dec 21, 2023 •

edited

Loading

agbpatro Jan 3, 2024

patrickdemers6 Jan 3, 2024

patrickdemers6 commented Jan 8, 2024

agbpatro commented Jan 8, 2024

hauserkristof commented Jan 17, 2024

agbpatro left a comment

agbpatro Jan 22, 2024

agbpatro Jan 23, 2024

patrickdemers6 Jan 24, 2024

agbpatro Jan 24, 2024

patrickdemers6 Jan 24, 2024

patrickdemers6 Jan 24, 2024

create http dispatcher #91

Are you sure you want to change the base?

create http dispatcher #91

Conversation

patrickdemers6 commented Dec 20, 2023 • edited Loading

Description

Type of change

Checklist:

Additional Notes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickdemers6 Dec 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickdemers6 commented Jan 8, 2024

agbpatro commented Jan 8, 2024

hauserkristof commented Jan 17, 2024

agbpatro left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickdemers6 commented Dec 20, 2023 •

edited

Loading

patrickdemers6 Dec 21, 2023 •

edited

Loading