Adjustments to send monitoring data directly to ES #11541

andsel · 2020-01-27T11:57:02Z

This PR is intended to implements the actions of issue #11573, collect the changes requested to internal collector to adapt to version 7.7 of ES monitoring documents.

The key point of the changes:

deprecate xpack.monitoring.* and exposes monitoring.* with same extensions.
if is found some xpack.monitoring.* setting and monitoring.* it make Logstash to exit
switch endpoint, shape documents as done previously from ES exporter
extract the list of cluster_uuid from output-elasticsearch defined in any pipeline, sending monitoring information for each of the cluster_uuid. if not defined it brings from setting monitoring.cluster_uuid, if also that is not defined put a default string empty string.

robbavey

@andsel The data shape changes make sense, but I think we need to make sure we are consistent with the rest of the stack in how we expose the direct shipping option to users.

robbavey · 2020-02-06T21:59:36Z

config/logstash.yml

@@ -236,6 +236,7 @@
 #xpack.monitoring.elasticsearch.sniffing: false
 #xpack.monitoring.collection.interval: 10s
 #xpack.monitoring.collection.pipeline.details.enabled: true
+#xpack.monitoring.collection.write_direct.enabled: true


I'm not sure this is what we want here - if we are to be consistent with beats, they introduced a new class of settings under monitoring.* for their direct shipping feature, which is also called out in the meta issue for the migration to Metricbeat.

I generally agree with you @robbavey, I would also change the name from write_direct to direct_shipping, but I have some doubts to clear up:

to start monitoring we have the xpack.monitoring.enabled settings that has to be enabled to use the internal collectors. So we could potentially have a monitoring.collection.direct_shipping.enabled (more specific) that require a more general xpack.monitoring.enabled to work. But xpack is an extension pack, so we would have something in monitoring.* path the require something form extension pack xpack.monitoring.* to work. I would expect that both stay at the same level, what do you think?

If you look at what the Beats team did - they have three options for monitoring, Legacy internal pointing to production ES cluster, regular internal monitoring pointing to monitoring ES cluster and Metricbeat collection.

How they appear to have structured it, is to have the Legacy monitoring being configured by settings in the xpack.monitoring namespace, and the 'direct' monitoring being configured by settings in the monitoring namespace, rather than having a specific setting for direct_shipping. To enable legacy monitoring, xpack.monitoring.enabled would be used, and to use direct monitoring, monitoring.enabled

Nice, deprecating the Legacy they redirect to the new and removed the documentation for the settings redirecting to the new internal collectors for reference. So we could:

keep the handling of old xpack.monitoring settings for back compatibility

copy the old setting in xpack.monitoring.* to the monitoring.* where monitoring.enabled means "use the ship direct way"

adapt the documentation to reflect this

Yes, I think that makes sense.

@robbavey with commit 5084eb1 I've done the rename, please, let me know what do you think about it

I think the rename makes sense - @chrisronline, @ycombinator, does this make sense to you, and is it consistent with the rest of the stack in terms of adding direct to monitoring cluster functionality?

Yes, monitoring.* settings is exactly what we're pushing for consistency across the stack 👍

robbavey

I'm happy with the general structure of the settings that were added, but before committing, I think a few things should be added in addition to those mentioned in the review:

I'd like to see an integration test with the new shape of data being sent to the _bulk endpoint in Elasticsearch.
I also think there should be a test for - that would at least log, but probably disallow - configurations that have set values for monitoring.* and xpack.monitoring.* in their logstash.yml

robbavey · 2020-02-11T22:18:53Z

config/logstash.yml

@@ -216,6 +216,23 @@
 # Default is false
 # pipeline.separate_logs: false
 #
+# Enables internal collector to shipping directly to Elasticsearch


Maybe change this to something like 'Enable monitoring via internal collector to an Elasticsearch monitoring cluster`

robbavey · 2020-02-11T22:20:01Z

config/logstash.yml

+#monitoring.elasticsearch.ssl.verification_mode: certificate
+#monitoring.elasticsearch.sniffing: false
+#monitoring.collection.interval: 10s
+#monitoring.collection.pipeline.details.enabled: true
 # ------------ X-Pack Settings (not applicable for OSS build)--------------


I think we might want to consider removing the xpack settings here, and just have the monitoring settings. We should also make a note to surface this in release notes and blogs for the 7.7.0 release

robbavey · 2020-02-11T22:46:44Z

x-pack/lib/monitoring/monitoring.rb

+
+      def monitoring_endpoint
+        if LogStash::MonitoringExtension.use_direct_shipping?(LogStash::SETTINGS)
+          "/_bulk/monitoring"


Is this correct? Testing this code against 7.6.0 generates an error, and glancing at the existing beats code that does this, it appears to send to a bare _bulk endpoint (rather than _bulk/monitoring, and send a calculated index name with the request.

robbavey · 2020-02-11T22:49:08Z

x-pack/lib/template.cfg.erb

@@ -20,7 +20,7 @@ output {
  <% else %>
    hosts => <%= es_hosts %>
  <% end %>
-    bulk_path => "/_monitoring/bulk?system_id=logstash&system_api_version=<%= system_api_version %>&interval=1s"
+    bulk_path => "<%= monitoring_endpoint %>?system_id=logstash&system_api_version=<%= system_api_version %>&interval=1s"


I don't think the query parameters are going to work

robbavey · 2020-02-11T22:49:27Z

x-pack/lib/template.cfg.erb

@@ -20,7 +20,7 @@ output {
  <% else %>
    hosts => <%= es_hosts %>
  <% end %>
-    bulk_path => "/_monitoring/bulk?system_id=logstash&system_api_version=<%= system_api_version %>&interval=1s"
+    bulk_path => "<%= monitoring_endpoint %>?system_id=logstash&system_api_version=<%= system_api_version %>&interval=1s"
    manage_template => false
    document_type => "%{[@metadata][document_type]}"
    index => ""


I suspect we may need to have a value here for the _bulk endpoint

ycombinator · 2020-02-12T14:14:50Z

I'd like to see an integration test with the new shape of data being sent to the _bulk endpoint in Elasticsearch.

FWIW, Beats has an integration test that tests collecting+sending monitoring data with xpack.monitoring.* settings and then with monitoring.* settings, and then compares both results to make sure they're structurally the same: https://github.com/elastic/beats/blob/69cd8d89dfebec5410746e086fe0c772f58d2662/libbeat/tests/system/test_monitoring.py#L95-L154.

I also think there should be a test for - that would at least log, but probably disallow - configurations that have set values for monitoring.* and xpack.monitoring.* in their logstash.yml

We have a check like this in Beats: https://github.com/elastic/beats/blob/df17e269c0e4db12b3d7b093445108717eade52c/libbeat/monitoring/monitoring.go#L88-L104

andsel · 2020-02-12T16:38:40Z

Thanks for the suggestion @ycombinator I'll create such a test because actually we miss 4 properties in the new document cluster_uuid , timestamp, interval_ms and source_node and I can't see the Logstash data in Kibana, I think maily because we don't set the cluster_uuid

ycombinator · 2020-02-12T16:53:00Z

I think mainly because we don't set the cluster_uuid.

Yes, I think you are almost certainly right about this.

And this brings up an important point, actually. When we built the logstash module in Metricbeat for externally monitoring Logstash nodes, we had to answer the same question — what value should we assign to the cluster_uuid field in the docs created by Metricbeat and indexed directly into the monitoring cluster? The solution we came up with was two-fold:

First, check if any of the Logstash pipelines running on the node being monitored used an Elasticsearch output plugin. If they do, get the cluster_uuid for that Elasticsearch cluster and use it in monitoring documents.
In the absence of a cluster_uuid from step 1 above, fallback to the value of the monitoring.cluster_uuid setting recently introduced in Conditionally add monitoring structure to top-level of output in API #11106.

Maybe some of the above is already known but I thought I'd mention it anyway, just to be safe. Internal collection + shipping directly to monitoring cluster should follow the same logic to be consistent with Metricbeat (external) collection + shipping directly to the monitoring cluster.

ycombinator · 2020-02-12T17:27:10Z

actually we miss 4 properties in the new document cluster_uuid , timestamp, interval_ms and source_node

I checked through the source code for the logstash module in Metricbeat, specifically the metricsets that create type:logstash_stats and type:logstash_state documents in .monitoring-logstash-* indices. Here are the relevant sections of code:

For type:logstash_stats documents: https://github.com/elastic/beats/blob/56c93e536c14abccbd14c64427e6023a865833b5/metricbeat/module/logstash/node_stats/data_xpack.go#L194-L203
For type:logstash_state documents: https://github.com/elastic/beats/blob/56c93e536c14abccbd14c64427e6023a865833b5/metricbeat/module/logstash/node/data_xpack.go#L45-L54

As you can see:

timestamp: is set to the current time / now.
interval_ms: is set to how frequently the monitoring data is collected. In the case of the logstash module in Metricbeat the default is 10s, so the default value of interval_ms works out to 10000.
source_node: is not indexed at all. This is OK because the Stack Monitoring UI doesn't actually used this field.

Historically this field has been added by the production Elasticsearch cluster (when using the legacy monitoring method of internal collection + shipping to production cluster). It was meant to contain information about the Elasticsearch node on the production cluster through which the monitoring data was passing, en route to the monitoring cluster. The idea was to capture this information in case we needed it for debugging issues with the routing of the monitoring data.

But when monitoring data is shipped directly to the monitoring cluster, either via internal collection (this PR) or Metricbeat collection, there is no concept of source node so it makes sense to simply omit this field. Of course, your comparison/parity tests will have to account for this omission.
cluster_uuid: we talked about how to determine that in the previous comment.

Hope this helps!

chrisronline · 2020-02-13T21:53:09Z

One more thing to note here. The Elasticsearch code does the date formatting (to determine the YYYY-MM-DD part of the monitoring index) using UTC time (instead of the server default time). I'm going to mirror this in the Kibana code, and recommend doing this here as well.

robbavey

Getting closer - my main concerns here are around the setting of the cluster uuid

config/logstash.yml

x-pack/lib/monitoring/inputs/metrics/state_event_factory.rb

x-pack/lib/monitoring/inputs/metrics/stats_event_factory.rb

x-pack/lib/monitoring/monitoring.rb

x-pack/lib/monitoring/outputs/elasticsearch_monitoring.rb

robbavey · 2020-02-20T21:08:08Z

x-pack/lib/monitoring/inputs/metrics/stats_event_factory.rb

+    def self.format_cluster_uuid(stats)
+      result = stats.extract_metrics([:stats, :pipelines, :main, :config], :cluster_uuids)
+      if result && !result[:cluster_uuids].empty?
+        result[:cluster_uuids][0]


I wonder about setting precedence here - we might want to have the monitoring.cluster_uuid override the value from cluster_uuids, particularly if there are >1 elasticsearch outputs pointing to different clusters.

We might want to log when we detect >1 distinct cluster_uuid values.

I agree with you, explicit user defined values should have precedence on implicitly extracted values

The Stack Monitoring UI is centered around the notion of an Elasticsearch cluster. User's first choose an Elasticsearch cluster, then drill deeper into Elasticsearch, Logstash, etc. monitoring data associated with that cluster. So the purpose of the cluster_uuid field in monitoring documents is to associate the monitoring data in the document with an ES cluster in the Stack Monitoring UI.

In terms of Logstash pipelines with multiple ES outputs, I think it might make sense to show that Logstash pipeline under each of those ES clusters in the UI. The only way to pull this off today is to index the same document multiple times, but with a different cluster_uuid value each time. We do this in the Metricbeat logstash module code: https://github.com/elastic/beats/blob/523528b7516adbda814d8770a2e259a4464023b0/metricbeat/module/logstash/node/data_xpack.go#L83-L91.

Happy to go a different direction here but either way we should probably keep the Metricbeat logstash module implementation and the implementation in this PR consistent.

so we could use give precedence to the monitoring.cluster_uuid settings if defined us that. If not defined, collect the list of cluster_uuid from all ES output plugins and change the metric input to fire one event for each cluster_uuid discovered

Actually, we are doing it the other way around in the Metricbeat logstash module. First precedence is given to cluster_uuid(s) discovered from the Logstash pipeline. If there are none, then second precedence is given to the monitoring.cluster_uuid setting in logstash.yml. If that is also empty, set the value to an empty string ("").

@ycombinator Interesting! I agree with the first notion that Logstash data is shown for each of the clusters, good to keep consistency with the Logstash module.

For the second point, does this mean effectively that monitoring.cluster_uuid is only applicable when no elasticsearch output is defined? Do you log when monitoring.cluster_uuid is ignored?

For the second point, does this mean effectively that monitoring.cluster_uuid is only applicable when no elasticsearch output is defined?

Yes, that's how we've been treating it in the Metricbeat logstash module: monitoring.cluster_uuid is a fallback only in cases when something more specific (an output in a specific pipeline on the node) can't provide the cluster UUID.

Do you log when monitoring.cluster_uuid is ignored?

We do not, but perhaps we could, at least at a DEBUG level. I'm happy to mirror in the Metricbeat logstash module whatever you decide to do about logging here in this PR.

ok, cool. In that case, I think I would prefer that we log that (ideally only once, rather than each time a metric is sent...), and also document the fact that monitoring.cluster_uuid is only relevant when no elasticsearch output is defined, and will be ignored when at least one es out is present.

I followed your suggestions. With commit eebfcd9 I switched the retrieval of cluster_uuid to be lazy and done only once in the input metric plugin instead of each event generation.

With commit 25beb49 I switched the cluster_uuid to be a list, retrieved by all elasticsearch output plugins, so that the same monitoring event is sent for each cluster_uuid.

@ycombinator I added a log line when we use the cluster_uuids instead of the setting from config file

What is left out is document better what's the intended use of monitoring.cluster_uuidsetting.

robbavey · 2020-02-20T21:08:33Z

x-pack/lib/monitoring/inputs/metrics/stats_event_factory.rb

+        if LogStash::SETTINGS.set?("monitoring.cluster_uuid")
+          LogStash::SETTINGS.get("monitoring.cluster_uuid")
+        else
+          "<UNDEFINED cluster>"


I believe this needs to be blank, rather than <UNDEFINED cluster> - if the cluster_uuid field is blank, then the stack monitoring page in Kibana will show a page with a list of clusters to select from, including a "standalone" cluster. The monitoring entries for a Logstash instance where the cluster_uuid cannot be found are located on the "standalone" cluster page.

I think we should also log here.

config/logstash.yml

robbavey · 2020-02-24T16:13:12Z

x-pack/lib/monitoring/inputs/metrics/state_event_factory.rb

@@ -5,16 +5,30 @@
 module LogStash; module Inputs; class Metrics;
  class StateEventFactory
    require "logstash/config/lir_serializer"
-    def initialize(pipeline)
+
+    def initialize(pipeline, snapshot, cluster_uuid, collection_interval = 10)


snapshot parameter no longer needed

robbavey · 2020-02-24T16:16:18Z

x-pack/lib/monitoring/inputs/metrics.rb

      update_stats(snapshot)
-      update_states
+      update_states(snapshot)


snapshot no longer needed for state updates

robbavey

Getting close! The snapshot parameter can be removed from throughout the update_state flow, and I have a nit/suggestion on logging.

x-pack/lib/monitoring/inputs/metrics.rb

andsel · 2020-02-25T11:12:39Z

Jenkins test this please

andsel · 2020-02-25T13:34:55Z

Jenkins test this

robbavey

Mostly nits

config/logstash.yml

robbavey · 2020-02-25T16:52:17Z

x-pack/lib/monitoring/inputs/metrics.rb

@@ -105,15 +106,28 @@ def stop
    end

    def update(snapshot)
+      if LogStash::MonitoringExtension.use_direct_shipping?(LogStash::SETTINGS)


What happens in the case of configuration reloads? Will new cluster_uuids be picked up?

In case direct shipping is enabled and a user change the ES output configuration in one of the pipelines those new cluster_uuids aren't brought up by the monitoring pipeline because the monitoring pipeline is not restarted.
Actually when the agent converge_state_and_update it reloads the changed pipelines and update_metrics but doesn't reload also the monitoring pipeline, should we scatter a refresh of that pipeline also?

However this behaveior is almost aligned to what it's done for not direct shipping, because in that case the ES to ship data is loaded from logstash.yml and that ES is responsible to enrich the data with cluster_uuid, so that configuration to be changed force the user to restart Logstash

I think I'm ok with this, and we can always follow up if this becomes an issue.

Its slightly different from the non-direct shipping case, as the logstash.yml changes would be a change to the location of the monitoring data, whereas the changes drawn from reloaded plugins would be a change to where that data is shown in the monitoring cluster.

robbavey · 2020-02-25T16:53:32Z

x-pack/lib/monitoring/inputs/metrics.rb

+      if result && !result[:cluster_uuids].empty?
+        cluster_uuids = result[:cluster_uuids]
+        @logger.debug("Elasticserach output cluster_uuids discovered: ", :cluster_uuids => cluster_uuids)
+        @logger.info("Found cluster_uuid from elasticsearch output plugins, ignoring setting monitoring.cluster_uuid")


Is this info statement still needed?

It's redundant respecting the debug, but probably as a user that configure ES outputs and also monitoring.cluster_uuid I would like to know why the settings in logstash.yml is not taken into account. I would keep the info logging, enriching with the discovered list of cluster_uuids provided by ES output plugins.

x-pack/spec/monitoring/inputs/metrics_spec.rb

x-pack/qa/integration/monitoring/es_documents_structure_validation_spec.rb

andsel · 2020-02-26T13:48:09Z

Jenkins test this

robbavey

Very close! Just another nitpicky comment/suggestion on logging

robbavey · 2020-02-26T14:26:34Z

x-pack/lib/monitoring/inputs/metrics.rb

+      result = stats.extract_metrics([:stats, :pipelines, :main, :config], :cluster_uuids)
+      if result && !result[:cluster_uuids].empty?
+        cluster_uuids = result[:cluster_uuids]
+        @logger.info("Found cluster_uuid from elasticsearch output plugins, ignoring setting monitoring.cluster_uuid. Using cluster_uuids: ", :cluster_uuids => cluster_uuids)


I still have mixed feelings about mentioning ignoring the cluster_uuid setting when it has not been set. As long as we document the setting appropriately, the cluster_uuid setting shouldn't be of concern to most users and we should probably only draw attention to it when it is being used - maybe with an info message if it is being used, and a warning message if it is overridden by the values drawn from the plugins.

I agree with you, citing something that the user hasn't configured is not good, reworded with a7e9c38

robbavey · 2020-02-26T14:29:30Z

x-pack/lib/monitoring/inputs/metrics.rb

@@ -105,15 +106,28 @@ def stop
    end

    def update(snapshot)
+      if LogStash::MonitoringExtension.use_direct_shipping?(LogStash::SETTINGS)


I think I'm ok with this, and we can always follow up if this becomes an issue.

Its slightly different from the non-direct shipping case, as the logstash.yml changes would be a change to the location of the monitoring data, whereas the changes drawn from reloaded plugins would be a change to where that data is shown in the monitoring cluster.

robbavey

LGTM

… cluster Close 11573

elasticsearch-bot · 2020-02-28T14:26:22Z

Andrea Selva merged this into the following branches!

Branch	Commits
master	`3695580`

… cluster Close 11573 Fixes elastic#11541

… cluster Close 11573 Added check on HTTP server before asking for monitoring data in unit test Fixes elastic#11541

… cluster Close 11573 Added check on HTTP server before asking for monitoring data in unit test Fixes #11541 Fixes #11641

… cluster Close 11573 (#11641) Added check on HTTP server before asking for monitoring data in unit test Fixes #11541

…S monitoring cluster During the development of PR elastic#11541 to direct ship monitoring data to an monitoring ES cluster without hopping through a production ES cluster, the settings for elasticsearch ouput was cloned into a version without the `xpack` prefix. Since that feature has been removed the settings should also be removed from the Docker image

…S monitoring cluster During the development of PR #11541 to direct ship monitoring data to an monitoring ES cluster without hopping through a production ES cluster, the settings for elasticsearch ouput was cloned into a version without the `xpack` prefix. Since that feature has been removed the settings should also be removed from the Docker image

This was referenced Jan 31, 2020

[7.7] [Monitoring] Collection changes #11403

Closed

[Meta Issue] Internal collector adjustments for 7.7 #11573

Closed

andsel self-assigned this Feb 4, 2020

andsel mentioned this pull request Feb 6, 2020

[Doc] added description of monitoring setting write_direct #11586

Closed

andsel marked this pull request as ready for review February 6, 2020 08:27

andsel requested a review from robbavey February 6, 2020 08:27

robbavey requested changes Feb 7, 2020

View reviewed changes

robbavey requested changes Feb 11, 2020

View reviewed changes

andsel requested a review from robbavey February 17, 2020 15:17

robbavey requested changes Feb 20, 2020

View reviewed changes

robbavey reviewed Feb 24, 2020

View reviewed changes

robbavey requested changes Feb 24, 2020

View reviewed changes

x-pack/lib/monitoring/inputs/metrics.rb Outdated Show resolved Hide resolved

x-pack/lib/monitoring/inputs/metrics.rb Outdated Show resolved Hide resolved

karenzone mentioned this pull request Feb 24, 2020

[Doc]Rework monitoring docs #11614

Closed

andsel requested a review from robbavey February 25, 2020 09:57

robbavey requested changes Feb 25, 2020

View reviewed changes

robbavey reviewed Feb 26, 2020

View reviewed changes

robbavey approved these changes Feb 26, 2020

View reviewed changes

andsel force-pushed the feature/adaptations_to_internal_collector_for_7.7 branch 2 times, most recently from 64d33f9 to 3ae0fc4 Compare February 27, 2020 11:27

Adaptations to internal collector to send data directly to monitoring…

61d1634

… cluster Close 11573

andsel force-pushed the feature/adaptations_to_internal_collector_for_7.7 branch from 3ae0fc4 to 61d1634 Compare February 28, 2020 14:04

elasticsearch-bot closed this in 3695580 Feb 28, 2020

andsel added a commit to andsel/logstash that referenced this pull request Feb 28, 2020

Adaptations to internal collector to send data directly to monitoring…

fa62840

… cluster Close 11573 Fixes elastic#11541

andsel mentioned this pull request Feb 28, 2020

Backport of PR #11541 to 7.7 #11641

Merged

andsel added the v8.0.0 label Feb 28, 2020

andsel mentioned this pull request Mar 4, 2020

Fixed timeformat that introduced flaky tests depending on time #11652

Closed

andsel added a commit to andsel/logstash that referenced this pull request Mar 11, 2020

Adaptations to internal collector to send data directly to monitoring…

32f8725

… cluster Close 11573 Added check on HTTP server before asking for monitoring data in unit test Fixes elastic#11541

elasticsearch-bot pushed a commit that referenced this pull request Mar 12, 2020

Adaptations to internal collector to send data directly to monitoring…

d901616

… cluster Close 11573 Added check on HTTP server before asking for monitoring data in unit test Fixes #11541 Fixes #11641

roaksoax pushed a commit that referenced this pull request Mar 13, 2020

Adaptations to internal collector to send data directly to monitoring…

7116d39

… cluster Close 11573 (#11641) Added check on HTTP server before asking for monitoring data in unit test Fixes #11541

robbavey mentioned this pull request Apr 13, 2020

Fix hard coded constraint on pipeline name for metrics #11777

Closed

This was referenced Jul 29, 2020

Remove settings that was intended to direct ship monitoring data to ES monitoring cluster #12152

Merged

Doc:Fix link to monitoring docs #11960

Merged

Adjustments to send monitoring data directly to ES #11541

Adjustments to send monitoring data directly to ES #11541

Conversation

andsel commented Jan 27, 2020 • edited Loading

robbavey left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robbavey left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ycombinator commented Feb 12, 2020 • edited Loading

andsel commented Feb 12, 2020

ycombinator commented Feb 12, 2020 • edited Loading

ycombinator commented Feb 12, 2020 • edited Loading

chrisronline commented Feb 13, 2020

robbavey left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ycombinator Feb 21, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andsel Feb 24, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robbavey left a comment

Choose a reason for hiding this comment

andsel commented Feb 25, 2020

andsel commented Feb 25, 2020

robbavey left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andsel Feb 26, 2020 • edited Loading

Choose a reason for hiding this comment

andsel commented Feb 26, 2020

robbavey left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robbavey left a comment

Choose a reason for hiding this comment

elasticsearch-bot commented Feb 28, 2020

andsel commented Jan 27, 2020 •

edited

Loading

ycombinator commented Feb 12, 2020 •

edited

Loading

ycombinator commented Feb 12, 2020 •

edited

Loading

ycombinator commented Feb 12, 2020 •

edited

Loading

ycombinator Feb 21, 2020 •

edited

Loading

andsel Feb 24, 2020 •

edited

Loading

andsel Feb 26, 2020 •

edited

Loading