Add Logstash output for Functionbeat

[bczifra]

Describe the enhancement:
Add a Logstash output for Functionbeat

Describe a specific use case for the enhancement or feature:
Elasticsearch clusters may be inaccessible from the cloud where Functionbeat runs. In such situations, it would be helpful to output to Logstash which could then forward the events as necessary.

An example flow: Functionbeat -> Logstash in internet -> Kafka queues in internet -> Logstash in intranet -> ... -> Elasticsearch.

related: #9866

[kylegoch]

This would also be useful for cases were you are using beats elsewhere and they all go to Logstash.

Having a beat that doesn't ship to Logstash seems silly, but I digress.

[cph1c06]

Shipment to Logstash is necessary function for Functionbeat as users can manipulate data using methods/parameters they had been familiar with for years.

[L-F-Escobar]

can functionbeat installed on aws cloudwatch/lambda send logs from a specific log group to logstash?

Within the functionbeat.yml file I see a logstash output section. Currently I am sending logs directly to my elastic cloud with my id/password. How can I send logs to logstash for data enrichment and then to kibana for visualization?

This is my functionbeat.yml file

functionbeat.provider.aws.deploy_bucket: "functionbeat-lambdas"

functionbeat.provider.aws.functions:
  - name: cloudwatch
    enabled: true
    type: cloudwatch_logs

# Description of the method to help identify them when you run multiples functions.
description: "lambda function for cloudwatch log ingestion"

# List of cloudwatch log group registered to that function.
triggers:
  - log_group_name: /log-grp-1
  - log_group_name: /log-grp-2




#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 1


#============================= Elastic Cloud==================================

cloud.id: "testing:dXMtd2VzdC0xLmF3cy5mb3VuZC5pbyQxNGNkODY0ZTlkOGU0NmY1OGZlZWRiOGU5MWRhMjJiNSQ0MWU4Zjk4ODVlMDE0OThhYWEwMzkwOTkyNjI4NmZjOQ=="
cloud.auth: "elastic:{ES_PWD}"



#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

[kvch]

Just to clear the confusion, Logstash and other outputs are part of the reference configuration is incorrect. They are going to be removed in a future release because none of them is supported apart from ES at the moment.

[L-F-Escobar]

@kvch So I cannot use functionbeat to ship cloudwatch logs to logstash?

[kvch]

Yes, exactly. Only ES output is supported ATM.

[Randy-312]

I need this to be an official part of my supported platform, so that I can use functionbeat.

[CloudViking86]

Agree, Logstash support would be nice

Edit:
Weird that a user that haven't participated in the conversation just gave me a thumbs down, anyway grateful for the implementation :)