Error Writing to Kinesis on EMR spinup

[ryangeno]

Hi -

I'm seeing this issue when we startup logstash as a Step in EMR spinup to forward logs to Kinesis. The strange thing is that once the EMR cluster is 'Ready', restarting logstash connects to Kinesis just fine. Might be an AWS issue which we're exploring, but I'm wondering if this issue has been logged before.

The error:
{:timestamp=>"2016-12-07T19:36:17.136000+0000", :message=>"Error writing event to Kinesis", :exception=>com.amazonaws.services.kinesis.producer.DaemonException: The child process has been shutdown and can no longer accept messages., :level=>:warn}

Kinesis Output Config
kinesis {
stream_name => "${KINESIS_STREAM:XXX}"
region => "us-east-1"
randomized_partition_key => true
aggregation_enabled => false
max_pending_records => 10000
}

I can provide additional information if needed.

rg

[samcday]

@ryangeno that looks like you're missing authentication for the Kinesis plugin. KPL tends to just crap out without a useful message in that case.