[ingest] - Error handling from simulate API

[BigFunger]

Summary

The initial version of the ingest server and client changes did not have logic to deal with when the simulate ES api would throw an error. It was written to always assume success. This is bad for two reasons: First, because things do not always go smoothly, and second because the gsub and grok processors that are coming in a later PR will throw an error if there is an invalid regular expression passed to them.

Changes

• Added the gsub processor
  • I needed to add this processor so that we can initiate an error by sending an invalid regular expression
• Added the dirtyProcessorId to the pipeline class. This will keep track of the processor that set the pipeline.dirty property to true. This way, if an error is raised, we know which processor caused it to happen.
• [edit] Added logic to consume the new error format from the simulate api to determine with processor caused the compile error
• [edit] Added throw/catch logic for any errors that come back that aren't compile errors.
• Changed the way that the kibana ingest api returns results.
  • Previously the kibana simulate api would return results for each processor, whether an error or success is returned from the es ingest api.
  • Now, when the es simulate returns a data error, the kibana simulate api returns results for each processor up to and including the processor that generated a data error.
  • Now, when the es simulate does not return an error, the kibana simulate api returns results for each processor
  • Now, when the es simulate api returns a pipeline compile error, the kibana simulate api returns a result for only the processor that was set as the dirty processor.
• Moved the logic that propagates api results to entire pipeline into the pipeline class
  • Previously, this logic existed in the simulate api.
• Changed the way that 'nested' errors appear.
  • Previously, when there was an error in a middle processor of the pipeline, all subsequent processors would also be set to an error state.
  • Now, when the above mentioned situation occurs, they are just set to a 'locked' state
• Added front end logic to handle when a pipeline-compile error occurs
  • Now each processor except the processor that returns the error is 'locked'
  • All ui that could alter the pipeline is locked until the pipeline-compile error occurs
    • Reordering processors
    • Adding processors
    • Removing processors (except processor that is causing the error)
    • Changing input data
• Fixed bug where the last character entered into a pipeline would appear to not be processed
  • There was a premature optimization that would cause this issue. This should now be resolved.

[Bargs]

Pipeline output becomes blank as soon as Gsub is added:

[BigFunger]

@Bargs wrote

Pipeline output becomes blank as soon as Gsub is added

This behavior was not introduced with this PR, and is something I plan on addressing in a future PR.

[Bargs]

Created a ticket to track it #6679

[Bargs]

Keys are optional by default

[Bargs]

This needs additional api tests under test/unit/api/ingest/_simulate.js to test the updated behaviour, and a new test suite for Gsub under test/unit/api/ingest/processors/_gsub

[Bargs]

Fixed bug where the last character entered into a pipeline would appear to not be processed

In the future, please split unrelated fixes like this into their own PR.

[Bargs]

This will test the set processor schema, not gsub

[Bargs]

@BigFunger I'm done reviewing the current code, passing back to you for updates. Please create a ticket for cleaning up the dirty_processor logic once ES has improved their errors, and link to the ES ticket for it.

[Bargs]

@BigFunger Will you rebase this on feature/ingest? That'll get Spencer's ES fix into the branch and should get the tests running again.

Also, does this PR block the addition of the other (non-regex based) processors? Or could I be reviewing those while you're making updates to this PR?

[BigFunger]

@Bargs, it does block the addition of the other processors, because this PR redefines the base flow of how processors and the pipeline work.

[Bargs]

@BigFunger Since elastic/elasticsearch#17260 is already merged, let's remove the dirty_processor tracking from this PR. That should fix the problem with compilation error'd processors getting locked and should clean the code up nicely.

[BigFunger]

Changes made per our discussion today and your above comments. I updated the description in the initial comment to reflect.

[Bargs]

There's a test failure in __tests__/process_es_ingest_simulate_error.js, otherwise lgtm