agent: failed to dispatch actions, error: acknowledge 12 actions

[mtojek]

I got the agent re-enrolled, but couldn't manage to run it:

2020-04-01T12:13:10+02:00 INFO	stateresolver.go:47	New State ID is ZNxNzneq
2020-04-01T12:13:10+02:00 INFO	stateresolver.go:48	Converging state requires execution of 0 step(s)
2020-04-01T12:13:10+02:00 INFO	stateresolver.go:61	Updating internal state
2020-04-01T12:13:10+02:00 DEBUG	action_dispatcher.go:92	Successfully dispatched action: 'action_id: b8acb674-589e-489e-9005-2fcc06a2f4ec, type: CONFIG_CHANGE'
2020-04-01T12:13:10+02:00 DEBUG	client.go:170	Request method: POST, path: /api/ingest_manager/fleet/agents/454f5dc0-7401-11ea-ab82-e145acb79802/acks
2020-04-01T12:13:10+02:00 ERROR	fleet_gateway.go:143	failed to dispatch actions, error: acknowledge 12 actions '[action_id: 8e5d0a3d-9c72-41fd-b1fd-51a053194a32, type: CONFIG_CHANGE action_id: b8acb674-589e-489e-9005-2fcc06a2f4ec, type: CONFIG_CHANGE action_id: b8acb674-589e-489e-9005-2fcc06a2f4ec, type: CONFIG_CHANGE action_id: b8acb674-589e-489e-9005-2fcc06a2f4ec, type: CONFIG_CHANGE action_id: b8acb674-589e-489e-9005-2fcc06a2f4ec, type: CONFIG_CHANGE action_id: b8acb674-589e-489e-9005-2fcc06a2f4ec, type: CONFIG_CHANGE action_id: b8acb674-589e-489e-9005-2fcc06a2f4ec, type: CONFIG_CHANGE action_id: b8acb674-589e-489e-9005-2fcc06a2f4ec, type: CONFIG_CHANGE action_id: b8acb674-589e-489e-9005-2fcc06a2f4ec, type: CONFIG_CHANGE action_id: b8acb674-589e-489e-9005-2fcc06a2f4ec, type: CONFIG_CHANGE action_id: b8acb674-589e-489e-9005-2fcc06a2f4ec, type: CONFIG_CHANGE action_id: b8acb674-589e-489e-9005-2fcc06a2f4ec, type: CONFIG_CHANGE]' for agent '454f5dc0-7401-11ea-ab82-e145acb79802' failed: Status code: 400, Kibana returned an error: Bad Request, message: all actions should belong to current agent
	acknowledge 12 actions '[action_id: 8e5d0a3d-9c72-41fd-b1fd-51a053194a32, type: CONFIG_CHANGE action_id: b8acb674-589e-489e-9005-2fcc06a2f4ec, type: CONFIG_CHANGE action_id: b8acb674-589e-489e-9005-2fcc06a2f4ec, type: CONFIG_CHANGE action_id: b8acb674-589e-489e-9005-2fcc06a2f4ec, type: CONFIG_CHANGE action_id: b8acb674-589e-489e-9005-2fcc06a2f4ec, type: CONFIG_CHANGE action_id: b8acb674-589e-489e-9005-2fcc06a2f4ec, type: CONFIG_CHANGE action_id: b8acb674-589e-489e-9005-2fcc06a2f4ec, type: CONFIG_CHANGE action_id: b8acb674-589e-489e-9005-2fcc06a2f4ec, type: CONFIG_CHANGE action_id: b8acb674-589e-489e-9005-2fcc06a2f4ec, type: CONFIG_CHANGE action_id: b8acb674-589e-489e-9005-2fcc06a2f4ec, type: CONFIG_CHANGE action_id: b8acb674-589e-489e-9005-2fcc06a2f4ec, type: CONFIG_CHANGE action_id: b8acb674-589e-489e-9005-2fcc06a2f4ec, type: CONFIG_CHANGE]' for agent '454f5dc0-7401-11ea-ab82-e145acb79802' failed: Status code: 400, Kibana returned an error: Bad Request, message: all actions should belong to current agent
2020-04-01T12:13:10+02:00 DEBUG	fleet_gateway.go:146	FleetGateway is sleeping, next update in 2s
2020-04-01T12:13:12+02:00 DEBUG	fleet_gateway.go:126	FleetGateway calling Checkin API

Dispatch actions got accumulated.

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ph]

Taking a look

[nchaulet]

@ph I go a similar error when I renrolled an agent, I think is trying to ack actions from a previous enrollment it is possible?

[ph]

Forgot to update this, I can reproduce it quite easily that issue. We save the last good action from fleet and replay it. When you reenroll we do not correctly clean the local state, so when the agent is up again you cannot restore the saved states.

[ph]

I am close to have a fix for this, in the mean time delete the action_store.yml if you have this issue.

[ph]

With the change with RPC this will change.

[EricDavisX]

I don't see a referenced RPC change PR or any timeframe, can we confirm if we intend this to be fixed for Alpha? It seems it will be a high pain point, but maybe we can live with it if we must - if not fixed, can we get a few notes added to the below doc how we can manually rectify the agent locally?
https://docs.google.com/document/d/1_HhrkWlJVYNZo3L49iuvki7wWCQS72VUiogKQxLsuck/edit#heading=h.uftks4gnnbvm

[michalpristas]

created a PR to get it fixed before RPC restruct