Deleting all recordings corrupts list of recordings

[hraftery]

I think this was triggered by issuing kapacitor delete recordings * because initially that didn't work either and the issue appeared immediately afterwards.

Now when I issue kapacitor list recordings I get:

failed to list recordings with pattern "": no key exists

I can see the following appear in kapacitor.log:

[httpd] 127.0.0.1 - - [03/Apr/2017:08:44:48 +1000] "GET /kapacitor/v1/recordings?fields=type&fields=size&fields=date&fields=status&limit=100&offset=0&pattern= HTTP/1.1" 400 92 "-" "KapacitorClient" fa11ae1f-17f5-11e7-8332-000000000000 16340

Strangely enough, I still seem to be able to record and replay recordings just fine.

[nathanielc]

@hraftery I have been able to reproduce this bug. I'll put together a fix, for now if you remove your entire kapacitor.db file and start fresh the issue will go away. Obviously that will delete all your tasks and other data as well.

[hraftery]

Great, thanks. Was just going to confirm the bug on a different installation with this simple reproduction method:

$ kapacitor list recordings
...list of recordings...
$ kapacitor delete recordings *
$ kapacitor list recordings
...list of recordings hasn't changed!...
$ kapacitor delete recordings 9*
$ kapacitor list recordings
failed to list recordings with pattern "": no key exists

Anyway to export the existing tasks ready for re-import, prior to deleting kapacitor.db?

[nathanielc]

Anyway to export the existing tasks ready for re-import, prior to deleting kapacitor.db?

First make a copy of the db file. Then it should be pretty straightforward to put together a script to download and save the JSON for each of the tasks from the API. Then you can POST that JSON back for each task (minus some read only fields). See https://docs.influxdata.com/kapacitor/v1.2/api/api/#tasks

If anything goes wrong just restore your copy of the db file and everything will be back(bug included).

[hraftery]

I can now confirm that this process works like something like this:

curl "http://127.0.0.1:9092/kapacitor/v1/tasks/task1" > task1.json
curl "http://127.0.0.1:9092/kapacitor/v1/tasks/task2" > task2.json
sudo mv /var/lib/kapacitor/kapacitor.db /var/lib/kapacitor/kapacitor.db.backup
sudo service kapacitor restart
curl -H "Content-Type: application/json" --data @task1.json http://127.0.0.1:9092/kapacitor/v1/tasks
curl -H "Content-Type: application/json" --data @task2.json http://127.0.0.1:9092/kapacitor/v1/tasks

with a couple of caveats:

1. of course, restarting kapacitor kills any UDFs you're running, so state is lost.
2. all the old recordings come back! I assume this is intended behaviour, but it does mean issuing a bunch of painful kapacitor delete recording $rid statements.

[nathanielc]

@hraftery Yes, those statements look good as a work around with the current version. PR #1296 should provide a permanent fix. The steps once that is merged and released will be:

# Perform a back just in case
kapacitor backup "kapacitor-$(date +%s).db"
# Rebuild the recordings index to fix the underlying issue
curl -X POST http://localhost:9092/kapacitor/v1/storage/stores/recordings --data-binary '{"action":"rebuild"}'
# Now you can safely delete all recordings without issue
kapacitor delete recordings \*

The issue stems from stale and invalid indexes existing in the database from previous versions of Kapacitor. The fix is to provide an API endpoint to rebuild indexes , which is what is happening above.