All databases lost after restart

[Raniz85]

Our database server restarted yesterday and when it came back online it had lost all track of the databases.

The databases are there on the disk and there is a lot of log messages about reading files from the databases, but the databases does not show up with the SHOW DATABASES command.

I've poked around and the databases aren't mentioned in meta/meta.db. I recreated the database structure on my own computer and copied meta.db to the server. That worked so far as the databases now show up with SHOW DATABASES, but none of the old data can be found via queries.

I've upgraded InfluxDB from 1.3.6 to 1.4.2 in my attempts to have it find the old databases, but that doesn't seem to have made any difference.

Logs from when InfluxDB first started after the server restart and my last attempt to fix it by copying meta.db from my computer are attached below.

InfluxDB log after server restart
InfluxDB log after copying meta.db

[jsternberg]

Unfortunately, it seems the meta information got hosed somehow. Here are some recovery steps.

# Stop the database.
$ service influxdb stop
# Perform a backup of the data, ensure you have enough space.
$ tar -C /var/lib/influxdb -czf influxdb.backup.tar.gz .
# This file is useless now.
$ rm /var/lib/influxdb/meta/meta.db
# Export the raw data to line protocol.
$ influx_inspect export -datadir /var/lib/influxdb/data -out <outfile>
# Delete the old data.
$ rm -r /var/lib/influxdb/{meta,data,wal}
# Restart the database.
$ service influxdb start
# If you have custom retention policies (such as shard duration or other ones different from the defaults), create those now.
# Then use influx -import to import the data you exported previously.
$ influx -import -path <path>

Make sure to backup the data on disk before attempting this so you can try again if something goes wrong. There is also a bug in influx -import that was very recently fixed in #9423. That fix hasn't reached a release yet. If you cannot compile it from source, you can look inside of the import file and split the file manually each time you see CONTEXT-DATABASE in the file to avoid that issue.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity. Please reopen if this issue is still important to you. Thank you for your contributions.

[tdschmidl]

# Stop the database.
$ service influxdb stop
# Perform a backup of the data, ensure you have enough space.
$ tar -C /var/lib/influxdb -czf influxdb.backup.tar.gz .
# This file is useless now.
$ rm /var/lib/influxdb/meta/meta.db
# Export the raw data to line protocol.
$ influx_inspect export -datadir /var/lib/influxdb/data -out <outfile>
# Delete the old data.
$ rm -r /var/lib/influxdb/{meta,data,wal}
# Restart the database.
$ service influxdb start
# If you have custom retention policies (such as shard duration or other ones different from the defaults), create those now.
# Then use influx -import to import the data you exported previously.
$ influx -import -path <path>

This process helped me out after powerloss of our VM host - thank you!