Include size of snapshot in snapshot metadata

[robin13]

Feature: add snapshot size to snapshot metadata.

While the snapshot size (as a snapshot) on disk may not have much relevance because of the additive nature of snapshots (segments may already exist in the repository, so data transferred may be less than the total, and deleting a snapshot may not release all of the disk space because some segments may be retained for other snapshots...), it should be easy to capture the total size of the snapshot as it will require disk space to be restored, and this is an information which can be very useful when trying to determine before executing a restore, whether there is enough disk space on the target cluster.

[luizgpsantos]

It seems that the snapshot status endpoint exposes this information in stats.total_size field:

GET /_snapshot/my_backup/snapshot_1/_status?human

[ywelsch]

@luizgpsantos that's different, that's just the size of the increment.

@tlrx and I looked at the code and think this should be simple to compute and add to the stats.

[vladimirdolzhenko]

There are properties total_size_in_bytes and processed_size_in_bytes in snapshot stats.

In fact total_size_in_bytes shows difference size between previous snapshot and current one, processed_size_in_bytes shows the progress of making snapshot - so it in in a range between 0 and total_size_in_bytes.

If we add another property, e.g. accumulated_size_in_bytes - it will confuse usage even more. Instead of that I'd like to break the semantic of total_size_in_bytes - it will reflect indeed total size of snapshot, while another property diff_size will reflect difference size between previous snapshot and current one (processed_size_in_bytes will be in a range 0 and diff_size)

Another question: is it worth to align number_of_files to be consistent with total_size_in_bytes and diff_size - it will require adding one more extra property like diff_number_of_files ?

[vladimirdolzhenko]

To summarize the change:

stats (for op like GET /_snapshot/backup1/snapshot_2/_status?human) looks

before:

"stats": {
"number_of_files": 8,
"processed_files": 8,
"total_size": "4.6kb",
"total_size_in_bytes": 4797,
"processed_size": "4.6kb",
"processed_size_in_bytes": 4797,
"start_time_in_millis": 1523882347327,
"time": "225ms",
"time_in_millis": 225
}

after:

"stats": {
"incremental_file_count": 8,
"total_file_count": 11,
"processed_file_count": 8,
"incremental_size": "4.6kb",
"incremental_size_in_bytes": 4797,
"total_size": "8kb",
"total_size_in_bytes": 8276,
"processed_size": "4.6kb",
"processed_size_in_bytes": 4797,
"start_time_in_millis": 1523882347327,
"time": "225ms",
"time_in_millis": 225
}