Enable BucketListDB by default in captive-core

[SirTyson]

What problem does your feature solve?

Recently, core added the EXPERIMENTAL_BUCKETLIST_DB flag. When this flag is set, stellar-core uses the BucketList for key value lookup for most queries instead of a SQL database. This reduces disk usage by over 50% since state does not need to be duplicated between a SQL database and the BucketList. BucketListDB also has up to 4x lookup performance compared to SQL. Finally, startup is significantly faster. Because we query buckets directly, we can skip most BucketApply logic since we no longer need to apply buckets to a SQL database.

While this feature is still hidden behind an experimental flag, we think it's ready to be turned on by default when captive-core is using on disk mode. BucketListDB is not currently compatible in in-memory mode.

What would you like to see?

We would like to add EXPERIMENTAL_BUCKETLIST_DB=true to the default captive-core configuration when Horizon is run with the --captive-core-use-db flag. In addition to enabling BucketListDB, we will also need to tune some low-level optimization parameters for captive-core. There are two configuration parameters we will need to tune to the Horizon SKU, EXPERIMENTAL_BUCKETLIST_DB_INDEX_PAGE_SIZE_EXPONENT and EXPERIMENTAL_BUCKETLIST_DB_INDEX_CUTOFF. EXPERIMENTAL_BUCKETLIST_DB_INDEX_PAGE_SIZE_EXPONENT should be set to the machine storage device's physical page size. Currently, the default page size is 16KB, which is the optimal value for AWS NVMe storage. If Horizon nodes typically use a different storage class like EBS, this value will have to be changed. Unfortunately, during testing we found that page sizes reported by the OS are incorrect on AWS. The best way to figure out this value is just by trial and error with our microbenchmarking tool Tracy.
EXPERIMENTAL_BUCKETLIST_DB_INDEX_CUTOFF controls how much memory BucketListDB uses. A higher EXPERIMENTAL_BUCKETLIST_DB_INDEX_CUTOFF means more RAM usage but faster lookup performance.

I'm currently working on getting a Horizon SKU test machine running where I can fine tune these values. In the mean time, I'll put out a PR so anyone else can also run a captive-core test, but it should not be merged until we find the optimal configuration settings.