Skip to content

Commit

Permalink
[PLAT-8859] Refactoring backup controller for create and restore to a…
Browse files Browse the repository at this point in the history 
…dd helper methods that allow calling from non API context.

Summary:
Refactoring backup controller for create and restore to add helper methods that allow calling from non API ocntext.
This will make calling these functions from the backup reconciler much easier and allow us to not remimplement this code.
This will also keep backup/restore functionality consistent between operator and the API clients.

Test Plan:
itest
build

Fixed Failing UT

```
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[info] Test com.yugabyte.yw.common.backuprestore.BackupHelperTest.testRestorePreflightWithoutBackupObjectNFSNonYBC(false, true) [0] started
[info] Test com.yugabyte.yw.common.backuprestore.BackupHelperTest.testRestorePreflightWithoutBackupObjectNFSNonYBC(true, false) [1] started
[info] Test com.yugabyte.yw.common.backuprestore.BackupHelperTest.testRestorePreflightWithoutBackupObjectNFSNonYBC(true, true) [2] started
[info] Test com.yugabyte.yw.common.backuprestore.BackupHelperTest.testRestorePreflightWithoutBackupObjectNFSNonYBC(false, false) [3] started
[info] Test com.yugabyte.yw.common.backuprestore.BackupHelperTest.testValidateBackupRequestYCQLSameKeyspaceInvalid started
[info] Test com.yugabyte.yw.common.backuprestore.BackupHelperTest.testValidateBackupRequestYCQLSameKeyspaceValid started
[info] Test com.yugabyte.yw.common.backuprestore.BackupHelperTest.testValidateBackupRequestYCQLDifferentKeyspaceValid started
[info] Test com.yugabyte.yw.common.backuprestore.BackupHelperTest.testRestorePreflightResponseWithYSQLBackupObjectExists(true, YB_CONTROLLER) [0] started
[info] Test com.yugabyte.yw.common.backuprestore.BackupHelperTest.testRestorePreflightResponseWithYSQLBackupObjectExists(false, YB_CONTROLLER) [1] started
[info] Test com.yugabyte.yw.common.backuprestore.BackupHelperTest.testRestorePreflightResponseWithYSQLBackupObjectExists(true, YB_BACKUP_SCRIPT) [2] started
[info] Test com.yugabyte.yw.common.backuprestore.BackupHelperTest.testRestorePreflightResponseWithYSQLBackupObjectExists(false, YB_BACKUP_SCRIPT) [3] started
[info] Test com.yugabyte.yw.common.backuprestore.BackupHelperTest.testRestorePreflightWithoutBackupObjectYBCWithYCQLIndexTables(backup/ybc_success_file_with_index_tables.json, true, false) [0] started
[info] Test com.yugabyte.yw.common.backuprestore.BackupHelperTest.testRestorePreflightWithoutBackupObjectYBCWithYCQLIndexTables(backup/ybc_success_file_with_index_tables.json, true, true) [1] started
[info] Test com.yugabyte.yw.common.backuprestore.BackupHelperTest.testRestorePreflightWithoutBackupObjectYBCWithYCQLIndexTables(backup/ybc_success_file_with_index_tables.json, false, true) [2] started
[info] Test com.yugabyte.yw.common.backuprestore.BackupHelperTest.testRestorePreflightWithoutBackupObjectYBCWithYCQLIndexTables(backup/ybc_success_file_with_index_tables.json, false, false) [3] started
[info] Test com.yugabyte.yw.common.backuprestore.BackupHelperTest.testRestorePreflightWithoutBackupObjectS3NonYBC(false, true) [0] started
[info] Test com.yugabyte.yw.common.backuprestore.BackupHelperTest.testRestorePreflightWithoutBackupObjectS3NonYBC(true, false) [1] started
[info] Test com.yugabyte.yw.common.backuprestore.BackupHelperTest.testRestorePreflightWithoutBackupObjectS3NonYBC(true, true) [2] started
[info] Test com.yugabyte.yw.common.backuprestore.BackupHelperTest.testRestorePreflightWithoutBackupObjectS3NonYBC(false, false) [3] started
[info] Test com.yugabyte.yw.common.backuprestore.BackupHelperTest.testValidateMapToRestoreWithUniverseNonRedisYBC_OverwriteYCQL started
[info] Test com.yugabyte.yw.common.backuprestore.BackupHelperTest.testValidateMapToRestoreWithUniverseNonRedisYBC_OverwriteYSQL started
[info] Test com.yugabyte.yw.common.backuprestore.BackupHelperTest.testRestorePreflightResponseWithYCQLBackupObjectExists(true, YB_CONTROLLER) [0] started
[info] Test com.yugabyte.yw.common.backuprestore.BackupHelperTest.testRestorePreflightResponseWithYCQLBackupObjectExists(false, YB_CONTROLLER) [1] started
[info] Test com.yugabyte.yw.common.backuprestore.BackupHelperTest.testRestorePreflightResponseWithYCQLBackupObjectExists(true, YB_BACKUP_SCRIPT) [2] started
[info] Test com.yugabyte.yw.common.backuprestore.BackupHelperTest.testRestorePreflightResponseWithYCQLBackupObjectExists(false, YB_BACKUP_SCRIPT) [3] started
[info] Test com.yugabyte.yw.common.backuprestore.BackupHelperTest.testValidateMapToRestoreWithUniverseNonRedisYBC_NoOverwriteYCQL started
[info] Test com.yugabyte.yw.common.backuprestore.BackupHelperTest.testValidateMapToRestoreWithUniverseNonRedisYBC_NoOverwriteYSQL started
[info] Test run finished: 0 failed, 0 ignored, 27 total, 86.92s
[info] Passed: Total 27, Failed 0, Errors 0, Passed 27
```

Reviewers: #yba-api-review, hzare

Reviewed By: #yba-api-review, hzare

Subscribers: sneelakantan, hzare, yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D27207
  • Loading branch information
@amannijhawan
amannijhawan committed Jul 26, 2023
1 parent 404d4c5 commit a68e98c
Show file tree
Hide file tree
Showing 5 changed files with 267 additions and 201 deletions.
1 change: 1 addition & 0 deletions managed/build.sbt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,7 @@ libraryDependencies ++= Seq(
"com.squareup.okhttp3" % "mockwebserver" % "4.9.2" % Test,
"io.grpc" % "grpc-testing" % "1.48.0" % Test,
"io.zonky.test" % "embedded-postgres" % "2.0.1" % Test,
"org.springframework" % "spring-test" % "5.3.9" % Test,
)

// Clear default resolvers.
Expand Down
212 changes: 211 additions & 1 deletion managed/src/main/java/com/yugabyte/yw/common/backuprestore/BackupHelper.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,15 @@
import static com.yugabyte.yw.common.Util.getUUIDRepresentation;
import static java.lang.Math.max;
import static play.mvc.Http.Status.BAD_REQUEST;
import static play.mvc.Http.Status.CONFLICT;
import static play.mvc.Http.Status.INTERNAL_SERVER_ERROR;
import static play.mvc.Http.Status.PRECONDITION_FAILED;

import com.google.common.collect.Lists;
import com.google.inject.Inject;
import com.google.inject.Singleton;
import com.yugabyte.yw.commissioner.Commissioner;
import com.yugabyte.yw.commissioner.tasks.subtasks.DeleteBackupYb;
import com.yugabyte.yw.common.PlatformServiceException;
import com.yugabyte.yw.common.StorageUtil;
import com.yugabyte.yw.common.StorageUtilFactory;
Expand All @@ -21,17 +24,28 @@
import com.yugabyte.yw.common.config.UniverseConfKeys;
import com.yugabyte.yw.common.customer.config.CustomerConfigService;
import com.yugabyte.yw.common.services.YBClientService;
import com.yugabyte.yw.forms.BackupRequestParams;
import com.yugabyte.yw.forms.BackupRequestParams.KeyspaceTable;
import com.yugabyte.yw.forms.BackupTableParams;
import com.yugabyte.yw.forms.DeleteBackupParams;
import com.yugabyte.yw.forms.DeleteBackupParams.DeleteBackupInfo;
import com.yugabyte.yw.forms.PlatformResults.YBPTask;
import com.yugabyte.yw.forms.RestoreBackupParams;
import com.yugabyte.yw.forms.RestoreBackupParams.BackupStorageInfo;
import com.yugabyte.yw.forms.RestorePreflightParams;
import com.yugabyte.yw.forms.RestorePreflightResponse;
import com.yugabyte.yw.models.Backup;
import com.yugabyte.yw.models.Backup.BackupCategory;
import com.yugabyte.yw.models.Backup.BackupState;
import com.yugabyte.yw.models.Customer;
import com.yugabyte.yw.models.CustomerTask;
import com.yugabyte.yw.models.Universe;
import com.yugabyte.yw.models.configs.CustomerConfig;
import com.yugabyte.yw.models.configs.CustomerConfig.ConfigState;
import com.yugabyte.yw.models.configs.data.CustomerConfigData;
import com.yugabyte.yw.models.configs.data.CustomerConfigStorageData;
import com.yugabyte.yw.models.helpers.TaskType;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
Expand All @@ -42,6 +56,7 @@
import java.util.Set;
import java.util.UUID;
import java.util.function.Function;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.collections4.CollectionUtils;
Expand All @@ -58,25 +73,33 @@
@Slf4j
@Singleton
public class BackupHelper {
private static final String VALID_OWNER_REGEX = "^[\\pL_][\\pL\\pM_0-9]*$";

private YbcManager ybcManager;
private YBClientService ybClientService;
private CustomerConfigService customerConfigService;
private RuntimeConfGetter confGetter;
private StorageUtilFactory storageUtilFactory;
@Inject Commissioner commissioner;

@Inject
public BackupHelper(
YbcManager ybcManager,
YBClientService ybClientService,
CustomerConfigService customerConfigService,
RuntimeConfGetter confGetter,
StorageUtilFactory storageUtilFactory) {
StorageUtilFactory storageUtilFactory,
Commissioner commisssioner) {
this.ybcManager = ybcManager;
this.ybClientService = ybClientService;
this.customerConfigService = customerConfigService;
this.confGetter = confGetter;
this.storageUtilFactory = storageUtilFactory;
// this.commissioner = commissioner;
}

public String getValidOwnerRegex() {
return VALID_OWNER_REGEX;
}

public void validateIncrementalScheduleFrequency(
Expand All @@ -100,6 +123,193 @@ public void validateIncrementalScheduleFrequency(
}
}

public UUID createBackupTask(UUID customerUUID, BackupRequestParams taskParams) {
Customer customer = Customer.getOrBadRequest(customerUUID);
// Validate universe UUID
Universe universe = Universe.getOrBadRequest(taskParams.getUniverseUUID(), customer);
taskParams.customerUUID = customerUUID;

if (universe
.getConfig()
.getOrDefault(Universe.TAKE_BACKUPS, "true")
.equalsIgnoreCase("false")) {
throw new PlatformServiceException(BAD_REQUEST, "Taking backups on the universe is disabled");
}

if (universe.getUniverseDetails().updateInProgress) {
throw new PlatformServiceException(
CONFLICT,
String.format(
"Cannot run Backup task since the universe %s is currently in a locked state.",
taskParams.getUniverseUUID().toString()));
}
if ((universe.getLiveTServersInPrimaryCluster().size() < taskParams.parallelDBBackups)
|| taskParams.parallelDBBackups <= 0) {
throw new PlatformServiceException(
BAD_REQUEST,
String.format(
"invalid parallel backups value provided for universe %s",
universe.getUniverseUUID()));
}
validateBackupRequest(taskParams.keyspaceTableList, universe, taskParams.backupType);

if (taskParams.timeBeforeDelete != 0L && taskParams.expiryTimeUnit == null) {
throw new PlatformServiceException(BAD_REQUEST, "Please provide time unit for backup expiry");
}

if (taskParams.storageConfigUUID == null) {
throw new PlatformServiceException(
BAD_REQUEST, "Missing StorageConfig UUID: " + taskParams.storageConfigUUID);
}
CustomerConfig customerConfig =
customerConfigService.getOrBadRequest(customerUUID, taskParams.storageConfigUUID);

if (!customerConfig.getState().equals(ConfigState.Active)) {
throw new PlatformServiceException(
BAD_REQUEST, "Cannot create backup as config is queued for deletion.");
}
if (taskParams.baseBackupUUID != null) {
Backup previousBackup =
Backup.getLastSuccessfulBackupInChain(customerUUID, taskParams.baseBackupUUID);
if (!universe.isYbcEnabled()) {
throw new PlatformServiceException(
BAD_REQUEST, "Incremental backup not allowed for non-YBC universes");
} else if (previousBackup == null) {
throw new PlatformServiceException(
BAD_REQUEST, "No previous successful backup found, please trigger a new base backup.");
}
validateStorageConfigOnBackup(customerConfig, previousBackup);
} else {
validateStorageConfig(customerConfig);
}
UUID taskUUID = commissioner.submit(TaskType.CreateBackup, taskParams);
log.info("Submitted task to universe {}, task uuid = {}.", universe.getName(), taskUUID);
CustomerTask.create(
customer,
taskParams.getUniverseUUID(),
taskUUID,
CustomerTask.TargetType.Backup,
CustomerTask.TaskType.Create,
universe.getName());
log.info("Saved task uuid {} in customer tasks for universe {}", taskUUID, universe.getName());
return taskUUID;
}

public UUID createRestoreTask(UUID customerUUID, RestoreBackupParams taskParams) {
Customer customer = Customer.getOrBadRequest(customerUUID);
taskParams.backupStorageInfoList.forEach(
bSI -> {
if (StringUtils.isNotBlank(bSI.newOwner)
&& !Pattern.matches(VALID_OWNER_REGEX, bSI.newOwner)) {
throw new PlatformServiceException(
BAD_REQUEST, "Invalid owner rename during restore operation");
}
});

taskParams.customerUUID = customerUUID;
taskParams.prefixUUID = UUID.randomUUID();
UUID universeUUID = taskParams.getUniverseUUID();
Universe universe = Universe.getOrBadRequest(universeUUID, customer);
if (CollectionUtils.isEmpty(taskParams.backupStorageInfoList)) {
throw new PlatformServiceException(BAD_REQUEST, "Backup information not provided");
}
validateRestoreOverwrites(taskParams.backupStorageInfoList, universe, taskParams.category);
CustomerConfig customerConfig =
customerConfigService.getOrBadRequest(customerUUID, taskParams.storageConfigUUID);
if (!customerConfig.getState().equals(ConfigState.Active)) {
throw new PlatformServiceException(
BAD_REQUEST, "Cannot restore backup as config is queued for deletion.");
}
// Even though we check with default location below, this is needed to validate
// regional locations, because their validity is not known to us when we send restore
// request with a config.
validateStorageConfig(customerConfig);
CustomerConfigStorageData configData =
(CustomerConfigStorageData) customerConfig.getDataObject();

storageUtilFactory
.getStorageUtil(customerConfig.getName())
.validateStorageConfigOnLocationsList(
configData,
taskParams
.backupStorageInfoList
.parallelStream()
.map(bSI -> bSI.storageLocation)
.collect(Collectors.toSet()));

if (taskParams.category.equals(BackupCategory.YB_CONTROLLER) && !universe.isYbcEnabled()) {
throw new PlatformServiceException(
BAD_REQUEST, "Cannot restore the ybc backup as ybc is not installed on the universe");
}
UUID taskUUID = commissioner.submit(TaskType.RestoreBackup, taskParams);
CustomerTask.create(
customer,
universeUUID,
taskUUID,
CustomerTask.TargetType.Universe,
CustomerTask.TaskType.Restore,
universe.getName());
return taskUUID;
}

public List<YBPTask> createDeleteBackupTasks(
UUID customerUUID, DeleteBackupParams deleteBackupParams) {
Customer customer = Customer.getOrBadRequest(customerUUID);
List<YBPTask> taskList = new ArrayList<>();
for (DeleteBackupInfo deleteBackupInfo : deleteBackupParams.deleteBackupInfos) {
UUID backupUUID = deleteBackupInfo.backupUUID;
Backup backup = Backup.maybeGet(customerUUID, backupUUID).orElse(null);
if (backup == null) {
log.error("Can not delete {} backup as it is not present in the database.", backupUUID);
} else {
if (Backup.IN_PROGRESS_STATES.contains(backup.getState())) {
log.error(
"Backup {} is in the state {}. Deletion is not allowed",
backupUUID,
backup.getState());
} else {
UUID storageConfigUUID = deleteBackupInfo.storageConfigUUID;
if (storageConfigUUID == null) {
// Pick default backup storage config to delete the backup if not provided.
storageConfigUUID = backup.getBackupInfo().storageConfigUUID;
}
if (backup.isIncrementalBackup() && backup.getState().equals(BackupState.Completed)) {
// Currently, we don't allow users to delete successful standalone incremental backups.
// They can only delete the full backup, along which all the incremental backups
// will also be deleted.
log.error(
"Cannot delete backup {} as it in {} state",
backup.getBackupUUID(),
backup.getState());
continue;
}
BackupTableParams params = backup.getBackupInfo();
params.storageConfigUUID = storageConfigUUID;
backup.updateBackupInfo(params);
DeleteBackupYb.Params taskParams = new DeleteBackupYb.Params();
taskParams.customerUUID = customerUUID;
taskParams.backupUUID = backupUUID;
taskParams.deleteForcefully = deleteBackupParams.deleteForcefully;
UUID taskUUID = commissioner.submit(TaskType.DeleteBackupYb, taskParams);
log.info("Saved task uuid {} in customer tasks for backup {}.", taskUUID, backupUUID);
String target =
!StringUtils.isEmpty(backup.getUniverseName())
? backup.getUniverseName()
: String.format("univ-%s", backup.getUniverseUUID().toString());
CustomerTask.create(
customer,
backup.getUniverseUUID(),
taskUUID,
CustomerTask.TargetType.Backup,
CustomerTask.TaskType.Delete,
target);
taskList.add(new YBPTask(taskUUID, taskParams.backupUUID));
}
}
}
return taskList;
}

public void validateStorageConfigOnBackup(Backup backup) {
CustomerConfig config =
customerConfigService.getOrBadRequest(
Expand Down
Loading

0 comments on commit a68e98c

Please sign in to comment.