Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 Search Improvements #1391

Merged
merged 26 commits into from
Jan 31, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
57f3ffe
allows setting storage class at upload time
wsorenson Dec 14, 2016
e26e8dc
allow each additional file to specify s3 storage class
wsorenson Dec 14, 2016
391295c
add debug stmt
wsorenson Dec 14, 2016
bb55ae8
fix merge conflicts
ssalinas Jan 6, 2017
13d2fe9
Move S3 file configuration to SingularityService
ssalinas Jan 4, 2017
84e112b
jets3t -> aws sdk
ssalinas Jan 9, 2017
fcde16e
use int here
ssalinas Jan 9, 2017
9db459b
Merge pull request #1394 from HubSpot/jets_to_aws
ssalinas Jan 9, 2017
0c9659a
cleanup needs to know defaults for transition to new format
ssalinas Jan 10, 2017
508ee73
cleaner check for default service log name
ssalinas Jan 10, 2017
6811cb0
more working on getting executor cleanup to run proerply
ssalinas Jan 10, 2017
3e0d3cf
this should be in cleanup configuration now s3
ssalinas Jan 10, 2017
ace58e7
add support for extra buckets to search for logs
ssalinas Jan 11, 2017
74e07a9
Implement pagination for S3 endpoint using continuation tokens
ssalinas Jan 9, 2017
8242609
Merge pull request #1395 from HubSpot/s3_pagination
ssalinas Jan 11, 2017
8d52018
Merge branch 's3_rework' of github.com:HubSpot/Singularity into s3_re…
ssalinas Jan 11, 2017
8861141
search the additional buckets when present
ssalinas Jan 11, 2017
c04a98e
do not search prefixes that will return duplicate results
ssalinas Jan 11, 2017
95d40fd
don't need group here after all
ssalinas Jan 17, 2017
ddb6e43
fix prefix trim
ssalinas Jan 17, 2017
9de457d
need to use length + startsWith, not contains
ssalinas Jan 17, 2017
8d157e3
multipart upload needs key as well
ssalinas Jan 18, 2017
6fb4c2c
Also add file arg
ssalinas Jan 18, 2017
5b23fe9
Set metadata here as well
ssalinas Jan 20, 2017
d426dea
user metadata is separate
ssalinas Jan 23, 2017
5e6d768
add docs for s3 changes
ssalinas Jan 31, 2017
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions Docs/releases/0.14.0.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
## Changes in `0.14.0`

Check out the [0.14.0 milestone](https://github.com/HubSpot/Singularity/issues?q=milestone%3A0.14.0+is%3Aclosed) to see new features / bugfixes in detail.

## Configuration Changes

[#1391](https://github.com/HubSpot/Singularity/pull/1391) include a rework of some of the S3 settings in Singularity. If you use the `SingularityExecutor` or `SingularityExecutorCleanup` modules and use the S3 upload features, you will need an update to your configuration. The fields for specifying which files to upload have been moved out of the `SingularityExecutor`. An example below shows all fields that would move.

Old Configuration (gets removed from `SingularityExecutor` and `SingularityExecutorCleanup` yaml files)
```yaml
executor:
s3UploaderBucket: my-logs-bucket
s3UploaderKeyPattern: "%requestId/%Y/%m/%taskId_%index-%s-%filename"
s3UploaderAdditionalFiles:
- access.log
s3StorageClass: "STANDARD_IA"
applyS3StorageClassAfterBytes: 75000
```

New Configuration (if not already present for use with S3 log searching)
```yaml
# in SingularityExecutorCleanup yaml configuration
executorCleanup:
defaultS3Bucket: my-logs-bucket
s3KeyFormat: "%requestId/%Y/%m/%taskId_%index-%s-%filename"
s3StorageClass: "STANDARD_IA"
applyS3StorageClassAfterBytes: 75000
s3UploaderAdditionalFiles:
- filename: access.log
# The default directory in the executor was set to 'logs', now it must be manually specified
# If not specified, the directory to search for log files will be the task app directory in the sandbox
directory: logs

# in SingularityService yaml configuration
s3:
s3Bucket: my-logs-bucket
s3KeyFormat: "%requestId/%Y/%m/%taskId_%index-%s-%filename"
s3StorageClass: "STANDARD_IA"
applyS3StorageClassAfterBytes: 75000
s3UploaderAdditionalFiles:
- filename: access.log
# The default directory in the executor was set to 'logs', now it must be manually specified
# If not specified, the directory to search for log files will be the task app directory in the sandbox
directory: logs
```

**NOTE** - To upgrade smoothly, it is strongly recommended to deploy `SingularityService` and the `SingularityExecutorCleanup` *before* deploying the `SingularityExecutor`
13 changes: 2 additions & 11 deletions SingularityBase/src/main/java/com/hubspot/deploy/ExecutorData.java
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@ public class ExecutorData {
private final Optional<Long> sigKillProcessesAfterMillis;
private final Optional<Integer> maxTaskThreads;
private final Optional<Boolean> preserveTaskSandboxAfterFinish;
private final Optional<String> loggingS3Bucket;
private final Optional<Integer> maxOpenFiles;
private final Optional<Boolean> skipLogrotateAndCompress;
private final Optional<List<S3ArtifactSignature>> s3ArtifactSignatures;
Expand All @@ -39,8 +38,7 @@ public ExecutorData(@JsonProperty("cmd") String cmd, @JsonProperty("embeddedArti
@JsonProperty("s3Artifacts") List<S3Artifact> s3Artifacts, @JsonProperty("successfulExitCodes") List<Integer> successfulExitCodes, @JsonProperty("user") Optional<String> user,
@JsonProperty("runningSentinel") Optional<String> runningSentinel, @JsonProperty("extraCmdLineArgs") List<String> extraCmdLineArgs, @JsonProperty("loggingTag") Optional<String> loggingTag,
@JsonProperty("loggingExtraFields") Map<String, String> loggingExtraFields, @JsonProperty("sigKillProcessesAfterMillis") Optional<Long> sigKillProcessesAfterMillis,
@JsonProperty("maxTaskThreads") Optional<Integer> maxTaskThreads, @JsonProperty("preserveTaskSandboxAfterFinish") Optional<Boolean> preserveTaskSandboxAfterFinish,
@JsonProperty("loggingS3Bucket") Optional<String> loggingS3Bucket, @JsonProperty("maxOpenFiles") Optional<Integer> maxOpenFiles,
@JsonProperty("maxTaskThreads") Optional<Integer> maxTaskThreads, @JsonProperty("preserveTaskSandboxAfterFinish") Optional<Boolean> preserveTaskSandboxAfterFinish, @JsonProperty("maxOpenFiles") Optional<Integer> maxOpenFiles,
@JsonProperty("skipLogrotateAndCompress") Optional<Boolean> skipLogrotateAndCompress, @JsonProperty("s3ArtifactSignatures") Optional<List<S3ArtifactSignature>> s3ArtifactSignatures,
@JsonProperty("logrotateFrequency") Optional<SingularityExecutorLogrotateFrequency> logrotateFrequency) {
this.cmd = cmd;
Expand All @@ -56,7 +54,6 @@ public ExecutorData(@JsonProperty("cmd") String cmd, @JsonProperty("embeddedArti
this.sigKillProcessesAfterMillis = sigKillProcessesAfterMillis;
this.maxTaskThreads = maxTaskThreads;
this.preserveTaskSandboxAfterFinish = preserveTaskSandboxAfterFinish;
this.loggingS3Bucket = loggingS3Bucket;
this.maxOpenFiles = maxOpenFiles;
this.skipLogrotateAndCompress = skipLogrotateAndCompress;
this.s3ArtifactSignatures = s3ArtifactSignatures;
Expand All @@ -65,7 +62,7 @@ public ExecutorData(@JsonProperty("cmd") String cmd, @JsonProperty("embeddedArti

public ExecutorDataBuilder toBuilder() {
return new ExecutorDataBuilder(cmd, embeddedArtifacts, externalArtifacts, s3Artifacts, successfulExitCodes, runningSentinel, user, extraCmdLineArgs, loggingTag,
loggingExtraFields, sigKillProcessesAfterMillis, maxTaskThreads, preserveTaskSandboxAfterFinish, loggingS3Bucket, maxOpenFiles, skipLogrotateAndCompress, s3ArtifactSignatures, logrotateFrequency);
loggingExtraFields, sigKillProcessesAfterMillis, maxTaskThreads, preserveTaskSandboxAfterFinish, maxOpenFiles, skipLogrotateAndCompress, s3ArtifactSignatures, logrotateFrequency);
}

@ApiModelProperty(required=true, value="Command for the custom executor to run")
Expand Down Expand Up @@ -133,11 +130,6 @@ public Optional<Boolean> getPreserveTaskSandboxAfterFinish() {
return preserveTaskSandboxAfterFinish;
}

@ApiModelProperty(required=false, value="Override the default bucket used by the S3Uploader to store log files")
public Optional<String> getLoggingS3Bucket() {
return loggingS3Bucket;
}

@ApiModelProperty(required=false, value="Maximum number of open files the task process is allowed")
public Optional<Integer> getMaxOpenFiles() {
return maxOpenFiles;
Expand Down Expand Up @@ -174,7 +166,6 @@ public String toString() {
.add("sigKillProcessesAfterMillis", sigKillProcessesAfterMillis)
.add("maxTaskThreads", maxTaskThreads)
.add("preserveTaskSandboxAfterFinish", preserveTaskSandboxAfterFinish)
.add("loggingS3Bucket", loggingS3Bucket)
.add("maxOpenFiles", maxOpenFiles)
.add("skipLogrotateAndCompress", skipLogrotateAndCompress)
.add("s3ArtifactSignatures", s3ArtifactSignatures)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,15 +21,14 @@ public class ExecutorDataBuilder {
private Optional<Long> sigKillProcessesAfterMillis;
private Optional<Integer> maxTaskThreads;
private Optional<Boolean> preserveTaskSandboxAfterFinish;
private Optional<String> loggingS3Bucket;
private Optional<Integer> maxOpenFiles;
private Optional<Boolean> skipLogrotateAndCompress;
private Optional<List<S3ArtifactSignature>> s3ArtifactSignatures;
private Optional<SingularityExecutorLogrotateFrequency> logrotateFrequency;

public ExecutorDataBuilder(String cmd, List<EmbeddedArtifact> embeddedArtifacts, List<ExternalArtifact> externalArtifacts, List<S3Artifact> s3Artifacts, List<Integer> successfulExitCodes,
Optional<String> runningSentinel, Optional<String> user, List<String> extraCmdLineArgs, Optional<String> loggingTag, Map<String, String> loggingExtraFields,
Optional<Long> sigKillProcessesAfterMillis, Optional<Integer> maxTaskThreads, Optional<Boolean> preserveTaskSandboxAfterFinish, Optional<String> loggingS3Bucket,
Optional<Long> sigKillProcessesAfterMillis, Optional<Integer> maxTaskThreads, Optional<Boolean> preserveTaskSandboxAfterFinish,
Optional<Integer> maxOpenFiles, Optional<Boolean> skipLogrotateAndCompress, Optional<List<S3ArtifactSignature>> s3ArtifactSignatures, Optional<SingularityExecutorLogrotateFrequency> logrotateFrequency) {
this.cmd = cmd;
this.embeddedArtifacts = embeddedArtifacts;
Expand All @@ -44,7 +43,6 @@ public ExecutorDataBuilder(String cmd, List<EmbeddedArtifact> embeddedArtifacts,
this.sigKillProcessesAfterMillis = sigKillProcessesAfterMillis;
this.maxTaskThreads = maxTaskThreads;
this.preserveTaskSandboxAfterFinish = preserveTaskSandboxAfterFinish;
this.loggingS3Bucket = loggingS3Bucket;
this.maxOpenFiles = maxOpenFiles;
this.skipLogrotateAndCompress = skipLogrotateAndCompress;
this.s3ArtifactSignatures = s3ArtifactSignatures;
Expand All @@ -57,7 +55,7 @@ public ExecutorDataBuilder() {

public ExecutorData build() {
return new ExecutorData(cmd, embeddedArtifacts, externalArtifacts, s3Artifacts, successfulExitCodes, user, runningSentinel, extraCmdLineArgs, loggingTag, loggingExtraFields,
sigKillProcessesAfterMillis, maxTaskThreads, preserveTaskSandboxAfterFinish, loggingS3Bucket, maxOpenFiles, skipLogrotateAndCompress, s3ArtifactSignatures, logrotateFrequency);
sigKillProcessesAfterMillis, maxTaskThreads, preserveTaskSandboxAfterFinish, maxOpenFiles, skipLogrotateAndCompress, s3ArtifactSignatures, logrotateFrequency);
}

public Optional<String> getLoggingTag() {
Expand Down Expand Up @@ -177,15 +175,6 @@ public ExecutorDataBuilder setPreserveTaskSandboxAfterFinish(Optional<Boolean> p
return this;
}

public Optional<String> getLoggingS3Bucket() {
return loggingS3Bucket;
}

public ExecutorDataBuilder setLoggingS3Bucket(Optional<String> loggingS3Bucket) {
this.loggingS3Bucket = loggingS3Bucket;
return this;
}

public Optional<Integer> getMaxOpenFiles() {
return maxOpenFiles;
}
Expand Down Expand Up @@ -238,7 +227,6 @@ public String toString() {
", sigKillProcessesAfterMillis=" + sigKillProcessesAfterMillis +
", maxTaskThreads=" + maxTaskThreads +
", preserveTaskSandboxAfterFinish=" + preserveTaskSandboxAfterFinish +
", loggingS3Bucket=" + loggingS3Bucket +
", maxOpenFiles=" + maxOpenFiles +
", skipLogrotateAndCompress=" + skipLogrotateAndCompress +
", s3ArtifactSignatures=" + s3ArtifactSignatures +
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,28 +13,30 @@
import com.google.common.collect.Sets;

public class SingularityS3FormatHelper {
public static final String DEFAULT_GROUP_NAME = "default";

private static final List<String> DISALLOWED_FOR_TASK = ImmutableList.of("%index", "%s", "%filename", "%fileext");
private static final List<String> DISALLOWED_FOR_DEPLOY = ImmutableList.copyOf(Iterables.concat(DISALLOWED_FOR_TASK, ImmutableList.of("%host")));
private static final List<String> DISALLOWED_FOR_REQUEST = ImmutableList.copyOf(Iterables.concat(DISALLOWED_FOR_DEPLOY, ImmutableList.of("%tag", "%deployId")));

public static String getS3KeyFormat(String s3KeyFormat, String requestId) {
public static String getS3KeyFormat(String s3KeyFormat, String requestId, String group) {
s3KeyFormat = s3KeyFormat.replace("%requestId", requestId);
s3KeyFormat = s3KeyFormat.replace("%group", group);

return s3KeyFormat;
}

public static String getS3KeyFormat(String s3KeyFormat, String requestId, String deployId, Optional<String> loggingTag) {
s3KeyFormat = getS3KeyFormat(s3KeyFormat, requestId);
public static String getS3KeyFormat(String s3KeyFormat, String requestId, String deployId, Optional<String> loggingTag, String group) {
s3KeyFormat = getS3KeyFormat(s3KeyFormat, requestId, group);

s3KeyFormat = s3KeyFormat.replace("%tag", loggingTag.or(""));
s3KeyFormat = s3KeyFormat.replace("%deployId", deployId);

return s3KeyFormat;
}

public static String getS3KeyFormat(String s3KeyFormat, SingularityTaskId taskId, Optional<String> loggingTag) {
s3KeyFormat = getS3KeyFormat(s3KeyFormat, taskId.getRequestId(), taskId.getDeployId(), loggingTag);
public static String getS3KeyFormat(String s3KeyFormat, SingularityTaskId taskId, Optional<String> loggingTag, String group) {
s3KeyFormat = getS3KeyFormat(s3KeyFormat, taskId.getRequestId(), taskId.getDeployId(), loggingTag, group);

s3KeyFormat = s3KeyFormat.replace("%host", taskId.getSanitizedHost());
s3KeyFormat = s3KeyFormat.replace("%taskId", taskId.toString());
Expand Down Expand Up @@ -118,8 +120,8 @@ private static String getDayOrMonth(int value) {
return String.format("%02d", value);
}

public static Collection<String> getS3KeyPrefixes(String s3KeyFormat, String requestId, String deployId, Optional<String> tag, long start, long end) {
String keyFormat = getS3KeyFormat(s3KeyFormat, requestId, deployId, tag);
public static Collection<String> getS3KeyPrefixes(String s3KeyFormat, String requestId, String deployId, Optional<String> tag, long start, long end, String group) {
String keyFormat = getS3KeyFormat(s3KeyFormat, requestId, deployId, tag, group);

keyFormat = trimTaskId(keyFormat, requestId + "-" + deployId);

Expand All @@ -136,8 +138,8 @@ private static String trimTaskId(String s3KeyFormat, String replaceWith) {
return s3KeyFormat;
}

public static Collection<String> getS3KeyPrefixes(String s3KeyFormat, String requestId, long start, long end) {
s3KeyFormat = getS3KeyFormat(s3KeyFormat, requestId);
public static Collection<String> getS3KeyPrefixes(String s3KeyFormat, String requestId, long start, long end, String group) {
s3KeyFormat = getS3KeyFormat(s3KeyFormat, requestId, group);

s3KeyFormat = trimTaskId(s3KeyFormat, requestId);

Expand Down Expand Up @@ -206,8 +208,8 @@ private static Collection<String> getS3KeyPrefixes(String s3KeyFormat, List<Stri
return keyPrefixes;
}

public static Collection<String> getS3KeyPrefixes(String s3KeyFormat, SingularityTaskId taskId, Optional<String> tag, long start, long end) {
String keyFormat = getS3KeyFormat(s3KeyFormat, taskId, tag);
public static Collection<String> getS3KeyPrefixes(String s3KeyFormat, SingularityTaskId taskId, Optional<String> tag, long start, long end, String group) {
String keyFormat = getS3KeyFormat(s3KeyFormat, taskId, tag, group);

return getS3KeyPrefixes(keyFormat, DISALLOWED_FOR_TASK, start, end);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,75 +7,33 @@
import com.wordnik.swagger.annotations.ApiModelProperty;

@ApiModel( description = "Represents a task sandbox file that was uploaded to S3" )
public class SingularityS3Log {
public static final String LOG_START_S3_ATTR = "starttime";
public static final String LOG_END_S3_ATTR = "endtime";

public class SingularityS3Log extends SingularityS3LogMetadata {
private final String getUrl;
private final String key;
private final long lastModified;
private final long size;
private final String downloadUrl;
private final Optional<Long> startTime;
private final Optional<Long> endTime;

@JsonCreator
public SingularityS3Log(@JsonProperty("getUrl") String getUrl, @JsonProperty("key") String key, @JsonProperty("lastModified") long lastModified, @JsonProperty("size") long size, @JsonProperty("downloadUrl") String downloadUrl,
@JsonProperty("startTime") Optional<Long> startTime, @JsonProperty("endTime") Optional<Long> endTime) {
super(key, lastModified, size, startTime, endTime);
this.getUrl = getUrl;
this.key = key;
this.lastModified = lastModified;
this.size = size;
this.downloadUrl = downloadUrl;
this.startTime = startTime;
this.endTime = endTime;
}

@ApiModelProperty("URL to file in S3")
public String getGetUrl() {
return getUrl;
}

@ApiModelProperty("S3 key")
public String getKey() {
return key;
}

@ApiModelProperty("Last modified time")
public long getLastModified() {
return lastModified;
}

@ApiModelProperty("File size (in bytes)")
public long getSize() {
return size;
}

@ApiModelProperty("URL to file in S3 containing headers that will force file to be downloaded instead of viewed")
public String getDownloadUrl() {
return downloadUrl;
}

@ApiModelProperty("Time the log file started being written to")
public Optional<Long> getStartTime() {
return startTime;
}

@ApiModelProperty("Time the log file was finished being written to")
public Optional<Long> getEndTime() {
return endTime;
}

@Override
public String toString() {
return "SingularityS3Log{" +
"getUrl='" + getUrl + '\'' +
", key='" + key + '\'' +
", lastModified=" + lastModified +
", size=" + size +
", downloadUrl='" + downloadUrl + '\'' +
", startTime=" + startTime +
", endTime=" + endTime +
'}';
"} " + super.toString();
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
package com.hubspot.singularity;

import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.google.common.base.Optional;
import com.wordnik.swagger.annotations.ApiModelProperty;

public class SingularityS3LogMetadata {
public static final String LOG_START_S3_ATTR = "starttime";
public static final String LOG_END_S3_ATTR = "endtime";

private final String key;
private final long lastModified;
private final long size;
private final Optional<Long> startTime;
private final Optional<Long> endTime;

@JsonCreator
public SingularityS3LogMetadata(@JsonProperty("key") String key, @JsonProperty("lastModified") long lastModified, @JsonProperty("size") long size,
@JsonProperty("startTime") Optional<Long> startTime, @JsonProperty("endTime") Optional<Long> endTime) {
this.key = key;
this.lastModified = lastModified;
this.size = size;
this.startTime = startTime;
this.endTime = endTime;
}

@ApiModelProperty("S3 key")
public String getKey() {
return key;
}

@ApiModelProperty("Last modified time")
public long getLastModified() {
return lastModified;
}

@ApiModelProperty("File size (in bytes)")
public long getSize() {
return size;
}

@ApiModelProperty("Time the log file started being written to")
public Optional<Long> getStartTime() {
return startTime;
}

@ApiModelProperty("Time the log file was finished being written to")
public Optional<Long> getEndTime() {
return endTime;
}

@Override
public String toString() {
return "SingularityS3Log{" +
"key='" + key + '\'' +
", lastModified=" + lastModified +
", size=" + size +
", startTime=" + startTime +
", endTime=" + endTime +
'}';
}
}
Loading