Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

optional pod selector labels #98

Merged
merged 3 commits into from
Jun 29, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions charts/core-dump-handler/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,7 @@ The agent pod has the following environment variables and these are all set by t
* INTERVAL - The amount of time in milliseconds between each check of the core dump folder for files to upload.
* SCHEDULE - A CRON formatted string [See cron library](https://github.com/mvniekerk/tokio-cron-scheduler#usage).
* USE_INOTIFY - Set a listener for the coredump folder can be used in conjunction with SCHEDULE
* COMP_POD_SELECTOR_LABEL - Optional selector label to filter pods that have core dump collection enabled. Default (empty) disables filter and enables collection for all. E.g. when selector label is set as "my.org/batch-workload" only pods that have a label named "my.org/batch-workload" (any value) will be enabled for core dump collection.

### Secrets

Expand Down Expand Up @@ -274,6 +275,7 @@ Composer
namespace - the namespace the pod is associated with.

* logLength: The amount of lines to take from the crashing pod. (Default 500)
* podSelectorLabel: Enable composer only if pod has label matching the specified selector. (Default "" matches all pods)

Daemonset
* hostDirectory: Maps to the HOST_DIR environment variable (Default "/var/mnt/core-dump-handler")
Expand Down
1 change: 1 addition & 0 deletions charts/core-dump-handler/ci/inotify-manage-store.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ composer:
# Double curlies are required otherwise helm trys to parse the string
filenameTemplate: "{{uuid}}-dump-{{timestamp}}-{{hostname}}-{{exe_name}}-{{pid}}-{{signal}}"
logLength: 500
podSelectorLabel: ""

daemonset:
name: "core-dump-handler"
Expand Down
1 change: 1 addition & 0 deletions charts/core-dump-handler/ci/interval-manage-store.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ composer:
# Double curlies are required otherwise helm trys to parse the string
filenameTemplate: "{{uuid}}-dump-{{timestamp}}-{{hostname}}-{{exe_name}}-{{pid}}-{{signal}}"
logLength: 500
podSelectorLabel: ""

daemonset:
name: "core-dump-handler"
Expand Down
1 change: 1 addition & 0 deletions charts/core-dump-handler/ci/schedule-no-manage-store.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ composer:
# Double curlies are required otherwise helm trys to parse the string
filenameTemplate: "{{uuid}}-dump-{{timestamp}}-{{hostname}}-{{exe_name}}-{{pid}}-{{signal}}"
logLength: 500
podSelectorLabel: ""

daemonset:
name: "core-dump-handler"
Expand Down
1 change: 1 addition & 0 deletions charts/core-dump-handler/ci/tolerations.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ composer:
# Double curlies are required otherwise helm trys to parse the string
filenameTemplate: "{{uuid}}-dump-{{timestamp}}-{{hostname}}-{{exe_name}}-{{pid}}-{{signal}}"
logLength: 500
podSelectorLabel: ""

daemonset:
name: "core-dump-handler"
Expand Down
2 changes: 2 additions & 0 deletions charts/core-dump-handler/templates/daemonset.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ spec:
value: {{ .Values.composer.ignoreCrio | quote }}
- name: COMP_CRIO_IMAGE_CMD
value: {{ .Values.composer.crioImageCmd }}
- name: COMP_POD_SELECTOR_LABEL
value: {{ .Values.composer.podSelectorLabel }}
- name: DEPLOY_CRIO_CONFIG
value: {{ .Values.daemonset.deployCrioConfig | quote }}
- name: CRIO_ENDPOINT
Expand Down
3 changes: 3 additions & 0 deletions charts/core-dump-handler/values.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,9 @@
},
"crioImageCmd": {
"type": "string"
},
"podSelectorLabel": {
"type": "string"
}
},
"required": [
Expand Down
1 change: 1 addition & 0 deletions charts/core-dump-handler/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ composer:
logLevel: "Warn"
filenameTemplate: "{uuid}-dump-{timestamp}-{hostname}-{exe_name}-{pid}-{signal}"
logLength: 500
podSelectorLabel: ""

daemonset:
name: "core-dump-handler"
Expand Down
5 changes: 3 additions & 2 deletions core-dump-agent/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -472,11 +472,12 @@ fn create_env_file(host_location: &str) -> Result<(), std::io::Error> {
"{uuid}-dump-{timestamp}-{hostname}-{exe_name}-{pid}-{signal}".to_string()
});
let log_length = env::var("LOG_LENGTH").unwrap_or_else(|_| "500".to_string());
let pod_selector_label = env::var("COMP_POD_SELECTOR_LABEL").unwrap_or_default();
info!("Creating {} file with LOG_LEVEL={}", destination, loglevel);
let mut env_file = File::create(destination)?;
let text = format!(
"LOG_LEVEL={}\nIGNORE_CRIO={}\nCRIO_IMAGE_CMD={}\nUSE_CRIO_CONF={}\nFILENAME_TEMPLATE={}\nLOG_LENGTH={}\n",
loglevel, ignore_crio, crio_image, use_crio_config, filename_template, log_length
"LOG_LEVEL={}\nIGNORE_CRIO={}\nCRIO_IMAGE_CMD={}\nUSE_CRIO_CONF={}\nFILENAME_TEMPLATE={}\nLOG_LENGTH={}\nPOD_SELECTOR_LABEL={}\n",
loglevel, ignore_crio, crio_image, use_crio_config, filename_template, log_length, pod_selector_label
);
info!("Writing composer .env \n{}", text);
env_file.write_all(text.as_bytes())?;
Expand Down
2 changes: 1 addition & 1 deletion core-dump-agent/tests/basic.rs
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ fn basic() -> Result<(), std::io::Error> {
"FILENAME_TEMPLATE={uuid}-dump-{timestamp}-{hostname}-{exe_name}-{pid}-{signal}"
));
assert!(env_content.contains("LOG_LENGTH=500"));
assert_eq!(env_content.lines().count(), 6);
assert_eq!(env_content.lines().count(), 7);
//TODO: [No9] Test uploading of a corefile
//TODO: [No9] Test remove option
//TODO: [No9] Test sweep option
Expand Down
3 changes: 3 additions & 0 deletions core-dump-composer/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ pub struct CoreConfig {
pub crictl_config_path: PathBuf,
pub log_level: String,
pub log_length: u32,
pub pod_selector_label: String,
pub use_crio_config: bool,
pub ignore_crio: bool,
pub image_command: ImageCommand,
Expand Down Expand Up @@ -85,6 +86,7 @@ impl CoreConfig {
let mut base_path = env::current_exe()?;
base_path.pop();

let pod_selector_label = env::var("POD_SELECTOR_LABEL").unwrap_or_default();
let log_level = env::var("LOG_LEVEL").unwrap_or_default();
let ignore_crio = env::var("IGNORE_CRIO")
.unwrap_or_else(|_| "false".to_string())
Expand Down Expand Up @@ -122,6 +124,7 @@ impl CoreConfig {

Ok(CoreConfig {
log_level,
pod_selector_label,
ignore_crio,
dot_env_path,
image_command,
Expand Down
19 changes: 19 additions & 0 deletions core-dump-composer/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,25 @@ fn main() -> Result<(), anyhow::Error> {
}
};

// match the label filter if there's one, and skip the whole process if it doesn't match
if !cc.pod_selector_label.is_empty() {
debug!(
"Pod selector specified. Will record only if pod has label {}",
&cc.pod_selector_label
);
let pod_labels = pod_object["labels"].as_object().unwrap();
// check if pod_labels has pod_selector_label
if pod_labels.get(&cc.pod_selector_label).is_none() {
info!(
"Skipping pod as it did not match selector label {}",
&cc.pod_selector_label
);
process::exit(0);
}
} else {
debug!("No pod selector specified, selecting all pods");
}

let namespace = pod_object["metadata"]["namespace"]
.as_str()
.unwrap_or("unknown");
Expand Down