Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pre-configure Splunk to speed up testing #948

Merged
merged 3 commits into from
Oct 5, 2023
Merged

Pre-configure Splunk to speed up testing #948

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d602a69
[chore] allow validation of examples allowing multi-documents
atoulme Oct 5, 2023
b5c7d8d
[chore] use splunk.yml to configure splunk on startup
atoulme Oct 5, 2023
69dd758
Merge branch 'main' into splunk_preconfigured
atoulme Oct 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 1 addition & 13 deletions .github/workflows/functional_test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jobs:
env:
CI_SPLUNK_PORT: 8089
CI_SPLUNK_USERNAME: admin
CI_SPLUNK_HEC_TOKEN: a6b5e77f-d5f6-415a-bd43-930cecb12959
CI_SPLUNK_HEC_TOKEN: "00000000-0000-0000-0000-0000000000000"
CI_SPLUNK_PASSWORD: helloworld
CI_INDEX_EVENTS: ci_events
CI_INDEX_METRICS: ci_metrics
Expand Down Expand Up @@ -73,18 +73,6 @@ jobs:
until kubectl logs splunk --tail=2 | grep -q 'Ansible playbook complete'; do
sleep 1;
done
export CI_SPLUNK_HOST=$(kubectl get pod splunk --template={{.status.podIP}})
# Setup Indexes
curl -k -u $CI_SPLUNK_USERNAME:$CI_SPLUNK_PASSWORD https://$CI_SPLUNK_HOST:$CI_SPLUNK_PORT/services/data/indexes -d name=$CI_INDEX_EVENTS -d datatype=event
curl -k -u $CI_SPLUNK_USERNAME:$CI_SPLUNK_PASSWORD https://$CI_SPLUNK_HOST:$CI_SPLUNK_PORT/services/data/indexes -d name=ns-anno -d datatype=event
curl -k -u $CI_SPLUNK_USERNAME:$CI_SPLUNK_PASSWORD https://$CI_SPLUNK_HOST:$CI_SPLUNK_PORT/services/data/indexes -d name=pod-anno -d datatype=event
curl -k -u $CI_SPLUNK_USERNAME:$CI_SPLUNK_PASSWORD https://$CI_SPLUNK_HOST:$CI_SPLUNK_PORT/services/data/indexes -d name=$CI_INDEX_METRICS -d datatype=metric
# Enable HEC services
curl -X POST -u $CI_SPLUNK_USERNAME:$CI_SPLUNK_PASSWORD -k https://$CI_SPLUNK_HOST:$CI_SPLUNK_PORT/servicesNS/nobody/splunk_httpinput/data/inputs/http/http/enable
# Create new HEC token
curl -X POST -u $CI_SPLUNK_USERNAME:$CI_SPLUNK_PASSWORD -k -d "name=splunk_hec_token&token=a6b5e77f-d5f6-415a-bd43-930cecb12959&disabled=0&index=main&indexes=main,ci_events,ci_metrics,ns-anno,pod-anno" https://$CI_SPLUNK_HOST:$CI_SPLUNK_PORT/servicesNS/nobody/splunk_httpinput/data/inputs/http
# Restart Splunk
curl -k -u $CI_SPLUNK_USERNAME:$CI_SPLUNK_PASSWORD https://$CI_SPLUNK_HOST:$CI_SPLUNK_PORT/services/server/control/restart -X POST

- name: Deploy splunk-otel-collector chart
run: |
Expand Down
5 changes: 2 additions & 3 deletions .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@ repos:
exclude: "^examples|^test"
- id: check-yaml
# Can't check source yaml since it has go templates in it.
# Can't check the examples because example demos contain multiple Kubernetes objects defined
# in a single file.
exclude: "^helm-charts|^test|^examples"
exclude: "^helm-charts"
args: [ --allow-multiple-documents ]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just noting this is also in https://github.com/signalfx/splunk-otel-collector-chart/pull/946/files

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes this PR was built off that branch as it requires this change.

- id: check-added-large-files
52 changes: 52 additions & 0 deletions ci_scripts/k8s-splunk.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,47 @@
apiVersion: v1
kind: ConfigMap
Copy link
Contributor

@jvoravong jvoravong Oct 5, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might suggest exposing k8s-splunk.yml more to users as a demo in the examples read me with something like this. Not required though.

Deploy local Splunk Enterprise Demo

Deploy the docker image for Splunk Enterprise to a cluster.

curl https://raw.githubusercontent.com/signalfx/splunk-otel-collector-chart/main/ci_scripts/k8s-splunk.yml | kubectl apply -f -

Use these Helm Chart values to send data to the Splunk Enterprise Demo endpoint.

# Splunk Platform required parameters
clusterName: splunk
splunkPlatform:
  token: "00000000-0000-0000-0000-0000000000000"
  index: main

metadata:
name: splunk-config
namespace: default
data:
default.yml: |
splunk:
conf:
indexes:
directory: /opt/splunk/etc/apps/search/local
content:
ci_events:
coldPath: $SPLUNK_DB/ci_events/colddb
datatype: event
homePath: $SPLUNK_DB/ci_events/db
maxTotalDataSizeMB: 512000
thawedPath: $SPLUNK_DB/ci_events/thaweddb
ci_metrics:
coldPath: $SPLUNK_DB/ci_metrics/colddb
datatype: metric
homePath: $SPLUNK_DB/ci_metrics/db
maxTotalDataSizeMB: 512000
thawedPath: $SPLUNK_DB/ci_metrics/thaweddb
ci_traces:
coldPath: $SPLUNK_DB/ci_traces/colddb
datatype: event
homePath: $SPLUNK_DB/ci_traces/db
maxTotalDataSizeMB: 512000
thawedPath: $SPLUNK_DB/ci_traces/thaweddb
ns-anno:
coldPath: $SPLUNK_DB/ns-anno/colddb
datatype: event
homePath: $SPLUNK_DB/ns-anno/db
maxTotalDataSizeMB: 512000
thawedPath: $SPLUNK_DB/ns-anno/thaweddb
pod-anno:
coldPath: $SPLUNK_DB/pod-anno/colddb
datatype: event
homePath: $SPLUNK_DB/pod-anno/db
maxTotalDataSizeMB: 512000
thawedPath: $SPLUNK_DB/pod-anno/thaweddb
---
apiVersion: v1
kind: Pod
metadata:
name: splunk
Expand Down Expand Up @@ -31,3 +74,12 @@ spec:
value: helloworld
- name: SPLUNK_LAUNCH_CONF
value: OPTIMISTIC_ABOUT_FILE_LOCKING=1
- name: SPLUNK_HEC_TOKEN
value: 00000000-0000-0000-0000-0000000000000
volumeMounts:
- name: config-volume
mountPath: /tmp/defaults/
volumes:
- name: config-volume
configMap:
name: splunk-config
19 changes: 3 additions & 16 deletions test/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,20 +27,6 @@
# Start a new terminal concole, run following command and keep it running in the background
kubectl port-forward pods/splunk 8089

# Setup Indexes
curl -k -u admin:helloworld https://localhost:8089/services/data/indexes -d name=ci_events -d datatype=event
curl -k -u admin:helloworld https://localhost:8089/services/data/indexes -d name=ns-anno -d datatype=event
curl -k -u admin:helloworld https://localhost:8089/services/data/indexes -d name=pod-anno -d datatype=event

# Enable HEC services
curl -X POST -u admin:helloworld -k https://localhost:8089/servicesNS/nobody/splunk_httpinput/data/inputs/http/http/enable

# Create new HEC token
curl -X POST -u admin:helloworld -k -d "name=splunk_hec_token&token=a6b5e77f-d5f6-415a-bd43-930cecb12959&disabled=0&index=main&indexes=main,ci_events,ns-anno,pod-anno" https://localhost:8089/servicesNS/nobody/splunk_httpinput/data/inputs/http

# Restart Splunk
curl -k -u admin:helloworld https://localhost:8089/services/server/control/restart -X POST

# Start a new terminal console, forward local port 8000 to the port on Splunk pod (for debugging)
kubectl port-forward pods/splunk 8000
You can then visit Splunk web page: https://localhost:8000
Expand Down Expand Up @@ -80,7 +66,8 @@
```
python -m pytest \
--splunkd-url https://localhost:8089 \
--splunk-user admin --splunk-password helloworld \
--splunk-user admin \
--splunk-password helloworld \
-p no:warnings -s
```
**Options are:**
Expand All @@ -94,5 +81,5 @@

--splunk-password
* Description: splunk user password
* Default: changeme
* Default: helloworld

2 changes: 1 addition & 1 deletion test/config_yaml_files/agent_tests_values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ splunkPlatform:
endpoint:
insecureSkipVerify: true
# token: ${CI_SPLUNK_HEC_TOKEN}
token: "a6b5e77f-d5f6-415a-bd43-930cecb12959"
token: "00000000-0000-0000-0000-0000000000000"
logsEnabled: true
metricsEnabled: false
metricsIndex:
Expand Down