ECI-393 Add Networking and Policy Sub-Modules #20

sva91 · 2024-12-04T20:11:22Z

What

Add Networking Sub-Module for Log Forwarding Setup
Add Policy sub-module for Log Forwarding Setup
Includes the following two enhancements

Show VCN and Subnet fields only if create_vcn is not selected.
Do not show tenancy, region and compartment to reduce confusion

Why

Setting up infrastructure for log forwarding

Testing

Ran the setup via terraform apply, as well as ORM to observe that this creates resources as expected.
https://cloud.oracle.com/resourcemanager/stacks/ocid1.ormstack.oc1.iad.amaaaaaaehwejaia7uc56y7gzw4mb2xv5zsiytzts3dmbvmgmnezooqlqb3a?region=us-ashburn-1

datadog-logs-oci-orm/logs-function/README.md

datadog-logs-oci-orm/modules/policy/main.tf

bengillmandd · 2024-12-12T16:09:43Z

datadog-logs-oci-orm/logs-function/README.md

+First, install the required dependencies listed in the `requirements.txt` file. You can do this using `pip`:
+
+```sh
+pip install -r requirements.txt


Not sure if this should be done as a part of this PR but IMO it is not clear to customers where they should run this. eg cloud shell? IMO the docs we currently have could use more hand-holding, for example how does one create a boilerplate python function.

For the purpose of this project, this readme can be considered applicable only for developers who are making changes to the python function. That includes mostly the Datadog team, but if there are external contributors, they can benefit from these steps as well. These steps shouldn't be executed from within customer's OCI environment.

bengillmandd · 2024-12-12T16:16:22Z

datadog-logs-oci-orm/logs-function/func.py

+DD_SOURCE = "oracle_cloud"  # Adding a source name.
+DD_SERVICE = "OCI Logs"  # Adding a service name.
+DD_TIMEOUT = 10 * 60  # Adding a timeout for the Datadog API call.
+DD_BATCH_SIZE = 1000  # Adding a batch size for the Datadog API call.


Is there a max size that can be sent to DD logs endpoint? I see that this batch is 1000 logs but I'm curious if we might need to enforce a size in bytes that are sent via the API?

I think the max size for the batch size is 1000, as enforced by Datadog. Since, logs are compressed before calling API, I think the byte limit shouldn't be a problem. Here are the limits: https://docs.datadoghq.com/api/latest/logs/#send-logs

The byte limit of 5Mb is on uncompressed payload. Even if sent compressed, it is uncompressed on EvP intake and then evaluated for its size.

Ideally, we can control byte limit by ensuring not more than 5Mb sent through connector hub

bengillmandd · 2024-12-12T16:26:37Z

datadog-logs-oci-orm/logs-function/func.py

+    return os.environ.get("DD_COMPRESS", "true").lower() == "true"
+
+
+def _compress_payload(payload: list[dict]):


might be good to have ther return type defined here so it's clear this can return two different types of data

bengillmandd · 2024-12-12T17:47:40Z

datadog-logs-oci-orm/logs-function/func.py

+    This function receives the logging json and invokes the Datadog endpoint
+    for ingesting logs. https://docs.cloud.oracle.com/en-us/iaas/Content/Logging/Reference/top_level_logging_format.htm#top_level_logging_format
+    If this Function is invoked with more than one log the function go over
+    each log and invokes the Datadog endpoint for ingesting one by one.


Are the logs ingested one by one? It looks like they are ingested in batches of 1000?

Yes, they are ingested in batches of 1000. When the function gets invoked, and if there are more than 1000 logs to be transmitted to Datadog, one function invocation will break them into batches of 1000, and make multiple API calls to datadog.

* Add Function App Module * Add container registry login functionality * reviewers comments * add service user optionality * ECI-394 Add function module and complete container registry module. (#22) * build and push image * Add function module * uncomment accidental comments * remove unnecessary code * Update datadog-logs-oci-orm/modules/containerregistry/locals.tf

datadog-logs-oci-orm/logs-function/requirements.txt

…respective pipeline

ECI [OCI] include more log attributes and assign "DD_SOURCE" based on OCI service

* working draft commit * Logging Module * uncomment modules * simplify file removal * Rename Logging Module to ResourceDiscovery * reviewer comments ECI-395

This reverts commit 5c1cb3b.

ECI-453 [OCI] add redaction option for sensitive log fields

* prepare tuples for log forwarding * Testing Issues resolution * uncomment code * rename data variable * move log search outside compartment in logging module * update output * simplify resource creation * Sva91/eci 397 handle audit log configs (#32) * ECI-397 Add Audit Log Forwarding * Uncomment audit log name * reviewer comment * ECI-398 Add connectorhub module (#35) * ECI-398 Add connectorhub module * fix bugs * simplify audit log input * uncomment modules * Sva91/ECI-399 E2E logs private beta update stack (#36) * modify as per updated requirements * remove resource prefix from policy * fix bug * fix login issue * Reviewer comments ECI-399 * Reviewer comments * reviewer comments ECI-396

kanishktripathi

few comments. We can also discuss about the logging module

datadog-logs-oci-orm/variables.tf

kanishktripathi · 2025-02-06T18:22:02Z

datadog-logs-oci-orm/modules/functionapp/variables.tf

+variable "datadog_tags" {
+  type        = string
+  default     = ""
+  description = "The tags to be sent with the logs. The tags should be in the format key1:value1,key2:value2"


Are we checking if the tags are in correct format?

kanishktripathi · 2025-02-06T18:23:15Z

.gitignore

@@ -1,3 +1,9 @@
 *.pyc
 **/__pycache__/**
 .idea/
+**/.terraform/**


Might be good to have this part merged now since it'll benefit anyone running terraform locally

kanishktripathi · 2025-02-06T20:02:22Z

datadog-logs-oci-orm/logging_services.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+
+output_file="oci_logging_services.json"


Thoughts on doing it through python where we have more flexibility like exception handling?

kanishktripathi · 2025-02-06T20:08:16Z

datadog-logs-oci-orm/logs-function/func.py

+DD_SOURCE = "oracle_cloud"  # Adding a source name.
+DD_SERVICE = "OCI Logs"  # Adding a service name.
+DD_TIMEOUT = 10 * 60  # Adding a timeout for the Datadog API call.
+DD_BATCH_SIZE = 1000  # Adding a batch size for the Datadog API call.


The byte limit of 5Mb is on uncompressed payload. Even if sent compressed, it is uncompressed on EvP intake and then evaluated for its size.

Ideally, we can control byte limit by ensuring not more than 5Mb sent through connector hub

datadog-logs-oci-orm/main.tf

kanishktripathi · 2025-02-06T20:51:14Z

datadog-logs-oci-orm/logging_services.sh

+echo "$response" > "$output_file"
+
+# Output the response in a valid JSON map for Terraform's external data source
+content=$(jq -c . < "$output_file") # Ensure the file's content is compact JSON


Can the sample output of this be documented?

Also, it'll be more readable to use python for better handling and for exception handling. If not immediately but as a follow-up

kanishktripathi · 2025-02-06T21:13:41Z

datadog-logs-oci-orm/modules/resourcediscovery/search_resources.sh

+    while [ "$next_page" != "null" ]; do
+        if [ "$next_page" == "init" ]; then
+            # First query without next page token
+            response=$(oci search resource structured-search --query-text \


There might be a trade-off between doing this search for single compartment or doing it for multiple compartments like:
(compartmentId = 'id1' || compartmentId = 'id2')
Can save on the API calls.

kanishktripathi · 2025-02-06T21:53:58Z

datadog-logs-oci-orm/modules/logging/data.tf

+  log_group_ids_string = join(",", [for lg in local.flat_log_groups : lg.id])
+}
+
+data "external" "logs_outside_compartment" {


It will be good to explain the steps being done here and in the underlying script. The reason to fetch log resources for other compartments.

kanishktripathi · 2025-02-06T22:01:03Z

datadog-logs-oci-orm/modules/logging/data.tf

+}
+
+# Step 1: Fetch all compartments in the tenancy
+data "oci_identity_compartments" "all_compartments" {


If this module runs for every logging compartment in a loop
in datadog-logs-oci-orm/main.tf, then is this step run every time? If yes, this can be minimized by running once

sva91 added 3 commits November 26, 2024 14:48

Init Function and Networking module

c4c6f69

Add VCN Module with create vcn option

0df16b8

add policy module

4442a73

sva91 marked this pull request as ready for review December 4, 2024 20:23

sva91 added the eci label Dec 4, 2024

sva91 commented Dec 4, 2024

View reviewed changes

datadog-logs-oci-orm/logs-function/README.md Show resolved Hide resolved

rheei reviewed Dec 5, 2024

View reviewed changes

datadog-logs-oci-orm/modules/policy/main.tf Outdated Show resolved Hide resolved

sva91 requested a review from a team December 5, 2024 21:57

sva91 changed the title ~~Sva91/log forwarding automation modules~~ ECI-393 Add Networking and Policy Sub-Modules Dec 10, 2024

bengillmandd reviewed Dec 12, 2024

View reviewed changes

datadog-datadog-staging-us1-all bot reviewed Dec 26, 2024

View reviewed changes

datadog-logs-oci-orm/logs-function/requirements.txt Show resolved Hide resolved

datadog-datadog-prod-us1 bot reviewed Dec 26, 2024

View reviewed changes

datadog-logs-oci-orm/logs-function/requirements.txt Show resolved Hide resolved

datadog-staging-us1-crawler-test bot reviewed Dec 26, 2024

View reviewed changes

datadog-logs-oci-orm/logs-function/requirements.txt Show resolved Hide resolved

rheei and others added 15 commits January 13, 2025 12:08

include type and oracle attributes when forwarding logs

cfcd696

specify the oci service as the dd source for directing logs to their …

9f10305

…respective pipeline

add unit test

184cac0

handle potential OOB error

2bff425

check for None

1558106

Merge pull request #29 from DataDog/rheei/add-more-log-attributes

a6a999a

ECI [OCI] include more log attributes and assign "DD_SOURCE" based on OCI service

ECI-395 Search Resources for Logging (#24)

178ccf0

* working draft commit * Logging Module * uncomment modules * simplify file removal * Rename Logging Module to ResourceDiscovery * reviewer comments ECI-395

add logic for redacdting fields

5c1cb3b

Revert "add logic for redacdting fields"

64b65ea

This reverts commit 5c1cb3b.

redact sensitive log fields

cc56e1c

Merge pull request #33 from DataDog/rheei/eci-453-add-redaction-option

408d97b

ECI-453 [OCI] add redaction option for sensitive log fields

add default for http endpoint

3fa502d

Assign compartment variable to Logging section

87036c4

conditional data log group

0aa8288

sva91 added 2 commits February 6, 2025 14:38

remove prevent destroy

ab4d4de

remove conditional creation

a64eec9

kanishktripathi reviewed Feb 6, 2025

View reviewed changes

sva91 added 5 commits February 6, 2025 17:20

add endpoints for other sites

3e9df07

update variable validation

80060dc

execute policy resources only in home region

e303fe7

add function shape variable

d3f449f

add support for tenancy compartment

8bcfa58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ECI-393 Add Networking and Policy Sub-Modules #20

ECI-393 Add Networking and Policy Sub-Modules #20

sva91 commented Dec 4, 2024 •

edited

Loading

bengillmandd Dec 12, 2024

sva91 Dec 26, 2024

bengillmandd Dec 12, 2024

sva91 Dec 26, 2024

kanishktripathi Feb 6, 2025

bengillmandd Dec 12, 2024

bengillmandd Dec 12, 2024

sva91 Dec 26, 2024

kanishktripathi left a comment

kanishktripathi Feb 6, 2025

kanishktripathi Feb 6, 2025

kanishktripathi Feb 6, 2025

kanishktripathi Feb 6, 2025

kanishktripathi Feb 6, 2025

kanishktripathi Feb 6, 2025

kanishktripathi Feb 6, 2025

kanishktripathi Feb 6, 2025 •

edited

Loading

		return os.environ.get("DD_COMPRESS", "true").lower() == "true"


		def _compress_payload(payload: list[dict]):

		@@ -0,0 +1,15 @@
		#!/bin/bash

		output_file="oci_logging_services.json"

ECI-393 Add Networking and Policy Sub-Modules #20

Are you sure you want to change the base?

ECI-393 Add Networking and Policy Sub-Modules #20

Conversation

sva91 commented Dec 4, 2024 • edited Loading

What

Why

Testing

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kanishktripathi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kanishktripathi Feb 6, 2025 • edited Loading

Choose a reason for hiding this comment

sva91 commented Dec 4, 2024 •

edited

Loading

kanishktripathi Feb 6, 2025 •

edited

Loading