Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(cdp): Added legacy plugins worker #27835

Open
wants to merge 33 commits into
base: master
Choose a base branch
from
Open

feat(cdp): Added legacy plugins worker #27835

wants to merge 33 commits into from

Conversation

benjackwhite
Copy link
Contributor

@benjackwhite benjackwhite commented Jan 23, 2025 •
edited
Loading

Problem

We want to fast migrate off of plugins to HogFunctions and one way we can do this is by just inlining the plugins and using HogFunctions as the base layer for the actual work

Changes

  • Adds the plugin worker which simulates the plugin server

TODO

  • Detect these "plugin" destinations and queue to cyclotron as "plugin"
  • Add migration command (will test properly and have follow up work for sure
  • Collect logs
  • Fix for customer.io person checking (we could just use persons instead and remove the storage)

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Does this work well for both Cloud and self-hosted?

How did you test this code?

@posthog-bot
Copy link
Contributor

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

  • chromium: 0 added, 1 modified, 0 deleted (diff for shard 1)
  • webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 27, 2025 •
edited
Loading

Size Change: +5 B (0%)

Total Size: 1.16 MB

ℹ️ View Unchanged
Filename Size Change
frontend/dist/toolbar.js 1.16 MB +5 B (0%)

compressed-size-action

@posthog-bot
Copy link
Contributor

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

  • chromium: 0 added, 1 modified, 0 deleted (diff for shard 1)
  • webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

@benjackwhite benjackwhite marked this pull request as ready for review January 27, 2025 17:24
github-advanced-security[bot]
github-advanced-security bot found potential problems Jan 27, 2025
View reviewed changes
posthog/cdp/migrations.py
"icon_url": plugin_config.plugin.icon,
}

print("Attempting to create hog function", data) # noqa: T201

Check failure

Code scanning / CodeQL

Clear-text logging of sensitive information High

This expression logs
sensitive data (secret)
Loading
as clear text.

Copilot Autofix AI about 10 hours ago

To fix the problem, we should avoid logging the entire data object directly. Instead, we can log only non-sensitive parts of the data object or use a method to sanitize the data before logging. This ensures that any sensitive information is not exposed in the logs.

The best way to fix the problem without changing existing functionality is to create a sanitized version of the data object that excludes any sensitive information and log this sanitized version instead. We can achieve this by creating a helper function that removes or masks sensitive fields from the data object.

Suggested changeset 1
posthog/cdp/migrations.py

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/posthog/cdp/migrations.py b/posthog/cdp/migrations.py
--- a/posthog/cdp/migrations.py
+++ b/posthog/cdp/migrations.py
@@ -8,2 +8,9 @@
 
+def sanitize_data(data):
+    sanitized = data.copy()
+    if "inputs" in sanitized:
+        for key, value in sanitized["inputs"].items():
+            if value.get("secret", False):
+                sanitized["inputs"][key]["value"] = "[REDACTED]"
+    return sanitized
 
@@ -88,4 +95,5 @@
 
-        print("Attempting to create hog function", data)  # noqa: T201
-        print(json.dumps(data, indent=2))  # noqa: T201
+        sanitized_data = sanitize_data(data)
+        print("Attempting to create hog function", sanitized_data)  # noqa: T201
+        print(json.dumps(sanitized_data, indent=2))  # noqa: T201
 
EOF
@@ -8,2 +8,9 @@

def sanitize_data(data):
sanitized = data.copy()
if "inputs" in sanitized:
for key, value in sanitized["inputs"].items():
if value.get("secret", False):
sanitized["inputs"][key]["value"] = "[REDACTED]"
return sanitized

@@ -88,4 +95,5 @@

print("Attempting to create hog function", data) # noqa: T201
print(json.dumps(data, indent=2)) # noqa: T201
sanitized_data = sanitize_data(data)
print("Attempting to create hog function", sanitized_data) # noqa: T201
print(json.dumps(sanitized_data, indent=2)) # noqa: T201

Copilot is powered by AI and may make mistakes. Always verify output.
Positive Feedback
Negative Feedback

Provide additional feedback

Please help us improve GitHub Copilot by sharing more details about this comment.

Please select one or more of the options
posthog/cdp/migrations.py
}

print("Attempting to create hog function", data) # noqa: T201
print(json.dumps(data, indent=2)) # noqa: T201

Check failure

Code scanning / CodeQL

Clear-text logging of sensitive information High

This expression logs
sensitive data (secret)
Loading
as clear text.

Copilot Autofix AI about 10 hours ago

To fix the problem, we need to ensure that sensitive information is not logged in clear text. The best way to do this is to remove or mask the sensitive information before logging. Specifically, we should avoid logging the entire data object directly and instead log only non-sensitive parts of it. We can create a sanitized version of the data object that excludes or masks sensitive fields before logging.

  1. Identify the lines where sensitive information is being logged.
  2. Create a sanitized version of the data object that excludes or masks sensitive fields.
  3. Log the sanitized version instead of the original data object.
Suggested changeset 1
posthog/cdp/migrations.py

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/posthog/cdp/migrations.py b/posthog/cdp/migrations.py
--- a/posthog/cdp/migrations.py
+++ b/posthog/cdp/migrations.py
@@ -88,4 +88,8 @@
 
-        print("Attempting to create hog function", data)  # noqa: T201
-        print(json.dumps(data, indent=2))  # noqa: T201
+        sanitized_data = data.copy()
+        for input_key in sanitized_data["inputs"]:
+            if sanitized_data["inputs"][input_key].get("secret", False):
+                sanitized_data["inputs"][input_key]["value"] = "[REDACTED]"
+        print("Attempting to create hog function", sanitized_data)  # noqa: T201
+        print(json.dumps(sanitized_data, indent=2))  # noqa: T201
 
@@ -98,3 +102,3 @@
 
-    print(hog_functions)  # noqa: T201
+    print([{"name": hf.name, "id": hf.id} for hf in hog_functions])  # noqa: T201
 
@@ -102,3 +106,3 @@
         print("Dry run, not creating hog functions")  # noqa: T201
-        return hog_functions
+        return [{"name": hf.name, "id": hf.id} for hf in hog_functions]
 
@@ -110,3 +114,3 @@
 
-    print("Done")  # noqa: T201
+    print("Done creating hog functions")  # noqa: T201
 
EOF
@@ -88,4 +88,8 @@

print("Attempting to create hog function", data) # noqa: T201
print(json.dumps(data, indent=2)) # noqa: T201
sanitized_data = data.copy()
for input_key in sanitized_data["inputs"]:
if sanitized_data["inputs"][input_key].get("secret", False):
sanitized_data["inputs"][input_key]["value"] = "[REDACTED]"
print("Attempting to create hog function", sanitized_data) # noqa: T201
print(json.dumps(sanitized_data, indent=2)) # noqa: T201

@@ -98,3 +102,3 @@

print(hog_functions) # noqa: T201
print([{"name": hf.name, "id": hf.id} for hf in hog_functions]) # noqa: T201

@@ -102,3 +106,3 @@
print("Dry run, not creating hog functions") # noqa: T201
return hog_functions
return [{"name": hf.name, "id": hf.id} for hf in hog_functions]

@@ -110,3 +114,3 @@

print("Done") # noqa: T201
print("Done creating hog functions") # noqa: T201

Copilot is powered by AI and may make mistakes. Always verify output.
Positive Feedback
Negative Feedback

Provide additional feedback

Please help us improve GitHub Copilot by sharing more details about this comment.

Please select one or more of the options
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@meikelmosby meikelmosby Awaiting requested review from meikelmosby

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@benjackwhite @posthog-bot