Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v5.25.0 proposal #4849

Merged
merged 52 commits into from
Nov 6, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
8e4e88d
fix WAF update rules version (#4798)
simon-id Oct 17, 2024
235039f
[ASM] Ssrf handle request options (#4791)
iunanua Oct 18, 2024
dd20345
don't update waf version if it dosn't exist (#4801)
simon-id Oct 18, 2024
cb24aeb
Identify span metrics from OpenTelemetry libraries with 'otel.library…
mcculls Oct 21, 2024
556398d
Fix: esbuild plugin when requiring esm files (#4774)
crysmags Oct 21, 2024
f11858c
Implement Config Consistency (#4725)
khanayan123 Oct 22, 2024
23b32dc
[CI] Enable Fastify suite.js (#4771)
watson Oct 22, 2024
df4ebcf
Fix yarn.lock (#4809)
watson Oct 22, 2024
e0b5a9a
also audit devDependencies (#4807)
bengl Oct 22, 2024
4b5d0a5
Update native-appsec to 8.2.1 (#4810)
uurien Oct 22, 2024
bc2f06a
[DI] Adhere to maxCollectionSize limit in snapshots (#4780)
watson Oct 22, 2024
f300586
refactor system tests (#4811)
wconti27 Oct 22, 2024
04dfa40
Add Support For Overriding GRPC Error Statuses (#4800)
khanayan123 Oct 22, 2024
1b494b8
add requirements json with native deps and denylist (#4753)
rochdev Oct 22, 2024
6d2b07e
Add support for Azure App Services tags in profiler (#4803)
szegedi Oct 23, 2024
9213f37
[test visibility] Add errors in retried tests in mocha (#4813)
juan-fernandez Oct 23, 2024
d7a951c
[DI] Guard against invalid probe config and related edge-cases (#4816)
watson Oct 23, 2024
532fd3d
Support Node 23 in the profiler (#4815)
szegedi Oct 24, 2024
8140844
Protect some lines in text_map.js (#4820)
uurien Oct 24, 2024
7528014
Separating Plugin Tests to Their Own CI Run (#4822)
crysmags Oct 24, 2024
242dbc2
Update @datadog/native-iast-taint-tracking (#4824)
uurien Oct 25, 2024
6634f1b
feat(kafkajs): add kafka cluster id to spans and dsm metrics (#4808)
wconti27 Oct 25, 2024
b9f6dfe
[DI] Refactor integration tests (#4817)
watson Oct 28, 2024
5ad6a71
Profiler shouldn't retry some HTTP requests when sending profiles (#4…
szegedi Oct 28, 2024
4af469e
[test visibility] Add dynamic instrumentation logs writer for test vi…
juan-fernandez Oct 28, 2024
f524af5
Replace manual.keep tag usage with an specific method to keep the tra…
iunanua Oct 29, 2024
4a32123
Update WAF recommended rules to v1.13.2 (#4834)
CarlesDD Oct 29, 2024
58fc85c
[DI] Adhere to maxFieldCount limit in snapshots (#4829)
watson Oct 29, 2024
14f21ba
[ASM] multer instrumentation (#4781)
iunanua Oct 29, 2024
287898f
[MLOB-1524] feat(llmobs): Introduce LLM Observability SDK (#4742)
sabrenner Oct 29, 2024
4b3e10a
[DI] Drop snapshot if JSON payload is too large (#4818)
watson Oct 30, 2024
fe37123
Add support for exit spans in Code Origin for Spans (#4772)
watson Oct 30, 2024
ab5d20a
update native metrics to 3.0.1 (#4838)
rochdev Oct 30, 2024
2ae6f73
add dsm for google pub sub (#3855)
wconti27 Oct 30, 2024
86d7c3a
Fix header injection vulnerability detection for access-control-allow…
CarlesDD Oct 30, 2024
57895ac
Defend against ref being undefined (#4831)
szegedi Oct 30, 2024
b82610e
[MLOB-1562] feat(llmobs): add openai integration (#4840)
sabrenner Oct 30, 2024
994bea2
upgrade to latest @azure/functions version (#4845)
khanayan123 Oct 30, 2024
9a30230
fix(ci): revert typescript 5.0 for docs tests (#4846)
sabrenner Oct 31, 2024
17b26f5
add some clarity in CONTRIBUTING.md (#4850)
bengl Nov 1, 2024
e39717d
add `yarn env <plugin-name>` (#4852)
bengl Nov 1, 2024
a41adb6
Fix amqp instrumentation (#4839)
piochelepiotr Nov 1, 2024
70133d4
Exploit prevention Shell injection (#4792)
uurien Nov 4, 2024
d2dfff3
Profiling code to presume at least Node 16 (#4335)
szegedi Nov 4, 2024
123703d
Baggage support (#4563)
ida613 Nov 4, 2024
407531d
fix incompatibilities with node 16 for 4.49.0 release (#4854)
rochdev Nov 4, 2024
562a129
Onboarding tests: simple installer scenario (#4855)
robertomonteromiguel Nov 5, 2024
244c032
Support url.parse, url.URL.parse and new url.URL for IAST taint track…
uurien Nov 5, 2024
8a4221d
[test visibility] Simple dynamic instrumentation - test visibility cl…
juan-fernandez Nov 6, 2024
2136eb4
simplify baggage code and add test case (#4858)
ida613 Nov 6, 2024
8368519
update AWS payload extraction rules (#4859)
tlhunter Nov 6, 2024
7ce6e06
v5.25.0
rochdev Nov 6, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/appsec.yml
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ jobs:
express:
runs-on: ubuntu-latest
env:
PLUGINS: express|body-parser|cookie-parser
PLUGINS: express|body-parser|cookie-parser|multer
steps:
- uses: actions/checkout@v4
- uses: ./.github/actions/node/setup
Expand Down
49 changes: 49 additions & 0 deletions .github/workflows/llmobs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
name: LLMObs

on:
pull_request:
push:
branches: [master]
schedule:
- cron: '0 4 * * *'

concurrency:
group: ${{ github.workflow }}-${{ github.ref || github.run_id }}
cancel-in-progress: true

jobs:
sdk:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: ./.github/actions/testagent/start
- uses: ./.github/actions/node/setup
- uses: ./.github/actions/install
- uses: ./.github/actions/node/18
- run: yarn test:llmobs:sdk:ci
- uses: ./.github/actions/node/20
- run: yarn test:llmobs:sdk:ci
- uses: ./.github/actions/node/latest
- run: yarn test:llmobs:sdk:ci
- if: always()
uses: ./.github/actions/testagent/logs
- uses: codecov/codecov-action@v3

openai:
runs-on: ubuntu-latest
env:
PLUGINS: openai
steps:
- uses: actions/checkout@v4
rochdev marked this conversation as resolved.
Show resolved Hide resolved
- uses: ./.github/actions/testagent/start
- uses: ./.github/actions/node/setup
- uses: ./.github/actions/install
- uses: ./.github/actions/node/oldest
- run: yarn test:llmobs:plugins:ci
shell: bash
- uses: ./.github/actions/node/latest
- run: yarn test:llmobs:plugins:ci
shell: bash
- uses: codecov/codecov-action@v3
rochdev marked this conversation as resolved.
Show resolved Hide resolved
- if: always()
uses: ./.github/actions/testagent/logs
77 changes: 75 additions & 2 deletions .github/workflows/plugins.yml
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,14 @@ jobs:
- uses: actions/checkout@v4
- uses: ./.github/actions/plugins/test

body-parser:
runs-on: ubuntu-latest
env:
PLUGINS: body-parser
steps:
- uses: actions/checkout@v4
- uses: ./.github/actions/plugins/test

bunyan:
runs-on: ubuntu-latest
env:
Expand Down Expand Up @@ -259,6 +267,14 @@ jobs:
- run: yarn test:plugins:ci
- uses: codecov/codecov-action@v2

cookie-parser:
runs-on: ubuntu-latest
env:
PLUGINS: cookie-parser
steps:
- uses: actions/checkout@v4
- uses: ./.github/actions/plugins/test

couchbase:
strategy:
matrix:
Expand Down Expand Up @@ -366,7 +382,22 @@ jobs:
express:
runs-on: ubuntu-latest
env:
PLUGINS: express|body-parser|cookie-parser
PLUGINS: express
steps:
- uses: actions/checkout@v4
- uses: ./.github/actions/plugins/test

express-mongo-sanitize:
runs-on: ubuntu-latest
services:
mongodb:
image: circleci/mongo
ports:
- 27017:27017
env:
PLUGINS: express-mongo-sanitize
PACKAGE_NAMES: express-mongo-sanitize
SERVICES: mongo
steps:
- uses: actions/checkout@v4
- uses: ./.github/actions/plugins/test
Expand Down Expand Up @@ -548,6 +579,23 @@ jobs:
- uses: actions/checkout@v4
- uses: ./.github/actions/plugins/test

mariadb:
runs-on: ubuntu-latest
services:
mysql:
image: mariadb:10.4
env:
MYSQL_ALLOW_EMPTY_PASSWORD: 'yes'
MYSQL_DATABASE: 'db'
ports:
- 3306:3306
env:
PLUGINS: mariadb
SERVICES: mariadb
steps:
- uses: actions/checkout@v4
- uses: ./.github/actions/plugins/test

memcached:
runs-on: ubuntu-latest
services:
Expand Down Expand Up @@ -641,12 +689,29 @@ jobs:
ports:
- 3306:3306
env:
PLUGINS: mysql|mysql2|mariadb # TODO: move mysql2 to its own job
PLUGINS: mysql
SERVICES: mysql
steps:
- uses: actions/checkout@v4
- uses: ./.github/actions/plugins/test

mysql2:
runs-on: ubuntu-latest
services:
mysql:
image: mariadb:10.4
env:
MYSQL_ALLOW_EMPTY_PASSWORD: 'yes'
MYSQL_DATABASE: 'db'
ports:
- 3306:3306
env:
PLUGINS: mysql2
SERVICES: mysql2
steps:
- uses: actions/checkout@v4
- uses: ./.github/actions/plugins/test

net:
runs-on: ubuntu-latest
env:
Expand Down Expand Up @@ -934,6 +999,14 @@ jobs:
- uses: actions/checkout@v4
- uses: ./.github/actions/plugins/test

url:
runs-on: ubuntu-latest
env:
PLUGINS: url
steps:
- uses: actions/checkout@v4
- uses: ./.github/actions/plugins/test

when:
runs-on: ubuntu-latest
env:
Expand Down
1 change: 0 additions & 1 deletion .github/workflows/system-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,6 @@ jobs:
uses: DataDog/system-tests/.github/workflows/compute-workflow-parameters.yml@main
with:
library: nodejs
scenarios: CROSSED_TRACING_LIBRARIES
scenarios_groups: essentials

system-tests:
Expand Down
10 changes: 9 additions & 1 deletion .gitlab-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,16 @@ onboarding_tests_installer:
parallel:
matrix:
- ONBOARDING_FILTER_WEBLOG: [test-app-nodejs,test-app-nodejs-container]
SCENARIO: [ INSTALLER_AUTO_INJECTION, SIMPLE_AUTO_INJECTION_PROFILING ]
SCENARIO: [ SIMPLE_INSTALLER_AUTO_INJECTION, SIMPLE_AUTO_INJECTION_PROFILING ]

onboarding_tests_k8s_injection:
variables:
WEBLOG_VARIANT: sample-app

requirements_json_test:
rules:
- when: on_success
variables:
REQUIREMENTS_BLOCK_JSON_PATH: ".gitlab/requirements_block.json"
REQUIREMENTS_ALLOW_JSON_PATH: ".gitlab/requirements_allow.json"

2 changes: 2 additions & 0 deletions .gitlab/prepare-oci-package.sh
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,5 @@ fi
echo -n $JS_PACKAGE_VERSION > packaging/sources/version

cd packaging

cp ../requirements.json sources/requirements.json
19 changes: 19 additions & 0 deletions .gitlab/requirements_allow.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
[
{"name": "min glibc x64", "filepath": "/some/path", "args": [], "envars": [], "host": {"os": "linux", "arch": "x64", "libc": "glibc:2.17"}},
{"name": "ok glibc x64", "filepath": "/some/path", "args": [], "envars": [], "host": {"os": "linux", "arch": "x64", "libc": "glibc:2.23"}},
{"name": "high glibc x64", "filepath": "/some/path", "args": [], "envars": [], "host": {"os": "linux", "arch": "x64", "libc": "glibc:3.0"}},
{"name": "musl x64", "filepath": "/some/path", "args": [], "envars": [], "host": {"os": "linux", "arch": "x64", "libc": "musl:1.2.2"}},
{"name": "min glibc arm64", "filepath": "/some/path", "args": [], "envars": [], "host": {"os": "linux", "arch": "arm64", "libc": "glibc:2.17"}},
{"name": "ok glibc arm64", "filepath": "/some/path", "args": [], "envars": [], "host": {"os": "linux", "arch": "arm64", "libc": "glibc:2.27"}},
{"name": "glibc x86","filepath": "/some/path", "args": [], "envars": [], "host": {"os": "linux", "arch": "x86", "libc": "glibc:2.19"}},
{"name": "musl arm","filepath": "/some/path", "args": [], "envars": [], "host": {"os": "linux", "arch": "arm", "libc": "musl:1.2.2"}},
{"name": "musl arm64", "filepath": "/some/path", "args": [], "envars": [], "host": {"os": "linux", "arch": "arm64", "libc": "musl:1.2.2"}},
{"name": "musl x64", "filepath": "/some/path", "args": [], "envars": [], "host": {"os": "linux", "arch": "x64", "libc": "musl:1.2.2"}},
{"name": "musl x86", "filepath": "/some/path", "args": [], "envars": [], "host": {"os": "linux", "arch": "x86", "libc": "musl:1.2.2"}},
{"name": "windows x64", "filepath": "/some/path", "args": [], "envars": [], "host": {"os": "windows", "arch": "x64"}},
{"name": "windows x86", "filepath": "/some/path", "args": [], "envars": [], "host": {"os": "windows", "arch": "x86"}},
{"name": "macos x64", "filepath": "/some/path", "args": [], "envars": [], "host": {"os": "darwin", "arch": "x64"}},
{"name": "macos arm64", "filepath": "/some/path", "args": [], "envars": [], "host": {"os": "darwin", "arch": "arm64"}},
{"name": "node app", "filepath": "/pathto/node", "args": ["/pathto/node", "./app.js"], "envars": [], "host": {"os": "linux", "arch": "x64", "libc": "glibc:2.40"}},
{"name": "ts-node app", "filepath": "/pathto/ts-node", "args": ["/pathto/ts-node", "./app.js"], "envars": [], "host": {"os": "linux", "arch": "x64", "libc": "glibc:2.40"}}
]
11 changes: 11 additions & 0 deletions .gitlab/requirements_block.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
[
{"name": "unsupported 2.x glibc x64","filepath": "/some/path", "args": [], "envars": [], "host": {"os": "linux", "arch": "x64", "libc": "glibc:2.16"}},
{"name": "unsupported 1.x glibc x64","filepath": "/some/path", "args": [], "envars": [], "host": {"os": "linux", "arch": "x64", "libc": "glibc:1.22"}},
{"name": "unsupported 2.x.x glibc x64","filepath": "/some/path", "args": [], "envars": [], "host": {"os": "linux", "arch": "x64", "libc": "glibc:2.16.9"}},
{"name": "unsupported 2.x glibc arm64","filepath": "/some/path", "args": [], "envars": [], "host": {"os": "linux", "arch": "arm64", "libc": "glibc:2.16"}},
{"name": "unsupported 2.x.x glibc x64","filepath": "/some/path", "args": [], "envars": [], "host": {"os": "linux", "arch": "arm64", "libc": "glibc:2.16.9"}},
{"name": "unsupported 2.x.x glibc x86","filepath": "/some/path", "args": [], "envars": [], "host": {"os": "linux", "arch": "x86", "libc": "glibc:2.17"}},
{"name": "npm","filepath": "/pathto/node", "args": ["/pathto/node", "/pathto/npm-cli.js"], "envars": [], "host": {"os": "linux", "arch": "x64", "libc": "glibc:2.40"}},
{"name": "yarn","filepath": "/pathto/node", "args": ["/pathto/node", "/pathto/yarn.js"], "envars": [], "host": {"os": "linux", "arch": "x64", "libc": "glibc:2.40"}},
{"name": "pnpm","filepath": "/pathto/node", "args": ["/pathto/node", "/pathto/pnpm.cjs"], "envars": [], "host": {"os": "linux", "arch": "x64", "libc": "glibc:2.40"}}
]
67 changes: 42 additions & 25 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,12 +72,18 @@ Eventually we plan to look into putting these permission-required tests behind a

## Development Requirements

Since this project supports multiple Node versions, using a version
manager such as [nvm](https://github.com/creationix/nvm) is recommended.
Since this project supports multiple Node.js versions, using a version manager
such as [nvm](https://github.com/creationix/nvm) is recommended. If you're
unsure which version of Node.js to use, just use the latest version, which
should always work.

We use [yarn](https://yarnpkg.com/) for its workspace functionality, so make sure to install that as well.
We use [yarn](https://yarnpkg.com/) 1.x for its workspace functionality, so make sure to install that as well. The easist way to install yarn 1.x with with npm:

To install dependencies once you have Node and yarn installed, run:
```sh
$ npm install -g yarn
```

To install dependencies once you have Node and yarn installed, run this in the project directory:

```sh
$ yarn
Expand All @@ -91,23 +97,42 @@ $ yarn
The `pg-native` package requires `pg_config` to be in your `$PATH` to be able to install.
Please refer to [the "Install" section](https://github.com/brianc/node-postgres/tree/master/packages/pg-native#install) of the `pg-native` documentation for how to ensure your environment is configured correctly.

### Setup

Before running _plugin_ tests, the data stores need to be running.
The easiest way to start all of them is to use the provided
docker-compose configuration:
### Plugin Tests

```sh
$ docker-compose up -d -V --remove-orphans --force-recreate
$ yarn services
```
Before running _plugin_ tests, the supporting docker containers need to be running. You _can_ attempt to start all of them using docker-compose, but that's a drain on your system, and not all the images will even run at all on AMD64 devices.

> **Note**
> The `aerospike`, `couchbase`, `grpc` and `oracledb` instrumentations rely on
> native modules that do not compile on ARM64 devices (for example M1/M2 Mac)
> - their tests cannot be run locally on these devices.

### Unit Tests
Instead, you can follow this procedure for the plugin you want to run tests for:

1. Check the CI config in `.github/workflows/plugins.yml` to see what the appropriate values for the `SERVICES` and `PLUGINS` environment variables are for the plugin you're trying to test (noting that not all plugins require `SERVICES`). For example, for the `amqplib` plugin, the `SERVICES` value is `rabbitmq`, and the `PLUGINS` value is `amqplib`.
2. Run the appropriate docker-compose command to start the required services. For example, for the `amqplib` plugin, you would run: `docker compose up -d rabbitmq`.
3. Run `yarn services`, with the environment variables set above. This will install any versions of the library to be tested against into the `versions` directory, and check that the appropriate services are running prior to running the test.
4. Now, you can run `yarn test:plugins` with the environment variables set above to run the tests for the plugin you're interested in.

To wrap that all up into a simple few lines of shell commands, here is all of the above, for the `amqplib` plugin:

```sh
# These are exported for simplicity, but you can also just set them inline.
export SERVICES="rabbitmq" # retrieved from .github/workflows/plugins.yml
export PLUGINS="amqplib" # retrieved from .github/workflows/plugins.yml

docker compose up -d $SERVICES
yarn services

yarn test:plugins # This one actually runs the tests. Can be run many times.
```

You can also run the tests for multiple plugins at once by separating them with a pipe (`|`) delimiter. For example, to run the tests for the `amqplib` and `bluebird` plugins:

```sh
PLUGINS="amqplib|bluebird" yarn test:plugins
```

### Other Unit Tests

There are several types of unit tests, for various types of components. The
following commands may be useful:
Expand All @@ -124,17 +149,6 @@ $ yarn test:instrumentations
Several other components have test commands as well. See `package.json` for
details.

To test _plugins_ (i.e. components in `packages/datadog-plugin-XXXX`
directories, set the `PLUGINS` environment variable to the plugin you're
interested in, and use `yarn test:plugins`. If you need to test multiple
plugins you may separate then with a pipe (`|`) delimiter. Here's an
example testing the `express` and `bluebird` plugins:

```sh
PLUGINS="express|bluebird" yarn test:plugins
```


### Linting

We use [ESLint](https://eslint.org) to make sure that new code
Expand All @@ -146,6 +160,9 @@ To run the linter, use:
$ yarn lint
```

This also checks that the `LICENSE-3rdparty.csv` file is up-to-date, and checks
dependencies for vulnerabilities.


### Benchmarks

Expand Down
1 change: 1 addition & 0 deletions LICENSE-3rdparty.csv
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ require,retry,MIT,Copyright 2011 Tim Koschützki Felix Geisendörfer
require,rfdc,MIT,Copyright 2019 David Mark Clements
require,semver,ISC,Copyright Isaac Z. Schlueter and Contributors
require,shell-quote,mit,Copyright (c) 2013 James Halliday
dev,@apollo/server,MIT,Copyright (c) 2016-2020 Apollo Graph, Inc. (Formerly Meteor Development Group, Inc.)
dev,@types/node,MIT,Copyright Authors
dev,autocannon,MIT,Copyright 2016 Matteo Collina
dev,aws-sdk,Apache 2.0,Copyright 2012-2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
Expand Down
2 changes: 1 addition & 1 deletion docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ services:
- KAFKA_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093
- KAFKA_CONTROLLER_QUORUM_VOTERS=1@127.0.0.1:9093
- KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER
- KAFKA_CLUSTER_ID=r4zt_wrqTRuT7W2NJsB_GA
- CLUSTER_ID=5L6g3nShT-eMCtK--X86sw
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
- KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT
- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
Expand Down
2 changes: 1 addition & 1 deletion docs/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"main": "typedoc.js",
"scripts": {
"build": "typedoc ../index.d.ts && ./add-redirects.sh",
"pretest": "tsc -p . && tsc test",
"pretest": "tsc -p . && tsc --types node test",
"test": "node test"
},
"license": "BSD-3-Clause",
Expand Down
Loading
Loading