Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vdk-dag: improve DAGs docs and example #1984

Merged
merged 41 commits into from
May 9, 2023

Conversation

yonitoo
Copy link
Contributor

@yonitoo yonitoo commented Apr 27, 2023

What:
Improve DAGs user-facing documentation and example based on the feedback and discussions with users.
The feedback we got from users is:

  • The Requirements section in the example links VDK DAGs to the VEP, it should rather link to the README. > Addressed.
  • Would it be possible to run the example with a DB of choice or am I required to use Trino? > Addressed.
  • The team name has to be added on several occasions and this could lead to some consistency issues. > Addressed.
  • The DAG-specific configuration variables are not very visible in the README. > Addressed.

Signed-off-by: Yoan Salambashev ysalambashev@vmware.com

Signed-off-by: Yoan Salambashev <ysalambashev@vmware.com>
yonitoo and others added 2 commits April 27, 2023 16:09
examples/dag-with-args-example/README.md Outdated Show resolved Hide resolved
examples/dag-with-args-example/README.md Outdated Show resolved Hide resolved
projects/vdk-plugins/vdk-meta-jobs/README.md Outdated Show resolved Hide resolved
yonitoo and others added 2 commits May 3, 2023 11:57
Signed-off-by: Yoan Salambashev <ysalambashev@vmware.com>
@yonitoo yonitoo marked this pull request as ready for review May 3, 2023 09:03
ivakoleva and others added 21 commits May 3, 2023 12:03
We need a feature flag for skipping OAuth2, so the frontend can 
operate with a Control Service where security is disabled. It is the
default use case when having quickstart-vdk for local demo or dev
installation.

This is a stacked PR onto auth configurations organized from
#1957
Introduced authSkipped flag defaulting to false, due security standards.
The top right corner with username and logout dropdown are rendered 
in case auth is enabled.

Testing done: verified both cases locally; added testing

---------

Signed-off-by: ivakoleva <iva.koleva@clearcode.bg>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Add new 'details' buttons to make error messages more visible. Note that
these messages are not very frequent
Changes are needed in three pages:
1. Manage Data Jobs grid
2. Data Job Executions tab grid
3. Details page - timeline (on both manage and explore sections)

In manage page grid and executions grid, 'details' buttons will appear
when the execution status is 'User Error', 'Platform Error' or
'Skipped'.

When displaying in details page timeline, I show the entire error
message without using signpost.

Testing done: verified locally


![image](https://user-images.githubusercontent.com/127503989/234296967-d776f764-3aa3-420d-9d66-6b9ee1ca6886.png)

![image](https://user-images.githubusercontent.com/127503989/234296997-033995e9-db01-4a8c-b100-b29e49ef46ba.png)

![image](https://user-images.githubusercontent.com/127503989/234297037-88b69bd6-ed7c-4df7-a303-e8fd2e2db3fb.png)
What: 
Fix typo in plugin name of DAGs example README.md

Signed-off-by: Yoan Salambashev <ysalambashev@vmware.com>
…2 to 1.9.3 in /projects/control-service/projects/pipelines_control_service (#1987)

Bumps
[org.junit.platform:junit-platform-commons](https://github.com/junit-team/junit5)
from 1.9.2 to 1.9.3.
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a
href="https://github.com/junit-team/junit5/commits">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=org.junit.platform:junit-platform-commons&package-manager=gradle&previous-version=1.9.2&new-version=1.9.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [org.junit.platform:junit-platform-suite-api](https://github.com/junit-team/junit5) from 1.9.2 to 1.9.3.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/commits)

---
updated-dependencies:
- dependency-name: org.junit.platform:junit-platform-suite-api
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
…2.457 in /projects/control-service/projects/pipelines_control_service (#1985)

Bumps
[com.amazonaws:aws-java-sdk-ecr](https://github.com/aws/aws-sdk-java)
from 1.12.455 to 1.12.457.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md">com.amazonaws:aws-java-sdk-ecr's
changelog</a>.</em></p>
<blockquote>
<h1><strong>1.12.457</strong> <strong>2023-04-26</strong></h1>
<h2><strong>Amazon OpenSearch Ingestion</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>Initial release for OpenSearch Ingestion</li>
</ul>
</li>
</ul>
<h1><strong>1.12.456</strong> <strong>2023-04-25</strong></h1>
<h2><strong>AWS DataSync</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>This release adds 13 new APIs to support AWS DataSync Discovery
GA.</li>
</ul>
</li>
</ul>
<h2><strong>AWS Directory Service</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>New field added in AWS Managed Microsoft AD DescribeSettings
response and regex pattern update for UpdateSettings value. Added length
validation to RemoteDomainName.</li>
</ul>
</li>
</ul>
<h2><strong>Amazon Chime SDK Messaging</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>Remove non actionable field from UpdateChannelReadMarker and
DeleteChannelRequest. Add precise exceptions to DeleteChannel and
DeleteStreamingConfigurations error cases.</li>
</ul>
</li>
</ul>
<h2><strong>Amazon Connect Service</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>Amazon Connect, Contact Lens Evaluation API release including
ability to manage forms and to submit contact evaluations.</li>
</ul>
</li>
</ul>
<h2><strong>Amazon Pinpoint</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>Adds support for journey runs and querying journey execution metrics
based on journey runs. Adds execution metrics to campaign activities.
Updates docs for Advanced Quiet Time.</li>
</ul>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/aws/aws-sdk-java/commit/6306a0895b9f1849351d2365eede1163c508fb4b"><code>6306a08</code></a>
AWS SDK for Java 1.12.457</li>
<li><a
href="https://github.com/aws/aws-sdk-java/commit/d0c7a4c3ee0f9b88dc646905cfbade0c8e61e2c2"><code>d0c7a4c</code></a>
Update GitHub version number to 1.12.457-SNAPSHOT</li>
<li><a
href="https://github.com/aws/aws-sdk-java/commit/dea8f2d050934be8df527891dd080e51f06c0527"><code>dea8f2d</code></a>
AWS SDK for Java 1.12.456</li>
<li><a
href="https://github.com/aws/aws-sdk-java/commit/7b3b8d3b5e5a09d2621b00b57b05c8baa98e73c7"><code>7b3b8d3</code></a>
Update GitHub version number to 1.12.456-SNAPSHOT</li>
<li>See full diff in <a
href="https://github.com/aws/aws-sdk-java/compare/1.12.455...1.12.457">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=com.amazonaws:aws-java-sdk-ecr&package-manager=gradle&previous-version=1.12.455&new-version=1.12.457)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…2.457 in /projects/control-service/projects/pipelines_control_service (#1981)

Bumps
[com.amazonaws:aws-java-sdk-sts](https://github.com/aws/aws-sdk-java)
from 1.12.454 to 1.12.457.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md">com.amazonaws:aws-java-sdk-sts's
changelog</a>.</em></p>
<blockquote>
<h1><strong>1.12.457</strong> <strong>2023-04-26</strong></h1>
<h2><strong>Amazon OpenSearch Ingestion</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>Initial release for OpenSearch Ingestion</li>
</ul>
</li>
</ul>
<h1><strong>1.12.456</strong> <strong>2023-04-25</strong></h1>
<h2><strong>AWS DataSync</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>This release adds 13 new APIs to support AWS DataSync Discovery
GA.</li>
</ul>
</li>
</ul>
<h2><strong>AWS Directory Service</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>New field added in AWS Managed Microsoft AD DescribeSettings
response and regex pattern update for UpdateSettings value. Added length
validation to RemoteDomainName.</li>
</ul>
</li>
</ul>
<h2><strong>Amazon Chime SDK Messaging</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>Remove non actionable field from UpdateChannelReadMarker and
DeleteChannelRequest. Add precise exceptions to DeleteChannel and
DeleteStreamingConfigurations error cases.</li>
</ul>
</li>
</ul>
<h2><strong>Amazon Connect Service</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>Amazon Connect, Contact Lens Evaluation API release including
ability to manage forms and to submit contact evaluations.</li>
</ul>
</li>
</ul>
<h2><strong>Amazon Pinpoint</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>Adds support for journey runs and querying journey execution metrics
based on journey runs. Adds execution metrics to campaign activities.
Updates docs for Advanced Quiet Time.</li>
</ul>
</li>
</ul>
<h1><strong>1.12.455</strong> <strong>2023-04-24</strong></h1>
<h2><strong>AWS Elemental MediaConvert</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>This release introduces a noise reduction pre-filter, linear
interpolation deinterlace mode, video pass-through, updated default job
settings, and expanded LC-AAC Stereo audio bitrate ranges.</li>
</ul>
</li>
</ul>
<h2><strong>Amazon Appflow</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>Increased the max length for RefreshToken and AuthCode from 2048 to
4096.</li>
</ul>
</li>
</ul>
<h2><strong>Amazon CodeCatalyst</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>Documentation updates for Amazon CodeCatalyst.</li>
</ul>
</li>
</ul>
<h2><strong>Amazon Elastic Compute Cloud</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>API changes to AWS Verified Access related to identity providers'
information.</li>
</ul>
</li>
</ul>
<h2><strong>Amazon Rekognition</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>Added new status result to Liveness session status.</li>
</ul>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/aws/aws-sdk-java/commit/6306a0895b9f1849351d2365eede1163c508fb4b"><code>6306a08</code></a>
AWS SDK for Java 1.12.457</li>
<li><a
href="https://github.com/aws/aws-sdk-java/commit/d0c7a4c3ee0f9b88dc646905cfbade0c8e61e2c2"><code>d0c7a4c</code></a>
Update GitHub version number to 1.12.457-SNAPSHOT</li>
<li><a
href="https://github.com/aws/aws-sdk-java/commit/dea8f2d050934be8df527891dd080e51f06c0527"><code>dea8f2d</code></a>
AWS SDK for Java 1.12.456</li>
<li><a
href="https://github.com/aws/aws-sdk-java/commit/7b3b8d3b5e5a09d2621b00b57b05c8baa98e73c7"><code>7b3b8d3</code></a>
Update GitHub version number to 1.12.456-SNAPSHOT</li>
<li><a
href="https://github.com/aws/aws-sdk-java/commit/92a322f01b6d9467285400f26f1a6b1176885c08"><code>92a322f</code></a>
AWS SDK for Java 1.12.455</li>
<li><a
href="https://github.com/aws/aws-sdk-java/commit/d080d6c99ac991aa804803a057ab8040b724ec36"><code>d080d6c</code></a>
Update GitHub version number to 1.12.455-SNAPSHOT</li>
<li>See full diff in <a
href="https://github.com/aws/aws-sdk-java/compare/1.12.454...1.12.457">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=com.amazonaws:aws-java-sdk-sts&package-manager=gradle&previous-version=1.12.454&new-version=1.12.457)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…12.457 in /projects/control-service/projects/pipelines_control_service (#1980)

Bumps
[com.amazonaws:aws-java-sdk-core](https://github.com/aws/aws-sdk-java)
from 1.12.455 to 1.12.457.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md">com.amazonaws:aws-java-sdk-core's
changelog</a>.</em></p>
<blockquote>
<h1><strong>1.12.457</strong> <strong>2023-04-26</strong></h1>
<h2><strong>Amazon OpenSearch Ingestion</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>Initial release for OpenSearch Ingestion</li>
</ul>
</li>
</ul>
<h1><strong>1.12.456</strong> <strong>2023-04-25</strong></h1>
<h2><strong>AWS DataSync</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>This release adds 13 new APIs to support AWS DataSync Discovery
GA.</li>
</ul>
</li>
</ul>
<h2><strong>AWS Directory Service</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>New field added in AWS Managed Microsoft AD DescribeSettings
response and regex pattern update for UpdateSettings value. Added length
validation to RemoteDomainName.</li>
</ul>
</li>
</ul>
<h2><strong>Amazon Chime SDK Messaging</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>Remove non actionable field from UpdateChannelReadMarker and
DeleteChannelRequest. Add precise exceptions to DeleteChannel and
DeleteStreamingConfigurations error cases.</li>
</ul>
</li>
</ul>
<h2><strong>Amazon Connect Service</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>Amazon Connect, Contact Lens Evaluation API release including
ability to manage forms and to submit contact evaluations.</li>
</ul>
</li>
</ul>
<h2><strong>Amazon Pinpoint</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>Adds support for journey runs and querying journey execution metrics
based on journey runs. Adds execution metrics to campaign activities.
Updates docs for Advanced Quiet Time.</li>
</ul>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/aws/aws-sdk-java/commit/6306a0895b9f1849351d2365eede1163c508fb4b"><code>6306a08</code></a>
AWS SDK for Java 1.12.457</li>
<li><a
href="https://github.com/aws/aws-sdk-java/commit/d0c7a4c3ee0f9b88dc646905cfbade0c8e61e2c2"><code>d0c7a4c</code></a>
Update GitHub version number to 1.12.457-SNAPSHOT</li>
<li><a
href="https://github.com/aws/aws-sdk-java/commit/dea8f2d050934be8df527891dd080e51f06c0527"><code>dea8f2d</code></a>
AWS SDK for Java 1.12.456</li>
<li><a
href="https://github.com/aws/aws-sdk-java/commit/7b3b8d3b5e5a09d2621b00b57b05c8baa98e73c7"><code>7b3b8d3</code></a>
Update GitHub version number to 1.12.456-SNAPSHOT</li>
<li>See full diff in <a
href="https://github.com/aws/aws-sdk-java/compare/1.12.455...1.12.457">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=com.amazonaws:aws-java-sdk-core&package-manager=gradle&previous-version=1.12.455&new-version=1.12.457)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Add a link to VDK Design Principles to Contributing guide
Upgrade lineage to a 'beta' version

Testing done: locally
Bumps [com.amazonaws:aws-java-sdk-ecr](https://github.com/aws/aws-sdk-java) from 1.12.457 to 1.12.459.
- [Release notes](https://github.com/aws/aws-sdk-java/releases)
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md)
- [Commits](aws/aws-sdk-java@1.12.457...1.12.459)

---
updated-dependencies:
- dependency-name: com.amazonaws:aws-java-sdk-ecr
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Currently, there are two groups of configurations to set the vdk and job
base images used in data job deployments. The old way, through the
`datajobs.deployment.dataJobBaseImage` and `datajobs.vdk.image`, and the
new way, through the support for multiple python versions in data job
deployments.

The two groups of configurations are in conflict, as the old
configuration options take precedence and cleanly migrating to the
support for multiple python versions is practically impossible.

This change removes the default values for the
`datajobs.deployment.supportedPythonVersions` and
`datajobs.deployment.defaultPythonVersion` properties, and updates the
service logic so that when these properties are set in an environment,
they take precedence over the old configuration.

Testing Done: CI/CD

---------

Signed-off-by: Andon Andonov <andonova@vmware.com>
Co-authored-by: github-actions <>
As part of the initiative to support multiple python versions for data
job deployments, we need to make sure that users can configure their
data jobs to use specific python versions, by setting the
`python_version` property in the job's **config.ini** file.

This change introduces functionality in the vdk-control-cli client to
read the python_version configuration set in the job's config.ini file,
and deploy the job through `vdk deploy ...`

Testing Done: New and existing tests.

Signed-off-by: Andon Andonov <andonova@vmware.com>
Co-authored-by: Miroslav Ivanov <40535952+mivanov1988@users.noreply.github.com>
….9.3 in /projects/control-service/projects/pipelines_control_service (#1995)

Bumps
[org.junit.jupiter:junit-jupiter-api](https://github.com/junit-team/junit5)
from 5.9.2 to 5.9.3.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/junit-team/junit5/releases">org.junit.jupiter:junit-jupiter-api's
releases</a>.</em></p>
<blockquote>
<p>JUnit 5.9.3 = Platform 1.9.3 + Jupiter 5.9.3 + Vintage 5.9.3</p>
<p>See <a
href="http://junit.org/junit5/docs/5.9.3/release-notes/">Release
Notes</a>.</p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/junit-team/junit5/commit/bd7d03f517dfca3730fbd123c5fd4d4b70930129"><code>bd7d03f</code></a>
Release 5.9.3</li>
<li><a
href="https://github.com/junit-team/junit5/commit/2afe30a0765c55a371828b216059cb62aa673dda"><code>2afe30a</code></a>
Finalize 5.9.3 release notes</li>
<li><a
href="https://github.com/junit-team/junit5/commit/8116548985b39b8e0b24a780d6f26ed560b7b94a"><code>8116548</code></a>
Prepare release notes for 5.9.3</li>
<li><a
href="https://github.com/junit-team/junit5/commit/468958d647d8f5c3cad075ee79ee91033356ad2b"><code>468958d</code></a>
Move release note for <a
href="https://redirect.github.com/junit-team/junit5/issues/3235">#3235</a>/<a
href="https://redirect.github.com/junit-team/junit5/issues/3264">#3264</a>
to 5.9.3</li>
<li><a
href="https://github.com/junit-team/junit5/commit/6c02c060d3a2c89f481475b8142aa543e3816772"><code>6c02c06</code></a>
Polish contribution</li>
<li><a
href="https://github.com/junit-team/junit5/commit/e575ea3e9c7f3ce95c794accbd81904a1ce92599"><code>e575ea3</code></a>
Do not parse {displayName} for <a
href="https://github.com/ParameterizedTest"><code>@​ParameterizedTest</code></a>
using MessageFormat</li>
<li><a
href="https://github.com/junit-team/junit5/commit/82f1f4be2ff43b8d445f970ae8de659cab2022d5"><code>82f1f4b</code></a>
Treat overloaded local &amp; external <a
href="https://github.com/MethodSource"><code>@​MethodSource</code></a>
factory methods equally</li>
<li><a
href="https://github.com/junit-team/junit5/commit/5b4a00046bbba4d2c5f9ea48cf59e598fee37c8a"><code>5b4a000</code></a>
Polish 5.9.3 Release Notes</li>
<li><a
href="https://github.com/junit-team/junit5/commit/f6f1afff075db92010871e841cc756ad19dc720a"><code>f6f1aff</code></a>
Move release notes entry for <a
href="https://redirect.github.com/junit-team/junit5/issues/3130">#3130</a>
to 5.9.3</li>
<li><a
href="https://github.com/junit-team/junit5/commit/b732a2bd05fe2a0a3a5ff01bdf6ad241a0c9434c"><code>b732a2b</code></a>
Fix issues with <a
href="https://github.com/MethodSource"><code>@​MethodSource</code></a>
local factory method lookups</li>
<li>Additional commits viewable in <a
href="https://github.com/junit-team/junit5/compare/r5.9.2...r5.9.3">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=org.junit.jupiter:junit-jupiter-api&package-manager=gradle&previous-version=5.9.2&new-version=5.9.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Reverts #1992

---------

Co-authored-by: github-actions <>
# Why
We want to be able to change the frontend config depending on the
situation it is deployed in.
This lets users provide their own appConfig file to the frontend. 
An example of this is toggling auth on and off. 

# What
Use k8s volumemounts to load in the config file.

# How has this been tested?
Deployed the helm locally.

---------

Signed-off-by: murphp15 <murphp15@tcd.ie>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
…2.459 in /projects/control-service/projects/pipelines_control_service (#1999)

Bumps
[com.amazonaws:aws-java-sdk-sts](https://github.com/aws/aws-sdk-java)
from 1.12.457 to 1.12.459.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md">com.amazonaws:aws-java-sdk-sts's
changelog</a>.</em></p>
<blockquote>
<h1><strong>1.12.459</strong> <strong>2023-04-28</strong></h1>
<h2><strong>AWS Direct Connect</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>This release corrects the jumbo frames MTU from 9100 to 8500.</li>
</ul>
</li>
</ul>
<h2><strong>AWS IoT</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>This release allows AWS IoT Core users to specify a TLS security
policy when creating and updating AWS IoT Domain Configurations.</li>
</ul>
</li>
</ul>
<h2><strong>AWS SimSpace Weaver</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>Added a new CreateSnapshot API. For the StartSimulation API,
SchemaS3Location is now optional, added a new SnapshotS3Location
parameter. For the DescribeSimulation API, added SNAPSHOT_IN_PROGRESS
simulation state, deprecated SchemaError, added new fields: StartError
and SnapshotS3Location.</li>
</ul>
</li>
</ul>
<h2><strong>AWS WAFV2</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>You can now associate a web ACL with a Verified Access
instance.</li>
</ul>
</li>
</ul>
<h2><strong>Amazon Appflow</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>Adds Jwt Support for Salesforce Credentials.</li>
</ul>
</li>
</ul>
<h2><strong>Amazon Athena</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>You can now use capacity reservations on Amazon Athena to run SQL
queries on fully-managed compute capacity.</li>
</ul>
</li>
</ul>
<h2><strong>Amazon Elastic File System</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>This release adds PAUSED and PAUSING state as a returned value for
DescribeReplicationConfigurations response.</li>
</ul>
</li>
</ul>
<h2><strong>Amazon Managed Grafana</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>This release adds support for the grafanaVersion parameter in
CreateWorkspace.</li>
</ul>
</li>
</ul>
<h2><strong>Amazon Rekognition</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>Added support for aggregating moderation labels by video segment
timestamps for Stored Video Content Moderation APIs and added additional
information about the job to all Stored Video Get API responses.</li>
</ul>
</li>
</ul>
<h2><strong>Amazon WorkSpaces</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>Added Windows 11 to support Microsoft_Office_2019</li>
</ul>
</li>
</ul>
<h1><strong>1.12.458</strong> <strong>2023-04-27</strong></h1>
<h2><strong>AWS IoT Core Device Advisor</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>AWS IoT Core Device Advisor now supports MQTT over WebSocket. With
this update, customers can run all three test suites of AWS IoT Core
Device Advisor - qualification, custom, and long duration tests - using
Signature Version 4 for MQTT over WebSocket.</li>
</ul>
</li>
</ul>
<h2><strong>AWS Lambda</strong></h2>
<ul>
<li>
<h3>Features</h3>
<ul>
<li>Add Java 17 (java17) support to AWS Lambda</li>
</ul>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/aws/aws-sdk-java/commit/f62768449ad6c670c4fb1c688f0b83dc06340109"><code>f627684</code></a>
AWS SDK for Java 1.12.459</li>
<li><a
href="https://github.com/aws/aws-sdk-java/commit/f76ce68e6ec824381b9f7bc1f9679fd891e3199d"><code>f76ce68</code></a>
Update GitHub version number to 1.12.459-SNAPSHOT</li>
<li><a
href="https://github.com/aws/aws-sdk-java/commit/9299a3dc7214a52d002760004005021206e72c74"><code>9299a3d</code></a>
AWS SDK for Java 1.12.458</li>
<li><a
href="https://github.com/aws/aws-sdk-java/commit/e00705767289038f4f095a55005dbcb035245eef"><code>e007057</code></a>
Update GitHub version number to 1.12.458-SNAPSHOT</li>
<li>See full diff in <a
href="https://github.com/aws/aws-sdk-java/compare/1.12.457...1.12.459">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=com.amazonaws:aws-java-sdk-sts&package-manager=gradle&previous-version=1.12.457&new-version=1.12.459)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Paul Murphy <murphp15@tcd.ie>
Fix bug in function `trimArraysToNElements` in
`util-helpers.plugins.js`

Signed-off-by: gorankokin <gorankokin@gmail.com>
Bump vdk/shared minor version to `1.1.0`
Bump vdk/data-pipelines minor version to `1.1.0`

Signed-off-by: gorankokin <gorankokin@gmail.com>
clarify build steps

Co-authored-by: Miroslav Ivanov <40535952+mivanov1988@users.noreply.github.com>
@yonitoo yonitoo requested a review from hzhristova as a code owner May 3, 2023 09:03
@yonitoo yonitoo requested a review from gorankokin as a code owner May 3, 2023 09:03
yonitoo added 4 commits May 3, 2023 12:04
Signed-off-by: Yoan Salambashev <ysalambashev@vmware.com>
Signed-off-by: Yoan Salambashev <ysalambashev@vmware.com>
Signed-off-by: Yoan Salambashev <ysalambashev@vmware.com>
Copy link
Contributor

@ivakoleva ivakoleva left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reading the PR description:

Improve DAGs user-facing documentation and example based on the feedback and discussions with users

I don't understand what the user feedback was

yonitoo added 4 commits May 4, 2023 16:30
Signed-off-by: Yoan Salambashev <ysalambashev@vmware.com>
Signed-off-by: Yoan Salambashev <ysalambashev@vmware.com>
Signed-off-by: Yoan Salambashev <ysalambashev@vmware.com>
@yonitoo
Copy link
Contributor Author

yonitoo commented May 4, 2023

Reading the PR description:

Improve DAGs user-facing documentation and example based on the feedback and discussions with users

I don't understand what the user feedback was

It was not properly described. Changed the description to address this.

@yonitoo yonitoo requested a review from gabrielgeorgiev1 May 4, 2023 14:43
yonitoo and others added 5 commits May 5, 2023 12:44
Signed-off-by: Yoan Salambashev <ysalambashev@vmware.com>
Signed-off-by: Yoan Salambashev <ysalambashev@vmware.com>
Signed-off-by: Yoan Salambashev <ysalambashev@vmware.com>
Copy link
Contributor

@gabrielgeorgiev1 gabrielgeorgiev1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks great

yonitoo added 2 commits May 9, 2023 11:50
Signed-off-by: Yoan Salambashev <ysalambashev@vmware.com>
@yonitoo yonitoo enabled auto-merge (squash) May 9, 2023 08:52
@yonitoo yonitoo merged commit 289d6b7 into main May 9, 2023
@yonitoo yonitoo deleted the person/ysalambashev/vdk-dag-improve-docs-and-example branch May 9, 2023 09:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants