Release 0.2.15 (#790)

# Description

Please describe the change you have made.

## Checklist

- [ ] Tests added/updated.
- [ ] Run Demo Job Locally.
- [ ] Documentation updated.
- [ ] Changelogs updated in
[CHANGELOG.cdf-tk.md](https://github.com/cognitedata/toolkit/blob/main/CHANGELOG.cdf-tk.md).
- [ ] Template changelogs updated in
[CHANGELOG.templates.md](https://github.com/cognitedata/toolkit/blob/main/CHANGELOG.templates.md).
- [ ] Version bumped.

[_version.py](https://github.com/cognitedata/toolkit/blob/main/cognite/cognite_toolkit/_version.py)
and

[pyproject.toml](https://github.com/cognitedata/toolkit/blob/main/pyproject.toml)
per [semantic versioning](https://semver.org/).

.github/workflows/build.yml

@@ -37,8 +37,6 @@ jobs:
       - name: Linting and static code checks try 2
         if: steps.pre-commit-first-run.outcome != 'success'
         uses: pre-commit/action@v3.0.1
-
-
   test:
     name: Run tests
     runs-on: ubuntu-latest
@@ -71,9 +69,8 @@ jobs:
       - name: Run pytest
         env:
           IS_GITHUB_ACTIONS: "true"
-        run: pytest tests -n8
+        run: pytest tests/test_unit -n8
   coverage:
-    needs: test
     name: Create coverage report
     runs-on: ubuntu-latest
     environment: dev

.pre-commit-config.yaml

@@ -12,7 +12,7 @@ repos:
           - --fixable=E,W,F,I,T,RUF,TID,UP
           - --target-version=py39
       - id: ruff-format
-    rev: v0.5.1
+    rev: v0.5.3
 
   - repo: https://github.com/igorshubovych/markdownlint-cli
     rev: v0.41.0

.vscode/launch.json

@@ -109,6 +109,14 @@
       "console": "integratedTerminal",
       "justMyCode": false
     },
+    {
+        "name": "Python: TS - 3D annotation function test",
+        "type": "debugpy",
+        "request": "launch",
+        "program": "./cognite_toolkit/cognite_modules/examples/cdf_data_pipeline_3d_valhall/functions/fn_context_3d_oid_fileshare_asset/handler.py",
+        "console": "integratedTerminal",
+        "justMyCode": false
+    },
     {
       "name": "Python: init",
       "type": "debugpy",

CHANGELOG.cdf-tk.md

@@ -15,6 +15,45 @@ Changes are grouped as follows:
 - `Fixed` for any bug fixes.
 - `Security` in case of vulnerabilities.
 
+## [0.2.15] - 2024-07-22
+
+### Added
+
+- [Feature Preview] Support for uploading `3DModel` resource to CDF. Note this is the metadata about a 3D model
+  Turn on the feature by running `cdf-tk features set model-3d --enable`.
+
+### Fixed
+
+- Running `cdf-tk deploy` after a failed build would raise an incorrect `ToolkitNotADirectoryError`,
+  instead of a `ToolkitFileNotFoundError` for the `_build_enviroment.yaml` file. This is now fixed.
+- When running `cdf-tk deploy` with `Functions` that have not explicitly set `cpu`, `memory`, or `runtime`,
+  would always be classified as changed. This is now fixed.
+- [Feature Preview] When dumping assets to `csv`, headers are no longer repeated for each 1000 asset.
+- [Feature Preview] When dumping assets to `parquet`, you can now dump more than 1000 assets without
+  getting the error `TypeError: write() got an unexpected keyword argument 'mode'`.
+- [Feature Preview] When dumping assets to `parquet/csv`, the Toolkit now keeps all asset in memory until it finds
+  all unique metadata keys. This is to ensure that header is correct in the resulting `parquet/csv` file.
+- In the `config.[env].yaml`, the `name` parameter in the `environment` section is no longer required.
+  This was supposed to be remove in `0.2.0a4`.
+- If you run `cdf-tk build --env dev`, and then `cdf-tk deploy -env prod` the Toolkit will
+  now raise a `ToolkitEnvError`.
+- If you run `cdf-tk build`, the Toolkit will no longer complain about resources that exist in
+  CDF but not in the build directory (given that the Toolkit has access to CDF).
+- If you deploy a data model that already exists in CDF, the API will silently fail to update the data model if
+  there are any changes to the views in the data model. The Toolkit will now verify that the update of data models
+  was successful and raise an error if it was not.
+
+### Changed
+
+- When running `cdf-tk deploy` for a function the Toolkit checked that it could import the function code
+  before deploying the function. This is now removed. The reason is that the toolkit is often run in a
+  different Python environment than the function code. This made this check unnecessarily restrictive
+  as it would fail even though the function code was correct due to missing dependencies.
+- [Feature Preview] Instead of using `functionExternalID`+`cron` expression to identify a function schedule,
+  the Toolkit now uses `functionExternalID`+`name`. This is to avoid the Toolkit to create multiple schedules
+  for the same function if the cron expression is changed and allow to have multiple schedules with the same
+  cron expression for the same function. To enable this feature, run `cdf-tk features set fun-schedule --enable`.
+
 ## [0.2.14] - 2024-07-15
 
 ### Fixed

CHANGELOG.templates.md

@@ -15,6 +15,13 @@ Changes are grouped as follows:
 - `Fixed` for any bug fixes.
 - `Security` in case of vulnerabilities.
 
+## [0.2.15] - 2024-07-22
+
+### Added
+
+- Module `cognite_modules/example/cdf_data_pipeline_3d_valhall` 3D contextualization
+  example pipeline. Reading 3D nodes and matching to asset hierarchy
+
 ## [0.2.14] - 2024-07-15
 
 No changes to templates.

cognite_toolkit/_cdf.py

@@ -23,6 +23,16 @@
 
     setup_asset_loader.setup_asset_loader()
 
+if FeatureFlag.is_enabled(Flags.MODEL_3D):
+    from cognite_toolkit._cdf_tk.prototypes import setup_3D_loader
+
+    setup_3D_loader.setup_model_3d_loader()
+
+if FeatureFlag.is_enabled(Flags.FUN_SCHEDULE):
+    from cognite_toolkit._cdf_tk.prototypes import modify_function_schedule
+
+    modify_function_schedule.modify_function_schedule_loader()
+
 if FeatureFlag.is_enabled(Flags.ROBOTICS):
     from cognite_toolkit._cdf_tk.prototypes import setup_robotics_loaders
 
@@ -317,11 +327,19 @@ def build(
     ] = False,
 ) -> None:
     """Build configuration files from the module templates to a local build directory."""
+    ToolGlobals: Union[CDFToolConfig, None] = None
+    with contextlib.redirect_stdout(None), contextlib.suppress(Exception):
+        # Remove the Error message from failing to load the config
+        # This is verified in check_auth
+        ToolGlobals = CDFToolConfig()
+
     cmd = BuildCommand()
     if ctx.obj.verbose:
         print(ToolkitDeprecationWarning("cdf-tk --verbose", "cdf-tk build --verbose").get_message())
     cmd.run(
-        lambda: cmd.execute(verbose or ctx.obj.verbose, Path(source_dir), Path(build_dir), build_env_name, no_clean)
+        lambda: cmd.execute(
+            verbose or ctx.obj.verbose, Path(source_dir), Path(build_dir), build_env_name, no_clean, ToolGlobals
+        )
     )
 
 
@@ -336,13 +354,14 @@ def deploy(
         ),
     ] = "./build",
     build_env_name: Annotated[
-        str,
+        Optional[str],
         typer.Option(
             "--env",
             "-e",
-            help="CDF project environment to build for. Defined in environments.yaml.",
+            help="CDF project environment to use for deployment. This is optional and "
+            "if passed it is used to verify against the build environment",
         ),
-    ] = "dev",
+    ] = None,
     interactive: Annotated[
         bool,
         typer.Option(
@@ -413,13 +432,14 @@ def clean(
         ),
     ] = "./build",
     build_env_name: Annotated[
-        str,
+        Optional[str],
         typer.Option(
             "--env",
             "-e",
-            help="CDF project environment to use for cleaning.",
+            help="CDF project environment to use for cleaning. This is optional and "
+            "if passed it is used to verify against the build environment",
         ),
-    ] = "dev",
+    ] = None,
     interactive: Annotated[
         bool,
         typer.Option(

cognite_toolkit/_cdf_tk/_migration.yaml

@@ -1,8 +1,13 @@
-- version: 0.2.14
+- version: 0.2.15
   cognite_modules: {}
   resources: {}
   tool: {}
   cognite_modules_hash: ""
+- version: 0.2.14
+  cognite_modules: {}
+  resources: {}
+  tool: {}
+  cognite_modules_hash: "051b3541ad0f5504d8b58aa21d9f404b0710e5dbc97990bd7d41983f2ffb177c"
 - version: 0.2.13
   cognite_modules: {}
   resources: {}

cognite_toolkit/_cdf_tk/commands/build.py

@@ -21,6 +21,7 @@
 from rich.panel import Panel
 
 from cognite_toolkit._cdf_tk._parameters import ParameterSpecSet
+from cognite_toolkit._cdf_tk.client import ToolkitClient
 from cognite_toolkit._cdf_tk.commands._base import ToolkitCommand
 from cognite_toolkit._cdf_tk.constants import (
     _RUNNING_IN_BROWSER,
@@ -75,6 +76,7 @@
     UnknownResourceTypeWarning,
 )
 from cognite_toolkit._cdf_tk.utils import (
+    CDFToolConfig,
     calculate_str_or_file_hash,
     iterate_modules,
     module_from_path,
@@ -93,7 +95,20 @@
 
 
 class BuildCommand(ToolkitCommand):
-    def execute(self, verbose: bool, source_path: Path, build_dir: Path, build_env_name: str, no_clean: bool) -> None:
+    def __init__(self, *args: Any, **kwargs: Any) -> None:
+        super().__init__(*args, **kwargs)
+        self.existing_resources_by_loader: dict[type[ResourceLoader], set[Hashable]] = defaultdict(set)
+        self.instantiated_loaders: dict[type[ResourceLoader], ResourceLoader] = {}
+
+    def execute(
+        self,
+        verbose: bool,
+        source_path: Path,
+        build_dir: Path,
+        build_env_name: str,
+        no_clean: bool,
+        ToolGlobals: CDFToolConfig | None = None,
+    ) -> None:
         if not source_path.is_dir():
             raise ToolkitNotADirectoryError(str(source_path))
 
@@ -122,6 +137,7 @@ def execute(self, verbose: bool, source_path: Path, build_dir: Path, build_env_n
             system_config=system_config,
             clean=not no_clean,
             verbose=verbose,
+            ToolGlobals=ToolGlobals,
         )
 
     def build_config(
@@ -132,6 +148,7 @@ def build_config(
         system_config: SystemYAML,
         clean: bool = False,
         verbose: bool = False,
+        ToolGlobals: CDFToolConfig | None = None,
     ) -> dict[Path, Path]:
         is_populated = build_dir.exists() and any(build_dir.iterdir())
         if is_populated and clean:
@@ -192,7 +209,7 @@ def build_config(
             for warning in warnings:
                 print(f"    {warning.get_message()}")
 
-        state = self.process_config_files(source_dir, module_directories, build_dir, config, verbose)
+        state = self.process_config_files(source_dir, module_directories, build_dir, config, verbose, ToolGlobals)
 
         build_environment = config.create_build_environment(state.hash_by_source_path)
         build_environment.dump_to_file(build_dir)
@@ -207,6 +224,7 @@ def process_config_files(
         build_dir: Path,
         config: BuildConfigYAML,
         verbose: bool = False,
+        ToolGlobals: CDFToolConfig | None = None,
     ) -> _BuildState:
         state = _BuildState.create(config)
         for module_dir, source_paths in module_directories:
@@ -263,7 +281,7 @@ def process_config_files(
                             # Copy the file as is, not variable replacement
                             shutil.copyfile(source_path, destination)
 
-        self._check_missing_dependencies(state, project_config_dir)
+        self._check_missing_dependencies(state, project_config_dir, ToolGlobals)
         return state
 
     def _validate_function_directory(
@@ -317,18 +335,36 @@ def _replace_variables_validate_to_build_directory(
             if self.print_warning:
                 print(str(file_warnings))
 
-    def _check_missing_dependencies(self, state: _BuildState, project_config_dir: Path) -> None:
+    def _check_missing_dependencies(
+        self, state: _BuildState, project_config_dir: Path, ToolGlobals: CDFToolConfig | None = None
+    ) -> None:
         existing = {(resource_cls, id_) for resource_cls, ids in state.ids_by_resource_type.items() for id_ in ids}
         missing_dependencies = set(state.dependencies_by_required.keys()) - existing
         for resource_cls, id_ in missing_dependencies:
             if self._is_system_resource(resource_cls, id_):
                 continue
+            if ToolGlobals and self._resource_exists_in_cdf(ToolGlobals.toolkit_client, resource_cls, id_):
+                continue
             required_by = {
                 (required, path.relative_to(project_config_dir))
                 for required, path in state.dependencies_by_required[(resource_cls, id_)]
             }
             self.warn(MissingDependencyWarning(resource_cls.resource_cls.__name__, id_, required_by))
 
+    def _resource_exists_in_cdf(self, client: ToolkitClient, loader_cls: type[ResourceLoader], id_: Hashable) -> bool:
+        """Check is the resource exists in the CDF project. If there are any issues assume it does not exist."""
+        if id_ in self.existing_resources_by_loader[loader_cls]:
+            return True
+        with contextlib.suppress(Exception):
+            if loader_cls not in self.instantiated_loaders:
+                self.instantiated_loaders[loader_cls] = loader_cls(client, None)
+            loader = self.instantiated_loaders[loader_cls]
+            retrieved = loader.retrieve([id_])
+            if retrieved:
+                self.existing_resources_by_loader[loader_cls].add(id_)
+                return True
+        return False
+
     @staticmethod
     def _is_system_resource(resource_cls: type[ResourceLoader], id_: Hashable) -> bool:
         """System resources are deployed to all CDF project and should not be checked for dependencies."""

cognite_toolkit/_cdf_tk/commands/clean.py

@@ -210,7 +210,7 @@ def execute(
         self,
         ToolGlobals: CDFToolConfig,
         build_dir_raw: str,
-        build_env_name: str,
+        build_env_name: str | None,
         dry_run: bool,
         include: list[str],
         verbose: bool,

cognite_toolkit/_cdf_tk/commands/deploy.py

@@ -21,6 +21,7 @@
     ResourceCreationError,
     ResourceUpdateError,
     ToolkitDeployResourceError,
+    ToolkitFileNotFoundError,
     ToolkitNotADirectoryError,
     UploadFileError,
 )
@@ -62,7 +63,7 @@ def execute(
         self,
         ToolGlobals: CDFToolConfig,
         build_dir_raw: str,
-        build_env_name: str,
+        build_env_name: str | None,
         dry_run: bool,
         drop: bool,
         drop_data: bool,
@@ -71,28 +72,30 @@ def execute(
     ) -> None:
         # Override cluster and project from the options/env variables
         build_dir: Path = Path(build_dir_raw)
-        if not build_dir.exists():
+        if not build_dir.is_dir():
             raise ToolkitNotADirectoryError(
                 "The build directory does not exists. Did you forget to run `cdf-tk build` first?"
             )
-        build_ = BuildEnvironment.load(read_yaml_file(build_dir / BUILD_ENVIRONMENT_FILE), build_env_name, "deploy")
+        build_environment_file_path = build_dir / BUILD_ENVIRONMENT_FILE
+        if not build_environment_file_path.is_file():
+            raise ToolkitFileNotFoundError(
+                f"Could not find build environment file '{BUILD_ENVIRONMENT_FILE}' in '{build_dir}'. "
+                "Did you forget to run `cdf-tk build` first?"
+            )
+
+        build_ = BuildEnvironment.load(read_yaml_file(build_environment_file_path), build_env_name, "deploy")
+
         build_.set_environment_variables()
+
         errors = build_.check_source_files_changed()
         for error in errors:
             self.warn(error)
         if errors:
             raise ToolkitDeployResourceError(
-                "One or more source files have been modified since the last build. " "Please rebuild the project."
+                "One or more source files have been modified since the last build. Please rebuild the project."
             )
 
-        print(
-            Panel(
-                f"[bold]Deploying config files from {build_dir_raw} to environment {build_env_name}...[/]", expand=False
-            )
-        )
-
-        if not build_dir.is_dir():
-            raise ToolkitNotADirectoryError(f"'{build_dir_raw}'. Did you forget to run `cdf-tk build` first?")
+        print(Panel(f"[bold]Deploying config files from {build_dir} to environment {build_.name}...[/]", expand=False))
 
         if not _RUNNING_IN_BROWSER:
             print(ToolGlobals.as_string())

cognite_toolkit/_cdf_tk/commands/featureflag.py

@@ -22,6 +22,14 @@ class Flags(Enum):
     ASSETS: ClassVar[dict[str, Any]] = {"visible": True, "description": "Enables the support for loading assets"}
     NO_NAMING: ClassVar[dict[str, Any]] = {"visible": True, "description": "Disables the naming convention checks"}
     ROBOTICS: ClassVar[dict[str, Any]] = {"visible": False, "description": "Enables the robotics sub application"}
+    FUN_SCHEDULE: ClassVar[dict[str, Any]] = {
+        "visible": True,
+        "description": "The resource function schedule use name instead of cron to uniquely identify the schedule",
+    }
+    MODEL_3D: ClassVar[dict[str, Any]] = {
+        "visible": True,
+        "description": "Enables the support for loading 3D Model resources",
+    }
 
 
 class FeatureFlag: