Release 0.3.0a6 (#974)

# Description

Please describe the change you have made.

## Checklist

- [ ] Tests added/updated.
- [ ] Run Demo Job Locally.
- [ ] Documentation updated.
- [ ] Changelogs updated in
[CHANGELOG.cdf-tk.md](https://github.com/cognitedata/toolkit/blob/main/CHANGELOG.cdf-tk.md).
- [ ] Template changelogs updated in
[CHANGELOG.templates.md](https://github.com/cognitedata/toolkit/blob/main/CHANGELOG.templates.md).
- [ ] Version bumped.

[_version.py](https://github.com/cognitedata/toolkit/blob/main/cognite/cognite_toolkit/_version.py)
and

[pyproject.toml](https://github.com/cognitedata/toolkit/blob/main/pyproject.toml)
per [semantic versioning](https://semver.org/).

.gitignore

@@ -301,3 +301,5 @@ tests/data/run_data/build_info.dev.yaml
 orgdir*
 module_upgrade/cdf.toml
 my_org
+# Org used by bootcamp
+ice-cream-dataops

CHANGELOG.cdf-tk.md

@@ -15,6 +15,18 @@ Changes are grouped as follows:
 - `Fixed` for any bug fixes.
 - `Security` in case of vulnerabilities.
 
+## [0.3.0a6] - 2024-09-20
+
+### Added
+
+- Added flag `--modules`/`-m` to select which modules to build in the `cdf build` command.
+- The `cdf build` command no longer requires `config.[env].yaml` to be present. If it is not present, the Toolkit
+  will use the default values for the environment.
+
+### Fixed
+
+- If you removed `metadata` in any resource file, the Toolkit would not update this change in CDF. This is now fixed.
+
 ## [0.3.0a5] - 2024-09-18
 
 ### Added
@@ -25,6 +37,8 @@ Changes are grouped as follows:
   [CAUTION] The current implementation will always redeploy the source, even if it has not changed. This will be
   fixed in a future release.
 - Added support for resource type `Sequence` in the `classic` folder.
+- Added parameter `--selected` to the `cdf build` command to overwrite the `selected` in the `config.[env].yaml` file.
+- Made the `config.[env].yaml` optional when running the `cdf build command.
 
 ### Changed
 

CHANGELOG.templates.md

@@ -15,6 +15,10 @@ Changes are grouped as follows:
 - `Fixed` for any bug fixes.
 - `Security` in case of vulnerabilities.
 
+## [0.3.0a6] - 2024-09-20
+
+Added `bootcamp` modules `ice_cream_api` and `oee`.
+
 ## [0.3.0a5] - 2024-09-18
 
 No changes to templates.

cdf.toml

@@ -4,17 +4,22 @@
 # when you log in to Cognite Data Fusion.
 # Note any string without white-space is allowed.
 default_organization_dir = "cognite_toolkit"
-default_env = "dev"
+#default_env = "dev"
 
-[cdf.feature_flags]
+[feature_flags]
 graphql = true
 asset-dump = true
 timeseries-dump = true
 import-cmd = true
 modules-app = true
 hosted-extractors = true
 
+[plugins]
+run = true
+pull = true
+dump = true
+
 [modules]
 # This is the version of the modules. It should not be changed manually.
 # It will be updated by the 'cdf module upgrade' command.
-version = "0.3.0a5"
+version = "0.3.0a6"

cognite_toolkit/_builtin_modules/bootcamp/README.md

@@ -0,0 +1,6 @@
+# Bootcamp module
+
+These are the initial configs and python functions used for the bootcamp.
+[Docs](https://docs.cdf-bootcamp.cogniteapp.com/)
+[Repo](https://github.com/cognitedata/cognite-data-fusion-bootcamp)
+[Configs repo](https://github.com/cognitedata/bootcamp-toolkit-config)

cognite_toolkit/_builtin_modules/bootcamp/default.config.yaml

@@ -0,0 +1,27 @@
+CDF_PROJECT: <YOUR CDF PROJECT> # ex. cdf-bootcamp-01-test
+IDP_CLIENT_ID: ${IDP_CLIENT_ID} # replaced by .env
+IDP_CLIENT_SECRET: ${IDP_CLIENT_SECRET} # replaced by .env
+IDP_TENANT_ID: ${IDP_TENANT_ID} # replaced by .env
+ # This is related to the cluster where the CDF project is hosted.
+CDF_URL: https://westeurope-1.cognitedata.com
+IDP_SCOPES: https://westeurope-1.cognitedata.com/.default
+IDP_TOKEN_URL: ${IDP_TOKEN_URL} # replaced by .env
+
+# Groups
+# OBJECT ID FOR TEST/PROD DATA DEVELOPER ENTRA GROUP
+data_developer_source_id: <NotSet>
+# OBJECT ID FOR TEST/PROD DATA PIPELINE OEE ENTRA GROUP
+data_pipeline_oee_source_id: <NotSet>
+# OBJECT ID FOR TEST/PROD ADMIN TK ENTRA GROUP
+readwrite_source_id: <NotSet>
+# OBJECT ID FOR TEST/PROD ADMIN TK ENTRA GROUP
+readonly_source_id: <NotSet>
+# OBJECT ID FOR TEST/PROD ADMIN TK ENTRA GROUP
+icapi_extractors_source_id: <NotSet>
+
+# Client Ids
+data_pipeline_oee_client_id: ${DATA_PIPELINE_OEE_CLIENT_ID} # replaced by .env
+data_pipeline_oee_client_secret: ${DATA_PIPELINE_OEE_CLIENT_SECRET} # replaced by .env
+
+icapi_extractors_client_id: ${ICAPI_EXTRACTORS_CLIENT_ID} # replaced by .env
+icapi_extractors_client_secret: ${ICAPI_EXTRACTORS_CLIENT_SECRET} # replaced by .env

cognite_toolkit/_builtin_modules/bootcamp/ice_cream_api/data_sets/icapi.DataSet.yaml

@@ -0,0 +1,3 @@
+externalId: {{ icapi_ds_external_id }}
+name: Ice Cream API
+description: Data set for the Ice Cream API

cognite_toolkit/_builtin_modules/bootcamp/ice_cream_api/default.config.yaml

@@ -0,0 +1 @@
+icapi_ds_external_id: ds_icapi

cognite_toolkit/_builtin_modules/bootcamp/ice_cream_api/extraction_pipelines/ep_icapi_assets.yaml

@@ -0,0 +1,8 @@
+externalId: ep_icapi_assets
+name: Ice Cream API Assets
+description: Assets source extraction pipeline for the Ice Cream API
+rawTables:
+  - dbName: ice_cream_api
+    tableName: icapi_assets_extractor
+source: Ice Cream API
+dataSetExternalId: {{ icapi_ds_external_id }}

cognite_toolkit/_builtin_modules/bootcamp/ice_cream_api/extraction_pipelines/ep_icapi_datapoints.yaml

@@ -0,0 +1,8 @@
+externalId: ep_icapi_datapoints
+name: Ice Cream API Data Points
+description: Data Points source extraction pipeline for the Ice Cream API
+rawTables:
+  - dbName: State Store
+    tableName: icapi_datapoints_extractor
+source: Ice Cream API
+dataSetExternalId: {{ icapi_ds_external_id }}

cognite_toolkit/_builtin_modules/bootcamp/ice_cream_api/extraction_pipelines/ep_icapi_timeseries.yaml

@@ -0,0 +1,5 @@
+externalId: ep_icapi_timeseries
+name: Ice Cream API Time Series
+description: Time Series source extraction pipeline for the Ice Cream API
+source: Ice Cream API
+dataSetExternalId: {{ icapi_ds_external_id }}

cognite_toolkit/_builtin_modules/bootcamp/ice_cream_api/functions/functions.yaml

@@ -0,0 +1,45 @@
+- name: Ice Cream API Assets Extractor
+  externalId: icapi_assets_extractor
+  owner: Bootcamp Team
+  description: Extracts Asset data from the Ice Cream API
+  metadata:
+    version: "1.0"
+  runtime: py311
+  functionPath: ./handler.py
+  envVars:
+    CDF_PROJECT: {{ CDF_PROJECT }}
+    CDF_URL: {{ CDF_URL }}
+    IDP_TENANT_ID: {{ IDP_TENANT_ID }}
+    IDP_CLIENT_ID: {{ icapi_extractors_client_id }}
+    IDP_CLIENT_SECRET: {{ icapi_extractors_client_secret }}
+    IDP_SCOPES: {{ IDP_SCOPES }}
+- name: Ice Cream API DataPoints Extractor
+  externalId: icapi_datapoints_extractor
+  owner: Bootcamp Team
+  description: Extracts DataPoints data from the Ice Cream API
+  metadata:
+    version: "1.0"
+  runtime: py311
+  functionPath: ./handler.py
+  envVars:
+    CDF_PROJECT: {{ CDF_PROJECT }}
+    CDF_URL: {{ CDF_URL }}
+    IDP_TENANT_ID: {{ IDP_TENANT_ID }}
+    IDP_CLIENT_ID: {{ icapi_extractors_client_id }}
+    IDP_CLIENT_SECRET: {{ icapi_extractors_client_secret }}
+    IDP_SCOPES: {{ IDP_SCOPES }}
+- name: Ice Cream API TimeSeries Extractor
+  externalId: icapi_timeseries_extractor
+  owner: Bootcamp Team
+  description: Extracts TimeSeries data from the Ice Cream API
+  metadata:
+    version: "1.0"
+  runtime: py311
+  functionPath: ./handler.py
+  envVars:
+    CDF_PROJECT: {{ CDF_PROJECT }}
+    CDF_URL: {{ CDF_URL }}
+    IDP_TENANT_ID: {{ IDP_TENANT_ID }}
+    IDP_CLIENT_ID: {{ icapi_extractors_client_id }}
+    IDP_CLIENT_SECRET: {{ icapi_extractors_client_secret }}
+    IDP_SCOPES: {{ IDP_SCOPES }}

cognite_toolkit/_builtin_modules/bootcamp/ice_cream_api/functions/icapi_assets_extractor/config.py

@@ -0,0 +1,18 @@
+from dataclasses import dataclass
+
+from cognite.extractorutils.configtools import BaseConfig
+
+
+@dataclass
+class DestConfig:
+    database: str
+    table: str
+
+@dataclass
+class ExtractorConfig:
+    api_url: str
+    dest: DestConfig
+
+@dataclass
+class Config(BaseConfig):
+    extractor: ExtractorConfig

cognite_toolkit/_builtin_modules/bootcamp/ice_cream_api/functions/icapi_assets_extractor/extractor_config.yaml

@@ -0,0 +1,26 @@
+extractor:
+  dest:
+    database: ice_cream_api
+    table: icapi_assets_extractor
+
+  api_url: https://ice-cream-factory.inso-internal.cognite.ai
+
+cognite:
+  host: ${CDF_URL}
+  project: ${CDF_PROJECT}
+
+  idp-authentication:
+    tenant: ${IDP_TENANT_ID}
+
+    client-id: ${IDP_CLIENT_ID}
+    secret: ${IDP_CLIENT_SECRET}
+    scopes:
+      - ${IDP_SCOPES}
+
+  extraction_pipeline:
+    external-id: ep_icapi_assets
+
+logger:
+  console:
+    level: INFO
+type: local

cognite_toolkit/_builtin_modules/bootcamp/ice_cream_api/functions/icapi_assets_extractor/handler.py

@@ -0,0 +1,45 @@
+from io import StringIO
+from threading import Event
+
+import pandas
+from cognite.client import CogniteClient
+from cognite.extractorutils import Extractor
+from cognite.extractorutils.statestore import AbstractStateStore
+from config import Config
+from ice_cream_factory_api import IceCreamFactoryAPI
+
+
+def run_extractor(
+    client: CogniteClient, states: AbstractStateStore, config: Config, stop_event: Event
+) -> None:
+    ice_cream_api = IceCreamFactoryAPI(base_url=config.extractor.api_url)
+
+    sites_csv = ice_cream_api.get_sites_csv()
+    sites_df = pandas.read_csv(
+        StringIO(sites_csv),
+        sep=",",
+        usecols=["name", "external_id", "description", "metadata", "parent_external_id"]
+    )
+
+    client.raw.rows.insert_dataframe(
+        dataframe=sites_df,
+        db_name=config.extractor.dest.database,
+        table_name=config.extractor.dest.table,
+        ensure_parent=True
+    )
+
+def handle(client: CogniteClient = None, data = None):
+    if data:
+        config_file_path = data.get("config_file_path", "extractor_config.yaml")
+    else:
+        config_file_path = "extractor_config.yaml"
+
+    with Extractor(
+        name="icapi_assets_extractor",
+        description="An extractor that ingest Assets from the Ice Cream Factory API to CDF clean",
+        config_class=Config,
+        version="1.0",
+        config_file_path=config_file_path,
+        run_handle=run_extractor,
+    ) as extractor:
+        extractor.run()

cognite_toolkit/_builtin_modules/bootcamp/ice_cream_api/functions/icapi_assets_extractor/ice_cream_factory_api.py

@@ -0,0 +1,37 @@
+from typing import Dict, Union
+
+from requests import Response, Session, adapters
+
+
+class IceCreamFactoryAPI:
+    """Class for Ice Cream Factory API."""
+
+    def __init__(self, base_url: str):
+        self.base_url = base_url
+        self.adapter = adapters.HTTPAdapter(max_retries=3)
+        self.session = Session()
+        self.session.mount("https://", self.adapter)
+
+    def get_response(
+        self, headers: Dict[str, str], url_suffix: str, params: Dict[str, Union[str, int, float]] = {}
+    ) -> Response:
+        """
+        Get response from API.
+
+        Args:
+            headers: request header
+            url_suffix: string to add to base url
+            params: query parameters
+        """
+
+        response = self.session.get(f"{self.base_url}/{url_suffix}", headers=headers, timeout=40, params=params)
+        response.raise_for_status()
+        return response
+
+    def get_sites_csv(self):
+        """
+        Get a dataframe for all sites from the Ice Cream API's site/{city}/csv endpoint
+        """
+        response = self.get_response(headers={}, url_suffix="site/all/csv")
+
+        return response.text

cognite_toolkit/_builtin_modules/bootcamp/ice_cream_api/functions/icapi_assets_extractor/requirements.txt

@@ -0,0 +1,3 @@
+cognite-extractor-utils >= 7.2.2
+orjson >= 3.10.5
+pandas

cognite_toolkit/_builtin_modules/bootcamp/ice_cream_api/functions/icapi_datapoints_extractor/config.py

@@ -0,0 +1,22 @@
+from dataclasses import dataclass, field
+from typing import List
+
+from cognite.extractorutils.configtools import BaseConfig, RawStateStoreConfig, StateStoreConfig
+
+
+@dataclass
+class ExtractorConfig:
+    api_url: str
+    backfill: bool
+    data_set_ext_id: str
+    hours: int
+    sites: List[str]
+    state_store: StateStoreConfig = field(
+        default_factory=StateStoreConfig(
+            raw=RawStateStoreConfig(database=None, table=None)
+        )
+    )
+
+@dataclass
+class Config(BaseConfig):
+    extractor: ExtractorConfig

cognite_toolkit/_builtin_modules/bootcamp/ice_cream_api/functions/icapi_datapoints_extractor/extractor_config.yaml

@@ -0,0 +1,35 @@
+cognite:
+  extraction_pipeline:
+    external-id: ep_icapi_datapoints
+  host: ${CDF_URL}
+  idp-authentication:
+    client-id: ${IDP_CLIENT_ID}
+    scopes:
+    - ${IDP_SCOPES}
+    secret: ${IDP_CLIENT_SECRET}
+    tenant: ${IDP_TENANT_ID}
+  project: ${CDF_PROJECT}
+extractor:
+  api_url: https://ice-cream-factory.inso-internal.cognite.ai
+  backfill: false
+  data_set_ext_id: ds_icapi
+  hours: 1
+  sites:
+  - Houston
+  - Oslo
+  - Kuala Lumpur
+  - Hannover
+  - Nuremberg
+  - Marseille
+  - Sao Paulo
+  - Chicago
+  - Rotterdam
+  - London
+  state_store:
+    raw:
+      database: State Store
+      table: icapi_datapoints_extractor
+logger:
+  console:
+    level: INFO
+type: local