feat: Upgrading python version to 3.11, adding support for 3.11 as we…

…ll. (feast-dev#4159) * Upgrading python version to 3.11, adding support for 3.11 as well. Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * Upgrading python version to 3.11, adding support for 3.11 as well. Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * chore: Bump macOS runners to macos-13 (feast-dev#4152) bump macos runner to 13 Signed-off-by: tokoko <togurg14@freeuni.edu.ge> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * chore: Use pixi to lock python dependencies in a single command (feast-dev#4114) use pixi to lock python dependencies in a single command Signed-off-by: tokoko <togurg14@freeuni.edu.ge> Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * Trying to fix the lint error after python upgrade. - error: Call to abstract method "__init__" of "Provider" with trivial body via super() is unsafe [safe-super] Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> * Adding only the integration tests to run on 3.11 Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> --------- Signed-off-by: Lokesh Rangineni <lokeshforjava@gmail.com> Co-authored-by: Tornike Gurgenidze <togurg14@freeuni.edu.ge>
tmihalac · May 3, 2024 · addf683 · addf683
1 parent 3a64aa8
commit addf683
Show file tree

Hide file tree

Showing 27 changed files with 1,298 additions and 175 deletions.
diff --git a/.github/fork_workflows/fork_pr_integration_tests_aws.yml b/.github/fork_workflows/fork_pr_integration_tests_aws.yml
@@ -33,21 +33,21 @@ jobs:
         id: image-tag
         run: echo "::set-output name=DOCKER_IMAGE_TAG::`git rev-parse HEAD`"
       - name: Cache Public ECR Image
-        id: lambda_python_3_9
+        id: lambda_python_3_11
         uses: actions/cache@v2
         with:
           path: ~/cache
-          key: lambda_python_3_9
+          key: lambda_python_3_11
       - name: Handle Cache Miss (pull public ECR image & save it to tar file)
         if: steps.cache-primes.outputs.cache-hit != 'true'
         run: |
           mkdir -p ~/cache
-          docker pull public.ecr.aws/lambda/python:3.9
-          docker save public.ecr.aws/lambda/python:3.9 -o ~/cache/lambda_python_3_9.tar
+          docker pull public.ecr.aws/lambda/python:3.11
+          docker save public.ecr.aws/lambda/python:3.11 -o ~/cache/lambda_python_3_11.tar
       - name: Handle Cache Hit (load docker image from tar file)
         if: steps.cache-primes.outputs.cache-hit == 'true'
         run: |
-          docker load -i ~/cache/lambda_python_3_9.tar
+          docker load -i ~/cache/lambda_python_3_11.tar
       - name: Build and push
         env:
           ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
@@ -67,7 +67,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: [ "3.9" ]
+        python-version: [ "3.11" ]
         os: [ ubuntu-latest ]
     env:
       OS: ${{ matrix.os }}

diff --git a/.github/fork_workflows/fork_pr_integration_tests_gcp.yml b/.github/fork_workflows/fork_pr_integration_tests_gcp.yml
@@ -9,7 +9,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: [ "3.9" ]
+        python-version: [ "3.11" ]
         os: [ ubuntu-latest ]
     env:
       OS: ${{ matrix.os }}

diff --git a/.github/fork_workflows/fork_pr_integration_tests_snowflake.yml b/.github/fork_workflows/fork_pr_integration_tests_snowflake.yml
@@ -9,7 +9,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: [ "3.9" ]
+        python-version: [ "3.11" ]
         os: [ ubuntu-latest ]
     env:
       OS: ${{ matrix.os }}

diff --git a/.github/workflows/build_wheels.yml b/.github/workflows/build_wheels.yml
@@ -59,7 +59,7 @@ jobs:
       - name: Setup Python
         uses: actions/setup-python@v3
         with:
-          python-version: "3.9"
+          python-version: "3.11"
           architecture: x64
       - name: Setup Node
         uses: actions/setup-node@v3
@@ -86,7 +86,7 @@ jobs:
         id: setup-python
         uses: actions/setup-python@v3
         with:
-          python-version: "3.10"
+          python-version: "3.11"
           architecture: x64
       - name: Setup Node
         uses: actions/setup-node@v3
@@ -137,7 +137,7 @@ jobs:
     strategy:
       matrix:
         os: [ubuntu-latest,  macos-13 ]
-        python-version: ["3.9", "3.10"]
+        python-version: ["3.9", "3.10", "3.11"]
         from-source: [ True, False ]
     env:
       # this script is for testing servers

diff --git a/.github/workflows/java_master_only.yml b/.github/workflows/java_master_only.yml
@@ -25,7 +25,7 @@ jobs:
         uses: actions/setup-python@v3
         id: setup-python
         with:
-          python-version: "3.9"
+          python-version: "3.11"
           architecture: x64
       - name: Authenticate to Google Cloud
         uses: 'google-github-actions/auth@v1'
@@ -95,7 +95,7 @@ jobs:
     if: github.repository == 'feast-dev/feast'
     runs-on: ubuntu-latest
     env:
-      PYTHON: 3.9
+      PYTHON: 3.11
     steps:
       - uses: actions/checkout@v4
         with:
@@ -110,7 +110,7 @@ jobs:
         uses: actions/setup-python@v3
         id: setup-python
         with:
-          python-version: 3.9
+          python-version: 3.11
           architecture: x64
       - name: Get pip cache dir
         id: pip-cache

diff --git a/.github/workflows/java_pr.yml b/.github/workflows/java_pr.yml
@@ -88,7 +88,7 @@ jobs:
         uses: actions/setup-python@v3
         id: setup-python
         with:
-          python-version: "3.9"
+          python-version: "3.11"
           architecture: x64
       - name: Authenticate to Google Cloud
         uses: 'google-github-actions/auth@v1'
@@ -111,7 +111,7 @@ jobs:
     runs-on: ubuntu-latest
     needs: unit-test-java
     env:
-      PYTHON: 3.9
+      PYTHON: 3.11
     steps:
       - uses: actions/checkout@v4
         with:
@@ -128,7 +128,7 @@ jobs:
           architecture: x64
       - uses: actions/setup-python@v3
         with:
-          python-version: '3.9'
+          python-version: '3.11'
           architecture: 'x64'
       - uses: actions/cache@v2
         with:
@@ -158,7 +158,7 @@ jobs:
         uses: actions/setup-python@v3
         id: setup-python
         with:
-          python-version: 3.9
+          python-version: 3.11
           architecture: x64
       - name: Get pip cache dir
         id: pip-cache

diff --git a/.github/workflows/linter.yml b/.github/workflows/linter.yml
@@ -6,14 +6,14 @@ jobs:
   lint-python:
     runs-on: [ubuntu-latest]
     env:
-      PYTHON: 3.9
+      PYTHON: 3.11
     steps:
       - uses: actions/checkout@v4
       - name: Setup Python
         id: setup-python
         uses: actions/setup-python@v5
         with:
-          python-version: "3.9"
+          python-version: "3.11"
           architecture: x64
       - name: Upgrade pip version
         run: |

diff --git a/.github/workflows/master_only.yml b/.github/workflows/master_only.yml
@@ -30,21 +30,21 @@ jobs:
         id: image-tag
         run: echo "::set-output name=DOCKER_IMAGE_TAG::`git rev-parse HEAD`"
       - name: Cache Public ECR Image
-        id: lambda_python_3_9
+        id: lambda_python_3_11
         uses: actions/cache@v2
         with:
           path: ~/cache
-          key: lambda_python_3_9
+          key: lambda_python_3_11
       - name: Handle Cache Miss (pull public ECR image & save it to tar file)
         if: steps.cache-primes.outputs.cache-hit != 'true'
         run: |
           mkdir -p ~/cache
-          docker pull public.ecr.aws/lambda/python:3.9
-          docker save public.ecr.aws/lambda/python:3.9 -o ~/cache/lambda_python_3_9.tar
+          docker pull public.ecr.aws/lambda/python:3.11
+          docker save public.ecr.aws/lambda/python:3.11 -o ~/cache/lambda_python_3_11.tar
       - name: Handle Cache Hit (load docker image from tar file)
         if: steps.cache-primes.outputs.cache-hit == 'true'
         run: |
-          docker load -i ~/cache/lambda_python_3_9.tar
+          docker load -i ~/cache/lambda_python_3_11.tar
       - name: Build and push
         env:
           ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
@@ -65,7 +65,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.9", "3.10" ]
+        python-version: ["3.9", "3.10", "3.11"]
         os: [ ubuntu-latest ]
     env:
       OS: ${{ matrix.os }}

diff --git a/.github/workflows/nightly-ci.yml b/.github/workflows/nightly-ci.yml
@@ -36,7 +36,7 @@ jobs:
         uses: actions/setup-python@v5
         id: setup-python
         with:
-          python-version: "3.9"
+          python-version: "3.11"
           architecture: x64
       - name: Set up AWS SDK
         uses: aws-actions/configure-aws-credentials@v1
@@ -89,21 +89,21 @@ jobs:
         id: image-tag
         run: echo "::set-output name=DOCKER_IMAGE_TAG::`git rev-parse HEAD`"
       - name: Cache Public ECR Image
-        id: lambda_python_3_9
+        id: lambda_python_3_11
         uses: actions/cache@v4
         with:
           path: ~/cache
-          key: lambda_python_3_9
+          key: lambda_python_3_11
       - name: Handle Cache Miss (pull public ECR image & save it to tar file)
-        if: steps.lambda_python_3_9.outputs.cache-hit != 'true'
+        if: steps.lambda_python_3_11.outputs.cache-hit != 'true'
         run: |
           mkdir -p ~/cache
-          docker pull public.ecr.aws/lambda/python:3.9
-          docker save public.ecr.aws/lambda/python:3.9 -o ~/cache/lambda_python_3_9.tar
+          docker pull public.ecr.aws/lambda/python:3.11
+          docker save public.ecr.aws/lambda/python:3.11 -o ~/cache/lambda_python_3_11.tar
       - name: Handle Cache Hit (load docker image from tar file)
-        if: steps.lambda_python_3_9.outputs.cache-hit == 'true'
+        if: steps.lambda_python_3_11.outputs.cache-hit == 'true'
         run: |
-          docker load -i ~/cache/lambda_python_3_9.tar
+          docker load -i ~/cache/lambda_python_3_11.tar
       - name: Build and push
         env:
           ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
@@ -124,7 +124,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: [ "3.9" ]
+        python-version: [ "3.11" ]
         os: [ ubuntu-latest ]
     env:
       OS: ${{ matrix.os }}

diff --git a/.github/workflows/pr_integration_tests.yml b/.github/workflows/pr_integration_tests.yml
@@ -47,21 +47,21 @@ jobs:
         id: image-tag
         run: echo "::set-output name=DOCKER_IMAGE_TAG::`git rev-parse HEAD`"
       - name: Cache Public ECR Image
-        id: lambda_python_3_9
+        id: lambda_python_3_11
         uses: actions/cache@v2
         with:
           path: ~/cache
-          key: lambda_python_3_9
+          key: lambda_python_3_11
       - name: Handle Cache Miss (pull public ECR image & save it to tar file)
         if: steps.cache-primes.outputs.cache-hit != 'true'
         run: |
           mkdir -p ~/cache
-          docker pull public.ecr.aws/lambda/python:3.9
-          docker save public.ecr.aws/lambda/python:3.9 -o ~/cache/lambda_python_3_9.tar
+          docker pull public.ecr.aws/lambda/python:3.11
+          docker save public.ecr.aws/lambda/python:3.11 -o ~/cache/lambda_python_3_11.tar
       - name: Handle Cache Hit (load docker image from tar file)
         if: steps.cache-primes.outputs.cache-hit == 'true'
         run: |
-          docker load -i ~/cache/lambda_python_3_9.tar
+          docker load -i ~/cache/lambda_python_3_11.tar
       - name: Build and push
         env:
           ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
@@ -86,7 +86,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: [ "3.10" ]
+        python-version: [ "3.9", "3.10", "3.11" ]
         os: [ ubuntu-latest ]
     env:
       OS: ${{ matrix.os }}

diff --git a/.github/workflows/pr_local_integration_tests.yml b/.github/workflows/pr_local_integration_tests.yml
@@ -19,7 +19,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: [ "3.10" ]
+        python-version: [ "3.11" ]
         os: [ ubuntu-latest ]
     env:
       OS: ${{ matrix.os }}

diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml
@@ -160,7 +160,7 @@ jobs:
           architecture: x64
       - uses: actions/setup-python@v3
         with:
-          python-version: '3.9'
+          python-version: '3.11'
           architecture: 'x64'
       - uses: actions/cache@v2
         with:

diff --git a/.github/workflows/unit_tests.yml b/.github/workflows/unit_tests.yml
@@ -7,7 +7,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: [ "3.9", "3.10" ]
+        python-version: [ "3.9", "3.10", "3.11"]
         os: [ ubuntu-latest, macos-13 ]
         exclude:
           - os: macos-13

diff --git a/Makefile b/Makefile
@@ -357,7 +357,7 @@ kill-trino-locally:
 	cd ${ROOT_DIR}; docker stop trino
 
 install-protoc-dependencies:
-	pip install --ignore-installed protobuf==4.23.4 "grpcio-tools>=1.56.2,<2" mypy-protobuf==3.1.0
+	pip install --ignore-installed protobuf==4.24.0 "grpcio-tools>=1.56.2,<2" mypy-protobuf==3.1.0
 
 install-feast-ci-locally:
 	pip install -e ".[ci]"

diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
@@ -106,8 +106,7 @@
   * [Google Cloud Platform](reference/providers/google-cloud-platform.md)
   * [Amazon Web Services](reference/providers/amazon-web-services.md)
   * [Azure](reference/providers/azure.md)
-* [Batch Materialization Engines](reference/batch-materialization/README.md)
-  * [Bytewax](reference/batch-materialization/bytewax.md)
+* [Batch Materialization Engines](reference/batch-materialization/README.md)  
   * [Snowflake](reference/batch-materialization/snowflake.md)
   * [AWS Lambda (alpha)](reference/batch-materialization/lambda.md)
   * [Spark (contrib)](reference/batch-materialization/spark.md)

diff --git a/docs/how-to-guides/running-feast-in-production.md b/docs/how-to-guides/running-feast-in-production.md
@@ -57,28 +57,9 @@ To keep your online store up to date, you need to run a job that loads feature d
 Out of the box, Feast's materialization process uses an in-process materialization engine. This engine loads all the data being materialized into memory from the offline store, and writes it into the online store. 
 
 This approach may not scale to large amounts of data, which users of Feast may be dealing with in production.
-In this case, we recommend using one of the more [scalable materialization engines](./scaling-feast.md#scaling-materialization), such as the [Bytewax Materialization Engine](../reference/batch-materialization/bytewax.md), or the [Snowflake Materialization Engine](../reference/batch-materialization/snowflake.md).
+In this case, we recommend using one of the more [scalable materialization engines](./scaling-feast.md#scaling-materialization), such as [Snowflake Materialization Engine](../reference/batch-materialization/snowflake.md).
 Users may also need to [write a custom materialization engine](../how-to-guides/customizing-feast/creating-a-custom-materialization-engine.md) to work on their existing infrastructure.  
 
-The Bytewax materialization engine can run materialization on an existing Kubernetes cluster. An example configuration of this in a `feature_store.yaml` is as follows:
-
-```yaml
-batch_engine:
-  type: bytewax
-  namespace: bytewax
-  image: bytewax/bytewax-feast:latest
-  env:
-    - name: AWS_ACCESS_KEY_ID
-      valueFrom:
-        secretKeyRef:
-          name: aws-credentials
-          key: aws-access-key-id
-    - name: AWS_SECRET_ACCESS_KEY
-      valueFrom:
-        secretKeyRef:
-          name: aws-credentials
-          key: aws-secret-access-key
-```
 
 ### 2.2 Scheduled materialization with Airflow