Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add ibis-based duckdb offline store #3822

Closed
wants to merge 2 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/auto_assign.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ assignees:
- woop
- tsotnet
- achals
- adchia
- felixwang9817

# A number of assignees to add to the pull request
Expand Down
10 changes: 5 additions & 5 deletions .github/fork_workflows/fork_pr_integration_tests_aws.yml
Original file line number Diff line number Diff line change
Expand Up @@ -83,15 +83,15 @@ jobs:
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
with:
# pull_request_target runs the workflow in the context of the base repo
# as such actions/checkout needs to be explicit configured to retrieve
# code from the PR.
ref: refs/pull/${{ github.event.pull_request.number }}/merge
submodules: recursive
- name: Setup Python
uses: actions/setup-python@v2
uses: actions/setup-python@v3
id: setup-python
with:
python-version: ${{ matrix.python-version }}
Expand All @@ -109,9 +109,6 @@ jobs:
aws-region: us-west-2
- name: Use AWS CLI
run: aws sts get-caller-identity
- name: Upgrade pip version
run: |
pip install --upgrade "pip>=21.3.1,<22.1"
- name: Get pip cache dir
id: pip-cache
run: |
Expand All @@ -126,6 +123,9 @@ jobs:
key: ${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-${{ hashFiles(format('**/py{0}-ci-requirements.txt', env.PYTHON)) }}
restore-keys: |
${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-
- name: Upgrade pip version
run: |
pip install --upgrade "pip>=21.3.1,<22.3"
- name: Install pip-tools
run: pip install pip-tools
- name: Install apache-arrow on ubuntu
Expand Down
10 changes: 5 additions & 5 deletions .github/fork_workflows/fork_pr_integration_tests_gcp.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,15 +25,15 @@ jobs:
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
with:
# pull_request_target runs the workflow in the context of the base repo
# as such actions/checkout needs to be explicit configured to retrieve
# code from the PR.
ref: refs/pull/${{ github.event.pull_request.number }}/merge
submodules: recursive
- name: Setup Python
uses: actions/setup-python@v2
uses: actions/setup-python@v3
id: setup-python
with:
python-version: ${{ matrix.python-version }}
Expand All @@ -53,9 +53,6 @@ jobs:
project_id: ${{ secrets.GCP_PROJECT_ID }}
- name: Use gcloud CLI
run: gcloud info
- name: Upgrade pip version
run: |
pip install --upgrade "pip>=21.3.1,<22.1"
- name: Get pip cache dir
id: pip-cache
run: |
Expand All @@ -70,6 +67,9 @@ jobs:
key: ${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-${{ hashFiles(format('**/py{0}-ci-requirements.txt', env.PYTHON)) }}
restore-keys: |
${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-
- name: Upgrade pip version
run: |
pip install --upgrade "pip>=21.3.1,<23.2"
- name: Install pip-tools
run: pip install pip-tools
- name: Install apache-arrow on ubuntu
Expand Down
11 changes: 5 additions & 6 deletions .github/fork_workflows/fork_pr_integration_tests_snowflake.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,15 +25,15 @@ jobs:
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
with:
# pull_request_target runs the workflow in the context of the base repo
# as such actions/checkout needs to be explicit configured to retrieve
# code from the PR.
ref: refs/pull/${{ github.event.pull_request.number }}/merge
submodules: recursive
- name: Setup Python
uses: actions/setup-python@v2
uses: actions/setup-python@v3
id: setup-python
with:
python-version: ${{ matrix.python-version }}
Expand All @@ -43,10 +43,6 @@ jobs:
uses: actions/setup-go@v2
with:
go-version: 1.18.0

- name: Upgrade pip version
run: |
pip install --upgrade "pip>=21.3.1,<22.1"
- name: Get pip cache dir
id: pip-cache
run: |
Expand All @@ -61,6 +57,9 @@ jobs:
key: ${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-${{ hashFiles(format('**/py{0}-ci-requirements.txt', env.PYTHON)) }}
restore-keys: |
${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-
- name: Upgrade pip version
run: |
pip install --upgrade "pip>=21.3.1,<23.2"
- name: Install pip-tools
run: pip install pip-tools
- name: Install apache-arrow on ubuntu
Expand Down
45 changes: 23 additions & 22 deletions .github/workflows/build_wheels.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ jobs:
highest_semver_tag: ${{ steps.get_highest_semver.outputs.highest_semver_tag }}
steps:
- name: Checkout
uses: actions/checkout@v2
uses: actions/checkout@v3
with:
persist-credentials: false
- name: Get release version
Expand Down Expand Up @@ -55,14 +55,14 @@ jobs:
name: Build wheels
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
- name: Setup Python
uses: actions/setup-python@v2
uses: actions/setup-python@v3
with:
python-version: "3.8"
architecture: x64
- name: Setup Node
uses: actions/setup-node@v2
uses: actions/setup-node@v3
with:
node-version: '17.x'
registry-url: 'https://registry.npmjs.org'
Expand All @@ -72,24 +72,24 @@ jobs:
run: |
python -m pip install build
python -m build --wheel --outdir wheelhouse/
- uses: actions/upload-artifact@v2
- uses: actions/upload-artifact@v3
with:
name: wheels
path: ./wheelhouse/*.whl

build-source-distribution:
name: Build source distribution
runs-on: macos-10.15
runs-on: macos-latest
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
- name: Setup Python
id: setup-python
uses: actions/setup-python@v2
uses: actions/setup-python@v3
with:
python-version: "3.10"
architecture: x64
- name: Setup Node
uses: actions/setup-node@v2
uses: actions/setup-node@v3
with:
node-version: '17.x'
registry-url: 'https://registry.npmjs.org'
Expand All @@ -105,7 +105,7 @@ jobs:
- name: Build
run: |
python3 setup.py sdist
- uses: actions/upload-artifact@v2
- uses: actions/upload-artifact@v3
with:
name: wheels
path: dist/*
Expand All @@ -120,7 +120,7 @@ jobs:
env:
REGISTRY: feastdev
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
- name: Set up QEMU
uses: docker/setup-qemu-action@v1
- name: Set up Docker Buildx
Expand All @@ -136,7 +136,7 @@ jobs:
needs: [build-python-wheel, build-source-distribution, get-version]
strategy:
matrix:
os: [ubuntu-latest, macos-10.15 ]
os: [ubuntu-latest, macos-latest ]
python-version: [ "3.8", "3.9", "3.10"]
from-source: [ True, False ]
env:
Expand All @@ -156,7 +156,7 @@ jobs:
steps:
- name: Setup Python
id: setup-python
uses: actions/setup-python@v2
uses: actions/setup-python@v3
with:
python-version: ${{ matrix.python-version }}
architecture: x64
Expand All @@ -165,7 +165,7 @@ jobs:
name: wheels
path: dist
- name: Install OS X dependencies
if: matrix.os == 'macos-10.15'
if: matrix.os == 'macos-latest'
run: brew install coreutils
- name: Install wheel
if: ${{ !matrix.from-source }}
Expand All @@ -192,11 +192,12 @@ jobs:
echo "$VERSION_OUTPUT from installed wheel is not in the correct format or doesn't have the right version $VERSION."
exit 1
fi
- name: Smoke test
run: |
feast init test_repo
cd test_repo/feature_repo
feast apply
echo "$TEST_SCRIPT" > run-and-wait.sh
bash run-and-wait.sh feast serve
bash run-and-wait.sh feast ui
# This is temporarily disabled.
# - name: Smoke test
# run: |
# feast init test_repo
# cd test_repo/feature_repo
# feast apply
# echo "$TEST_SCRIPT" > run-and-wait.sh
# bash run-and-wait.sh feast serve
# bash run-and-wait.sh feast ui
21 changes: 10 additions & 11 deletions .github/workflows/java_master_only.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,11 @@ jobs:
MAVEN_CACHE: gs://feast-templocation-kf-feast/.m2.2020-08-19.tar
REGISTRY: gcr.io/kf-feast
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
with:
submodules: 'true'
- name: Setup Python
uses: actions/setup-python@v2
uses: actions/setup-python@v3
id: setup-python
with:
python-version: "3.8"
Expand Down Expand Up @@ -53,7 +53,7 @@ jobs:
if: github.repository == 'feast-dev/feast'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
with:
submodules: 'true'
- name: Lint java
Expand All @@ -63,7 +63,7 @@ jobs:
if: github.repository == 'feast-dev/feast'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
with:
submodules: 'true'
- name: Set up JDK 11
Expand All @@ -86,7 +86,7 @@ jobs:
${{ runner.os }}-ut-maven-
- name: Test java
run: make test-java-with-coverage
- uses: actions/upload-artifact@v2
- uses: actions/upload-artifact@v3
with:
name: java-coverage-report
path: ${{ github.workspace }}/docs/coverage/java/target/site/jacoco-aggregate/
Expand All @@ -97,7 +97,7 @@ jobs:
env:
PYTHON: 3.8
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
with:
submodules: 'true'
- name: Set up JDK 11
Expand All @@ -107,14 +107,11 @@ jobs:
java-package: jdk
architecture: x64
- name: Setup Python (to call feast apply)
uses: actions/setup-python@v2
uses: actions/setup-python@v3
id: setup-python
with:
python-version: 3.8
architecture: x64
- name: Upgrade pip version
run: |
pip install --upgrade pip
- name: Get pip cache dir
id: pip-cache
run: |
Expand All @@ -129,9 +126,11 @@ jobs:
key: ${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-${{ hashFiles(format('**/py{0}-ci-requirements.txt', env.PYTHON)) }}
restore-keys: |
${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-pip-
- name: Upgrade pip version
run: |
pip install --upgrade "pip>=21.3.1,<23.2"
- name: Install pip-tools
run: pip install pip-tools

- name: Install Python dependencies
run: make install-python-ci-dependencies
- uses: actions/cache@v2
Expand Down
Loading
Loading