Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Support blob type sources and GCS as an example of such source. #1366

Merged
merged 9 commits into from
Jul 30, 2024
Merged

feat: Support blob type sources and GCS as an example of such source. #1366

merged 9 commits into from
Jul 30, 2024

Conversation

alan-kut
Copy link
Contributor

This PR

  • adds support for blob type sources
  • adds support for GCS using blob source

Notes

In a similar fashion other cloud storages supported by go-cloud blob can be added.

How to test

@alan-kut alan-kut requested a review from a team as a code owner July 22, 2024 14:01
@dosubot dosubot bot added the size:XL This PR changes 500-999 lines, ignoring generated files. label Jul 22, 2024
@netlify Netlify
Copy link

netlify bot commented Jul 22, 2024 •
edited
Loading

✅ Deploy Preview for polite-licorice-3db33c ready!

Name Link
🔨 Latest commit 7d2f912
🔍 Latest deploy log https://app.netlify.com/sites/polite-licorice-3db33c/deploys/66a90cb1b0ff8a0008cfe27b
😎 Deploy Preview https://deploy-preview-1366--polite-licorice-3db33c.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@alan-kut
Add support for blob type sources.
598a4c1 
Signed-off-by: Alan Kutniewski <kutniewski@google.com>
@alan-kut
Support for GCS source using Blob sync.
0c7412a 
Signed-off-by: Alan Kutniewski <kutniewski@google.com>
@alan-kut alan-kut changed the title Support blob type sources and GCS as an example of such source. feat: Support blob type sources and GCS as an example of such source. Jul 22, 2024
@beeme1mr
Copy link
Member

Hey @alan-kut, thanks for the PR. I'll test this out E2E tomorrow.

@alan-kut
Copy link
Contributor Author

Thanks. I fixed lint error and data race.

Few assumptions I made in this code:

  • It will return error from Sync and Resync if the blob cannot be accessed (authorization error, doesn't exist or other problems)
  • It will log warning when sync on cron cannot access the blob
  • It won't fetch blob and send updated data if the updated time of the blob hasn't changed (except for ReSync)

toddbaert
toddbaert reviewed Jul 23, 2024
View reviewed changes
core/go.mod Show resolved Hide resolved
beeme1mr
beeme1mr approved these changes Jul 23, 2024
View reviewed changes
Copy link
Member

@beeme1mr beeme1mr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've tested this locally and it works great. My only concern is the size of the change detection log message.

Could you please sign off your comments and update the sync config docs when you get a moment?

core/pkg/sync/blob/blob_sync.go Outdated Show resolved Hide resolved
renovate bot and others added 2 commits July 24, 2024 10:37
@renovate @alan-kut
fix(deps): update module connectrpc.com/otelconnect to v0.7.1 (open-f…
15d4070 
…eature#1367)

[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
|
[connectrpc.com/otelconnect](https://togithub.com/connectrpc/otelconnect-go)
| `v0.7.0` -> `v0.7.1` |
[![age](https://developer.mend.io/api/mc/badges/age/go/connectrpc.com%2fotelconnect/v0.7.1?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/go/connectrpc.com%2fotelconnect/v0.7.1?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/go/connectrpc.com%2fotelconnect/v0.7.0/v0.7.1?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/go/connectrpc.com%2fotelconnect/v0.7.0/v0.7.1?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|

---

### Release Notes

<details>
<summary>connectrpc/otelconnect-go
(connectrpc.com/otelconnect)</summary>

###
[`v0.7.1`](https://togithub.com/connectrpc/otelconnect-go/releases/tag/v0.7.1)

[Compare
Source](https://togithub.com/connectrpc/otelconnect-go/compare/v0.7.0...v0.7.1)

This is a bug-fix release that addresses a race condition when closing a
stream.

#### What's Changed

##### Bugfixes

- Fix data race in streaming client close by
[@&#8203;emcfarlane](https://togithub.com/emcfarlane) in
[#&#8203;173](https://togithub.com/connectrpc/otelconnect-go/issues/173)

#### New Contributors

- [@&#8203;gvacaliuc](https://togithub.com/gvacaliuc) made their first
contribution in
[#&#8203;163](https://togithub.com/connectrpc/otelconnect-go/issues/163)
- [@&#8203;ytnsym](https://togithub.com/ytnsym) made their first
contribution in
[#&#8203;176](https://togithub.com/connectrpc/otelconnect-go/issues/176)
- [@&#8203;drice-buf](https://togithub.com/drice-buf) made their first
contribution in
[#&#8203;178](https://togithub.com/connectrpc/otelconnect-go/issues/178)

**Full Changelog**:
connectrpc/otelconnect-go@v0.7.0...v0.7.1

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View the
[repository job
log](https://developer.mend.io/github/open-feature/flagd).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40MzguMCIsInVwZGF0ZWRJblZlciI6IjM3LjQzOC4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
@alan-kut
Fix lint and data race
224a656 
Signed-off-by: Alan Kutniewski <kutniewski@google.com>
@alan-kut
Update docs with the information about GCS sync source
186822e 
Signed-off-by: Alan Kutniewski <kutniewski@google.com>
@alan-kut
Copy link
Contributor Author

I changed the log to debug, signed off commit, updated the docs (in a new commit).

Should I squash the commits somehow?
Rebase them on main?

tangenti
tangenti reviewed Jul 24, 2024
View reviewed changes
core/pkg/sync/blob/blob_sync.go Show resolved Hide resolved
core/pkg/sync/blob/blob_sync.go Show resolved Hide resolved
core/pkg/sync/builder/syncbuilder.go Show resolved Hide resolved
@alan-kut
Simplify the blob sync by using bucket.Download and validate bucket and
18952d8 
boject in the init.

Signed-off-by: Alan Kutniewski <kutniewski@google.com>
@beeme1mr
Copy link
Member

I've reviewed all the licensing issues detected by FOSSA. The library is licensed BSD-3, but comments in the code referred to other licenses, which triggered a false positive.

@alan-kut alan-kut requested a review from tangenti July 25, 2024 07:57
@alan-kut
Fix log message
75c2ce9 
Signed-off-by: Alan Kutniewski <kutniewski@google.com>
@toddbaert
Copy link
Member

I changed the log to debug, signed off commit, updated the docs (in a new commit).

Should I squash the commits somehow? Rebase them on main?

Commits are squashed automatically, everything will be clean in main.

toddbaert
toddbaert approved these changes Jul 25, 2024
View reviewed changes
Copy link
Member

@toddbaert toddbaert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very cool addition.

@beeme1mr beeme1mr merged commit 21f2c9a into open-feature:main Jul 30, 2024
15 checks passed
@github-actions github-actions bot mentioned this pull request Jul 30, 2024
Merged
@beeme1mr beeme1mr mentioned this pull request Aug 2, 2024
Open
beeme1mr pushed a commit that referenced this pull request Aug 22, 2024
@github-actions
chore: release main (#1356)
b6c18f3 
🤖 I have created a release *beep* *boop*
---


<details><summary>flagd: 0.11.2</summary>

##
[0.11.2](flagd/v0.11.1...flagd/v0.11.2)
(2024-08-22)


### 🐛 Bug Fixes

* **deps:** update module buf.build/gen/go/open-feature/flagd/grpc/go to
v1.5.1-20240215170432-1e611e2999cc.1
([#1372](#1372))
([ae24595](ae24595))
* **deps:** update module github.com/open-feature/flagd/core to v0.10.1
([#1355](#1355))
([8fcfb14](8fcfb14))
* **deps:** update module golang.org/x/net to v0.28.0
([#1380](#1380))
([239a432](239a432))
* **deps:** update module golang.org/x/sync to v0.8.0
([#1378](#1378))
([4804c17](4804c17))


### 🧹 Chore

* **deps:** update dependency go to v1.22.6
([#1297](#1297))
([50b92c1](50b92c1))
* **deps:** update golang docker tag to v1.23
([#1382](#1382))
([abb5ca3](abb5ca3))
* improve gRPC sync service shutdown behavior
([#1375](#1375))
([79d9085](79d9085))
</details>

<details><summary>flagd-proxy: 0.6.5</summary>

##
[0.6.5](flagd-proxy/v0.6.4...flagd-proxy/v0.6.5)
(2024-08-22)


### 🐛 Bug Fixes

* **deps:** update module buf.build/gen/go/open-feature/flagd/grpc/go to
v1.5.1-20240215170432-1e611e2999cc.1
([#1372](#1372))
([ae24595](ae24595))
* **deps:** update module github.com/open-feature/flagd/core to v0.10.1
([#1355](#1355))
([8fcfb14](8fcfb14))
* **deps:** update module golang.org/x/net to v0.28.0
([#1380](#1380))
([239a432](239a432))
* **deps:** update module golang.org/x/sync to v0.8.0
([#1378](#1378))
([4804c17](4804c17))


### 🧹 Chore

* **deps:** update dependency go to v1.22.6
([#1297](#1297))
([50b92c1](50b92c1))
* **deps:** update golang docker tag to v1.23
([#1382](#1382))
([abb5ca3](abb5ca3))


### 📚 Documentation

* **flagd-proxy:** removed invalid grpc prefix from uri config
([4911697](4911697))
</details>

<details><summary>core: 0.10.2</summary>

##
[0.10.2](core/v0.10.1...core/v0.10.2)
(2024-08-22)


### 🐛 Bug Fixes

* **deps:** update module buf.build/gen/go/open-feature/flagd/grpc/go to
v1.5.1-20240215170432-1e611e2999cc.1
([#1372](#1372))
([ae24595](ae24595))
* **deps:** update module connectrpc.com/otelconnect to v0.7.1
([#1367](#1367))
([184915b](184915b))
* **deps:** update module
github.com/open-feature/open-feature-operator/apis to v0.2.44
([#1368](#1368))
([0c68726](0c68726))
* **deps:** update module golang.org/x/crypto to v0.26.0
([#1379](#1379))
([05f6658](05f6658))
* **deps:** update module golang.org/x/mod to v0.20.0
([#1377](#1377))
([797d7a4](797d7a4))
* **deps:** update module golang.org/x/sync to v0.8.0
([#1378](#1378))
([4804c17](4804c17))


### ✨ New Features

* add 'watcher' interface to file sync
([#1365](#1365))
([61fff43](61fff43))
* added new grpc sync config option to allow setting max receive message
size. ([#1358](#1358))
([bed077b](bed077b))
* Support blob type sources and GCS as an example of such source.
([#1366](#1366))
([21f2c9a](21f2c9a))


### 🧹 Chore

* **deps:** update dependency go to v1.22.6
([#1297](#1297))
([50b92c1](50b92c1))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Signed-off-by: OpenFeature Bot <109696520+openfeaturebot@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Kavindu-Dodan Kavindu-Dodan Kavindu-Dodan left review comments

@beeme1mr beeme1mr beeme1mr approved these changes

@toddbaert toddbaert toddbaert approved these changes

@tangenti tangenti Awaiting requested review from tangenti

@kamil-nowosad kamil-nowosad Awaiting requested review from kamil-nowosad

Assignees
No one assigned
Labels
size:XL This PR changes 500-999 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@alan-kut @beeme1mr @toddbaert @kamil-nowosad @Kavindu-Dodan @tangenti