Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add telemetry on datastore #56

Merged
merged 4 commits into from
Sep 21, 2022
Merged

Add telemetry on datastore #56

merged 4 commits into from
Sep 21, 2022

Conversation

devbugging
Copy link
Contributor

Add telemtry on datastore.

For contributor use:

  • Targeted PR against correct branch (see CONTRIBUTING.md)
  • Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
  • Code follows the standards mentioned here.
  • Updated relevant documentation
  • Re-reviewed Files changed in the Github PR explorer
  • Added appropriate labels

@devbugging devbugging changed the base branch from master to staging September 21, 2022 09:38
@devbugging devbugging merged commit 9ac7f6c into staging Sep 21, 2022
DylanTinianov added a commit that referenced this pull request Oct 14, 2022
@devbugging @DylanTinianov
Merge staging into master
7f3c75d 
* Add telemetry on datastore (#56)

* Add debug logs

* Add runtime calculation logging

* add more log

* add datastore telemtry

Co-authored-by: Dylan Tinianov <dylantinianov@gmail.com>

* add better telemtry (#58)

* Implement SQL storage (#59)

* implemented sql storage

* remove persistence specific models

* remove unused storage impl

* fix storage interface

* update account controller to changes

* update project controller

* update storage interface

* update scripts controller

* update transactions controller

* update embed controller

* fix project

* init storage

* fix API changes

* test todo

* mod tidy

* move sql to top

* fix models - remove logs

* dissable telemtry

* remove unused fields

* improve init

* fix tests

* fix tests

* add postresql driver

* postgresql client init

* postgresql client init

* update server with const sql

* add host

* add logging

* debug logging

* more debug logs

* more debug logs

* add sentry panic

* clean up code

* update version

* change cors

* Revert "change cors"

This reverts commit fe6c2ea.

* set verison on sentry

* fix update methods

* fix update methods

* clean up code

* clean up code

* clean up stale code (#63)

* Test bugfixes (#65)

* fix factory methods

* add sqlite db to gitignore

* fix test

* fix update

* check and standarize update

* refactor account update

* validate update

* fix missing value

* store init test

* fix execution index

* Cache invalidation and mutex improvements (#70)

* fix factory methods

* add sqlite db to gitignore

* fix test

* fix update

* check and standarize update

* refactor account update

* validate update

* fix missing value

* store init test

* fix execution index

* rename shadowed var

* mutex changes tests

* refactor mutex sync map

* fix test store init

* simplify mutex

* simplify mutex

* refactor cache

* refactor cache

* add eviction handler

* add logger

* bug reproduction

* refactor create init accounts

* track cache eviction

* disable cache test

* added get batch accounts

* optimize get account

* batch get accounts

* log len and cap exe

* add explicit order

* add logging

* add logging

* add logging

* add logging

* clean code

* add extra log for state recreation issue

* fix test

* add missing headers

* State recreation bugfix (#78)

* no version fix

* fix order clause

* added state test

* changed make for pgs store

* temp disable cache

* enable cache

* bugfix recreation failure

* bugfix recreation failure

* added more comments explaning cache

* Emulator pool (#79)

* increase idle connections

* update emulator

* clean up mutex

* get accounts test

* add to cache

* add instance pool

* add tests for instance pool

* integrate instance pool

* create pool

* tidy

* Fix linter (#83)

* Fix linter

* Include CI on staging

* Update linter

* Fix Test_DeployContracts

* Improvement/pool error handling (#80)

* Refill pool when empty

* Fill pool async

* Remove pool refill

* Parallelize test cases (#84)

* Testing/add replica testing (#95)

* Add replica testing

* Found potential cached emulator issue

* Update replica testing

* Update dockerfile

* Create replica tests

* Undo dockerfile change

* temp log

* add version log

* allow all origins

* allow all origins

* origins fix

* test deploy

* update allowed origins

* use env for allowed origins

* explicit order (#96)

* Change cache to thread safe lru (#77)

* Change cache to thread safe lru

* Add cache tests

* Add cache test with executions

* Verify execution block heights

* Fix linter

* Fix linter

* Add error handling to emulatorCache

* Remove cache evicted messages

* Improve cache error handling

* Fix comment

* Update cache error handling

* Update cache error handling

* Update cache error handling

* Add disabled emulator test

* Revert "Parallelize test cases (#84)" (#97)

This reverts commit 91772f0.

Co-authored-by: Dylan Tinianov <dylantinianov@gmail.com>
@DylanTinianov DylanTinianov deleted the improvement/debug-staging-telemetry branch November 15, 2022 17:00
bthaile added a commit that referenced this pull request Sep 9, 2024
@bluesign @devbugging
@DylanTinianov @turbolent @Latkes @dependabot @bthaile
update API to Cadence 1.0 (#365)
a8ad5d3 
* Merge staging into master

* Add telemetry on datastore (#56)

* Add debug logs

* Add runtime calculation logging

* add more log

* add datastore telemtry

Co-authored-by: Dylan Tinianov <dylantinianov@gmail.com>

* add better telemtry (#58)

* Implement SQL storage (#59)

* implemented sql storage

* remove persistence specific models

* remove unused storage impl

* fix storage interface

* update account controller to changes

* update project controller

* update storage interface

* update scripts controller

* update transactions controller

* update embed controller

* fix project

* init storage

* fix API changes

* test todo

* mod tidy

* move sql to top

* fix models - remove logs

* dissable telemtry

* remove unused fields

* improve init

* fix tests

* fix tests

* add postresql driver

* postgresql client init

* postgresql client init

* update server with const sql

* add host

* add logging

* debug logging

* more debug logs

* more debug logs

* add sentry panic

* clean up code

* update version

* change cors

* Revert "change cors"

This reverts commit fe6c2ea.

* set verison on sentry

* fix update methods

* fix update methods

* clean up code

* clean up code

* clean up stale code (#63)

* Test bugfixes (#65)

* fix factory methods

* add sqlite db to gitignore

* fix test

* fix update

* check and standarize update

* refactor account update

* validate update

* fix missing value

* store init test

* fix execution index

* Cache invalidation and mutex improvements (#70)

* fix factory methods

* add sqlite db to gitignore

* fix test

* fix update

* check and standarize update

* refactor account update

* validate update

* fix missing value

* store init test

* fix execution index

* rename shadowed var

* mutex changes tests

* refactor mutex sync map

* fix test store init

* simplify mutex

* simplify mutex

* refactor cache

* refactor cache

* add eviction handler

* add logger

* bug reproduction

* refactor create init accounts

* track cache eviction

* disable cache test

* added get batch accounts

* optimize get account

* batch get accounts

* log len and cap exe

* add explicit order

* add logging

* add logging

* add logging

* add logging

* clean code

* add extra log for state recreation issue

* fix test

* add missing headers

* State recreation bugfix (#78)

* no version fix

* fix order clause

* added state test

* changed make for pgs store

* temp disable cache

* enable cache

* bugfix recreation failure

* bugfix recreation failure

* added more comments explaning cache

* Emulator pool (#79)

* increase idle connections

* update emulator

* clean up mutex

* get accounts test

* add to cache

* add instance pool

* add tests for instance pool

* integrate instance pool

* create pool

* tidy

* Fix linter (#83)

* Fix linter

* Include CI on staging

* Update linter

* Fix Test_DeployContracts

* Improvement/pool error handling (#80)

* Refill pool when empty

* Fill pool async

* Remove pool refill

* Parallelize test cases (#84)

* Testing/add replica testing (#95)

* Add replica testing

* Found potential cached emulator issue

* Update replica testing

* Update dockerfile

* Create replica tests

* Undo dockerfile change

* temp log

* add version log

* allow all origins

* allow all origins

* origins fix

* test deploy

* update allowed origins

* use env for allowed origins

* explicit order (#96)

* Change cache to thread safe lru (#77)

* Change cache to thread safe lru

* Add cache tests

* Add cache test with executions

* Verify execution block heights

* Fix linter

* Fix linter

* Add error handling to emulatorCache

* Remove cache evicted messages

* Improve cache error handling

* Fix comment

* Update cache error handling

* Update cache error handling

* Update cache error handling

* Add disabled emulator test

* Revert "Parallelize test cases (#84)" (#97)

This reverts commit 91772f0.

Co-authored-by: Dylan Tinianov <dylantinianov@gmail.com>

* Create security scanning (#112)

* remove turbolent

* Create codeql.yml (#114)

Please approve this request to configure Code-QL. The submitted template is set for scans on Pull/Push and 0700 hours daily in the Master Branch for the Go Language.  If you have any questions please contact me @Latkes

* remove unused test lib (#136)

* Add error rate metrics to v1 (#149)

* Add metrics for error rate

* Counter server errors for error rate

* Separate server and user errors

* Not authorized errors are user errors

* Catch invalid graphql requests

* Catch invalid graphql requests

* Update flow-emulator to v0.42.0 (#163)

* Bump github.com/ethereum/go-ethereum from 1.9.13 to 1.10.22 (#167)

* Bump github.com/ethereum/go-ethereum from 1.9.13 to 1.10.22

Bumps [github.com/ethereum/go-ethereum](https://github.com/ethereum/go-ethereum) from 1.9.13 to 1.10.22.
- [Release notes](https://github.com/ethereum/go-ethereum/releases)
- [Commits](ethereum/go-ethereum@v1.9.13...v1.10.22)

---
updated-dependencies:
- dependency-name: github.com/ethereum/go-ethereum
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

* Set cache to nil if lru is nil

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Dylan Tinianov <dylantinianov@gmail.com>

* Create dependabot.yml (#166)

Add Dependabot for flow-emulator and cadence

* Bump actions/setup-go from 2 to 3 (#168)

Bumps [actions/setup-go](https://github.com/actions/setup-go) from 2 to 3.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](actions/setup-go@v2...v3)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump actions/cache from 1 to 3 (#169)

Bumps [actions/cache](https://github.com/actions/cache) from 1 to 3.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](actions/cache@v1...v3)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump golangci/golangci-lint-action from 3.2.0 to 3.3.1 (#170)

Bumps [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) from 3.2.0 to 3.3.1.
- [Release notes](https://github.com/golangci/golangci-lint-action/releases)
- [Commits](golangci/golangci-lint-action@v3.2.0...v3.3.1)

---
updated-dependencies:
- dependency-name: golangci/golangci-lint-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump github.com/onflow/cadence from 0.31.0 to 0.31.3 (#172)

Bumps [github.com/onflow/cadence](https://github.com/onflow/cadence) from 0.31.0 to 0.31.3.
- [Release notes](https://github.com/onflow/cadence/releases)
- [Changelog](https://github.com/onflow/cadence/blob/master/CHANGELOG.md)
- [Commits](onflow/cadence@v0.31.0...v0.31.3)

---
updated-dependencies:
- dependency-name: github.com/onflow/cadence
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump actions/checkout from 2 to 3 (#171)

Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@v2...v3)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* update packages to support cadence 1.0

* update to use newest previews, had to remove storage, need to get new appoarch to getting user storage

* test fixes

* merge2

* log and flowkit fixes

* remove prints

* tests

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Gregor G <75445744+sideninja@users.noreply.github.com>
Co-authored-by: Dylan Tinianov <dylantinianov@gmail.com>
Co-authored-by: Bastian Müller <bastian@axiomzen.co>
Co-authored-by: Aaron Moreno <36414888+Latkes@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tom Haile <bradford.haile@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@psiemens psiemens Awaiting requested review from psiemens

@turbolent turbolent Awaiting requested review from turbolent

@janezpodhostnik janezpodhostnik Awaiting requested review from janezpodhostnik

@DylanTinianov DylanTinianov Awaiting requested review from DylanTinianov

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@devbugging @DylanTinianov