🌱 cron: generalize and expose worker (6/n) #2317

spencerschrock · 2022-09-30T17:48:37Z

What kind of change does this PR introduce?

Part of a series of refactors to generalize the cron infrastructure to enable re-use by the ossf criticality score project

PR title follows the guidelines defined in our pull request documentation

What is the current behavior?

The worker logic is scorecard specific and can't be re-used.

What is the new behavior (if this is a feature change)?**

The general logic has been separated from the scorecard specific logic.
Criticality should be able to import the required modules and create their own worker which implements the interface.

The following packages are (re) exposed in cron:

config
data
worker

Tests for the changes have been added (for bug fixes/features)

more of a refactor of existing code. the worker release test should catch any major errors.

Which issue(s) this PR fixes

NONE

Special notes for your reviewer

Does this PR introduce a user-facing change?

For user-facing changes, please add a concise, human-readable release note to
the release-note

(In particular, describe what changes users might need to make in their
application as a result of this pull request.)

NONE

Signed-off-by: Spencer Schrock <sschrock@google.com>

spencerschrock · 2022-09-30T17:51:01Z

Not sure if Friday is the best time to merge something like this, but wanted to get it up for discussion for any changes.
@oliverchang @calebbrown fyi

codecov · 2022-09-30T17:55:49Z

Codecov Report

Merging #2317 (b1b9601) into main (900701c) will not change coverage.
The diff coverage is n/a.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2317   +/-   ##
=======================================
  Coverage   40.61%   40.61%           
=======================================
  Files         112      112           
  Lines        8831     8831           
=======================================
  Hits         3587     3587           
  Misses       4984     4984           
  Partials      260      260

github-actions · 2022-09-30T18:04:25Z

Integration tests success for
[22d12ad]
(https://github.com/ossf/scorecard/actions/runs/3160647323)

Signed-off-by: Spencer Schrock <sschrock@google.com>

spencerschrock · 2022-10-04T17:16:33Z

876acb0 will cause metadata to be populated to the BigQuery results. Might be worth spinning this off in it's own PR

github-actions · 2022-10-04T17:26:28Z

Integration tests success for
[9389f67]
(https://github.com/ossf/scorecard/actions/runs/3184068892)

github-actions · 2022-10-15T02:14:37Z

Stale pull request message

cron/worker/worker.go

Signed-off-by: Spencer Schrock <sschrock@google.com>

github-actions · 2022-10-19T17:24:29Z

Integration tests success for
[19ec3d7]
(https://github.com/ossf/scorecard/actions/runs/3283512444)

azeemshaikh38 · 2022-10-19T17:46:30Z

876acb0 will cause metadata to be populated to the BigQuery results. Might be worth spinning this off in it's own PR

Oh, so this was the bug all along. Yeah doing this in a separate PR SGTM.

This reverts commit 876acb0. Will send separate PR. Signed-off-by: Spencer Schrock <sschrock@google.com>

github-actions · 2022-10-19T18:24:38Z

Integration tests success for
[580d634]
(https://github.com/ossf/scorecard/actions/runs/3283903143)

Signed-off-by: Spencer Schrock <sschrock@google.com>

github-actions · 2022-10-19T20:55:50Z

Integration tests success for
[b1b9601]
(https://github.com/ossf/scorecard/actions/runs/3284833174)

* WIP Signed-off-by: Spencer Schrock <sschrock@google.com> * Appease linter. Signed-off-by: Spencer Schrock <sschrock@google.com> * Update Makefile for worker Signed-off-by: Spencer Schrock <sschrock@google.com> * Extract already completed request sanity check. Signed-off-by: Spencer Schrock <sschrock@google.com> * Add worker test. Signed-off-by: Spencer Schrock <sschrock@google.com> * remove logger from worker interface Signed-off-by: Spencer Schrock <sschrock@google.com> * move cron data and worker out of cron/internal Signed-off-by: Spencer Schrock <sschrock@google.com> * Move config out of internal. Signed-off-by: Spencer Schrock <sschrock@google.com> * Document worker interface. Signed-off-by: Spencer Schrock <sschrock@google.com> * Fix typo which prevented metadata from going to cron job. Signed-off-by: Spencer Schrock <sschrock@google.com> * Address feedback. Signed-off-by: Spencer Schrock <sschrock@google.com> * Revert "Fix typo which prevented metadata from going to cron job." This reverts commit 876acb0. Will send separate PR. Signed-off-by: Spencer Schrock <sschrock@google.com> * Fix linter. Signed-off-by: Spencer Schrock <sschrock@google.com> Signed-off-by: Spencer Schrock <sschrock@google.com> Signed-off-by: latortuga <latortugaaaa>

* WIP Signed-off-by: Spencer Schrock <sschrock@google.com> * Appease linter. Signed-off-by: Spencer Schrock <sschrock@google.com> * Update Makefile for worker Signed-off-by: Spencer Schrock <sschrock@google.com> * Extract already completed request sanity check. Signed-off-by: Spencer Schrock <sschrock@google.com> * Add worker test. Signed-off-by: Spencer Schrock <sschrock@google.com> * remove logger from worker interface Signed-off-by: Spencer Schrock <sschrock@google.com> * move cron data and worker out of cron/internal Signed-off-by: Spencer Schrock <sschrock@google.com> * Move config out of internal. Signed-off-by: Spencer Schrock <sschrock@google.com> * Document worker interface. Signed-off-by: Spencer Schrock <sschrock@google.com> * Fix typo which prevented metadata from going to cron job. Signed-off-by: Spencer Schrock <sschrock@google.com> * Address feedback. Signed-off-by: Spencer Schrock <sschrock@google.com> * Revert "Fix typo which prevented metadata from going to cron job." This reverts commit 876acb0. Will send separate PR. Signed-off-by: Spencer Schrock <sschrock@google.com> * Fix linter. Signed-off-by: Spencer Schrock <sschrock@google.com> Signed-off-by: Spencer Schrock <sschrock@google.com> Signed-off-by: nathaniel.wert <nathaniel.wert@kudelskisecurity.com>

* WIP Signed-off-by: Spencer Schrock <sschrock@google.com> * Appease linter. Signed-off-by: Spencer Schrock <sschrock@google.com> * Update Makefile for worker Signed-off-by: Spencer Schrock <sschrock@google.com> * Extract already completed request sanity check. Signed-off-by: Spencer Schrock <sschrock@google.com> * Add worker test. Signed-off-by: Spencer Schrock <sschrock@google.com> * remove logger from worker interface Signed-off-by: Spencer Schrock <sschrock@google.com> * move cron data and worker out of cron/internal Signed-off-by: Spencer Schrock <sschrock@google.com> * Move config out of internal. Signed-off-by: Spencer Schrock <sschrock@google.com> * Document worker interface. Signed-off-by: Spencer Schrock <sschrock@google.com> * Fix typo which prevented metadata from going to cron job. Signed-off-by: Spencer Schrock <sschrock@google.com> * Address feedback. Signed-off-by: Spencer Schrock <sschrock@google.com> * Revert "Fix typo which prevented metadata from going to cron job." This reverts commit 876acb0. Will send separate PR. Signed-off-by: Spencer Schrock <sschrock@google.com> * Fix linter. Signed-off-by: Spencer Schrock <sschrock@google.com> Signed-off-by: Spencer Schrock <sschrock@google.com>

spencerschrock added 12 commits September 20, 2022 13:26

WIP

f7e7e77

Signed-off-by: Spencer Schrock <sschrock@google.com>

Appease linter.

0ba8ba0

Signed-off-by: Spencer Schrock <sschrock@google.com>

Update Makefile for worker

f174eca

Signed-off-by: Spencer Schrock <sschrock@google.com>

Extract already completed request sanity check.

7a28653

Signed-off-by: Spencer Schrock <sschrock@google.com>

Add worker test.

dcfa11a

Signed-off-by: Spencer Schrock <sschrock@google.com>

Merge branch 'main' into cron-worker-refactor

f5c84ca

Signed-off-by: Spencer Schrock <sschrock@google.com>

remove logger from worker interface

3636448

Signed-off-by: Spencer Schrock <sschrock@google.com>

Merge branch 'main' into cron-worker-refactor

16409d5

Signed-off-by: Spencer Schrock <sschrock@google.com>

move cron data and worker out of cron/internal

5588e03

Signed-off-by: Spencer Schrock <sschrock@google.com>

Move config out of internal.

805f4a7

Signed-off-by: Spencer Schrock <sschrock@google.com>

Document worker interface.

9d88b27

Signed-off-by: Spencer Schrock <sschrock@google.com>

Merge branch 'main' into cron-worker-refactor

22d12ad

Signed-off-by: Spencer Schrock <sschrock@google.com>

spencerschrock requested review from olivekl, azeemshaikh38, justaugustus, laurentsimon and naveensrinivasan as code owners September 30, 2022 17:48

spencerschrock temporarily deployed to integration-test September 30, 2022 17:49 Inactive

spencerschrock added 2 commits October 4, 2022 10:10

Fix typo which prevented metadata from going to cron job.

876acb0

Signed-off-by: Spencer Schrock <sschrock@google.com>

Merge branch 'main' into cron-worker-refactor

9389f67

Signed-off-by: Spencer Schrock <sschrock@google.com>

spencerschrock temporarily deployed to integration-test October 4, 2022 17:12 Inactive

github-actions bot added the no-pr-activity label Oct 15, 2022

azeemshaikh38 approved these changes Oct 19, 2022

View reviewed changes

cron/worker/worker.go Show resolved Hide resolved

Merge branch 'main' into cron-worker-refactor

c472d45

Signed-off-by: Spencer Schrock <sschrock@google.com>

Address feedback.

19ec3d7

Signed-off-by: Spencer Schrock <sschrock@google.com>

spencerschrock temporarily deployed to integration-test October 19, 2022 17:00 Inactive

spencerschrock removed the no-pr-activity label Oct 19, 2022

Revert "Fix typo which prevented metadata from going to cron job."

580d634

This reverts commit 876acb0. Will send separate PR. Signed-off-by: Spencer Schrock <sschrock@google.com>

spencerschrock temporarily deployed to integration-test October 19, 2022 18:06 Inactive

azeemshaikh38 enabled auto-merge (squash) October 19, 2022 18:29

spencerschrock added 2 commits October 19, 2022 13:30

Fix linter.

4a2b4c1

Signed-off-by: Spencer Schrock <sschrock@google.com>

Merge branch 'main' into cron-worker-refactor

b1b9601

Signed-off-by: Spencer Schrock <sschrock@google.com>

auto-merge was automatically disabled October 19, 2022 20:39
Head branch was pushed to by a user without write access

spencerschrock temporarily deployed to integration-test October 19, 2022 20:39 Inactive

azeemshaikh38 enabled auto-merge (squash) October 19, 2022 20:50

azeemshaikh38 merged commit f979097 into ossf:main Oct 19, 2022

spencerschrock deleted the cron-worker-refactor branch May 24, 2023 17:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🌱 cron: generalize and expose worker (6/n) #2317

🌱 cron: generalize and expose worker (6/n) #2317

spencerschrock commented Sep 30, 2022

spencerschrock commented Sep 30, 2022

codecov bot commented Sep 30, 2022 •

edited

Loading

github-actions bot commented Sep 30, 2022

spencerschrock commented Oct 4, 2022

github-actions bot commented Oct 4, 2022

github-actions bot commented Oct 15, 2022

github-actions bot commented Oct 19, 2022

azeemshaikh38 commented Oct 19, 2022

github-actions bot commented Oct 19, 2022

github-actions bot commented Oct 19, 2022

🌱 cron: generalize and expose worker (6/n) #2317

🌱 cron: generalize and expose worker (6/n) #2317

Conversation

spencerschrock commented Sep 30, 2022

What kind of change does this PR introduce?

What is the current behavior?

What is the new behavior (if this is a feature change)?**

Which issue(s) this PR fixes

Special notes for your reviewer

Does this PR introduce a user-facing change?

spencerschrock commented Sep 30, 2022

codecov bot commented Sep 30, 2022 • edited Loading

Codecov Report

github-actions bot commented Sep 30, 2022

spencerschrock commented Oct 4, 2022

github-actions bot commented Oct 4, 2022

github-actions bot commented Oct 15, 2022

github-actions bot commented Oct 19, 2022

azeemshaikh38 commented Oct 19, 2022

github-actions bot commented Oct 19, 2022

github-actions bot commented Oct 19, 2022

codecov bot commented Sep 30, 2022 •

edited

Loading