Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add maintenance window logic #115

Merged
merged 3 commits into from
Feb 29, 2024
Merged

Add maintenance window logic #115

merged 3 commits into from
Feb 29, 2024

Conversation

ericaporter
Copy link
Contributor

Relating to this ticket

We need the ability to do maintenance on on a live streamed BiqQuery dataset, such as register_events_production.

This PR adds a configuration item for a maintenance window.

Redis logic - WIP

  • On reaching the start maintenance window start timestamp, DfE Analytics will start queueing items with a schedule time of:
    end_timestamp + number of seconds elapsed since start_timestamp
    The queued items will end up queued in Redis.
  • Once the end of the maintenance window is reached, the queue items will be processed and sent to BigQuery and a First In First Out (FIFO) base.

Copy link
Collaborator

@asatwal asatwal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comments in code on suggestions for implementation.

lib/dfe/analytics.rb Show resolved Hide resolved
lib/dfe/analytics.rb Show resolved Hide resolved
config/locales/en.yml Outdated Show resolved Hide resolved
Comment on lines 21 to 26
if DfE::Analytics.within_maintenance_window?
# Delaying / Queueing events - the redis bit
Rails.logger.info('Within BigQuery maintenance window. Events will be queued for later processing.')
return
end

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to remove this block here and move into the do method above.

Suggested change
if DfE::Analytics.within_maintenance_window?
# Delaying / Queueing events - the redis bit
Rails.logger.info('Within BigQuery maintenance window. Events will be queued for later processing.')
return
end

In the do method replace lines 13-17 with the following:

   if DfE::Analytics.within_maintenance_window?
     set(wait_until: DfE::Analytics.next_scheduled_time_after_maintenance_window).perform_later(events)
   elsif DfE::Analytics.async?
     perform_later(events)
   else
     perform_now(events)
   end

Copy link
Collaborator

@asatwal asatwal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good @ericaporter - Just have a minor comment about adding another test.

spec/dfe/analytics/send_events_spec.rb Show resolved Hide resolved
@ericaporter ericaporter merged commit b38abea into main Feb 29, 2024
7 checks passed
@ericaporter ericaporter deleted the config-scheduled-downtime branch February 29, 2024 14:53
@ericaporter ericaporter mentioned this pull request Feb 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants