v0.21.1

Release notes for v0.21.1

Changelog since v0.21.0

This release fixes an issue where terraform would fail to apply if redis authentication is disabled (which is the default).

Changes by Kind

Infrastructure fixes

• Allow for redis auth to be disabled and terraform to apply successfully (#1751, @mikehelmick)

Buf Fixes and Improvements

• Basic coverage for /codes pages (#1746, @whaught)
• Include multiline and whitespace in search in SMS preview to fix a bug where an SMS message contained newline characters. This did not affect delivered SMSes, only the preview window. (#1745, @sethvargo)

Dependencies

Added

Nothing has changed.

Changed

Nothing has changed.

Removed

Nothing has changed.

v0.21.0

Release notes for v0.21.0

If redis auth is not enabled, please use v0.21.1 instead

Upgrade notes

• This release introduces the ability for the verification server to pull statistics from a key server. This functionality is off by default, and can be enabled by a realm administrator. There are new configuration values for the default key server and audience to use.

  • KEY_SERVER_URL - base URL for the key server to pull stats from, the /v1/stats path will be appended to this. You must set this value in our terraform configuration for your deployment for this functionality to work.
  • KEY_SERVER_STATS_AUDIENCE - default value is the same as default value from the exposure-notifications-server

• Authenticated SMS

  • This is a new feature under development, it is off by default and subject to change without notice.

Changes by Kind

Statistics

Key-server stats

• Add CSV format for key-server stats file (#1691, @whaught)
• Add TEK age & onset-upload distribution chart (#1669, @whaught)
• Adjustable data smoothing (#1705, @whaught)
• Average issue-claim chart (#1710, @whaught)
• Claim age distribution chart added to UI (#1709, @whaught)
• Dev seed script randomizes the issue to claim time (#1731, @whaught)
• Emit key-server stats as json for charting (#1661, @whaught)
• Gives stats puller services the correct KMS permissions. (#1686, @mikehelmick)
• Redraw all charts with window resize. Move some js to application.js (#1700, @whaught)
• Show distributions as 7 day sums (#1698, @whaught)
• Show publish requests by OS (#1665, @whaught)
• Slider tick marks for issue-claim age chart (#1719, @whaught)
• Stats - Claim age distribution chart isn't resized when the scope is changed. (#1720, @mikehelmick)
• Styling updates for statistics page & about links (#1724, @whaught)
• Use UTC time in stats-puller test (#1740, @sethvargo)
• Add random key-server stats data in dev seed script (#1651, @whaught)
• Key server histograms have better controls and don't resize as days change (#1718, @mikehelmick)
• Logic for executing the v1/stats request and storing results (#1621, @whaught)
• Padding for stats sliders (#1725, @whaught)

Verification server stats

• Add code issue to claim age average and distribution stat (#1675, @whaught)
• Add invalid codes and full token graphs to stats (#1641, @sethvargo)
• Add new stats for codes_invalid and tokens to responses (#1631, @sethvargo)
• Increment code-issue stats at the end of issuance logic. This avoids recording [known] failures. (#1638, @whaught)
• Make codes/stats bit of seeding optional (#1628, @sethvargo)
• Make seed script also optionally verify codes and claim tokens (#1629, @sethvargo)
• Move API key stats into the API and display invalid claim attempts on API keys page (#1646, @sethvargo)

Authenticated SMS (new feature that is off by default)

• Add SMS signing functionality. This functionality is off by default, as Google and Apple are still developing the necessary client-side features to support it. (#1696, @sethvargo)
• Add database model for managed keys for signing SMS messages. (#1649, @mikehelmick)
• Implement first pass at SMS signature algorithm package (#1650, @sethvargo)
• Add utility for verifying SMS signatures (#1721, @sethvargo)
• Create SMSSigning config and instantiate SMS signing key manager where needed (#1673, @sethvargo)
• Give Admin API signer verifier permissions on the for the keyring containing SMS singing keys. (#1704, @mikehelmick)
• Standardize response codes and add tests to SMS keys (#1672, @sethvargo)
• Switch to short date with less base64-encoding in returned SMS signature (#1722, @sethvargo)
• Realm admins can create/rotate SMS signing keys and enable authenticated SMS. (#1668, @mikehelmick)

Other SMS changes

• Move SMS templates into the SMS tab (originally under codes) (#1734, @sethvargo)
• Use the same SMS provider for all messages, cache locally for 5 minutes to improve performance (#1674, @sethvargo)
• Display a preview of how SMSes could be split across multiple messages at 153 character boundaries. (#1737, @sethvargo)

Test coverage

• Add more tests for cacher package (#1659, @sethvargo)
• Add more tests for routes (#1657, @sethvargo)
• Add more unit tests for internal/project package (#1656, @sethvargo)
• Fix e2e test for unknown user-agent header (#1695, @whaught)
• Add some more admin tests (#1660, @sethvargo)
• Add tests for api package (#1658, @sethvargo)
• Run e2e enx-redirector tests as part of CI (#1732, @sethvargo)

Terraform changes

• Add optional authentication to Redis. The default behavior remains unchanged, but a new Terraform variable redis_enable_auth exists to opt-in to Redis authentication. Opting in can cause downtime, so if you choose to enable it, we recommend doing i...

v0.20.0

Release notes for exposure-notifications-verification-server v0.20.0

Documentation

Changelog since v0.19.0

⚠️ Upgrade notes ⚠️

Deployment Notes

This version introduces both Binary Authorization and multiple new services. To help ensure a successful upgrade, operators should taint the build step before running Terraform. This will ensure new services are built and existing services are attested with the proper signatures for authorization.

terraform taint module.en.null_resource.build

Note this will increase the duration of the Terraform run to about 10 minutes. Upon a successful run, continue with the upgrade process as normal.

Key rotation

This release introduces unattended signing key rotation for both verification token and verification certificate signing.

All key rotation is done, by default, every 30 days.

Verification token key rotation

Keys are now backed by the database, seeded by the key ring that the current token signing key version is on.
Until the new rotate job runs, the legacy key version will be used for signing. Once the rotate job has been
run, there will be a new signing key used to sign new verification tokens. The legacy signing key will be honored
as long as it is still configured. v0.21.0 will move to only database-backed token signing keys.

Verification certificate key rotation

In this release, realm admins will be allowed to configure their signing keys for automatic rotation.
It is important that this realm's key server be configured to use the JWKS public key discovery document
for the health authority signing keys.

Verification signing keys are created, but not made active for at least 1 hour so that there is time
for the key server to import the public key.

Changes by Kind

Breaking changes

• Potentially breaking! - Source token signing keys from the database. This completes the move of system token signing keys from environment variables to the database. This change attempts to be backward compatible, but server are encouraged to test changes in an isolated environment before upgrading production systems. (#1602, @sethvargo)

Features

• Add intelligence for redirecting post-login (#1550, @sethvargo)
• Add audits for saving a Token Signing Key (#1601, @sethvargo)
• Add password-reset and email-verification email templates to realm settings (#1555, @whaught)
• Add rotation service and schema for rotating token signing keys (#1597, @sethvargo)
• Added support for Twilio messaging services (#1526, @whaught)
• Adds database column for realm level auto key rotation settings. (#1594, @mikehelmick)
• Adds the stats-puller service to run every hour. (#1603, @whaught)
• Advanced searching is a new feature for user search that allows for more granular searching. It's backwards-compatible with the former search syntax, but now includes the ability to filter searches by fields. Specifically, you can now query like "name:foo" or "can:APIKeyWrite" to perform advanced queries. Bulk permission management is a new feature that allows people with UserWrite permissions to modify other users' permissions in bulk in their realm. (#1525, @sethvargo)
• Allow JWK lookup URLs to work with region codes in addition to IDs (#1559, @sethvargo)
• Allow for legacy signing key config for tokens to be used during the upgrade to DB backed tokens. (#1624, @mikehelmick)
• Allow for realm admins to enable automatic verification certificate key rotation. Keys are rotated every 30 days, and given 1h soak time for upstream key servers to pull public keys via public key discovery. (#1614, @mikehelmick)
• Realm stats: Slider control to select date range to display. Trend lines are shown for codes issued and codes claimed. (#1598, @mikehelmick)
• Stop bulk-issue client if the Twilio SMS queue is full (#1571, @whaught)
• Support testType field as input for bulk-upload (#1531, @whaught)
• The bulk-uploader will skip conflict or already-succeeded lines when re-using a log csv from a previous attempt (#1530, @whaught)

General bug fixes and improvements

• Add locking to stats-puller service. Refactor cleanup -> lock (#1608, @whaught)
• Add optional enx-redirect tests to e2e-runner (#1585, @sethvargo)
• Add tests for cleanup (#1600, @sethvargo)
• Check session expiration on login (#1549, @sethvargo)
• Clean up nits from verification certification rotation (#1618, @sethvargo)
• Cleanup for key-server stats (#1615, @whaught)
• Correct padding (#1541, @whaught)
• Defensively check deleted_at fields in realm/user join tables for memberships (#1565, @sethvargo)
• Delete any orphaned entries before creating foreignkey. This handles an extreme edge case where an entry was manually deleted from the database before the FK constraint existed. (#1533, @sethvargo)
• Delete users who are deleted (#1566, @sethvargo)
• Display realm_id in system admin event logs (#1605, @sethvargo)
• Don't return http.Response when unneeded (#1586, @sethvargo)
• Drop foreignkey constraints if they already exist before creating (#1534, @sethvargo)
• Drop optional calculation of Daily Active Users (#1544, @sethvargo)
• Fix nits from signing key cleanup (#1619, @sethvargo)
• Garbage collection of references to deleted signing keys. (#1616, @mikehelmick)
• I8n strings for bulk-issue (#1509, @whaught)
• Log URLs and response codes in appsync errors, increase max size to 64kb (#1545, @sethvargo)
• Move associated site data responses into api package (#1577, @sethvargo)
• Move e2e client into internal/ (#1578, @sethvargo)
• New tables for storing key-server statistics (#1599, @whaught)
• Not-found and invalid for malformed realm_id in jw...

v0.19.2

Changes since v0.19.1

App-sync service

• Log URLs and response codes in appsync errors, increase max size to 64kb (#1545, @sethvargo)
• Remove leading slash in clients (#1547, @sethvargo)

Database migrations

• Drop foreignkey constraints if they already exist before creating (#1534, @sethvargo)
• Drop optional calculation of Daily Active Users (#1544, @sethvargo)

Monitoring, logging, and alerting

• Retry database pings in health checks (#1537, @sethvargo)
• Simplified alerting configuration to accommodate low-traffic services (#1535, @yuriatgoogle)

Bulk upload

• Correct padding (#1541, @whaught)
• Use outer padding for bulk-uploader client (#1542, @whaught)

Misc

• Display test coverage displayed on presubmit (#1540, @mikehelmick)

Dependencies

Added

Nothing has changed.

Changed

Nothing has changed.

Removed

Nothing has changed.

v0.19.1

Release notes for main

Documentation

Changelog since v0.19.0

Changes by Kind

New Features

• Added support for Twilio messaging services (#1526, @whaught)

• Advanced searching is a new feature for user search that allows for more granular searching. It's backwards-compatible with the former search syntax, but now includes the ability to filter searches by fields. Specifically, you can now query like "name:foo" or "can:APIKeyWrite" to perform advanced queries.

  Bulk permission management is a new feature that allows people with UserWrite permissions to modify other users' permissions in bulk in their realm. (#1525, @sethvargo)

Bulk Upload

• Support testType field as input for bulk-upload (#1531, @whaught)
• The bulk-uploader will skip conflict or already-succeeded lines when re-using a log csv from a previous attempt (#1530, @whaught)
• I8n strings for bulk-issue (#1509, @whaught)

Fixes

• Delete any orphaned entries before creating foreignkey. This handles an extreme edge case where an entry was manually deleted from the database before the FK constraint existed. (#1533, @sethvargo)
• Drop foreignkey constraints if they already exist before creating (#1534, @sethvargo)
• Monitoring: Add ClouldSchedulerJobFailed alert. (#1527, @yegle)
• Only emit an error when shouldCleanup actually fails (#1520, @sethvargo)
• Return NotFound for invalid UUIDs (#1523, @sethvargo)
• Switch modeler logs to debug (#1522, @sethvargo)

Dependencies

Added

Nothing has changed.

Changed

Nothing has changed.

Removed

Nothing has changed.

v0.19.0

Release notes for main

Documentation

Changelog since v0.18.0

Changes by Kind

Breaking change

• Breaking: To continue using the Terraform module, the following input variable is needed to avoid introducing a diff (#1513, @yegle)

  revision_annotations = {                                                       
      adminapi     = { "autoscaling.knative.dev/maxScale" : "1000" }
      apiserver    = { "autoscaling.knative.dev/maxScale" : "1000" }
      appsync      = { "autoscaling.knative.dev/maxScale" : "1000" }
      cleanup      = { "autoscaling.knative.dev/maxScale" : "1000" }
      e2e-runner   = { "autoscaling.knative.dev/maxScale" : "1000" }
      enx-redirect = { "autoscaling.knative.dev/maxScale" : "1000" }
      modeler      = { "autoscaling.knative.dev/maxScale" : "1000" }
  } 

Monitoring

• Improved SLO-based alerting reset time (#1294, @yuriatgoogle)
• Add SLO chart in Verificatoin Server dashboard. (#1315, @yegle)
• Monitoring: Add Availability SLO alert to all Cloud Run services. (#1351, @yegle)
• Add Latency SLO and related alert. Currently a rudimentary threshold (90% requests are returned in <10s) is set on all services. (#1361, @yegle)
• Modified service and SLOs configurations for better configurability (#1449, @yuriatgoogle)
• Latency alerting done via threshold, rather than SLO (#1510, @yuriatgoogle)

Role based authentication

• *Major change- Introduce Role-Based Access Controls (RBAC) to replace legacy user/admin roles. Existing users will retain their existing permissions, but new users will be able to have more granular permissions. This change involves an *irreversible database migration- and should be planned accordingly. We recommend system operators put the servers into maintenance mode before applying these migrations. (#1335, @sethvargo)
• Add descriptions for RBAC permissions (#1405, @sethvargo)
• Improve UX on permissions selection (#1435, @sethvargo)
• Require UserWrite for admin reset of another user's password (#1445, @whaught)
• Document implied permissions, enforce via javascript (#1418, @sethvargo)
• Setting write permissions automatically add required read permissions. (#1411, @mikehelmick)

Bulk issue codes

• Add SMS template selection to bulk uploader (#1370, @whaught)
• Clearer outer error message for batch issue API (#1369, @whaught)
• End to end test runner now tests bulk issue in a separate handler and scheduler (#1436, @mikehelmick)
• Remember the issuing user's last used SMS template. (#1379, @whaught)
• Fix logging error in batch issue API. (#1336, @mikehelmick)
• Fix off-by-one line numbers for bulk uploader (#1460, @whaught)
• Integration / e2e test cases for BatchIssueCode (#1376, @whaught)
• Show first 50 success / error cases for bulk-issue with UUIDs
  Allow download of log file for bulk-issue (#1458, @whaught)
• RBAC check for BulkIssue on the API (#1400, @whaught)
• Return a more detailed error when bulk issue is not enabled (#1331, @sethvargo)
• Adding more docs (#1365, @whaught)

Statistics

• Add more left padding to graphs for large y-axis numbers. (#1342, @sethvargo)
• Add statistics endpoints to adminapi (#1402, @sethvargo)
• Move stats calculations to be out-of-band (#1500, @sethvargo)
• Move user stat caching into the model (#1494, @sethvargo)
• Only display daily actives graph if enabled (#1398, @sethvargo)
• Serve user statistics via javascript (#1496, @sethvargo)
• Make daily active stats collection a realm configurable (#1396, @sethvargo)
• Introduce a new API key type for accessing statistics. The statistics endpoints of the admin API are currently in preview and are subject to change. (#1404, @sethvargo)
• Collect invalid codes and token statistics (backend only) (#1499, @sethvargo)

Internationalization

• Add ph translations (#1407, @sethvargo)
• Allow realm to have multiple SMS templates. Adds UI in realm settings to select and edit templates. (#1338, @whaught)
• Allow user to select an SMS template on code-issue. Add template label field to issueAPI. (#1352, @whaught)
• Enable postgres hstore. Add fields to realm to store multiple SMS templates and add validation for them. (#1325, @whaught)
• Japanese (ja) translation (#1303, @yuryu)

Redirect mobile apps

• Fix json output for iOS universal links (#1374, @whaught)
• Fixes to iOS Universal Links formatting (#1308, @whaught)
• Give admins the ability to disable AppStore redirects for apps (#1466, @sethvargo)
• Display OS on the mobile apps index table (#1484, @whaught)
• Switch to older Associated Domains format for iOS (#1307, @whaught)

Fixes

• Users added to realm don't get password reset email - only newly created firebase users do (#1395, @whaught)
• Wait for Redis connections to become available (#1419, [@sethvargo](https://github.com/sethvar...

v0.18.5

Release notes for v0.18.5

Documentation

Changelog since v0.18.4

Changes

• change 400 to 404 for NotFound (commit/492acce71d4af8568ff73999ca061f180356bd0c, @mikehelmick)
• Make SMS message failures info level (#1420, @sethvargo )
• Extract trace context if present and add to logger (#1422, @sethvargo )

Dependencies

Added

Nothing has changed.

Changed

Nothing has changed.

Removed

Nothing has changed.

v0.18.4

• cherry-pick of c5acc33 - have ENX redirect service treat unknown paths/domains as 404

v0.18.3

• scrub phone numbers (#1360), @mikehelmick)
• Return 400 (instead of 500) on Twilio errors (#1313, @sethvargo)
• fix json output (#1374)

v0.18.2

Release notes for v0.18.2

Documentation

Changelog since v0.18.1

Changes

• Switch to older Associated Domains format for iOS (#1307, @whaught)
• Update to iOS mobile apps format (#1308, @whaught)

Dependencies

Added

Nothing has changed.

Changed

Nothing has changed.

Removed

Nothing has changed.