Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add eligible segment filter for bandits #7533

Merged
merged 1 commit into from
Jan 7, 2021
Merged

Conversation

tmancey
Copy link
Collaborator

@tmancey tmancey commented Jan 7, 2021

Resolves brave/brave-browser#13395

Submitter Checklist:

  • There is a ticket for my issue.
  • Used Github auto-closing keywords in the commit message.
  • Wrote a good PR/commit description
  • Added appropriate labels (QA/Yes or QA/No; release-notes/include or release-notes/exclude; OS/...) to the associated issue
  • Checked the PR locally: npm run test -- brave_browser_tests, npm run test -- brave_unit_tests, npm run lint, npm run gn_check, npm run tslint
  • Ran git rebase master (if needed).
  • Requested a security/privacy review as needed.

Reviewer Checklist:

  • New files have MPL-2.0 license header.
  • Adequate test coverage exists to prevent regressions
  • Major classes, functions and non-trivial code blocks are well-commented
  • Changes in component dependencies are properly reflected in gn
  • Code follows the style guide
  • Test plan is specified in PR before merging

After-merge Checklist:

Test Plan:

Run build with features cli args, e.g. --enable-logging=stderr --vmodule="*/bat-native-ads/*"=9,"*/brave_ads/*"=9,"*/brave_user_model/*"=9,"*/bat_ads/*"=9 --brave-ads-staging --rewards=staging=true --disable-features="TextClassification" --enable-features="EpsilonGreedyBandit"

Case 1: Default models enabled (Text Classification and Purchase Intent)

  • Verify that Text Classification and Purchase Intent models are enabled and Epsilon Greedy Bandit is disabled: Text classification feature is enabled and Purchase intent feature is enabled
  • Trigger ad notification and verify that ads are served as usual via text classification
  • Process purchase intent segment and verify keywords are extracted:
    Extracting purchase intent signal from visited URL and Extracted purchase intent signal from visited URL
  • Verify that bandit is processing (despite the model being disabled, that is by design)
  • Verify that the model is not suggesting segments to the ad serve pipeline. Shouldn't see Exploring epsilon greedy bandit segments: ... or Exploiting epsilon greedy bandit segments: ...

Case 2: Variant models enabled (Epsilon Greedy Bandit and Purchase Intent: --disable-features="TextClassification" --enable-features="EpsilonGreedyBandit")

  • Verify that Epsilon Greedy Bandit and Purchase Intent models are enabled and Text Classification is disabled: Epsilon greedy bandit feature is enabled and Purchase intent feature is enabled and Text classification feature is disabled
  • Process purchase intent segment and verify keywords are extracted:
    Extracting purchase intent signal from visited URL and Extracted purchase intent signal from visited URL
  • Verify that bandit is processing
  • Verify that the model is suggesting segments to the ad serve pipeline Exploring epsilon greedy bandit segments: ... or Exploiting epsilon greedy bandit segments: ...

Case 3: Restart after one round of training and make sure state wasn't deleted in user Preferences JSON key epsilon_greedy_bandit_arms

@tmancey tmancey self-assigned this Jan 7, 2021
@tmancey tmancey force-pushed the issues/13395 branch 4 times, most recently from 7074c3d to cafefcf Compare January 7, 2021 09:39
Copy link
Contributor

@moritzhaller moritzhaller left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM++

@moritzhaller
Copy link
Contributor

moritzhaller commented Jan 7, 2021

CI failing on macos due to known intermittent rollup issues

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add eligible segment filter for bandits
2 participants