Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API: implement SRR and various maintenance/improvement tasks. #118

Merged
merged 51 commits into from
Oct 1, 2021

Conversation

stsievert
Copy link
Owner

@stsievert stsievert commented Jul 15, 2021

What does this PR implement?
It allows the following:

  • Implements an "synchronous round robin" scheme that searches n_search queries (instead of asynchronously).
  • Specifying the triplets indices in validation sampling.
    • ...by specifying queries: [[0, 1, 2], [3, 4, 5]] in init.yaml under the samplers heading.
  • Customizing the participant ID
    • ...by visiting http://localhost:8421/?puid=foobar.
  • Allow customizing samplers_per_user
    • Related: API: allow /query?ident=.
    • ...by specifying samplers_per_user in init.yaml.
  • Allow user to initialize embedding with pre-computed embedding
    • (which is especially relevant given the downloadable online embedding)
  • Doc changes:
    • Clarify sampling key in init.yml.
    • Change "ARR" to mean "async RR" instead of "adaptive RR"
    • Add initial embedding to offline embedding
  • Maint:
    • better error handling on dashboards/getting embeddings.
    • Better HTML serving of targets (prefetch instead of custom solution)
    • Adds torch.no_grad in score.

Backwards incompatible changes

  • Renames RandomSampling to Random to fit with naming scheme of other samplers.
    • (but warns if wrong file uploaded)
  • (internal) renames Runner to Sampler.

Reference issues/PRs
This closes #114 and #117.

@stsievert stsievert changed the title API: allow specification of triplets in validation API: allow customization of triplets in validation & participant ID Jul 15, 2021
@stsievert
Copy link
Owner Author

@crcox does this resolve your issue? I think this is similar to the external ID/ext_id you mentioned in #117.

@crcox
Copy link
Contributor

crcox commented Jul 15, 2021

Yes, that will do the trick.
Clarification on validation queries: is it something like

samplers: {
    Validation: {},
    queries: [[0, 1, 2], [3, 4, 5]]
}

or

samplers: {
    Validation: {
        queries: [[0, 1, 2], [3, 4, 5]]
    }
}

?

@stsievert
Copy link
Owner Author

stsievert commented Jul 15, 2021

basic.yaml shows an example of how to specify the indices. Here's a snippet:

samplers:
  ARR: {}
  Random: {}
  Validation:
    queries: [[0, 1, 2], [3, 4, 5], [6, 7, 8]]

(updated to reflect sampler renaming)

@crcox
Copy link
Contributor

crcox commented Jul 15, 2021

Perfect, thanks!

@stsievert stsievert changed the title API: allow customization of triplets in validation & participant ID API: various customizations (samplers, validation sampling, participant ID) Jul 19, 2021
* Rename RandomSampling to Random
* hasattr(self, 'get_queries') → self.run_get_queries
@stsievert stsievert changed the title API: various customizations (samplers, validation sampling, participant ID) API: implement ARRProxy and renaming, cleaning, improvements. Jul 26, 2021
@stsievert stsievert changed the title API: implement ARRProxy and renaming, cleaning, improvements. API: implement SRR and various maintenance/improvement tasks. Oct 1, 2021
@stsievert
Copy link
Owner Author

Most tests are passing locally, all except 3. That's a non-issue; I'm testing that errors get logged, and can't restart docker in the middle of the tests. If I could, those tests would pass.

@stsievert stsievert merged commit 4dfc1b3 into master Oct 1, 2021
@stsievert stsievert deleted the validation-specify-queries branch October 1, 2021 18:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Allow user to specify queries in validation sampling
2 participants