Vector PRF

[cmacdonald]

WIP.

Our results:

Results from Pyserini:

(I think they use FAISS as a retrieval backend)

[seanmacavaney]

Looks great, thanks! I prefer class-based Transformer definitions for stuff that isn't ad hoc. I'm happy to convert it to that format if you're okay with it?

[cmacdonald]

I guess the advantage of a transformer is that its parameters can be tuned in a gridsearch.

I think we should commit a notebook to the examples folder. I have one that can be associted.

[seanmacavaney]

Sounds good. Yeah, there are other benefits too, like you can get a clearer repr.

I think more extensive tests would also be good. I just put in one basic one to make sure I didn’t mess something up in the new version.

[seanmacavaney]

Thoughts on replacing this boilerplate with a decorator?

# BEFORE:

    def transform(self, inp: pd.DataFrame) -> pd.DataFrame:
        """Transforms the input DataFrame query-by-query."""
        return pt.apply.by_query(self.transform_by_query, add_ranks=False)(inp)

    def transform_by_query(self, inp: pd.DataFrame) -> pd.DataFrame:
        pta.validate.result_frame(inp, extra_columns=['query', 'query_vec', 'doc_vec'])
        ...

# AFTER:

    @pta.transform.by_query(add_ranks=False)
    def transform(self, inp: pd.DataFrame) -> pd.DataFrame:
        pta.validate.result_frame(inp, extra_columns=['query', 'query_vec', 'doc_vec'])
        ...

It could live in alpha as we refine on the idea/name/etc.

[cmacdonald]

sure.

[cmacdonald]

It's a fairly niche style, thus far. Perhaps setup a gist so we can continue discussion there. I guess this in effect making transform() methods that can be slightly specialised (eg guaranteed to be called once per query) without needing to have multiple more transform_*() methods in the API specification

[seanmacavaney]

Yeah, I'd rather not have any more transform_*() definitions in pt.Transformer if possible.

I'm also no stranger to just doing a groupby in the definition. It ends up being a few lines of boilerplate, but nothing crazy.

[seanmacavaney]

Alright, I'm happy with this version.

I went ahead and added @pta.transform.by_query to alpha. I'm happy to discuss further for a version to include in the core API, but I didn't want to hold up this PR on those discussions.

[cmacdonald]

ok, I'll make a separate PR with a notebook.

[cmacdonald]

Going to merge it?

[seanmacavaney]

Ok! Sounds like it's urgent.