Discussion: Idiom for loading DataFrames

[lintool]

In my original implementation I wrote a DataFrameLoader, but it seems to have rapidly fallen out of use... We should decide on the idiom we want for loading DataFrames.

Current implementation:

val df = RecordLoader.loadArchives("example.arc.gz", sc).extractImageDetailsDF()
// alternatively, extractValidPagesDF, extractHyperlinksDF, etc.

The downside of this is that the user has access to raw RDDs, which is what loadArchives returns... this is asking for trouble in mixing RDDs and DFs in unpredictable ways?

Another option would be to introduce a DF interface that does not give access to RDDs. Something like:

val df = DataFrameLoader.loadArchives("example.arc.gz", sc).images

The other nice feature is that we can have much shorter DF names like pages, links, images, image_links, etc. - don't need the DF part to disambiguate because DataFrameLoader makes this clear. One more nice features is the ability to selectively reduce scope down the road and hide RDDs from the user, as we move completely over to DFs.

I'm leaning towards this design, but would be happy to hear opinions from others...

[ianmilligan1]

The other nice feature is that we can have much shorter DF names like pages, links, images, image_links, etc. - don't need the DF part to disambiguate because DataFrameLoader makes this clear. One more nice features is the ability to selectively reduce scope down the road and hide RDDs from the user, as we move completely over to DFs.

I'm generally agnostic but this pushes me in the camp of having a DF-specific interface. The second syntax example you gave with the .images is very usable.

[ruebot]

Fine by me. I can see moving towards strict DataFrames helping out on the AUK side of things.

[jwli229]

+1 for strict dataframes and hiding away RDDs

[ruebot]

I think #350 hits this, and/or resolves it. I'll leave that to @lintool

[lintool]

👍

We can close this issue after #350 is merged.

[ianmilligan1]

Closed with e32ae17.