feature request: possibility to query about the progress

[dportabella]

related to #156
it would great to have the possibility to query about the progress, something such as:

case class WarcFileReport(file: File, started: Boolean, finished: Boolean, completed: Double)

def progressFn(warcFileReports: List[WarcFileReport]) {
  warcFileReports.foreach(println)
}

RecordLoader.loadArchives(file, sc, progressFnEvery = 1 hour, progressFn)
.foreach(archiveRecord => println(s"+++ ${archiveRecord.getUrl}"))

Is something like that feasible?

[ianmilligan1]

Thanks @dportabella - we'll investigate.

[ruebot]

I believe this be better resolved with just using the Spark web interface. That gives you some really nice detail, and recent versions of Spark have decent status indicators in the shell interface.

If it isn't, please feel free to re-open the issue.