GitHub - jhzab/CDXCreator: A CDX file creator written in Scala for Spark

jhzab / CDXCreator Public

Notifications You must be signed in to change notification settings
Fork 0
Star 1

A CDX file creator written in Scala for Spark

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 11 Commits
project		project
src/main/scala/de/l3s/CDXExtractor		src/main/scala/de/l3s/CDXExtractor
README		README
build.sbt		build.sbt

Repository files navigation

# CDXCreator

CDXCreator is a Spark application that creates CDX files from (W)ARC files.

It expects to be run under Spark and receives two arguments: '--input' and
'--output'. The former can also be a glob. Output is written in a "CSV" format
with " " as the seperator. The output filenames are random.

The files currently **don't** have a CDX header.