A demo project which will Read and Write a dataframe to MySQL DB.

Build System (Gradle)

------------------------------------------------------------
Essential tasks runnable from root project
------------------------------------------------------------

Build tasks
-----------
build - Assembles and tests this project.
clean - Deletes the build directory.
jar - Assembles a jar archive containing the main classes.

Help tasks
----------
tasks - Displays the tasks runnable from root project 'SparkReadWriteDF'.

IDE tasks
---------
cleanEclipse - Cleans all Eclipse files.
eclipse - Generates all Eclipse files.

Verification tasks
------------------
check - Runs all checks.
test - Runs the unit tests.

Other tasks
-----------
checkScoverage
jarScoverage

Input Resources

• Common Resources

  • record.json has JSON records which will be written to DB.
  • dbArgs.json MySQL DB Connection Details are present here.
  • sparkArgs.json Spark Engine Configuration Details are present here.

• For src/main/ specific

  • input.json contains input arguments for the EntryPoint.

• For src/test/ specific

  • testInput1.json contains input arguments for the EntryPointTest.
  • testInput1.fail.json contains input arguments for the EntryPointTest.
  • testInput2.json contains input arguments for the EntryPointTest.
  • testInput2.fail.json contains input arguments for the EntryPointTest.

How to run?

• Install Gradle 2.14 Build Tool (click here for reference)

• For eclipse, install Buildship 1.0.16 Plugin for Gradle 2.14 (click here for reference)

  NOTE: Installing Gradle 3.0 or above will give following error

  No such property: useAnt for class: org.gradle.api.tasks.scala.ScalaCompileOptions

  This is because useAnt support is removed from Gradle 3.0 onwards. So prefer sticiking to 2.14 version.

• Setup MySQL server as mentioned below separately

• Run any of the above mentioned gradle tasks, for example, run any one of the following line -

  gradle build
  
  gradle test
  
  gradle checkScoverage

How to setup MYSQL DB?

• Install MySQL (for example like this)
• Login To Root
• Create a new user (like this)
• Create a database (like this)
• Create a simple SQL table (like this)
• Update following JSON file with MySQL login details: dbArgs.json

Scala Unit Tests (ScalaTest Tool)

The above shown unit testing report is present at location build/reports/tests/

Scala Code Coverage (SCoverage Tool)

The above shown code coverage report is present at location build/reports/scoverage/