Add Databricks profile

[anthonysena]

Per feedback from @konstjar, I've added a new profile "webapi-databricks" to use the Databricks JDBC Driver reference. Some notes on this PR:

• Using the latest Databricks driver (v2.6.36) was problematic in Apache Tomcat 8.5.x. When using this driver, the servlet container would crash and WebAPI failed to start. v2.6.34 did not exhibit this behavior hence that is why it is referenced in the pom.xml
• Added com.databricks.client.jdbc.Driver to the list of supported drivers to support this newer JDBC driver.

[anthonysena]

From discussion with @konstjar on the Atlas WG call, we decided to create a "webapi-databricks" profile and retain the "webapi-spark" profile for backwards compatibility. I'll revise this PR accordingly.

[konstjar]

@anthonysena

My filling is that it is not enough to add the profile.

There are few places in WebAPI where we have some custom logic for Spark dialect:
https://github.com/OHDSI/WebAPI/blob/master/src/main/java/org/ohdsi/webapi/cohortcharacterization/DropCohortTableListener.java#L52

WebAPI/src/main/java/org/ohdsi/webapi/util/CancelableJdbcTemplate.java

Line 81 in 3f9e90c

if (stmt.getConnection().getMetaData().getURL().startsWith("jdbc:spark")) {

WebAPI/src/main/java/org/ohdsi/webapi/cohortcharacterization/GenerateCohortCharacterizationTasklet.java

Line 106 in 3f9e90c

if (source.getSourceDialect().equals("spark")) {

WebAPI/src/main/java/org/ohdsi/webapi/util/PreparedStatementRenderer.java

Line 291 in 3f9e90c

if (targetDialect.equals("spark")) {

WebAPI/src/main/java/org/ohdsi/webapi/pathway/PathwayStatisticsTasklet.java

Line 173 in 3f9e90c

if (source.getSourceDialect().equals("spark")) {

I think that we should add "jdbc:databricks" handling there. In the past we had few issues with Characterization and Pathway analysis executions.

[anthonysena]

Thanks @konstjar - you've raised some important points here with respect to the code base and the "spark" profile.

Recalling our conversation on the Atlas WG call, I thought that we wanted to adopt the "databricks" profile since some users may be using the "spark" driver which may target an older build of Apache Spark for Databricks while others may need the newer driver? When setting up the new or old Spark data source in Atlas/WebAPI, the source would be configured with the dbms == "spark" but the connection string might differ based on the driver. So in this way, the only check that requires extension is the one that is looking at the JDBC connection string, correct? In that case we can have a static method to detect the dialect based on either "databricks" or "spark" in the JDBC connection string and that would work?

[anthonysena]

@konstjar - I've updated this PR with the idea that when a person configures a data source to use spark/databricks, they would create the entry in the source table with the dialect == spark and for the connection string they could either provide the jdbc:spark or jdbc:databricks connection details. So, the checks in the code that you pointed out earlier in this PR remain the same with the exception of the one that checks the connection string details. I've modified this to check both jdbc URL formats and that should hopefully handle backwards compatibility. If you agree, this should be ready to merge.

[konstjar]

@anthonysena Looks good to me.

[anthonysena]

Thanks @konstjar - can you approve the PR then?

[konstjar]

LGTM