saveAsBigQueryTable exception with StructType column

[andreclaudino]

I am trying to save data using saveAsBigQueryTable, but this is resulting in an error. I simplified the dataframe for a single column, an StructType one, this is the schema:

root
 |-- contexto: struct (nullable = true)
 |    |-- b2wChannel: string (nullable = true)
 |    |-- b2wDeviceType: string (nullable = true)
 |    |-- b2wEPar: string (nullable = true)
 |    |-- b2wOpn: string (nullable = true)
 |    |-- b2wPid: string (nullable = true)
 |    |-- b2wSid: string (nullable = true)
 |    |-- b2wUid: string (nullable = true)
 |    |-- customerId: string (nullable = true)
 |    |-- salesSolution: string (nullable = true

When using other fields (of other types) the dataset is saved as expected, so, the error is in StructType columns.

This is the stack

[error] (run-main-0) java.lang.NoSuchMethodError: org.json4s.JsonDSL$.seq2jvalue(Lscala/collection/Traversable;Lscala/Function1;)Lorg/json4s/JsonAST$JArray;
[error] java.lang.NoSuchMethodError: org.json4s.JsonDSL$.seq2jvalue(Lscala/collection/Traversable;Lscala/Function1;)Lorg/json4s/JsonAST$JArray;
[error] 	at com.samelamin.spark.bigquery.converters.SchemaConverters$$anonfun$typeToJson$3.apply(SchemaConverters.scala:66)
[error] 	at com.samelamin.spark.bigquery.converters.SchemaConverters$$anonfun$typeToJson$3.apply(SchemaConverters.scala:66)
[error] 	at org.json4s.JsonDSL$JsonAssoc.$tilde(JsonDSL.scala:90)
[error] 	at com.samelamin.spark.bigquery.converters.SchemaConverters$.typeToJson(SchemaConverters.scala:66)
[error] 	at com.samelamin.spark.bigquery.converters.SchemaConverters$.com$samelamin$spark$bigquery$converters$SchemaConverters$$fieldToJson(SchemaConverters.scala:84)
[error] 	at com.samelamin.spark.bigquery.converters.SchemaConverters$$anonfun$1.apply(SchemaConverters.scala:87)
[error] 	at com.samelamin.spark.bigquery.converters.SchemaConverters$$anonfun$1.apply(SchemaConverters.scala:87)
[error] 	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
[error] 	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
[error] 	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
[error] 	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
[error] 	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
[error] 	at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
[error] 	at com.samelamin.spark.bigquery.converters.SchemaConverters$.SqlToBQSchema(SchemaConverters.scala:87)
[error] 	at com.samelamin.spark.bigquery.BigQueryDataFrame.saveAsBigQueryTable(BigQueryDataFrame.scala:41)
[error] 	at ml.freight.prepare.CarregaPedidos$.delayedEndpoint$ml$freight$prepare$CarregaPedidos$1(CarregaPedidos.scala:24)
[error] 	at ml.freight.prepare.CarregaPedidos$delayedInit$body.apply(CarregaPedidos.scala:6)
[error] 	at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
[error] 	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
[error] 	at scala.App$$anonfun$main$1.apply(App.scala:76)
[error] 	at scala.App$$anonfun$main$1.apply(App.scala:76)
[error] 	at scala.collection.immutable.List.foreach(List.scala:392)
[error] 	at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
[error] 	at scala.App$class.main(App.scala:76)
[error] 	at ml.freight.prepare.CarregaPedidos$.main(CarregaPedidos.scala:6)
[error] 	at ml.freight.prepare.CarregaPedidos.main(CarregaPedidos.scala)
[error] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error] 	at java.lang.reflect.Method.invoke(Method.java:498)

[samelamin]

The problem is most probably in transforming the struct type to the bq schema

Can you provide a sample json record of the broken file so it can help the investigation