Different read and write number of rows

[celoibarros]

Hi there,

I am getting some differences when reading from BQ and writing to SQL Server.
I tried locally, if i set

.master("local") the read dataframe as 160k rows and after the dataframe write i only get 80k in SQL Server table.

If i run the same with .master("local[*]") and get the same number in the read and write.

But when i run the code in the cluster
--master "yarn"
--deploy_mode "cluster"
i am still getting differences.

Do you have any idea what is happening?
It looks like some "partitions" are not being write.

Best regards.