[BUG]: Timestamps can no longer be cast to long

[AndreasVolkmann]

Describe the bug
In version 0.9.0 we were able to run the following:

row.GetAs<long>(timestampColumn);

Where the target column is of type Timestamp.

However, in later versions the same code throws the following exception:

System.InvalidCastException: Unable to cast object of type 'Microsoft.Spark.Sql.Types.Timestamp' to type 'System.Int64'.
    at Microsoft.Spark.Utils.TypeConverter.ConvertTo[T](Object obj)
   at Microsoft.Spark.Sql.Row.GetAs[T](String columnName)

This PR may be related: #428.

It starts breaking in version 0.11.0 and I also tried in the latest version.

To Reproduce
I wrote a test to reproduce this in my project:

[TestMethod]
public void Convert_Timestamp_ToLong()
{
	// Arrange
	const string columnName = "c";
	var structType = new StructType(new[] { new StructField(columnName, new TimestampType()) });
	var dateTime = DateTime.Parse("2021-04-06 10:20:30");
	var ticks = (long)dateTime.Subtract(new DateTime(1970, 1, 1)).TotalMicroseconds();
	Console.WriteLine(ticks);
	var values = new object[] { ticks };
	var row = CreateInstance<Row>(values, structType);

	// Act
	var result = row.GetAs<long>(columnName);

	// Assert
	result.Should().Be(ticks);
}


private static T CreateInstance<T>(params object[] args)
{
	var type = typeof(T);
	var instance = type.Assembly.CreateInstance(
		type.FullName!,
		false,
		BindingFlags.Instance | BindingFlags.NonPublic,
		null,
		args,
		null,
		null);
	return (T)instance;
}

Expected behavior
Expected the cast to succeed as before.

[imback82]

@suhsteve Can you help when you get a chance?

[suhsteve]

The reason why this worked in 0.9.0 was because we didn't have complete support for TimestampType and we were just passing the raw unpickled/pickled long values to and from spark. The PR #428 you linked added support for the Timestamp type.

Looking at your example, it looks like row.GetAs<long>(timestampColumn) may be equivalent to something like row.GetAs<Timestamp>(timestampColumn).GetIntervalInMicroseconds() .

Since GetIntervalInMicroseconds() is internal, we either make this public or allow Timestamp columns to be converted to long when using Row.GetAs<> by adding support in Microsoft.Spark.Utils.TypeConverter ? I'm partial to the second approach.

What do you think @imback82 ?

[imback82]

Thanks @suhsteve for looking into this.

@AndreasVolkmann Can you just work with C#'s DateTime object? You could get the Timestamp object first as @suhsteve was suggesting row.GetAs<Timestamp>(timestampColumn) and just call ToDateTime:

spark/src/csharp/Microsoft.Spark/Sql/Types/Timestamp.cs

Line 118 in a6f4e91

public DateTime ToDateTime() => _dateTime;

, and you can convert DateTime object to any value you want to work with.

[imback82]

I am removing a bug label since casting the timestamp to long is not supported since 0.11; the current version is more type-safe.

[AndreasVolkmann]

@imback82 We worked around it by calling the internal functions that Steve mentioned.
It would definitely be great to have either option 1 or 2 from @suhsteve implemented.

[imback82]

Since GetIntervalInMicroseconds is simple to implement, can you introduce it in your app? We already expose DateTime object via ToDateTime. We are trying to maintain API parity for types that are available in Scala, Python, and .NET, and minimize .NET specific APIs if possible.