Sorting on a String in WebUI table (sometimes) fails and generates error

[pete-petey]

I tried simpler tables, with just a string and it dod not generate the experience / error below.

Here is the one that did (repeatedly).
With this script, sorting in the table UI on anything other than the two strings (USym & Exchange) is fine. If I sort on either of those columns, I get the error cited below.

The query:

from deephaven.TableTools import timeTable
from deephaven import SortPair
from deephaven import ComboAggregateFactory as caf
from deephaven.conversion_utils import NULL_DOUBLE
import random

Symbols = ["SPY", "PFF", "FB", "DIS", "USO", "XLF", "XLU", "RBLX", "REM", "REZ", "GME", "HDSN"]
Exchanges = ["NYSE", "INET", "EDGX", "BATS", "EDGA" , "MEMX", "BATY", "XOTC", "OTCU"]
priceMap = {"PFF":37.50, "SPY":395.00, "USO":40.00, "XLF":31.50, "XLU":62.50, "REM":33.33, "REZ":71.33, "HDSN":1.85, "FB":270.00, "DIS":190.00, "GME":17.50, "RBLX":70}


def pricerFunc (USym,  E):
   _aPrice = priceMap[USym]
   if(_aPrice ==  NULL_DOUBLE):
       priceMap[USym] = abs(E)/100.0
   else:
       priceMap[USym] = (_aPrice*100 + (int)(E/2000))/100.0
   return priceMap[USym]


def distributerFunc(cnt):
    n = random.randint(0,(cnt*(cnt+1))/2) +1
    for i in range(cnt):
      n = n - (cnt-i)
      if (n <= 0):
          return i
          
x = distributerFunc(len(Symbols))
y = len(Exchanges)
TickingTable = timeTable("00:00:00.013").tail(100000).update("A=random.randint(0,10)", "B=random.randint(0,2)", "C=random.randint(0,50)", "D= ((int) (random.randint(0,100))/100.0 - 0.5) * 20000.0", "USym=Symbols[x-1]", "Size=Math.max(((int) random.randint(0, 11))*100, ((int) random.randint(0, 149))+1)", "Price=(double)pricerFunc(USym, D)", "TradeVal=Size*Price", "Exchange = Exchanges[(int)C%y]", "I=ii").dropColumns("C", "D")
Cumulative = TickingTable.update("TradeVal = Size * Price").by(caf.AggCombo(caf.AggSum("Volume = Size", "TradeVal"), caf.AggWAvg("Size", "VWAP=Price")), "USym", "Exchange").sort(SortPair.descending("TradeVal"))

Here is the error:

Sheduler-Concurrent-2 | i.d.g.s.SessionState      | Internal Error '45d932bf-e3db-4259-81f2-b58994417ccc' io.deephaven.base.verify.RequirementFailure: Requirement failed: required Comparable.class.isAssignableFrom(sortColumns[ii].getType()) || sortColumns[ii].getType().isPrimitive(), instead sortColumnNames[ii] == "Exchange", sortColumns[ii].getType() == class io.deephaven.db.tables.remote.preview.DisplayWrapper.
        at io.deephaven.base.verify.Require.fail(Require.java:108)
        at io.deephaven.base.verify.Require.requirement(Require.java:169)
        at io.deephaven.base.verify.Require.requirement(Require.java:175)
        at io.deephaven.db.v2.SortOperation.<init>(SortOperation.java:55)
        at io.deephaven.db.v2.QueryTable.sort(QueryTable.java:2229)
        at io.deephaven.grpc_api.table.ops.SortTableGrpcImpl.create(SortTableGrpcImpl.java:97)
        at io.deephaven.grpc_api.table.ops.SortTableGrpcImpl.create(SortTableGrpcImpl.java:20)
        at io.deephaven.grpc_api.table.TableServiceGrpcImpl$BatchExportBuilder.doExport(TableServiceGrpcImpl.java:450)
        at io.deephaven.grpc_api.table.TableServiceGrpcImpl.lambda$null$6(TableServiceGrpcImpl.java:319)
        at io.deephaven.grpc_api.session.SessionState$ExportObject.doExport(SessionState.java:824)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at io.deephaven.grpc_api.runner.DeephavenApiServerModule$ThreadFactory.lambda$newThread$0(DeephavenApiServerModule.java:147)
        at java.lang.Thread.run(Thread.java:748)

[nbauernfeind]

I believe #936 would fix this issue. Thanks for the report; I will use this example to validate that work.

[nbauernfeind]

I originally misunderstood what exactly the error was on this issue.

I boiled this down to a smaller example:

from deephaven import empty_table
Symbols = ["SPY", "PFF", "FB", "DIS", "USO", "XLF", "XLU", "RBLX", "REM", "REZ", "GME", "HDSN"]
t = empty_table(len(Symbols)).update("Sym = Symbols[i]")

The larger issue here is that the column is of type PyObject, which gets transformed into DisplayWrapper. Sort does not like either of these types. However, it does work if you explicitly cast the symbol to a String.

t = empty_table(len(Symbols)).update("Sym = (String)Symbols[i]")

If we want, we could make the DisplayWrapper class sortable; this is very easy to do:

diff --git a/engine/table/src/main/java/io/deephaven/engine/table/impl/preview/DisplayWrapper.java b/engine/table/src/main/java/io/deephaven/engine/table/impl/preview/DisplayWrapper.java
index 1e097040e..656006e76 100644
--- a/engine/table/src/main/java/io/deephaven/engine/table/impl/preview/DisplayWrapper.java
+++ b/engine/table/src/main/java/io/deephaven/engine/table/impl/preview/DisplayWrapper.java
@@ -1,6 +1,7 @@
 package io.deephaven.engine.table.impl.preview;
 
 import io.deephaven.configuration.Configuration;
+import org.jetbrains.annotations.NotNull;
 import org.jetbrains.annotations.Nullable;
 
 import java.io.Serializable;
@@ -9,7 +10,7 @@ import java.io.Serializable;
  * Wraps Objects that cannot be displayed (e.g. not serializable or an unknown class) and allows them to be displayed as
  * a String.
  */
-public class DisplayWrapper implements Serializable {
+public class DisplayWrapper implements Serializable, Comparable<DisplayWrapper> {
     private static final int MAX_CHARACTERS =
             Configuration.getInstance().getIntegerWithDefault("DisplayWrapper.maxCharacters", 10000);
     private final String displayString;
@@ -47,4 +48,9 @@ public class DisplayWrapper implements Serializable {
 
         return false;
     }
+
+    @Override
+    public int compareTo(@NotNull DisplayWrapper o) {
+        return toString().compareTo(o.toString());
+    }
 }

I'll reach out to @rcaudy for more guidance.

[nbauernfeind]

This was fixed by #4073.