The JSON library is very slow

[konnectr]

Problem:

import time
import json
import orjson
import ujson
import simplejson

def benchmark(name, dumps, loads):
    start = time.time()
    for i in range(3000000):
        result = dumps(m)
        loads(result)
    print(name, time.time() - start)

if __name__ == "__main__":
    m = {
        "timestamp": 1556283673.1523004,
        "task_uuid": "0ed1a1c3-050c-4fb9-9426-a7e72d0acfc7",
        "task_level": [1, 2, 1],
        "action_status": "started",
        "action_type": "main",
        "key": "value",
        "another_key": 123,
        "and_another": ["a", "b"],
    }

    benchmark("Python", json.dumps, json.loads)
    benchmark("ujson", ujson.dumps, ujson.loads)

    # orjson only outputs bytes, but often we need unicode:
    benchmark("orjson", lambda s: str(orjson.dumps(s), "utf-8"), orjson.loads)

    benchmark('simplejson', simplejson.dumps, simplejson.loads)

Results:

Python 11.381623983383179
ujson 5.0240020751953125
orjson 2.138978958129883
simplejson 15.721292972564697

Redash uses simplejson, which is very slow
Solution:
Use another library to serialize and deserialize json

[justinclift]

As long as it doesn't break things for our existing users, then it sounds like a decent idea. 😄

[arikfr]

Replacing the library should be relatively easy as in most places we use our own wrapper (utils.json_dumps /utils.json_loads), so we need to update in a single place.

I remember that simplejson supports the ignore_nan option which I think the built in one didn't.