Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TypeError: expected bytes, str found #14

Closed
petric3 opened this issue Apr 8, 2021 · 3 comments · Fixed by #16
Closed

TypeError: expected bytes, str found #14

petric3 opened this issue Apr 8, 2021 · 3 comments · Fixed by #16

Comments

@petric3
Copy link

petric3 commented Apr 8, 2021

Im not computer science guy (mechanical eng.), so hopefully the question is not too off. My project involves streaming millions of data (speed is very important) via websocket client delivered as string messages containing a system's operational parameters in json/dict format. One pushed message is a list (actual list in string) containing several dictionaries through which we iterate and analyse in real time. A 3 abbreviated separately pushed messages would look sth like this:

[{"t":1611064750942,"Signal003":"Q","Signal003_MODE":1,"Signal003_TARGET":11,"Signal003_NEXT":8,"Signal003_IRV":9.05,"Signal003_ONOFF":9.06,"Signal003_TEMP":5,"Signal001_PRES":98, ,"Signal001_ENT":3}]
[{"t":1611064750943,"Signal033":"T","Signal003_MODE":1,"Signal003_TARGET":5170,"Signal001_ONOFF":19,"Signal001_TEMP":91.28,"s":6445,"Signal033_IRV":[12],"Signal003_ENT":3},{"Signal003":"T","Signal003_MODE":2,"Signal003_TARGET":5171,"Signal001_ONOFF":8,"Signal001_TEMP":9.04,"s":100,"Signal033_IRV":[12],"t":1611064750943,"Signal003_ENT":3}]
[{"t":1611064750943,"Signal065":"T","Signal003_MODE":3,"Signal003_TARGET":5172,"Signal001_ONOFF":8,"Signal001_TEMP":9.04,"s":1000,"Signal033_IRV":[12],"Signal003_ENT":3},{"Signal003":"T","Signal003_MODE":1,"Signal003_TARGET":5173,"Signal001_ONOFF":8,"Signal001_TEMP":9.04,"s":100,"Signal033_IRV":[12],"t":1611064750943,"Signal003_ENT":3},{"Signal003":"T","Signal003_MODE":1,"Signal003_TARGET":7116,"Signal001_ONOFF":12,"Signal001_TEMP":9.04,"s":10,"Signal033_IRV":[14,12,37,41],"t":1611064750943,Signal003_ENT":3},{"Signal003":"T","Signal003_MODE":1,"Signal003_TARGET":961,"Signal001_ONOFF":19,"Signal001_TEMP":9.04,"s":4,"Signal033_IRV":[14,12,37,41],"t":1611064750943,Signal003_ENT":3},{"Signal003":"T","Signal003_MODE":1,"Signal003_TARGET":962,"Signal001_ONOFF":19,"Signal001_TEMP":9.04,"s":10,"Signal033_IRV":[14,12,37,41],"t":1611064750943,"Signal003_ENT":3}]

On your frontpage, I saw the example and if understood it correctly, the message should be in byte format, hence I understand the TypeError Im getting:

json_bytes = b'''
{
  "foo": [1,2,[3]]
}
'''

Would it be possible to parse a string message within the cysimdjson library as we have no influence on the type of the pushed message from the websocket? I hope the question is not too off, but since websockets libraries in conjunction with json formats are widely used, I was thinking the problem might be worth looking into. Also, I can imagine that one could do the string conversion somehow in python, but C speed would probably be affected. At the moment, I use orjson, which is pretty fast and working well, but looking at your results, it ignites the interest. Regardless of the answer, thank you for the efforts with the library.

@ateska
Copy link
Contributor

ateska commented Apr 8, 2021

I see. Can you please also provide the whole error output you are getting.
We intentionally avoid any unnecessary type checks since it costs a performance but if this is doable in e.g. exception handler, it could be added easily.

@petric3
Copy link
Author

petric3 commented Apr 8, 2021

Thanks for the reply. I completely understand the avoidance of type checks, so I don't expect too much, but maybe there's some work around or suggestion, thanks in advance. Here goes the error:
Traceback (most recent call last): File "/home/oem/Syncthing/Projects/Pump_diagnostics_TWU21/JSN_TEST/json_study.py", line 66, in <module> benchmark0(F"{'cysimdjson_parser':>15}", parser.parse) File "/home/oem/Syncthing/Projects/Pump_diagnostics_TWU21/JSN_TEST/json_study.py", line 50, in benchmark0 jsn = loads(m) File "cysimdjson/cysimdjson.pyx", line 359, in cysimdjson.JSONParser.parse TypeError: expected bytes, str found

ateska added a commit that referenced this issue Apr 9, 2021
@ateska ateska linked a pull request Apr 9, 2021 that will close this issue
@petric3
Copy link
Author

petric3 commented Apr 10, 2021

First test trial with parser.parse_string() works well, so the issue is fit to be closed. Then will test it thoroughly in coming week. Thank you, very much appreciated!

@petric3 petric3 closed this as completed Apr 10, 2021
ateska added a commit that referenced this issue Apr 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants