Cant decode token (invalid start byte)

[ghost]

Hi,
i try to discover the vacuum robot on macos.
mirobo discover does find it, but cannot decode or show the token:

(dustcloud-PBjoGnIX) bash-3.2$ mirobo discover --handshake true
INFO:miio.device:Sending discovery to with timeout of 5s..
WARNING:miio.device:error while reading discover results: 'utf-8' codec can't decode byte 0xb2 in position 3: invalid start byte
(dustcloud-PBjoGnIX) bash-3.2$ 5~

Is there some kind of workaround?
Or is it possible to write the token undecoded to a file?

[rytilahti]

Could you please try mirobo discover --debug --handshake true to get more output why it fails?

[ghost]

Great idea!
Thank you very mutch.

(dustcloud-PBjoGnIX) bash-3.2$ mirobo --debug discover --handshake true
INFO:miio.vacuum_cli:Debug mode active
INFO:miio.device:Sending discovery to <broadcast> with timeout of 5s..
DEBUG:miio.protocol:Unable to decrypt, returning raw bytes: b''
DEBUG:miio.device:Got a response: Container: 
    data = Container: 
        data =  (total 0)
        value =  (total 0)
        offset1 = 32
        offset2 = 32
        length = 0
    header = Container: 
        data = !1\x00 \x00\x00\x00\x00\x04o\x04\xb2\x00\x016\x9c (total 16)
        value = Container: 
            length = 32
            unknown = 0
            device_id = \x04o\x04\xb2 (total 4)
            ts = 1970-01-01 22:05:16
        offset1 = 0
        offset2 = 16
        length = 16
    checksum = KXY2qGP2N6jCu9NN (total 16)
WARNING:miio.device:error while reading discover results: 'utf-8' codec can't decode byte 0xb2 in position 3: invalid start byte

[rytilahti]

Looks like it doesn't work anymore in newer versions of construct, the last known version to work was construct==2.9.28 . You can downgrade the package for testing, and then update it back to 2.9.30 (in case some parts of the protocol are broken in 2.9.28, I'm not sure anymore what breaks and when).

[arekbulski]

It looks like a problem with one of String classes (there are 4 in construct) but I cant make what is the problem from just looking at this log. String classes recently added support for UTF16/32 but that clearly aint the problem, they also droped support for encoding=None but that aint either. Can you link me the exact source line that parses that string?

If its String or CString class, then encoding must be on this list: (utf-8 is not, utf8 is)
https://construct.readthedocs.io/en/latest/api/strings.html#construct.possiblestringencodings
If that happens to be the problem, send me a PR to include it.

[yawor]

All Construct code used for miio protocol is in miio.protocol module. There're no Strings in the Message struct.

The problem is with Message.header.value.device_id. It is defined as "device_id" / Hex(Bytes(4)). Up to Construct 2.9.28 this produced bytes in the decoded message, since 2.9.29 it produces construct.lib.hex.HexDisplayedBytes. When trying to decode() that object (miio/device.py at line 187) it raises the above exception.

[arekbulski]

Right, but you got it half wrong. Hex/HexDump still produce bytes or rather bytes-derived objects, its just that the content was changed. It used to produce hexlified bytes, now it produces the original bytes but with modified __repr__. See example:
https://construct.readthedocs.io/en/latest/misc.html#hex-and-hexdump

It seems I should expand the example:

>>> d = Hex(GreedyBytes)
>>> obj = d.parse(b"\x00\x00\x01\x02")
>>> obj
b'\x00\x00\x01\x02'
>>> print(obj)
unhexlify('00000102')

[arekbulski]

The code you pointed at:
https://github.com/rytilahti/python-miio/blob/master/miio/device.py#L187

Simple fix: (you probably need to adjust few other lines as well)

binascii.hexlify(m.header.value.device_id).decode(),

I will update Construct docs to mention this.

EDIT: For explanation, the decode method exists on returned bytes as well, but the contents is not hexlified and therefore contain values that arent "utf8" decodable.

[syssi]

Who is affected of this issue?

[arekbulski]

I updated the API page to be more accurate what Hex results in.
https://construct.readthedocs.io/en/latest/misc.html#hex-and-hexdump

If I may explain myself, the Hex class was never meant (by me) to transform bytes, it was meant to provide hex view of it. But I was sloppy and never got around to properly implementing it, until now. Thats why API change. Old implemenataion was just stale code that wasnt written by me.

[smartjx]

I'm facing the same issue when I try to connect a mi ir remote

root@hassbian:~# mirobo --debug discover --handshake true
INFO:miio.vacuum_cli:Debug mode active
INFO:miio.device:Sending discovery to with timeout of 5s..
DEBUG:miio.protocol:Unable to decrypt, returning raw bytes: b''
DEBUG:miio.device:Got a response: Container:
data = Container:
length = 0
value = (total 0)
offset1 = 32
data = (total 0)
offset2 = 32
header = Container:
length = 16
value = Container:
length = 32
unknown = 0
device_id = \x05;\xdc[ (total 4)
ts = 1970-01-01 22:07:47
offset1 = 0
data = !1\x00 \x00\x00\x00\x00\x05;\xdc[\x00\x0173 (total 16)
offset2 = 16
checksum = \xa5@yi\xcd\x0f\xb2yA\x1c\xc3\xe2b\xf5\xde\xbc (total 16)
WARNING:miio.device:error while reading discover results: 'utf-8' codec can't decode byte 0xdc in position 2: invalid continuation byte

[syssi]

@smartjx The issue will be fixed with the merge of https://github.com/rytilahti/python-miio/pull/226/files

[smartjx]

I've done pip3 install construct==2.9.31 but seems this issue has not been resolved

[syssi]

Please post logs and use construct 2.9.30.

[smartjx]

@syssi thank you for your reply.
My logs has not changed after I update to construct 2.9.31.
I don't know python. Could you please let me know if I can replace those files on my raspberry or I have to wait for new version of python-miio ? Thank you again.

[syssi]

SSH to your raspberry pi and provide the output of these commands:

sudo find / -name miio
sudo find / -name construct

[smartjx]

@syssi thank you for you guide.
I've replaced those 3 files you updated. But still there are issue when using mi ir remote.
Home Assistant shows Unable to discover a device

MIIO log :

root@hassbian:/tmp# mirobo --debug discover --handshake true
INFO:miio.vacuum_cli:Debug mode active
INFO:miio.device:Sending discovery to with timeout of 5s..
DEBUG:miio.protocol:Unable to decrypt, returning raw bytes: b''
DEBUG:miio.device:Got a response: Container:
data = Container:
offset2 = 32
value = (total 0)
length = 0
offset1 = 32
data = (total 0)
header = Container:
offset2 = 16
value = Container:
length = 32
unknown = 0
device_id = \x05;\xdc[ (total 4)
ts = 1970-01-02 20:45:15
length = 16
offset1 = 0
data = !1\x00 \x00\x00\x00\x00\x05;\xdc[\x00\x02u[ (total 16)
checksum = \xa5@yi\xcd\x0f\xb2yA\x1c\xc3\xe2b\xf5\xde\xbc (total 16)
INFO:miio.device: IP 192.168.1.73 (ID: 053bdc5b) - token: b'a5407969cd0fb279weeeg3e262f5debc'
INFO:miio.device:Discovery done

HA Log

Token not accepted by device : Unable to discover the device 192.168.1.73
‎8‎:‎33‎ ‎PM components/remote/xiaomi_miio.py (ERROR)
Unable to discover a device at address 192.168.1.73
‎8‎:‎33‎ ‎PM components/remote/xiaomi_miio.py (ERROR)
error while reading discover results: _decode() takes 3 positional arguments but 4 were given
‎8‎:‎33‎ ‎PM components/remote/xiaomi_miio.py (WARNING)`

[arekbulski]

error while reading discover results: _decode() takes 3 positional arguments but 4 were given

Thats definately the known Adapter/path bug, just hang on until #226 is merged.

[syssi]

@arekbulski The issue is already "fixed" in release 0.3.6. The error shouldn't occur because of the *args, **kwargs. PR #226 won't help here.

[smartjx]

@syssi
My current environment is following
Home Assistant 0.63.2
python-miio 0.3.6 with (chuangmi_ir.py,device.py,protocol.py replaced by your lastest submit)
construct 2.9.30
Linux hassbian 4.9.41-v7+
Python 3.5.3
Xiaomi IR Remote
Raspberry Pi 3 B

It seems 0.3.6 still has this issue.

I do found an issue open in HA github

home-assistant/core#12471

[arekbulski]

Of course it does. The */** hack that you dare to call a "patch" is failing.

[syssi]

I don't think so. It looks like there are multiple python-miio and/or construct versions installed.

@smartjx Please provide the output of

sudo find / -name miio
sudo find / -name construct

It will point to multiple installs.

[smartjx]

yes I do found multiple installs.

[smartjx]

@syssi
After I updated the *.py files with your latest submit , my Xiaomi ir remote works.

Thank you 👍

root@hassbian:/tmp/python-miio-master/miio# mv . /srv/homeassistant/lib/python3.5/site-packages/miio/