Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Windows10 - UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte #571

Closed
symonjim opened this issue Aug 16, 2022 · 15 comments
Closed

Windows10 - UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte #571

symonjim opened this issue Aug 16, 2022 · 15 comments

Comments

@symonjim
Copy link

symonjim commented Aug 16, 2022 •
edited by amotl
Loading

Followed steps to install mqttwarn on my W10 machine. Everything seemed to work including mqttwarn make-xxx commands. Finally tried running mqttwarn by itself and got strange error messages. Any help?

pip install --upgrade mqttwarn
mqttwarn make-config > mqttwarn.ini
cat mqttwarn.ini
mqttwarn make-samplefuncs > samplefuncs.py
mqttwarn
Traceback (most recent call last):
  File "D:\Program Files (D)\Python37\Scripts\mqttwarn-script.py", line 33, in <module>
    sys.exit(load_entry_point('mqttwarn==0.28.1', 'console_scripts', 'mqttwarn')())
  File "d:\program files (d)\python37\lib\site-packages\mqttwarn\commands.py", line 93, in run
    run_mqttwarn()
  File "d:\program files (d)\python37\lib\site-packages\mqttwarn\commands.py", line 132, in run_mqttwarn
    config = load_configuration(name=scriptname)
  File "d:\program files (d)\python37\lib\site-packages\mqttwarn\configuration.py", line 196, in load_configuration
    return Config(configfile, defaults=defaults)
  File "d:\program files (d)\python37\lib\site-packages\mqttwarn\configuration.py", line 38, in __init__
    self.read_file(f)
  File "d:\program files (d)\python37\lib\configparser.py", line 717, in read_file
    self._read(f, source)
  File "d:\program files (d)\python37\lib\configparser.py", line 1014, in _read
    for lineno, line in enumerate(fp, start=1):
  File "d:\program files (d)\python37\lib\codecs.py", line 714, in __next__
    return next(self.reader)
  File "d:\program files (d)\python37\lib\codecs.py", line 645, in __next__
    line = self.readline()
  File "d:\program files (d)\python37\lib\codecs.py", line 558, in readline
    data = self.read(readsize, firstline=True)
  File "d:\program files (d)\python37\lib\codecs.py", line 504, in read
    newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
@amotl
Copy link
Member

amotl commented Aug 17, 2022 •
edited
Loading

Dear @symonjim,

thank you for writing in. It looks like that this is a specific issue when mqttwarn tries to open the generated configuration file mqttwarn.ini. Do you have any clues why chr(255) could be the first character in the generated file on Windows 10?

For the sake of getting the whole picture, may I ask if you are using the cmd.exe shell, or PowerShell?

With kind regards,
Andreas.

@jpmens
Copy link
Collaborator

jpmens commented Aug 17, 2022

That sounds a bit like a (broken) BOM. Which text editor have you been using, @symonjim, for editing the mqttwarn.ini file?

@amotl
Copy link
Member

amotl commented Aug 17, 2022 •
edited
Loading

Do you have any clues why chr(255) could be the first character in the generated file on Windows?

I see, pylint-dev/pylint#1663 has many insights into the problem and also offers a solution.

Windows 10 produces a file with UTF-16 encoding when using the default redirection operator, which causes this problem when trying to read it back with assuming UTF-8.

When using PowerShell, you might try this command to generate the default configuration file:

mqttwarn make-config | Out-File -Encoding utf8 mqttwarn.ini

Another proposed solution in the discussion is to invoke chcp 65001 before running mqttwarn make-config > mqttwarn.ini, using either cmd.exe, or PowerShell.

It would be sweet if you could evaluate both proposed solutions and report back to us, so that we can improve the mqttwarn documentation correspondingly.

@amotl amotl changed the title invalid start byte? Windows10 - UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte Aug 17, 2022
@symonjim
Copy link
Author

symonjim commented Aug 17, 2022 via email

@symonjim
Copy link
Author

symonjim commented Aug 17, 2022 via email •
edited by amotl
Loading

@symonjim
Copy link
Author

symonjim commented Aug 17, 2022 •
edited by amotl
Loading

Oops. That long file inclusion was not a good idea.

I used notepad to rewrite the ini file making sure it was output utf8. Ran mqttwarn and got the following new error messages.

PS D:\Program Files (D)\mqttwarn-main\mqttwarn-main> mqttwarn
Traceback (most recent call last):
  File "D:\Program Files (D)\Python37\Scripts\mqttwarn-script.py", line 33, in <module>
    sys.exit(load_entry_point('mqttwarn==0.28.1', 'console_scripts', 'mqttwarn')())
  File "d:\program files (d)\python37\lib\site-packages\mqttwarn\commands.py", line 93, in run
    run_mqttwarn()
  File "d:\program files (d)\python37\lib\site-packages\mqttwarn\commands.py", line 132, in run_mqttwarn
    config = load_configuration(name=scriptname)
  File "d:\program files (d)\python37\lib\site-packages\mqttwarn\configuration.py", line 196, in load_configuration
    return Config(configfile, defaults=defaults)
  File "d:\program files (d)\python37\lib\site-packages\mqttwarn\configuration.py", line 106, in __init__
    self.functions = load_functions(functions_file)
  File "d:\program files (d)\python37\lib\site-packages\mqttwarn\util.py", line 219, in load_functions
    py_mod = imp.load_source(mod_name, filepath)
  File "d:\program files (d)\python37\lib\imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 724, in exec_module
  File "<frozen importlib._bootstrap_external>", line 860, in get_code
  File "<frozen importlib._bootstrap_external>", line 791, in source_to_code
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ValueError: source code string cannot contain null bytes
PS D:\Program Files (D)\mqttwarn-main\mqttwarn-main>

@amotl
Copy link
Member

amotl commented Aug 17, 2022 via email •
edited
Loading

@symonjim
Copy link
Author

symonjim commented Aug 17, 2022 •
edited by amotl
Loading

I did that with the .py files. I opened them in notepad and immediately did a "save as" choosing utf8 and overwriting the original. Interestingly all the files I had worked with had said they were going to save as utf8 until I got to samplefuncs.py which said it would save as utf16. I changed that to utf8 and...presto! It works!

PS D:\Program Files (D)\mqttwarn-main\mqttwarn-main> mqttwarn
2022-08-17 18:28:26,596 INFO     [mqttwarn.commands         ] Starting mqttwarn
2022-08-17 18:28:26,597 INFO     [mqttwarn.commands         ] Log level is DEBUG
2022-08-17 18:28:26,597 DEBUG    [mqttwarn.core             ] Trying to load built-in service "file" from "file"
2022-08-17 18:28:26,598 DEBUG    [mqttwarn.core             ] Trying to load service "file" from file "d:\program files (d)\python37\lib\site-packages\mqttwarn\services\file.py"
2022-08-17 18:28:26,599 INFO     [mqttwarn.core             ] Successfully loaded service "file"
2022-08-17 18:28:26,599 DEBUG    [mqttwarn.core             ] Trying to load built-in service "log" from "log"
2022-08-17 18:28:26,600 DEBUG    [mqttwarn.core             ] Trying to load service "log" from file "d:\program files (d)\python37\lib\site-packages\mqttwarn\services\log.py"
2022-08-17 18:28:26,601 INFO     [mqttwarn.core             ] Successfully loaded service "log"
2022-08-17 18:28:26,601 DEBUG    [mqttwarn.core             ] Attempting connection to MQTT broker localhost:1883...
2022-08-17 18:28:26,601 DEBUG    [mqttwarn.core             ] Setting LWT to clients/mqttwarn...
2022-08-17 18:28:28,672 INFO     [mqttwarn.core             ] Publishing status information to mqttwarn/$SYS
2022-08-17 18:28:28,673 INFO     [mqttwarn.core             ] Starting 1 worker threads
2022-08-17 18:28:28,678 DEBUG    [mqttwarn.core             ] Job queue has 0 items to process
2022-08-17 18:28:28,679 DEBUG    [mqttwarn.core             ] Connected to MQTT broker, subscribing to topics...
2022-08-17 18:28:28,680 DEBUG    [mqttwarn.core             ] Cleansession==False; previous subscriptions for clientid mqttwarn remain active on broker
2022-08-17 18:28:28,682 DEBUG    [mqttwarn.core             ] Subscribing to hello/1 (qos=0)
2022-08-17 18:28:28,682 DEBUG    [mqttwarn.core             ] Subscribing to owntracks/+/+ (qos=0)
2022-08-17 18:28:28,683 DEBUG    [mqttwarn.core             ] Subscribing to test/robustness-1 (qos=0)
2022-08-17 18:28:28,683 DEBUG    [mqttwarn.core             ] Subscribing to test/topic-targets-dynamic (qos=0)
2022-08-17 18:28:28,684 DEBUG    [mqttwarn.core             ] Subscribing to test/topic-targets-func (qos=0)

and mosquitto seemed to react. Now I just need to make some adjustments for my setup and tell it what to do when it gets a message.

Thanks for all your help.

Jim

@symonjim
Copy link
Author

symonjim commented Aug 18, 2022 •
edited by amotl
Loading

It worked fine until I added my smtp config and topic to the mqttwarn.ini file. Then it gave me another esoteric error message about int having no length??

I added:

[config:smtp]
server   = 'localhost:25'
sender   = "mqttwarn <symonjim@localhost>"
starttls = False
targets = {
    'symonjim'  : ['symonjim@sybr.net']
    }

and:

[IOTgadgettopic]
topic   = IOTgadgettopic
targets = smtp:symonjim

and got this when it failed:

2022-08-17 22:04:55,000 DEBUG    [mqttwarn.core             ] Successfully loaded service "smtp"
2022-08-17 22:04:55,761 DEBUG    [mqttwarn.core             ] Attempting connection to MQTT broker localhost:1883...
2022-08-17 22:04:55,762 DEBUG    [mqttwarn.core             ] Setting LWT to clients/mqttwarn...
2022-08-17 22:04:57,833 ERROR    [mqttwarn.core             ] Cannot connect to MQTT broker at localhost:1883: object of type 'int' has no len()

hmm...

@amotl
Copy link
Member

amotl commented Aug 18, 2022

Hi Jim,

got this when it failed:
Cannot connect to MQTT broker at localhost:1883: object of type 'int' has no len()

That's weird, but I am confident that we will also find out about the root cause. The corresponding place in the code where this is happening is:

https://github.com/jpmens/mqttwarn/blob/f4e6e0888a2345bb3b0bf7a490081b1e48185609/mqttwarn/core.py#L629-L634

I am assuming that again something might be going wrong with correctly parsing and interpreting the configuration file on your end. I can confirm that, when inserting the snippets you shared into the mqttwarn.ini on my machine, I don't observe such an error.

Currently, I have no idea what exactly might be going wrong on your end. Two things come to mind how to proceed:

a) Maybe the culprit is different newline characters on Windows after saving the configuration file? It is hard to believe, because you already edited and saved it manually beforehand, right?

b) Can you verify everything works flawlessly again, when you remove the snippets you added?

With kind regards,
Andreas.

@amotl amotl mentioned this issue Aug 18, 2022
Merged
@symonjim
Copy link
Author

I found it. Had nothing to do with the smtp stuff I had inserted. I had also put in these lines without single quotes. Single quotes fixed it.

Now I just need to figure out how to specify qos=1.

Fails with that odd error message:

username = mqttwarn
password = 123456789

Works fine:

username = 'mqttwarn'
password = '123456789'

@symonjim
Copy link
Author

symonjim commented Aug 18, 2022 •
edited by amotl
Loading

Go that. Now this. I guess I need to just plug away at it and not keep clogging up this site with each of my steps.

2022-08-18 10:45:37,047 DEBUG    [mqttwarn.core             ] Message received on IOTgadgettopic:   "This is MQTTX1 with gadget topic with another try"
2022-08-18 10:45:37,048 DEBUG    [mqttwarn.core             ] Section [IOTgadgettopic] matches message on IOTgadgettopic. Processing...
2022-08-18 10:45:37,051 DEBUG    [mqttwarn.core             ] Cannot decode JSON object, payload=  "This is MQTTX1 with gadget topic with another try": dictionary update sequence element #0 has length 1; 2 is required
2022-08-18 10:45:37,053 DEBUG    [mqttwarn.core             ] Message on IOTgadgettopic going to smtp:mqttwarn
2022-08-18 10:45:37,053 DEBUG    [mqttwarn.core             ] New `smtp:mqttwarn' job: IOTgadgettopic
2022-08-18 10:45:37,054 DEBUG    [mqttwarn.core             ] Processor #0 is handling: `smtp' for mqttwarn
2022-08-18 10:45:37,054 INFO     [mqttwarn.core             ] Invoking service plugin for `smtp'
2022-08-18 10:45:37,056 DEBUG    [mqttwarn.services.smtp    ] *** MODULE=d:\program files (d)\python37\lib\site-packages\mqttwarn\services\smtp.py: service=smtp, target=mqttwarn
2022-08-18 10:45:37,056 ERROR    [mqttwarn.core             ] Cannot invoke service for `smtp'
Traceback (most recent call last):
  File "d:\program files (d)\python37\lib\site-packages\mqttwarn\core.py", line 517, in processor
    notified = timeout(module.plugin, (srv, st))
  File "d:\program files (d)\python37\lib\site-packages\mqttwarn\util.py", line 133, in timeout
    raise it.exception
  File "d:\program files (d)\python37\lib\site-packages\mqttwarn\util.py", line 121, in run
    self.result = func(*args, **kwargs)
  File "d:\program files (d)\python37\lib\site-packages\mqttwarn\services\smtp.py", line 24, in plugin
    username    = item.config['username']
KeyError: 'username'
2022-08-18 10:45:37,063 WARNING  [mqttwarn.core             ] Notification of smtp for `IOTgadgettopic' FAILED or TIMED OUT
2022-08-18 10:45:37,066 DEBUG    [mqttwarn.core             ] Job queue has 0 items to process

@amotl amotl mentioned this issue Aug 18, 2022
Closed
@amotl
Copy link
Member

amotl commented Aug 18, 2022

Hi again,

I am glad that the issues with the configuration files seem to be resolved now.

not keep clogging up this site with each of my steps

Don't worry, please keep reporting. Every detail counts to improve the software and/or its documentation in one way or another. On the very last detail you reported, I've just created #573.

Cheers,
Andreas.

@amotl amotl mentioned this issue Aug 21, 2022
Merged
@amotl
Copy link
Member

amotl commented Aug 21, 2022

Hi again,

I've just investigated the situation on my workstation using Racker, and wanted to report back about the outcome on Windows10, mainly as a reference for others running into the same problem. #576 also tries to reproduce the problem on CI. In order to finally resolve this issue, we should improve the documentation correspondingly, to make Windows users aware of it.

With kind regards,
Andreas.

Introduction

Per pylint-dev/pylint#1663, we discovered that:

When using the default redirection operator > on Windows 10, it produces a file with UTF-16 encoding. This obviously causes problems when trying to read it back with assuming the UTF-8 character encoding.

Out-File and > / >> create "Unicode" - UTF-16LE-files by default

Apparently, the problem only happens with PowerShell, it does not occur with Bash nor does it happen with cmd.exe.

Non-solutions

  • Using the chcp command for changing the Code page of the active console did not work.
  • Using [Console]::OutputEncoding = [System.Text.Encoding]::GetEncoding(65001) did not work.

Solution I

Use the Out-File cmdlet.

mqttwarn make-config | Out-File -Encoding utf8 mqttwarn.ini

Solution II

Adjust the settings in $PSDefaultParameterValues. This will be active for all subsequent invocations.

$PSDefaultParameterValues['Out-File:Encoding'] = 'utf8'
mqttwarn make-config > mqttwarn.ini

Reset this setting:

$PSDefaultParameterValues.Remove('Out-File:Encoding')

You can also adjust the setting for all cmdlets, not only Out-File:

$PSDefaultParameterValues['*:Encoding'] = 'utf8'

In order to make those settings persistent, you may want to add them to your $PROFILE (current user only) or $PROFILE.AllUsersCurrentHost (all users) file.

References

@amotl
Copy link
Member

amotl commented Aug 21, 2022

7b42d6c improves the README correspondingly, so let me close this issue. Thanks again for making us aware of it, Jim.

@amotl amotl closed this as completed Aug 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jpmens @amotl @symonjim