Last hotfix broke luna for me: ParseError

[kbr-]

Hi,

0.6.1 alpha works well, 0.6.2 (for which you forgot to change the version in addon.xml btw.) doesn't.
Here's what happens:

13:33:29 962.675781 T:1751024624 NOTICE: [script.luna] [CoreService] - initialized
13:33:30 963.831177 T:1751024624 ERROR: /home/osmc/.kodi/addons/script.module.requests/lib/requests/packages/urllib3/connectionpool.py:791: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification
is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
InsecureRequestWarning)
13:33:30 963.855835 T:1751024624 NOTICE: [script.luna] Trying to re-encode received XML as UTF-16
13:33:30 963.874756 T:1751024624 ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
- NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
Error Type: <class 'xml.etree.ElementTree.ParseError'>
Error Contents: not well-formed (invalid token): line 1, column 1
Traceback (most recent call last):
File "/home/osmc/.kodi/addons/script.luna/addon.py", line 203, in
game_controller.get_games()
File "/home/osmc/.kodi/addons/script.luna/resources/lib/controller/gamecontroller.py", line 18, in get_games
game_list = self.moonlight_helper.list_games()
File "/home/osmc/.kodi/addons/script.luna/resources/lib/util/moonlighthelper.py", line 135, in list_games
return RequiredFeature('nvhttp').request().get_app_list()
File "/home/osmc/.kodi/addons/script.luna/resources/lib/nvhttp/nvhttp.py", line 146, in get_app_list
applist = self.get_app_list_from_string(response.content)
File "/home/osmc/.kodi/addons/script.luna/resources/lib/nvhttp/nvhttp.py", line 151, in get_app_list_from_string
applist_root = ET.ElementTree(ET.fromstring(self.re_encode_string(xml_string))).getroot()
File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1300, in XML
parser.feed(text)
File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1642, in feed
self._raiseerror(v)
File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1506, in _raiseerror
raise err
ParseError: not well-formed (invalid token): line 1, column 1
-->End of Python script error report<--
13:33:30 964.082031 T:1956852656 ERROR: GetDirectory - Error getting plugin://script.luna/
13:33:30 964.082336 T:1956852656 ERROR: CGUIMediaWindow::GetDirectory(plugin://script.luna/) failed

[wackerl91]

Heya! Thanks for letting me know (also for the fact that I didn't change the versions). I'll have a look at this as soon as I can and will let you know if I should require more information.
I hope sticking with 0.6.1. is not too big of an inconvenience for you.

[kbr-]

Not an inconvenience at all. I am only grateful for the great addon.

[Langoor2]

@wackerl91 thats not an problem at all, im glad the older version is working! good luck with your exames!
im just really glad that this addon exsists ^^

[wackerl91]

@kbr- @Langoor2 @Spitwod

I've had a look at this and while I can't reproduce the error on my machines (Note: I can't update GFE on my notebook past 2.10.x as my notebook's GPU is not supported and I had to patch the exe to get it running) I might have a somewhat 'hacky' solution for now by catching the error you guys were seeing and re-trying the old behaviour present in 0.6.1.

However before releasing it I'd like to make sure that this is indeed working. The only file which has changed is nvhttp and for testing purposes you should be able to just replace it:

cd ~/.kodi/addons/script.luna/resources/lib/nvhttp
mv nvhttp.py nvhttp.bk
// (if present, remove nvhttp.pyc and nvhttp.pyo:
rm nvhttp.py*
// )
nano nvhttp.py
// -> copy the contents of above linked file into the editor (might lag a bit while it's adding lines), once it's done press CTRL + X -> Y -> Enter
// Permissions should already be set properly, but just to make sure:
chmod 644 nvhttp.py

If Luna was running for some reason close it completely (go back to Kodi's 'programs' view), start it again and try whatever you were doing when it failed during 0.6.2.
Note: from what I could gather by your logs this error happened when Luna tried to build the game list, so you probably should delete the entire game cache (that's an option in the settings) and have Luna re-build it (happens automatically once you enter the 'games' view).
Should it still fail for whatever reason it will dump the entire XML string that led to this error to the logs (slightly above the 'ValueError: ETree is not set.' exception that should be present in this case) so I can debug this further if necessary.

Please let me know how it goes or if you have additional questions :)

Specifically @Langoor2: You mentioned that you're unable to set a controller map in your original issue, but didn't provide logs for this particular problem. If you're able to get your game list (i.e. the above change did indeed fix the error present in the logs) but you're still unable to set a controller map I'd kindly ask you to open a new issue so we can investigate that further. Setting a controller map should not be related to the game list in any way, though to be fair I can't rule that out completely as Luna's current architecture grew on demand and wasn't prepared to handle all that stuff it currently does - read: it's a huge mess, which is why I'm rebuilding it more or less from ground up.

[Kampher]

Hi Wacker,

Thank you very much for this amazing addon. I've been using this for about two weeks as a complete Raspberry Pi newbie, and it's been working great.

I had this exact error, and the fix above completely worked for about five minutes. When I tried to access my games list again, however, I got booted from Luna and couldn't make it back to the addon without getting the "Check log for errors" message again.

Edit: Sorry to edit again. I've tried reinstalling everything from scratch, and I've been able to reproduce the error.

If I install Luna, but make sure that nothing is paired, and then use your fix, I can get into Luna but get the error that it can't find any games (Which makes sense, because I'm not paired to my computer). As soon as I pair, I can't get back into Luna and I just get the "Check the log" error again.

Hope that helps.

[wackerl91]

Heya @Kampher,
in general (to the other guys, too) I'd like to thank you guys very much for all the appreciation you're showing towards Luna, that's great and I really enjoy building something that a lot of people find useful (even though I guess my userbase isn't that large, but I got no statistics about that - Luna isn't talking home :) ). Even more so because you guys are really nice and not demanding that I fix stuff the very instant it was reported! So thanks for your understanding as well.

Back to the ugly stuff, though: I tried following the steps you listed above to the letter and I still can't reproduce this on my end. This is what I did:

1. uninstalled my current dev version
2. reset known hosts on my notebook (the 'Forget Devices' button in GFE)
3. deleted everything leftover from my previous Luna installation (essentially deleted everything in ~/.kodi/userdata/addon_data/script.luna/.storage/ and ~/.cache/moonlight/)
4. patched the nvhttp.py file using the above mentioned changes
5. started Luna, got an error because I had a typo in my host's IP address (request timed out)
6. corrected the IP address and paired to my notebook from outside the addon
7. Luna then started to fetch the game list and parse the information
8. entered the game view to make sure all my games showed up properly, they did

There's two things I'd like to note about this:

• I'm running GFE 2.11.2.66 on my notebook
• My guess is that it's failing at step 7 for you, essentially preventing you from launching Luna (this usually only happens when the storage file was never created - which it isn't if Luna dies in the process)

Unfortunately this means we have to go full circle on this, and believe me: I like that even less than you do ;)
This means that I'll need some more information:

• the GFE version on your gaming machine
• the relevant portion of the logs

In case you don't know (you mentioned being new to all of this) you can just follow the log output inside the terminal window by calling tail -f .kodi/temp/kodi.log. After doing this just make Luna fail again (you seem to have some experience with this now :P (scnr)) and you should see something along the lines of:

10:43:57 2760670.250000 T:1550840816   ERROR: /home/osmc/.kodi/addons/script.module.requests/lib/requests/packages/urllib3/connectionpool.py:791: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
                                              InsecureRequestWarning)
10:43:57 2760670.250000 T:1550840816  NOTICE: [script.luna] Trying to re-encode received XML as UTF-16
10:43:57 2760670.250000 T:1550840816  NOTICE: [script.luna] Re-encode failed, trying to decode as UTF-8
10:43:57 2760670.250000 T:1550840816  NOTICE: [script.luna] Trying to encode as: UTF-16
10:43:57 2760670.250000 T:1550840816  NOTICE: [script.luna] Trying to get information for game: Tom Clancy's The Division

The first error (unsecure request warning) can be ignored, that's just in there because I needed to disable request validation for requests made to the GFE host because the certificate in use would fail this validation (I'd need to write my own validator for this step and will do so at some point in the future, but it's only a minor inconvenience and nothing to worry about right now).

In your case it should(!) fail after the second last line and instead log an error saying "Building ETree from XML failed: ParseError" followed by the XML it tries to parse at that point.
You can then copy the whole shebang and paste it into a pastebin and just leave the permalink in your next comment (btw, no need to login at pastebin, you can just use the options beneath the textarea, e.g. for setting it as unlisted so only people with the link can access it, and hit the 'Create Paste' button - it then redirects you to your newly created paste where you can just copy the link from the URL).
Please make sure to include the XML containing all of your games, as this is most likely why it's failing and I need that information to throw Luna at it until she's able to get her job done.

If I'm not right with above assumption then just send me the entire log from the last action Luna logged (and therefore successfully completed) and the entire error stack trace (please use pastebin for this as well, as the formatting for longer logs in GitHub is a pain to read).

Should you get a parse error but it's not being followed by an XML string however then we need to wait for the other three guys to try the very same while I'll think of some easy to follow instructions so we can get that information.

[Kampher]

Hi @wackerl91,

Ok, so I tried your steps and it still won't work for me. Here's the info you need.

1. It looks like I'm running GFE version 2.11.4.0
2. Here is the error\

I hope this gives you a clue as to what might be going wrong. I tried version 0.6.1 but I'm getting the same results.

Thanks again for a great addon! Like I mentioned earlier, I had no problems with this until I reinstalled a fresh OSMC onto a new SD card and reinstalled everything. Thank you for the support.

[wackerl91]

Heya @Kampher,
thanks for that information, I was able to track down a bug using your log.

This entire encode / decode stuff is such a mess without strict standards and it's obviously giving me a hard time, especially since GFE sometimes specifies UTF-8 as encoding and sometimes UTF-16.

Anyway, I took another shot at it by just trying to decode and then encode the string in every possible format (in this context at least). If the decode fails as ASCII, UTF-8 and UTF-16 it will ultimately fail, log the offending string and raise an exception. Similarly, if I'm not able to encode the string as either UTF-8 or UTF-16 it will again log the string and raise an exception.

If the string passes the decode / encode stage but still can't be interpreted properly it will fail for good and log the offending string.

Again this change works for me (using special characters in at least one game's name). I'd be glad if you could give it a try:

cd ~/.kodi/addons/script.luna/resources/lib/nvhttp/
mv nvhttp.py nvhttp.bk2
wget https://raw.githubusercontent.com/wackerl91/luna/2a9d14beacd40e0640c4f31490ba95db5f074b66/resources/lib/nvhttp/nvhttp.py nvhttp.py

I'm so sorry that this turned out to be such a clusterf..., yet I want to thank you for your patience and help while resolving this issue (hopefully for good this time!) - really means a lot.

P.S.: Please make sure to include some lines above the actual error next time. I'm logging pretty much (though not as much as I'd like to, but this will change in the future) which helps me to put errors in context. It wasn't an issue this time as it's pretty clear where this is happening, just a heads-up for the future :) Most devs love to have as much context as possible.
BTW, if you're not tailing the log (which one usually doesn't) you can also display the last x lines using tail -n [x] [logfile], e.g. for the last 200 lines: tail -n 200 ~/.kodi/temp/kodi.log ;) You can also 'select' only the last 200 lines Luna logged: tail -n 200 ~/.kodi/temp/kodi.log | grep script.luna
Edit: Sorry if you knew this already - as always, I can't guess how much experience different people have and I just want to make sure :)

[Kampher]

That did the trick! Thanks for such a speedy and well thought out reply.

Next time I'll be sure to add some more to the error log. I figured you didn't want the other lines because you mentioned that you knew of the errors mentioned above.

[wackerl91]

Awesome! Admittedly it's kind of a crowbar solution, but I'm glad it's working now. Since I'm not at home right now I'll make sure to publish a bug fix release either tonight or tomorrow.
Thanks again for helping me figure this out and tirelessly retrying the solutions I came up with and providing relevant information. Really appreciate it!

If you should have any other issues or feature requests in the future please let me know :) Cheers!

[wackerl91]

Luna 0.6.3 contains the fix for this. Again, thanks for all your help and understanding. If you encounter any other issues and / or have feature requests (or anything else, really) just let me know. It was a pleasure talking to you guys!