Fix Ukraine production parser #5770
Conversation
parsers/UA.py
Outdated
| conn = http.client.HTTPSConnection("ua.energy") | ||
| payload = f"action=get_data_oes&report_date={target_date}&type=day" | ||
| headers = { | ||
| "Content-Type": "application/x-www-form-urlencoded", | ||
| "User-Agent": "PostmanRuntime/7.32.3", | ||
| } | ||
| conn.request("POST", "/wp-admin/admin-ajax.php", payload, headers) | ||
| res = conn.getresponse() | ||
| response = json.loads(res.read().decode("utf-8")) |
There was a problem hiding this comment.
Been trying to wrap my head around why this works and request don't but without success so this is fine but maybe add a comment or two as to why we are using it instead of requests?
There was a problem hiding this comment.
I feel like this should work with requests, does it really not?
There was a problem hiding this comment.
I replicated the code and changed the headers to what is used here in requests but I'm just getting 403 http codes.
And honestly I can't explain why right now.
If I have the time someday I might dig into it again because as far as I know it should work.
There was a problem hiding this comment.
Same here. Interesting thought, requests work in small PoC script!
from requests import Session
url = "https://ua.energy/wp-admin/admin-ajax.php"
postdata = {"action": "get_data_oes", "report_date": "20.07.2023", "type": "day"}
response = Session().post(
url, postdata, headers={"User-Agent": "PostmanRuntime/7.32.3"}
)
print(response.json())
But when I copy exact same lines to parser, poetry run fails because of 403.
I have no idea how and why it works like this, so the maximum that I can do is add a comment saying: #TODO: someone, please look at this and explain why it works like this.
There was a problem hiding this comment.
There are something going on that's for sure.
But I'd suggest putting a comment like this:
# We are using HTTP.client because Request returns 403 http codes.
# TODO: Look into why requests are returning 403 http codes while HTTP.client works.There was a problem hiding this comment.
Actually I got an HTTP 403 Forbidden when running that proof-of-concept script in my computer for the first time (and also when I retry it). I'm located in Canada with a normal Canadian IP address, Python 3.6.9 and requests==2.27.1.
The error message says that it's from Cloudflare, so maybe they have paranoia settings tuned high in Cloudflare, I wouldn't be surprised.
Error message includes:
Why have I been blocked?
This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.
Which could be anything.
Going to https://ua.energy/ in a browser also gives me that error message.
I would guess that ultimately it's not a question of using requests or http.client, and that you got it working using http.client or the PoC script is a fluke.
There was a problem hiding this comment.
I think that maybe this service just blocks all connections not from Europe AND connection from all over the world that uses most popular library in python to make requests, requests. Strange thing to do, why would you create an API and block developers from accessing it automatically, but ok, we'll work with what they give us. I'm located in Poland and while working on that was using Ukrainian VPN. There is no difference in behavior, same problems with requests and same successes with http.client.
There was a problem hiding this comment.
Yeah I can't explain it either, but ua.energy is also available to me from Sweden.
Seems very strange.
parsers/UA.py
Outdated
| @@ -75,10 +79,21 @@ def fetch_production( | |||
| row["production"][v] = 0.0 | |||
There was a problem hiding this comment.
While not related to your changes we have a standardized way to do this now with the new EventList classes.
You would need to import and use the following to change it:
from electricitymap.contrib.lib.models.event_lists import (
ProductionBreakdownList,
)
from electricitymap.contrib.lib.models.events import (
ProductionMix,
StorageMix,
)
This is more of a optional feature, but I'd like to change it in here to avoid another PR.
If you don't want to do it would you have any objections to me changing it in your PR personally?
There was a problem hiding this comment.
If you can, please, go ahead and make changes in my PR. I tried to compare it to other parses to get a grasp of how it should be used, but I'm not sure, so, I think it will be better if you do it.
|
I pushed my changes, let me know if there is any questions about them. After you have taken a look I'll approve and merge it. |
Issue
Issue #3981. Not sure if it should be closed, still have to look into ENTSO-E parser for consumption data.
Description
Fixes the production parser for UA. Have to mock User-Agent for parser and switch to use
http.clientinstead ofrequests, becauserequestsadds something to header that makes Ukrainian server recognize parser as a bot.Also, not sure why this parser was not able to parse past dates, seems pretty straightforward to me, so fixed that too.
Double check
poetry run test_parser "zone_key"poetry run test_parser "zone_key" --target_datetime "date"pnpx prettier --write .(with no result since no changes to front-end were done) andpoetry run formatto format my changes.