-
Notifications
You must be signed in to change notification settings - Fork 187
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
query_crossborder_flows
fails in August 2021 for DE
->NL
#144
Comments
Here is the full error message btw: ---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
/var/folders/d9/6_lw279s459fpbps9x1mxd6m0000gn/T/ipykernel_18198/2060622407.py in <module>
1 start = pd.Timestamp(str(int(2021_08_01)), tz='Europe/Amsterdam')
2 end = pd.Timestamp(str(int(2021_09_01)), tz='Europe/Amsterdam')
----> 3 client.query_crossborder_flows("DE", "NL", start=start, end=end)
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/entsoe/decorators.py in year_wrapper(start, end, *args, **kwargs)
62 for _start, _end in blocks:
63 try:
---> 64 frame = func(*args, start=_start, end=_end, **kwargs)
65 except NoMatchingDataError:
66 logging.debug(f"NoMatchingDataError: between {_start} and {_end}")
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/entsoe/entsoe.py in query_crossborder_flows(self, country_code_from, country_code_to, start, end, **kwargs)
1180 start=start,
1181 end=end)
-> 1182 ts = parse_crossborder_flows(text)
1183 ts = ts.tz_convert(area_from.tz)
1184 ts = ts.truncate(before=start, after=end)
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/entsoe/parsers.py in parse_crossborder_flows(xml_text)
229 series = pd.Series(dtype = 'object')
230 for soup in _extract_timeseries(xml_text):
--> 231 series = series.append(_parse_crossborder_flows_timeseries(soup))
232 series = series.sort_index()
233 return series
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/entsoe/parsers.py in _parse_crossborder_flows_timeseries(soup)
659 series = pd.Series(index=positions, data=flows)
660 series = series.sort_index()
--> 661 series.index = _parse_datetimeindex(soup)
662
663 return series
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/core/generic.py in __setattr__(self, name, value)
5498 try:
5499 object.__getattribute__(self, name)
-> 5500 return object.__setattr__(self, name, value)
5501 except AttributeError:
5502 pass
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/_libs/properties.pyx in pandas._libs.properties.AxisProperty.__set__()
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/core/series.py in _set_axis(self, axis, labels, fastpath)
557 if not fastpath:
558 # The ensure_index call above ensures we have an Index object
--> 559 self._mgr.set_axis(axis, labels)
560
561 # ndarray compatibility
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/core/internals/managers.py in set_axis(self, axis, new_labels)
214 def set_axis(self, axis: int, new_labels: Index) -> None:
215 # Caller is responsible for ensuring we have an Index object.
--> 216 self._validate_set_axis(axis, new_labels)
217 self.axes[axis] = new_labels
218
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pandas/core/internals/base.py in _validate_set_axis(self, axis, new_labels)
55
56 elif new_len != old_len:
---> 57 raise ValueError(
58 f"Length mismatch: Expected axis has {old_len} elements, new "
59 f"values have {new_len} elements"
ValueError: Length mismatch: Expected axis has 22 elements, new values have 85 elements |
Hi thanks for your elaborate report! I will look into it when I get home/have time. Is this possibly related to #133?
Also you're welcome thanks for the kind words!
Sent from ProtonMail mobile
…-------- Original Message --------
On 23 Nov 2021, 11:16, Charel Felten wrote:
Here is the full error message btw:
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
ValueError
Traceback
(
most
recent
call
last
)
/
var
/
folders
/
d9
/
6_l
w279s459fpbps9x1mxd6m0000gn
/
T
/
ipykernel_18198
/
2060622407.
py
in
<
module
>
1
start
=
pd
.
Timestamp
(
str
(
int
(
2021_08_01
)),
tz
=
'Europe/Amsterdam'
)
2
end
=
pd
.
Timestamp
(
str
(
int
(
2021_09_01
)),
tz
=
'Europe/Amsterdam'
)
-
-
-
-
>
3
client
.
query_crossborder_flows
(
"DE"
,
"NL"
,
start
=
start
,
end
=
end
)
/
Library
/
Frameworks
/
Python
.
framework
/
Versions
/
3.9
/
lib
/
python3
.
9
/
site
-
packages
/
entsoe
/
decorators
.
py
in
year_wrapper
(
start
,
end
,
*
args
,
**
kwargs
)
62
for
_start
,
_end
in
blocks
:
63
try
:
-
-
-
>
64
frame
=
func
(
*
args
,
start
=
_start
,
end
=
_end
,
**
kwargs
)
65
except
NoMatchingDataError
:
66
logging
.
debug
(
f"NoMatchingDataError: between
{
_start
}
and
{
_end
}
"
)
/
Library
/
Frameworks
/
Python
.
framework
/
Versions
/
3.9
/
lib
/
python3
.
9
/
site
-
packages
/
entsoe
/
entsoe
.
py
in
query_crossborder_flows
(
self
,
country_code_from
,
country_code_to
,
start
,
end
,
**
kwargs
)
1180
start
=
start
,
1181
end
=
end
)
->
1182
ts
=
parse_crossborder_flows
(
text
)
1183
ts
=
ts
.
tz_convert
(
area_from
.
tz
)
1184
ts
=
ts
.
truncate
(
before
=
start
,
after
=
end
)
/
Library
/
Frameworks
/
Python
.
framework
/
Versions
/
3.9
/
lib
/
python3
.
9
/
site
-
packages
/
entsoe
/
parsers
.
py
in
parse_crossborder_flows
(
xml_text
)
229
series
=
pd
.
Series
(
dtype
=
'object'
)
230
for
soup
in
_extract_timeseries
(
xml_text
):
-
-
>
231
series
=
series
.
append
(
_parse_crossborder_flows_timeseries
(
soup
))
232
series
=
series
.
sort_index
()
233
return
series
/
Library
/
Frameworks
/
Python
.
framework
/
Versions
/
3.9
/
lib
/
python3
.
9
/
site
-
packages
/
entsoe
/
parsers
.
py
in
_parse_crossborder_flows_timeseries
(
soup
)
659
series
=
pd
.
Series
(
index
=
positions
,
data
=
flows
)
660
series
=
series
.
sort_index
()
-
-
>
661
series
.
index
=
_parse_datetimeindex
(
soup
)
662
663
return
series
/
Library
/
Frameworks
/
Python
.
framework
/
Versions
/
3.9
/
lib
/
python3
.
9
/
site
-
packages
/
pandas
/
core
/
generic
.
py
in
__setattr__
(
self
,
name
,
value
)
5498
try
:
5499
object
.
__getattribute__
(
self
,
name
)
->
5500
return
object
.
__setattr__
(
self
,
name
,
value
)
5501
except
AttributeError
:
5502
pass
/
Library
/
Frameworks
/
Python
.
framework
/
Versions
/
3.9
/
lib
/
python3
.
9
/
site
-
packages
/
pandas
/
_libs
/
properties
.
pyx
in
pandas
.
_libs
.
properties
.
AxisProperty
.
__set__
()
/
Library
/
Frameworks
/
Python
.
framework
/
Versions
/
3.9
/
lib
/
python3
.
9
/
site
-
packages
/
pandas
/
core
/
series
.
py
in
_set_axis
(
self
,
axis
,
labels
,
fastpath
)
557
if
not
fastpath
:
558
# The ensure_index call above ensures we have an Index object
-
-
>
559
self
.
_mgr
.
set_axis
(
axis
,
labels
)
560
561
# ndarray compatibility
/
Library
/
Frameworks
/
Python
.
framework
/
Versions
/
3.9
/
lib
/
python3
.
9
/
site
-
packages
/
pandas
/
core
/
internals
/
managers
.
py
in
set_axis
(
self
,
axis
,
new_labels
)
214
def
set_axis
(
self
,
axis
:
int
,
new_labels
:
Index
)
-
>
None
:
215
# Caller is responsible for ensuring we have an Index object.
-
-
>
216
self
.
_validate_set_axis
(
axis
,
new_labels
)
217
self
.
axes
[
axis
]
=
new_labels
218
/
Library
/
Frameworks
/
Python
.
framework
/
Versions
/
3.9
/
lib
/
python3
.
9
/
site
-
packages
/
pandas
/
core
/
internals
/
base
.
py
in
_validate_set_axis
(
self
,
axis
,
new_labels
)
55
56
elif
new_len
!=
old_len
:
-
-
-
>
57
raise
ValueError
(
58
f"Length mismatch: Expected axis has
{
old_len
}
elements, new "
59
f"values have
{
new_len
}
elements"
ValueError
:
Length
mismatch
:
Expected
axis
has
22
elements
,
new
values
have
85
elements
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, [view it on GitHub](#144 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/ABEC63WKXBCLX2HFTKTY4QDUNNSYRANCNFSM5ITDSPTQ).
Triage notifications on the go with GitHub Mobile for [iOS](https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675) or [Android](https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub).
|
query_crossborder_flows
fails in August 2021query_crossborder_flows
fails in August 2021 for DE
->NL
I'm not sure its related to #137 given that user seems to have incorrect data whereas in my case it just fails. Altough interestingly start = pd.Timestamp(str(int(2021_08_01)), tz='Europe/Amsterdam')
end = pd.Timestamp(str(int(2021_09_01)), tz='Europe/Amsterdam')
client.query_crossborder_flows("CZ", "AT", start=start, end=end) returns
which is indeed a |
Yes it is true that 15 min trading was introduced this year for that border. It is very possible that that breaks some assumption of the code.
Sent from ProtonMail mobile
…-------- Original Message --------
On 23 Nov 2021, 11:28, Charel Felten wrote:
I'm not sure its related to [#137](#137) given that user seems to have incorrect data whereas in my case it just fails.
Altough interestingly
start
=
pd
.
Timestamp
(
str
(
int
(
2021_08_01
)),
tz
=
'Europe/Amsterdam'
)
end
=
pd
.
Timestamp
(
str
(
int
(
2021_09_01
)),
tz
=
'Europe/Amsterdam'
)
client
.
query_crossborder_flows
(
"CZ"
,
"AT"
,
start
=
start
,
end
=
end
)
returns
2021-08-01 00:00:00+02:00 1103.0
2021-08-01 01:00:00+02:00 1121.0
2021-08-01 02:00:00+02:00 1062.0
2021-08-01 03:00:00+02:00 1037.0
2021-08-01 04:00:00+02:00 945.0
...
2021-08-31 19:00:00+02:00 899.0
2021-08-31 20:00:00+02:00 821.0
2021-08-31 21:00:00+02:00 766.0
2021-08-31 22:00:00+02:00 1054.0
2021-08-31 23:00:00+02:00 969.0
Freq: 60T, Length: 744, dtype: float64
which is indeed a DataFrame, which makes me think that the issue I have is potentially only present from DE->NL queries. I added this to the title of the issue.
—
You are receiving this because you commented.
Reply to this email directly, [view it on GitHub](#144 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/ABEC63UEXBNU75FBTAACNT3UNNUDXANCNFSM5ITDSPTQ).
Triage notifications on the go with GitHub Mobile for [iOS](https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675) or [Android](https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub).
|
It gets a bit weirder: I narrowed the problem down to the 1st of August 2021. Because from August 2nd onwards, we get the correct result (15min interval) start = pd.Timestamp(year=2021, month=8, day=2, tz='Europe/Amsterdam')
end = pd.Timestamp(year=2021, month=9, day=1, tz='Europe/Amsterdam')
client.query_crossborder_flows("DE", "NL", start=start, end=end) gives the correct looking df:
The problem seems to be only for 1st of august, for example: start = pd.Timestamp(year=2021, month=8, day=1, hour=10, tz='Europe/Amsterdam')
end = pd.Timestamp(year=2021, month=8, day=1, hour=15, tz='Europe/Amsterdam')
client.query_crossborder_flows("DE", "NL", start=start, end=end) gives again: ValueError: Length mismatch: Expected axis has 5 elements, new values have 17 elements However what is really interesting:Querying every single hour of august 1st individually does work: for h1, h2 in zip(range(23), range(1,24)):
start = pd.Timestamp(year=2021, month=8, day=1, hour=h1, tz='Europe/Amsterdam')
end = pd.Timestamp(year=2021, month=8, day=1, hour=h2, tz='Europe/Amsterdam')
print(client.query_crossborder_flows("DE", "NL", start=start, end=end)) gives
To me it looks like the frequency is changed from |
hi @charelF thank you for your thorough investigation. That seems like a very good reasoning of why it goes wrong. The question is what would be the best fix. This is a one time occurence so maybe just put in a static fix would be fine. But it is possible that this will occur on more borders. I am not sure yet what is the best solution. Do you have a good idea? |
Unfortunately I am both new to this library and also new to this domain, so I do not feel qualified to give an answer on how to best fix this in the library (particularly since the call stack is still somewhat obscure to me, in particular the That being said, I did more investigations: July 31st 2021We have normal 1 hour intervals August 2nd 2021Normal 15 min intervals August 1st 2021Oh no... so it looks like a mix between 1 hour intervals and 15 min intervals --> This is probably where the error comes from, so I assume its an issue from entsoe I checked, an this issue is not present for any of the other borders of With these insights, I would say this is an error from entsoe, so probably a manual fix would apply here, since one can assume this should not happen again... |
Thanks, I am happy to pay some of the time this library saved me back! |
hi @charelF I have poked some people and they are looking into it. ill update the issue when I know more |
hi @MattEwen this seems like a consistent issue then. I will open a support ticket with entsoe. |
hi @MattEwen I have received a reply it is fixed now on dutch borders but the other borders they are sill looking into. |
thanks, thats nice to hear! |
Problem
There seems to be a problem with crossborder flows
query_crossborder_flows
.In particular, querying over DE -> NL crossborder flows over summer 2021, there is a bug in august.
I assume it happens because the frequency in July is hourly data, and in September its quarterly (15min) data. Probably somewhere in august it switches from hourly to quarterly, which in turn messes up the dataframe and causes the error
Information
Reproduction
July 2021: Correct
returns the expected output:
September 2021: Correct
Similarly,
returns the expected output:
August 2021: Problem
However this one fails:
with
Lastly, I just wanted to thank all contributors of this library, it has saved me enormous amount of time and headaches I had working with the entsoe XML API. Thanks!
The text was updated successfully, but these errors were encountered: