Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create abstract base class for inverters #183

Merged
merged 25 commits into from
Aug 28, 2024

Conversation

mduffin95
Copy link
Contributor

@mduffin95 mduffin95 commented Aug 15, 2024

Pull Request

Description

  • Aims to create an abstraction across different inverter types to simplify dealing with different models. This should also help with testing as we can treat "mock" inverters and real ones in the same way.
  • Aims to centralise pulling configuraiton from environment variables. Again this should help with testing as a settings object can be instantiated without needing env variables if required.

How Has This Been Tested?

Kind of difficult to test as I don't have access to all of these inverter types. However I have managed to get access to a GivEnergy system and I've managed to test this one out.

It would be good to mock out some API calls to various inverters so that they can be tested.

  • Yes

If your changes affect data processing, have you plotted any changes? i.e. have you done a quick sanity check?

  • Yes

Checklist:

  • My code follows OCF's coding style guidelines
  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • I have checked my code and corrected any misspellings

@mduffin95 mduffin95 marked this pull request as ready for review August 15, 2024 22:46
Copy link

codecov bot commented Aug 15, 2024

Codecov Report

Attention: Patch coverage is 54.66667% with 34 lines in your changes missing coverage. Please review.

Project coverage is 58.33%. Comparing base (a9f0c79) to head (6038636).
Report is 97 commits behind head on main.

Files Patch % Lines
quartz_solar_forecast/inverters/solarman.py 42.10% 11 Missing ⚠️
quartz_solar_forecast/inverters/enphase.py 35.71% 9 Missing ⚠️
quartz_solar_forecast/inverters/givenergy.py 40.00% 6 Missing ⚠️
quartz_solar_forecast/inverters/solis.py 60.00% 6 Missing ⚠️
quartz_solar_forecast/data.py 83.33% 2 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (a9f0c79) and HEAD (6038636). Click for more details.

HEAD has 1 upload less than BASE
Flag BASE (a9f0c79) HEAD (6038636)
3 2
Additional details and impacted files
@@             Coverage Diff             @@
##             main     #183       +/-   ##
===========================================
- Coverage   79.31%   58.33%   -20.98%     
===========================================
  Files          18       23        +5     
  Lines         551      984      +433     
===========================================
+ Hits          437      574      +137     
- Misses        114      410      +296     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@peterdudfield
Copy link
Contributor

Thanks @mduffin95, this is very neat, Really appreciate all this work youve done.

@aryanbhosale would you like to review this?

@aryanbhosale
Copy link
Member

Thanks @mduffin95, this is very neat, Really appreciate all this work youve done.

@aryanbhosale would you like to review this?

Yup sure, I'll have a look at this right now

@aryanbhosale
Copy link
Member

Thanks @mduffin95, this is very neat, Really appreciate all this work youve done.

@aryanbhosale would you like to review this?

Hey @peterdudfield I've added some reviews , do you want me to review something again or is this fine?

@mduffin95
Copy link
Contributor Author

@aryanbhosale I can't see any comments. Have you submitted the review?

@aryanbhosale
Copy link
Member

@aryanbhosale I can't see any comments. Have you submitted the review?

Hey yes I've added the reviews, check the file changes tab probably it'll show up there

@mduffin95
Copy link
Contributor Author

mduffin95 commented Aug 16, 2024

@aryanbhosale any comments added should show up on this page. You'd also show up in the reviewers section if you'd submitted a review. Do the comments say "pending" next to them for you?

@aryanbhosale
Copy link
Member

@aryanbhosale any comments added should show up on this page. You'd also show up in the reviewers section if you'd submitted a review. Do the comments say "pending" next to them for you?
Yeah they say pending next to mine

@mduffin95
Copy link
Contributor Author

Yeah they say pending next to mine

@aryanbhosale this means you need to submit the review before I can see the comments.

Copy link
Member

@aryanbhosale aryanbhosale left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @mduffin95 ,thanks for this, I've put some reviews - mostly on separation of concerns (model classes from processing) and making it compatible with the existing api functionality

@@ -144,61 +135,11 @@ def format_nwp_data(df: pd.DataFrame, nwp_source:str, site: PVSite):
)
return data_xr

def fetch_enphase_data() -> Optional[pd.DataFrame]:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why have these fetch functions for different inverter brands been removed?

Copy link
Contributor Author

@mduffin95 mduffin95 Aug 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The functionality has just been moved to the get_data method for each inverter.

quartz_solar_forecast/data.py Show resolved Hide resolved
quartz_solar_forecast/data.py Show resolved Hide resolved
quartz_solar_forecast/data.py Show resolved Hide resolved
quartz_solar_forecast/inverters/enphase.py Show resolved Hide resolved
quartz_solar_forecast/inverters/enphase.py Show resolved Hide resolved
quartz_solar_forecast/inverters/enphase.py Show resolved Hide resolved
quartz_solar_forecast/inverters/givenergy.py Show resolved Hide resolved
quartz_solar_forecast/pydantic_models.py Show resolved Hide resolved
quartz_solar_forecast/pydantic_models.py Show resolved Hide resolved
@aryanbhosale
Copy link
Member

Yeah they say pending next to mine

@aryanbhosale this means you need to submit the review before I can see the comments.

Ah my bad, thank you for that(bad network on my side so didn't get submitted)

Copy link
Member

@aryanbhosale aryanbhosale left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have these been tested @mduffin95 - with the api and the dashboard?

@aryanbhosale
Copy link
Member

image
image
image
image
image
image

I get these errors with enphase

@aryanbhosale
Copy link
Member

image image image image image image

I get these errors with enphase

It's the same issue with all the inverters - not just enphase

@mduffin95
Copy link
Contributor Author

It looks like that's because the env file has multiple variables set up. I only tried it with variables for GivEnergy. If we want to allow that I should be able to tweak the settings.

@aryanbhosale
Copy link
Member

It looks like that's because the env file has multiple variables set up. I only tried it with variables for GivEnergy. If we want to allow that I should be able to tweak the settings.

@mduffin95 error with enphase:
INFO: 127.0.0.1:49020 - "POST /forecast/ HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/venv/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 406, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/venv/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/venv/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/venv/lib/python3.11/site-packages/starlette/applications.py", line 123, in call
await self.middleware_stack(scope, receive, send)
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in call
raise exc
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in call
await self.app(scope, receive, _send)
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/venv/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in call
await self.app(scope, receive, send)
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/venv/lib/python3.11/site-packages/starlette/routing.py", line 754, in call
await self.middleware_stack(scope, receive, send)
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/venv/lib/python3.11/site-packages/starlette/routing.py", line 774, in app
await route.handle(scope, receive, send)
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/venv/lib/python3.11/site-packages/starlette/routing.py", line 295, in handle
await self.app(scope, receive, send)
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/venv/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/venv/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
response = await f(request)
^^^^^^^^^^^^^^^^
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/venv/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/venv/lib/python3.11/site-packages/fastapi/routing.py", line 193, in run_endpoint_function
return await run_in_threadpool(dependant.call, **values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/venv/lib/python3.11/site-packages/starlette/concurrency.py", line 42, in run_in_threadpool
return await anyio.to_thread.run_sync(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/venv/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread
return await future
^^^^^^^^^^^^
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 859, in run
result = context.run(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/api/app/api.py", line 45, in forecast
predictions_with_live = run_forecast(site=site, ts=timestamp)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/quartz_solar_forecast/forecast.py", line 112, in run_forecast
return predict_ocf(site, None, ts, nwp_source)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/quartz_solar_forecast/forecast.py", line 30, in predict_ocf
pv_xr = make_pv_data(site=site, ts=ts)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/quartz_solar_forecast/data.py", line 183, in make_pv_data
live_generation_kw = site.get_inverter().get_data(ts)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/quartz_solar_forecast/inverters/enphase.py", line 32, in get_data
return get_enphase_data(self.__settings)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/quartz_solar_forecast/inverters/enphase.py", line 163, in get_enphase_data
access_token = get_enphase_access_token(settings)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/aryan/OneDrive - BITS Pilani K K Birla Goa Campus/Desktop/myLab4Prac/GSoC_openclimatefix/Open-Source-Quartz-Solar-Forecast/quartz_solar_forecast/inverters/enphase.py", line 106, in get_enphase_access_token
conn.request(
File "/usr/lib/python3.11/http/client.py", line 1303, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.11/http/client.py", line 1314, in _send_request
self.putrequest(method, url, **skips)
File "/usr/lib/python3.11/http/client.py", line 1148, in putrequest
self._validate_path(url)
File "/usr/lib/python3.11/http/client.py", line 1248, in _validate_path
raise InvalidURL(f"URL can't contain control characters. {url!r} "
http.client.InvalidURL: URL can't contain control characters. "/oauth/token?grant_type=authorization_code&redirect_uri=https://api.enphaseenergy.com/oauth/redirect_uri&code=client_id='CLIENT_ID' system_id='SYSTEM_ID' api_key='API_KEY' client_secret='CLIENT_SECRET'" (found at least ' ')

GivEnergy plot with live kw might be inconsistent because of an extra/1000 - please check your code since the output should not be this far from the forecasted one
image

for Solis it doesnt fetch any live data: (Error retrieving Solis data: Timeout error occurred)
image

Solarman: this is the same as givenergy, please look out for the extra /1000 which is causing this inconsistency
image

i think once these errors are solved this can be merged in

@mduffin95
Copy link
Contributor Author

please look out for the extra /1000 which is causing this inconsistency

@aryanbhosale have you checked this functionality on the main branch? My change doesn't alter this part of the code.

Regarding this issue:

URL can't contain control characters. "/oauth/token?grant_type=authorization_code&redirect_uri=https://api.enphaseenergy.com/oauth/redirect_uri&code=client_id='CLIENT_ID' system_id='SYSTEM_ID' api_key='API_KEY' client_secret='CLIENT_SECRET'"

I've pushed a chane that I think might fix this. If it doesn't work do you think you might be able to take a quick look to find the fix yourself? As I don't have enphase credentials it's difficult for me to do.

@aryanbhosale
Copy link
Member

please look out for the extra /1000 which is causing this inconsistency

@aryanbhosale have you checked this functionality on the main branch? My change doesn't alter this part of the code.

Regarding this issue:

URL can't contain control characters. "/oauth/token?grant_type=authorization_code&redirect_uri=https://api.enphaseenergy.com/oauth/redirect_uri&code=client_id='CLIENT_ID' system_id='SYSTEM_ID' api_key='API_KEY' client_secret='CLIENT_SECRET'"

I've pushed a chane that I think might fix this. If it doesn't work do you think you might be able to take a quick look to find the fix yourself? As I don't have enphase credentials it's difficult for me to do.

What do you think could be the issue here?

@aryanbhosale
Copy link
Member

@mduffin95 everything works currently but with enphase its a little weird,
this is the streamlit ui:
image

even after i type the url, when i go back to my CLI, i see this:
image

This shouldnt happen, if im using something in the API/dashboard context everything should be done on the dashboard side itself, and when im using the CLI context, it should show up in the CLI, there shouldnt be a mismatch bw the 2, the code for this is currently in the recently pushed main branch of this repo

anyway if i paste the url in the cli after its prompted, it says this:
enphase.py", line 115, in get_enphase_access_token
access_token = data_json["access_token"]
~~~~~~~~~^^^^^^^^^^^^^^^^
KeyError: 'access_token'

We're using env vars to store the access/refresh token as of now when using the cli context, and conditionally checking if they exist, then dont go into the CLI context(in the current main branch in enphase.py)

Would be great if you could fix this, everything else works perfectly fine now

@aryanbhosale
Copy link
Member

one more thing @mduffin95 , when No Inverter is chosen, the columns should have only power_kw like this:
image

currently it has both power_kw and power_no_live_kw, and theyre both the same which doesnt make sense to have, it would be good if that change is made too before merging this in

@aryanbhosale
Copy link
Member

Great work here @mduffin95 ! It's fixed and works as expected. Thanks a lot for this.
@peterdudfield I'm happy for this to be merged in, would you like to have a look at it before we merge this in?

@aryanbhosale
Copy link
Member

hey @mduffin95 , do you mind having a look at the current docs for the api, as well as the dashboard_2 docs, and see if your changes affect anything there? and update it accordingly, it would be great if we could get this done in the same relevant PR

@mduffin95
Copy link
Contributor Author

I've added a line to the docs about making sure new inverter models extend AbstractInverter. Other than this I don't think any changes are required really because this is a refactoring and doesn't add new functionality.

@aryanbhosale
Copy link
Member

I've added a line to the docs about making sure new inverter models extend AbstractInverter. Other than this I don't think any changes are required really because this is a refactoring and doesn't add new functionality.

Thank you for this

Copy link
Member

@Sukh-P Sukh-P left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work on this!

@aryanbhosale
Copy link
Member

Nice work on this!

Can this be merged?

@Sukh-P
Copy link
Member

Sukh-P commented Aug 28, 2024

Nice work on this!

Can this be merged?

If yourself and @peterdudfield are happy with it too then yes I would say go ahead, maybe squash and merge as it's quite a few commits

@aryanbhosale
Copy link
Member

Nice work on this!

Can this be merged?

If yourself and @peterdudfield are happy with it too then yes I would say go ahead, maybe squash and merge as it's quite a few commits

I'm happy with it too, just waiting for @peterdudfield to give his comments over this before merging this in

@Sukh-P
Copy link
Member

Sukh-P commented Aug 28, 2024

In fact I would say go ahead now, you worked on this most @aryanbhosale so your approval was the key thing, so feel free to merge @mduffin95! Thanks again

@aryanbhosale aryanbhosale merged commit 3e11e48 into openclimatefix:main Aug 28, 2024
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants