pvlib for Julia

[langestefan]

Hi!

I have been an active user of pvlib for a couple of years and really love the project. I have used it both commercially and for personal projects.

Recently, I have moved most of my projects from Python to Julia because of the modelling, scientific computing and autodifferentiation (AD) ecosystems. Julia has some shortcomings of course, but for the most part I don't regret the move.

Except for one bit, which is that pvlib is not supported in Julia. I can get around that by calling python from Julia, but having a native Julia solution would be much better. Julia and photovoltaic simulations seem like a great match, so I made this post on the Julia discourse a while back. It got a little bit of traction and some discussions, and also @kevinsa5 responded. I have had a small discussion with him, and he suggested opening a discussion here.

As to concrete plans: I would like to start a 'pvlib for Julia' from scratch, written in Julia, but making use of the best ideas of pvlib as much as possible. Pvlib has had a long history and I don't have to / want to reinvent the wheel. However, I also don't want to have any cross-calling from Julia to Python, that is not the goal of this project.

The core features I have in mind right now will be:

• Modularity. Everything will be built on top of ModelingToolkit.jl. ModelingToolkit is an acausal modelling framework, such as Modelica. However, there is a much larger emphasis on numerical algorithms and performance. The benefit of this approach is that once components are built we don't have to write any electrical equations by hand: they will be built symbolically, simplified and then passed to (symbolic-) numerical algorithms for solving. This enables very high flexibility when defining and configuring systems. For example, I always found working with AC/DC optimizers in pvlib a bit of a pain in the ass, this will now be possible.
• Native autodifferentiation support. This will enable gradient-based optimization, sensitivity analysis, model-based machine learning etc. There is a massive ongoing effort in the Julia community to harmonize autodifferentiation across all packages, see JuliaDiff. In Julia you write your code first, and then you add autodifferentiation. This is distinctively different from Python, where you first choose a package (JAX, PyTorch, ...) and then you write your code in that framework. Hence I think it would be a very big task to add autodifferentiation support to pvlib-python, while in Julia it would be feasible. This is partly inspired by the paper dPV: An End-to-End Differentiable Solar-Cell Simulator which implemented these ideas at the cell level in JAX.
• Data analysis. Next to a simulation package it would be nice to have a data analysis package with more advanced features based on recent research efforts. This is inspired by Solar Data Tools, which uses a signal decomposition framework to support shading and soiling loss estimations from unlabeled PV data.
• Speed. Julia is fast, very fast, and much faster than Python. That means we no longer need any C backbones such as numpy. Everything can be natively written in Julia, even the heavy numerical algorithms, and it will be fast.

As I am just one person and can only work in my free time it doesn't make much sense that I work on this in isolation. It would be much more effective if it could be hosted under the umbrella of pvlib and lead by someone with more experience than me. Hence why I wanted to start this discussion here. I understand this is not a small effort, but now seems the right time to start.

I am very keen to hear your thoughts and opinions!

ps. As to the current state, I am already able to simulate IV curves with a basic single diode model defined in ModellingToolkit. Check the discourse post linked above if you're curious.

[markcampanelli]

Hi Stefan, Fortune favors the bold! I have heard good things about Julia. 🙂 My new saying is: “Python is a great language. This includes great for hanging oneself or one’s whole dev team.” A few comments about design decisions in pvlib that I find problematic: - The ModelChain wrapper does not seem to be easily/obviously extended with new models and the “configuration discovery” ends up being more opaque than convenient (IMHO). I’m very curious if the ModelingToolkit.jl would provide a better solution here. - It seems to have an “algorithm first” mindset over an “interface first” mindset. Given the size and scope, both anre equally important IMHO. If I want to drop in my own flavor of single-diode model, then it’s hard to know what interface I must adopt to ensure integration with the ModelChain is highly likely to be successful without a wandering into a bigger, interface-breaking refactor. (I learned that there is a DC-model plug-in mechanism in pvlib, but it just seems to get buried in the presentation.) - Pandas goes too deep (again, IMHO), into the lowest levels of almost all functions. Combining this with Python’s duck typing can make development painful and error prone at worst and inelegant at best. Also, what if I want to use polars instead? A time-stamped data provider interface would be great. (To be fair, this is more of an issue in the greater scientific Python community, but I wonder if Julia is more elegant/flexible here.) - Typing. Gradual typing is nice, but for all that attention to typing detail one doesn’t see any performance boost in Python. It’s kinda like putting lipstick on a pig. Another thing to consider: How might using Julia suffer from having a smaller “surrounding ecosystem” than Python, such as web-server, web-application, etc. packages. For example, can one write a lambda function in AWS using Julia? If not, then can one instead compile and deploy a Julia-based binary to AWS Lambda? (Maybe Julia has grown significantly and is not so deficient here?) Also, I’m curious how this effort might be taken on without fracturing/diluting the open-software development side of the solar PV modeling community. I know a major problem I have is finding time to contribute to pvlib while I still have to pay a mortgage and raise kids. For all of my misgivings about using Python in the large project that pvlib has become, I still greatly appreciate the time commitment by the maintainers, as well as Python’s popularity working to their advantage with regards to open source contribution. Happy to discuss further in this thread, and I may even have a real life project or two to develop as test use cases. (I am very concerned about my bandwidth, however.) Mark Campanelli LinkedIn <https://www.linkedin.com/in/markcampanelli/> Intelligent Measurement Systems LLC <https://intelligentmeasurementsystems.com> Try PVfit <https://pvfit.app> today!

…

On Fri, Dec 20, 2024 at 06:13 Stefan de Lange ***@***.***> wrote: Hi! I have been an active user of pvlib for a couple of years and really love the project. I have used it both commercially and for personal projects. Recently, I have moved most of my projects from Python to Julia because of the modelling, scientific computing and autodifferentiation (AD) ecosystems. Julia has some shortcomings of course, but for the most part I don't regret the move. Except for one bit, which is that pvlib is not supported in Julia. I can get around that by calling python from Julia, but having a native Julia solution would be much better. Julia and photovoltaic simulations seem like a great match, so I made this post on the Julia discourse <https://discourse.julialang.org/t/pvlib-for-julia/118952> a while back. It got a little bit of traction and some discussions, and also @kevinsa5 <https://github.com/kevinsa5> responded. I have had a small discussion with him, and he suggested opening a discussion here. As to concrete plans: I would like to start a 'pvlib for Julia' from scratch, written in Julia, but making use of the best ideas of pvlib as much as possible. Pvlib has had a long history and I don't have to / want to reinvent the wheel. However, I also don't want to have any cross-calling from Julia to Python, that is not the goal of this project. The core features I have in mind right now will be: - Modularity. Everything will be built on top of ModelingToolkit.jl <https://github.com/SciML/ModelingToolkit.jl>. ModelingToolkit is an acausal modelling framework, such as Modelica. However, there is a much larger emphasis on numerical algorithms and performance. The benefit of this approach is that once components are built we don't have to write any electrical equations by hand: they will be built symbolically, simplified and then passed to (symbolic-) numerical algorithms for solving. This enables very high flexibility when defining and configuring systems. For example, I always found working with AC/DC optimizers in pvlib a bit of a pain in the ass, this will now be possible. - Native autodifferentiation support. This will enable gradient-based optimization, sensitivity analysis, model-based machine learning etc. There is a massive ongoing effort in the Julia community to harmonize autodifferentiation across all packages, see JuliaDiff <https://github.com/JuliaDiff/>. In Julia you write your code first, and then you add autodifferentiation. This is distinctively different from Python, where you first choose a package (JAX, PyTorch, ...) and then you write your code in that framework. Hence I think it would be a very big task to add autodifferentiation support to pvlib-python, while in Julia it would be feasible. This is partly inspired by the paper dPV: An End-to-End Differentiable Solar-Cell Simulator <https://arxiv.org/abs/2105.06305> which implemented these ideas at the cell level in JAX. - Data analysis. Next to a simulation package it would be nice to have a data analysis package with more advanced features based on recent research efforts. This is inspired by Solar Data Tools <https://solar-data-tools.readthedocs.io/en/main/>, which uses a signal decomposition framework to support shading and soiling loss estimations from unlabeled PV data. - Speed. Julia is fast, very fast, and much faster than Python. That means we no longer need any C backbones such as numpy. Everything can be natively written in Julia, even the heavy numerical algorithms, and it will be fast. As I am just one person and can only work in my free time it doesn't make much sense that I work on this in isolation. It would be much more effective if it could be hosted under the umbrella of pvlib and lead by someone with more experience than me. Hence why I wanted to start this discussion here. I understand this is not a small effort, but now seems the right time to start. I am very keen to hear your thoughts and opinions! ps. As to the current state, I am already able to simulate IV curves with a basic single diode model defined in ModellingToolkit. Check the discourse post linked above if you're curious. — Reply to this email directly, view it on GitHub <#2337>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAISX4ZX6PNNOGG5WLH7LUT2GQJWTAVCNFSM6AAAAABT7DTMBSVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZXG4YTSOJVG4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[langestefan]

Thanks, great points!

I’m very curious if the ModelingToolkit.jl would provide a better solution here.

Not necessarily. This is mostly a software design problem, having the right abstraction level and clearly defined interfaces is required to make it easy to extend the components of the ModelChain. This will be a matter of defining the right abstract types, their interfaces, and associated tests.

Pandas goes too deep

Completely agree with this. Pandas API is very unintuitive to me and working with indexes is a PITA. Actually, this was the main reason I decided to trash my last pvlib project. I even made an issue about it a while back, though in hindsight I didn't word it that tactfully: #1926

Another thing to consider: How might using Julia suffer from having a smaller “surrounding ecosystem” than Python, such as web-server, web-application, etc. packages.

That's a valid concern, but for now I would say the focus should be on the modelling aspect, that's where Julia shines.

Also, I’m curious how this effort might be taken on without fracturing/diluting the open-software development side of the solar PV modeling community.

My hope is that by making PV simulations more accessible in Julia, the Julia community which mostly consists of scientists and engineers will want to start contributing as well. With proper software design we can make it easier to contribute, and perhaps we can reignite interest in some old pvlib contributors as well.

[kandersolar]

Splitting development effort is my main concern as well. I think the other maintainers would agree that pvlib-python already suffers from too little developer attention. I know I would want to get sucked into a fun new julia version of pvlib, and I am reluctant to sign on to taking that time away from pvlib-python. Especially when I already committed to maintaining solarfactors but have largely failed to deliver on that promise.

[langestefan]

Okay let me clear that up then, I am not requesting any development effort from pvlib core developers. But I understand your concern, is there a possible compromise? To me there is a lot of value in having a pvlib.jl, with the project being hosted under the pvlib organization. But I am also aware its perhaps too early for that, this post was mainly intended to gauge interest and get some ideas / start discussions.

[adriesse]

Interesting discussion and ideas. I think there are two distinct aspects to the proposal:

1. Using the language Julia
2. Using a modelling framework

Is there a modelling framework in Python that could be used to fulfill a subset of the wish list?

[langestefan]

There are some comparisons made with alternatives here: https://docs.sciml.ai/ModelingToolkit/stable/comparison/

I think one of the strengths of Modelingtoolkit.jl is that it integrates perfectly with the wider SciML/Julia ecosystem. Python packages generally live more in isolation.

It would anyway be a very large effort to rewrite much of core pvlib functionality in a structural /symbolic modelling language, without much benefit.

[kandersolar]

Another point worth considering: I believe julia's compiler is nearing readiness for producing "general purpose" shared library binaries, along the lines of what you could produce with C++ or Rust. Once that functionality on the julia tooling side is ready, it means a julia port of pvlib could be "easily" wrapped and used in other programming languages.

IIRC, we have heard from some companies that they like pvlib, but can't integrate python into their in-house software, so they either don't use pvlib at all, or have to reimplement their pvlib-based prototypes in whatever language their own software uses. Providing a "language-agnostic" version of pvlib by compiling to shared libraries could make it accessible to a substantial segment of users that are currently prevented from using it because of programming language barriers. Of course, it would take quite a lot of work to get to that point...

[echedey-ls]

I had a long comment redacted from a few days ago but I was hesitant to publish it. It was exactly that point of shared libraries/ABI/bindings/foreign function interfaces or whatever you wanna call it. When I first saw Julia introductions I didn't have a clue that compiled shared libraries were a planned thing.

In any case, I support whatever language that uses Apache Arrow (or other alternatives, if any) as the underlying data engine, and bindings are provided for other languages. I recommend having a look at it, as it supports 12 languages if I haven't forgotten how to count. https://arrow.apache.org/docs/index.html

I dream with Python being the sandbox to try and configure simulations and then that heavy-duty computations are done in a resources-efficient language.

All my support goes to shared libs.

[markcampanelli]

I think you folks might be onto something here :).

Using Julia instead of, say, Rust (or god forbid C/C++) for a binary-compatible PV "core computational library" might be a good way forward. This approach would stick with a scientific-computing friendly language (like Python), but with the added benefit of performant typing and compilation. In the longer run, pvlib-python (or other "wrappers", including something potentially also Julia-based) could provide the higher-level "ModelChain", iotools, pandas-connector, etc. functionalities. While I'm fairly sure Julia could do higher-level stuff now/soon too, I would suggest staying close to the metal in a core computation library, kinda like the original Matllab version (!). (Please, no object orientation...pretty please?)

Would this work for users that are otherwise currently blocked by Python when they move beyond prototype to production? Let's ask around!

Does anyone have a sense as to what the bindings generation process looks like, say if we wanted to make a "tracer bullet" where some existing Python code in pvlib calls a function in a Julia-compiled binary?

[markcampanelli]

I don't like to be a wet blanket, but here is a counter-argument to writing a core computation library in Julia in close cooperation with pvlib. I am wondering why DoE wouldn't say: If users cannot take pvlib-python to production, then why don't we refocus our resources back into SAM, which is (presumably) performant, has a lower-level SDK, and has Python bindings?

[markcampanelli]

@langestefan

Julia has some shortcomings of course, but for the most part I don't regret the move.

Would you mind listing these here or linking to another discussion if you already did this elsewhere?

[langestefan]

For me the negatives of switching were:

1. Even though code execution is fast, pre-compilation can be slow. So the first time you run a piece of code, it will pre-compile, and generally it will be slower. There's a guide here by Tim Holy that explains how to tackle this: https://julialang.org/blog/2021/01/precompile_tutorial/
2. Error messages can be difficult to read. Julia stacktraces are long, and reading them can be hard. It's somewhat of a skill you will have to develop in my opinion. We can tackle this by catching errors and displaying user-friendly error messages. This will really help users.
3. Julia packages are developed with a 'do one thing, but do it well' kind of philosophy. Very similar to Linux. The upside of that is that in general packages compose very well, packages that were never explicitly designed to work together work flawlessly. Examples are everything that conforms to the Tables.jl interface works so well together. The downside is that you may have to load / install many packages. For example when I first started working with DataFrames.jl I wanted to use timezone-aware timestamps. To make that work I needed DataFrames.jl, TimeZones.jl and Dates.jl. These last two packages operate completely independently of DataFrames.jl, that is very different from using Pandas or Polars so you have to get used to that.
4. Since Julia community is smaller than Python's naturally there will be less functionality, examples and documentation. However, since the Julia community mostly consists of engineers and scientists, I have found almost everything I use to be of high quality. The community is also very nice and helpful, for example I often post questions on https://discourse.julialang.org/ and I always get answers of word-class experts on almost any topic. In fact the Julia community being so amazing is one of the biggest reasons I don't regret the switch.

[markcampanelli]

@langestefan Ok, so when Julia encounters a new type argument to a “polymorphically” defined function, it must somehow determine if that type has a compatible interface before the function can be compiled. Do you know of a good reference that explains this process (to a newbie)?

[langestefan]

Not entirely sure what you mean. Are you asking about multiple dispatch?

[markcampanelli]

Your question answered my question :). I'm such a Julia newbie that I didn't even know it was called "multiple dispatch".

I quickly found this: https://discourse.julialang.org/t/understanding-multiple-dispatch/76601

A question that comes to me is if multiple dispatch can be extended to the aforementioned "general purpose" shared library binaries? (I would be very surprised if it could be done simply.) That is, does one have to somehow nail down the types when generating said binaries?