FreeCAD stubs

[tritao]

Hello @ostr00000, thanks for your comment and review in the FreeCAD Pythons bindings PR.

I have been taking a deeper look at your work in this repo and it really is outstanding. 👏

As you're now aware I've also been trying to improve the FreeCAD Python typing situation, went through a slightly different route since I wasn't fully aware of how advanced your work on this repository was, but it's probably for the best as I think the direct typing stubs from the binding stubs has a few advantages.

One of the goals of my work has always been to try and introduce more Python typing into the FreeCAD project code and I believe that requires having a more direct integration in the core.

After the Python interface files are merged into the core, I was going to start looking for the other missing pieces, but you seem to have taken care of such things here already.

At this point I am wondering if when we introduce such Python stubs in the core, if you think that it should use such files directly at typing stubs or if we should input such files into your pre-existing generator and re-use the same process you have.

Or if you think it could be cleaner to refactor some things in the core to further ease the work of future maintenance and generation of complete and accurate typings.

So overall I think it could be very beneficial if we could collaborate on this as you have a lot of experience working on this problem and on the spirit of open-source, would be best to continue from all the great work you have done here.

[ostr00000]

your work in this repo and it really is outstanding. 👏

Thanks, but there are so many dirty regex 😃
(on the other hand, comparing to other extracting methods like fully parsing code, the regex are quite fast).

Actually, the repo started as a quick script to generate helper stubs for my personal usage. Then I read somewhere on the FreeCAD forum about .xml files, so I improved this code.
And at this moment I decided that I would be waste not to share my work with others, so I put it into a git repo. Then there was a request for easy install, so I uploaded it to PyPI. And later I have been gradually improving the code.

Recently, I started refactoring code to parse the original c++ code (to avoid regex/macros/etc.) and extract the python type using the same heuristics I developed in the regex version.
I am mainly Python developer, so I have to learn a lot about c++ (especially about clang internals).
The work is currently in a proof of concept stage, because I write it when I feel like it and when I have free time, so it is not pushed to this repo.

---

At this point I am wondering if when we introduce such Python stubs in the core, if you think that it should use such files directly at typing stubs or if we should input such files into your pre-existing generator and re-use the same process you have.

I believe the best place for stubs is to be together with the c++ code. There were even a few ideas to integrate this code with FreeCAD repo, but I decided to dismiss the decision in time and stabilize code a bit more.
The reasons are mainly:

• always up-to-date code (today there was exactly that kind of issue - makeLoft has wrong keyword arguments #34),
• possible errors are fixed together in cpp code (sometimes I reported issues about wrong code in FreeCAD repo, but it was not always fixed).

So in a long run, I would be ok, to make this repo archived.

---

I was going to start looking for the other missing pieces

I believe these pieces may be missing when the FreeCAD will start uses stub to generate cpp:

1. Dynamic property and getter/setter types - this may be very tricky, because (if I remember correctly), some properties are added dynamically in cpp code instead of declaration in xml. On the other hand, few years ago python descriptors (__get__/__set__) were not recognized in typing system, but now it should be easier to implement it as a descriptor helper class.
2. rich comparison/number protocols - The problem here is the following: cpp code requires a certain entry in a structure (Python docs), but the implementation frequently raises an exception (for example, you can call both __add__ and __mul__, but __mul__ may raise an exception). So probably this generator code should be more fragmented. This may be also tricky for many overloads, so maybe it should be written manually (example for vector that needs overload).
3. Some modules are not accessible for a normal Python import. I tried to hack it using private modules to not import it directly. But maybe this is a bug in FreeCAD cpp code? see this comment for more details.
4. Some module attributes are added dynamically in cpp/python code. For example FreeCAD.GuiUp. I do not know how stubs code will deal with it when generating code, but these module variables are very frequently used in python code.
5. As a part of stub distribution package, I provide a normal python package with constants. I almost never use direct string, but prefer to use it through a variable/const (this allows to avoid typos). This part of code may be easy to extract/move, because its whole content is almost in a single file. This is actually not related to subs, but I believe it is worth to give python dev all constant hidden somewhere in cpp in a single Python module.
6. The most difficult part is this handcrafted stub file. This is a Protocol class for FreeCAD Proxy object and FreeCADGui ViewProvider object. Despite my efforts, these subs are not 100% valid, because the python dev do not have to override every method. It is something like an optional protocol that currently does not exist in Python. I was recently involved in a discussion about optional protocols (and later the discussion moved to python forum). This is a proposition to a new PEP, so it has a long way to become a part of python, so I propose to just copy this handcrafted file. Maybe it would be nice if we can import this and use standard python inheritance/nominal typing, instead of requiring structural typing? (I remember when I meet a FreeCAD code the first time, I had a huge problem how to find all available methods for Proxy object).
7. Listing raised exceptions in docstrings - this is a very, very optional. Also, I think that recently the number of possible exception has been reduced. But sometimes a module from FreeCAD Ext uses a non-standard exception.

As you can see, there are many small issues that I tried to not bother about it the FreeCAD devs, but probably now it is a good time to fix it.

What do you think about each of these points/steps?
Are these infos useful, or you would like to know something more specific?

[tritao]

Recently, I started refactoring code to parse the original c++ code (to avoid regex/macros/etc.) and extract the python type using the same heuristics I developed in the regex version.
I am mainly Python developer, so I have to learn a lot about c++ (especially about clang internals).
The work is currently in a proof of concept stage, because I write it when I feel like it and when I have free time, so it is not pushed to this repo.

That's a great idea. I have a lot of experience with such an approach (developed https://github.com/mono/CppSharp) along with Clang internals, so let me know if I can provide any assistance.

About your other points, thanks very much, this was the kind of details I was looking for. I will investigate in more detail so I can provide more feedback.

Do you think it will still beneficial to have libclang-based approach even with the Python stubs being provided directly by FreeCAD? Is there any information that we need from the C++ files that is not in the bindings? I have not looked into everything in detail so I am not sure.

But maybe even that was the case maybe it could still be beneficial to have the libclang-based tools to make sure all the signatures are correct?

[ostr00000]

Do you think it will still beneficial to have libclang-based approach even with the Python stubs being provided directly by FreeCAD?

I think I would be better to focus on stub first. And later, if we notice we are missing something that cannot be generated, consider scanning cpp code.

Is there any information that we need from the C++ files that is not in the bindings?

Definitely some points from above:

• 1 - dynamic properties - but maybe this should be declared in stubs, and we generate some an initialization method in cpp for it? I'm not sure, but it is possible that this code must be defined in a core cpp class, and not in a python wrapper, so it may be challenging.
• 4 - variables assigned to the module in cpp - probably only a few variables (like Control, ActiveDocumment). If it will be possible to create .pyi file with arbitrary variable on module level, then this is not needed.
• 5 - searching for const values - but as I explained above this is one of the first code I have written - it is about 100 python lines.

make sure all the signatures are correct?

I think that cpp compiler should be good enough to detect most mistakes.
The only vulnerable point may be Python parsing functions (like PyArg_ParseTuple, PyArg_ParseTupleAndKeywords), because these functions take an arbitrary number of arguments.
From my experience, the mistakes are frequent in these functions.
Anyway, the generated code should be much safer than current code.

[Andrei-Pozolotin]

@tritao , @ostr00000 :

Do you guys have an idea how to let more people contribute to the stubs
in order to move the refactoring of this project faster?

Perhaps you could agree on some core visitor-pattern based skeleton
and let various meta-info extractor approaches to co-exist,
regardless if they are regex-parse or clang-ast or manual-code based?

And, could you share the ownership of the project between yourselves,
so the demands on each maintainer would be less?

[tritao]

Right now I am focused on merging the Python interface stubs inside FreeCAD (starting with FreeCAD/FreeCAD#19450, and will be sending more for the rest of FreeCAD modules).

I think after that it's probably a good idea to update this repository so it can take advantage of that. And there will probably be some errors during this process so would be great to come up with a process to make sure there are no regressions.

Wish I had more concrete proposals at this time, but don't really know yet, that's why I create this issue to ask for some help from @ostr00000 as he has the most experience for this. I am open for any collaboration, in fact I would prefer it. Open to suggestions on how to proceed.

Eventually I think the best thing would be to integrate this work into FreeCAD, so the typing stubs are generated as part of the official build, where we all can share "ownership" and more importantly so FreeCAD core source code can take advantage of the typing as well.

[Andrei-Pozolotin]

@tritao :

João: Thank you for your work on:

• Base: Modernize Python bindings APIs FreeCAD/FreeCAD#19450

I had the following question for a long time but was afraid to ask 😄 :

Do you think it is feasible to convince FreeCAD core team to give up on
the whole PyCXX https://cxx.sourceforge.net/ disaster idea,
and migrate FreeCAD to the Cython instead so there will be no
gap at all between the python and the c/cpp, with the added benefit
of possibility to reduce the amount of needless cpp code in the project by 80% ?

https://cython.readthedocs.io/en/stable/src/userguide/wrapping_CPlusPlus.html

[tritao]

Feasible, probably yes, if we can come up with a set of convincing technical advantages. But it would be quite some work, so it needs to be worth it from a technical perspective. I don't have any previous experience with Cython, so at the moment I can't tell if it would be advisable or not.
But I do like the idea, it does seem like it has the pontetial to remove a lot of code and complexity.

Btw I also don't know that much about PyCXX, so why do you refer to it as disaster? Is it because it is not very well mantained?