Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatic binding generation #264

Closed
wants to merge 131 commits into from
Closed

Automatic binding generation #264

wants to merge 131 commits into from

Conversation

madsmtm
Copy link
Owner

@madsmtm madsmtm commented Sep 8, 2022

This is very similar to using bindgen, but I went with creating things from scratch because:

  1. bindgen is kinda intimidating, knowing so little about libclang, so I started here first; I'd still like to upstream some of this, but I'll start here for now.
  2. This only has to run on my machine every once in a while, whereas bindgen has to handle all C code under the sun. A few manual fixes and such are acceptable.
  3. I'd like to run this multiple times for different targets (iOS, macOS, tvOS, ... - GNUStep will probably suffer a bit here), and then merge the result using #[cfg] so that we can ship one, ready-to-use library.
  4. I'd like to enrich the output using separate files (.apinotes or similar), and such a thing will probably never belong in bindgen
  5. Other things that are harder to integrate into something existing, for example availability attributes.

That said, I have taken inspiration from the existing Objective-C implementation in bindgen, credits to @simlay.

In the end, I'm envisioning something like this (so objc2 would be the Apple equivalent to the windows crate):

objc2/src/
  foundation/
    generated/
      ...
    unsafe_fns.toml // Or `Foundation.apinotes`, or similar
    string.rs
    mutable_string.rs
    ...
  appkit/...
  core_data/...
  core_display/...
  // And so on, for the Apple frameworks that expose an Objective-C interface

See #85 (comment) for a bit of details on what Swift does.

@madsmtm madsmtm added the enhancement New feature or request label Sep 8, 2022
This was referenced Sep 8, 2022
@madsmtm
Copy link
Owner Author

madsmtm commented Sep 9, 2022

I estimated the size of the code if we included all of Apple's frameworks (assuming Rust and C have similar line ratios); it would be around 13MB zipped, 50MB unzipped. For reference, the windows crate is ~14MB zipped, ~200MB unzipped.

@madsmtm
Copy link
Owner Author

madsmtm commented Sep 9, 2022

@madsmtm madsmtm mentioned this pull request Sep 17, 2022
4 tasks
@madsmtm
Copy link
Owner Author

madsmtm commented Sep 19, 2022

I probably won't put this under objc2, instead I'll create a new crate with the frameworks. Possible names, in order of my current preference:

  1. apple, is an empty crate owned by @zu1kd, have sent them an email to discuss name transfer
  2. icrate, a play on Apple's "i"-prefixing (iPhone/iPad/...)
  3. ilib, similar to above
  4. frameworks, ambiguous since also commonly used for "web frameworks"
  5. appel, wordplay

@madsmtm
Copy link
Owner Author

madsmtm commented Sep 23, 2022

Example of how I want at least some of AppKit to look like: https://github.com/rust-windowing/winit/tree/fafdedfb7d3a7370ca4b01108f7713b685633164/src/platform_impl/macos/appkit

@madsmtm
Copy link
Owner Author

madsmtm commented Oct 4, 2022

init methods always return the same type as the receiver: https://clang.llvm.org/docs/AutomaticReferenceCounting.html#related-result-types

How can we model this nicely?

@madsmtm
Copy link
Owner Author

madsmtm commented Oct 5, 2022

@madsmtm
Copy link
Owner Author

madsmtm commented Oct 5, 2022

Idea: Use (parts of) Swift's test suite (for example this one for errors: https://github.com/apple/swift/blob/7123d2614b5f222d03b3762cb110d27a9dd98e24/test/Inputs/clang-importer-sdk/usr/include/errors.h)

@madsmtm madsmtm force-pushed the header-translator branch 4 times, most recently from dedf7d4 to 420f9a0 Compare October 9, 2022 01:28
@madsmtm
Copy link
Owner Author

madsmtm commented Oct 10, 2022

Clang has "module maps" for figuring out what to import from Objective-C headers - we should use those!

@ericmarkmartin
Copy link

Hey,

I'm really excited about this enhancement and would find it super useful for some work I'm doing on binding the AuthenticationServices framework. That in mind, I have a couple of high-level questions.

  1. do you have a rough ETA for when this thing might land? If it's soon-ish I'll probably hold off on writing my own bindings but otherwise I might just proceed on my own.
  2. if you think this is still a little ways out, is there any way I could help to bring this around sooner?

@madsmtm
Copy link
Owner Author

madsmtm commented Oct 31, 2022

Currently I'm trying to push forwards on an initial version that actually compiles Foundation and AppKit.

Primary missing items are:

Timeline on this is unknown, but I have quite a lot of free time this week, so I should be able to get pretty far. I can probably give you a better estimate at the end of the week, if I'm not done by then.

Once I have this finished, there'll be lots of smaller things to work on, which I could probably use some help with, including:

  • Actual type-safety (creating newtypes instead of type-aliases for enums, typedefs and so on, allowing us to later on mark various methods as safe)
  • Extern functions
  • Inline functions (requires quite a lot, since we effectively have to translate C code to Rust. But we can probably get quite far with a few tricks)
  • Block support
  • Protocols: Add extern_protocol! macro and ProtocolType trait #250
  • "out" parameters: Handling objects passed as "out parameters" #277
  • Autogenerated documentation (using code comments, and ideally data from developer.apple.com)
  • Possibly a more "swifty" translation (tweaking method names, making enums associated constants). Probably better to do after we have autogenerated documentation, since there is great value in the current design where you can very easily find what you're looking for

I'll try to see if I can figure out a good way for you / somebody else to contribute with some of that, when I get there.

@madsmtm
Copy link
Owner Author

madsmtm commented Nov 1, 2022

I could use some guidance though: I'll admit I'm not the best at git, and I'm unsure if storing the generated result in git is the best solution? It does make audits (security and just general) easier, at the cost of blowing up the amount files we store. An alternative could also be to create a new repo just for it.

But what do you think?

Copy link
Collaborator

@simlay simlay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could use some guidance though: I'll admit I'm not the best at git, and I'm unsure if storing the generated result in git is the best solution? It does make audits (security and just general) easier, at the cost of blowing up the amount files we store. An alternative could also be to create a new repo just for it.

But what do you think?

Putting it in a different repo isn't a bad idea but I think in general what I'd say is that the generated stuff in tracking should be either for some kind of sys-crate (or icrate) or for integration tests. Bindgen keeps the input headers and the expectations in git and verifies that they match as expected in CI and the output does compile. Using the full frameworks to test is useful to know breadth of functionality but I believe it can change from macOS/SDK version to version.

After reviewing some stuff, it seems like the header-translator crate is meant to run as a binary and then either store the output in the crate and tracking. I tend to like using a generator crate in a build script.

header-translator/src/main.rs Outdated Show resolved Hide resolved
});
}

fn output_files(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should you want to use this crate as a build script, I think output_files (and other stuff) will need to go in the lib.rs

@madsmtm
Copy link
Owner Author

madsmtm commented Nov 1, 2022

I tend to like using a generator crate in a build script

Yeah, I'm aware of that approach, my desire with header-translator/icrate was to do things a bit differently than bindgen, to improve:

  • Compilation speed, since the work is only done once on my local machine, and not on every user's machine (here I'm hoping that download speed will not be a limiting factor)
  • Documentation, since items that are only available on one platform would be hidden behind appropriate cfgs
  • The need to have a specific SDK version installed (would help with cross-compiling on Linux)

I believe it can change from macOS/SDK version to version.

I think my desire was to manually keep icrate up to date with new SDK versions. I'm currently compiling for the macOS 12.3 SDK, on a machine running macOS 10.14.6.

@notgull
Copy link
Contributor

notgull commented Dec 16, 2022

Well, I was writing an automatic binding generator for AppKit, only to find out that you're doing it already; and from a glance it looks like your version is way ahead of mine! :-)

My only complaint would be that, in my opinion, having the setup use Deref in order to emulate inheritance isn't hygenic. If it were up to me, I'd do something like what gtk-rs does:

// Have this trait.
pub trait IsA<T> {}

pub struct NSDate { /* .. */ }
impl IsA<NSDate> for NSDate {}
impl IsA<NSObject> for NSObject {}

// Extension trait.
pub trait NSDateExt : IsA<NSDate> { /* ... */ }
impl<T: IsA<NSDate>> NSDateExt for T {}

That being said, I'm very excited to see this once it reaches readiness. Please let me know if there's anything I can do for this.

madsmtm added a commit that referenced this pull request Dec 21, 2022
See full history in 1c4c875.

Add initial header translation

Closer to usable

Mostly works with NSCursor

Mostly works with NSAlert.h

Refactor a bit

AppKit is now parse-able

handle reserved keywords

Handle protocols somewhat

Handle the few remaining entity kinds

Works with Foundation

Cleanup

Refactor

Refactor Method to (almost) be PartialEq

Parse more things

Parse NSConsumed

Verify memory management

More work

Fix reserved keywords

Refactor statements

Add initial availability

Prepare RustType

Split RustType up in parse and ToToken part

Add icrate

Add config files

Temporarily disable protocol generation

Generate files

Add initial generated files for Foundation

Skip "index" header

Add basic imports

Allow skipping methods

Properly emit `unsafe`

Make classes public

Rename objc feature flag

Improve imports somewhat

Further import improvements

Handle basic typedefs

Improve generics handling

Improve pointers to objects

Refactor RustType::TypeDef

Mostly support generics

Refactor config setup

Small fixes

Support nested generics

Comment out a bit of debug logging

Emit all files

Parse sized integer types

Parse typedefs that map to other typedefs

Appease clippy

Add `const`-parsing for RustType::Id

Parse Objective-C in/out/inout/bycopy/byref/oneway qualifiers

Fix `id` being emitted when it actually specifies a protocol

Make AppKit work again

Parse all qualifiers, in particular lifetime qualifiers

More consistent ObjCObjectPointer parsing

Validate some lifetime attributes

Fix out parameters (except for NSError)

Assuming we find a good solution to #277

Refactor Stmt objc declaration parsing

Clean up how return types work

Refactor property parsing

Fixes their order to be the same as in the source file

Add support for functions taking NSError as an out parameter

Assuming we do #276

Change icrate directory layout

Refactor slightly

Refactor file handling to allow for multiple frameworks simultaneously

Put method output inside an extern_methods! call

We'll want this no matter what, since it'll allow us to extend things with availability attributes in the future.

Use extern_methods! functionality

To cut down on the amount of code, which should make things easier to review and understand.

This uses features which are not actually yet done, see #244.

Not happy with the formatting either, but not sure how to fix that?

Manually fix the formatting of attribute macros in extern_methods!

Add AppKit bindings

Speed things up by optionally formatting at the end instead

Prepare for parsing more than one SDK

Specify a minimum deployment target

Document SDK situation

Parse headers on iOS as well

Refactor stmt parsing a bit

Remove Stmt::FileImport and Stmt::ItemImport

These are not nearly enough to make imports work well anyhow, so I'll rip it out and find a better solution

Do preprocessing step explicitly as the first thing

Refactor so that file writing is done using plain Display

Allows us to vastly improve the speed, as well as allowing us to make the output much prettier wrt. newlines and such in the future (proc_macro2 / quote output is not really meant to be consumed by human eyes)

Improve whitespace in generated files and add warning header

Don't crash on unions

Add initial enum parsing

Add initial enum expr parsing

Add very simple enum output

Fix duplicate enum check

Improve enum expr parsing

This should make it easier for things to work on 32-bit platforms

Add static variable parsing

Add a bit of WIP code

Add function parsing

Fix generic struct generation

Make &Class as return type static

Trim unnecessary parentheses

Fix generics default parameter

Remove methods that are both instance and class methods

For now, until we can solve this more generally

Skip protocols that are also classes

Improve imports setups

Bump recursion limit

Add MacTypes.h type translation

Fix int64_t type translation

Make statics public

Fix init methods

Make __inner_extern_class allowing trailing comma in generics

Attempt to improve Rust's parsing speed of icrate

Custom NSObject

TMP import

Remove NSProxy

Temporarily remove out parameter setup

Add struct support

Add partial support for "related result types"

Refactor typedef parsing a bit

Output remaining typedefs

Fix Option<Sel> and *mut bool

Fix almost all remaining type errors in Foundation

Skip statics whoose value we cannot find

Fix anonymous enum types

Fix AppKit duplicate methods

Add CoreData

Properly fix imports

Add `abstract` keyword

Put enum and static declarations behind a macro

Add proper IncompleteArray parsing

Refactor type parsing

Make NSError** handling happen in all the places that it does with Swift

Refactor Ty a bit more

Make Display for RustType always sound

Add support for function pointers

Add support for block pointers

Add extern functions

Emit protocol information

We can't parse it yet though, see #250

Make CoreData compile

Make AppKit compile

Add support for the AuthenticationServices framework

Do clang < v13 workarounds without modifying sources

Refactor Foundation fixes
@madsmtm
Copy link
Owner Author

madsmtm commented Dec 21, 2022

My only complaint would be that, in my opinion, having the setup use Deref in order to emulate inheritance isn't hygenic. If it were up to me, I'd do something like what gtk-rs does:

// Have this trait.
pub trait IsA<T> {}

pub struct NSDate { /* .. */ }
impl IsA<NSDate> for NSDate {}
impl IsA<NSObject> for NSObject {}

// Extension trait.
pub trait NSDateExt : IsA<NSDate> { /* ... */ }
impl<T: IsA<NSDate>> NSDateExt for T {}

Yeah, still very much experimenting with what's the best approach!

I'm not sure what you mean by "isn't hygenic", would you care to elaborate?

That being said, I'm very excited to see this once it reaches readiness. Please let me know if there's anything I can do for this.

Sure, there will be fairly soon.

I'm done with the rebase and do actually have something workable over in #308 now, once that is merged it should become easier to collaborate.

madsmtm added a commit that referenced this pull request Dec 23, 2022
See full history in 1c4c875.

Add initial header translation

Closer to usable

Mostly works with NSCursor

Mostly works with NSAlert.h

Refactor a bit

AppKit is now parse-able

handle reserved keywords

Handle protocols somewhat

Handle the few remaining entity kinds

Works with Foundation

Cleanup

Refactor

Refactor Method to (almost) be PartialEq

Parse more things

Parse NSConsumed

Verify memory management

More work

Fix reserved keywords

Refactor statements

Add initial availability

Prepare RustType

Split RustType up in parse and ToToken part

Add icrate

Add config files

Temporarily disable protocol generation

Generate files

Add initial generated files for Foundation

Skip "index" header

Add basic imports

Allow skipping methods

Properly emit `unsafe`

Make classes public

Rename objc feature flag

Improve imports somewhat

Further import improvements

Handle basic typedefs

Improve generics handling

Improve pointers to objects

Refactor RustType::TypeDef

Mostly support generics

Refactor config setup

Small fixes

Support nested generics

Comment out a bit of debug logging

Emit all files

Parse sized integer types

Parse typedefs that map to other typedefs

Appease clippy

Add `const`-parsing for RustType::Id

Parse Objective-C in/out/inout/bycopy/byref/oneway qualifiers

Fix `id` being emitted when it actually specifies a protocol

Make AppKit work again

Parse all qualifiers, in particular lifetime qualifiers

More consistent ObjCObjectPointer parsing

Validate some lifetime attributes

Fix out parameters (except for NSError)

Assuming we find a good solution to #277

Refactor Stmt objc declaration parsing

Clean up how return types work

Refactor property parsing

Fixes their order to be the same as in the source file

Add support for functions taking NSError as an out parameter

Assuming we do #276

Change icrate directory layout

Refactor slightly

Refactor file handling to allow for multiple frameworks simultaneously

Put method output inside an extern_methods! call

We'll want this no matter what, since it'll allow us to extend things with availability attributes in the future.

Use extern_methods! functionality

To cut down on the amount of code, which should make things easier to review and understand.

This uses features which are not actually yet done, see #244.

Not happy with the formatting either, but not sure how to fix that?

Manually fix the formatting of attribute macros in extern_methods!

Add AppKit bindings

Speed things up by optionally formatting at the end instead

Prepare for parsing more than one SDK

Specify a minimum deployment target

Document SDK situation

Parse headers on iOS as well

Refactor stmt parsing a bit

Remove Stmt::FileImport and Stmt::ItemImport

These are not nearly enough to make imports work well anyhow, so I'll rip it out and find a better solution

Do preprocessing step explicitly as the first thing

Refactor so that file writing is done using plain Display

Allows us to vastly improve the speed, as well as allowing us to make the output much prettier wrt. newlines and such in the future (proc_macro2 / quote output is not really meant to be consumed by human eyes)

Improve whitespace in generated files and add warning header

Don't crash on unions

Add initial enum parsing

Add initial enum expr parsing

Add very simple enum output

Fix duplicate enum check

Improve enum expr parsing

This should make it easier for things to work on 32-bit platforms

Add static variable parsing

Add a bit of WIP code

Add function parsing

Fix generic struct generation

Make &Class as return type static

Trim unnecessary parentheses

Fix generics default parameter

Remove methods that are both instance and class methods

For now, until we can solve this more generally

Skip protocols that are also classes

Improve imports setups

Bump recursion limit

Add MacTypes.h type translation

Fix int64_t type translation

Make statics public

Fix init methods

Make __inner_extern_class allowing trailing comma in generics

Attempt to improve Rust's parsing speed of icrate

Custom NSObject

TMP import

Remove NSProxy

Temporarily remove out parameter setup

Add struct support

Add partial support for "related result types"

Refactor typedef parsing a bit

Output remaining typedefs

Fix Option<Sel> and *mut bool

Fix almost all remaining type errors in Foundation

Skip statics whoose value we cannot find

Fix anonymous enum types

Fix AppKit duplicate methods

Add CoreData

Properly fix imports

Add `abstract` keyword

Put enum and static declarations behind a macro

Add proper IncompleteArray parsing

Refactor type parsing

Make NSError** handling happen in all the places that it does with Swift

Refactor Ty a bit more

Make Display for RustType always sound

Add support for function pointers

Add support for block pointers

Add extern functions

Emit protocol information

We can't parse it yet though, see #250

Make CoreData compile

Make AppKit compile

Add support for the AuthenticationServices framework

Do clang < v13 workarounds without modifying sources

Refactor Foundation fixes
@madsmtm
Copy link
Owner Author

madsmtm commented Dec 23, 2022

A possible source of at least a bit of docs is also https://github.com/xamarin/apple-api-docs.

In general, Xamarin has done a lot of work in this space, I should try to take a look at their stuff at some point

@madsmtm madsmtm added the A-framework Affects the framework crates and the translator for them label Dec 23, 2022
@madsmtm
Copy link
Owner Author

madsmtm commented Dec 23, 2022

#308 is done 🎉

Will try to release a new version of icrate to crates.io today, so that you can play with it. I've opened tagged issues for some of the remaining stuff - do free to comment on any of them if you have suggestions for how to do something differently (especially ergonomics-wise, the hardest part in all of this is in the design work ;) )

@madsmtm madsmtm closed this Dec 23, 2022
@madsmtm madsmtm deleted the header-translator branch December 23, 2022 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-framework Affects the framework crates and the translator for them enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants