Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support publishing Flutter plugin + Dart standalone hybrid packages #3563

Open
dcharkes opened this issue Sep 12, 2022 · 13 comments
Open

Support publishing Flutter plugin + Dart standalone hybrid packages #3563

dcharkes opened this issue Sep 12, 2022 · 13 comments
Labels
type-enhancement A request for a change that isn't a bug

Comments

@dcharkes
Copy link
Contributor

I was under the impression that we could publish Flutter plugins that also run in Dart standalone (ignoring the Flutter plugin code) by a pubspec with:

name: mylib_source
description: A new Flutter FFI plugin project.
version: 0.0.1
homepage:

environment:
  sdk: ">=2.17.0 <3.0.0"
  # no flutter:

dependencies:

dev_dependencies:

flutter:
  plugin:
    platforms:
      android:
        package: com.example.mylib_source
        ffiPlugin: true
      ios:
        ffiPlugin: true
      macos:
        ffiPlugin: true
      linux:
        ffiPlugin: true
      windows:
        ffiPlugin: true

This works locally with both the Flutter and Dart standalone SDK. (For example dart pub get && dart test and flutter pub get && flutter test, where dart is not from the Flutter SDK.)

However, pub does not support publishing this:

$ dart pub publish --dry-run
// ...
Package validation found the following error:
* pubspec.yaml allows Flutter SDK version 1.9.x, which does not support the flutter.plugin.platforms key.
  Please consider increasing the Flutter SDK requirement to ^1.10.0 (environment.sdk.flutter)
  
  See https://flutter.dev/docs/development/packages-and-plugins/developing-packages#plugin
Sorry, your package is missing a requirement and can't be published yet.
For more information, see: https://dart.dev/tools/pub/cmd/pub-lish.

Adding the flutter key makes dart pub get fail:

Resolving dependencies... 
Because jni requires the Flutter SDK, version solving failed.

Flutter users should run `flutter pub get` instead of `dart pub get`.

As far as I can see, the only workaround is to publish two packages.

cc @mahesh-hegde @jonasfj

edit: duplicate of #2606

@mahesh-hegde
Copy link

mahesh-hegde commented Sep 12, 2022

For me the result depends only on which SDK dart command is from.

mahesh:~/Code/jnigen/jnigen/example/pdfbox_plugin/dart_example [jni_refactor=] 
$ export PATH=$HOME/.local/dart-sdk/bin:$PATH
mahesh:~/Code/jnigen/jnigen/example/pdfbox_plugin/dart_example [jni_refactor=] 
$ dart pub get
Resolving dependencies... 
Because pdf_info depends on jni from path which requires the Flutter SDK, version solving failed.

Flutter users should run `flutter pub get` instead of `dart pub get`.
[1]
mahesh:~/Code/jnigen/jnigen/example/pdfbox_plugin/dart_example [jni_refactor=] 
$ export PATH=$HOME/.local/flutter/bin:$PATH
mahesh:~/Code/jnigen/jnigen/example/pdfbox_plugin/dart_example [jni_refactor=] 
$ dart pub get
Resolving dependencies... (2.6s)
Got dependencies!

It's confirmed also by the CI behaviour. The check in which I used setup-dart action failed. I updated it to use flutter-action and it passed.

@jonasfj
Copy link
Member

jonasfj commented Sep 13, 2022

Context: We did this years ago because older versions of the Flutter SDK didn't support flutter.plugin.platforms.

I suppose we could tweak the behavior such that the Flutter SDK constraint is expressed as a minimum Dart SDK constraint.

@sigurdm, any thoughts?

@sigurdm
Copy link
Contributor

sigurdm commented Sep 14, 2022

A few thoughts:

We need to have some story to tell here, but I'm not entirely sure how this kind of module should be published.

I feel it is somewhat surprising for a package that works as a dart-only package has a flutter.plugin.platforms key.

Could it be that there should be two packages, one having the dart ffi part, and another one importing the other, and wrapping it as a flutter plugin?

Even if that is technically feasible there would be downsides to that:

  • branding a single package is easier
  • publishing a single package is easier

@dcharkes
Copy link
Contributor Author

@sigurdm another downside is that all your downstream packages have to do the same.

I feel it is somewhat surprising for a package that works as a dart-only package has a flutter.plugin.platforms key.

The contract is that the plugin is used to bundle the same native code for the same Dart API. But that contract is not enforced.

dnys1 added a commit to dnys1/amplify-flutter that referenced this issue Dec 18, 2022
Adds workaround for dart-lang/pub#3563 which requires using Flutter's `dart` command if a package lists a Flutter SDK constraint even if the package does not depend on Flutter.
dnys1 added a commit to dnys1/amplify-flutter that referenced this issue Dec 18, 2022
Adds workaround for dart-lang/pub#3563 which requires using Flutter's `dart` command if a package lists a Flutter SDK constraint even if the package does not depend on Flutter.
dnys1 added a commit to dnys1/amplify-flutter that referenced this issue Dec 18, 2022
Adds workaround for dart-lang/pub#3563 which requires using Flutter's `dart` command if a package lists a Flutter SDK constraint even if the package does not depend on Flutter.
dnys1 added a commit to dnys1/amplify-flutter that referenced this issue Dec 18, 2022
Adds workaround for dart-lang/pub#3563 which requires using Flutter's `dart` command if a package lists a Flutter SDK constraint even if the package does not depend on Flutter.
dnys1 added a commit to aws-amplify/amplify-flutter that referenced this issue Jan 10, 2023
* chore(aft): Make base commands sync

There isn't any value with these being async since it's okay to block in this context. And sync makes everything easier to work with.

commit-id:6743d9d4

* feat(aft): Changelog/version commands

commit-id:0c17379d

* fix(aws_common): Logger initialization

Fixes an issue in logger initialization where the initial plugin is not registered to the root plugin.

commit-id:063fbfe0

* fix(aft): Versioning algorithm and performance

commit-id:c45852ad

* chore(aft): Clean up

commit-id:de10a06e

* chore(aft): Update README

commit-id:8261f867

* chore(aft): Update tests

Expand e2e tests to incude changelog/pubspec changes

commit-id:e9757240

* chore(aft): Update dependencies

commit-id:4509edf2

* chore(aft): Bump dependencies

commit-id:14f44d17

* chore(aft): Enforce changelog update includes commits

Changelog updates should only be made with a non-empty list of commits.

commit-id:4eccafce

* chore(aft): Add `promptYesNo` helper

commit-id:8544885f

* chore(aft): Refine `isExample`

Refine the definition of `isExample` to be more precise and not require workarounds.

commit-id:51cc0ac0

* chore(aft): Change `aft version` to `aft version-bump`

commit-id:0526b477

* chore(aft): Remove from mono_repo

The package libgit2dart, while it can be used in Dart-only packages tricks pub into thinking it has a dependency on Flutter. mono_repo cannot handle this discrepancy.

commit-id:ec27a8d2

* fix(aft): Publish constraints

Fixes constraints around publishing checks and which packages to consider for publishing.

commit-id:937ee042

* chore(aft): Publish command checks

Improves publish command checks by removing `pubspec_overrides` and not splitting the pre-publish and publish commands for a package.

* chore(aft): Add components and define version bump types

* chore(aft): Clean up `deps` command

Fix logging and merge `pub.dev` logic with `publish`

* chore(aft): Add placeholders for version bump commit

* chore(aft): Update logging settings

* test(aft): Update e2e tests

* More updates

* chore(aft): More cleanup

* chore(aft): Add propagation option

* chore(aft): Remove `changelog` command

* chore(aft): Clean up

* chore(aft): Update wording in `aft.yaml`

* chore(aft): Link packages before version bump

* chore(aft): Follow Dart SemVer strategy

Use build tag (`+`) for patch releases.

* chore(aft): Fix analysis errors

* fix(aft): Submodule `libgit2dart`

Adds workaround for dart-lang/pub#3563 which requires using Flutter's `dart` command if a package lists a Flutter SDK constraint even if the package does not depend on Flutter.

* fix(authenticator): ARB syntax

Remove unncessary commas

* chore(aft): Remove setup step

* chore(aft): Copy libgit2 from

Instead of trying to install it (since the latest version is not available in apt)

* chore(aft): Reset after patch

* chore(aft): Copy lib to /usr

* fix(aft): `dev_dependency` conflicts

When `dev_dependencies` contains versions of the published packages different than those on pub.dev (for example when using path deps or `any`), this causes issues during pre-publish verification.

By simply keeping these constraints up-to-date as well, we lose nothing (since they are overridden in local development via linking), but gain stronger confidence in the pre-publish step.

* chore(aft): Change workflow path for libgit2

* chore(aft): Fix tests in CI

* chore(aft): Bump dependency

Co-authored-by: Dillon Nys <dillon.andre.nys@gmail.com>
@sigurdm sigurdm added type-enhancement A request for a change that isn't a bug and removed feature-request labels Sep 11, 2023
@dnys1
Copy link

dnys1 commented Apr 3, 2024

The requirement that Flutter be installed and used simply by virtue of the environment specifier creates a whole host of issues right now. It makes it very difficult to publish hybrid packages, which is creating a lot of sprawl in the ecosystem. But worse, it means that any project which has just one transitive dependency on such a package must have Flutter installed and use that only.

This negatively impacts the ability to run CI, compile dart for the server, use in Docker, etc. where the Flutter SDK is much less portable and unnecessary because the package may not even use Flutter.

IMO, the presence of an environment specifier should be modified to mean if running with Flutter pub, then this requirement holds.

The requirement for Flutter to be installed and used should only come from an explicit dependency on a Flutter SDK package.

@jonasfj
Copy link
Member

jonasfj commented May 16, 2024

@dnys1, @dcharkes what would happen if we simply removed environment.flutter from pubspec.yaml?

I suspect that dart pub publish would fail, because of some validations. We could disable those using dart pub publish --skip-validation. And maybe tweak them to go away.

Though in general it is a bit scary to let people make a top-level flutter section without a environment.flutter SDK constraint.

Just curious.

@dnys1
Copy link

dnys1 commented May 16, 2024

Hey @jonasfj, that's correct. As you mention, attempting to publish the following pubspec is unsupported.

name: hybrid_pkg
# ...

environment:
  sdk: ^3.0.0

dependencies:
  # Dart-only deps

flutter:
  plugin:
    platforms:
      ios:
        ffiPlugin: true

In this case, pub requires the presence of environment.flutter. I agree that that is a good design choice. However, as soon as you add that, you can no longer run dart pub anything from a standalone install--you must use the Dart SDK from Flutter.

This is true of the root package or if any transitive dependencies follow this format (which is required for publishing). It essentially colors the entire dependency tree to Flutter.

Importantly, this is despite the fact that the Flutter SDK is not actually needed. There can be zero dependencies/dev dependencies on Flutter in the graph and this requirement still holds.

The flutter section is currently the only way to signal to the Flutter tooling that when this package is used in a Flutter environment, it handles native code appropriately. This allows, for example, iOS/Android code to exist alongside Dart-FFI code in a single package and function in both worlds.

Basically, I don't believe that the presence of either flutter or environment.flutter sections should ever mandate the use of the Flutter SDK, only a direct/transitive dependency on an SDK-vended package. The requirement that environment.flutter be included if a flutter section is present seems totally reasonable, it just throws a major wrench in the works currently.

@jonasfj
Copy link
Member

jonasfj commented May 16, 2024

Hmm, it's technically possible to publishing without environment.flutter by using dart pub publish --skip-validation.

This is obviously not intended :)

And I don't know if Flutter will read the flutter section without a flutter SDK dependency. It probably will 🙈

Also SDK detection in Pana (which affects pub.dev UI) might be off. Not sure.


On topic: I think it's rather risky to change existing semantics for environment.flutter.
Though I can see how it's attractive.
If we want to go that way we should explore how many packages and versions would be affected.

Or maybe we do need a mechanism to indicate that flutter is optional. Or simply allow omitting environment.flutter when publishing.

Or maybe we need to wait for the native assets work @dcharkes is working on.

@dnys1
Copy link

dnys1 commented May 16, 2024

Hmm, it's technically possible to publishing without environment.flutter by using dart pub publish --skip-validation.

This is obviously not intended :)

Good to know! Though I agree there should be a happy path for it.

And I don't know if Flutter will read the flutter section without a flutter SDK dependency. It probably will 🙈

It does 🙂

Also SDK detection in Pana (which affects pub.dev UI) might be off. Not sure.

Yep 😅 dart-lang/pana#1351


Changing the semantics of environment.flutter is the most obvious path to me if it remains a (loose) requirement for publishing. Removing the requirement could work, too, except for cases where there truly is a requirement for a certain version of Flutter (though I can't think of a time this couldn't be conveyed solely via the sdk constraint).

It's true that there are few instances where you need this level of hybridization, but it would be very handy if there was first-class support for these kinds of packages. In the past, the only way was to have two separate packages, a Dart one with FFI, and a Flutter one which was a very thin wrapper (but which maybe used method channels and other SDK code). With native assets, there is no longer a need to use Flutter in the actual Dart code, only for the platform-specific configuration which lives outside lib and which the Dart tool will happily ignore.

@Levi-Lesches
Copy link
Contributor

Levi-Lesches commented Sep 4, 2024

Adding my vote here. Until native assets are fully supported by Dart and Flutter, the flutter: section is needed.

Even once it is, we would still need the following for federated plugins:

flutter:
  plugin:
    implements: flutter_local_notifications

While we may see more and more method channels being replaced by FFI, I don't think federation is going anywhere anytime soon. There is little value to a wrapper package, and it only adds bloat to pubspecs, monorepos, build + analyzer CI's, and Pub in terms of eating up more good names for packages.

I think Flutter 1.9.0 is old enough at this point that it shouldn't be an issue. At the very least, Pub can check that environment.sdk uses a version of Dart that is too new for Flutter 1.9.0 (pre null-safety) and therefore it is impossible for this warning to be relevant.

EDIT @jonasfj If it's decided that this warning should be removed, I would gladly open a PR as this is a blocker on a few of my projects

@jonasfj
Copy link
Member

jonasfj commented Sep 12, 2024

@sigurdm, @dcharkes maybe we should talk next week, I'm not entirely sure what we should do here.
But I'm increasingly convinced we should try to come up with some sort of solution.

@mkustermann
Copy link
Member

(Thinking out loud here)

A package may want to work in different environments (dart sdk, flutter sdk - maybe 3rd party sdks in the future). The language has support - e.g. via conditional imports - that allows the package's code paths to behave differently in different environments.

We'd want such a package to:

  • declare which environments it supports running in
  • define which versions of those environments it supports
  • may have environment specific configuration, assets, etc (e.g. encoded in pubspec.yaml or via hook/{build,link}.dart)

So it would seem natural that a pubspec.yaml that looks like this

environment:
  sdk: ^a.b.c
  flutter: ^x.y.z

gets attached the following meaning: This package is usable in normal Dart SDK in version ^a.b.c as well as in Flutter SDK in version ^x.y.z
=> i.e. the presence of environment.flutter doesn't imply it's a requirement, but the constraint applies iff used in that environment

A SDK like the Flutter SDK is basically a wrapper around our lower-level tools (e.g. pub, dart2js, dart2wasm, vm, core libraries, ...). It provides it's own command line interface as well as API surface (e.g. dart:ui). The SDK's package get/upgrade functionality would work on that SDKs environment key.

=> That's probably the key issue here: A flutter packages get will choose different transitive package versions than dart pub get would choose.
=> The consequence here would be that flutters SDK would operate on .flutter_tool/package_config.json and dart SDK would operate on .dart_tool/flutter_config.json.

(Thinking out loud further)

Having a separate version resolution would also allow environment-specific dependencies, so the package when used in flutter may depend on package:flutter_icons but when used in standalone sdk may depend on package:standalone_icons.

One could encode such environment specific imports in pubspec.yaml

environment:
  sdk: ^a.b.c
  flutter: ^x.y.z
dependencies:
  flutter_icons:
    - environment: flutter
    - >= ...
  standalone_icons:
    - environment: sdk
    - >= ...

(without this, the package couldn't depend on flutter-only packages, but all transitive deps would need to support both environments - which may not make sense for them)

(Thinking out loud further)

Any tool that is depending on package_config.json may have to choose between different ones now. That may include analyzer.

@jonasfj
Copy link
Member

jonasfj commented Oct 14, 2024

(just rambling off Martins thoughts)

We should probably drop the name environment if we completely change the semantics.
We could express it like:

dart: 3.5.1   # <-- language version, nothing more.
sdks:         # Supported SDKs
 - dart: ^3.7.0       # <-- Dart is supported
 - flutter: ^3.28.0   # <-- Flutter is supported

I'm not sure we should want SDK-specific dependencies.

=> That's probably the key issue here: A flutter packages get will choose different transitive package versions than dart pub get would choose.

I wonder if there are ways to solve that problem. Ways to allow import "package:flutter/..." similar to dart:io on web, or configurable imports based on the SDK (and an ability for pub deal with that). Ending my ramblings here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type-enhancement A request for a change that isn't a bug
Projects
None yet
Development

No branches or pull requests

7 participants