Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add @capacitor-mlkit/selfie-segmentation package #66

Merged
merged 10 commits into from
Sep 11, 2023

Conversation

robingenz
Copy link
Member

@robingenz robingenz commented Sep 2, 2023

Pull request checklist

Please check if your PR fulfills the following requirements:

  • The changes have been tested successfully.
  • A changeset has been created (npm run changeset).

@trancee
Copy link
Contributor

trancee commented Sep 3, 2023

@robingenz I have just noticed when testing the implementation, that without the enableRawSizeMask, the app will crash because it has to return too many data points. If I enable the raw mask size, it returns 64KB of data points (256 x 256) which works.
But I wonder how this is helpful for segmenting the selfie picture. Looking at the samples from Google, they apply the segmentation mask to the input image and return the segmented picture. This might be more useful for this package, what do you think?

@robingenz
Copy link
Member Author

@trancee Yes, you're right. Do you already have an idea how best to implement it?
My suggestion would be the following:

/**
 * @since 5.2.0
 */
export interface ProcessImageOptions {
  /**
   * The local path to the image file.
   *
   * @since 5.2.0
   */
  path: string;
}

/**
 * @since 5.2.0
 * @see https://developers.google.com/android/reference/com/google/mlkit/vision/segmentation/SegmentationMask
 */
export interface ProcessImageResult {
  /**
   * Returns a mask that indicates the foreground and background segmentation.
   *
   * @since 5.2.0
   */
  mask: number[];
  /**
   * Returns the width of the mask.
   *
   * @since 5.2.0
   */
  width: number;
  /**
   * Returns the height of the mask.
   *
   * @since 5.2.0
   */
  height: number;
  /**
   * The path to the generated foreground image file.
   * 
   * @since 5.2.0
   */
  foregroundImagePath: string;
  /**
   * The path to the generated background image file.
   * 
   * @since 5.2.0
   */
  backgroundImagePath: string;
}

mask would always be the rawSizeMask and the images could be saved in the tmp folder and the filenames could be {filename}-foreground.{extension} and {filename}-background.{extension}.

What do you think?

@trancee
Copy link
Contributor

trancee commented Sep 3, 2023

mask would always be the rawSizeMask and the images could be saved in the tmp folder and the filenames could be {filename}-foreground.{extension} and {filename}-background.{extension}.

What do you think?

Yes, I think that's a good idea. I will try to implement it this way.

@robingenz robingenz marked this pull request as ready for review September 9, 2023 09:53
@robingenz
Copy link
Member Author

@trancee Is this PR ready for merge?

@trancee
Copy link
Contributor

trancee commented Sep 9, 2023

@trancee Is this PR ready for merge?

I will update the iOS implementation first, then it should be ready.

* add definition

* changed to enableRawSizeMask

* feat: add Android implementation

* fixed minor issues

* feat(ios): add implementation

* refactor: return selfie segmented image

* feat: added confidence threshold

* lint(android)

* cleaned up code

* feat(ios): add implementation

* feat(ios): fixed missing import

* refactoring

---------

Co-authored-by: Robin Genz <mail@robingenz.dev>
@robingenz robingenz added the feature Feature label Sep 11, 2023
@robingenz
Copy link
Member Author

@trancee Can we merge that as well?

@trancee
Copy link
Contributor

trancee commented Sep 11, 2023

@trancee Can we merge that as well?

Yes, I am done with the implementation. Good to merge!

@robingenz
Copy link
Member Author

@trancee Thank you again for your contributions! 😉

@robingenz robingenz merged commit 2e33530 into main Sep 11, 2023
@robingenz robingenz deleted the feat/selfie-segmentation branch September 11, 2023 14:44
@github-actions github-actions bot mentioned this pull request Sep 11, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 10, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants