[FEATURE REQUEST] Firebase Custom ML

[wazini-john]

Add capability to load custom ML Models

[bharat-biradar]

@wazini-john We have added support for firebase custom model for object detection but not published yet you can get the code from master branch, for image labelling and ink recogniser we will update them get back to you.

[wazini-john]

When can we expect it in the prod ?

[bharat-biradar]

@wazini-john Sorry I had some personal things to do. I'll add it for the image labeling by the end of the day.

[bharat-biradar]

@wazini-john Wait for the PR to be reviewed and merged.

[francislions]

@bharat-biradar I'm currently working with your library pointing to the master branch in pubspec, because i want to use the Firebase custom model.
I have two problems:

• I have tried the remote model download and the image labeler, it works for the first time, but when i try to load another model, the new model loaded doesn't work, it seems that the old model is still there like it was cached, and the image labeler gives me the same label i was receiving before
• i have tried the local model, i have saved a file under the assets directory and have made the entry in pubsec for the asset, but when i use the labeler i receive the "FileNotFoundException". I use this statement in the image labeler to load the file CustomImageLabelerOptions(customModel: CustomLocalModel.asset, customModelPath: 'identity_document_20210624_hi.tflite')

[bharat-biradar]

@francislions Did you try calling close() for the previous instance before loading another model.

[francislions]

I haven't called close(), i didn't know about it. But i restarted the app and rebuilded it sometimes, but nothing changed. I had to delete the app and rebuild to load another model.
I'm going to try with close() and i'll let you know if it works.

[francislions]

Now i called close() on the image labeler instance and it works.
Do you have more info about the case with the model bundled with the app?
Another question, i have seen the image labeler is marked to support Android, but for iOS is in 'yet' state, so in iOS the image labeler for custom model doesn't work?

[mugglmenzel]

Thanks for adding custom ML support to the google_ml_kit package.

However, the remote model manager does not work for me.
While downloading the model with the deprecated firebase_ml_custom package works, it does not work with google_ml_kit.

Following is how it works with firebase_ml_custom:

final remoteModel = FirebaseCustomRemoteModel("tflite-model");
await FirebaseModelManager.instance.download(remoteModel, FirebaseModelDownloadConditions());
File modelFile = await FirebaseModelManager.instance.getLatestModelFile(remoteModel);
await GoogleMlKit.vision
        .imageLabeler(CustomImageLabelerOptions(
            customModel: CustomLocalModel.file,
            customModelPath: modelFile.path,
            confidenceThreshold: 0))
        .processImage(InputImage.fromFilePath(imagePath));

The following does not work, suing the model manager from google_ml_kit:

await GoogleMlKit.vision.remoteModelManager().downloadModel("tflite-model", isWifiRequired: false);
await GoogleMlKit.vision
        .imageLabeler(CustomRemoteLabelerOption(
            confidenceThreshold: 0, 
            modelName: "tflite-model"))
        .processImage(InputImage.fromFilePath(imagePath));

Firebase SDK version: 27.1.0
Platform: Android

Am I missing anything or using the library incorrectly? Do I need to configure anything around firebase (custom ML) access specifically for google_ml_kit?

[bharat-biradar]

@mugglmenzel On which platform did this error occur iOS or android?

[mugglmenzel]

I am testing on Android. The behavior is that the Future<String> never returns on downloadModel().

[wazini-john]

@mugglmenzel Did it work ?

[fbernaly]

We added this feature in these PRs for both iOS and Android:

• Add firebase remote model support for image labelling #69
• Add RemoteModelManager for iOS #99

We released both in 0.7.0. Could you confirm that that is solving your request to close this issue?

[mugglmenzel]

Thanks, I tested it now.
Unfortunately, the model download throws a Platformexception now:

Unhandled Exception: PlatformException(error, com.google.mlkit.common.MlKitException: Model downloading failed due to error code: 400 from Android DownloadManager, null, null)

[mugglmenzel]

I could resolve the issue now and it's working perfectly! Thank you!

Issue seems to be around cached files. I had uninstalled the app from my qemu device and did a cold boot. After a reinstall during the flutter build, the model download worked flawlessly and so did loading a ImageLabeler.
Maybe this helps others who are migrating from FirebaseModelManager.

[janoshr]

Will other types of custom ml models be supported in the future?

The issue I am facing is that my model does not work with image/video/text data or similar and it requires just the list of processed features.

There are two popular choices apart from this package for this problem as far as I know

• tflite
• tflite_flutter

The first one provides no support for generic models, while the 2nd is no longer maintained, and the maintainer cannot be reached.

So my question is will there be a generic custom tensorflow lite model support?