Baal Dashboard

[Dref360]

👏 to @nitish1295 for starting the discussion on this!

The idea would be to create an app that can showcase Baal capabilities.

Right now we have the idea of:

• MLFlow to store/retrieve logs.
• A frontend that presents model performance, uncertain examples, etc (potentially in Flutter)
• A labeling interface (inside or outside the app).

My personal preference would be to separate the labeling interface from this app as in a real-case scenario labelers wouldn't look at the metrics. This would make this project more versatile as well as it can be "plugged-in" in any labeling interface on earth. I can be convinced that for a V1, we can support classification labeling in the app.

---

From @nitish1295

At this point, in my current PR #245 taking inputs from human oracle is pretty basic.

I am not sure about the direction of baal but maybe something along the following lines can help generate a lot of interest. Of course this is just a suggestion and would need significant effort. Basically a minimal UI in which we can see where we are going after each active learning, this initial tab/page of UI could start very basic with some basic data sources such as S3, Azure SQL or file system etc. from where data has to be loaded and other inputs like model,problem type and heuristic and the following can be the next tab/page where user can get into labeling via active learning.

I read your discussion about label studio but I have very limited experience with that but based on a quick look does not seem that they offer all the metrics etc while you are labeling.

If you do plan on doing something along these lines do count me in @Dref360

[nitish1295]

My personal preference would be to separate the labeling interface from this app as in a real-case scenario labelers wouldn't look at the metrics. This would make this project more versatile as well as it can be "plugged-in" in any labeling interface on earth.

The point about making a labelling interface which can be "plugged-in" anywhere does sound useful. Looking for some clarity on this:

When you talk about decoupling/separating labelling interface from the application does that mean we pick up something pre-built like labelling studio as our labelling interface and do things similar to what you have refrenced in #105 OR do we build a custom interface which can be "plugged-in" anywhere.

Also now that you have mentioned a decoupling of the labelling interface. I am thinking of this app as 2 different components:

• A labelling interface - This can be built seprately/or we can use exisiting solutions. But this interface can be embedded/plugged in anywhere. Even in our own main app which we create
• Main App - Main app can have metrics etc. and the capability so that we can plug in/embed our own labelling interface which we mentioned above

That way we get a labelling interface which can be used anywhere along with baal + an app which can run baal,metrics,labelling based on the interface we designed(something native to baal). Then based on the requirements people can decide whether to use the app or the labelling interface only

A suggestion based on what I have mentioned above:

Maybe we can look more into Flutter but seems like label studio does offer flexibility to integrate with other frameworks. Refer https://labelstud.io/guide/frontend.html. We just embed labelling interface from labelling studio into our app

The 🍔 button on top right can let you choose metrics + data etc. if you want them and that way we might not have to build the metrics in the first iteration itself :

If this work then we might not have to re-invent the wheel for the labelling interface and can just use Label Studio and embed it in our main app.

Some other things to ponder over. I know these are questions far beyond the scope of what we have now but if we have answers to some of them we might be able to design the app in a way so that these can be accommodated in the future if needed

• We might not need it right now but how do you feel about Data version control along with ML flow. Maybe also add DVC to the mix so that people can keep track of where there current train and pool are in different experiments and reterive and keep track of them easily
• Do you think you would ever reach a point where you might need people to do AL over their phones in an app. Of course the backend for AL will run somehere on cloud and the app will send API calls
• Do you think you want to allow people to run the AL "backend" on a cloud instance such as GCP/Azure/AWS while the fronted runs on their system and again the fronted basically communicate based on API calls

[Dref360]

1. I'm also looking at DVC for my current job! That would be ideal if we can "restart" an experiment from a checkpoint, with a new heuristic.
   1. For a v1, I think we can trust Baal logging, it keeps track of what was labelled and when.
2. Maybe practitioners would like to see the metrics from the app, but labelling from mobile seems a bit farfetched. So I would use this app as a monitoring tool. From my experience, we reviewed metrics with the labelling team once a week.
3. Yes that would be ideal, if we use MLFlow to store metrics, we can easily query the MLFlow database from the app.

I'm quite busy till next week (end of the year rush and all that), but I will have a system design by Friday next week. And we can get started after New Year.

[nitish1295]

@Dref360 Let me know when can we start working on this

[Dref360]

Hello!

I did a bit of digging over the holidays and I think we can use the following workflow:

1. Users launch a labelling task with Baal and log metrics to their MLFlow server (with a Run ID)
2. Baal Dashboard can be configured at launch to connect to the MLFlow server.
3. A webapp or an app can then connect to Baal Dashboard and see metrics for a specific Run ID.

What I understand from MLFlow is that:

1. We can log any metrics (float) pretty easily
2. We can log other things (array, objects) using JSON files.

IMO the next steps should be:

1. Make a proposal on what to log from Baal to MLFlow (metrics, uncertainty, what else?)
2. Create a repo in Baal-org and add you as a contributor.
3. Create a set of fake MLFlow runs
4. Add logging to MLFlow to Baal from the specs.

[nitish1295]

Thanks for making this!

Based on what you have shown above, I am assuming the baal dashboard server back-end can run based on a REST API which will receive requests from the frontend. So on each request something along the the following will execute:

model.train()

model.eval()

mc_dropout()

get_uncertanity()

send_request_frontend() # API call

update_baal_data()

Let me know if this is how you plan to implement this.

Also we can definitely move this to a separate repo but before that you might want to look at #250

[Dref360]

Hello!

I created the first step of the repo. Very simple, but it is the most I can do this week.

We can use fake data for now.

Loom: loom.com/share/c9904efc97894af8b5783a8a2535566e

Once we get metrics to the frontend, we can start charting with d3.js or Apex Chart.

I gave you (@nitish1295) and @parmidaatg write access to that repo.

baal-org/baal-dashboard

Let's use Loom as an async communication tool.