Reinforcement learning

[jarnmo]

I've looked into the available documentation and examples, but haven't been able to figure out if it is possible to use the ML.NET in its current state for (non-deep) reinforcement learning. If it is possible, I'd be thankful for any hints on how to implement a simple case. In case reinforcement learning is not possible atm, what exactly is missing and are there any plans on implementing the missing pieces?

Thanks!

[GalOshri]

Reinforcement learning is not yet available in ML.NET. One critical component that is currently missing is "exploration" (trying out different actions to see how the reward changes) and how this affects data collection and model training. This is something that would be interesting to explore in the future.

What is the scenario you have in mind?

[jarnmo]

I see. My scenario is basically to use it as a part of building a game AI.

[GalOshri]

Got it. A few more questions: what is the complexity of the game? How many actions does the AI have (the number of choices available to it at each step)? Does the list of actions change throughout the course of the game or is it static?

[jarnmo]

Well, what I'm actually trying to do, is to build a framework which would be used to write game rules (gameplay code) and would make it as easy as possible to add AI actors as well. In the end anything but a trivial AI would probably have to be build of several components, some ML and some utilizing more traditional features of the framework. The end goal is to allow building games of any practical complexity, but no 'academic restrictions' apply here. RL can only be a piece in the AI puzzle and for example 'goal hacking' is certainly an option if it makes the AI behave in a desirable way :).

While a game AI can be extremely complex, I'm hoping to find ways to split the problem domains into smaller pieces that could then be handled by individual 'RL pipelines'. Maybe even just learn to estimate the value of taking each action in a given situation individually. I think the Microsoft's Mrs PacMan AI used this kind of an approach.

It's certainly common that the list of possible actions an actor can take changes throughout the course of a game, but if the value of each action (for each target) would be evaluated by it's own NN I guess the changes would be a matter of updating the list of evaluated NNs (and whatever logic coordinates between them).

This is all very much just an idea atm, but I guess even a very simple RL implementation would allow meaningful experimentation. I might look into CNTK and their C# bindings now, but as I have no practical experience from ML I wouldn't mind something a bit easier to approach..

[axmand]

To Implement RL such as DQN, I need to train neural network after an observation.
however, i don't know how to split train data to train the neural network by 'TextLoader'.
Is there any way to gradually input training data?
//
well, i found this in:

machinelearning/test/Microsoft.ML.Tests/CollectionDataSourceTests.cs

Line 29 in f6c6f5b

Assert.NotNull(CollectionDataSource.Create(new List<Input>() { new Input { Number1 = 1, String1 = "1" } }));

[ghost]

Any news about some RL support?

[codemzs]

CC: @wschin

[chris3713]

Would be nice too see this implemented some time.

[codemzs]

We don't have RL support in the short-term but it is something we are considering for long-term.

[InDieTasten]

Was the closing of this issue intended? How is this tracked from now on?

[codemzs]

Our short term goals are around ONNX, DNN training and AutoML. RL is something we have considered and is on our list but just not something we plan to work on this year. Feel free to send a PR on RL and we will happily review.

[InDieTasten]

I understand this is not a priority right now. I would just like to get updates to for this particular feature.

Is there another issue for tracking progress in this area, or how do I get updates to things "on your list"? Where is "your list", if it's not the GitHub issues?

[codemzs]

You will get updates when this issue will be reopened. Please read our roadmap and feel free to ask questions on gitter regarding specific feature updates . Thank you for your interest.

[wandrzej]

Our short term goals are around ONNX, DNN training and AutoML. RL is something we have considered and is on our list but just not something we plan to work on this year. Feel free to send a PR on RL and we will happily review.

@codemzs
I was wondering how this fits with the RL work that Unity is doing. They have their ML-agents, which uses TF for the training part, but for inference they developed Barracuda which is based on DirectML and allows use of ONNX models. Since Unity uses C# this looks like a pretty obvious case for using ML.NET with Unity.
What is currently missing in ML.NET to implement some general model like PPO?

[david-uk-hash]

@codemzs any update on RL?

[codemzs]

@david-uk-hash Thanks for checking back. Unfortunately, we do not plan to support RL in the near future. This is an open source project so we encourage and welcome contributions.

[chunky]

Is RL on your roadmap at all? I have a model developed in VB.NET that I currently use with Python for RL via some unpleasant hijinks; being able to stay within a single process would have some really significant benefits.

[david-uk-hash]

Even a fully featured Neural Network API in C#, will make our life's easier as PPO or Neat can be implemented easily.

[halheinrich]

My interest is in evaluating different variants of backgammon and assigning skill rating to each - with the goal of finding the variant that most rewards skill.

Skill level 0 is random moves.
Skill level 1 beats level 0 75% of the time.
Skill level 2 beats level 1 75% of the time.
...

[johnwc]

We're looking to use it for climate control in small warehouse type building. Having large fans that pull heat out, with ducts bringing outside air in. Need the RL to read inside & outside temp/humidity along with fan's speed & CFM, to maintain temp & humidity inside.

[michaelgsharp]

@luisquintanilla

[MiheevN]

Reinforcement learning required for an MMO game already being developed in pure .Net (no Unity)

Creation of the intellect of characters and different machines.
Thought she was already there. But it turned out we had to wait.

[michaelgsharp]

@luisquintanilla any thoughts on this?

[torronen]

@luisquintanilla Any ideas reinforcement learning will be, might be, or probably not is on the ML.NET 3.0 roadmap?

[luisquintanilla]

@torronen Not for 3.0. We're still learning about this scenario.

[jez9999]

@torronen Not for 3.0. We're still learning about this scenario.

This would seem to contradict the following bug: #5918

Which says that there most certainly is a plan being followed for deep learning support. Does it include reinforcement learning?

[luisquintanilla]

@jez9999 Could you clarify how #5918 contradicts the statement in this issue?

There is a plan for deep learning which we're actively working on. We're still learning about reinforcement learning and not planning on implementing anything during the ML.NET 3.0 timeframe (Nov 2023). So to answer your second question - our current plan for deep learning doesn't include reinforcement learning.

[jez9999]

OK, thought that integration with TorchSharp/pyTorch implied reinforcement learning because it seems you can do it with pyTorch.

[jez9999]

And by the way, you should have reinforcement learning high on the agenda. A common use for machine learning is building game AIs, which I want to do, so I totally need it.

[Bonifatius94]

@luisquintanilla @jez9999 RL isn't only for game AI development, but also for any kind of automation that involves solving a Markov Decision Process. For example, I'm currently researching on training a self-driving e-scooter with the PPO algorithm regarding autonomous driving. But there are various other applications like continuous power plant / factory control. Also in robotics, there's a huge research field on autonomous agents for motion control and decision making. Or let's say you wanna develop any kind of personal assistant that helps to be more productive at work. There will be plenty of demand for RL training.

IMO, behavioral AI systems for decision making are the next step, especially in combination with ChatGPT-empowered expert systems. Every big company will start to enhance their workers' boring repetitive tasks with AI systems within the next 5-10 years. So, there should be a huge market for a well-performing training framework that allows to train a lot faster than this very slow Python crap due to the support for hardware intrinsics, etc. Just think about it for a moment. You've got a great opportunity.

[decriptor]

@luisquintanilla does that mean that most of #5918 should be completed in ML.NET 3.0?

[Ceeeeed]

I'm also interested in RL. This needs to be in ML.NET.

[GeorgeS2019]

@michaelgsharp

@torronen Not for 3.0. We're still learning about this scenario.

Sept 2023 Update for .NET Deep RL

Do add the deep learning label to this issue.

We are in 2023, this project has ONLY 3 issues labeled with deep learning :-)

[sirhamsteralot]

Chipping in to say that I too am currently in the progress of planning out an AI for a game and would like to utilise ML.Net for Reinforcement learning. The inclusion would be mostly on the decision making front of the game AI for me however robotics/self driving rc cars and the like would also be applicable usecases.

Are there any updates? I see that it still has the label future, considering this seems like one of the more active topics on the roadmap I hope this can be picked up relatively soon

[Flem100]

I have been wanting to use ML.Net with Unity for a long time. I want to use it instead of https://github.com/Unity-Technologies/ml-agents, which I don't like to use.
I got ML.net 3.0 to work with Unity and then I learn that ML.net does not have reinforcement learning.
My current goal is to do the same as in this video:
https://www.youtube.com/watch?v=L_4BPjLBF4E
using ML.net in Unity.

[GeorgeS2019]

First, this is OUTSIDE the scope of Microsoft ML.NET team.

Second, this topic has been discussed in the Godot communities, those involved in Godot reinforcement learning, using c# with or without TorchSharp.

The video requires 3D Godot environment requirements which is outside the scope and of interest of ML.NET team, as well as outside the team working on TorchSharp.

[InDieTasten]

@GeorgeS2019 Nobody expects ML.NET to feature a 3D environment for agent training.

What we want is for ML.NET to provide an interface to generate neural networks so we can run a fitness function on them and return that fitness value back to ML.NET to adjust the model. There are obviously many different algorithms that could be used under the hood.

With that kind of interface, it would be easy to use any simulation environment, no matter if it's Unity, Godot, or something else entirely. We can spawn agents into environments or entire environments entirely as part of the fitness function.

[Flem100]

@GeorgeS2019 I hope the ML.NET people have a different perspective. I would not think it's in Microsoft's best interest to send people Godot's way instead of Unity, because it's so heavily connected to .NET, which is one of the main reasons I prefer it over anything else. Everything I create in .NET, more less, natively works in Unity. Niiice :)

[GeorgeS2019]

The video suggests a solution similar to Unity for achieving the RL to deliver similar solution in c# beyond the Unity environment.

Unity RL is based on python, which means currently no satisfying RL framework for c# unless very serious effort being invested.

[Flem100]

@GeorgeS2019 Sadly yes, you are are right. But maybe this framework is what we need, at least for now:
https://github.com/asieradzk/RL_Matrix
He is actually going to have a Community standup tomorrow: https://www.youtube.com/watch?v=Aoyj7qzyung
He is going to talk about using his framework in Godot, but I just emailed him and asked about Unity and he responded that he will include an example for Unity as well :D

[GeorgeS2019]

@Flem100

Try using the RL_Matrix. I assume you have not tried Unity RL and know what is the learning experience and why there are so many Unity RL projects in github and why Unity is used in so many e.g. Master projects.

For c#, we need a solution, which will help the transition and easy adoption of an RL framework from people who have used the Unity RL.

If possible, it is feasible to port Unity RL to e.g. Godot RL.

The implementation of RL_Matrix, those seem the best, the design for future choice is to be discussed.

[GeorgeS2019]

@InDieTasten

for ML.NET to provide an interface to generate neural networks so we can run a fitness function on them and return that fitness value back to ML.NET to adjust the model.

Before we can get there, we need someone with expertise on how e.g. pytorch address that.

Is any of the Pytorch-based RL frameworks out there addressing that using an interface as you wish?

[asieradzk]

For c#, we need a solution, which will help the transition and easy adoption of an RL framework from people who have used the Unity RL.

We can use all of the Unity's ml-agents sensor components with RLMatrix, all you need to do is use the toolkit. I wrote a wrapper converting ray sensor component and its only couple dozen lines. If there's demand I'll convert all Unity's ml-agents sensor components - they are great one of my favourite things that came out of ml-agents.

https://x.com/adisiera/status/1839015234742210779

in this vid its the same sensor component and environment that was used with ml-agents before.