Request for raycasting

[jaspercayne]

Being able to raycast from the agent's eye position at a block would be a large benefit for being able to detect what the agent can actually "see", as opposed to the observation grid which allows the agent to see blocks that might be obscured. As an even better improvement, being able to return information about the block hit by the ray would provide an easy way to gather information about the world from a distance without needing an obscenely large observation grid. Even better if the raycast could also return information about mobs that have been hit as well, allowing the agent to react to mobs before they are basically on top of it; which would be of particular use for skeletons whose ranged attack might be able to strike the agent from outside the observation grid, leaving them essentially defenseless.

[timhutton]

Yes, this fits in with a few conversations we've had over the last few days - people want to avoid the vision problem and don't want to have to use the grid observation with the player's yaw. But one thing I can't see yet: what sort of format would you want the data returned in?

[jaspercayne]

Maybe JSON, since that seems to be what most things are when dealing with
potentially large/different types of data. Something to the effect of
{ XYZ, block name, metadata}
Should be sufficient in my opinion, but if the same raycast could also
detect dropped item entities, there may also need to be a int for how many
items are in the stack. For mobs, maybe
{XYZ, mob type, facing}
I don't know if it would be possible, but maybe also a boolean state if the
mob is aggressive or not.

On Wed, Jul 13, 2016, 10:57 PM Tim Hutton notifications@github.com wrote:

Yes, this fits in with a few conversations we've had over the last few
days - people want to avoid the vision problem and don't want to have to
use the grid observation with the player's yaw. But one thing I can't see
yet: what sort of format would you want the data returned in?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#184 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AGHiBUTIn3lzaRWJzFo7w3l33srkyAL3ks5qVaWTgaJpZM4JLndD
.

[DaveyBiggers]

Minecraft itself keeps track of the object (block or entity) that is directly in the line of sight (eg the block that gets the black highlight). We can add this as an observation (indeed, we used to do this in an earlier version).

[jaspercayne]

That would be fantastic! That would solve the issue of being able to see
blocks that it shouldn't see completely. It might take some fine tuning on
our parts to get the agent to look directly at what we want, but that is
our problem, not yours. Would that include information about mobs as well,
or would that need to be a different thing?

On Thu, Jul 14, 2016, 6:16 AM David Bignell notifications@github.com
wrote:

Minecraft itself keeps track of the object (block or entity) that is
directly in the line of sight (eg the block that gets the black highlight).
We can add this as an observation (indeed, we used to do this in an earlier
version).

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#184 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AGHiBd89VJo1wwglhmxSySlUA3JaHVMfks5qVgxjgaJpZM4JLndD
.

[DaveyBiggers]

It includes mobs, though (if memory serves) it doesn't include floating items. Hopefully we can do something about that though.

[jaspercayne]

All things in time. Just being able to identify threats and keep away from
them/defend itself is a massive leap forward in the agent's ability to
survive.

On Thu, Jul 14, 2016, 10:05 AM David Bignell notifications@github.com
wrote:

It includes mobs, though (if memory serves) it doesn't include floating
items. Hopefully we can do something about that though.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#184 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AGHiBTc0mbZA6g3JwGsZ1SAND6VP06jRks5qVkIqgaJpZM4JLndD
.

[brunodalarosa]

I think this will be really useful and i'm looking forward to it! News about when this'll be released?
Also, you guys from MS don't see it as a important feature to Malmo? I mean, i read somewhere that the project used to have something like that but now it haven't. Why it have been discontinued?

[DaveyBiggers]

@bcesarg6 it just fell through the cracks during some refactoring - it's always been on the list to get reinstated. As you can see, we added this issue to the Camel milestone, so the intention was to have it done by Camel's due-date. Hopefully we're still on track for that, though it might end up getting descoped to Dolphin if we run out of time.

[DaveyBiggers]

Have added ObservationFromRay - it will return a json object called "LineOfSight" - see the XSD for details. At the moment it has a range of 50 blocks, directly along the player's line of sight. We can easily extend this to allow users to specify the range / angle themselves, and to request multiple angles. (So you can have eyes in the back of your head, for example...) For the time being, I think this default behaviour is the most useful.

[jaspercayne]

Would this be able to detect something smaller than a single block; say
maybe a torch, while you are aiming at the block space, or would you need
to have the ray actually collide with the hitbox of the torch? Also could I
request the ability to feed it a boolean to allow raycasting through
transparent blocks? Otherwise the agent basically has the same visual
capabilities of a creeper, which could get very messy...

On Mon, Jul 25, 2016, 10:34 AM David Bignell notifications@github.com
wrote:

Have added ObservationFromRay - it will return a json object called
"LineOfSight" - see the XSD for details. At the moment it has a range of 50
blocks, directly along the player's line of sight. We can easily extend
this to allow users to specify the range / angle themselves, and to request
multiple angles. (So you can have eyes in the back of your head, for
example...) For the time being, I think this default behaviour is the most
useful.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#184 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGHiBct9VEA9oeTQSKMHlcbga5wjMZr1ks5qZMmFgaJpZM4JLndD
.

[DaveyBiggers]

It should work with torches, and small things like flowers etc, but yes, the ray has to collide with the hitbox. It's essentially the same code Minecraft uses to determine what the player will interact with when they attack/use. Not sure about transparent blocks, will have to do some investigation.