Overall Questions

[ryankauk]

Hey there, I think this project is great just have a few questions for clarifications and remarks.

1. I'm new to react now so I'm wondering why the react reconciler is necessary for this project?

2. I'm just wondering why this package uses react-spring, babylonjs has animations built in. Do the animations not work well with react? I just think since babylonjs is already a heavy package, the necessities to get it working with react should be as minimal as possible. Maybe if using react-spring it would be a plugin to react babylonjs or a submodule of it.

Opinions and Remarks

1. I think the canvas should be decoupled from the engine, I'd like to use Engine and scene basically as a provider and place the canvas where I would like so that I can still grab context to the engine and scene when outside of canvas. As well, I'm not able to put html markup inside child of Engine, Scene etc I'm assuming this is because of react reconciler? That means I have two completely separate trees, one for html and one for babylonjs, which would become cumbersome for me since I can't inject pages from nextjs.

For example:
I'm using nextjs and would like to put the scene and engine in my _app.tsx but only want the canvas a certain width and placing html content into my pages as well as the markup inside to trigger objects.

Let me know your thoughts! :)

[brianzinn]

1. The react-reconciler originally wasn't a direct dependency - when the project first started it was using reactDOM renderer (that uses react-reconciler) and building a canvas and everything in a much different way. Using react-reconciler lets us easily discover the component hierarchy and manage scene objects automatically. This also allows us also to define our own host components - like <box ../> with lowercase even though it looks like DOM.

2. That is a good question. It was added this week and perhaps ought to be a separate library. If you are not using it then you can ignore the peer dependency warnings I thought. The react-spring hooks are very powerful and I haven't had any time to make anything equivalent using the built-in animations. I use the built-in BabylonJS animations for my own projects imperatively. I will update my sample projects to make sure it is kept as minimal as possible and consider your idea of plugin/submodule.

--

1. The canvas is decoupled from the Engine to some extent, but would need more testing. In react there is a concept of a portal, where you can render outside the DOM of the parent hierarchy. I did some experimenting a while back and you can pass in an HTML canvas using the Engine prop portalCanvas.
   https://github.com/brianzinn/react-babylonjs/blob/master/src/Engine.tsx#L46
   There is definitely some work needed there, but you are the first person to ask for it, if that is what you are after.
   Additionally, you can pass the following canvas props through from Engine: 'touch-action', height, width, id, style. That allows you to define a custom canvas width or use CSS also to define canvas properties (ie: '#your-canvas-id: {width: '?'}').

[ryankauk]

Great thanks so much for the info!

I'll try playing around with these parameters and look more into react-dom and react-reconciler and play around with them to see if I can find a way to connect nextjs with this. I think it will be a powerful combo.

[keverw]

I've had some questions too, just been watching the project though but planning to start using it for an idea I had.

1. authoritative servers - you can use nullEngine and Physics, be nice to be able to declaratively do things like React on the server side, but I got a feeling had to manually do that with BJS directly. However since server only really cares about where objects and players are for physics, not like the server version needs to have colors, click binding and stuff set so that saves a bit of code since the nullEngine one would just to be handing the physics and not playable on the server. Client would have the richer fuller experience.

2. Then the babylon GUI kinda disappoints me since used to rich HTML and CSS. I know on Desktop and mobile a html/css overlay is prefect. but on VR looks like that hasn't been supported much until recently but still bleeding edge. I don't have anything to test it with but I'm hoping to get my hands on a Oculus Quest but always out of stock, but being able to support that at some point would be cool. https://github.com/immersive-web/dom-overlays/blob/master/explainer.md, https://www.chromestatus.com/feature/6048666307526656 and https://immersive-web.github.io/dom-overlays/

   It sounds like you'd put the overlay in a certain div, and what's transparent would show the world, then it would blend it but have to pass the info into the WebXR context.

   On a large-FoV VR headset, the overlay may appear as a rectangle floating in space that's kept in front of the user, but isn't necessarily strictly head-locked (in order to improve comfort). This rectangle may be smaller than the maximum FoV if necessary to ensure that the corners and edges remain easily visible.

[brianzinn]

@keverw the BabylonJS GUI is not as complete as HTML/CSS! The DOM overlays will be great when they are available. With Babylon GUI you can project onto a plane as a texture. This works in VR/XR. This GUI is on a plane and can be placed anywhere in the scene (in this example, I keep moving the plane to be in front of the camera) - give it a try in VR with your controls:
https://brianzinn.github.io/react-babylonjs/?path=/story/gui--with-2-dui

[keverw]

@brianzinn Yeah I seen that demo on the computer, can't wait to try VR though in general once I get my hardware. Totally agree not as complete as HTML/CSS, but with the overlay thing should help since less of a learning curve. I think the overlay for WebXR is available now but behind a flag (For Android, but since Oculus Quest uses Android under the hood not sure if it made it to that yet - looking forward to getting one myself to learn more about VR and give a try at writing some code for it) but still bleeding edge since introduced March this year. So overtime hopefully more stable and can be supported with react-babylonjs, not sure if it'd require much changes to babylonjs as it sounded like just passing in options but unsure...

I guess wanting to use this for the server side probably wouldn't be very easy? Probably requires the dom to use it with react-babylonjs and nullEngine? Would be cool though but I'm guessing most likely on server will just use BJS directly since limited things would be tracked on the server and manually adding/removing stuff.

[ryankauk]

@brianzinn So I've done some reading with the react-reconciler and react-dom and have a few notes for discussion.

1. I personally think that the engine and scene should be disconnected from the reconciler. Personally I wouldn't mind imperatively defining my scenes in regular react-dom while just using this package for injection of the engine, scene setup, and react hooks, making the whole custom declarative renderer optional this could give the package traffic while allowing custom usage.

2. I think if using react-reconciler react-babylonjs should provide the whole interface, including the createPortal. This way the user can choose where to place it. I think a good place for an example is react-three-fiber

3. I think for dev purposes as well the devdependencies should be shaved down and ported to their respective "dev tools". I forked the repo and was going to start moving some things around but found it has a heavy setup cost. Mainly because there's a lot packages for things like the stories and build process. Could possibly move them into their own packages just nested inside this package so that way their dependencies only have to be installed when using them. react-babylonjs/stories/package.json and react-babylonjs/tools/package.json.

I think overall this package has everything it needs, just some separation is needed. I think for this aspect react-three-fiber would be a good place to base the user facing api structure on.

[ryankauk]

I did some adjustment to separate the engine and scene and I can now do something like

export const RootScene = ({ children }) => {
    const id = 'portal-canvas';


    return (
        <Engine portalCanvas={id}>
            <Scene onSceneMount={({ scene }) => {
                console.log('on scene mount')

                scene.clearColor = new Color4(0, 0, 0, 1)
                // This creates and positions a free camera(non - mesh)

                var camera = new ArcRotateCamera("camera12", 0, 0, 5, Vector3.Zero(), scene);
                camera.minZ = 0;
                camera.maxZ = 50;
                camera.lowerRadiusLimit = 2;
                camera.upperRadiusLimit = 50;

                // This targets the camera to scene origin
                camera.setTarget(Vector3.Zero());

                camera.position = new Vector3(0, 0, 22);

                camera.wheelPrecision = 50;


            }}>
                <canvas id={id}></canvas>
                {children}
            </Scene>
        </Engine>

    );
}

[brianzinn]

One thing this library wants to do is abstract away enough what is going on that you wouldn't need to create your own DOM canvas element. The Engine flows through typical properties to the auto-created canvas element. You could do the same currently like this:

export const RootScene = ({ children }) => {
    const id = 'portal-canvas';

    return (
        <Engine canvasId={id}>
            <Scene onSceneMount={({ scene }) => {
                console.log('on scene mount')
                // rest here
            }}>
                {children}
            </Scene>
        </Engine>
    );
}

The HostConfig renderer for the react-reconciler is separate from the Scene and Engine components:
https://github.com/brianzinn/react-babylonjs/blob/master/src/ReactBabylonJSHostConfig.ts

The Scene is a functional component and creates the renderer, which is what renders the children. It will not render regular reactDOM elements like div. You can still create and compose your children components in there - using babylonjs JSX intrinsic elements (like <box ../>, etc.) as host components, functional components or React.Components or any combination thereof. Here is where that happens:
https://github.com/brianzinn/react-babylonjs/blob/master/src/Scene.tsx#L134

I do think there is always room for improvement and if I can figure out what is missing or how to make a cleaner separation then I will definitely welcome changes. I've shared with you there the key places. Your fork is even with master, so I couldn't see the changes/improvements you made.

I like your idea to split the project to multiple package.json - it will reduce a lot the regular dev dependencies, so will investigate that. Your original comment about react-spring is a good one as well -- already hookex wanted to split as a separate project, but I wanted to try out in main repo first.

[ryankauk]

For me I have a custom layout so that they're siblings in html, where as if using the package the child you pass to engine are direct children to the canvas since.

Sorry when I was referring to the reconciler I meant removing the reconciler creation from scene ie.

react-babylonjs/src/Scene.tsx

Line 134 in aa26d55

const renderer = ReactReconciler(ReactBabylonJSHostConfig);

Possibly could move it to a provider component or just providing an option to disable reconciliation. This way this package is still usable on a custom level if someone can't use the reconciler like in my case

The issue here is that I'm using next.js which internally uses ReactDOM and when using reacts context they're scoped per reconciler, and from what I can, there doesn't seem to be a solution yet to share context. In this sense I have no option but to use reactDOM so that next.js can do it's thing. I also would still like the normal html components because of seo purposes.

[brianzinn]

I know what you mean now hopefully now that you mention the context!! It needs something like a bridge to bring the context over.
Do you mean like this issue for konva?
konvajs/react-konva#188

[ReinierRothuis]

Hi @brianzinn , great library, we have been using it for a few weeks now. We are encountering the issue @ryankauk is referring to regarding the contexts as well. Was about to post a separate issue on it, but it seems they are related. (If you want I can create a separate issue?).

We are using some graphql courtesy of the apollo graphql client and react hooks to fetch some data determining the scenes. It seems like the Apollo (Context) Provider doesn't provide it's context to the components within the Engine. I think this is because the context is scoped per reconciler, as @ryankauk was mentioning in his previous comment, losing things like cache. So a bridge of some sort seems like a welcome new feature. If you'd point me in the right direction I can develop this myself but I haven't a clue where to start. Or if there is a way that we can use only one react reconciler, that would also be a solution.

[brianzinn]

@ReinierRothuis groetjes! You won't be able to use one react-reconciler, because ReactDOM will be used for your regular website, while the canvas and BabylonJS scene are rendered with a separate one that understands the BabylonJS specific elements like camera, lights, geometries, etc. Here is a working example (modified from react-konva issue) - I will add this as a storybook playground and to readme this week:

const ThemeContext = React.createContext({
    color: Color3.Red(),
    position: Vector3.Zero(),
    name: 'default context'
});

const ThemedBox = () => {
    const ctx = React.useContext(ThemeContext);
    return (
      <box name={ctx.name} position={ctx.position}>
          <standardMaterial diffuseColor={ctx.color} specularColor={Color3.Black()} />
      </box>
    );
  };

const EngineScene = () => (
  <div style={{ flex: 1, display: 'flex' }}>
    <ThemeContext.Consumer>
    {value => (
        <Engine antialias adaptToDeviceRatio canvasId='babylonJS' >
            <Scene>
                <ThemeContext.Provider value={value}>
                    <arcRotateCamera name="arc" target={Vector3.Zero()} minZ={0.001}
                        alpha={-Math.PI / 4} beta={(Math.PI / 4)} radius={5 }
                    />
                    <hemisphericLight name='light1' intensity={0.7} direction={Vector3.Up()} />
                    <ground name='ground1' width={6} height={6} subdivisions={2} />
                    <ThemedBox />
                </ThemeContext.Provider>
            </Scene>
        </Engine>
    )}
    </ThemeContext.Consumer>
  </div>
)

export const BridgedContext = () => (
    <ThemeContext.Provider value={{
        color: Color3.FromHexString('#FFAF00'),
        position: new Vector3(0, 1, 0),
        name: 'testing'
    }}>
      <EngineScene />
    </ThemeContext.Provider>
);

[ReinierRothuis]

Thanks a lot! We worked it out with your tips. We didn't need to build a bridge, we could just instantiate another ApolloProvider inside of the engine using the method above and bridge the client over that way.

With apollo it goes like this:

<ApolloConsumer>
  {(client): ReactElement => (
    <Engine>
          <Scene>
            <ApolloProvider client={client}>
                   <RestOfYourComponents/>
            </ApolloProvider>
          </Scene>
        </Engine>
      )}
</ApolloConsumer>