Interactivity in IJulia, resp. WebGL-support

[oschulz]

Are there any plans to support interactivity and things like rotating 3D-plots in IJulia? I guess WebGL would be they way to go, but I don't know about the state of WebGL-support in GLVisualize.

I think this would be a very important feature - more and more, people will run computations on large (and powerful) remote machines - and while there are solutions for remote-display OpenGL, they all require quite a bit of machinery on both server and client side.

[mkborregaard]

PlotlyJS is interactive in IJulia (and Atom), and is based on WebGL

[oschulz]

PlotlyJS is interactive in IJulia (and Atom), and is based on WebGL

True - I was refering to GLVisualize-based MakiE-plotting. :-)

PlotlyJS isn't really usable for plots with lot's of data points though, it get's very slow. I think WebGL can do much better, in principle, but we have little influence on how plotly.js uses it - I guess plots with a lot of data is just not a priority for them. I have a feeling plotly.js is more focused on "shiny" plots than scatter-plots with n*10^6 points or histograms with 64000 bins (and I need those).

Also, I'm hoping for things like data-interaction in the future (e.g. select some data in the plot, and trigger some action, e.g. a new, more detailed plot, and things like that).

[SimonDanisch]

How is using WebGL solving the issue that you describe? It seems orthogonal to visualizing data on remote machines. WebGL is really just OpenGL running in a browser on the clients machine - so it's just strictly needed when you want to put visualizations on a website. I guess you mention WebGL since it feels like it solves the issue - while in fact it runs on your laptop and most frameworks that use WebGL already have some infrastructure to stream data from the server. What we really just need to satisfy this use case with the current MakiE + OpenGL, is ability to serialize signals and send the data via some streaming protocol to the client. A first prototype should be relatively straight forward to implement.
I wonder if @timholy already has some tools for that :)

[oschulz]

It seems orthogonal to visualizing data on remote machines

I wouldn't call it orthogonal - i'd say it's a great way for speedy, graphics rich visualization of data without any software dependencies on the client side (except for a modern web-browser, of course). Of course it's just OpenGL (or a subset of it?), but to my knowledge not all OpenGL frameworks do allow for WebGL output out of the box.

WebGL is really just OpenGL running in a browser on the clients machine

I know - but still, a MakiE backend like GLVisualize would need to support WebGL output, right? Especially for smooth integration with IJulia?

via some streaming protocol to the client

Shouldn't Jupyter/IJulia take care of data transport between server and client? Or am I missing something here?

[SimonDanisch]

Shouldn't Jupyter/IJulia take care of data transport between server and client? Or am I missing something here?

Well, I was talking about a non web based approach ;) That's what I meant, that you're conflating the need for WebGL and enabling your use case.
I probably didn't formulate it well, but I meant there is a way of doing all this smoothly, and possibly much more performant, with a julia desktop application, not using any web framework.

WebGL would still be nice, but I think for your use case it almost has no advantages besides that you only need a browser... Which is obviously pretty cool!
But also hard to make it performant and one would need to write large parts in javascript. It's actually surprisingly complicated to do all the bridging between julia, javascript, the server, the client, the browser etc... So for now I'd like to rely on e.g. plotly for use cases that really need to be web based, and just have a way for streaming data from a server to a local opengl application (Makie) for your use case!

[timholy]

Is MakiE set up for remote operation? Or is VirtualGL necessary? JuliaGL/GLVisualize.jl#146

[timholy]

Ah, I see you're wondering about streaming. No, we haven't set that up yet.

[oschulz]

Well, I was talking about a non web based approach ;)
That's what I meant, that you're conflating the need for WebGL and enabling your use case.

Oh, sure, remote accelerated visualization can be (and is) done without WebGL - but like I said, it usually requires quite a bit of machinery on both sides, that's often not so easy to set up for the user.

And IJulia is very popular - and with JupyterLab it's becoming a fully featured remote working environment sufficient for many applications.

Also, with IJulia one can make fully interactive presentiations (slides) via RISE/Reveal.js - and MakiE graphics in them would be awesome.

[mkborregaard]

But @oschulz the whole point of MakiE is to work towards replacing Plots, at which point it would just be a question of passing the plot to the plotlyjs backend.

[SimonDanisch]

@oschulz I know all of that... But note, that with IJulia and a browser it's also remote, rendering is still happening on the client side, so you might as well do the rendering in Julia - and in Julia you could avoid the main bottleneck of plotly, namely the messed up json communication ;)

And of course, a solution would only be nice if it's as easy as Pkg.add("MakiE") - which I hope we can do.
And actually, after helping to teach a course to 150 people using Plotly, IJulia and GLVisualize (which is the most problematic dependency of MakiE) - GLVisualize was one of the easier packages to set up for the students (who all weren't computer science students).

As I said, it's pretty difficult to get performance out of IJulia + WebGL remote rendering especially because of the communication - it's probably also the reason why you perceive plotly as slow.
I'd be surprised if the performance is actually because they messed up the WebGL rendering.

But overall, MakiE is still mainly a one man job - I hope this will change, but until then I have to really prioritize what to work on.
So, all the nice arguments you're bringing are not very meaningful if I don't have the time and knowledge to make it work.

And I feel much more inclined and able to make fast remote rendering happening with the streaming tools we already have handy in Julia instead of diving into javascript programming and creating tools that can not be developed further as easily by Julia programmers. I think we could make a first nice prototype in roughly one day.

Please also note, that MakiE already works in IJulia - just not with WebGL yet.

[oschulz]

But @oschulz the whole point of MakiE is to work towards replacing Plots, at which point it would just be a question of passing the plot to the plotlyjs backend.

Sure, but for some things I have hope that the GLVisualize backend will perform a lot better than PlotlyJS. :-)

[oschulz]

But overall, MakiE is still mainly a one man job - I hope this will change, but until then I have to really prioritize what to work on.

I understand, and I didn't mean to be critical - I just wanted to set up this issue as a future nice-to-have. :-) I guess it has more to do with GLVisualize than with MakiE itself, anyhow?

[oschulz]

it's probably also the reason why you perceive plotly as slow. I'd be surprised if the performance is actually because they messed up the WebGL rendering.

In my case, it is definitely because of plotly itself (though data streaming is also an issue) - it's things like zooming in on the plot and so on, and with plotlyjs, all of that should happen client-side, without server interaction, right?

I really liked your "Drawing 2.7 billion Points in 10s" blog article, @SimonDanisch. I'm pretty sure plotly couldn't handle even a small fraction of that, no matter how the data got into it (of course, Jupyter couldn't stream that amount of data in the first place ...).

[SimonDanisch]

In my case, it is definitely because of plotly itself

Good to know! :) Would be interesting to see if they just wrote slow shaders, or if it's a problem with WebGL in general.

I guess it has more to do with GLVisualize than with MakiE itself, anyhow?

Well, not really, that's what I meant by saying the visualization part is orthogonal. I just mentioned that, because a solution involving MakiE and fast plotting via OpenGL will also depend on GLVisualize.

The biggest problem is getting the data to whatever machine you're using to look at the data - whether you end up visualizing it in WebGL or with OpenGL in GLVisualize doesn't really matter (well, OpenGL might be faster so in that sense it matters).

Why the visualization framework seems to matter is, because Plotly already solved the communication part internally.

With plotly it's possible to assemble the visualization on the server and then just "serve" the javascript to the client, with all the data embedded in javascript.

If we make the MakiE scene serializable, which shouldn't be that difficult, this should also work quite, possibly with better and faster streaming :) If we manage to make signals work remotely, we could easily implement the use cases you speak about, and would have the ability to only update parts of the plot (and only serialize & sent parts of the data). I wonder if that would already make it work in IJulia automatically. Never tried using an IJulia notebook remotely.

I really liked your Drawing 2.7 billion Points in 10s blog article,

Thanks :) Although that's a bad example, since the whole visualization part is pure Julia, and visualizing it is just a matter of displaying an Image. The updating of a big image quickly is probably the only noteworthy achievement of MakiE in that article ;) Which would be very cool if it just works ™ over a remote connection!

[SimonDanisch]

huh, I just started up a notebook - how would I even start a session on another server?

[oschulz]

huh, I just started up a notebook - how would I even start a session on another server?

What I (and many people I work with) do is to start Jupyter on a remote server (e.g. a beefy compute server or the head-node of a compute cluster), then then access remotely via an SSH tunnel (since there's of course always a firewall that prevents direct access to the Jupyter port on such systems). It's very convenient. With JupyterLab, it basically become a complete remote computing environment, including file-browing, code-editing, shells, etc.

This way, one can also do things like live presentations and demos, with code running real-time on a powerful system, even if the PC in the meeting room has nothing on it but a web browser. Beyond presentations, this also enables very productive analysis sessions with a group of people (where code and plots with be changed, re-run, etc.) - from anywhere.

[oschulz]

access remotely via an SSH tunnel

This isn't necessary, of course, if there's a JupyterHub installation or so. I'd like to see what we can do in that direction at our institute.

[oschulz]

If we make the MakiE scene serializable, which shouldn't be that difficult, this should also work

Alternatively, could ThreeJS.jl (though it seems inactive) become a backend for MakiE, long term, in addition to GLVisualize? Though maybe ThreeJS is not low-level enough?

[SimonDanisch]

I see, that seems handy and works quite different from what I expected.
ThreeJS is an option, but I think you can also only feed it data via json - but one could at least load your own shaders. Anyone is welcome to help with the integration of ThreeJS :) From what I remember, ThreeJS also isn't in the best state right now :(

[oschulz]

We could also look into offering JSROOT as a backend (maybe also for Plots). CERN ROOT itself is a large, complex C++ beast, but JSROOT itself doesn't depend on it - it only needs the objects to be plotted as JSON. ROOT generates the JSON by serializing it's C++ objects, but I think the JSON serializations could be generated directly in pure Julia without too much trouble. Plots like this one are pretty awesome (JSROOT uses three.js for 3D-plots). :-)

[SimonDanisch]

Actually, I was just recently forced to look further into julia's web stack and implementing the kind of interactive use cases for IJulia + Atom.
Turns out that @shashi and @JobJob have done a great job since the last time I checked, and it's actually relatively straight forward to implement quite a lot of the GLVisualize functionality with ThreeJS + WebIO.jl.
I will soon publish some initial prototypes! :)

[oschulz]

Awesome, thanks!

[oschulz]

If possible, it would be great to support the HTML5 Gamepad API at some point, since that'll automatically enable using input devices like the 3Dconnexion SpaceNavigator.

[SimonDanisch]

Well whoever needs that is more then welcome to figure out how to call the correct javascript with WebIO.jl

[oschulz]

Yes, i had hope the WebIO.jl would make things like that possible, in the future. Also, could WebIO give us interactivity with server-side rendering, long-term? I know I've been all about WebGL here, but in some use cases (complex plots, beefy GPU in server, weak GPU in client), server-side rendering (via VirtualGL?) may be preferable - if the user can still zoom, rotate, and interact in general.

[SimonDanisch]

well, that should mostly work already.. just need to pick up the mouse events client side with webio and stream the images rendered on the server to the client - the last part should already be implemented

[oschulz]

Sounds great!

The downside is, of course, that interactivity will only work while the Juypter/IJulia server is running. Whereas WebGL plots can still be interactive in a saved notebook (or even in a notebook converted to HTML, or in nbviewer). Also, WebGL-interaction is latency-free. So I think both server-side and client-side rendering would be great to have, in the long run.

[EricForgy]

Fast forward 15 months...

Hi guys 👋

I just read the thread above. Cool stuff 👍

In the meantime, progress with and interest in Makie continues. I just announced a very simple WIP package - Figures.jl - to create "Matlab like" figure windows in a browser. In one of the responses, I said:

Oh yes. My dream is to allow Makie to work with browsers 😍 This is a babystep in that direction 🙂

Figures currently just works with plotly.js (not PlotlyJS), but I think the concept vaguely resembles what Makie might look like with a WebGL backend except I did things in reverse. There is already a nice plotly.js API for the browser and, starting with that, I implemented a large part of the API in native Julia (so now, e.g., input errors can be caught on the Julia side as opposed to lobbing JS over to the browser and hoping for the best). With Makie, the nice native Julia API is already there and we need to make that work in the browser 🙂

Communication between Julia and the browser is currently done with websockets and JSON, which I think is fine for Plotly, but we probably want something more performant for WebGLMakie, or whatever we call the WebGL backend. BSON seems like a natural choice or maybe Protobuf.

Simon, if you (or anyone else, Tim?) could maybe help with a high-level roadmap how you imagine Makie might work with a WebGL backend, that would be awesome. I've only just started skimming around the Makie code and see three primary repos:

1. Makie.jl
2. AbstractPlotting.jl
3. GLMakie.jl

GLMakie describes itself as an "OpenGL backend for Makie", which sounds like a good place to start 🙂

This line in particular fills me with hope:

AbstractPlotting.register_backend!(GLBackend())

It seems to me like if we implement a WebGLBackend(), things should just work™️.

I have no idea how much work it would be to implement WebGLBackend though 😅

In any case, I'm happy to help out if I can 🙏

Cheers

PS: It sounds like ThreeJS.jl might also provide some hints so I'll check that out too 👍

[SimonDanisch]

WGLMakie does this now

[oschulz]

WGLMakie does this now

Yay!