Support multiple workers / instances behind load balancer / reverse proxy (for example Gunicorn or similar)

[aersam]

This is a proposal after having discussed an issue in #1193

Currently nicegui cannot be used behind gunicorn or another reverse proxy without affinity support. This may lead to confusion and also I think it's straightforward to remove that limitation.

Reverse Proxies are quite common with FastAPI/Python since otherwise the Python GIL limits you quite a bit and a single long running request can block others (if not using async/await, sometimes you can't)

I think it would be possible for nicegui to implement support for running behind a reverse proxy without too much effort, but maybe I'm wrong. This is because once the websocket connection is established, only a single worker is involved anyway.
One would need persist the client(id/data) to disk on initial page render and then read it from there if needed. After the websocket handshake, everything works with the websocket which implies staying on the same worker. So only one hand-over between workers/processes is required.

Do you think that's something to consider? Or do I simply things too much?

[rodja]

Thanks for bringing this up @aersam. We would love to remove this limitation. But it might not be so simple.

One would need persist the client(id/data) to disk on initial page render and then read it from there if needed.

The data part may become tricky. Let us consider a simple scenario with a load balancer / reverse proxy in the front and two workers / instances named A and B in the back. The correct page-builder function (decorated with @ui.page) is executed when requesting the initial page. For example on worker A. While execution, arbitrary state can be created on the worker. For example opening an usb camara device or creating a database connection. After loading the html response the client establishes a websocket connection but may be routed to worker B by the load balancer. I do not know any way to transfer the state from A to B.

Last week @falkoschindler voiced a possible alternative approach: what if the page builder does not execute the decorated function directly but only serves a super tiny, generic html page which establishes a websocket connection. Then the the page builder is executed after the websocket connection is established. We have not found the time to investigate this approach. Anybody interested in giving it a shot?

[aersam]

I am unsure how much time I find for it, but will do this sooner or later as we will need it

[rodja]

If we're running the page builder on incoming websocket instead of incoming http request, we need to abandon the ability to return Starlette/FastAPI responses. And we won't have access to the request parameters any longer.

Another problem with multiple workers is the loss of global state. For example in a simple chat app, only the users on the same worker would see each other.

[aersam]

I think people are aware of the loss of global state if working with multiple workers, that's the same thing in every other framework, too.

Access to request parameter and not being able to return Responses is a big thing to me, however. It's the only meaningful place to do authentication, so I guess we need a different solution.

What if one could add a parameter sideeffect=False to ui.page which would allow to transfer the state between worker on init? Since the Id is strictly increment only, I guess it should work to just reexecute it on another worker. The first worker would somehow need to be able to remove it's state after some timeout since he won't receive the connection

Or allow something more involved which allows splitting the logic in two handlers:

@ui.page('/documentation')
def documentation_page(page) -> None:
    @page.pre_render(req: Request) ## no access to events/client. used for SEO/access to parameters etc
    def pre_render():
        ui.html("Loading...") # html would be removed on render
        if not req.bla
            return Response() # if a response object is returned, no connection wanted
        return None # maybe allow a json-dumpable dict that is passed to render?, even on other worker

    @page.render()
    def render():
        add_head_html()
        menu = side_menu()
        add_header(menu)
        ui.add_head_html('<style>html {scroll-behavior: auto;}</style>')
        with ui.column().classes('w-full p-8 lg:p-16 max-w-[1250px] mx-auto'):
            section_heading('Reference, Demos and more', '*NiceGUI* Documentation')
            documentation.create_full()

But looks too complicated... Would need to be optional to use

[rodja]

If we could auto-discover and reach all other workers we may be able to route the websocket stream to the correct worker.

Or we simply write a tutorial on how to use a load balancer like Traefik which supports session affinity instead of plain gunicorn.

[aersam]

That being said I do not have a good solution yet and I don't think there is an easy solution

[rodja]

I'm not too confident you never run into a blocking problem in a larger production app.

Sure, it's not easy. But I think it's still the best option for running a larger production app. In our robotics framework RoSys which uses NiceGUI as a basis we can activate asyncio warnings for tasks which block to long: https://github.com/zauberzeug/rosys/blob/main/rosys/analysis/asyncio_warnings.py

Also because many libraries just don't support async/await, just imagine some people doing some unwise pandas stuff.

Pandas operations are mostly not I/O bound but cpu bound and need to be executed in a separate process (see run.cpu_bound in RoSys).

For scenarios like these, gunicorn is a big help

I disagree. Gunicorn covers up the inherent problem of having blocked the event loop. It will still affect everything running on the specific worker. Hence a portion of your users will still experience the lag.

[v479038280]

If we can learn from streamlit about how to handle such delay?

[aersam]

@rodja: Well you're right in that's the proper way of doing things. I just have a quite a special group of people doing python that's not mainly devs but more coming from a data background without too much knowledge on resource management. Gunicorn is a bit of a duplicate assurance to me that no stupid thing like a memory leak or so blows up everything. If I have 4 workers and one is blown up, gunicorn will restart it and it will save me lot's of tickets, but not all. Of course it's not a solution, it's a workaround.

However I must admit that I did not see how complex this will be and therefore it's maybe just simpler to use a real load balancer with affinity support. What you should be doing however is to have a page to direct users to in case they experience the handshake error, to have some guidance there

[rodja]

Thanks for explaining @aersam. I totally understand the issue but am not sure how to fix it. I've created a new entry in our FAQs: https://github.com/zauberzeug/nicegui/wiki/FAQs#how-to-avoid-the-reloading-because-handshake-failed-javascript-error-message-which-results-in-a-full-page-reload

[Stefan2016]

Thanks for explaining @aersam. I totally understand the issue but am not sure how to fix it. I've created a new entry in our FAQs: https://github.com/zauberzeug/nicegui/wiki/FAQs#how-to-avoid-the-reloading-because-handshake-failed-javascript-error-message-which-results-in-a-full-page-reload

Do you have experience setting up session affinity with ngnix and nicegui? Would be interesting to see an example.

[rodja]

This feature request is more about supporting multiple workers for proxies which do not support session affinity. But nginx does. See #3038 for a discussion about such a setup.

[barrywhart]

Has anyone considered the possibility of storing the Client instances in a data store that is shared across workers / instances? In the case of multiple gunicorn workers, the Client instances could be stored in memory in a separate singleton process or in a data store such as shelve or SQLite.

The clients would be serialized using pickle, dill, cloudpickle. (Note that dill and cloudpickle have better support than pickle for serializing code such as Python lambdas.)

For scaling across multiple servers, the Client instances could be stored in Redis or a traditional database.

I understand the earlier point about side effects, but I think this can be minimized or avoided through documentation. The basic guidance would be to that state is only accessible via fields of the Element objects that make up the page. For my use case, the application is a pretty traditional web application that ultimately triggers some long-running automation jobs (e.g. applying Terraform changes).

[rodja]

As I wrote in #1539 (comment):

While execution, arbitrary state can be created on the worker. For example opening an usb camara device or creating a database connection.

So you do not only need to sync the client instances but everything. Mabye in very defined special scenarios it may work... but I think it its nothing we could support from the general NiceGUI library side.

[barrywhart]

very defined special scenarios it may work

I think this (well-defined state) is the typical case for many applications. How many web applications have a USB camera device connected to the server?

Regardless, perhaps the way to approach this would be to enhance NiceGUI to support pluggable state backends. Then people could select or build a state backend based on their needs. Another advantage of this approach is that the best backend could vary based on the scale of the application. E.g. an application with a handful of users may choose to use a SQL backend, while a high-volume application might use Redis or a NoSQL backend.

[barrywhart]

Related: Last year, I contributed a pluggable backend for MLflow (PR link). In that case, it was for authentication, not for state management.

I mention this because, if there's potential interest in this idea, I may be able to contribute.

I think this capability could significantly expand the potential number of users for NiceGUI. It's a really great package, and I'd love it if it were suitable for higher-volume applications.

[T2XX]

can we using SSE technology to avoid websocket http upgrade? so that we can use gunicorn works.

[T2XX]

see example

[rodja]

SSE (Server-Sent Events) are unidirectional and hence are not suited for interactive applications as far as I know. Also they would have the same problem as websockts: the state for one client must be completely managed in one worker and can not be shared between processes.