Can worker/cluster resource usage be limited?

[pedrovgp]

Is there anyway to limit worker resource usage, or cluster resource usage, for that matter?

I am running django Q in the same machine as my Django project and it would be very useful to make sure it won't degrade app performance more than a certain amount.

By the way, incredible project. It alone justified migrating to Django 2 and Python 3.8.

[Koed00]

You can use cpu-affinity to set which worker will use which cores on your machine. For example; 2 workers with affinity 1 , would use core 1 for worker 1 and core 2 for worker 2. That would leave the other cores available for your main app.

If you're on Linux you can ofcourse use ulimit or something like monit to reign in the cluster.

I do have it on my todo list to at some point implement a resources module and have a way to limit memory and cpu usage.

[pedrovgp]

Thanks for the tips!

[randlet]

I'm not sure if this is exactly the right issue to bring this up or not, but the resource usage of Django-Q is also a bit of an issue for me too. The main problem is that my Django project has some large dependencies and each Python process is ~140+MB. Even with only a single worker, running qlcuster results in 5 processes which means Django-Q is taking up ~750MB for me. With the following config:

Q_CLUSTER = {
    'name': 'foo',
    'workers': 1,
    'timeout': 60,
    'catch_up': True,
    'recycle': 5,
    'compress': False,
    'save_limit': 250,
    'queue_limit': 500,
    'cpu_affinity': 1,
    'label': 'Django Q',
    'orm': 'default',
    'sync': False,
}

Running python manage.py qcluster I get:

3511  someuser  20   0  742.6m 206.4m   4.6m S   1.0 10.6 345:30.93 /path/to/venv/bin/python /path/to/project/manage.py qcluster
16751 someuser  20   0  676.9m 143.6m   6.7m S   0.0  7.3   0:00.00 /path/to/venv/bin/python /path/to/project/manage.py qcluster
23664 someuser  20   0  676.6m 142.4m   5.6m S   0.0  7.3 126:24.03 /path/to/venv/bin/python /path/to/project/manage.py qcluster
23681 someuser  20   0  675.9m 139.9m   4.9m S   0.0  7.2   0:27.06 /path/to/venv/bin/python /path/to/project/manage.py qcluster
23602 someuser  20   0  674.9m 136.1m   2.1m S   0.0  7.0   0:02.35 /path/to/venv/bin/python /path/to/project/manage.py qcluster

I wonder if it would be possible to run just the worker in the apps virtualenv and then qcluster in its own much lighter weight virtualenv? (I don't know enough about the architecture yet to know if this makes sense).

However, I want to echo @pedrovgp comments that this is a minor gripe and django-q is a fantastic project!

[Koed00]

That is the correct amount of processes, but they should all share the same memory space initially and only claim more memory when they diverge. I have a production server that runs a 250Mb image with 4 works at around 320Mb memory. After a while the memory usuage will go up to around 800Mb and then the recycle will usually drop it back to around the 350Mb mark. This is the cost of carrying around the entire django code base in the cluster.

How did you arrive at 750Mb? Most linux utilities will show full memory usage per child process, even though most of the memory is shared. You should look at the actual memory usage. Compare it with the cluster off and on.

That's said, I do think that I could tweak the memory usage a bit. I haven't found a good way of doing it yet.

[randlet]

Thanks for the response!

To get that number I just added up the "RES" column from running top. That's the 6th column in above table (sorry I forgot the header in the table above). My understanding is that this is the amount of actual RAM used by a process but I could definitely be mistaken on that.

I just tried the following:

ubuntu@ip-10-0-1-186:~$ free -h                                                                                                                                                                                    
              total        used        free      shared  buff/cache   available                                                                                                                                    
Mem:           1.9G        1.0G        259M        1.1M        626M        713M                                                                                                                                    
Swap:            0B          0B          0B                                                                                                                                                                        
ubuntu@ip-10-0-1-186:~$ sudo supervisorctl stop django-q                                                                                                                                                           
django-q: stopped                                                                                                                                                                                                  
ubuntu@ip-10-0-1-186:~$ free -h                                                                                                                                                                                    
              total        used        free      shared  buff/cache   available                                                                                                                                    
Mem:           1.9G        758M        570M        1.0M        625M        1.0G                                                                                                                                    
Swap:            0B          0B          0B

So I think was definitely mistaken here since this looks like total usage is ~300MB which makes a lot more sense. Sorry for wasting your time and thanks again for your time.

[Koed00]

Not a waste of time at all. I enjoyed double checking the code and reminding myself how I wrote this back then.
And I'm sure this conversation will help someone else some day.

[Koed00]

FYI I have a PR ready #457 that will add a memory limit for workers. Once workers hit this limit they recycle to release any extra memory. I still need to test it a bit more and get some coverage on it, but it's on the way.