Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How is logging handled #209

Closed
tawanda opened this issue Nov 30, 2016 · 2 comments
Closed

How is logging handled #209

tawanda opened this issue Nov 30, 2016 · 2 comments

Comments

@tawanda
Copy link

tawanda commented Nov 30, 2016

Hi Thanks for the great project, celery was giving me nightmares

I have 2 questions

  1. if i run a cluster with supervisor and pipe stdout to the log file for the supervisor process will all workers log to the same file and will that cause any conflicts if workers try and write to the same log file at the same time. Should i disable logging all together or there is no problem?

  2. if i run clusters on multiple servers, will i still be ok with scheduled tasks. As in they wont repeat. I know thats why celery runs beat in its own process. How are you handling this

Thanks again for the great work

@Koed00
Copy link
Owner

Koed00 commented Nov 30, 2016

Hi there.

  1. You should be ok there. All workers use the same logging bucket. It either set up it's own or uses the one you set up for Django.

  2. There is a small chance that multiple clusters pick up the same schedule. Currently the only way to prevent this is by disabling the scheduler on all but one cluster server.

@tawanda
Copy link
Author

tawanda commented Dec 1, 2016

Thanks for getting back to me

if i use a central cache server to lock do you think i can get away with running the scheduler on all instances

e.g

def scheduled_task():
    lock_id = "something unique"
    lock_expire = 60 * 5  # five minutes

    acquire_lock = lambda: cache.add(lock_id, "true", lock_expire)
    release_lock = lambda: cache.delete(lock_id)
    
    if acquire_lock():
        # do some things here .. 
        release_lock()        
    
    return True

I use elastic beanstalk so all my servers will have the same settings, if you think this could work then that would be sweet at lease if the tasks are 10ms from each other when they are recieved

@tawanda tawanda closed this as completed Mar 3, 2017
@tawanda tawanda reopened this Oct 10, 2017
@tawanda tawanda closed this as completed Oct 10, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants