Getting 'can't pickle lock objects' using async()

[mistalaba]

I am trying to send emails using django-q, but am getting a "TypeError: can't pickle lock objects" error trying the below. It works without the async. Sorry if I'm stupid, but I can't get this to work :/

Method for sending single email

def send_single_correspondence(subject, text_content, html_content, from_email, recipient, connection:
    msg = EmailMultiAlternativesWithEncoding(subject, unicode(text_content), from_email, [recipient], connection=connection)
    msg.attach_alternative(html_content, "text/html")
    msg.send()

and the async call

def send_board_correspondence(correspondence_instance):
    ...
    for recipient in recipients:
        async(send_single_correspondence, subject, text_content, html_content, from_email, recipient[0], connection)

Probably not important, this is just a class that's overridden to handle attachents better:

class EmailMultiAlternativesWithEncoding(EmailMultiAlternatives):
    def _create_attachment(self, filename, content, mimetype=None):
        """
        Converts the filename, content, mimetype triple into a MIME attachment
        object. Use self.encoding when handling text attachments.
        """
        if mimetype is None:
            mimetype, _ = mimetypes.guess_type(filename)
            if mimetype is None:
                mimetype = DEFAULT_ATTACHMENT_MIME_TYPE
        basetype, subtype = mimetype.split('/', 1)
        if basetype == 'text':
            encoding = self.encoding or settings.DEFAULT_CHARSET
            attachment = SafeMIMEText(smart_str(content,
                settings.DEFAULT_CHARSET), subtype, encoding)
        else:
            # Encode non-text attachments with base64.
            attachment = MIMEBase(basetype, subtype)
            attachment.set_payload(content)
            encoders.encode_base64(attachment)
        if filename:
            try:
                filename = filename.encode('ascii')
            except UnicodeEncodeError:
                filename = Header(filename, 'utf-8').encode()
            attachment.add_header('Content-Disposition', 'attachment',
                                   filename=filename)
        return attachment

The variables sent to send_single_correspondence are:
subject: u'Async sending 12:23'
text_content: 'Message body'
html_content: u'<p>Message body</p>'
from_email: u'fromemail@example.com'
recipient: u'toemail@example.com'
connection: <django.core.mail.backends.smtp.EmailBackend object at 0xb49aa0c>

Traceback:

Traceback (most recent call last):
  File "/home/martin/.virtualenvs/fysiografen/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/media/sf_Projects/fysiografen/fysiografen/utils.py", line 194, in wrapper
    result = f(*args, **kwds)
  File "/media/sf_Projects/fysiografen/board/views.py", line 62, in correspondence_send
    send_board_correspondence(instance)
  File "/media/sf_Projects/fysiografen/fysiografen/mail_utils.py", line 135, in send_board_correspondence
    async(send_single_correspondence, subject, text_content, html_content, from_email, recipient[0], connection, correspondence_instance.attachments.all())
  File "/home/martin/.virtualenvs/fysiografen/local/lib/python2.7/site-packages/django_q/tasks.py", line 41, in async
    pack = signing.SignedPackage.dumps(task)
  File "/home/martin/.virtualenvs/fysiografen/local/lib/python2.7/site-packages/django_q/signing.py", line 24, in dumps
    serializer=PickleSerializer)
  File "/home/martin/.virtualenvs/fysiografen/local/lib/python2.7/site-packages/django/core/signing.py", line 111, in dumps
    data = serializer().dumps(obj)
  File "/home/martin/.virtualenvs/fysiografen/local/lib/python2.7/site-packages/django_q/signing.py", line 40, in dumps
    return pickle.dumps(obj)
  File "/home/martin/.virtualenvs/fysiografen/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle lock objects

And here's the faulting line:

/home/martin/.virtualenvs/fysiografen/lib/python2.7/copy_reg.py in _reduce_ex
raise TypeError, "can't pickle %s objects" % base.name

Local vars
self
<thread.lock object at 0xadb9320>

base
<type 'thread.lock'>

proto
0

Perhaps I'm going about this the wrong way? It seems that I'm missing some information on the restrictions for django-q...?

Thanks for looking into this!

[Koed00]

Good morning.

I think the connection object is the problem. django.core.mail.backends.smtp.EmailBackend uses thread locking and that makes it impossible to be pickled (serialized).That's just a pickle limitation. In this case you can probably just omit the connection to the EmailMultiAlternativesWithEncoding call and let the worker create a fresh one or add a default to your function:

def send_single_correspondence(subject, text_content, html_content, from_email, recipient, connection=get_connection()):

It's also better to use a dotted string path to your mail function e.g. 'fysiografen.mail_utils.send_single_correspondence', this will make sure it gets freshly imported when called by the worker.

[mistalaba]

In this case you can probably just omit the connection to the EmailMultiAlternativesWithEncoding call and let the worker create a fresh one or add a default to your function

Since I'm most likely won't change the connection (sending these through Mandrill), I put the get_connection() in the send_single_correspondence and omitted it completely in the call later.
This worked like a charm!

It's also better to use a dotted string path to your mail function e.g. 'fysiografen.mail_utils.send_single_correspondence', this will make sure it gets freshly imported when called by the worker.

Thank you for the tip, I didn't know this!

Thanks so much for your work, django-q is so much nicer to work with than celery!

[brettbeeson]

Hi Legends, if you're searching "Django Q pickle error" and end of here, this thread helped me fix my error.
For others, I found out a few things the hard way:

• Careful sending (via async arguments) complex objects. I was pass a complex object in so I could run self.method() in the queue. I changed to sending the object id (in the database) and look it up in a normal (static) function.
• Careful returning objects too. I was returning a complex object and it couldn't pickle. I changed to return a simple integer or string.
  Thinking about multi-process queues, and pickles, it seems better to pass and return simple native objects, or nothing.