Misspelled address in 'Bcc' field generates hourly duplicate emails for the valid recipient

[radarsymphony]

Summary

When a user misspells an address in a 'Bcc' field, the valid recipients in the 'To' field receive hourly duplicates of the email.

Steps to reproduce

1. Create a New Conversation from within a mailbox.
2. Optional: Add text for body and subject-line.
3. In the ‘To Field’, add a valid address (the Recipient).
4. In the ‘Bcc Field’, add a misspelled address.
   
5. Send.
6. The (initial) email arrives in the Recipient’s inbox.
7. In this example, a duplicate email is received 5 minutes later, and then another duplicate every hour after that.
   

What is the current bug behaviour?

If a user misspells a Bcc'd address, the main recipient will keep receiving duplicate emails (until the job queue is cleared).
As expected, the following error appears within ‘Outgoing Email’ and ‘Send Errors’ in the logs:
Send error. Expected response code 354 but got code "554", with message "554 5.5.1 Error: no valid recipients ".

What is the expected or correct behaviour?

If an address is misspelled, other recipients should only receive one email, even if the undelivered email is added to a job queue and attempted a few more times.
In an additional test, the valid and misspelled addresses were swapped. No email came through to the Bcc'd recipient (this is good and expected), the job just sat in the queue until cleared.

Possible fixes

Is there a way to keep track of successful deliveries for each recipient separately?

Interim Solution:

1. In mailboxes, Click on ‘Manage’ → ‘System’
2. Scroll to the ‘Background Jobs’ section.
3. Click ‘Cancel’ for the job that is repeatedly sending an email.
   

Note: From the subheading under “Background Jobs”, it looks like this issue would have resolved itself after some time limit:
“Queued and failed jobs are cleaned automatically once in a while. No need to worry or delete them manually.”
Question: How often are queued and failed jobs auto-cleaned from the system?

[freescout-helpdesk]

Which mail server are you using?

[radarsymphony]

We are using postfix, but we can recreate the issue with others as the problem arose outside of an internal message.

[freescout-helpdesk]

You won't be able to reproduce this on https://demo.freescout.net/

It looks like the problem here is that the mail server sending emails returns an error immediately if the recipient address does not exist. This is a strange behavior. Normally mail server accepts an email for delivery, returns successful status code and if the email can not be delivered, sends a bounce message to the sender.

[tiredofit]

Interesting, I am able to replicate this as well on my own instance. I utilize a postfix (MTA) (lookup against LDAP) which forwards via dagent to Kopano Mail services (Formerly Zarafa) It seems that if a user is not in the usertable it instantly rejects, which I feel is the right way to do things, otherwise a user could effectively DoS a mail server with recipients that don't exist and clog their network bandwidth/fill their queue.

I also tried it with a MX in front of it ie Postal MTA -> Postfix (LDAP Lookup) -> Kopano.

[fvdm]

@freescout-helpdesk

It looks like the problem here is that the mail server sending emails returns an error immediately if the recipient address does not exist. This is a strange behavior. Normally mail server accepts an email for delivery, returns successful status code and if the email can not be delivered, sends a bounce message to the sender.

Bounce mails are not okay anymore thanks to back scatter spam and tight mail integrations. When a user does not exist the receiving MTA immediately returns a permanent error, allowing the sending MTA to handle it in a clean way.

[freescout-helpdesk]

@radarsymphony do you have "IMAP Folder To Save Outgoing Replies" configured in the mailbox "Fetching Emails" settings?

[radarsymphony]

@freescout-helpdesk That field is currently blank under our "Fetching Emails" settings. As in, we don't have that option active, but could implement it.

[freescout-helpdesk]

This issue is not trivial.

With servers like in this case, which are returning an error when one of the recipients can not receive the email, on PHP side there is no way to find out which emails were actually sent and which - not.

For example, if an email to a customer contains 2 CC addresses and 3 BCC addresses. Your mail server delivers an email to all BCC and to one CC address, but one CC address fails, your mail server returns a failure response without specifying which recipients succeeded and which - not:

Send error. Expected response code 354 but got code "554", with message "554 5.5.1 Error: no valid recipients ".

By the way Gmail/GSuite and other popular mail services behave normally.

[freescout-helpdesk]

What we can do is to collect most popular mail server responses designating delivery failure to one or more recipients (like 554 5.5.1 Error: no valid recipients) and consider the email successfully delivered in such cases and not try to resend it.

So, please post in this issue error messages you are receiving (see "Outgoing Emails" or "Manage > Logs > Send Errors" ), when FreeScout sends duplicate emails.

[radarsymphony]

Hi @freescout-helpdesk
When the issue is triggered, the following error gets added to "Manage > Logs > Send Errors":

Expected response code 354 but got code "554", with message "554 5.5.1 Error: no valid recipients "; File: /www/html/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Transport/AbstractSmtpTransport.php (457)

[fulldecent]

Can this please be replicated in the new 1.8 version?

[freescout-helpdesk]

Closing for no activity.

[dabesa]

I just got this behavior and we sent dozen of emails to the same recipients that were in BCC.
The email we sent was like this:
From:
To:
cc:
bcc:

Each recipient in BCC got dozens of emails until I killed the processes in the FreeScout queue.

Do you need any information for this?

Thanks

[freescout-helpdesk]

Can you send error messages for this email from Manage > Logs >Send Errors.

[dabesa]

Hi, I don't find that option:

Where is it?

[freescout-helpdesk]

Try to check App Logs.

[dabesa]

There's no error log under there:

[dabesa]

Under queue-jobs I see the tasks Processed:

I see a laravel error, around the time I killed the job:

Call to undefined method App\Misc\Helper::checkRequiredExtensions() {"userId":1,"email":"","exception":"[object] (Symfony\Component\Debug\Exception\FatalThrowableError(code: 0): Call to undefined method App\Misc\Helper::checkRequiredExtensions() at /home/cafeanal/helpdesk.cafe-analog.nl/app/Http/Controllers/SystemController.php:34)

[dabesa]

@freescout-helpdesk was it helpful somehow?

[malejoavilas]

What we can do is to collect most popular mail server responses designating delivery failure to one or more recipients (like 554 5.5.1 Error: no valid recipients) and consider the email successfully delivered in such cases and not try to resend it.

So, please post in this issue error messages you are receiving (see "Outgoing Emails" or "Manage > Logs > Send Errors" ), when FreeScout sends duplicate emails.

Hello,

I'm getting this errors when I send an email from a Ticket.

SQLSTATE[42S22]: Column not found: 1054 Unknown column 'smtp_queue_id' in 'field list' (SQL: insert into send_logs (thread_id, message_id, email, mail_type, status, customer_id, user_id, status_message, smtp_queue_id, updated_at, created_at) values (3843, reply-3843-7ac3f1e2687b4b60@mobiera.net, m.avila@mobiera.com, 1, 1, 6, , , 65A9D200EC, 2023-12-11 16:21:03, 2023-12-11 16:21:03)); File: /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connection.php (664)

[nooblag]

Hi there, yes there may be a possible regression on this issue?

I experienced the same sort of issue today on version 1.8.117. Messages with BCC recipients that fail caused the whole conversation to resend, which meant successful recipients also received it again every minute (which is when the cronjob for artisan schedule:run is set to operate)...

[adam3278]

Hello,
I see this bug is live again. My worker made a small mistake in bcc addresses section and as a result he received 70 emails to his worker email (you cannot send a message with all recipients as bcc in freescout, so he typed his own email in To), until I cancelled the background job in Freescout. Ok, this would be expected, but maybe 2-3 times and for addresses, where the message wasn’t delivered. I hope, that all of the bccs didn’t received the repeating message every hour of whole weekend. How to set the retry limit?

Queue job logs:
‘’’
Expected response code 354 but got code "503", with message "503-All RCPT commands were rejected with this error:
503-501 <'dar***@A***.com'>: malformed address: '> may not follow <'dar***@A***.com
503 Valid RCPT command must precede DATA
" {"exception":"[object] (Swift_TransportException(code: 503): Expected response code 354 but got code "503", with message "503-All RCPT commands were rejected with this error:
503-501 <'dar***@A***.com'>: malformed address: '> may not follow <'dar***@A***.com
503 Valid RCPT command must precede DATA
" at /www/html/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Transport/AbstractSmtpTransport.php:457)
[stacktrace]
#0 /www/html/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Transport/AbstractSmtpTransport.php(341): Swift_Transport_AbstractSmtpTransport->assertResponseCode('503-All RCPT co...', Array)
#1 /www/html/overrides/swiftmailer/swiftmailer/lib/classes/Swift/Transport/EsmtpTransport.php(305): Swift_Transport_AbstractSmtpTransport->executeCommand('DATA

', Array, Array, false, NULL)
#2 /www/html/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Transport/AbstractSmtpTransport.php(390): Swift_Transport_EsmtpTransport->executeCommand('DATA

', Array, Array)
#3 /www/html/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Transport/AbstractSmtpTransport.php(497): Swift_Transport_AbstractSmtpTransport->doDataCommand(Array)
#4 /www/html/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Transport/AbstractSmtpTransport.php(525): Swift_Transport_AbstractSmtpTransport->doMailTransaction(Object(Swift_Message), 'biuro@prowater....', Array, Array)
#5 /www/html/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Transport/AbstractSmtpTransport.php(207): Swift_Transport_AbstractSmtpTransport->sendBcc(Object(Swift_Message), 'biuro@prowater....', Array, Array)
#6 /www/html/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Mailer.php(71): Swift_Transport_AbstractSmtpTransport->send(Object(Swift_Message), Array)
#7 /www/html/vendor/laravel/framework/src/Illuminate/Mail/Mailer.php(451): Swift_Mailer->send(Object(Swift_Message), Array)
#8 /www/html/vendor/laravel/framework/src/Illuminate/Mail/Mailer.php(235): Illuminate\Mail\Mailer->sendSwiftMessage(Object(Swift_Message))
#9 /www/html/vendor/laravel/framework/src/Illuminate/Mail/Mailable.php(119): Illuminate\Mail\Mailer->send('emails/customer...', Array, Object(Closure))
#10 /www/html/vendor/laravel/framework/src/Illuminate/Mail/Mailer.php(250): Illuminate\Mail\Mailable->send(Object(Illuminate\Mail\Mailer))
#11 /www/html/vendor/laravel/framework/src/Illuminate/Mail/Mailer.php(205): Illuminate\Mail\Mailer->sendMailable(Object(App\Mail\ReplyToCustomer))
#12 /www/html/vendor/laravel/framework/src/Illuminate/Mail/PendingMail.php(99): Illuminate\Mail\Mailer->send(Object(App\Mail\ReplyToCustomer))
#13 /www/html/app/Jobs/SendReplyToCustomer.php(293): Illuminate\Mail\PendingMail->send(Object(App\Mail\ReplyToCustomer))
#14 [internal function]: App\Jobs\SendReplyToCustomer->handle()
#15 /www/html/overrides/laravel/framework/src/Illuminate/Container/BoundMethod.php(28): call_user_func_array(Array, Array)
#16 /www/html/overrides/laravel/framework/src/Illuminate/Container/BoundMethod.php(87): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
#17 /www/html/overrides/laravel/framework/src/Illuminate/Container/BoundMethod.php(27): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Application), Array, Object(Closure))
#18 /www/html/overrides/laravel/framework/src/Illuminate/Container/Container.php(549): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Array, Array, NULL)
#19 /www/html/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(94): Illuminate\Container\Container->call(Array)
#20 /www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(114): Illuminate\Bus\Dispatcher->Illuminate\Bus\{closure}(Object(App\Jobs\SendReplyToCustomer))
#21 /www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(102): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(App\Jobs\SendReplyToCustomer))
#22 /www/html/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(98): Illuminate\Pipeline\Pipeline->then(Object(Closure))
#23 /www/html/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(48): Illuminate\Bus\Dispatcher->dispatchNow(Object(App\Jobs\SendReplyToCustomer), false)
#24 /www/html/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(76): Illuminate\Queue\CallQueuedHandler->call(Object(Illuminate\Queue\Jobs\DatabaseJob), Array)
#25 /www/html/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(320): Illuminate\Queue\Jobs\Job->fire()
#26 /www/html/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(270): Illuminate\Queue\Worker->process('database', Object(Illuminate\Queue\Jobs\DatabaseJob), Object(Illuminate\Queue\WorkerOptions))
#27 /www/html/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(114): Illuminate\Queue\Worker->runJob(Object(Illuminate\Queue\Jobs\DatabaseJob), 'database', Object(Illuminate\Queue\WorkerOptions))
#28 /www/html/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(100): Illuminate\Queue\Worker->daemon('database', 'emails,default,...', Object(Illuminate\Queue\WorkerOptions))
#29 /www/html/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(84): Illuminate\Queue\Console\WorkCommand->runWorker('database', 'emails,default,...')
#30 [internal function]: Illuminate\Queue\Console\WorkCommand->handle()
#31 /www/html/overrides/laravel/framework/src/Illuminate/Container/BoundMethod.php(28): call_user_func_array(Array, Array)
#32 /www/html/overrides/laravel/framework/src/Illuminate/Container/BoundMethod.php(87): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
#33 /www/html/overrides/laravel/framework/src/Illuminate/Container/BoundMethod.php(27): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Application), Array, Object(Closure))
#34 /www/html/overrides/laravel/framework/src/Illuminate/Container/Container.php(549): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Array, Array, NULL)
#35 /www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(183): Illuminate\Container\Container->call(Array)
#36 /www/html/vendor/symfony/console/Command/Command.php(255): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle))
#37 /www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(169): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle))
#38 /www/html/vendor/symfony/console/Application.php(992): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#39 /www/html/vendor/symfony/console/Application.php(255): Symfony\Component\Console\Application->doRunCommand(Object(Illuminate\Queue\Console\WorkCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#40 /www/html/vendor/symfony/console/Application.php(148): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#41 /www/html/vendor/laravel/framework/src/Illuminate/Console/Application.php(88): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#42 /www/html/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(121): Illuminate\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#43 /www/html/artisan(60): Illuminate\Foundation\Console\Kernel->handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#44 {main}
"}
‘’’

[freescout-help]

"malformed address" error has been added to no_retry_mail_errors list, it will be published in the next release.

[adam3278]

Thank you! You’re huge