Implementation Help: Folder::idle() function running in background. - Laravel

[catabozan]

I want to listen for incoming emails using the idle() function in my Laravel project.

The problem is that the idle() function never finishes running and the rest of the code is never executed.

I need to call the idle() function for each user, since I must listen for incoming emails for all of my users. (each user authenticates with its own IMAP credentials)

T've tried to set up a job for each user and dispatch it to a queue, but then the job never ends (and since the numbers of users changes, the number of workers must change), and I can't find a way to stop the job when needed.

I also need a way to stop the execution of the idle() function for a specific user, for example if the user deletes its account.

Is there a way to run that piece of code in the background, without it interfering with the code execution, with the possibility to stop the execution of a specific instance of the process?

[HelloSebastian]

A continuous loop runs in the idle method and therefore blocks the following program code.

I have discovered two solutions for me so far, but I am still looking for the perfect solution.

You can create a new PHP process for each folder.

You can create a command that reads your active users from the database. Then you would have to create a new process for each user and their folders. Maybe it is possible to control only the inbox via the idle method.

So that the started processes are also stopped again when the user is no longer active, you must track them.

I loaded all users from the database every 2 minutes and compared them with an array of active users. Based on the changes I either started or stopped processes.

// $userIds -> from db, $activeUserIds -> from array
$insertions = array_diff($userIds, $activeUserIds);
$deletions = array_diff($activeUserIds, $userIds);


foreach ($insertions as $insertionId) {
    foreach ($users as $user) {
        if ($user->getId() == $insertionId) {
            $this->startUpdateFlagsProcess($user);
            $this->activeUsers[] = $user;
        }
    }
}

foreach ($deletions as $deletionId) {
    foreach ($this->activeUsers as $key => $activeUser) {
        if ($deletionId == $activeUser->getId()) {
            $this->endUpdateFlagsProcess($activeUser);
            unset($this->activeUsers[$key]);
        }
    }
}

// ensure all process are running
foreach ($this->updateFlagsProcesses as $accountId => $process) {
    $this->ensureProcessRun($process, $accountId);
}

foreach ($this->fetchEmailProcesses as $accountId => $process) {
    $this->ensureProcessRun($process, $accountId);
}

I use symfony proccess component for process management. The processes are stored in arrays with the account IDs as index.

This could of course become a lot of processes, which is not necessarily performant. I am still at the beginning and am testing with two accounts.

ReactPHP

The second option I am working on is to use ReactPHP. There you can start multiple "event loops" that don't block the program code. But here I can't give you any more details if it really works.

Maybe the information will help you to find a solution.

[catabozan]

Thanks, I'll take a look at the Symphony process component and ReactPHP.

Edit: Could you provide the repo(or a more detailed example) of how you managed to create the processes?

[HelloSebastian]

Sure.

The program code I have already written is in a while(true) {} loop.

Symfony proccess component:
https://symfony.com/doc/current/components/process.html

ReactPHP Event Loop:
https://reactphp.org/event-loop/

I use symfony in my project, so I don't know if there is also a library from laravel for process management.

In my data structure, a user can have multiple accounts and an account can have multiple mailboxes.

The startUpdateFlagsProcessmethod:

// use Symfony\Component\Process\Process;

private function startUpdateFlagsProcess(User $user)
{
    foreach ($user->getAccounts() as $account) {
        $process = new Process(["php", "/Users/sebastian/Sites/mail-client/bin/console", "app:email:update-flags", $account->getId()], timeout: null);
        $process->start();
        $this->updateFlagsProcesses[$account->getId()] = $process;
    }
}

The endUpdateFlagsProcess method

// use Symfony\Component\Process\Process;

private function endUpdateFlagsProcess(User $user)
{
    foreach ($user->getAccounts() as $account) {
        if (isset($this->updateFlagsProcesses[$account->getId()])) {
            $this->updateFlagsProcesses[$account->getId()]->stop();

            unset($this->updateFlagsProcesses[$account->getId()]);
        } else {
            echo "Process was not started.";
        }
    }
}

In the ensureProccessRun method I check the respective array of processes. If the process is no longer running because of an error, it is restarted.

I can't help you much with ReactPHP yet, as I'm just trying it out myself today. However, it seems to be very promising, because everything can be processed in one process or command.

[catabozan]

I gave ReactPHP a try, but it seems like because the idle() function never stops running, the loop gets stuck and the rest of the code never gets executed.

$client = (new ClientManager())
    ->make([ ... ]);

/** @var Folder $folder */
$folder = $client->getFolders()[0];

$loop = Loop::get();

$loop->addTimer(0, function () use ($folder) {
    $folder->idle(function (Message $email) {
        echo $email->uid . "\n";
    });
});
$loop->run();

// never reaches this line
echo "aaa";

The script runs until $loop->run() and gets stuck. Do you have any solution in mind / resources you could point me to?

[HelloSebastian]

Yup, that's exactly what I found out today too.

My current state is that you either have to realize with multiple processes or not use the idle method.

One problem with implementing the idle method with ReactPHP is also that you need a new instance of Client for each mailbox, since only one mailbox can be selected at a time. So several connections to the IMAP server would have to be made at the same time.

Please correct me if this is not correct.

I am currently testing several alternative ways to the idle method.

In PR #201 I added a method getByUidGreaterThanEqual($uid) which returns all messages with an equal or greater UID. Per mailbox you would then have to store the last known UID and could then retrieve this again and again via ReactPHP in a timer.

Additionally, I'm currently facing the challenge of sychronizing already collected messages (e.g. updating the flags (read, answer, ...)). This problem is not handled by the idle method at all. So in my use case I would have to do a realization via timers anyway.

[Webklex]

Hi @catabozan ,
the idle method is supposed to never end. It "listens" on an open connection for new mails and therefor has to run forever. If you are using laravel, you could use jobs / queue workers to maintain these connections (be aware to have enough free workers) and reopen them in case of any failures.
You could also use a command as described here: https://www.php-imap.com/frameworks/laravel/commands and have it called for each inbox you want to "monitor".

Best regards,

[catabozan]

Thanks for your help. I would have to summon lots of workers to use the queue, and I think it would be quite difficult to manage all the jobs. The problem is that I want to listen for incoming emails for all my users at the same time, and I would like to cancel a job if, for example, a user deletes its account.

I'm thinking about creating a separate script that would only listen for incoming emails and ping the Laravel server through a webhook.

This is the script

<?php

require __DIR__ . '/../vendor/autoload.php';
use Webklex\PHPIMAP\ClientManager;
use Webklex\PHPIMAP\Folder;
use Webklex\PHPIMAP\Message;

// required
$emailCredentials = json_decode(getenv("LISTENER_EMAIL_CREDENTIALS"), true);
$webhookUrl = getenv('LISTENER_WEBHOOK_URL');
$ID = getenv('LISTENER_ID');

// optional
$folderPath = getenv('LISTENER_FOLDER_PATH');
$debug = getenv('LISTENER_DEBUG') && getenv('LISTENER_DEBUG') !== 'false';

// if the required env variables are not set
if (
    ! $emailCredentials
    || ! key_exists('imap_host', $emailCredentials)
    || ! key_exists('imap_port', $emailCredentials)
    || ! key_exists('imap_username', $emailCredentials)
    || ! key_exists('imap_password', $emailCredentials)
    || ! $webhookUrl
    || ! $ID
) {
    exit(10); // custom exit code 
}

// if does not have a '/' at the end, add it
if (! (substr($webhookUrl, -1) === '/')) {
    $webhookUrl .= '/';
}

$webhookUrl .= $ID . '/';

if ($debug) {
    echo "-----\n";
    echo "Warning: Debug is Enabled!\n";
    echo "Base Webhook Url: $webhookUrl\n";
    echo "-----\n\n";
}

$client = (new ClientManager())
->make([
    'host'          => $emailCredentials['imap_host'],
    'port'          => (int) $emailCredentials['imap_port'],
    'encryption'    => 'ssl',
    'validate_cert' => true,
    'username'      => $emailCredentials['imap_username'],
    'password'      => $emailCredentials['imap_password'],
    'protocol'      => 'imap'
])->connect();

/** @var Folder $folder */
$folder = $folderPath && $folderPath !== "false"
    ? $client->getFolderByPath($folderPath)
    : $client->getFolders()[0];

$curl = curl_init();
curl_setopt($curl, CURLOPT_POST, true);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, false);
curl_setopt($curl, CURLOPT_VERBOSE, false);


// post request to http://domain.com/imap-webhook/{LISTENER_ID}/{uid}
$folder->idle(function (Message $message) use ($curl, $webhookUrl, $debug, $ID) {
    $uid = $message->getUid();
    $url = $webhookUrl . $uid;

    if ($debug) {
        echo "Ping to url: $url \n";
        echo "ID: $ID \n";
        echo "uid: $uid \n";
        echo "------------- \n";

        curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
    }

    curl_setopt($curl, CURLOPT_URL, $url);
    $resp = curl_exec($curl);

    if ($debug) {
        echo $resp . "\n";
    }
}, 1200, true);

and I run it like this

LISTENER_EMAIL_CREDENTIALS="{
    \"imap_host\": \"mail.domain.ro\",
    \"imap_port\": 993,
    \"imap_username\": \"test@domain.ro\",
    \"imap_password\": \"password123\"
}" LISTENER_ID=69 LISTENER_FOLDER_PATH=false LISTENER_DEBUG=true LISTENER_WEBHOOK_URL="https://domain.com/email-webhook" php src/index.php

I'm thinking about using supervisord with supervisorphp/supervisor to start a separate process of the script for each user, and use the ENV variables to identify and configure each process.

Is this a viable solution or is it too overkill?

[HelloSebastian]

I would have one more idea, but I have not tried it yet.

You could replace the idle method completely with ReactPHP. That means, instead of a while(true) loop, use a ReactPHP timer. Then you could start as many as you want in one process.

Should you try this, I would appreciate feedback.

About your idea:
How do you find out the processes that should be terminated?

[catabozan]

I haven't thought about using ReactPHP instead of while(true), this seems like a better idea than creating processes with supervisord.

How do you find out the processes that should be terminated?

I was thinking about using the LISTENER_ID (that would be, for example, the user_id) env variable to assign an ID to the process, I'm not sure how I would be able to do that. It is possible to change the number of processes running using supervisorphp/supervisor as it's stated in the second answer here. I was going through the supervisord docs to see if I can achieve this.

Should you try this, I would appreciate feedback.

Sure, I'll let you know :).

[catabozan]

I got it working by replacing the while(true) with $loop->addPeriodicTimer()

index.php

$client = (new ClientManager())
    ->make([ ... ])->connect();

/** @var Folder $folder */
$folder = $client->getFolders()[0];

$timer = ReactPHPFolder::from($folder)
    ->reactPHPIdle(function (Message $message) {
        echo "uid: {$message->uid}\n";
    }, 1200, true);

echo "test not in loop\n\n";

ReactPHPFolder.php

class ReactPHPFolder extends Folder
{
    protected function __construct(Folder $folder)
    {
        parent::__construct(
            $folder->getClient(),
            $folder->name,
            $folder->delimiter,
            self::createAttributesArray($folder)
        );
    }

    public static function from(Folder $folder): self
    {
        return new self($folder);
    }

    public function reactPHPIdle(callable $callback, $timeout = 1200, $auto_reconnect = false): TimerInterface
    {
        $this->client->getConnection()->setConnectionTimeout($timeout);

        $this->client->reconnect();
        $this->client->openFolder($this->path, true);
        $connection = $this->client->getConnection();

        $sequence = ClientManager::get('options.sequence', IMAP::ST_MSGN);
        $connection->idle();

        $timer = Loop::get()->addPeriodicTimer(0, function () use ($callback, &$connection, $sequence, $auto_reconnect) {
            try {
                $line = $connection->nextLine();
                if (($pos = strpos($line, "EXISTS")) !== false) {
                    $msgn = (int) substr($line, 2, $pos -2);
                    $connection->done();

                    $this->client->openFolder($this->path, true);
                    $message = $this->query()->getMessageByMsgn($msgn);
                    $message->setSequence($sequence);
                    $callback($message);

                    $event = $this->getEvent("message", "new");
                    $event::dispatch($message);

                    $connection->idle();
                }
            } catch (RuntimeException $e) {
                if (strpos($e->getMessage(), "connection closed") === false) {
                    throw $e;
                }
                if ($auto_reconnect === true) {
                    $this->client->reconnect();
                    $this->client->openFolder($this->path, true);

                    $connection = $this->client->getConnection();
                    $connection->idle();
                }
            }
        });

        return $timer;
    }

    protected static function createAttributesArray(Folder $folder): array
    {
        $attributes = [];

        if ($folder->no_inferiors) {
            $attributes[] = '\NoInferiors';
        }

        if ($folder->no_select) {
            $attributes[] = '\NoSelect';
        }

        if ($folder->marked) {
            $attributes[] = '\Marked';
        }

        if ($folder->referral) {
            $attributes[] = '\Referral';
        }

        if ($folder->has_children) {
            $attributes[] = '\HasChildren';
        }

        return $attributes;
    }
}

the reactPHPIdle() method returns the timer, so you can stop that particular loop at any time.

I think this should work.

[HelloSebastian]

This looks very good! :)

Have you already tried if it works with multiple folders of a client? My understanding is that this should not work directly, since only one folder can be "selected" at a time. Is that a requirement with you, that all folders of an account should be watched or only Inbox?

A small improvement:

$timer = Loop::addPeriodicTimer(...);

You don't need to call get() additionally.

May I use your ReactPHPFolder class in my project? :)

[catabozan]

May I use your ReactPHPFolder class in my project? :)

Sure :)

Have you already tried if it works with multiple folders of a client?

It's not a requirement, I only need to watch the inbox folder. But I think you could create multiple clients for the same credentials, one for each folder you want to watch.

$timer = Loop::addPeriodicTimer(...)

Thanks for the tip, I'll use the addPeriodicTimer() function :)

[catabozan]

@HelloSebastian I have encountered an issue with the use of ReactPHP - here is a new issue I opened: php-imap/issues/209.

Did you encounter this problem as well?