Using HPX's mutex and condition_variable leads to 99% CPU usage

[phil-skillwon]

Hello,
My application is based on the HPX runtime, and I have set these two configuration options:

hpx.max_idle_loop_count=1000
hpx.max_idle_backoff_time=1000

I used HPX-provided mutex and condition_variable to protect shared data, which is a std::queue<std::vector<std::byte>>. I created two HPX threads, with thread A as the producer and thread B as the consumer. Thread A reads data from PCIe-based XDMA and pushes it to the queue, then calls condition_variable::notify_one. Thread B calls my custom read function to read data from the shared queue within a specified timeout.

Here's how my custom read function looks:

ResultRs<vector<byte>, ErrorRs> Xdma::read(const int64_t timeout) 
{
	// size_t m_maxSizeCacheQue;
    // std::queue<std::vector<std::byte>> m_cacheQue;
	// hpx::mutex m_cacheLock;
    // hpx::condition_variable m_cacheCond;
	
    ResultRs<IntfCardFrame, ErrorRs> ret = ErrorRs::withCode(ErrorKind::Other);
    std::unique_lock guard(m_cacheLock);
    if (0 == timeout) 
    {
        if (true == m_cacheQue.empty()) 
        {
            return ErrorRs::withCode(ErrorKind::ResourceNotAvailable);
        }
    }
    else if (timeout < 0) 
    {
        m_cacheCond.wait(guard, [&]{
            return false == this->m_cacheQue.empty();
        });
    }
    else 
    {
        auto retBool = m_cacheCond.wait_for(guard, std::chrono::milliseconds(timeout), [&]{
            return false == this->m_cacheQue.empty();
        });

        if (false == retBool) 
        {
            return ErrorRs::withCode(ErrorKind::Timeout);
        }
    }

    auto item = std::move(m_cacheQue.front());
    m_cacheQue.pop();
    return item;
}

During testing, I noticed that even when the shared queue is empty, using mutex or condition_variable causes all CPU cores to reach 99% utilization.
I’ve encountered a similar issue before where using hpx::thread::sleep_for(1s) caused CPU usage to hit 80%. #6470
Is this behavior normal? I want to achieve minimal CPU usage when using mutex and condition_variable.

[hkaiser]

The HPX synchronization primitives (semaphore, condition_variable, etc.) suspend the running HPX (user-level) thread, not the underlying kernel (p-) thread. This is done to allow for other HPX threads to run while a particular one is blocked.

For this reason, the CPUs are still utilized 100% as the kernel threads that run the scheduling loop for HPX threads keep churning along.

If you want to reduce CPU utilization in case if no HPX work is available and ready to run, you can use

hpx::threads::set_scheduler_mode(hpx::threads::policies::scheduler_mode::fast_idle_mode);

first thing inside hpx_main that instructs the main scheduling loop to put the underlying kernel thread to sleep much faster than done by default.