OSquery fails to run after upgrade from 8.16.1 to 8.17.1

[belimawr]

For confirmed bugs, please report:

• Version: 8.17.1
• Operating System: Windows
• Discuss Forum URL: https://discuss.elastic.co/t/osquerybeat-permission-error-no-queries-possible/371400/2

Steps to Reproduce:

1. Install Elastic-Agent 8.16.1 with Osquery Manager and Osquery Logs integration
2. Wait for the Elastic-Agent to become healthy
3. Upgrade to 8.17.1
4. The Elastic-Agent might became unhealthy and you'll see errors from OSquery like this

W0204 10:53:17.582293   844 extensions.cpp:426] Will not autoload extension with unsafe directory permissions: C:\Program Files\Elastic\Agent\data\elastic-agent-8.17.1-b46c28\components\osquery-extension.exe

E0204 10:53:17.615715   844 shutdown.cpp:79] Cannot activate osq_config config plugin: Unknown registry plugin: osq_config

Then some errors communicating with it:

failed to connect, reconnect might be attempted, err: dialing pipe '\\.\pipe\elastic\osquery\2084fe12-fa4c-4555-a3b1-196c9a370738': open \\.\pipe\elastic\osquery\2084fe12-fa4c-4555-a3b1-196c9a370738: The system cannot find the file specified.

attempt 1 out of 11 failed, err: dialing pipe '\\.\pipe\elastic\osquery\2084fe12-fa4c-4555-a3b1-196c9a370738': open \\.\pipe\elastic\osquery\2084fe12-fa4c-4555-a3b1-196c9a370738: The system cannot find the file specified.

And other OSquery errors:

osquerybeat Run exited with error: I0204 10:53:20.079067 10368 init.cpp:413] osquery initialized [version=5.12.1]
I0204 10:53:20.081629 10368 dispatcher.cpp:78] Adding new service: UsersService (0000024ED6BD4240) to thread: 12900 (0000024ED6C14F40) in process 12916
I0204 10:53:20.081629 10368 dispatcher.cpp:78] Adding new service: GroupsService (0000024ED6BD5670) to thread: 8372 (0000024ED6C14D40) in process 12916
W0204 10:53:20.082434 10368 extensions.cpp:426] Will not autoload extension with unsafe directory permissions: C:\Program Files\Elastic\Agent\data\elastic-agent-8.17.1-b46c28\components\osquery-extension.exe
I0204 10:53:20.082934 10368 rocksdb.cpp:90] Opening RocksDB handle: osquery\osquery.db
I0204 10:53:20.091150  8372 groups_service.cpp:55] Groups cache initialized
I0204 10:53:20.097339 12900 users_service.cpp:149] Users cache initialized
I0204 10:53:20.120223 10368 dispatcher.cpp:78] Adding new service: ExtensionWatcher (0000024ED76F0260) to thread: 22012 (0000024ED76C45A0) in process 12916
I0204 10:53:20.120223 10368 dispatcher.cpp:78] Adding new service: ExtensionRunnerCore (0000024ED7706C00) to thread: 25828 (0000024ED76C4740) in process 12916
E0204 10:53:20.120223 10368 shutdown.cpp:79] Cannot activate osq_config config plugin: Unknown registry plugin: osq_config
I0204 10:53:20.120223 10368 dispatcher.cpp:149] Thread: 10368 requesting a stop
I0204 10:53:20.120223 10368 dispatcher.cpp:156] Service: 0000024ED6BD4240 has been interrupted
I0204 10:53:20.120223 10368 dispatcher.cpp:156] Service: 0000024ED6BD5670 has been interrupted
I0204 10:53:20.120223 10368 dispatcher.cpp:156] Service: 0000024ED76F0260 has been interrupted
I0204 10:53:20.120223 25828 interface.cpp:299] Extension manager service starting: \\.\pipe\elastic\osquery\2084fe12-fa4c-4555-a3b1-196c9a370738
I0204 10:53:20.144812 10368 dispatcher.cpp:156] Service: 0000024ED7706C00 has been interrupted
I0204 10:53:20.144812 10368 dispatcher.cpp:122] Thread: 10368 requesting a join
I0204 10:53:20.149353 10368 dispatcher.cpp:140] Service thread: 0000024ED76C4740 has joined
I0204 10:53:20.149353 10368 dispatcher.cpp:140] Service thread: 0000024ED76C45A0 has joined
I0204 10:53:20.149353 10368 dispatcher.cpp:140] Service thread: 0000024ED6C14D40 has joined
I0204 10:53:20.149353 10368 dispatcher.cpp:140] Service thread: 0000024ED6C14F40 has joined
I0204 10:53:20.149353 10368 dispatcher.cpp:144] Services and threads have been cleared: exit status 78

Out of 3 attempts using a Windows Server 2019, only the first failed, all other attempts worked fine.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[cmacknz]

Relevant docs: https://osquery.readthedocs.io/en/stable/deployment/extensions/

Extensions Binary Permissions
First, a note: the osquery agent will refuse to load an extension executable from the filesystem if the file's permissions allow write or modify by non-privileged accounts. Before loading an extension, change the owner of the your_extension.ext file to be the root account.

On Windows, because of permission inheritance, just changing the owner of a file is not sufficient. You must also change the owner of the parent directory, remove all inherited DACLs, and disable inheritance. For example, if your osquery extensions are in the .\Extensions directory, the following commands will set permissions that satisfy osquery:

icacls .\Extensions /setowner Administrators /t
icacls .\Extensions /grant Administrators:f /t
icacls .\Extensions /inheritance:r /t
icacls .\Extensions /inheritance:d /t

[cmacknz]

We need to test whether this happens with fresh 8.17.1 installs or exclusively with upgrades.

We don't have any internal reports of this yet since OSQuery is enabled in our internal InfoSec deployments of agent but I've reached out to double check.

[belimawr]

We need to test whether this happens with fresh 8.17.1 installs or exclusively with upgrades.

I'll try a fresh install and report here my findings.

[belimawr]

I've just tested a fresh install from the zip and Osquery works without any problem.

[cmacknz]

There are apparently 5 machines in our internal InfoSec Fleet that have this Will not autoload extension with unsafe directory permissions.

It feels like this problem doesn't happen every time, consistent with Tiago being able to reproduce it 1 out of 3 times as mentioned in our internal support case about this problem.

[belimawr]

I'm still having a hard time reproducing it, yesterday I wrote an integration test to keep trying this upgrade scenario, but even with that running multiple time I have not managed to reproduce/gather more information about the failure. :/