fix(backend): Fix .env file read contention on pyro connection setup

[majdyz]

On high concurrent pyro connections initialization, the .env file being on the high read contention causing connection establishment to fail.

Calling Secrets(), Config(), and Settings(), will always trigger a file read. And when this is executed too many times it will cause read contention and pass the file descriptor limit set by the OS. This will trigger retry and causes delay & thundering herd issues. In Unix this is errored out as:

• Too Many Open Files or
• There are only ... file descriptors (hard limit) available

To reproduce the error you can try to do the run locally and run hundreds of concurrent requests.

Changes 🏗️

Move all the usage of Config as a globally initialized variable so it's only called once per process.

Checklist 📋

For code changes:

• I have clearly listed my changes in the PR description
• I have made a test plan
• I have tested my changes according to the test plan:
  • ...

Example test plan

• Create from scratch and execute an agent with at least 3 blocks
• Import an agent from file upload, and confirm it executes correctly
• Upload agent to marketplace
• Import an agent from marketplace and confirm it executes correctly
• Edit an agent from monitor, and confirm it executes correctly

For configuration changes:

• .env.example is updated or already compatible with my changes
• docker-compose.yml is updated or already compatible with my changes
• I have included a list of my configuration changes in the PR description (under Changes)

Examples of configuration changes

• Changing ports
• Adding new services that need to communicate with each other
• Secrets or environment variable changes
• New or infrastructure changes such as databases

[qodo-merge-pro]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review API Changes Several API endpoints were changed from async to sync functions. Need to verify this won't impact request handling performance or concurrency. Configuration Issue The pyro_host variable is used before being defined when constructing the client URI. This could lead to connection errors. Logging Level Change Changed retry failure logging from info to error level. Verify this aligns with logging strategy and won't flood error logs.

[netlify]

✅ Deploy Preview for auto-gpt-docs canceled.

Name	Link
🔨 Latest commit	a73c63a
🔍 Latest deploy log	https://app.netlify.com/sites/auto-gpt-docs/deploys/67440e159b5e7800086aa5d6

[aarushik93]

@majdyz is this related to today's outage? Because we don't read from .env

[majdyz]

@aarushik93 unrelated, just something I found during the local test.

[aarushik93]

Can you explain a bit more, the PR description doesn't really explain to me what the issue is or what its fixing

[majdyz]

@aarushik93 Described in the PR like this:

On high concurrent pyro connections initialization, the .env file being on the high read contention causing connection establishment to fail.

Calling `Secrets()`, `Config()`, and `Settings()`, will always trigger a file read. And when this is executed too many times it will cause read contention and pass the file descriptor limit set by the OS. This will trigger retry and causes delay & thundering herd issues. In Unix this is errored out as:
* `Too Many Open Files` or
* `There are only ... file descriptors (hard limit) available`

Let me know if there is any part that needs to be clarified.

[aarushik93]

Gotcha, thanks that clears it up for me!