Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: run bacalhau in a separate container #438

Merged
merged 7 commits into from
Nov 20, 2024

Conversation

walkah
Copy link
Collaborator

@walkah walkah commented Nov 14, 2024

Summary

Previously, we bundled bacalhau and lilypad in a single container for resource providers. This meant that any crash/exit from lilypad would restart both bacalhau and lilypad. This could create lots of phone-home version checks for bacalhau. This PR also:

  • Adds a HEALTHCHECK to the bacalhau Dockerfile so that we can properly wait for it to boot before starting the RP
  • Moves local dev solver to port 8081 (to avoid conflicts with kubo HTTP gateway on 8080)

Task/Issue reference

Closes: https://github.com/Lilypad-Tech/internal/issues/339

Test plan

  • Integration tests should still run (using docker-compose.dev.yaml).
  • For RPs, grab the latest docker-compose.yaml file and try to run. You should still be able to run jobs.

@cla-bot cla-bot bot added the cla-signed label Nov 14, 2024
@github-actions github-actions bot added the fix label Nov 14, 2024
@walkah walkah marked this pull request as ready for review November 19, 2024 15:34
@walkah walkah requested a review from a team as a code owner November 19, 2024 15:34
@@ -208,7 +208,7 @@ func (executor *BacalhauExecutor) getJobID(
outputError := strings.Join(strings.Fields(strings.Join(splitOutputs[1:], " ")), " ")

if outputError != "" {
return "", fmt.Errorf("error running command %s -> %s, %s", deal.ID, outputError, runOutput)
log.Error().Msgf("error parsing output %s -> %s, %s", deal.ID, outputError, runOutput)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
log.Error().Msgf("error parsing output %s -> %s, %s", deal.ID, outputError, runOutput)
log.Error().Msgf("error found while parsing output %s -> %s, %s", deal.ID, outputError, runOutput)

Possible edit. It's not a parsing error, but an error we found while parsing.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have thoughts about it being ERR vs WARN?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tough call. On the one hand, the possible error we know about (around the version check) is more of a warning. On the other hand, could there be other errors that aren't a warning?

I'd say let's go with the common case we know about and use WARN. If we start seeing other errors, we can revisit, but this may not matter once we have upgraded to the latest Bacalhau.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, once we get fully over to the bacalhau http api, a lot of this code become moot. 👍

@bgins
Copy link
Contributor

bgins commented Nov 19, 2024

Wondering if we may want to update the solver SERVER_PORT to 8081 across the board for consistency. A few other places in the code that could be updated:

Port: GetDefaultServeOptionInt("SERVER_PORT", 8080), //nolint:gomnd

EXPOSE 8080

console.log('export SERVER_PORT=8080')
console.log('export SERVER_URL=http://localhost:8080')

lilypad/stack

Line 237 in fe48d8c

-p 8080:8080 \

Outside of the code, we would need to update:

  • Doppler solver projects
  • Cloudflare tunnels

@walkah
Copy link
Collaborator Author

walkah commented Nov 19, 2024

Wondering if we may want to update the solver SERVER_PORT to 8081 across the board for consistency.

My thinking (and I can be convinced otherwise) was that just in local dev / CI there's potential for port conflicts with kubo on 8080 but in deployments, it's running in a more isolated context. I can definitely to the other updates though!

@bgins
Copy link
Contributor

bgins commented Nov 19, 2024

Wondering if we may want to update the solver SERVER_PORT to 8081 across the board for consistency.

My thinking (and I can be convinced otherwise) was that just in local dev / CI there's potential for port conflicts with kubo on 8080 but in deployments, it's running in a more isolated context. I can definitely to the other updates though!

Yeah, that makes sense! Let's go with it.

Copy link
Contributor

@bgins bgins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Integration tests and jobs running over here with the new setup. ✨

@walkah walkah merged commit 25083a7 into main Nov 20, 2024
4 checks passed
@walkah walkah deleted the walkah/fix-separate-bacalhau-container branch November 20, 2024 19:32
walkah added a commit that referenced this pull request Dec 10, 2024
* fix: run bacalhau ins a separate container

* chore: bump kubo version

* new port for rate limit test

* don't try to call API during tests

* fix errors
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants