Merge dev -> master #1355
Merged
Merge dev -> master #1355
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…rack-sector-upload
It turns out that Chris 's hunch was right and that we are not tracking sector upload estimates at times we should be, even worse we currently don't track anything at times we should be penalising the host for being slow. I added explicit logging for when we penalise a host for being (super) slow, so going to follow this up on my node to be sure. Overall I thought it was a little tricky to come up with a good way to fix it and to write a solid unit and/or integration test for it. If we really want to check the exact errors we would need an integration test but that would involve a level of orchestration we are currently not capable of doing from an integration test. The unit test I came up with is not exactly what I was hoping for but it does cover all branches. Closes #1167
### Minor Changes - 49ff7435: The configuration page now recommends specific changes to settings to match with more hosts. Closes <#1137> - 49ff7435: Configuration settings that the user has not modified will be kept in sync with remote values that change. Closes <SiaFoundation/hostd#286> ### Patch Changes - 49ff7435: Fixed a bug where hovering over the max storage or upload price tooltips would crash the app.
Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.19.0 to 0.20.0. - [Commits](golang/sys@v0.19.0...v0.20.0) --- updated-dependencies: - dependency-name: golang.org/x/sys dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.19.0 to 0.20.0. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/golang/sys/commit/7d69d983c4522784860c781a0d7b80408fdc0cd1"><code>7d69d98</code></a> unix: extend support for z/OS</li> <li><a href="https://github.com/golang/sys/commit/77580903240cde87369d3ea876dbb47e76e48905"><code>7758090</code></a> cpu: add support for sve2 detection</li> <li><a href="https://github.com/golang/sys/commit/9a28524796a519b225fedd6aaaf4b1bf6c06c002"><code>9a28524</code></a> windows: drop go version tags for unsupported versions</li> <li><a href="https://github.com/golang/sys/commit/27dc90bcf4d797251506d82bd56307ca2c13d59c"><code>27dc90b</code></a> unix: update to Linux kernel 6.4</li> <li>See full diff in <a href="https://github.com/golang/sys/compare/v0.19.0...v0.20.0">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details>
Bumps [golang.org/x/term](https://github.com/golang/term) from 0.19.0 to 0.20.0. - [Commits](golang/term@v0.19.0...v0.20.0) --- updated-dependencies: - dependency-name: golang.org/x/term dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [golang.org/x/term](https://github.com/golang/term) from 0.19.0 to 0.20.0. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/golang/term/commit/46c790f81f1f50148a57f7ddf0c637b84ff2f0e6"><code>46c790f</code></a> go.mod: update golang.org/x dependencies</li> <li>See full diff in <a href="https://github.com/golang/term/compare/v0.19.0...v0.20.0">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details>
Signed-off-by: snoppy <michaleli@foxmail.com>
fix a typo `cient` -> `client`
The worker cache requires setting up the worker in a way we did not have
to do before, namely registering webhooks with the bus so the worker
gets notified when certain events take place on the bus. In single node
setups or when you're on localhost everything works fine but in a
distributed setup we are missing a way to define where the bus can reach
the worker API. I came up with `serviceAddress` for now but I've gone
back and forth quite a bit. I like `serviceAddress` because it ties in
closely with docker services, I think we should call it
`something-address` to be consistent with the rest of the config.
I verified it using the following `docker-compose` file:
```
version: "3.9"
services:
bus:
build:
context: .
dockerfile: ./docker/Dockerfile
container_name: bus
ports:
- "9981:9980"
environment:
- RENTERD_AUTOPILOT_ENABLED=false
- RENTERD_WORKER_ENABLED=false
- RENTERD_WORKER_REMOTE_ADDRS=http://worker:9980/api/worker
- RENTERD_WORKER_API_PASSWORD=test
- RENTERD_API_PASSWORD=test
- RENTERD_SEED=[seed here]
worker:
build:
context: .
dockerfile: ./docker/Dockerfile
container_name: worker
ports:
- "9982:9980"
environment:
- RENTERD_AUTOPILOT_ENABLED=false
- RENTERD_BUS_REMOTE_ADDR=http://bus:9980/api/bus
- RENTERD_BUS_API_PASSWORD=test
- RENTERD_WORKER_API_PASSWORD=test
- RENTERD_WORKER_SERVICE_ADDR=http://worker:9980
- RENTERD_API_PASSWORD=test
- RENTERD_SEED=[seed here]
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost:9980/api/worker || exit 1"]
interval: 30s
timeout: 10s
retries: 5
```
Fixes #1302
---------
Co-authored-by: Christopher Schinnerl <chris@sia.tech>
On `gompa` rclone wouldn't work via `localhost` anymore since `HostBucketEnabled` was set to `true`. Problem is without this setting, access per proper domain (e.g. `bucket.example.com`) doesn't work anymore but with it IPs and `localhost` will be rewritten incorrectly. The solution is to allow for configuring `HostBucketBases` which is already supported by `gofakes3` and allows for configuring only specific domains for which the bucket is extracted from the `host`. That way `localhost` and `127.0.0.1` still work but if specified `bucket.example.com` will also work.
`areq` has an alert for unexpected response size (very old host), figured getting rid of the exact math is probably ok, reused `maxMerkleProofResponseSize` to avoid duplicating the const
Added a few more migrations since they are rather small.
The sector roots RPC used to return all sector roots, this was fixed in SiaFoundation/hostd@e6dfc96 and released in https://github.com/SiaFoundation/hostd/releases/tag/v1.0.2-beta.4 . We should ignore the error for old hosts.
- Minor tweak to slab health update to speed it up by making use of indices - Make sure recording a host scans doesn't abort after 30 seconds when db is under heavy load - Make sure recording price tables doesn't abort after 30 seconds when db is under heavy load - Register new price tables in the background, unblocking operations faster and preventing the record operation from being cancelled after a timeout
There's a docs PR following this one, but this one contains some small
tweaks to the way we parse the config on startup. There's some small
pitfalls in our config currently when it comes to setting up `renterd`
as a cluster of nodes.
- setting `worker.Enabled` to `false` when remotes are configured makes
it easier to build conditionals
- appending `/api/worker` to the worker's external addr made it
inconsistent with `RENTERD_BUS_REMOTE_ADDR`
- seed should only be required if you have at least a bus or worker, the
autopilot doesn't require it
I purposefully kept it at that and did not try to refactor too much.
I think for v2 we should consider taking another look at our config
though.
It's not the worst but it's definitely not prettiest either.
This min. requirements cluster setup docker-compose file works.
```yaml
version: '3.9'
services:
bus:
build:
context: .
dockerfile: ./docker/Dockerfile
container_name: renterd_bus
environment:
- RENTERD_SEED=jump illness approve cry fabric alert fly post kind age keep bid
- RENTERD_API_PASSWORD=bus-pass
ports:
- "9980:9980"
- "9981:9981"
worker-1:
build:
context: .
dockerfile: ./docker/Dockerfile
container_name: renterd_worker-1
environment:
- RENTERD_AUTOPILOT_ENABLED=false
- RENTERD_SEED=jump illness approve cry fabric alert fly post kind age keep bid
- RENTERD_API_PASSWORD=worker-pass
- RENTERD_BUS_API_PASSWORD=bus-pass
- RENTERD_BUS_REMOTE_ADDR=http://bus:9980/api/bus
- RENTERD_WORKER_EXTERNAL_ADDR=http://worker-1:9980/api/worker
ports:
- "9982:9980"
depends_on:
- bus
worker-2:
build:
context: .
dockerfile: ./docker/Dockerfile
container_name: renterd_worker-2
environment:
- RENTERD_SEED=jump illness approve cry fabric alert fly post kind age keep bid
- RENTERD_API_PASSWORD=worker-pass
- RENTERD_BUS_API_PASSWORD=bus-pass
- RENTERD_BUS_REMOTE_ADDR=http://bus:9980/api/bus
- RENTERD_WORKER_EXTERNAL_ADDR=http://worker-2:9980/api/worker
ports:
- "9983:9980"
depends_on:
- bus
autopilot:
build:
context: .
dockerfile: ./docker/Dockerfile
container_name: renterd_autopilot
environment:
- RENTERD_API_PASSWORD=autopilot-pass
- RENTERD_BUS_API_PASSWORD=bus-pass
- RENTERD_BUS_REMOTE_ADDR=http://bus:9980/api/bus
- RENTERD_WORKER_API_PASSWORD=<worker-password>
- RENTERD_WORKER_REMOTE_ADDRS=http://worker-1:9980/api/worker;http://worker-2:9980/api/worker
ports:
- "9984:9980"
depends_on:
- bus
- worker-1
- worker-2
```
### Minor Changes - bdbc4d58: The slab migration failed alert now shows health and an object IDs list that includes the file context menu for each object. Closes <#1322> Co-authored-by: ChrisSchinnerl <ChrisSchinnerl@users.noreply.github.com>
I didn't bother writing it out but instead supplied an example compose file with the absolute minimum config requirements per service.
This PR adds a `subnets` column to the `hosts` table. This in an attempt to get rid of the redundant IP churn we saw on `arequipa`. This is under the assumption that the churn is related to the way we deal with intermittent dns resolve issues, which up until now we've treated as redundant IPs because the `ipFilter` returns a simple yes/no bool. Fixes #1342 --------- Co-authored-by: Christopher Schinnerl <chris@sia.tech>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.