Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error on line 58 in commander script #15

Closed
SebastianMoinhinwech opened this issue Feb 16, 2021 · 20 comments
Closed

Error on line 58 in commander script #15

SebastianMoinhinwech opened this issue Feb 16, 2021 · 20 comments

Comments

@SebastianMoinhinwech
Copy link

Hi,

on a clean debian i had had some problems (after switched to explicit root (su - root) everything worked well) the website is stuck on "Almost there! Now we will request a certificate for this website and restart one more time...". I restarted several times, i think the problem is, that the webserver in the docker image isn`t ready. But, here are the results from the logs:

Web Certificate Log

Di 16. Feb 13:31:15 CET 2021
Parsing account key...
Parsing CSR...
Found domains: host.domain.local
Getting directory...
Traceback (most recent call last):
File "/home/labca/acme_tiny.py", line 197, in
main(sys.argv[1:])
File "/home/labca/acme_tiny.py", line 193, in main
signed_crt = get_crt(args.account_key, args.csr, args.acme_dir, log=LOGGER, CA=args.ca, disable_check=args.disable_check, directory_url=args.directory_url, contact=args.contact)
File "/home/labca/acme_tiny.py", line 105, in get_crt
directory, _, _ = _do_request(directory_url, err_msg="Error getting directory")
File "/home/labca/acme_tiny.py", line 45, in _do_request
raise ValueError("{0}:\nUrl: {1}\nData: {2}\nResponse Code: {3}\nResponse: {4}".format(err_msg, url, data, code, resp_data))
ValueError: Error getting directory:
Url: http://host.domain.local
Data: None
Response Code: 502
Response:

<title>502 Bad Gateway</title>

502 Bad Gateway


nginx

LABCA Logs

labca_1 | 2021/02/16 12:33:56 errorHandler: ERROR! On line 58 in commander script
labca_1 | main._hostCommand(0xbd1c00, 0xc00027e0e0, 0xc00024e700, 0xb104bf, 0xc, 0x0, 0x0, 0x0, 0xb3b500)
labca_1 | /go/src/labca/main.go:1513 +0x5a5
labca_1 | main.finalHandler(0xbd1c00, 0xc00027e0e0, 0xc00024e700)
labca_1 | /go/src/labca/main.go:1860 +0xf9
labca_1 | net/http.HandlerFunc.ServeHTTP(0xb3ac58, 0xbd1c00, 0xc00027e0e0, 0xc00024e700)
labca_1 | /usr/local/go/src/net/http/server.go:2042 +0x44
labca_1 | main.authorized.func1(0xbd1c00, 0xc00027e0e0, 0xc00024e700)
labca_1 | /go/src/labca/main.go:2316 +0x230
labca_1 | net/http.HandlerFunc.ServeHTTP(0xc000130b20, 0xbd1c00, 0xc00027e0e0, 0xc00024e700)
labca_1 | /usr/local/go/src/net/http/server.go:2042 +0x44
labca_1 | github.com/gorilla/mux.(*Router).ServeHTTP(0xc00020a0c0, 0xbd1c00, 0xc00027e0e0, 0xc00024e500)
labca_1 | /go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210 +0xd3
labca_1 | net/http.serverHandler.ServeHTTP(0xc0001062a0, 0xbd1c00, 0xc00027e0e0, 0xc00024e500)
labca_1 | /usr/local/go/src/net/http/server.go:2843 +0xa3
labca_1 | net/http.(*conn).serve(0xc0002540a0, 0xbd32c0, 0xc00023df00)
labca_1 | /usr/local/go/src/net/http/server.go:1925 +0x8ad
labca_1 | created by net/http.(*Server).Serve
labca_1 | /usr/local/go/src/net/http/server.go:2969 +0x36c
labca_1 | 2021/02/16 12:33:56 http: superfluous response.WriteHeader call from main.finalHandler (main.go:1861)
labca_1 | 2021/02/16 12:34:11 ERROR: Message from server: 'ERROR! On line 58 in commander script
labca_1 | '
labca_1 | 2021/02/16 12:34:11 errorHandler: ERROR! On line 58 in commander script
labca_1 | main._hostCommand(0xbd1c00, 0xc00027e2a0, 0xc00024ed00, 0xb104bf, 0xc, 0x0, 0x0, 0x0, 0xb3b500)
labca_1 | /go/src/labca/main.go:1513 +0x5a5
labca_1 | main.finalHandler(0xbd1c00, 0xc00027e2a0, 0xc00024ed00)
labca_1 | /go/src/labca/main.go:1860 +0xf9
labca_1 | net/http.HandlerFunc.ServeHTTP(0xb3ac58, 0xbd1c00, 0xc00027e2a0, 0xc00024ed00)
labca_1 | /usr/local/go/src/net/http/server.go:2042 +0x44
labca_1 | main.authorized.func1(0xbd1c00, 0xc00027e2a0, 0xc00024ed00)
labca_1 | /go/src/labca/main.go:2316 +0x230
labca_1 | net/http.HandlerFunc.ServeHTTP(0xc000130d20, 0xbd1c00, 0xc00027e2a0, 0xc00024ed00)
labca_1 | /usr/local/go/src/net/http/server.go:2042 +0x44
labca_1 | github.com/gorilla/mux.(*Router).ServeHTTP(0xc00020a0c0, 0xbd1c00, 0xc00027e2a0, 0xc00024eb00)
labca_1 | /go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210 +0xd3
labca_1 | net/http.serverHandler.ServeHTTP(0xc0001062a0, 0xbd1c00, 0xc00027e2a0, 0xc00024eb00)
labca_1 | /usr/local/go/src/net/http/server.go:2843 +0xa3
labca_1 | net/http.(*conn).serve(0xc0002541e0, 0xbd32c0, 0xc00043c180)
labca_1 | /usr/local/go/src/net/http/server.go:1925 +0x8ad
labca_1 | created by net/http.(*Server).Serve
labca_1 | /usr/local/go/src/net/http/server.go:2969 +0x36c
labca_1 | 2021/02/16 12:34:11 http: superfluous response.WriteHeader call from main.finalHandler (main.go:1861)
labca_1 | 2021/02/16 12:34:39 GET /logs/web
labca_1 | 2021/02/16 12:34:39 GET /ws?logType=web
labca_1 | 2021/02/16 12:35:00 GET /logs/weberr
labca_1 | 2021/02/16 12:35:25 GET /logs/labca
labca_1 | 2021/02/16 12:35:25 GET /ws?logType=labca

Thanks for this project and your support.

@hakwerk
Copy link
Owner

hakwerk commented Feb 16, 2021

Could you please check the boulder logs: either via https://host.domain.local/admin/logs/boulder or on the commandline (cd /home/labca/boulder; docker-compose logs boulder).
For some reason the ACME application is not coming up.

@SebastianMoinhinwech
Copy link
Author

Mmh, First look at the logs and it seems the database server isn`t coming up:

boulder_1 | I210216192139 ocsp-responder mbD4lQM Loading OCSP Database for CA Cert: labca/test-ca.pem
boulder_1 | �[31m�[1mE210216192139 ocsp-responder t46r4gU [AUDIT] Could not connect to database: dial tcp: i/o timeout�[0m
boulder_1 | Could not connect to database: dial tcp: i/o timeout
boulder_1 |
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 |
boulder_1 |
boulder_1 | These processes exited early (check above for their output):
boulder_1 | '('./bin/ocsp-responder', '--config', 'labca/config/ocsp-responder.json')' with pid 380 exited 1
boulder_1 | I210216192139 log-validator kKG6cwA Caught SIGTERM
boulder_1 | I210216192139 log-validator htX9nwc Exiting
boulder_1 | 2021/02/16 19:21:39 Caught SIGTERM
boulder_1 | pebble-challtestsrv - 2021/02/16 19:21:39 Caught signals. Shutting down
boulder_1 | pebble-challtestsrv - 2021/02/16 19:21:39 http: Server closed
boulder_1 | pebble-challtestsrv - 2021/02/16 19:21:40 http: Server closed
boulder_1 | pebble-challtestsrv - 2021/02/16 19:21:40 http: Server closed
boulder_1 | pebble-challtestsrv - 2021/02/16 19:21:41 Shutting down management server
boulder_1 | pebble-challtestsrv - 2021/02/16 19:21:41 http: Server closed
boulder_1 | pebble-challtestsrv - 2021/02/16 19:21:41 Goodbye!
boulder_1 | 2021/02/16 19:21:41 Exiting

@SebastianMoinhinwech
Copy link
Author

boulder_1 | rsyslogd: error during parsing file /etc/rsyslog.d/boulder.rsyslog.conf, on or before line 1: omfile: warning: default template was already set via legacy directive - may lead to inconsistent results. [v8.1901.0 try https://www.rsyslog.com/e/2207 ] boulder_1 | rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted. boulder_1 | rsyslogd: activation of module imklog failed [v8.1901.0 try https://www.rsyslog.com/e/2145 ] boulder_1 | Starting enhanced syslogd: rsyslogd. boulder_1 | Connected to boulder-mysql:3306 boulder_1 | Database boulder_sa_test already exists - skipping create boulder_1 | goose: no migrations to run. current version: 20200609125504 boulder_1 | migrated boulder_sa_test database with ./sa/_db/ boulder_1 | added users to boulder_sa_test boulder_1 | Database boulder_sa_integration already exists - skipping create boulder_1 | goose: no migrations to run. current version: 20200609125504 boulder_1 | migrated boulder_sa_integration database with ./sa/_db/ boulder_1 | added users to boulder_sa_integration boulder_1 | created all databases boulder_1 | CKR_SLOT_ID_INVALID: Slot 0 does not exist. boulder_1 | Found slot 2091935369 with matching token label. boulder_1 | The key pair has been imported. boulder_1 | CKR_SLOT_ID_INVALID: Slot 1 does not exist. boulder_1 | Found slot 1306638887 with matching token label. boulder_1 | The key pair has been imported. boulder_1 | GOBIN=/go/src/github.com/letsencrypt/boulder/bin GO111MODULE=on go install -mod=vendor -tags "integration" ./... boulder_1 | cp /go/src/github.com/letsencrypt/boulder/bin/boulder-va /go/src/github.com/letsencrypt/boulder/bin/boulder-remoteva boulder_1 | pebble-challtestsrv - 2021/02/16 19:26:41 Creating HTTP-01 challenge server on 10.77.77.77:5002 boulder_1 | pebble-challtestsrv - 2021/02/16 19:26:41 Creating HTTPS HTTP-01 challenge server on 10.77.77.77:5001 boulder_1 | pebble-challtestsrv - 2021/02/16 19:26:41 Creating TCP and UDP DNS-01 challenge server on :8053 boulder_1 | pebble-challtestsrv - 2021/02/16 19:26:41 Creating TCP and UDP DNS-01 challenge server on :8054 boulder_1 | pebble-challtestsrv - 2021/02/16 19:26:41 Creating TLS-ALPN-01 challenge server on 10.88.88.88:5001 boulder_1 | pebble-challtestsrv - 2021/02/16 19:26:41 Answering A queries with 10.77.77.77 by default boulder_1 | pebble-challtestsrv - 2021/02/16 19:26:41 Starting challenge servers boulder_1 | pebble-challtestsrv - 2021/02/16 19:26:41 Starting management server on :8055 boulder_1 | I210216192641 mail-test-srv _LzLggc mail-test-srv: Got connection from 127.0.0.1:40726 boulder_1 | 2021/02/16 19:26:41 mail-test-srv: 127.0.0.1:40726: readline: EOF boulder_1 | 2021/02/16 19:26:42 ct-test-srv on :4501 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEKtnFevaXV/kB8dmhCNZHmxKVLcHX1plaAsY9LrKilhYxdmQZiu36LvAvosTsqMVqRK9a96nC8VaxAdaHUbM8EA== boulder_1 | 2021/02/16 19:26:42 ct-test-srv on :4512 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEFRu37ZRLg8lT4rVQwMwh4oAOpXb4Sx+9hgQ+JFCjmAv3oDV+sDOMsC7hULkGTn+LB5L1SRo/XIY4Kw5V+nFXgg== boulder_1 | 2021/02/16 19:26:42 ct-test-srv on :4510 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEyw1HymhJkuxSIgt3gqW3sVXqMqB3EFsXcMfPFo0vYwjNiRmCJDXKsR0Flp7MAK+wc3X/7Hpc8liUbMhPet7tEA== boulder_1 | 2021/02/16 19:26:42 ct-test-srv on :4500 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEYggOxPnPkzKBIhTacSYoIfnSL2jPugcbUKx83vFMvk5gKAz/AGe87w20riuPwEGn229hKVbEKHFB61NIqNHC3Q== boulder_1 | 2021/02/16 19:26:42 ct-test-srv on :4511 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEFRu37ZRLg8lT4rVQwMwh4oAOpXb4Sx+9hgQ+JFCjmAv3oDV+sDOMsC7hULkGTn+LB5L1SRo/XIY4Kw5V+nFXgg== boulder_1 | I210216192642 akamai-purger 8tvK_Q8 Versions: akamai-purger=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified) boulder_1 | I210216192642 boulder-remoteva 0MLlrww Versions: boulder-remoteva=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified) boulder_1 | Connecting to va1.boulder:9097 health service boulder_1 | I210216192642 boulder-remoteva 0MLlrww Versions: boulder-remoteva=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified) boulder_1 | Connecting to va1.boulder:9098 health service boulder_1 | Connecting to sa1.boulder:9095 health service boulder_1 | got error connecting to health service sa1.boulder:9095: grpc.health.v1.Health.Check timed out after 999 ms boulder_1 | Connecting to sa1.boulder:9095 health service boulder_1 | I210216192644 boulder-sa kcXg9QE Versions: boulder-sa=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified) boulder_1 | got error connecting to health service sa1.boulder:9095: grpc.health.v1.Health.Check timed out after 1000 ms boulder_1 | Connecting to sa1.boulder:9095 health service boulder_1 | I210216192646 ocsp-responder tLqf8AI Versions: ocsp-responder=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified) boulder_1 | I210216192646 ocsp-responder mbD4lQM Loading OCSP Database for CA Cert: labca/test-ca.pem boulder_1 | Connecting to nonce2.boulder:9101 health service boulder_1 | got error connecting to health service nonce2.boulder:9101: grpc.health.v1.Health.Check timed out after 998 ms boulder_1 | Connecting to nonce2.boulder:9101 health service boulder_1 | I210216192648 nonce-service l6_-0Q4 Versions: nonce-service=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified) boulder_1 | got error connecting to health service nonce2.boulder:9101: grpc.health.v1.Health.Check timed out after 1000 ms boulder_1 | Connecting to nonce2.boulder:9101 health service boulder_1 | I210216192649 boulder-publisher 6uKbhAo Versions: boulder-publisher=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified) boulder_1 | Connecting to publisher2.boulder:9091 health service boulder_1 | I210216192649 log-validator g5CP7Ac Waiting for /var/log/boulder-wfe2.log to appear... boulder_1 | I210216192649 log-validator 44ONuAY Waiting for /var/log/boulder-wfe.log to appear... boulder_1 | Connecting to publisher1.boulder:9091 health service boulder_1 | I210216192649 boulder-publisher 6uKbhAo Versions: boulder-publisher=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified) boulder_1 | got error connecting to health service publisher1.boulder:9091: grpc.health.v1.Health.Check timed out after 1000 ms boulder_1 | Connecting to publisher1.boulder:9091 health service boulder_1 | I210216192650 nonce-service l6_-0Q4 Versions: nonce-service=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified) boulder_1 | Connecting to nonce1.boulder:9101 health service boulder_1 | I210216192650 boulder-sa kcXg9QE Versions: boulder-sa=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified) boulder_1 | Connecting to sa2.boulder:9095 health service boulder_1 | I210216192651 boulder-va l_rfyQ8 Versions: boulder-va=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified) boulder_1 | Connecting to va2.boulder:9092 health service boulder_1 | I210216192651 boulder-va l_rfyQ8 Versions: boulder-va=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified) boulder_1 | Connecting to va1.boulder:9092 health service boulder_1 | Connecting to ca1.boulder:9093 health service boulder_1 | got error connecting to health service ca1.boulder:9093: grpc.health.v1.Health.Check timed out after 999 ms boulder_1 | Connecting to ca1.boulder:9093 health service boulder_1 | I210216192652 boulder-ca 5u-yxAg Versions: boulder-ca=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified) boulder_1 | I210216192652 boulder-ca 9OXLlwI loading hostname policy, sha256: 0be6f6061a1041f866b71037f1aa96e0335be2035e0e8657aa203129f2d13f8d boulder_1 | got error connecting to health service ca1.boulder:9093: grpc.health.v1.Health.Check timed out after 1000 ms boulder_1 | Connecting to ca1.boulder:9093 health service boulder_1 | got error connecting to health service ca1.boulder:9093: grpc.health.v1.Health.Check timed out after 1000 ms boulder_1 | Connecting to ca1.boulder:9093 health service boulder_1 | got error connecting to health service ca1.boulder:9093: grpc.health.v1.Health.Check timed out after 1000 ms boulder_1 | Connecting to ca1.boulder:9093 health service boulder_1 | got error connecting to health service ca1.boulder:9093: grpc.health.v1.Health.Check timed out after 1000 ms boulder_1 | Connecting to ca1.boulder:9093 health service boulder_1 | got error connecting to health service ca1.boulder:9093: grpc.health.v1.Health.Check timed out after 1000 ms boulder_1 | Connecting to ca1.boulder:9093 health service boulder_1 | got error connecting to health service ca1.boulder:9093: grpc.health.v1.Health.Check timed out after 1000 ms boulder_1 | Connecting to ca1.boulder:9093 health service boulder_1 | got error connecting to health service ca1.boulder:9093: grpc.health.v1.Health.Check timed out after 1000 ms boulder_1 | Connecting to ca1.boulder:9093 health service boulder_1 | got error connecting to health service ca1.boulder:9093: grpc.health.v1.Health.Check timed out after 1000 ms boulder_1 | Connecting to ca1.boulder:9093 health service boulder_1 | got error connecting to health service ca1.boulder:9093: grpc.health.v1.Health.Check timed out after 894 ms boulder_1 | timed out waiting for ca1.boulder:9093 health check boulder_1 | �[31m�[1mE210216192701 health-checker xIqPkwY [AUDIT] timed out waiting for ca1.boulder:9093 health check�[0m boulder_1 | Starting service mail-test-srv boulder_1 | Starting service akamai-test-srv boulder_1 | Starting service ct-test-srv boulder_1 | Starting service sd-test-srv boulder_1 | Starting service akamai-purger boulder_1 | Waiting for debug port 9666 (./bin/akamai-purger --config labca/config/akamai-purger.json) boulder_1 | Waiting for debug port 9666 (./bin/akamai-purger --config labca/config/akamai-purger.json) boulder_1 | Waiting for debug port 9666 (./bin/akamai-purger --config labca/config/akamai-purger.json) boulder_1 | Starting service boulder-remoteva-a boulder_1 | Starting service boulder-remoteva-b boulder_1 | Starting service boulder-sa-1 boulder_1 | Starting service ocsp-responder boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json) boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json) boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json) boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json) boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json) boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json) boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json) boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json) boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json) boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json) boulder_1 | Starting service nonce-service-zinc boulder_1 | Starting service boulder-publisher-2 boulder_1 | Starting service log-validator boulder_1 | Waiting for debug port 8016 (./bin/log-validator --config labca/config/log-validator.json) boulder_1 | Starting service boulder-publisher-1 boulder_1 | Starting service nonce-service-taro boulder_1 | Starting service boulder-sa-2 boulder_1 | Starting service boulder-va-2 boulder_1 | Starting service boulder-va-1 boulder_1 | Starting service boulder-ca-a boulder_1 | Error starting service boulder-ca-a: Command '['./bin/health-checker', '-addr', 'ca1.boulder:9093', '-config', 'labca/config/health-checker.json']' returned non-zero exit status 1. boulder_1 | I210216192701 boulder-ca kKG6cwA Caught SIGTERM boulder_1 | I210216192701 boulder-ca htX9nwc Exiting boulder_1 | I210216192701 boulder-va kKG6cwA Caught SIGTERM boulder_1 | I210216192701 boulder-va htX9nwc Exiting boulder_1 | I210216192701 boulder-va kKG6cwA Caught SIGTERM boulder_1 | I210216192701 boulder-va htX9nwc Exiting boulder_1 | I210216192701 boulder-sa kKG6cwA Caught SIGTERM boulder_1 | I210216192701 boulder-sa htX9nwc Exiting boulder_1 | I210216192701 nonce-service kKG6cwA Caught SIGTERM boulder_1 | I210216192701 nonce-service htX9nwc Exiting boulder_1 | I210216192701 boulder-publisher kKG6cwA Caught SIGTERM boulder_1 | I210216192701 boulder-publisher htX9nwc Exiting boulder_1 | I210216192701 log-validator kKG6cwA Caught SIGTERM boulder_1 | I210216192701 log-validator htX9nwc Exiting boulder_1 | I210216192701 boulder-publisher kKG6cwA Caught SIGTERM boulder_1 | I210216192701 boulder-publisher htX9nwc Exiting boulder_1 | I210216192701 nonce-service kKG6cwA Caught SIGTERM boulder_1 | I210216192701 nonce-service htX9nwc Exiting boulder_1 | I210216192701 ocsp-responder kKG6cwA Caught SIGTERM boulder_1 | I210216192702 boulder-sa kKG6cwA Caught SIGTERM boulder_1 | I210216192702 boulder-sa htX9nwc Exiting boulder_1 | I210216192702 boulder-remoteva kKG6cwA Caught SIGTERM boulder_1 | I210216192702 boulder-remoteva htX9nwc Exiting boulder_1 | I210216192702 boulder-remoteva kKG6cwA Caught SIGTERM boulder_1 | I210216192702 boulder-remoteva htX9nwc Exiting boulder_1 | I210216192702 akamai-purger kKG6cwA Caught SIGTERM boulder_1 | I210216192702 akamai-purger 2f7Ciwo Shutting down; queue is already empty. boulder_1 | I210216192702 akamai-purger htX9nwc Exiting boulder_1 | pebble-challtestsrv - 2021/02/16 19:27:02 Caught signals. Shutting down boulder_1 | 2021/02/16 19:27:02 Caught SIGTERM boulder_1 | pebble-challtestsrv - 2021/02/16 19:27:02 http: Server closed boulder_1 | pebble-challtestsrv - 2021/02/16 19:27:02 http: Server closed boulder_1 | pebble-challtestsrv - 2021/02/16 19:27:02 http: Server closed boulder_1 | pebble-challtestsrv - 2021/02/16 19:27:03 Shutting down management server boulder_1 | pebble-challtestsrv - 2021/02/16 19:27:03 http: Server closed boulder_1 | pebble-challtestsrv - 2021/02/16 19:27:03 Goodbye! boulder_1 | 2021/02/16 19:27:03 Exiting

@hakwerk
Copy link
Owner

hakwerk commented Feb 16, 2021

From

boulder_1 | rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.

it appears that you may be running in an unprivileged container?
If so, try it in a VM or in a privileged container.

@hakwerk
Copy link
Owner

hakwerk commented Feb 17, 2021

Hmmm, never mind that - I see the same message also in an instance that is working fine

@hakwerk
Copy link
Owner

hakwerk commented Feb 17, 2021

Maybe try running the updater manually (/home/labca/labca/install) to force refreshing everything

@SebastianMoinhinwech
Copy link
Author

Hi, did the update, but again, same result:

boulder_1 | rsyslogd: error during parsing file /etc/rsyslog.d/boulder.rsyslog.conf, on or before line 1: omfile: warning: default template was already set via legacy directive - may lead to inconsistent results. [v8.1901.0 try https://www.rsyslog.com/e/2207 ]
boulder_1 | rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
boulder_1 | rsyslogd: activation of module imklog failed [v8.1901.0 try https://www.rsyslog.com/e/2145 ]
boulder_1 | Starting enhanced syslogd: rsyslogd.
boulder_1 | Connected to boulder-mysql:3306
boulder_1 | Database boulder_sa_test already exists - skipping create
boulder_1 | goose: no migrations to run. current version: 20200609125504
boulder_1 | migrated boulder_sa_test database with ./sa/db/
boulder_1 | added users to boulder_sa_test
boulder_1 | Database boulder_sa_integration already exists - skipping create
boulder_1 | goose: no migrations to run. current version: 20200609125504
boulder_1 | migrated boulder_sa_integration database with ./sa/db/
boulder_1 | added users to boulder_sa_integration
boulder_1 | created all databases
boulder_1 | CKR_SLOT_ID_INVALID: Slot 0 does not exist.
boulder_1 | Found slot 2091935369 with matching token label.
boulder_1 | The key pair has been imported.
boulder_1 | CKR_SLOT_ID_INVALID: Slot 1 does not exist.
boulder_1 | Found slot 1306638887 with matching token label.
boulder_1 | The key pair has been imported.
boulder_1 | GOBIN=/go/src/github.com/letsencrypt/boulder/bin GO111MODULE=on go install -mod=vendor -tags "integration" ./...
boulder_1 | cp /go/src/github.com/letsencrypt/boulder/bin/boulder-va /go/src/github.com/letsencrypt/boulder/bin/boulder-remoteva
boulder_1 | pebble-challtestsrv - 2021/02/17 07:41:32 Creating HTTP-01 challenge server on 10.77.77.77:5002
boulder_1 | pebble-challtestsrv - 2021/02/17 07:41:32 Creating HTTPS HTTP-01 challenge server on 10.77.77.77:5001
boulder_1 | pebble-challtestsrv - 2021/02/17 07:41:32 Creating TCP and UDP DNS-01 challenge server on :8053
boulder_1 | pebble-challtestsrv - 2021/02/17 07:41:32 Creating TCP and UDP DNS-01 challenge server on :8054
boulder_1 | pebble-challtestsrv - 2021/02/17 07:41:32 Creating TLS-ALPN-01 challenge server on 10.88.88.88:5001
boulder_1 | pebble-challtestsrv - 2021/02/17 07:41:32 Answering A queries with 10.77.77.77 by default
boulder_1 | pebble-challtestsrv - 2021/02/17 07:41:32 Starting challenge servers
boulder_1 | pebble-challtestsrv - 2021/02/17 07:41:32 Starting management server on :8055
boulder_1 | 2021/02/17 07:41:32 ct-test-srv on :4510 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEyw1HymhJkuxSIgt3gqW3sVXqMqB3EFsXcMfPFo0vYwjNiRmCJDXKsR0Flp7MAK+wc3X/7Hpc8liUbMhPet7tEA==
boulder_1 | 2021/02/17 07:41:32 ct-test-srv on :4511 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEFRu37ZRLg8lT4rVQwMwh4oAOpXb4Sx+9hgQ+JFCjmAv3oDV+sDOMsC7hULkGTn+LB5L1SRo/XIY4Kw5V+nFXgg==
boulder_1 | 2021/02/17 07:41:32 ct-test-srv on :4512 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEFRu37ZRLg8lT4rVQwMwh4oAOpXb4Sx+9hgQ+JFCjmAv3oDV+sDOMsC7hULkGTn+LB5L1SRo/XIY4Kw5V+nFXgg==
boulder_1 | 2021/02/17 07:41:32 ct-test-srv on :4500 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEYggOxPnPkzKBIhTacSYoIfnSL2jPugcbUKx83vFMvk5gKAz/AGe87w20riuPwEGn229hKVbEKHFB61NIqNHC3Q==
boulder_1 | 2021/02/17 07:41:32 ct-test-srv on :4501 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEKtnFevaXV/kB8dmhCNZHmxKVLcHX1plaAsY9LrKilhYxdmQZiu36LvAvosTsqMVqRK9a96nC8VaxAdaHUbM8EA==
boulder_1 | I210217074132 mail-test-srv tO-DkAk mail-test-srv: Got connection from 127.0.0.1:34908
boulder_1 | 2021/02/17 07:41:32 mail-test-srv: 127.0.0.1:34908: readline: EOF
boulder_1 | I210217074133 akamai-purger 8tvK_Q8 Versions: akamai-purger=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to nonce2.boulder:9101 health service
boulder_1 | I210217074133 nonce-service l6
-0Q4 Versions: nonce-service=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | got error connecting to health service nonce2.boulder:9101: grpc.health.v1.Health.Check timed out after 1000 ms
boulder_1 | Connecting to nonce2.boulder:9101 health service
boulder_1 | I210217074134 nonce-service l6
-0Q4 Versions: nonce-service=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to nonce1.boulder:9101 health service
boulder_1 | I210217074134 ocsp-responder tLqf8AI Versions: ocsp-responder=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | I210217074134 ocsp-responder mbD4lQM Loading OCSP Database for CA Cert: labca/test-ca.pem
boulder_1 | I210217074135 log-validator g5CP7Ac Waiting for /var/log/boulder-wfe2.log to appear...
boulder_1 | I210217074135 log-validator 44ONuAY Waiting for /var/log/boulder-wfe.log to appear...
boulder_1 | I210217074135 boulder-remoteva 0MLlrww Versions: boulder-remoteva=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to va1.boulder:9098 health service
boulder_1 | I210217074135 boulder-remoteva 0MLlrww Versions: boulder-remoteva=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to va1.boulder:9097 health service
boulder_1 | Connecting to publisher1.boulder:9091 health service
boulder_1 | I210217074136 boulder-publisher 6uKbhAo Versions: boulder-publisher=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | got error connecting to health service publisher1.boulder:9091: grpc.health.v1.Health.Check timed out after 1000 ms
boulder_1 | Connecting to publisher1.boulder:9091 health service
boulder_1 | I210217074137 boulder-va l_rfyQ8 Versions: boulder-va=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to va2.boulder:9092 health service
boulder_1 | I210217074137 boulder-publisher 6uKbhAo Versions: boulder-publisher=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to publisher2.boulder:9091 health service
boulder_1 | Connecting to sa1.boulder:9095 health service
boulder_1 | I210217074137 boulder-sa kcXg9QE Versions: boulder-sa=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | got error connecting to health service sa1.boulder:9095: grpc.health.v1.Health.Check timed out after 1000 ms
boulder_1 | Connecting to sa1.boulder:9095 health service
boulder_1 | I210217074138 boulder-sa kcXg9QE Versions: boulder-sa=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to sa2.boulder:9095 health service
boulder_1 | Connecting to ca2.boulder:9093 health service
boulder_1 | I210217074139 boulder-ca 5u-yxAg Versions: boulder-ca=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | I210217074139 boulder-ca 9OXLlwI loading hostname policy, sha256: 0be6f6061a1041f866b71037f1aa96e0335be2035e0e8657aa203129f2d13f8d
boulder_1 | got error connecting to health service ca2.boulder:9093: grpc.health.v1.Health.Check timed out after 1000 ms
boulder_1 | Connecting to ca2.boulder:9093 health service
boulder_1 | got error connecting to health service ca2.boulder:9093: grpc.health.v1.Health.Check timed out after 1000 ms
boulder_1 | Connecting to ca2.boulder:9093 health service
boulder_1 | got error connecting to health service ca2.boulder:9093: grpc.health.v1.Health.Check timed out after 1000 ms
boulder_1 | Connecting to ca2.boulder:9093 health service
boulder_1 | got error connecting to health service ca2.boulder:9093: grpc.health.v1.Health.Check timed out after 1000 ms
boulder_1 | Connecting to ca2.boulder:9093 health service
boulder_1 | got error connecting to health service ca2.boulder:9093: grpc.health.v1.Health.Check timed out after 1000 ms
boulder_1 | Connecting to ca2.boulder:9093 health service
boulder_1 | got error connecting to health service ca2.boulder:9093: grpc.health.v1.Health.Check timed out after 999 ms
boulder_1 | Connecting to ca2.boulder:9093 health service
boulder_1 | got error connecting to health service ca2.boulder:9093: grpc.health.v1.Health.Check timed out after 1000 ms
boulder_1 | Connecting to ca2.boulder:9093 health service
boulder_1 | got error connecting to health service ca2.boulder:9093: grpc.health.v1.Health.Check timed out after 1000 ms
boulder_1 | Connecting to ca2.boulder:9093 health service
boulder_1 | got error connecting to health service ca2.boulder:9093: grpc.health.v1.Health.Check timed out after 1000 ms
boulder_1 | Connecting to ca2.boulder:9093 health service
boulder_1 | got error connecting to health service ca2.boulder:9093: grpc.health.v1.Health.Check timed out after 889 ms
boulder_1 | timed out waiting for ca2.boulder:9093 health check
boulder_1 | �[31m�[1mE210217074149 health-checker h9Hfjw0 [AUDIT] timed out waiting for ca2.boulder:9093 health check�[0m
boulder_1 | Starting service ct-test-srv
boulder_1 | Starting service akamai-test-srv
boulder_1 | Starting service mail-test-srv
boulder_1 | Starting service sd-test-srv
boulder_1 | Starting service akamai-purger
boulder_1 | Waiting for debug port 9666 (./bin/akamai-purger --config labca/config/akamai-purger.json)
boulder_1 | Waiting for debug port 9666 (./bin/akamai-purger --config labca/config/akamai-purger.json)
boulder_1 | Starting service nonce-service-zinc
boulder_1 | Starting service nonce-service-taro
boulder_1 | Starting service ocsp-responder
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Starting service log-validator
boulder_1 | Waiting for debug port 8016 (./bin/log-validator --config labca/config/log-validator.json)
boulder_1 | Waiting for debug port 8016 (./bin/log-validator --config labca/config/log-validator.json)
boulder_1 | Starting service boulder-remoteva-b
boulder_1 | Starting service boulder-remoteva-a
boulder_1 | Starting service boulder-publisher-1
boulder_1 | Starting service boulder-va-2
boulder_1 | Starting service boulder-publisher-2
boulder_1 | Starting service boulder-sa-1
boulder_1 | Starting service boulder-sa-2
boulder_1 | Starting service boulder-ca-b
boulder_1 | Error starting service boulder-ca-b: Command '['./bin/health-checker', '-addr', 'ca2.boulder:9093', '-config', 'labca/config/health-checker.json']' returned non-zero exit status 1.
boulder_1 | I210217074149 boulder-sa kKG6cwA Caught SIGTERM
boulder_1 | I210217074149 boulder-sa htX9nwc Exiting
boulder_1 | I210217074149 boulder-sa kKG6cwA Caught SIGTERM
boulder_1 | I210217074149 boulder-sa htX9nwc Exiting
boulder_1 | I210217074149 boulder-publisher kKG6cwA Caught SIGTERM
boulder_1 | I210217074149 boulder-va kKG6cwA Caught SIGTERM
boulder_1 | I210217074149 boulder-va htX9nwc Exiting
boulder_1 | I210217074149 boulder-publisher kKG6cwA Caught SIGTERM
boulder_1 | I210217074149 boulder-publisher htX9nwc Exiting
boulder_1 | I210217074149 boulder-remoteva kKG6cwA Caught SIGTERM
boulder_1 | I210217074149 boulder-remoteva htX9nwc Exiting
boulder_1 | I210217074149 boulder-remoteva kKG6cwA Caught SIGTERM
boulder_1 | I210217074149 boulder-remoteva htX9nwc Exiting
boulder_1 | I210217074149 log-validator kKG6cwA Caught SIGTERM
boulder_1 | I210217074149 log-validator htX9nwc Exiting
boulder_1 | I210217074149 ocsp-responder kKG6cwA Caught SIGTERM
boulder_1 | I210217074149 nonce-service kKG6cwA Caught SIGTERM
boulder_1 | I210217074149 nonce-service htX9nwc Exiting
boulder_1 | I210217074149 nonce-service kKG6cwA Caught SIGTERM
boulder_1 | I210217074149 nonce-service htX9nwc Exiting
boulder_1 | I210217074149 akamai-purger kKG6cwA Caught SIGTERM
boulder_1 | I210217074149 akamai-purger 2f7Ciwo Shutting down; queue is already empty.
boulder_1 | I210217074149 akamai-purger htX9nwc Exiting
boulder_1 | 2021/02/17 07:41:49 Caught SIGTERM
boulder_1 | pebble-challtestsrv - 2021/02/17 07:41:49 Caught signals. Shutting down
boulder_1 | pebble-challtestsrv - 2021/02/17 07:41:49 http: Server closed
boulder_1 | pebble-challtestsrv - 2021/02/17 07:41:50 http: Server closed
boulder_1 | pebble-challtestsrv - 2021/02/17 07:41:50 http: Server closed
boulder_1 | pebble-challtestsrv - 2021/02/17 07:41:51 Shutting down management server
boulder_1 | pebble-challtestsrv - 2021/02/17 07:41:51 http: Server closed
boulder_1 | 2021/02/17 07:41:51 Exiting
boulder_1 | pebble-challtestsrv - 2021/02/17 07:41:51 Goodbye!

@SebastianMoinhinwech
Copy link
Author

Ok, i think i miss the /labca/config/health-checker.json

root@linux5:/home/labca/boulder# ./bin/health-checker -addr ca2.boulder:9093 -config labca/config/health-checker.json
E210217084835 health-checker 54aw7Qg [AUDIT] failed to read json config: open labca/config/health-checker.json: no such file or directory
failed to read json config: open labca/config/health-checker.json: no such file or directory
root@linux5:/home/labca/boulder# cd labca/
root@linux5:/home/labca/boulder/labca# dir
root@linux5:/home/labca/boulder/labca#

@hakwerk
Copy link
Owner

hakwerk commented Feb 17, 2021

Does /home/labca/boulder_labca/config/health-checker.json exist? Docker-compose should map the folder /home/labca/boulder_labca to the correct location inside the docker container

@SebastianMoinhinwech
Copy link
Author

yes, this exists

root@linux5:/home/labca/boulder_labca/config# ls -l health*
-rw-r--r-- 1 labca labca 238 Feb 17 08:30 health-checker.json

@hakwerk
Copy link
Owner

hakwerk commented Feb 17, 2021

You could try to increase the timeout value in that health-checker.json to e.g. 10 seconds:

"timeout": "10s"

When I made mine smaller than the default at e.g. 100ms I see the same error messages as you.

@SebastianMoinhinwech
Copy link
Author

Nope, i think i setup a new fresh server and try again:

boulder_1 | I210217123213 boulder-publisher kKG6cwA Caught SIGTERM
boulder_1 | I210217123213 boulder-publisher htX9nwc Exiting
boulder_1 | I210217123213 ocsp-responder kKG6cwA Caught SIGTERM
boulder_1 | I210217123214 boulder-remoteva kKG6cwA Caught SIGTERM
boulder_1 | I210217123214 boulder-remoteva htX9nwc Exiting
boulder_1 | I210217123214 akamai-purger kKG6cwA Caught SIGTERM
boulder_1 | I210217123214 akamai-purger 2f7Ciwo Shutting down; queue is already empty.
boulder_1 | I210217123214 akamai-purger htX9nwc Exiting
boulder_1 | 2021/02/17 12:32:14 Caught SIGTERM
boulder_1 | pebble-challtestsrv - 2021/02/17 12:32:14 Caught signals. Shutting down
boulder_1 | pebble-challtestsrv - 2021/02/17 12:32:14 http: Server closed
boulder_1 | pebble-challtestsrv - 2021/02/17 12:32:14 http: Server closed
boulder_1 | pebble-challtestsrv - 2021/02/17 12:32:15 http: Server closed
boulder_1 | pebble-challtestsrv - 2021/02/17 12:32:15 Shutting down management server
boulder_1 | pebble-challtestsrv - 2021/02/17 12:32:15 http: Server closed
boulder_1 | pebble-challtestsrv - 2021/02/17 12:32:16 Goodbye!
boulder_1 | 2021/02/17 12:32:16 Exiting
boulder_1 | Starting enhanced syslogd: rsyslogdrsyslogd: error during parsing file /etc/rsyslog.d/boulder.rsyslog.conf, on or before line 1: omfile: warning: default template was already set via legacy directive - may lead to inconsistent results. [v8.1901.0 try https://www.rsyslog.com/e/2207 ]
boulder_1 | rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
boulder_1 | rsyslogd: activation of module imklog failed [v8.1901.0 try https://www.rsyslog.com/e/2145 ]
boulder_1 | Starting enhanced syslogd: rsyslogd.
boulder_1 | labca/entrypoint.sh: connect: Connection refused
boulder_1 | labca/entrypoint.sh: line 19: /dev/tcp/boulder-mysql/3306: Connection refused
boulder_1 | Thu Feb 18 07:19:11 UTC 2021 - still trying to connect to boulder-mysql:3306
boulder_1 | labca/entrypoint.sh: connect: Connection refused
boulder_1 | labca/entrypoint.sh: line 19: /dev/tcp/boulder-mysql/3306: Connection refused
boulder_1 | Thu Feb 18 07:19:12 UTC 2021 - still trying to connect to boulder-mysql:3306
boulder_1 | labca/entrypoint.sh: connect: Connection refused
boulder_1 | labca/entrypoint.sh: line 19: /dev/tcp/boulder-mysql/3306: Connection refused
boulder_1 | Thu Feb 18 07:19:13 UTC 2021 - still trying to connect to boulder-mysql:3306
boulder_1 | labca/entrypoint.sh: connect: Connection refused
boulder_1 | labca/entrypoint.sh: line 19: /dev/tcp/boulder-mysql/3306: Connection refused
boulder_1 | Thu Feb 18 07:19:14 UTC 2021 - still trying to connect to boulder-mysql:3306
boulder_1 | labca/entrypoint.sh: connect: Connection refused
boulder_1 | labca/entrypoint.sh: line 19: /dev/tcp/boulder-mysql/3306: Connection refused
boulder_1 | Thu Feb 18 07:19:15 UTC 2021 - still trying to connect to boulder-mysql:3306
boulder_1 | labca/entrypoint.sh: connect: Connection refused
boulder_1 | labca/entrypoint.sh: line 19: /dev/tcp/boulder-mysql/3306: Connection refused
boulder_1 | Thu Feb 18 07:19:16 UTC 2021 - still trying to connect to boulder-mysql:3306
boulder_1 | Connected to boulder-mysql:3306
boulder_1 | Database boulder_sa_test already exists - skipping create
boulder_1 | goose: no migrations to run. current version: 20200609125504
boulder_1 | migrated boulder_sa_test database with ./sa/db/
boulder_1 | added users to boulder_sa_test
boulder_1 | Database boulder_sa_integration already exists - skipping create
boulder_1 | goose: no migrations to run. current version: 20200609125504
boulder_1 | migrated boulder_sa_integration database with ./sa/db/
boulder_1 | added users to boulder_sa_integration
boulder_1 | created all databases
boulder_1 | CKR_SLOT_ID_INVALID: Slot 0 does not exist.
boulder_1 | Found slot 2091935369 with matching token label.
boulder_1 | The key pair has been imported.
boulder_1 | CKR_SLOT_ID_INVALID: Slot 1 does not exist.
boulder_1 | Found slot 1306638887 with matching token label.
boulder_1 | The key pair has been imported.
boulder_1 | GOBIN=/go/src/github.com/letsencrypt/boulder/bin GO111MODULE=on go install -mod=vendor -tags "integration" ./...
boulder_1 | cp /go/src/github.com/letsencrypt/boulder/bin/boulder-va /go/src/github.com/letsencrypt/boulder/bin/boulder-remoteva
boulder_1 | pebble-challtestsrv - 2021/02/18 07:21:17 Creating HTTP-01 challenge server on 10.77.77.77:5002
boulder_1 | pebble-challtestsrv - 2021/02/18 07:21:17 Creating HTTPS HTTP-01 challenge server on 10.77.77.77:5001
boulder_1 | pebble-challtestsrv - 2021/02/18 07:21:17 Creating TCP and UDP DNS-01 challenge server on :8053
boulder_1 | pebble-challtestsrv - 2021/02/18 07:21:17 Creating TCP and UDP DNS-01 challenge server on :8054
boulder_1 | pebble-challtestsrv - 2021/02/18 07:21:17 Creating TLS-ALPN-01 challenge server on 10.88.88.88:5001
boulder_1 | pebble-challtestsrv - 2021/02/18 07:21:17 Answering A queries with 10.77.77.77 by default
boulder_1 | pebble-challtestsrv - 2021/02/18 07:21:17 Starting management server on :8055
boulder_1 | pebble-challtestsrv - 2021/02/18 07:21:17 Starting challenge servers
boulder_1 | Connecting to publisher2.boulder:9091 health service
boulder_1 | I210218072118 boulder-publisher 6uKbhAo Versions: boulder-publisher=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | 2021/02/18 07:21:19 mail-test-srv: 127.0.0.1:47268: readline: EOF
boulder_1 | I210218072119 mail-test-srv rZjrxAY mail-test-srv: Got connection from 127.0.0.1:47268
boulder_1 | I210218072119 boulder-remoteva 0MLlrww Versions: boulder-remoteva=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to va1.boulder:9097 health service
boulder_1 | I210218072119 akamai-purger 8tvK_Q8 Versions: akamai-purger=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to sa1.boulder:9095 health service
boulder_1 | I210218072120 boulder-sa kcXg9QE Versions: boulder-sa=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to publisher1.boulder:9091 health service
boulder_1 | I210218072121 boulder-publisher 6uKbhAo Versions: boulder-publisher=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | I210218072122 log-validator g5CP7Ac Waiting for /var/log/boulder-wfe2.log to appear...
boulder_1 | I210218072122 log-validator 44ONuAY Waiting for /var/log/boulder-wfe.log to appear...
boulder_1 | I210218072122 boulder-remoteva 0MLlrww Versions: boulder-remoteva=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to va1.boulder:9098 health service
boulder_1 | I210218072123 boulder-va l_rfyQ8 Versions: boulder-va=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to va2.boulder:9092 health service
boulder_1 | I210218072123 ocsp-responder tLqf8AI Versions: ocsp-responder=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | I210218072123 ocsp-responder mbD4lQM Loading OCSP Database for CA Cert: labca/test-ca.pem
boulder_1 | 2021/02/18 07:21:23 ct-test-srv on :4501 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEKtnFevaXV/kB8dmhCNZHmxKVLcHX1plaAsY9LrKilhYxdmQZiu36LvAvosTsqMVqRK9a96nC8VaxAdaHUbM8EA==
boulder_1 | 2021/02/18 07:21:23 ct-test-srv on :4512 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEFRu37ZRLg8lT4rVQwMwh4oAOpXb4Sx+9hgQ+JFCjmAv3oDV+sDOMsC7hULkGTn+LB5L1SRo/XIY4Kw5V+nFXgg==
boulder_1 | 2021/02/18 07:21:23 ct-test-srv on :4500 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEYggOxPnPkzKBIhTacSYoIfnSL2jPugcbUKx83vFMvk5gKAz/AGe87w20riuPwEGn229hKVbEKHFB61NIqNHC3Q==
boulder_1 | 2021/02/18 07:21:23 ct-test-srv on :4510 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEyw1HymhJkuxSIgt3gqW3sVXqMqB3EFsXcMfPFo0vYwjNiRmCJDXKsR0Flp7MAK+wc3X/7Hpc8liUbMhPet7tEA==
boulder_1 | 2021/02/18 07:21:23 ct-test-srv on :4511 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEFRu37ZRLg8lT4rVQwMwh4oAOpXb4Sx+9hgQ+JFCjmAv3oDV+sDOMsC7hULkGTn+LB5L1SRo/XIY4Kw5V+nFXgg==
boulder_1 | Connecting to nonce1.boulder:9101 health service
boulder_1 | I210218072123 nonce-service l6
-0Q4 Versions: nonce-service=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | I210218072124 nonce-service l6
-0Q4 Versions: nonce-service=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to nonce2.boulder:9101 health service
boulder_1 | I210218072125 boulder-va l_rfyQ8 Versions: boulder-va=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to va1.boulder:9092 health service
boulder_1 | I210218072125 boulder-sa kcXg9QE Versions: boulder-sa=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to sa2.boulder:9095 health service
boulder_1 | Connecting to ca2.boulder:9093 health service
boulder_1 | I210218072125 boulder-ca 5u-yxAg Versions: boulder-ca=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | I210218072125 boulder-ca 9OXLlwI loading hostname policy, sha256: 0be6f6061a1041f866b71037f1aa96e0335be2035e0e8657aa203129f2d13f8d
boulder_1 | got error connecting to health service ca2.boulder:9093: grpc.health.v1.Health.Check timed out after 9999 ms
boulder_1 | Connecting to ca2.boulder:9093 health service
boulder_1 | got error connecting to health service ca2.boulder:9093: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ca2.boulder:9093 health service
boulder_1 | I210218072145 boulder-ca 5u-yxAg Versions: boulder-ca=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | I210218072145 boulder-ca 9OXLlwI loading hostname policy, sha256: 0be6f6061a1041f866b71037f1aa96e0335be2035e0e8657aa203129f2d13f8d
boulder_1 | Connecting to ca1.boulder:9093 health service
boulder_1 | Connecting to ra1.boulder:9094 health service
boulder_1 | I210218072151 boulder-ra -rX4vgQ Versions: boulder-ra=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | I210218072151 boulder-ra 9OXLlwI loading hostname policy, sha256: 0be6f6061a1041f866b71037f1aa96e0335be2035e0e8657aa203129f2d13f8d
boulder_1 | �[31m�[1mE210218072151 boulder-ra rpSTxAc [AUDIT] Couldn't load rate limit policies file: yaml: line 17: did not find expected key�[0m
boulder_1 | Couldn't load rate limit policies file: yaml: line 17: did not find expected key
boulder_1 | got error connecting to health service ra1.boulder:9094: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ra1.boulder:9094 health service
boulder_1 | got error connecting to health service ra1.boulder:9094: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ra1.boulder:9094 health service
boulder_1 | I210218072218 boulder-publisher _LKDgws time=2021-02-18T07:22:18.35382214Z
boulder_1 | I210218072219 boulder-remoteva i8ffjgo time=2021-02-18T07:22:19.417989943Z
boulder_1 | I210218072219 akamai-purger 8OPxxwg time=2021-02-18T07:22:19.783702108Z
boulder_1 | I210218072220 boulder-sa -sWy5QM time=2021-02-18T07:22:20.120173968Z
boulder_1 | I210218072221 log-validator w5e-wA0 time=2021-02-18T07:22:21.499221939Z
boulder_1 | got error connecting to health service ra1.boulder:9094: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ra1.boulder:9094 health service
boulder_1 | I210218072221 boulder-publisher 9tSEaAA time=2021-02-18T07:22:21.778994052Z
boulder_1 | I210218072222 boulder-remoteva 77Hf2Ag time=2021-02-18T07:22:22.862282455Z
boulder_1 | I210218072223 boulder-va svnvlAU time=2021-02-18T07:22:23.036197449Z
boulder_1 | I210218072223 ocsp-responder _YPvlg8 time=2021-02-18T07:22:23.44046145Z
boulder_1 | I210218072223 nonce-service 4a3E4Q0 time=2021-02-18T07:22:23.95074956Z
boulder_1 | I210218072224 nonce-service w6qgzg8 time=2021-02-18T07:22:24.847138533Z
boulder_1 | I210218072225 boulder-va nKGnqQo time=2021-02-18T07:22:25.039609786Z
boulder_1 | I210218072225 boulder-sa 0KuRnQI time=2021-02-18T07:22:25.286739601Z
boulder_1 | I210218072225 boulder-ca l6Xc3wI time=2021-02-18T07:22:25.739813229Z
boulder_1 | got error connecting to health service ra1.boulder:9094: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ra1.boulder:9094 health service
boulder_1 | got error connecting to health service ra1.boulder:9094: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ra1.boulder:9094 health service
boulder_1 | I210218072245 boulder-ca wOO72g8 time=2021-02-18T07:22:45.689232968Z
boulder_1 | got error connecting to health service ra1.boulder:9094: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ra1.boulder:9094 health service
boulder_1 | got error connecting to health service ra1.boulder:9094: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ra1.boulder:9094 health service
boulder_1 | got error connecting to health service ra1.boulder:9094: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ra1.boulder:9094 health service
boulder_1 | I210218072318 boulder-publisher -JDS7g0 time=2021-02-18T07:23:18.354113961Z
boulder_1 | I210218072319 boulder-remoteva z--y8wk time=2021-02-18T07:23:19.418474268Z
boulder_1 | I210218072319 akamai-purger j66Kyw4 time=2021-02-18T07:23:19.784070572Z
boulder_1 | I210218072320 boulder-sa try2mAg time=2021-02-18T07:23:20.120535016Z
boulder_1 | I210218072321 log-validator y7Ph4QM time=2021-02-18T07:23:21.499579006Z
boulder_1 | got error connecting to health service ra1.boulder:9094: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ra1.boulder:9094 health service
boulder_1 | I210218072321 boulder-publisher prKU3wU time=2021-02-18T07:23:21.779342213Z
boulder_1 | I210218072322 boulder-remoteva haSvlA0 time=2021-02-18T07:23:22.862579016Z
boulder_1 | I210218072323 boulder-va zumq0A0 time=2021-02-18T07:23:23.036712063Z
boulder_1 | I210218072323 ocsp-responder _7yI3QI time=2021-02-18T07:23:23.441941236Z
boulder_1 | I210218072323 nonce-service -JX4lAE time=2021-02-18T07:23:23.951403161Z
boulder_1 | I210218072324 nonce-service 8pmT_Qs time=2021-02-18T07:23:24.847476411Z
boulder_1 | I210218072325 boulder-va _tLc5Ao time=2021-02-18T07:23:25.04003745Z
boulder_1 | I210218072325 boulder-sa uMuy6Aw time=2021-02-18T07:23:25.287276618Z
boulder_1 | I210218072325 boulder-ca tpynqgU time=2021-02-18T07:23:25.740246714Z
boulder_1 | got error connecting to health service ra1.boulder:9094: grpc.health.v1.Health.Check timed out after 9894 ms
boulder_1 | Connecting to ra1.boulder:9094 health service
boulder_1 | got error connecting to health service ra1.boulder:9094: grpc.health.v1.Health.Check timed out after 0 ms
boulder_1 | �[31m�[1mE210218072331 health-checker pfeEkw4 [AUDIT] timed out waiting for ra1.boulder:9094 health check�[0m
boulder_1 | timed out waiting for ra1.boulder:9094 health check
boulder_1 | Waiting for debug port 8055 (pebble-challtestsrv --defaultIPv4 10.77.77.77 -defaultIPv6 --dns01 :8053,:8054 --management :8055 --http01 10.77.77.77:5002 -https01 10.77.77.77:5001 --tlsalpn01 10.88.88.88:5001)
boulder_1 | Starting service akamai-test-srv
boulder_1 | Starting service sd-test-srv
boulder_1 | Starting service boulder-publisher-2
boulder_1 | Starting service mail-test-srv
boulder_1 | Starting service boulder-remoteva-a
boulder_1 | Starting service akamai-purger
boulder_1 | Waiting for debug port 9666 (./bin/akamai-purger --config labca/config/akamai-purger.json)
boulder_1 | Waiting for debug port 9666 (./bin/akamai-purger --config labca/config/akamai-purger.json)
boulder_1 | Starting service boulder-sa-1
boulder_1 | Starting service log-validator
boulder_1 | Waiting for debug port 8016 (./bin/log-validator --config labca/config/log-validator.json)
boulder_1 | Waiting for debug port 8016 (./bin/log-validator --config labca/config/log-validator.json)
boulder_1 | Waiting for debug port 8016 (./bin/log-validator --config labca/config/log-validator.json)
boulder_1 | Waiting for debug port 8016 (./bin/log-validator --config labca/config/log-validator.json)
boulder_1 | Starting service boulder-publisher-1
boulder_1 | Starting service boulder-remoteva-b
boulder_1 | Starting service boulder-va-2
boulder_1 | Starting service ocsp-responder
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Starting service ct-test-srv
boulder_1 | Starting service nonce-service-taro
boulder_1 | Starting service nonce-service-zinc
boulder_1 | Starting service boulder-va-1
boulder_1 | Starting service boulder-sa-2
boulder_1 | Starting service boulder-ca-b
boulder_1 | Starting service boulder-ca-a
boulder_1 | Starting service boulder-ra-1
boulder_1 | Error starting service boulder-ra-1: Command '['./bin/health-checker', '-addr', 'ra1.boulder:9094', '-config', 'labca/config/health-checker.json']' returned non-zero exit status 1.
boulder_1 | I210218072331 boulder-ca kKG6cwA Caught SIGTERM
boulder_1 | I210218072331 boulder-ca htX9nwc Exiting
boulder_1 | I210218072331 boulder-ca kKG6cwA Caught SIGTERM
boulder_1 | I210218072331 boulder-ca htX9nwc Exiting
boulder_1 | I210218072331 boulder-sa kKG6cwA Caught SIGTERM
boulder_1 | I210218072331 boulder-sa htX9nwc Exiting
boulder_1 | I210218072331 boulder-va kKG6cwA Caught SIGTERM
boulder_1 | I210218072331 boulder-va htX9nwc Exiting
boulder_1 | I210218072331 nonce-service kKG6cwA Caught SIGTERM
boulder_1 | I210218072331 nonce-service htX9nwc Exiting
boulder_1 | I210218072331 nonce-service kKG6cwA Caught SIGTERM
boulder_1 | I210218072331 nonce-service htX9nwc Exiting
boulder_1 | I210218072331 ocsp-responder kKG6cwA Caught SIGTERM
boulder_1 | I210218072332 ocsp-responder htX9nwc Exiting
boulder_1 | I210218072332 boulder-va kKG6cwA Caught SIGTERM
boulder_1 | I210218072332 boulder-va htX9nwc Exiting
boulder_1 | I210218072332 boulder-remoteva kKG6cwA Caught SIGTERM
boulder_1 | I210218072332 boulder-remoteva htX9nwc Exiting
boulder_1 | I210218072332 boulder-publisher kKG6cwA Caught SIGTERM
boulder_1 | I210218072332 boulder-publisher htX9nwc Exiting
boulder_1 | I210218072332 log-validator kKG6cwA Caught SIGTERM
boulder_1 | I210218072332 log-validator htX9nwc Exiting
boulder_1 | I210218072332 boulder-sa kKG6cwA Caught SIGTERM
boulder_1 | I210218072332 boulder-sa htX9nwc Exiting
boulder_1 | I210218072332 akamai-purger kKG6cwA Caught SIGTERM
boulder_1 | I210218072332 akamai-purger 2f7Ciwo Shutting down; queue is already empty.
boulder_1 | I210218072332 akamai-purger htX9nwc Exiting
boulder_1 | I210218072332 boulder-remoteva kKG6cwA Caught SIGTERM
boulder_1 | I210218072332 boulder-remoteva htX9nwc Exiting
boulder_1 | I210218072332 boulder-publisher kKG6cwA Caught SIGTERM
boulder_1 | I210218072332 boulder-publisher htX9nwc Exiting
boulder_1 | 2021/02/18 07:23:32 Caught SIGTERM
boulder_1 | pebble-challtestsrv - 2021/02/18 07:23:32 Caught signals. Shutting down
boulder_1 | pebble-challtestsrv - 2021/02/18 07:23:32 http: Server closed
boulder_1 | pebble-challtestsrv - 2021/02/18 07:23:32 http: Server closed
boulder_1 | pebble-challtestsrv - 2021/02/18 07:23:33 http: Server closed
boulder_1 | pebble-challtestsrv - 2021/02/18 07:23:33 Shutting down management server
boulder_1 | pebble-challtestsrv - 2021/02/18 07:23:33 http: Server closed
boulder_1 | 2021/02/18 07:23:34 Exiting
boulder_1 | pebble-challtestsrv - 2021/02/18 07:23:34 Goodbye!

@SebastianMoinhinwech
Copy link
Author

So, fresh install, debian 10.8 this time english, same behaviour, only additional package is "ssh and basic" and webmin.

boulder_1 | 2021/02/19 09:44:01 ct-test-srv on :4500 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEYggOxPnPkzKBIhTacSYoIfnSL2jPugcbUKx83vFMvk5gKAz/AGe87w20riuPwEGn229hKVbEKHFB61NIqNHC3Q==
boulder_1 | 2021/02/19 09:44:01 ct-test-srv on :4510 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEyw1HymhJkuxSIgt3gqW3sVXqMqB3EFsXcMfPFo0vYwjNiRmCJDXKsR0Flp7MAK+wc3X/7Hpc8liUbMhPet7tEA==
boulder_1 | 2021/02/19 09:44:01 ct-test-srv on :4501 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEKtnFevaXV/kB8dmhCNZHmxKVLcHX1plaAsY9LrKilhYxdmQZiu36LvAvosTsqMVqRK9a96nC8VaxAdaHUbM8EA==
boulder_1 | I210219094401 mail-test-srv 9o-fuQE mail-test-srv: Got connection from 127.0.0.1:52334
boulder_1 | 2021/02/19 09:44:01 mail-test-srv: 127.0.0.1:52334: readline: EOF
boulder_1 | Connecting to sa1.boulder:9095 health service
boulder_1 | I210219094402 boulder-sa kcXg9QE Versions: boulder-sa=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to publisher2.boulder:9091 health service
boulder_1 | I210219094407 boulder-publisher 6uKbhAo Versions: boulder-publisher=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to nonce2.boulder:9101 health service
boulder_1 | I210219094411 nonce-service l6_-0Q4 Versions: nonce-service=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | I210219094412 boulder-publisher 6uKbhAo Versions: boulder-publisher=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to publisher1.boulder:9091 health service
boulder_1 | I210219094413 log-validator g5CP7Ac Waiting for /var/log/boulder-wfe2.log to appear...
boulder_1 | I210219094413 log-validator 44ONuAY Waiting for /var/log/boulder-wfe.log to appear...
boulder_1 | I210219094413 nonce-service l6_-0Q4 Versions: nonce-service=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to nonce1.boulder:9101 health service
boulder_1 | I210219094413 boulder-remoteva 0MLlrww Versions: boulder-remoteva=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to va1.boulder:9098 health service
boulder_1 | I210219094413 boulder-sa kcXg9QE Versions: boulder-sa=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to sa2.boulder:9095 health service
boulder_1 | I210219094414 ocsp-responder tLqf8AI Versions: ocsp-responder=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | I210219094414 ocsp-responder mbD4lQM Loading OCSP Database for CA Cert: labca/test-ca.pem
boulder_1 | I210219094414 boulder-remoteva 0MLlrww Versions: boulder-remoteva=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to va1.boulder:9097 health service
boulder_1 | Connecting to ca2.boulder:9093 health service
boulder_1 | I210219094415 boulder-ca 5u-yxAg Versions: boulder-ca=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | I210219094415 boulder-ca 9OXLlwI loading hostname policy, sha256: 0be6f6061a1041f866b71037f1aa96e0335be2035e0e8657aa203129f2d13f8d
boulder_1 | I210219094415 boulder-ca 5u-yxAg Versions: boulder-ca=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | I210219094415 boulder-ca 9OXLlwI loading hostname policy, sha256: 0be6f6061a1041f866b71037f1aa96e0335be2035e0e8657aa203129f2d13f8d
boulder_1 | Connecting to ca1.boulder:9093 health service
boulder_1 | I210219094417 ocsp-updater rpnyoA8 Versions: ocsp-updater=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | I210219094417 boulder-va l_rfyQ8 Versions: boulder-va=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to va1.boulder:9092 health service
boulder_1 | I210219094418 akamai-purger 8tvK_Q8 Versions: akamai-purger=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | I210219094418 boulder-va l_rfyQ8 Versions: boulder-va=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to va2.boulder:9092 health service
boulder_1 | Connecting to ra2.boulder:9094 health service
boulder_1 | I210219094419 boulder-ra -rX4vgQ Versions: boulder-ra=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | I210219094419 boulder-ra 9OXLlwI loading hostname policy, sha256: 0be6f6061a1041f866b71037f1aa96e0335be2035e0e8657aa203129f2d13f8d
boulder_1 | �[31m�[1mE210219094419 boulder-ra rpSTxAc [AUDIT] Couldn't load rate limit policies file: yaml: line 17: did not find expected key�[0m
boulder_1 | Couldn't load rate limit policies file: yaml: line 17: did not find expected key
boulder_1 | got error connecting to health service ra2.boulder:9094: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ra2.boulder:9094 health service
boulder_1 | got error connecting to health service ra2.boulder:9094: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ra2.boulder:9094 health service
boulder_1 | got error connecting to health service ra2.boulder:9094: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ra2.boulder:9094 health service
boulder_1 | I210219094448 boulder-ca 9OXLlwI loading hostname policy, sha256: 0be6f6061a1041f866b71037f1aa96e0335be2035e0e8657aa203129f2d13f8d
boulder_1 | I210219094449 boulder-ca 9OXLlwI loading hostname policy, sha256: 0be6f6061a1041f866b71037f1aa96e0335be2035e0e8657aa203129f2d13f8d
boulder_1 | got error connecting to health service ra2.boulder:9094: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ra2.boulder:9094 health service
boulder_1 | I210219094502 boulder-sa o_qX0Ak time=2021-02-19T09:45:02.977317002Z
boulder_1 | I210219094507 boulder-publisher vq3HFgA time=2021-02-19T09:45:07.874540275Z
boulder_1 | got error connecting to health service ra2.boulder:9094: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ra2.boulder:9094 health service
boulder_1 | I210219094511 nonce-service 18bkkQU time=2021-02-19T09:45:11.704897972Z
boulder_1 | I210219094512 boulder-publisher tOTC3go time=2021-02-19T09:45:12.111681223Z
boulder_1 | I210219094513 log-validator qK_EgQk time=2021-02-19T09:45:13.024161346Z
boulder_1 | I210219094513 nonce-service i82wxwM time=2021-02-19T09:45:13.100301464Z
boulder_1 | I210219094513 boulder-remoteva 49e3xAk time=2021-02-19T09:45:13.300977686Z
boulder_1 | I210219094513 boulder-sa 39-Dmw4 time=2021-02-19T09:45:13.48081026Z
boulder_1 | I210219094514 ocsp-responder 99Ot7Ak time=2021-02-19T09:45:14.238682457Z
boulder_1 | I210219094514 boulder-remoteva oPjk3g4 time=2021-02-19T09:45:14.311958667Z
boulder_1 | I210219094515 boulder-ca z6P14wY time=2021-02-19T09:45:15.123527708Z
boulder_1 | I210219094515 boulder-ca kaXzpgc time=2021-02-19T09:45:15.88740346Z
boulder_1 | I210219094517 ocsp-updater kaGtwQ8 time=2021-02-19T09:45:17.842120023Z
boulder_1 | I210219094517 boulder-va 0ong7ww time=2021-02-19T09:45:17.989920219Z
boulder_1 | I210219094518 akamai-purger io-Dswo time=2021-02-19T09:45:18.419518749Z
boulder_1 | I210219094518 boulder-va 4_zC-w8 time=2021-02-19T09:45:18.50592626Z
boulder_1 | got error connecting to health service ra2.boulder:9094: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ra2.boulder:9094 health service
boulder_1 | got error connecting to health service ra2.boulder:9094: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ra2.boulder:9094 health service
boulder_1 | got error connecting to health service ra2.boulder:9094: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ra2.boulder:9094 health service
boulder_1 | got error connecting to health service ra2.boulder:9094: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ra2.boulder:9094 health service
boulder_1 | got error connecting to health service ra2.boulder:9094: grpc.health.v1.Health.Check timed out after 9894 ms
boulder_1 | timed out waiting for ra2.boulder:9094 health check
boulder_1 | �[31m�[1mE210219094558 health-checker 5qzUjwU [AUDIT] timed out waiting for ra2.boulder:9094 health check�[0m
boulder_1 | Waiting for debug port 8055 (pebble-challtestsrv --defaultIPv4 10.77.77.77 -defaultIPv6 --dns01 :8053,:8054 --management :8055 --http01 10.77.77.77:5002 -https01 10.77.77.77:5001 --tlsalpn01 10.88.88.88:5001)
boulder_1 | Starting service sd-test-srv
boulder_1 | Starting service ct-test-srv
boulder_1 | Starting service mail-test-srv
boulder_1 | Starting service boulder-sa-1
boulder_1 | Starting service boulder-publisher-2
boulder_1 | Starting service nonce-service-zinc
boulder_1 | Starting service boulder-publisher-1
boulder_1 | Starting service log-validator
boulder_1 | Waiting for debug port 8016 (./bin/log-validator --config labca/config/log-validator.json)
boulder_1 | Waiting for debug port 8016 (./bin/log-validator --config labca/config/log-validator.json)
boulder_1 | Waiting for debug port 8016 (./bin/log-validator --config labca/config/log-validator.json)
boulder_1 | Waiting for debug port 8016 (./bin/log-validator --config labca/config/log-validator.json)
boulder_1 | Waiting for debug port 8016 (./bin/log-validator --config labca/config/log-validator.json)
boulder_1 | Waiting for debug port 8016 (./bin/log-validator --config labca/config/log-validator.json)
boulder_1 | Waiting for debug port 8016 (./bin/log-validator --config labca/config/log-validator.json)
boulder_1 | Starting service nonce-service-taro
boulder_1 | Starting service boulder-remoteva-b
boulder_1 | Starting service boulder-sa-2
boulder_1 | Starting service ocsp-responder
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Starting service boulder-remoteva-a
boulder_1 | Starting service akamai-test-srv
boulder_1 | Starting service boulder-ca-b
boulder_1 | Starting service boulder-ca-a
boulder_1 | Starting service ocsp-updater
boulder_1 | Waiting for debug port 8006 (./bin/ocsp-updater --config labca/config/ocsp-updater.json)
boulder_1 | Waiting for debug port 8006 (./bin/ocsp-updater --config labca/config/ocsp-updater.json)
boulder_1 | Waiting for debug port 8006 (./bin/ocsp-updater --config labca/config/ocsp-updater.json)
boulder_1 | Waiting for debug port 8006 (./bin/ocsp-updater --config labca/config/ocsp-updater.json)
boulder_1 | Waiting for debug port 8006 (./bin/ocsp-updater --config labca/config/ocsp-updater.json)
boulder_1 | Waiting for debug port 8006 (./bin/ocsp-updater --config labca/config/ocsp-updater.json)
boulder_1 | Waiting for debug port 8006 (./bin/ocsp-updater --config labca/config/ocsp-updater.json)
boulder_1 | Waiting for debug port 8006 (./bin/ocsp-updater --config labca/config/ocsp-updater.json)
boulder_1 | Starting service boulder-va-1
boulder_1 | Starting service akamai-purger
boulder_1 | Waiting for debug port 9666 (./bin/akamai-purger --config labca/config/akamai-purger.json)
boulder_1 | Waiting for debug port 9666 (./bin/akamai-purger --config labca/config/akamai-purger.json)
boulder_1 | Starting service boulder-va-2
boulder_1 | Starting service boulder-ra-2
boulder_1 | Error starting service boulder-ra-2: Command '['./bin/health-checker', '-addr', 'ra2.boulder:9094', '-config', 'labca/config/health-checker.json']' returned non-zero exit status 1.
boulder_1 | I210219094558 boulder-va kKG6cwA Caught SIGTERM
boulder_1 | I210219094558 boulder-va htX9nwc Exiting
boulder_1 | I210219094558 akamai-purger kKG6cwA Caught SIGTERM
boulder_1 | I210219094558 akamai-purger 2f7Ciwo Shutting down; queue is already empty.
boulder_1 | I210219094558 akamai-purger htX9nwc Exiting
boulder_1 | I210219094558 boulder-va kKG6cwA Caught SIGTERM
boulder_1 | I210219094558 boulder-va htX9nwc Exiting
boulder_1 | I210219094558 ocsp-updater kKG6cwA Caught SIGTERM
boulder_1 | I210219094558 ocsp-updater htX9nwc Exiting
boulder_1 | I210219094558 boulder-ca kKG6cwA Caught SIGTERM
boulder_1 | I210219094558 boulder-ca htX9nwc Exiting
boulder_1 | I210219094558 boulder-ca kKG6cwA Caught SIGTERM
boulder_1 | I210219094558 boulder-ca htX9nwc Exiting
boulder_1 | I210219094558 boulder-remoteva kKG6cwA Caught SIGTERM
boulder_1 | I210219094558 boulder-remoteva htX9nwc Exiting
boulder_1 | I210219094558 ocsp-responder kKG6cwA Caught SIGTERM
boulder_1 | I210219094559 boulder-sa kKG6cwA Caught SIGTERM
boulder_1 | I210219094559 boulder-sa htX9nwc Exiting
boulder_1 | I210219094559 boulder-remoteva kKG6cwA Caught SIGTERM
boulder_1 | I210219094559 nonce-service kKG6cwA Caught SIGTERM
boulder_1 | I210219094559 nonce-service htX9nwc Exiting
boulder_1 | I210219094559 log-validator kKG6cwA Caught SIGTERM
boulder_1 | I210219094559 log-validator htX9nwc Exiting
boulder_1 | I210219094559 boulder-publisher kKG6cwA Caught SIGTERM
boulder_1 | I210219094559 boulder-publisher htX9nwc Exiting
boulder_1 | I210219094559 nonce-service kKG6cwA Caught SIGTERM
boulder_1 | I210219094559 boulder-publisher kKG6cwA Caught SIGTERM
boulder_1 | I210219094559 boulder-publisher htX9nwc Exiting
boulder_1 | I210219094559 boulder-sa kKG6cwA Caught SIGTERM
boulder_1 | I210219094559 boulder-sa htX9nwc Exiting
boulder_1 | pebble-challtestsrv - 2021/02/19 09:45:59 Caught signals. Shutting down
boulder_1 | 2021/02/19 09:45:59 Caught SIGTERM
boulder_1 | pebble-challtestsrv - 2021/02/19 09:45:59 http: Server closed
boulder_1 | pebble-challtestsrv - 2021/02/19 09:45:59 http: Server closed
boulder_1 | pebble-challtestsrv - 2021/02/19 09:46:00 http: Server closed
boulder_1 | pebble-challtestsrv - 2021/02/19 09:46:00 Shutting down management server
boulder_1 | pebble-challtestsrv - 2021/02/19 09:46:00 http: Server closed
boulder_1 | pebble-challtestsrv - 2021/02/19 09:46:01 Goodbye!
boulder_1 | 2021/02/19 09:46:01 Exiting

@hakwerk
Copy link
Owner

hakwerk commented Feb 22, 2021

Can you check line 17 in /home/labca/boulder_labca/rate-limit-policies.yml - I see that your last two logs are complaining about that.
Your "host.domain.local" should be below that on line 18 (and on the last line). Does it perhaps contain any special characters?

@SebastianMoinhinwech
Copy link
Author

I can`t see my host in that file: linux5.itschnack.local

See cmd/shell.go for definitions of these rate limits.

certificatesPerName:
window: 2160h
threshold: 2
overrides:
ratelimit.me: 1
lim.it: 0
# Hostnames used by the letsencrypt client integration test.
le.wtf: 10000
le1.wtf: 10000
le2.wtf: 10000
le3.wtf: 10000
nginx.wtf: 10000
good-caa-reserved.com: 10000
bad-caa-reserved.com: 10000
ecdsa.le.wtf: 10000
must-staple.le.wtf: 10000
: 10000
: 10000
: 10000
: 10000
: 10000
: 10000
registrationOverrides:
101: 1000
registrationsPerIP:
window: 168h # 1 week
threshold: 10000
overrides:
127.0.0.1: 1000000
registrationsPerIPRange:
window: 168h # 1 week
threshold: 99999
overrides:
127.0.0.1: 1000000
pendingAuthorizationsPerAccount:
window: 168h # 1 week, should match pending authorization lifetime.
threshold: 150
invalidAuthorizationsPerAccount:
window: 5m
threshold: 3
newOrdersPerAccount:
window: 3h
threshold: 1500
certificatesPerFQDNSet:
window: 24h
threshold: 5
overrides:
le.wtf: 10000
le1.wtf: 10000
le2.wtf: 10000
le3.wtf: 10000
le.wtf,le1.wtf: 10000
good-caa-reserved.com: 10000
nginx.wtf: 10000
ecdsa.le.wtf: 10000
must-staple.le.wtf: 10000
: 10000
: 10000
: 10000
: 10000
: 10000
: 10000

@hakwerk
Copy link
Owner

hakwerk commented Feb 24, 2021

I've no idea how that could have happened, I'm not able to reproduce this in any way. If you would remove all the lines with only whitespace before the colon (": 10000") then at least the boulder-ra component should be able to start again.

I'll further investigate how to prevent and especially recover from this situation

@SebastianMoinhinwech
Copy link
Author

Hi, strange. I created a complete new debian 10.8 VM, this time with english localization and installed only webmin.

Same behaviour. I did all the tweaks mentioned here, but also no function (Also the remove of the :10000". I added my local fqdn in the top of the rate-limits. i copied some missing folders from boulder_ca to boulder/labca (config and grpc-certs).

boulder_1 | rsyslogd: error during parsing file /etc/rsyslog.d/boulder.rsyslog.conf, on or before line 1: omfile: warning: default template was already set via legacy directive - may lead to inconsistent results. [v8.1901.0 try https://www.rsyslog.com/e/2207 ]
boulder_1 | rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
boulder_1 | rsyslogd: activation of module imklog failed [v8.1901.0 try https://www.rsyslog.com/e/2145 ]
boulder_1 | Starting enhanced syslogd: rsyslogd.
boulder_1 | Connected to boulder-mysql:3306
boulder_1 | Database boulder_sa_test already exists - skipping create
boulder_1 | goose: no migrations to run. current version: 20200609125504
boulder_1 | migrated boulder_sa_test database with ./sa/db/
boulder_1 | added users to boulder_sa_test
boulder_1 | Database boulder_sa_integration already exists - skipping create
boulder_1 | goose: no migrations to run. current version: 20200609125504
boulder_1 | migrated boulder_sa_integration database with ./sa/db/
boulder_1 | added users to boulder_sa_integration
boulder_1 | created all databases
boulder_1 | CKR_SLOT_ID_INVALID: Slot 0 does not exist.
boulder_1 | Found slot 156883163 with matching token label.
boulder_1 | The key pair has been imported.
boulder_1 | CKR_SLOT_ID_INVALID: Slot 1 does not exist.
boulder_1 | Found slot 1114748740 with matching token label.
boulder_1 | The key pair has been imported.
boulder_1 | GOBIN=/go/src/github.com/letsencrypt/boulder/bin GO111MODULE=on go install -mod=vendor -tags "integration" ./...
boulder_1 | cp /go/src/github.com/letsencrypt/boulder/bin/boulder-va /go/src/github.com/letsencrypt/boulder/bin/boulder-remoteva
boulder_1 | pebble-challtestsrv - 2021/02/24 08:30:35 Creating HTTP-01 challenge server on 10.77.77.77:5002
boulder_1 | pebble-challtestsrv - 2021/02/24 08:30:35 Creating HTTPS HTTP-01 challenge server on 10.77.77.77:5001
boulder_1 | pebble-challtestsrv - 2021/02/24 08:30:35 Creating TCP and UDP DNS-01 challenge server on :8053
boulder_1 | pebble-challtestsrv - 2021/02/24 08:30:35 Creating TCP and UDP DNS-01 challenge server on :8054
boulder_1 | pebble-challtestsrv - 2021/02/24 08:30:35 Creating TLS-ALPN-01 challenge server on 10.88.88.88:5001
boulder_1 | pebble-challtestsrv - 2021/02/24 08:30:35 Answering A queries with 10.77.77.77 by default
boulder_1 | pebble-challtestsrv - 2021/02/24 08:30:35 Starting challenge servers
boulder_1 | pebble-challtestsrv - 2021/02/24 08:30:35 Starting management server on :8055
boulder_1 | I210224083035 log-validator g5CP7Ac Waiting for /var/log/boulder-wfe2.log to appear...
boulder_1 | I210224083035 log-validator 44ONuAY Waiting for /var/log/boulder-wfe.log to appear...
boulder_1 | Connecting to publisher1.boulder:9091 health service
boulder_1 | I210224083036 boulder-publisher 6uKbhAo Versions: boulder-publisher=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to sa1.boulder:9095 health service
boulder_1 | I210224083037 boulder-sa kcXg9QE Versions: boulder-sa=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | I210224083038 boulder-remoteva 0MLlrww Versions: boulder-remoteva=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to va1.boulder:9097 health service
boulder_1 | I210224083038 mail-test-srv 9qPqagA mail-test-srv: Got connection from 127.0.0.1:49726
boulder_1 | 2021/02/24 08:30:38 mail-test-srv: 127.0.0.1:49726: readline: EOF
boulder_1 | 2021/02/24 08:30:38 ct-test-srv on :4511 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEFRu37ZRLg8lT4rVQwMwh4oAOpXb4Sx+9hgQ+JFCjmAv3oDV+sDOMsC7hULkGTn+LB5L1SRo/XIY4Kw5V+nFXgg==
boulder_1 | 2021/02/24 08:30:38 ct-test-srv on :4512 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEFRu37ZRLg8lT4rVQwMwh4oAOpXb4Sx+9hgQ+JFCjmAv3oDV+sDOMsC7hULkGTn+LB5L1SRo/XIY4Kw5V+nFXgg==
boulder_1 | 2021/02/24 08:30:38 ct-test-srv on :4510 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEyw1HymhJkuxSIgt3gqW3sVXqMqB3EFsXcMfPFo0vYwjNiRmCJDXKsR0Flp7MAK+wc3X/7Hpc8liUbMhPet7tEA==
boulder_1 | 2021/02/24 08:30:38 ct-test-srv on :4501 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEKtnFevaXV/kB8dmhCNZHmxKVLcHX1plaAsY9LrKilhYxdmQZiu36LvAvosTsqMVqRK9a96nC8VaxAdaHUbM8EA==
boulder_1 | 2021/02/24 08:30:38 ct-test-srv on :4500 with pubkey MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEYggOxPnPkzKBIhTacSYoIfnSL2jPugcbUKx83vFMvk5gKAz/AGe87w20riuPwEGn229hKVbEKHFB61NIqNHC3Q==
boulder_1 | I210224083039 boulder-remoteva 0MLlrww Versions: boulder-remoteva=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to va1.boulder:9098 health service
boulder_1 | I210224083039 boulder-publisher 6uKbhAo Versions: boulder-publisher=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to publisher2.boulder:9091 health service
boulder_1 | I210224083040 ocsp-responder tLqf8AI Versions: ocsp-responder=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | I210224083040 ocsp-responder mbD4lQM Loading OCSP Database for CA Cert: labca/test-ca.pem
boulder_1 | Connecting to nonce1.boulder:9101 health service
boulder_1 | I210224083040 nonce-service l6
-0Q4 Versions: nonce-service=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | I210224083041 boulder-sa kcXg9QE Versions: boulder-sa=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to sa2.boulder:9095 health service
boulder_1 | I210224083042 nonce-service l6
-0Q4 Versions: nonce-service=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to nonce2.boulder:9101 health service
boulder_1 | Connecting to ca1.boulder:9093 health service
boulder_1 | I210224083042 boulder-ca 5u-yxAg Versions: boulder-ca=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | I210224083042 boulder-ca 9OXLlwI loading hostname policy, sha256: 0be6f6061a1041f866b71037f1aa96e0335be2035e0e8657aa203129f2d13f8d
boulder_1 | got error connecting to health service ca1.boulder:9093: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ca1.boulder:9093 health service
boulder_1 | got error connecting to health service ca1.boulder:9093: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ca1.boulder:9093 health service
boulder_1 | got error connecting to health service ca1.boulder:9093: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ca1.boulder:9093 health service
boulder_1 | got error connecting to health service ca1.boulder:9093: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ca1.boulder:9093 health service
boulder_1 | got error connecting to health service ca1.boulder:9093: grpc.health.v1.Health.Check timed out after 9921 ms
boulder_1 | Connecting to ca1.boulder:9093 health service
boulder_1 | I210224083132 boulder-va l_rfyQ8 Versions: boulder-va=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to va2.boulder:9092 health service
boulder_1 | I210224083132 akamai-purger 8tvK_Q8 Versions: akamai-purger=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | I210224083133 boulder-va l_rfyQ8 Versions: boulder-va=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | Connecting to va1.boulder:9092 health service
boulder_1 | I210224083133 boulder-ca 5u-yxAg Versions: boulder-ca=(Unspecified Unspecified) Golang=(go1.15.7) BuildHost=(Unspecified)
boulder_1 | I210224083133 boulder-ca 9OXLlwI loading hostname policy, sha256: 0be6f6061a1041f866b71037f1aa96e0335be2035e0e8657aa203129f2d13f8d
boulder_1 | Connecting to ca2.boulder:9093 health service
boulder_1 | I210224083135 log-validator lIPjxAo time=2021-02-24T08:31:35.692201758Z
boulder_1 | �[31m�[1mE210224083136 boulder-ca pLaPhgw [AUDIT] Failed to open orphaned certificate queue: leveldb: manifest corrupted (field 'comparer'): missing [file=MANIFEST-002965]�[0m
boulder_1 | Failed to open orphaned certificate queue: leveldb: manifest corrupted (field 'comparer'): missing [file=MANIFEST-002965]
boulder_1 | I210224083136 boulder-publisher ssjTvgE time=2021-02-24T08:31:36.850346894Z
boulder_1 | I210224083137 boulder-sa qL68rAw time=2021-02-24T08:31:37.409483707Z
boulder_1 | I210224083138 boulder-remoteva vsTZnAg time=2021-02-24T08:31:38.489524589Z
boulder_1 | I210224083139 boulder-remoteva nYGkrg0 time=2021-02-24T08:31:39.003388915Z
boulder_1 | I210224083139 boulder-publisher pojTzwk time=2021-02-24T08:31:39.208425837Z
boulder_1 | I210224083140 ocsp-responder j9uvxQY time=2021-02-24T08:31:40.63336142Z
boulder_1 | I210224083140 nonce-service o7Pa3Ao time=2021-02-24T08:31:40.883245041Z
boulder_1 | I210224083141 boulder-sa 9qyyhgE time=2021-02-24T08:31:41.961278372Z
boulder_1 | I210224083142 nonce-service 56PCvwE time=2021-02-24T08:31:42.143711999Z
boulder_1 | I210224083142 boulder-ca oeDP3As time=2021-02-24T08:31:42.570492571Z
boulder_1 | got error connecting to health service ca2.boulder:9093: grpc.health.v1.Health.Check timed out after 9999 ms
boulder_1 | Connecting to ca2.boulder:9093 health service
boulder_1 | got error connecting to health service ca2.boulder:9093: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ca2.boulder:9093 health service
boulder_1 | got error connecting to health service ca2.boulder:9093: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ca2.boulder:9093 health service
boulder_1 | got error connecting to health service ca2.boulder:9093: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ca2.boulder:9093 health service
boulder_1 | got error connecting to health service ca2.boulder:9093: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ca2.boulder:9093 health service
boulder_1 | I210224083232 boulder-va 67bM2Ag time=2021-02-24T08:32:32.477705967Z
boulder_1 | I210224083232 akamai-purger h82X-A4 time=2021-02-24T08:32:32.862682795Z
boulder_1 | I210224083233 boulder-va v42T1AE time=2021-02-24T08:32:33.075885647Z
boulder_1 | got error connecting to health service ca2.boulder:9093: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ca2.boulder:9093 health service
boulder_1 | I210224083235 log-validator jKjy7gM time=2021-02-24T08:32:35.696795194Z
boulder_1 | I210224083236 boulder-publisher 7vmUzgQ time=2021-02-24T08:32:36.851387189Z
boulder_1 | I210224083237 boulder-sa k8r26gU time=2021-02-24T08:32:37.410076252Z
boulder_1 | I210224083238 boulder-remoteva 0aWvngg time=2021-02-24T08:32:38.489792817Z
boulder_1 | I210224083239 boulder-remoteva -uTEoAk time=2021-02-24T08:32:39.003691477Z
boulder_1 | I210224083239 boulder-publisher 1IiZpgg time=2021-02-24T08:32:39.208804012Z
boulder_1 | I210224083240 ocsp-responder ypDT2AQ time=2021-02-24T08:32:40.633743894Z
boulder_1 | I210224083240 nonce-service osKQvQU time=2021-02-24T08:32:40.883600933Z
boulder_1 | I210224083241 boulder-sa -_zjtAw time=2021-02-24T08:32:41.961846035Z
boulder_1 | I210224083242 nonce-service 4Nr5jw0 time=2021-02-24T08:32:42.144115678Z
boulder_1 | I210224083242 boulder-ca pemKiQY time=2021-02-24T08:32:42.570797682Z
boulder_1 | got error connecting to health service ca2.boulder:9093: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ca2.boulder:9093 health service
boulder_1 | got error connecting to health service ca2.boulder:9093: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ca2.boulder:9093 health service
boulder_1 | got error connecting to health service ca2.boulder:9093: grpc.health.v1.Health.Check timed out after 10000 ms
boulder_1 | Connecting to ca2.boulder:9093 health service
boulder_1 | got error connecting to health service ca2.boulder:9093: grpc.health.v1.Health.Check timed out after 9892 ms
boulder_1 | Connecting to ca2.boulder:9093 health service
boulder_1 | got error connecting to health service ca2.boulder:9093: grpc.health.v1.Health.Check timed out after 0 ms
boulder_1 | timed out waiting for ca2.boulder:9093 health check
boulder_1 | �[31m�[1mE210224083313 health-checker h9Hfjw0 [AUDIT] timed out waiting for ca2.boulder:9093 health check�[0m
boulder_1 | Waiting for debug port 8055 (pebble-challtestsrv --defaultIPv4 10.77.77.77 -defaultIPv6 --dns01 :8053,:8054 --management :8055 --http01 10.77.77.77:5002 -https01 10.77.77.77:5001 --tlsalpn01 10.88.88.88:5001)
boulder_1 | Starting service log-validator
boulder_1 | Waiting for debug port 8016 (./bin/log-validator --config labca/config/log-validator.json)
boulder_1 | Waiting for debug port 8016 (./bin/log-validator --config labca/config/log-validator.json)
boulder_1 | Starting service sd-test-srv
boulder_1 | Starting service boulder-publisher-1
boulder_1 | Starting service boulder-sa-1
boulder_1 | Starting service boulder-remoteva-a
boulder_1 | Starting service mail-test-srv
boulder_1 | Starting service akamai-test-srv
boulder_1 | Starting service ct-test-srv
boulder_1 | Starting service boulder-remoteva-b
boulder_1 | Starting service boulder-publisher-2
boulder_1 | Starting service ocsp-responder
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Waiting for debug port 8005 (./bin/ocsp-responder --config labca/config/ocsp-responder.json)
boulder_1 | Starting service nonce-service-taro
boulder_1 | Starting service boulder-sa-2
boulder_1 | Starting service nonce-service-zinc
boulder_1 | Starting service boulder-ca-a
boulder_1 | Starting service boulder-va-2
boulder_1 | Starting service akamai-purger
boulder_1 | Waiting for debug port 9666 (./bin/akamai-purger --config labca/config/akamai-purger.json)
boulder_1 | Waiting for debug port 9666 (./bin/akamai-purger --config labca/config/akamai-purger.json)
boulder_1 | Waiting for debug port 9666 (./bin/akamai-purger --config labca/config/akamai-purger.json)
boulder_1 | Starting service boulder-va-1
boulder_1 | Starting service boulder-ca-b
boulder_1 | Error starting service boulder-ca-b: Command '['./bin/health-checker', '-addr', 'ca2.boulder:9093', '-config', 'labca/config/health-checker.json']' returned non-zero exit status 1.
boulder_1 | I210224083313 boulder-va kKG6cwA Caught SIGTERM
boulder_1 | I210224083313 boulder-va htX9nwc Exiting
boulder_1 | I210224083313 akamai-purger kKG6cwA Caught SIGTERM
boulder_1 | I210224083313 akamai-purger 2f7Ciwo Shutting down; queue is already empty.
boulder_1 | I210224083313 akamai-purger htX9nwc Exiting
boulder_1 | I210224083313 boulder-va kKG6cwA Caught SIGTERM
boulder_1 | I210224083313 boulder-va htX9nwc Exiting
boulder_1 | I210224083313 boulder-ca kKG6cwA Caught SIGTERM
boulder_1 | I210224083313 boulder-ca htX9nwc Exiting
boulder_1 | I210224083313 nonce-service kKG6cwA Caught SIGTERM
boulder_1 | I210224083313 nonce-service htX9nwc Exiting
boulder_1 | I210224083313 boulder-sa kKG6cwA Caught SIGTERM
boulder_1 | I210224083313 boulder-sa htX9nwc Exiting
boulder_1 | I210224083313 nonce-service kKG6cwA Caught SIGTERM
boulder_1 | I210224083313 nonce-service htX9nwc Exiting
boulder_1 | I210224083313 ocsp-responder kKG6cwA Caught SIGTERM
boulder_1 | I210224083313 boulder-va kKG6cwA Caught SIGTERM
boulder_1 | I210224083313 boulder-va htX9nwc Exiting
boulder_1 | I210224083313 akamai-purger kKG6cwA Caught SIGTERM
boulder_1 | I210224083313 akamai-purger 2f7Ciwo Shutting down; queue is already empty.
boulder_1 | I210224083313 akamai-purger htX9nwc Exiting
boulder_1 | I210224083313 boulder-va kKG6cwA Caught SIGTERM
boulder_1 | I210224083313 boulder-va htX9nwc Exiting
boulder_1 | I210224083313 boulder-ca kKG6cwA Caught SIGTERM
boulder_1 | I210224083313 boulder-ca htX9nwc Exiting
boulder_1 | I210224083313 nonce-service kKG6cwA Caught SIGTERM
boulder_1 | I210224083313 nonce-service htX9nwc Exiting
boulder_1 | I210224083313 boulder-sa kKG6cwA Caught SIGTERM
boulder_1 | I210224083313 boulder-sa htX9nwc Exiting
boulder_1 | I210224083313 nonce-service kKG6cwA Caught SIGTERM
boulder_1 | I210224083313 nonce-service htX9nwc Exiting
boulder_1 | I210224083313 ocsp-responder kKG6cwA Caught SIGTERM
boulder_1 | I210224083313 boulder-publisher kKG6cwA Caught SIGTERM
boulder_1 | I210224083313 boulder-publisher htX9nwc Exiting
boulder_1 | I210224083313 boulder-remoteva kKG6cwA Caught SIGTERM
boulder_1 | I210224083313 boulder-remoteva htX9nwc Exiting
boulder_1 | I210224083313 boulder-remoteva kKG6cwA Caught SIGTERM
boulder_1 | I210224083313 boulder-remoteva htX9nwc Exiting
boulder_1 | I210224083313 boulder-sa kKG6cwA Caught SIGTERM
boulder_1 | I210224083313 boulder-publisher kKG6cwA Caught SIGTERM
boulder_1 | I210224083313 boulder-publisher htX9nwc Exiting
boulder_1 | I210224083313 log-validator kKG6cwA Caught SIGTERM
boulder_1 | I210224083313 log-validator htX9nwc Exiting
boulder_1 | pebble-challtestsrv - 2021/02/24 08:33:13 Caught signals. Shutting down
boulder_1 | 2021/02/24 08:33:13 Caught SIGTERM
boulder_1 | pebble-challtestsrv - 2021/02/24 08:33:13 http: Server closed
boulder_1 | pebble-challtestsrv - 2021/02/24 08:33:14 http: Server closed
boulder_1 | pebble-challtestsrv - 2021/02/24 08:33:14 http: Server closed
boulder_1 | pebble-challtestsrv - 2021/02/24 08:33:15 Shutting down management server
boulder_1 | pebble-challtestsrv - 2021/02/24 08:33:15 http: Server closed
boulder_1 | pebble-challtestsrv - 2021/02/24 08:33:15 Goodbye!
boulder_1 | 2021/02/24 08:33:15 Exiting

hakwerk added a commit that referenced this issue Feb 28, 2021
hakwerk added a commit that referenced this issue Feb 28, 2021
hakwerk added a commit that referenced this issue Feb 28, 2021
hakwerk added a commit that referenced this issue Mar 3, 2021
* release/21.03:
  Prevent warning when updating LabCA
  Lower frequency of looking for file changes of hostname and rate-limit policies
  Do not wait for boulder startup when setup is not yet complete
  Wait longer for server to be ready before requesting first certificate
  Tweak description string of whitelist mode
  Fix config substitutions for whitelist mode (#15)
  Hold off on starting boulder until setup wizzard is completed
  Make installer more robust
  Make config substitutions more robust
  Stick to release tags even if there are newer commits on master
  Minor tweaks
  Fix query buildup when combining tables
@hakwerk
Copy link
Owner

hakwerk commented Mar 3, 2021

I think I finally have solved the problem. Please try updating your instance (by running /home/labca/labca/install again), hopefully it works now.
I also made some other improvements to make things more robust in general.

@SebastianMoinhinwech
Copy link
Author

Hi

thanks for your work and support. I did a fresh install of debian and installed it from beginning:

No errors in the ACME Log, but still stuck on reloading the page for creating new certificate pair.

So, i found this in the Certificate Log:

Thu 04 Mar 2021 09:28:41 AM CET
Parsing account key...
Parsing CSR...
Found domains: linux5.itschnack.local
Getting directory...
Directory found!
Registering account...
Already registered!
Creating new order...
Traceback (most recent call last):
File "/home/labca/acme_tiny.py", line 197, in
main(sys.argv[1:])
File "/home/labca/acme_tiny.py", line 193, in main
signed_crt = get_crt(args.account_key, args.csr, args.acme_dir, log=LOGGER, CA=args.ca, disable_check=args.disable_check, directory_url=args.directory_url, contact=args.contact)
File "/home/labca/acme_tiny.py", line 120, in get_crt
order, _, order_headers = _send_signed_request(directory['newOrder'], order_payload, "Error creating new order")
File "/home/labca/acme_tiny.py", line 59, in _send_signed_request
return _do_request(url, data=data.encode('utf8'), err_msg=err_msg, depth=depth)
File "/home/labca/acme_tiny.py", line 45, in _do_request
raise ValueError("{0}:\nUrl: {1}\nData: {2}\nResponse Code: {3}\nResponse: {4}".format(err_msg, url, data, code, resp_data))
ValueError: Error creating new order:
Url: https://linux5.itschnack.local/acme/new-order
Data: {"protected": "eyJ1cmwiOiAiaHR0cHM6Ly9saW51eDUuaXRzY2huYWNrLmxvY2FsL2FjbWUvbmV3LW9yZGVyIiwgImFsZyI6ICJSUzI1NiIsICJub25jZSI6ICJ6aW5jNUxXTEstWmQ0aXB6UGhZZ2pLZUJIaFgwdkdnZ3cyWHg5cmU1eUNWYWxBNCIsICJraWQiOiAiaHR0cHM6Ly9saW51eDUuaXRzY2huYWNrLmxvY2FsL2FjbWUvYWNjdC8yIn0", "payload": "eyJpZGVudGlmaWVycyI6IFt7InR5cGUiOiAiZG5zIiwgInZhbHVlIjogImxpbnV4NS5pdHNjaG5hY2subG9jYWwifV19", "signature": "dT7VZRBaZJUgTXcUOs-XKmFE2OxLNTMSjD8TlocM5XLZvgHU3vTmARJVKHr6P-UlW9SEEkCaVX886FxzBi8GTYAG3gqfX7L4T0Cq6AYcpEOFMC_ftYC3dfK68QxFfkbjgnd4A4x-PV5SiQjyYy-MK8kOkRWcNNe2XV-XNCeelqO9Ppg3_hPvCLzEFBFlE9343BFeC9L5lwe5EeSt-80kAQaBqOsOk9keT5yHuzz1ITxlXKGrY_OtmcsKNtjDMPxV0IVXsJAHzAO7ZneHGoodkdSspXQts69fqrXewg6TyNVqzH9bjgpIVzYbdDosPcaUS7G9Tqn9-7TdQFOTlFWircsd5EE4O0H9UPnSQWIvCdm92lwnEfRhyj1oBNCguqx4XWD2CaIAikGheRoPxD8WtuBwp0NIdsix_2NLsMT62ioa5eB06WQY7ZK0eHwShoehhORhW3GmMZqc8ktemeqek4HzEn6rxgNVpToXR0N0JK9VAbA1P2xmDklT2Wdqg7AcB5jGAtxBiHBvA52Y60jtfBjORvS76W08UDp7kJuhkkBAgtYlIMMFNA90C6gRW4TZJ_fCtIp5IHXBhBKQXzAdMKg017wpvvtiLCVLiBYd3UftTiTG9JJ8uORDHU1_yPzWKT8mkrZMOoHOmM6i2FiXxTj8UH1_YFN0jH9kYSMnY2g"}
Response Code: 400
Response: {
"type": "urn:ietf:params:acme:error:rejectedIdentifier",
"detail": "Error creating new order :: Cannot issue for "linux5.itschnack.local": Domain name does not end with a valid public suffix (TLD)",
"status": 400
}

Now i wondering, why i can`t create .local Certs with a private PKI :)

@SebastianMoinhinwech
Copy link
Author

Ok, got it, restarted the whole process and now i got it running! thank you very much, what i can see, looks very good!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants