gotosocial: 0.14.2 -> 0.15.0 #303321

jugendhacker · 2024-04-11T10:18:50Z

Description of changes

New release: https://github.com/superseriousbusiness/gotosocial/releases/tag/v0.15.0

Things done

Add a 👍 reaction to pull requests you find important.

misuzu

TestValidationTestSuite/TestValidateEmail is fixed now (thank you, @blakesmith!), we should remove it from skippedTests

misuzu · 2024-04-13T07:18:20Z

Looks like another flaky test

ok      github.com/superseriousbusiness/gotosocial/internal/processing/polls    0.385s
timestamp="11/04/2024 15:23:30.311" func=dereferencing.(*Dereferencer).RefreshStatusAsync.func1 level=ERROR msg="error enriching remote status: enrichStatus: error dereferencing http://fossbros-anonymous.io/users/foss_satan/statuses/01FVW7JHQFSFK166WWKR8CBA6M: Dereference: GET request to http://fossbros-anonymous.io/users/foss_satan/statuses/01FVW7JHQFSFK166WWKR8CBA6M failed: status=\"\" body=\"{\"error\":\"404 not found\"}\""
--- FAIL: TestStatusCreateTestSuite (6.56s)
    --- FAIL: TestStatusCreateTestSuite/TestProcessReplyToUnthreadedRemoteStatus (0.88s)
        create_test.go:276: 
                Error Trace:    /build/source/internal/processing/status/create_test.go:276
                Error:          Received unexpected error:
                                GetAPIStatus: error converting status: statusToFrontend: error converting status author: AccountToAPIAccountPublic: error counting statuses: database table is locked (262)
                Test:           TestStatusCreateTestSuite/TestProcessReplyToUnthreadedRemoteStatus
        create_test.go:277: 
                Error Trace:    /build/source/internal/processing/status/create_test.go:277
                Error:          Expected value not to be nil.
                Test:           TestStatusCreateTestSuite/TestProcessReplyToUnthreadedRemoteStatus
        panic.go:261: test panicked: runtime error: invalid memory address or nil pointer dereference
            goroutine 557 [running]:
            runtime/debug.Stack()
                /nix/store/mzg3cka0bbr5jq96ysymwziw74fnk22m-go-1.22.1/share/go/src/runtime/debug/stack.go:24 +0x5e
            github.com/stretchr/testify/suite.failOnPanic(0xc000ed31e0, {0x21caac0, 0x3e7eea0})
                /build/source/vendor/github.com/stretchr/testify/suite/suite.go:89 +0x37
            github.com/stretchr/testify/suite.Run.func1.1()
                /build/source/vendor/github.com/stretchr/testify/suite/suite.go:188 +0x299
            panic({0x21caac0?, 0x3e7eea0?})
                /nix/store/mzg3cka0bbr5jq96ysymwziw74fnk22m-go-1.22.1/share/go/src/runtime/panic.go:770 +0x132
            github.com/superseriousbusiness/gotosocial/internal/processing/status_test.(*StatusCreateTestSuite).TestProcessReplyToUnthreadedRemoteStatus(0xc00026c608)
                /build/source/internal/processing/status/create_test.go:282 +0x20f
            reflect.Value.call({0xc001374000?, 0xc000615ea0?, 0xc009d2eb40?}, {0x2493284, 0x4}, {0xc00c007f28, 0x1, 0x21e1cc0?})
                /nix/store/mzg3cka0bbr5jq96ysymwziw74fnk22m-go-1.22.1/share/go/src/reflect/value.go:596 +0xce5
            reflect.Value.Call({0xc001374000?, 0xc000615ea0?, 0x37d2f50?}, {0xc00c007f28?, 0xf?, 0x0?})
                /nix/store/mzg3cka0bbr5jq96ysymwziw74fnk22m-go-1.22.1/share/go/src/reflect/value.go:380 +0xb9
            github.com/stretchr/testify/suite.Run.func1(0xc000ed31e0)
                /build/source/vendor/github.com/stretchr/testify/suite/suite.go:202 +0x4a5
            testing.tRunner(0xc000ed31e0, 0xc000cb5e60)
                /nix/store/mzg3cka0bbr5jq96ysymwziw74fnk22m-go-1.22.1/share/go/src/testing/testing.go:1689 +0xfb
            created by testing.(*T).Run in goroutine 350
                /nix/store/mzg3cka0bbr5jq96ysymwziw74fnk22m-go-1.22.1/share/go/src/testing/testing.go:1742 +0x390
FAIL
FAIL    github.com/superseriousbusiness/gotosocial/internal/processing/status   9.361s
FAIL
error: builder for '/nix/store/aw52fm9577jdl83zp4gvziadg66nbznj-gotosocial-0.15.0.drv' failed with exit code 1;

Builds fine on my machine

(finished: must succeed: curl -sS -f http://localhost:8081/nodeinfo/2.0 | jq '.usage.users.total' | grep -q '^1$', in 0.06 seconds)
(finished: run the VM test script, in 17.70 seconds)
test script finished in 17.78s
cleanup
kill machine (pid 9)
qemu-kvm: terminating on signal 15 from pid 6 (/nix/store/gd3shnza1i50zn8zs04fa729ribr88m9-python3-3.11.8/bin/python3.11)
(finished: cleanup, in 0.01 seconds)
kill vlan (pid 7)
/nix/store/frdz9vsrz72gbyk3jxnpndh2ns00gv6n-vm-test-run-gotosocial

blakesmith · 2024-04-13T11:19:14Z

@misuzu I saw this database is locked test failure sporadically as well on early 0.14 versions, but could only get it to reproduce via ofborg runs (never on my local x86 linux machine). I'm not familiar with how ofborg is setup to know if there's any configuration that would lead to more likely failures.

Looks like something might be accidentally holding the lock to the sqlite database beyond the timeout, that's causing the build failure. By default, the gotosocial test suite runs with an in-memory sqlite database, so there must be some additional concurrency issues happening here during this test to cause the issue.

There was some recent work in superseriousbusiness/gotosocial#2774 to fix some "database is locked" bugs happening with a live instance, but not sure if this is related.

blakesmith · 2024-04-13T11:54:04Z

Looks like our gotosocial friends upstream have seen the same sporadic test suite failures due to in-memory sqlite. See: superseriousbusiness/gotosocial#2774 (comment).

I wonder if there's a way we can configure the tests to use a temporary persistent sqlite db on the filesystem (instead of in-memory). This would also run the same way it would run in production via PRAGMA journal_mode=WAL, which is better at handling concurrent readers / writers. The test suite supports testing via a persistent postgres instance, so this should theoretically be possible!

That being said, I'm going to:

Rerun ofborg checks for now. It would be nice to get to 0.15, since this is not a real failure in the package that needs fixing.
Investigate possible ways to run the test suite in a way that more accurately represents how the code will run when in production. It would be nice to not have this issue continue cropping up during package updates.
(Maybe): See if there are other ways to make the tests run more reliably with sqlite in-memory.

blakesmith · 2024-04-13T11:56:01Z

@ofborg test gotosocial

blakesmith · 2024-04-13T11:57:15Z

@ofborg build gotosocial

blakesmith · 2024-04-13T21:15:06Z

Update: I can get several tests to reliably fail in gotosocial's upstream test suite by running them repeatedly. Like I mentioned before, this seems like a broader issue with how the test suite runs via an in-memory sqlite database, which doesn't have the same concurrency semantics as when gotosocial runs on a real instance with sqlite, and WAL mode enabled.

With gotosocial checked out, you can pretty reliably get the test suite to fail by rerunning a test many times:

$ go test ./... -count=100 -run=TestStatusCreateTestSuite/TestProcessReplyToUnthreadedRemoteStatus

Leads to many test failures such as:

--- FAIL: TestStatusCreateTestSuite (0.90s)
    --- FAIL: TestStatusCreateTestSuite/TestProcessReplyToUnthreadedRemoteStatus (0.90s)
        create_test.go:276:
                Error Trace:    /home/blake/src/gotosocial/internal/processing/status/create_test.go:276
                Error:          Received unexpected error:
                                GetAPIStatus: error converting status: statusToFrontend: error converting status author: AccountToAPIAccountPublic: error counting statuses: database table is locked (262)
                Test:           TestStatusCreateTestSuite/TestProcessReplyToUnthreadedRemoteStatus
        create_test.go:277:
                Error Trace:    /home/blake/src/gotosocial/internal/processing/status/create_test.go:277
                Error:          Expected value not to be nil.
                Test:           TestStatusCreateTestSuite/TestProcessReplyToUnthreadedRemoteStatus
        panic.go:261: test panicked: runtime error: invalid memory address or nil pointer dereference
            goroutine 5830 [running]:
            runtime/debug.Stack()
                /nix/store/vxxlsbznhhkw1sb7qi5wki03d2sihpha-go-1.21.8/share/go/src/runtime/debug/stack.go:24 +0x5e
            github.com/stretchr/testify/suite.failOnPanic(0xc0a0a7cd00, {0x21370a0, 0x3ba7c80})
                /home/blake/src/gotosocial/vendor/github.com/stretchr/testify/suite/suite.go:89 +0x37
            github.com/stretchr/testify/suite.Run.func1.1()
                /home/blake/src/gotosocial/vendor/github.com/stretchr/testify/suite/suite.go:188 +0x25d
            panic({0x21370a0?, 0x3ba7c80?})
                /nix/store/vxxlsbznhhkw1sb7qi5wki03d2sihpha-go-1.21.8/share/go/src/runtime/panic.go:914 +0x21f
            github.com/superseriousbusiness/gotosocial/internal/processing/status_test.(*StatusCreateTestSuite).TestProcessReplyToUnthreadedRemoteStatus(0xc0a5080000)
                /home/blake/src/gotosocial/internal/processing/status/create_test.go:282 +0x22a
            reflect.Value.call({0xc000948880?, 0xc09d8f4a20?, 0xc0a72e1060?}, {0x23f5127, 0x4}, {0xc08f57be70, 0x1, 0xc08f57bd78?})
                /nix/store/vxxlsbznhhkw1sb7qi5wki03d2sihpha-go-1.21.8/share/go/src/reflect/value.go:596 +0xce7
            reflect.Value.Call({0xc000948880?, 0xc09d8f4a20?, 0xc0a5080000?}, {0xc08f57be70?, 0x535fad?, 0x357d5b0?})
                /nix/store/vxxlsbznhhkw1sb7qi5wki03d2sihpha-go-1.21.8/share/go/src/reflect/value.go:380 +0xb9
            github.com/stretchr/testify/suite.Run.func1(0xc0a0a7cd00)
                /home/blake/src/gotosocial/vendor/github.com/stretchr/testify/suite/suite.go:202 +0x467
            testing.tRunner(0xc0a0a7cd00, 0xc09d8cefc0)
                /nix/store/vxxlsbznhhkw1sb7qi5wki03d2sihpha-go-1.21.8/share/go/src/testing/testing.go:1595 +0xff
            created by testing.(*T).Run in goroutine 5829
                /nix/store/vxxlsbznhhkw1sb7qi5wki03d2sihpha-go-1.21.8/share/go/src/testing/testing.go:1648 +0x3ad

EDIT: However! I cannot get the tests to fail when running them with an actual sqlite database on the filesystem. This works:

GTS_DB_TYPE="sqlite" GTS_DB_ADDRESS="/tmp/test_gotosocial.db" go test ./... -count=100 -run=TestStatusCreateTestSuite/TestProcessReplyToUnthreadedRemoteStatus

@misuzu Does it make sense to run our check tests somehow with something temporary on the filesystem? I worry about portability, and state on the machine, but there might be other examples in nixpkgs that do something like this that I'm not aware of.

Will be digging into this more soon on the upstream side.

misuzu · 2024-04-14T07:33:18Z

Does it make sense to run our check tests somehow with something temporary on the filesystem?

I think it should be fine if it doesn't write anything to the $out. Something like this should do the trick:

env = {
  GTS_DB_TYPE = "sqlite";
  GTS_DB_ADDRESS = "/tmp/test_gotosocial.db";
};

blakesmith · 2024-04-14T11:53:22Z

Looks like running the full test suite with a persistent, on-disk sqlite has other issues. Several tests that fail due to state being carried across tests. One example (amongst many):

   --- FAIL: TestTokenTestSuite/TestRetrieveClientCredentialsOK (0.99s)
        log.go:148: test panicked: already exists
            goroutine 116 [running]:
            runtime/debug.Stack()
                /nix/store/vxxlsbznhhkw1sb7qi5wki03d2sihpha-go-1.21.8/share/go/src/runtime/debug/stack.go:24 +0x5e
            github.com/stretchr/testify/suite.failOnPanic(0xc002020820, {0x21c7600, 0xc001b88460})
                /home/blake/src/gotosocial/vendor/github.com/stretchr/testify/suite/suite.go:89 +0x37
            github.com/stretchr/testify/suite.Run.func1.1()
                /home/blake/src/gotosocial/vendor/github.com/stretchr/testify/suite/suite.go:188 +0x25d
            panic({0x21c7600?, 0xc001b88460?})
                /nix/store/vxxlsbznhhkw1sb7qi5wki03d2sihpha-go-1.21.8/share/go/src/runtime/panic.go:914 +0x21f
            github.com/superseriousbusiness/gotosocial/internal/log.Panic.func1()
                /home/blake/src/gotosocial/internal/log/log.go:148 +0x25
            github.com/superseriousbusiness/gotosocial/internal/log.Panic({0x0, 0x0}, {0xc000700a20, 0x1, 0x1})
                /home/blake/src/gotosocial/internal/log/log.go:150 +0x117
            github.com/superseriousbusiness/gotosocial/testrig.StandardDBSetup({0x2ac74e8, 0xc00217e960}, 0xc00221b2f0)
                /home/blake/src/gotosocial/testrig/db.go:144 +0x151
            github.com/superseriousbusiness/gotosocial/internal/api/auth_test.(*AuthStandardTestSuite).SetupTest(0xc00020d900)
                /home/blake/src/gotosocial/internal/api/auth/auth_test.go:95 +0x32a
            github.com/stretchr/testify/suite.Run.func1(0xc002020820)
                /home/blake/src/gotosocial/vendor/github.com/stretchr/testify/suite/suite.go:192 +0x1ee
            testing.tRunner(0xc002020820, 0xc000acecf0)
                /nix/store/vxxlsbznhhkw1sb7qi5wki03d2sihpha-go-1.21.8/share/go/src/testing/testing.go:1595 +0xff
            created by testing.(*T).Run in goroutine 59
                /nix/store/vxxlsbznhhkw1sb7qi5wki03d2sihpha-go-1.21.8/share/go/src/testing/testing.go:1648 +0x3ad

So we're back to where we started.

@misuzu Are we at the point where we should skip the upstream test suite during derivation build? I have a sneaking suspicion we will continue to play wack-a-mole with flakey tests until we can help the upstream gotosocial team with broader test reliability improvements.

misuzu · 2024-04-14T11:59:39Z

Are we at the point where we should skip the upstream test suite during derivation build?

Yeah, I guess we can run them manually on each update to make sure the package is not totally broken

jugendhacker · 2024-04-16T09:29:23Z

Okay, so should I disable all tests for now in this PR?

misuzu · 2024-04-16T12:47:39Z

Okay, so should I disable all tests for now in this PR?

Yes, please

pkgs/servers/gotosocial/default.nix

It was fixed by superseriousbusiness/gotosocial#2760

Tests seem to be very unstable currently, so we should do them manually for now, see: NixOS#303321 (comment)

misuzu

Thank you!

ofborg bot requested review from blakesmith and misuzu April 11, 2024 10:46

ofborg bot added 10.rebuild-darwin: 1-10 10.rebuild-darwin: 1 10.rebuild-linux: 1-10 10.rebuild-linux: 1 labels Apr 11, 2024

misuzu requested changes Apr 11, 2024

View reviewed changes

jugendhacker requested a review from misuzu April 11, 2024 14:35

blakesmith mentioned this pull request Apr 13, 2024

[bug/sqlite] Instance can get in a state where every db query returns database is locked (517) superseriousbusiness/gotosocial#2774

Closed

jugendhacker force-pushed the update-gotosocial branch from 84195f1 to f9961b3 Compare April 17, 2024 11:55

misuzu reviewed Apr 19, 2024

View reviewed changes

pkgs/servers/gotosocial/default.nix Outdated Show resolved Hide resolved

jugendhacker added 3 commits April 20, 2024 14:11

gotosocial: 0.14.2 -> 0.15.0

cb0ecfc

gotosocial: do not skip TestValidateEmail

d244dc9

It was fixed by superseriousbusiness/gotosocial#2760

gotosocial: do not run test for now

f92bbde

Tests seem to be very unstable currently, so we should do them manually for now, see: NixOS#303321 (comment)

jugendhacker force-pushed the update-gotosocial branch from f9961b3 to f92bbde Compare April 20, 2024 12:13

jugendhacker requested a review from misuzu April 20, 2024 12:13

misuzu approved these changes Apr 22, 2024

View reviewed changes

wegank added 12.approvals: 1 This PR was reviewed and approved by one reputable person 12.approved-by: package-maintainer This PR was reviewed and approved by a maintainer listed in the package labels Apr 23, 2024

pbsds merged commit 115bbb6 into NixOS:master Apr 24, 2024
26 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gotosocial: 0.14.2 -> 0.15.0 #303321

gotosocial: 0.14.2 -> 0.15.0 #303321

jugendhacker commented Apr 11, 2024

misuzu left a comment

misuzu commented Apr 13, 2024

blakesmith commented Apr 13, 2024 •

edited

Loading

blakesmith commented Apr 13, 2024

blakesmith commented Apr 13, 2024 •

edited

Loading

blakesmith commented Apr 13, 2024

blakesmith commented Apr 13, 2024 •

edited

Loading

misuzu commented Apr 14, 2024

blakesmith commented Apr 14, 2024 •

edited

Loading

misuzu commented Apr 14, 2024

jugendhacker commented Apr 16, 2024

misuzu commented Apr 16, 2024

misuzu left a comment

gotosocial: 0.14.2 -> 0.15.0 #303321

gotosocial: 0.14.2 -> 0.15.0 #303321

Conversation

jugendhacker commented Apr 11, 2024

Description of changes

Things done

misuzu left a comment

Choose a reason for hiding this comment

misuzu commented Apr 13, 2024

blakesmith commented Apr 13, 2024 • edited Loading

blakesmith commented Apr 13, 2024

blakesmith commented Apr 13, 2024 • edited Loading

blakesmith commented Apr 13, 2024

blakesmith commented Apr 13, 2024 • edited Loading

misuzu commented Apr 14, 2024

blakesmith commented Apr 14, 2024 • edited Loading

misuzu commented Apr 14, 2024

jugendhacker commented Apr 16, 2024

misuzu commented Apr 16, 2024

misuzu left a comment

Choose a reason for hiding this comment

blakesmith commented Apr 13, 2024 •

edited

Loading

blakesmith commented Apr 13, 2024 •

edited

Loading

blakesmith commented Apr 13, 2024 •

edited

Loading

blakesmith commented Apr 14, 2024 •

edited

Loading