Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DNM: Debugging simulation failure #3746

Closed
wants to merge 23 commits into from
Closed

Conversation

cwgoes
Copy link
Contributor

@cwgoes cwgoes commented Feb 26, 2019

Do not merge.

Ref #3735

@cwgoes
Copy link
Contributor Author

cwgoes commented Feb 26, 2019

The last operation with:

go test ./cmd/gaia/app -run TestFullGaiaSimulation -SimulationEnabled=true -SimulationNumBlocks=400 -SimulationVerbose=false -SimulationCommit=true -SimulationSeed=30 -v -timeout 24h

is

BeginBlock

resulting in

outstanding: 454560506.432459005262762026stake, rewards: 482536446.922005148438991902stake
Panic with err
 negative coin amount
goroutine 10 [running]:
runtime/debug.Stack(0xc004aaa490, 0x2, 0x2)
	/usr/lib/go/src/runtime/debug/stack.go:24 +0xa7
github.com/cosmos/cosmos-sdk/x/mock/simulation.SimulateFromSeed.func2(0xc000942020, 0xc0001ede00, 0xc000f666d0)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/x/mock/simulation/simulate.go:131 +0xb6
panic(0xc74780, 0xfd99e0)
	/usr/lib/go/src/runtime/panic.go:513 +0x1b9
github.com/cosmos/cosmos-sdk/types.DecCoins.Sub(0xc0012b18c0, 0x1, 0x1, 0xc0012b1580, 0x1, 0x1, 0x0, 0x0, 0xc000b05280)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/types/dec_coin.go:268 +0xbe
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.withdrawDelegationRewards(0xfe1180, 0xc000f66340, 0xc000b06230, 0xc000b06230, 0xfe1180, 0xc000f66390, 0xfe11c0, 0xc000f663a0, 0xc000b05280, 0x5, ...)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/x/distribution/keeper/delegation.go:106 +0x6f8
github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.WithdrawDelegationRewards(0xfe1180, 0xc000f66340, 0xc000b06230, 0xc000b06230, 0xfe1180, 0xc000f66390, 0xfe11c0, 0xc000f663a0, 0xc000b05280, 0x5, ...)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/x/distribution/keeper/keeper.go:62 +0x20c
github.com/cosmos/cosmos-sdk/x/distribution/simulation.CanWithdrawInvariant.func1(0xfe7f60, 0xc004704390, 0xc005260940, 0xc, 0xfe1180, 0xc000f66390)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/x/distribution/simulation/invariants.go:57 +0x329
github.com/cosmos/cosmos-sdk/x/distribution/simulation.AllInvariants.func1(0xfe7f60, 0xc004704390, 0xc005260940, 0xc, 0x36, 0x0)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/x/distribution/simulation/invariants.go:14 +0xbf
github.com/cosmos/cosmos-sdk/x/mock/simulation.PeriodicInvariant.func1(0xfe7f60, 0xc004704390, 0xc005260940, 0xc, 0x0, 0x0)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/x/mock/simulation/util.go:135 +0xc8
github.com/cosmos/cosmos-sdk/x/mock/simulation.assertAllInvariants(0xc000109700, 0xc000109a00, 0xc000f788a0, 0x5, 0x5, 0xdb7faf, 0xa, 0xc000942020)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/x/mock/simulation/util.go:23 +0x15c
github.com/cosmos/cosmos-sdk/x/mock/simulation.SimulateFromSeed(0xfefe20, 0xc000109700, 0xc000109a00, 0xf13910, 0x1e, 0xc0000fe2a0, 0xe, 0xe, 0xc000f788a0, 0x5, ...)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/x/mock/simulation/simulate.go:156 +0x19de
github.com/cosmos/cosmos-sdk/cmd/gaia/app.TestFullGaiaSimulation(0xc000109700)
	/home/cwgoes/working/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/sim_test.go:359 +0x339
testing.tRunner(0xc000109700, 0xf138a0)
	/usr/lib/go/src/testing/testing.go:827 +0xbf
created by testing.(*T).Run
	/usr/lib/go/src/testing/testing.go:878 +0x35c

@cwgoes
Copy link
Contributor Author

cwgoes commented Feb 26, 2019

BeginBlock allocation:

Simulating... block 53/400, operation 200/245. allocating tokens: sumPrecommitPower 2425321, totalPower 3437104, previousProposer FEBDC6079DE334E695E3673F3391C18AC6E4E31A, votes [{{[107 109 87 172 245 246 64 66 112 217 111 242 215 216 81 239 217 106 201 140] 593263 {} [] 0} true {} [] 0} {{[51 53 198 104 78 148 62 99 166 114 226 114 57 213 213 243 189 198 136 161] 593263 {} [] 0} true {} [] 0} {{[67 156 141 101 153 123 176 233 186 71 233 134 151 110 191 4 255 255 85 98] 140574 {} [] 0} true {} [] 0} {{[57 178 135 219 182 202 201 227 198 148 87 150 23 11 149 23 241 126 39 25] 1011783 {} [] 0} false {} [] 0} {{[32 136 51 194 44 138 21 169 0 70 89 189 24 13 233 206 138 113 138 217] 1098221 {} [] 0} true {} [] 0}]

@cwgoes
Copy link
Contributor Author

cwgoes commented Feb 26, 2019

Simulating... block 53/400, operation 200/245. allocating tokens: sumPrecommitPower 2425321, totalPower 3437104, previousProposer FEBDC6079DE334E695E3673F3391C18AC6E4E31A, votes [{{[107 109 87 172 245 246 64 66 112 217 111 242 215 216 81 239 217 106 201 140] 593263 {} [] 0} true {} [] 0} {{[51 53 198 104 78 148 62 99 166 114 226 114 57 213 213 243 189 198 136 161] 593263 {} [] 0} true {} [] 0} {{[67 156 141 101 153 123 176 233 186 71 233 134 151 110 191 4 255 255 85 98] 140574 {} [] 0} true {} [] 0} {{[57 178 135 219 182 202 201 227 198 148 87 150 23 11 149 23 241 126 39 25] 1011783 {} [] 0} false {} [] 0} {{[32 136 51 194 44 138 21 169 0 70 89 189 24 13 233 206 138 113 138 217] 1098221 {} [] 0} true {} [] 0}]
allocate to validator: val Validator
  Operator Address:           cosmosvaloper1l67uvpuauv6wd90rvuln8ywp3trwfcc6csx0hn
  Validator Consensus Pubkey: cosmosvalconspub1zcjduepqt2gf45pv33fls7c4n5ukzugj3r68z8jqpzhha0rd0qfepnetyxgs3ltmn7
  Jailed:                     true
  Status:                     Unbonding
  Tokens:                     978435872116
  Delegator Shares:           1074040715314.427507160608908083
  Description:                {mtKhGaArLP LZpHzksjbf iStyxpeFQq bZAnuFTQvl}
  Unbonding Height:           52
  Unbonding Completion Time:  3130-05-17 08:19:00 +0000 UTC
  Minimum Self Delegation:    1
  Commission:                 rate: 0.000000000000000000, maxRate: 0.000000000000000000, maxChangeRate: 0.000000000000000000, updateTime: 3130-05-13 20:58:16 +0000 UTC, tokens 1926535755.026056324633364950stake
allocate to validator: val Validator
  Operator Address:           cosmosvaloper1ddk40t847eqyyuxedled0kz3alvk4jvvjc34yd
  Validator Consensus Pubkey: cosmosvalconspub1zcjduepqnv44xpew6zfe0t7my0l2aqpqdjkfz745cnd7aq57gm9xxeyteupsrcuuud
  Jailed:                     false
  Status:                     Bonded
  Tokens:                     821305424644
  Delegator Shares:           821305424537.767010354351433199
  Description:                {uYzBQdMEbJ bxiDGgqJDk ABxLWIJsVX MfqYsmTBTm}
  Unbonding Height:           0
  Unbonding Completion Time:  1970-01-01 00:00:00 +0000 UTC
  Minimum Self Delegation:    1
  Commission:                 rate: 0.000000000000000000, maxRate: 0.000000000000000000, maxChangeRate: 0.000000000000000000, updateTime: 3130-05-14 14:22:30 +0000 UTC, tokens 488783546.089502163102858834stake
allocate to validator: val Validator
  Operator Address:           cosmosvaloper1xv6uv6zwjslx8fnjufern4w47w7udz9pz8cyp9
  Validator Consensus Pubkey: cosmosvalconspub1zcjduepq468fng9n8pcumxxfft0vh2puqvwqflwu77mfurfm78e8nxe4wtkqu2zkhr
  Jailed:                     false
  Status:                     Bonded
  Tokens:                     1056209516788
  Delegator Shares:           1056209516758.744243553817861383
  Description:                {tShhiQMqyn ZgVBXsWlGS LlbEhCaHVL uhdGZIBcqh}
  Unbonding Height:           0
  Unbonding Completion Time:  1970-01-01 00:00:00 +0000 UTC
  Minimum Self Delegation:    1
  Commission:                 rate: 0.000000000000000000, maxRate: 0.000000000000000000, maxChangeRate: 0.000000000000000000, updateTime: 3130-05-14 18:14:53 +0000 UTC, tokens 488783546.089502163102858834stake
allocate to validator: val Validator
  Operator Address:           cosmosvaloper1gwwg6eve0wcwnwj8axrfwm4lqnll74tz43my0d
  Validator Consensus Pubkey: cosmosvalconspub1zcjduepqhpcdx9t2u7sznwy6yugsss5he2wq6kd2hfwhhtpqu2yd2l00jlcqy8etds
  Jailed:                     false
  Status:                     Bonded
  Tokens:                     592640828671
  Delegator Shares:           592640828616.566285150862470016
  Description:                {kAOmtGclpZ CcCGCVaOsg zSduqvUcHu FfwzyJHfzm}
  Unbonding Height:           3
  Unbonding Completion Time:  3130-05-13 03:28:56 +0000 UTC
  Minimum Self Delegation:    1
  Commission:                 rate: 0.000000000000000000, maxRate: 0.000000000000000000, maxChangeRate: 0.000000000000000000, updateTime: 3130-05-14 22:31:19 +0000 UTC, tokens 115817534.901023113859504340stake
allocate to validator: val Validator
  Operator Address:           cosmosvaloper18xeg0kakety7835527tpwzu4zlchufcegv768l
  Validator Consensus Pubkey: cosmosvalconspub1addwnpepqgqjc4d984ure9lsyp5q0szazdmgsll2akykeefkevz9uhngpue7ut0v3lj
  Jailed:                     false
  Status:                     Bonded
  Tokens:                     694555613172
  Delegator Shares:           694555613046.265691868696022074
  Description:                {CrlSveUOEW VqnfMzYnJU SzDmOxgzQC cpXGhbdbDG}
  Unbonding Height:           42
  Unbonding Completion Time:  3130-05-16 12:14:59 +0000 UTC
  Minimum Self Delegation:    1
  Commission:                 rate: 0.000000000000000000, maxRate: 0.000000000000000000, maxChangeRate: 0.000000000000000000, updateTime: 3130-05-14 20:18:16 +0000 UTC, tokens 833598054.510520236781843100stake
allocate to validator: val Validator
  Operator Address:           cosmosvaloper1yzyr8s3v3g26jqzxtx73sr0fe698rzke8wt0c2
  Validator Consensus Pubkey: cosmosvalconspub1addwnpepqvhaxt4d49zppf0k73823wda99c5hc82uduksnwpm9qj8vwx2p4lguanjac
  Jailed:                     false
  Status:                     Bonded
  Tokens:                     1158606569179
  Delegator Shares:           1158606569157.987408001662616391
  Description:                {rXmQEcTtDS LDKHQFwCot yMlKmIsUvl IhWyKGwJSj}
  Unbonding Height:           26
  Unbonding Completion Time:  3130-05-15 03:15:57 +0000 UTC
  Minimum Self Delegation:    1
  Commission:                 rate: 0.000000000000000000, maxRate: 0.000000000000000000, maxChangeRate: 0.000000000000000000, updateTime: 3130-05-14 16:04:06 +0000 UTC, tokens 904813471.883395990024181480stake
remaining after: 250438521.500000008495388462stake

@cwgoes
Copy link
Contributor Author

cwgoes commented Feb 26, 2019

It is the withdrawal from the just-tombstoned validator that panics:

remaining after: 250438521.500000008495388462stake
withdraw from Validator
  Operator Address:           cosmosvaloper1l67uvpuauv6wd90rvuln8ywp3trwfcc6csx0hn
  Validator Consensus Pubkey: cosmosvalconspub1zcjduepqt2gf45pv33fls7c4n5ukzugj3r68z8jqpzhha0rd0qfepnetyxgs3ltmn7
  Jailed:                     true
  Status:                     Unbonding
  Tokens:                     978435872116
  Delegator Shares:           1074040715314.427507160608908083
  Description:                {mtKhGaArLP LZpHzksjbf iStyxpeFQq bZAnuFTQvl}
  Unbonding Height:           52
  Unbonding Completion Time:  3130-05-17 08:19:00 +0000 UTC
  Minimum Self Delegation:    1
  Commission:                 rate: 0.000000000000000000, maxRate: 0.000000000000000000, maxChangeRate: 0.000000000000000000, updateTime: 3130-05-13 20:58:16 +0000 UTC to Delegation:
  Delegator: cosmos1l67uvpuauv6wd90rvuln8ywp3trwfcc6ayj6mq
  Validator: cosmosvaloper1l67uvpuauv6wd90rvuln8ywp3trwfcc6csx0hn
  Shares:    269007827526.276427191360113449
Panic with err
 negative coin amount

@codecov
Copy link

codecov bot commented Feb 26, 2019

Codecov Report

Merging #3746 into develop will decrease coverage by 0.12%.
The diff coverage is 41.66%.

@@             Coverage Diff             @@
##           develop    #3746      +/-   ##
===========================================
- Coverage    61.15%   61.02%   -0.13%     
===========================================
  Files          189      190       +1     
  Lines        14052    14093      +41     
===========================================
+ Hits          8593     8600       +7     
- Misses        4921     4937      +16     
- Partials       538      556      +18

@cwgoes
Copy link
Contributor Author

cwgoes commented Feb 26, 2019

remaining after: 250438521.500000008495388462stake
at block height 54 found slash event: height 52, endingPeriod 184, rewards before: 106804023.007942852200393955stake
startingPeriod: 177
endingPeriod: 198
withdraw from Validator
  Operator Address:           cosmosvaloper1l67uvpuauv6wd90rvuln8ywp3trwfcc6csx0hn
  Validator Consensus Pubkey: cosmosvalconspub1zcjduepqt2gf45pv33fls7c4n5ukzugj3r68z8jqpzhha0rd0qfepnetyxgs3ltmn7
  Jailed:                     true
  Status:                     Unbonding
  Tokens:                     978435872116
  Delegator Shares:           1074040715314.427507160608908083
  Description:                {mtKhGaArLP LZpHzksjbf iStyxpeFQq bZAnuFTQvl}
  Unbonding Height:           52
  Unbonding Completion Time:  3130-05-17 08:19:00 +0000 UTC
  Minimum Self Delegation:    1
  Commission:                 rate: 0.000000000000000000, maxRate: 0.000000000000000000, maxChangeRate: 0.000000000000000000, updateTime: 3130-05-13 20:58:16 +0000 UTC to Delegation:
  Delegator: cosmos1l67uvpuauv6wd90rvuln8ywp3trwfcc6ayj6mq
  Validator: cosmosvaloper1l67uvpuauv6wd90rvuln8ywp3trwfcc6csx0hn
  Shares:    269007827526.276427191360113449
Panic with err
 negative coin amount
goroutine 8 [running]:

@cwgoes
Copy link
Contributor Author

cwgoes commented Feb 27, 2019

Closing in favor of #3759.

@cwgoes cwgoes closed this Feb 27, 2019
@cwgoes cwgoes deleted the cwgoes/back-to-truncation branch February 27, 2019 21:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants