Coinjoin chart #1640

ademuanthony · 2019-12-11T21:31:33Z

This PR is a collaboration between me and @raedah

We added a chart for coinjoin that shows the mixed amount by day block

We added a chart for the value of Anonymity Set to the Coin Supply chart

yarn.lock

chappjc · 2019-12-12T14:05:17Z

Shouldn't we replace "Coinjoin" with "CoinShuffle++" or something generic like "mixed coins" or "privacy" on these pages? Coinjoin really under sells where Decred has implemented (cspp).

raedah · 2019-12-16T16:39:57Z

CSPP is only part of the server client protocol and does not appear on chain. Coinjoin is the name of the transaction type that is being identified. Mixed coins or privacy are accurate but are more generic, less specific. Perhaps more generic is better since what is being identified is actually several specific transaction cases such as the connected sstx transactions.

chappjc · 2019-12-16T16:43:00Z

I see. From a marketing perspective, how about "anonymity set"?

chappjc · 2019-12-16T17:23:25Z

Actually, Privacy Participation would jive well with the other charts (e.g. Stake Participation). How about that?

buck54321

Partial review. Overall, this is pretty solid. charts_controller adds a lot of code to force together two data sets that I'm not convinced need to be plotted together, but hopefully you can set me straight. The javascript there can be simplified quite a bit, but I don't want to put too much energy into it until I'm sure that there's a good reason.

db/cache/charts.go

db/dcrpg/pgblockchain.go

buck54321 · 2019-12-18T14:09:42Z

public/js/controllers/charts_controller.js

@@ -154,7 +165,7 @@ function zipTvY (times, ys, yMult) {

 function zipIvY (ys, yMult, offset) {
  yMult = yMult || 1
-  offset = offset || 1
+  offset = offset || 1 // TODO: check for why offset is set to a default value of 1 when genesis block has a height of 0


db/dcrpg/upgrades.go

ademuanthony · 2020-01-22T18:10:03Z

Oh the 1.7.1 maint is because of this?

But I think this change will create far too many false positives. It also allows outputs with different values in the same tx to all be recognized as ticket funding outputs, but they would have to be the same value to be a mix.

I have rolled back these changes for now.
It is now noted and we can look into it later and provide a better solution

chappjc · 2020-01-31T13:58:18Z

Please have a look at the test failure:

 --- FAIL: TestChartsCache (0.00s)
##[error]    charts_test.go:53: DeepEqual: expected true, found false for Height after read
    --- FAIL: TestChartsCache/Read_from_an_existing_gob_encoded_file (0.00s)
##[error]        testing.go:864: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test

db/dcrpg/internal/txstmts.go

db/dcrpg/queries.go

db/dcrpg/pgblockchain.go

chappjc · 2020-01-31T15:03:20Z

db/dcrpg/queries.go

+		if totalMixed < 0 {
+			totalMixed *= -1
+		}


In what circumstances does this (totalMixed < 0) happen? Surely that indicates a bug in something else.

Please comment on this case.

I no longer have this issue, this was based on different implementation where my effort to remove spent vouts was leading the negative value. I ought to have removed it when the implementation was changed.

db/dcrpg/queries.go

chappjc · 2020-02-03T14:29:54Z

Thanks for refactoring to use Length. Just some trivial test failures related to code formatting (https://github.com/decred/dcrdata/runs/423024195#step:9:102). I'll do a final review and hopefully we can get this up on alpha today or tomorrow.

chappjc · 2020-02-03T21:13:14Z

For follow-up work, we should investigate triggering a fresh charts data update as part of the upgrade process rather than the usual charts data update on startup. The reason is that on startup I see"

[INF] DATD: Mainchain sync complete.
[INF] DATD: All ready, at height 420483.
[WRN] PSQL: charts.ValidateLengths: dataset at index 9 has mismatched length 0 != 420484
[WRN] PSQL: charts.ValidateLengths: dataset at index 10 has mismatched length 0 != 420484
[WRN] PSQL: ChartData.Lengthen: block data length mismatch detected. Truncating blocks length to 0
[INF] PSQL: Updating charts data...

It doesn't say what it is doing, but a look at top indicates it is dcrdata not postgres, which suggests it is one of the functions that operates on the retrieved data, probably appendPrivacyParticipation or maybe appendAnonymitySet.

chappjc · 2020-02-03T21:14:30Z

After about 5 minutes:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xc727a8]

goroutine 1 [running]:
github.com/decred/dcrdata/db/dcrpg/v5.appendAnonymitySet(0xc0075f41e0, 0xc00c11e100, 0xc030b6a2b0, 0x0)
	/home/jon/github/decred/dcrdata/db/dcrpg/queries.go:3726 +0x618
github.com/decred/dcrdata/db/cache/v3.(*ChartData).Update(0xc0075f41e0, 0x0, 0x0)
	/home/jon/github/decred/dcrdata/db/cache/charts.go:873 +0x3c1
github.com/decred/dcrdata/db/cache/v3.(*ChartData).Load(0xc0075f41e0, 0xc01f3a63c0, 0x2f, 0x0, 0x0)
	/home/jon/github/decred/dcrdata/db/cache/charts.go:728 +0x1a0
main._main(0x133cc60, 0xc0002a6180, 0x0, 0x0)
	/home/jon/github/decred/dcrdata/main.go:992 +0x3797
main.main()
	/home/jon/github/decred/dcrdata/main.go:58 +0x81

Looks like it got through privacy participation and crashed on anonymity set.

chappjc · 2020-02-03T21:27:23Z

Note that to reproduce that panic, you will have to run with master to get a chartscache.gob the has all the charts data except anon set and priv partic.

db/cache/charts.go

db/dcrpg/queries.go

include fee in ticket price in IsMixedSplitTx steal txsizes from dcrwallet/internal/txsizes return mixDenom from IsMixedSplitTx txWithTicketPrice: really ugly way to get txraw and price for same best block

Add TotalMixed to explorer/types.BlockInfo. Add MixCount and MixDenom to explorer/types.TxBasic, and remove them from explorer/types.TxInfo.

Added a window bin for a moving 28 days sum of the total mixed. Added the 28 days moving sum line to the circulation chart. Added percentage of mixed total to the legend. Persist chart visibility in url. 29.07 days moving sum. {rewardPeriod} total mixed moving sum. Fetched ticket mixed total at 1% tolerance. Filled empty spaces with 0s to maintain the length of the record set Add moving sum graph with a checkbox to the total mixed chart. Removed total mixed moving sum chart from block bin on circulation chart. Used Mixed for label instead of Total Mixed. Hide log scale for coinjoins. Added day range accumulator for chart data and used it for the reward period.

while giving the user the ability to zoom by all other options

Added support for block bin on the chart

to resolve the issue of spent tx showing in anonymity set chart

earlier testnet transactions paid higher fees, checking a range to accommodate those

set to chart.go

chappjc · 2020-02-04T23:21:01Z

Running well now. Fresh updating charts data is down to 20 seconds from >5 minutes. This is still several times longer than before, so we should look to optimizing this in future work.

Last issue, code formatting again: https://github.com/decred/dcrdata/runs/425480283#step:9:97

chappjc · 2020-02-04T23:31:19Z

public/js/controllers/charts_controller.js

    this.query.update(this.settings)
    this.settings.chart = this.settings.chart || 'ticket-price'
    this.zoomCallback = this._zoomCallback.bind(this)
    this.drawCallback = this._drawCallback.bind(this)
    this.limits = null
    this.lastZoom = null
+    this.visibility = []
+    if (this.settings.visibility) {
+      this.settings.visibility.split(',', -1).forEach(s => {


Actually, I do not care for how the URL ends up with %2C in it, but I don't have a better suggestion for this. Since the charts and controls are working, I don't want to delay this PR any longer. For a future issues. @buck54321 might also have a suggestion for this.

The visibility here is an array.
Some chart has multiple lines and checkboxes for toggling the visibility of any of the lines. The encoded value of the array is what lead to the %2C from visibility = [true, false].
During the iterations of the PR, I had a recommendation to preserve the visibility across page reload

I understand the reason, and I agree tha tit's good to preserve the settings in the URL query parameters, I'm just pointing out that in the URL, the query params are ugly as a result:

We can resolve this in the future though, but it will have to be very near future and definitely before release otherwise we will get broken urls when we change the scheme.

Use running sum to hold block anonymity sum and a map to track spending heights and remove value from the running some. Take daily average for day

chappjc · 2020-02-05T14:42:27Z

Letting it run for a while, I get warnings about dataset at index 10 when the charts do their occasional update:

[WRN] PSQL: charts.ValidateLengths: dataset at index 10 has mismatched length 420836 != 420837
[WRN] PSQL: ChartData.Lengthen: block data length mismatch detected. Truncating blocks length to 420836

Restarting dcrdata seems to repair things, but it looks like there will be a problem while running for long periods.

chappjc · 2020-02-05T15:22:51Z

db/dcrpg/internal/txstmts.go

+	SelectMixedVouts = `
+		SELECT vouts.value, fund_tx.block_height, spend_tx.block_height, vouts.tx_tree
+		FROM vouts
+		JOIN transactions AS fund_tx ON vouts.tx_hash=fund_tx.tx_hash
+		LEFT OUTER JOIN transactions AS spend_tx ON spend_tx_row_id=spend_tx.id
+		WHERE mixed=true and value>0 ORDER BY fund_tx.block_height;`


I thought you had a version of this query with a block height constraint like with SelectMixedTotalPerBlock. Related to your comment at https://github.com/decred/dcrdata/pull/1640/files#diff-87f7a577b6af45d0284e4c3fbb98ddf5R3685

Will you follow up on this in a separate PR? I want to merge this, but this change must be done soon as this is an expensive query that the server can't afford to have running often.

I will follow up in a separate PR

Excellent, thanks. And thanks for your persistence and patience in this PR.

chappjc · 2020-02-05T20:42:19Z

Letting it run for a while, I get warnings about dataset at index 10 when the charts do their occasional update:
[WRN] PSQL: charts.ValidateLengths: dataset at index 10 has mismatched length 420836 != 420837
[WRN] PSQL: ChartData.Lengthen: block data length mismatch detected. Truncating blocks length to 420836
Restarting dcrdata seems to repair things, but it looks like there will be a problem while running for long periods.

This does seem to be a consistent problem. Off by one in this dataset?

[WRN] PSQL: charts.ValidateLengths: dataset at index 10 has mismatched length 421007 != 421008
[WRN] PSQL: ChartData.Lengthen: block data length mismatch detected. Truncating blocks length to 421007

[WRN] PSQL: charts.ValidateLengths: dataset at index 10 has mismatched length 421059 != 421060
[WRN] PSQL: ChartData.Lengthen: block data length mismatch detected. Truncating blocks length to 421059

Filled anonymity set until it is the same length with block height and the max funding height is filled

chappjc reviewed Dec 11, 2019

View reviewed changes

yarn.lock Outdated Show resolved Hide resolved

buck54321 requested changes Dec 18, 2019

View reviewed changes

ademuanthony force-pushed the coinjoin-chart branch 2 times, most recently from 33a6809 to cb3f1c6 Compare December 23, 2019 05:32

ademuanthony force-pushed the coinjoin-chart branch from 5b7b3e6 to 4a0df68 Compare January 3, 2020 18:00

ademuanthony force-pushed the coinjoin-chart branch 2 times, most recently from e63bab0 to 43de7d5 Compare January 16, 2020 03:11

chappjc reviewed Jan 17, 2020

View reviewed changes

db/dcrpg/upgrades.go Outdated Show resolved Hide resolved

ademuanthony changed the title ~~[WIP] Coinjoin chart~~ Coinjoin chart Jan 22, 2020

ademuanthony force-pushed the coinjoin-chart branch 2 times, most recently from 36843ca to 4e13df5 Compare January 27, 2020 18:28

chappjc requested changes Jan 31, 2020

View reviewed changes

chappjc reviewed Feb 3, 2020

View reviewed changes

db/cache/charts.go Outdated Show resolved Hide resolved

chappjc reviewed Feb 3, 2020

View reviewed changes

db/dcrpg/queries.go Outdated Show resolved Hide resolved

chappjc and others added 6 commits February 4, 2020 14:55

views: tag mix tx on tx page

58a53e1

include fee in ticket price in IsMixedSplitTx steal txsizes from dcrwallet/internal/txsizes return mixDenom from IsMixedSplitTx txWithTicketPrice: really ugly way to get txraw and price for same best block

views: mix data on block page

d602c65

Add TotalMixed to explorer/types.BlockInfo. Add MixCount and MixDenom to explorer/types.TxBasic, and remove them from explorer/types.TxInfo.

Zoom coinjoins chart to the area with record for the 'all' option

36d96ee

while giving the user the ability to zoom by all other options

Track mixed amount staked and regular transaction in charts

70ba521

rm yarn.lock

f3deaf3

ademuanthony added 15 commits February 4, 2020 15:02

Used CoinShuffle++ as name for coinjoin chart

01eaa37

Anonymity set

a084f3b

Removed spent output from the anonymity set chart data.

03197f2

Added support for block bin on the chart

clean up

c07ee0f

Privacy participation

f65d2e1

show thousand separator on y label and remove min height from utox query

3e6a6be

chart updater for privacy participation

e423155

Remove anonymity set cache

04cef68

to resolve the issue of spent tx showing in anonymity set chart

Added range to split mixed tx computation

b6deeaa

earlier testnet transactions paid higher fees, checking a range to accommodate those

Added maintenance db upgrade to recompute mixed tx

901ad3f

Removed DB upgrade and fee range comparison in cspp

0b12132

Print 0s on the legend of privacy set chat instead of empty string

887ab97

Code refactoring. Moved day chart data calculation for anonymity

cb17bb3

set to chart.go

Added test for anonymity set

c6eaf4f

Code formatting

7d6037c

ademuanthony force-pushed the coinjoin-chart branch 2 times, most recently from 781ea38 to e3e2d6e Compare February 4, 2020 14:14

chappjc reviewed Feb 4, 2020

View reviewed changes

Refactored anonymity computation

8883e0c

Use running sum to hold block anonymity sum and a map to track spending heights and remove value from the running some. Take daily average for day

ademuanthony force-pushed the coinjoin-chart branch from e3e2d6e to 8883e0c Compare February 5, 2020 12:20

chappjc reviewed Feb 5, 2020

View reviewed changes

Filled missing gap in anonymity set chart data

ca6bdea

Filled anonymity set until it is the same length with block height and the max funding height is filled

chappjc merged commit 9ecad0c into decred:master Feb 7, 2020

chappjc mentioned this pull request Feb 10, 2020

mix chart queries need optimization #1689

Closed

chappjc mentioned this pull request Apr 8, 2020

coinjoin tx chart #1543

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Coinjoin chart #1640

Coinjoin chart #1640

ademuanthony commented Dec 11, 2019 •

edited

Loading

chappjc commented Dec 12, 2019

raedah commented Dec 16, 2019

chappjc commented Dec 16, 2019

chappjc commented Dec 16, 2019

buck54321 left a comment

buck54321 Dec 18, 2019 •

edited

Loading

ademuanthony commented Jan 22, 2020

chappjc commented Jan 31, 2020

chappjc Jan 31, 2020 •

edited

Loading

chappjc Feb 4, 2020

ademuanthony Feb 5, 2020

chappjc commented Feb 3, 2020

chappjc commented Feb 3, 2020 •

edited

Loading

chappjc commented Feb 3, 2020

chappjc commented Feb 3, 2020

chappjc commented Feb 4, 2020

chappjc Feb 4, 2020

ademuanthony Feb 5, 2020

chappjc Feb 5, 2020 •

edited

Loading

chappjc commented Feb 5, 2020

chappjc Feb 5, 2020 •

edited

Loading

ademuanthony Feb 5, 2020

chappjc Feb 5, 2020

chappjc commented Feb 5, 2020

Coinjoin chart #1640

Coinjoin chart #1640

Conversation

ademuanthony commented Dec 11, 2019 • edited Loading

chappjc commented Dec 12, 2019

raedah commented Dec 16, 2019

chappjc commented Dec 16, 2019

chappjc commented Dec 16, 2019

buck54321 left a comment

Choose a reason for hiding this comment

buck54321 Dec 18, 2019 • edited Loading

Choose a reason for hiding this comment

ademuanthony commented Jan 22, 2020

chappjc commented Jan 31, 2020

chappjc Jan 31, 2020 • edited Loading

Choose a reason for hiding this comment

chappjc Feb 4, 2020

Choose a reason for hiding this comment

ademuanthony Feb 5, 2020

Choose a reason for hiding this comment

chappjc commented Feb 3, 2020

chappjc commented Feb 3, 2020 • edited Loading

chappjc commented Feb 3, 2020

chappjc commented Feb 3, 2020

chappjc commented Feb 4, 2020

chappjc Feb 4, 2020

Choose a reason for hiding this comment

ademuanthony Feb 5, 2020

Choose a reason for hiding this comment

chappjc Feb 5, 2020 • edited Loading

Choose a reason for hiding this comment

chappjc commented Feb 5, 2020

chappjc Feb 5, 2020 • edited Loading

Choose a reason for hiding this comment

ademuanthony Feb 5, 2020

Choose a reason for hiding this comment

chappjc Feb 5, 2020

Choose a reason for hiding this comment

chappjc commented Feb 5, 2020

ademuanthony commented Dec 11, 2019 •

edited

Loading

buck54321 Dec 18, 2019 •

edited

Loading

chappjc Jan 31, 2020 •

edited

Loading

chappjc commented Feb 3, 2020 •

edited

Loading

chappjc Feb 5, 2020 •

edited

Loading

chappjc Feb 5, 2020 •

edited

Loading