add staking #7

buck54321 · 2019-09-17T21:02:32Z

Adds widespread but not-quite-complete support for all script types other that alt-sig types. Implementation is a mirror of parts of dcrd's txscript and dcrutil modules.

Adds a stake pool client (StakePool class) with methods for getting the purchase info, getting the stats, setting the voting address, and setting the vote bits.

New http module in util used to be the networking portion of dcrdata module, but is now shared among some others.

Adds ticket purchase support (VSP only). Also moves pydecred tests to a dedicated test file to clean up packages for import. This should be done in the crypto package too, but I'm leaving that for another PR.

Updated GUI interface to support stake pool handling and ticket purchasing.

The main goal of this effort was to reproduce dcrwallet's (*Wallet).purchaseTickets method. The mirrored method is implemented on the DcrdataBlockchain, with keys and internal addresses coming from a closure passed from an unlocked wallet.

Notably, no way to revoke a missed or expired ticket yet. Will follow up for that in another PR. This one is already way too big.

PR is still a draft for now while I continue testing and tweaking, but it's definitely ready for scrutiny. I'll add examples before removing from draft status, too.

buck54321 · 2019-11-13T22:19:05Z

accounts.py

        }
    @staticmethod
-    def __fromjson__(obj):
-        acct = Account(
+    def __fromjson__(obj, cls=None):


Passing the constructor like this is a little bit of a hack. Once I have more than one account, I'll put some more thought into this.

buck54321 · 2019-11-13T22:24:17Z

crypto/crypto.py

-    #         depth = obj["depth"],
-    #         childNum = obj["childNum"],
-    #         isPrivate = obj["isPrivate"],
-    #     )


ExtendedKey should never be saved or JSON-encoded. The key is serialized and encrypted before saving.

buck54321 · 2019-11-13T22:39:33Z

pydecred/account.py

+
+The DecredAccount inherits from the tinydecred base Account and adds staking 
+support.
+"""


In preparation for multi-asset, I've separated account into a base class and an inheriting Decred-specific class.

buck54321 · 2019-11-13T22:41:34Z

pydecred/account.py

+# In addition to the standard internal and external branches, we'll have a third
+# branch. This should help accomodate upcoming changes to dcrstakepool. See also
+# https://github.com/decred/dcrstakepool/pull/514
+STAKE_BRANCH = 2


I don't know if there is a best way to handle this. I've gone off-BIP a little bit to create a staking branch from which all ticket-related keys will be generated. Right now, only the zeroth key is used, but more will be used when accountless VSP is implemented.

I believe this was the method discussed. AFAIK still waiting on @jrick's opinion.

buck54321 · 2019-11-13T22:45:20Z

pydecred/account.py

+        addrs = self.unseenAddrs()
+        while addrs:
+            for addr in addrs:
+                for txid in blockchain.txsForAddr(addr):


This can potentially take a long time since the requests must be made one address at-a-time. Might consider breaking the sync into two sections, one quick but critical section and one deep sync section that can run in a background thread afterwards.

buck54321 · 2019-11-13T23:05:27Z

pydecred/dcrdata.py

                    scriptPubKey = changeScript,
                    amount = changeAmount*1e-8,
                    satoshis = changeAmount,
                ))

            return newTx, utxos, newUTXOs

-class TestDcrdata(unittest.TestCase):


All testing classes should be moved to separate modules.

buck54321 · 2019-11-13T23:10:02Z

ui/qutilities.py

    """
    w, lyt = makeWidget(QtWidgets.QWidget, HORIZONTAL)
    lyt.addWidget(wgt)
    w.setContentsMargins(l, t, r, b)
    return w

-
-class QSimpleTable(QtWidgets.QTableWidget):


Unused and cumbersome.

buck54321 · 2019-11-13T23:11:44Z

ui/screens.py

@@ -137,7 +145,7 @@ def showEvent(self, e):
    def closeEvent(self, e):
        self.hide()
        e.ignore()
-    def stack(self, w):
+    def stack_(self, w):


Trying to connect any common UI-modifying functions as Qt slots to simplify threaded operations.

buck54321 · 2019-11-13T23:17:54Z

ui/screens.py

+        is based purely on voting record, e.g. voted/(voted+missed). The sorting
+        and some initial filtering was already performed in setPools.
+        """
+        count = len(self.pools)//2+1


This is a simple but pretty dumb way to filter low-performing pools. Consider other options.

buck54321 · 2019-11-13T23:22:01Z

util/tinyjson.py

@@ -11,7 +11,7 @@
 def clsKey(cls):
    return cls.__qualname__

-def register(cls):
+def register(cls, tag=None):


This enables setting the __jsontag__ manually, instead of using the built in name generating algorithm. Needed if registering identically-named classes.

JoeGruffins · 2019-11-14T23:06:05Z

It keeps growing.

JoeGruffins

BIG.

So, this is a fist pass. It looks like accounts moved after I had left some comments, at least one was already addressed, but ignore any that are no longer pertinent...

When trying to run app.py now, it says my password is incorrect.

log

2019-11-15 15:30:10,441 app INFO <module>(523) configuration file at /home/joe/.local/share/TinyDecred/tinywallet.conf
2019-11-15 15:30:10,441 app INFO <module>(524) data directory at /home/joe/.local/share/TinyDecred
  File "app.py", line 185, in openWallet
    w = Wallet.openFile(path, pw)
  File "/home/joe/.local/lib/python3.8/site-packages/tinydecred/wallet/wallet.py", line 178, in openFile
    wallet = tinyjson.load(fileKey.decrypt(bytes.fromhex(wrapper["wallet"])).decode())
  File "/home/joe/.local/lib/python3.8/site-packages/tinydecred/util/tinyjson.py", line 37, in load
    return json.loads(s, object_hook=decoder)
  File "/usr/lib/python3.8/json/__init__.py", line 370, in loads
    return cls(**kw).decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.8/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
  File "/home/joe/.local/lib/python3.8/site-packages/tinydecred/util/tinyjson.py", line 30, in decoder
    return _types[obj["_jt_"]].__fromjson__(obj)
2019-11-15 15:30:14,836 app WARNING openWallet(191) exception encountered while attempting to open wallet: 'Account'
None

Going back to master I can proceed as normal.
I can also press the back arrow at any time and see this:

wallet

but then I cannot return to the password input screen.

I will try moving my wallet and making a new one.

~~Removing the wallet.db from the testnet folder has no affect.~~
~~After removing the wallet, I can still open it on master. Where is it?~~

That back arrow is also present in master

Found the real wallet file! Deleted and made a new one. Seems to be working alright now.
Will continue testing.

Should I be able to get to some staking screen? :P

I get to this screen, which way to staking?

wallet

JoeGruffins · 2019-11-14T01:06:02Z

accounts.py

+    def getUTXOs(self, requested, approve=None):
+        """
+        Find confirmed and mature UTXOs, smallest first, that sum to the
+        requested amount, in atoms.
+
+        Args:
+            requested (int): Required amount in atoms.
+            filter (func(UTXO) -> bool): Optional UTXO filtering function.


It looks like the second argument's name is "approve" and not "filter".

JoeGruffins · 2019-11-14T07:38:08Z

accounts.py

-        self.lastInternalIndex = idx
-        return addr
+        newAddrs = []
+        extAddrs, intAddrs = self.externalAddresses, self.internalAddresses


I'm not sure of the purpose of this. It's not a copy correct? Is it to obtain shorter names? If so you use self.internalAddresses once below.

pydecred/account.py

pydecred/txscript.py

xaur · 2019-11-15T15:17:39Z

Adds a stake pool client (StakePool class)

The rename to "VSP" was almost universal so how about using these terms in code as well?

JoeGruffins · 2019-11-16T05:01:31Z

pydecred/tests.py

+    def test_extract_script_addrs(self):
+        from tinydecred.pydecred import mainnet
+        scriptVersion = 0
+


This is really my first time looking at a python test, but it looks like you need to add:

if __name__ == '__main__': unittest.main()

To the bottom in order to test as per:
https://docs.python.org/3/library/unittest.html

After adding those lines, I got this:

test results

$ python3 pydecred/tests.py .Emsg: {'event': 'subscribeResp', 'message': {'success': True, 'request_event': 'address:Dcur2mcGjmENx4DhNqDctW5wJCVyT3Qeqkx', 'request_id': 6, 'data': 'subscribed to address:Dcur2mcGjmENx4DhNqDctW5wJCVyT3Qeqkx'}} msg: {'done': 'done'} . no stake pool credentials provided. skipping stake pool test s no stake pool credentials provided. skipping stake pool test s no stake pool credentials provided. skipping stake pool test s............... ====================================================================== ERROR: test_purchase_ticket (__main__.TestDcrdata) ---------------------------------------------------------------------- Traceback (most recent call last): File "pydecred/tests.py", line 1974, in test_purchase_ticket ticketPrice = self.stakeDiff() AttributeError: 'TestDcrdata' object has no attribute 'stakeDiff' ---------------------------------------------------------------------- Ran 21 tests in 10.503s FAILED (errors=1, skipped=3)

edit: It seems test can also be run from the command line like so, without adding to the file:

python3 -m unittest discover

The result is the same

I typically test by specifying the module, e.g.

python -m unittest crypto

Looks like you're looking into streamlining testing in #18, so I'll let you find the best solution for our needs.

stakeDiff error fixed.

buck54321 · 2019-11-18T14:07:26Z

I'll do class StakePool -> class VotingServiceProvider, but I'm gonna use stakePool many places internally because vsp is non-descriptive and votingServiceProvider is too long.

buck54321 · 2019-11-18T16:18:22Z

re wallet won't load: At this point of development, expect every major PR to break the wallet, so delete the wallet and reload with your mnemonic seed. In the future, we'll need to version the wallet and handle upgrades with more care.

JoeGruffins · 2019-11-20T05:27:37Z

Working great!
Just a few things.
One is that the "show another" button doesn't seem to do anything
Another is that % staked did not change until I restarted the wallet once, when starting from balance 0.
And just a feeling, but startup seems to take considerably longer after buying 34 tickets.
And right after purchasing a ticket, my the spent dcr is added to the totat amout before being marked as spent. This wallet only has 500.99 dcr in it, just after buying some tickets the amount shoots up for a short time (first I bought 11 tickets, this is right after buying 23 more):

wallet

You can also see that percent staked is 0 although 98% of funds have been staked. The correct percent shows after a restart.

One more very tiny thing, the vsp url doesn't seem to work if there is a trailing slash at the end, perhaps that can be removed automatically?

I just had some tickets that looked like they were bought, but apparently were not, the log:

log

2019-11-20 14:40:01,481 dcrdata ERROR broadcast(929) broadcast error: ('RequestError', 'Error encountered in requesting path https://testnet.dcrdata.org/insight/api/tx/send: HTTP Error 400: Bad Request\nNone')
2019-11-20 14:40:02,160 dcrdata ERROR broadcast(929) broadcast error: ('RequestError', 'Error encountered in requesting path https://testnet.dcrdata.org/insight/api/tx/send: HTTP Error 400: Bad Request\nNone')
2019-11-20 14:40:02,169 dcrdata INFO purchaseTickets(1346) published ticket 4d4de58ba2ac8ea794a24e3495a4aa2f1ecb523565fcb54efeeea48cbb075210
2019-11-20 14:40:02,893 dcrdata ERROR broadcast(929) broadcast error: ('RequestError', 'Error encountered in requesting path https://testnet.dcrdata.org/insight/api/tx/send: HTTP Error 400: Bad Request\nNone')
2019-11-20 14:40:02,902 dcrdata INFO purchaseTickets(1346) published ticket 30a0c3aca91dbb10e0095e5da9413216870497feea4595d10a9d37a60a29aaee
2019-11-20 14:40:03,630 dcrdata ERROR broadcast(929) broadcast error: ('RequestError', 'Error encountered in requesting path https://testnet.dcrdata.org/insight/api/tx/send: HTTP Error 400: Bad Request\nNone')
2019-11-20 14:40:03,638 dcrdata INFO purchaseTickets(1346) published ticket 1c57e8bd8e15d276fae28400b2f8f5dc78a080161770341a208d3e9f27b67f4b

I can't remember, but does this check to see if we are in the ticket buying window? Also, is the ticket price and such recalculated if there is a change from the time of entering the vsp screen?

Some more balance discrepancies, these don't need to be fixed now, but should make an issue to make syncing better.
Stake some ~36 tickets, close laptop for about a day, open to this amount:

open laptop

Restart wallet.

after restart

After sync completes. Presumably this is the correct amount.

after sync

Adds support for all script types other that alt-sig types. Implementation is a mirror of parts of txscript and dcrutil modules. Also adds basic stake pool client. Adds new http module in util, used in both dcrdata and stakepool.

Also moves pydecred tests to dedicated test file so they aren't evaluated on import.

Updated interface to support ticket purchases via VSP. No solo voting and still needs revocation support, but signing stake-P2PKH outputs of all types is now supported.

…thon conflict

buck54321 · 2019-11-22T13:40:09Z

pydecred/account.py

-        # Remove spent utxos from cache.
-        self.spendUTXOs(spentUTXOs)
-        # Add new UTXOs to set. These may be replaced with network-sourced
-        # UTXOs once the wallet receives an update from the BlockChain.
-        for utxo in newUTXOs:
-            self.addUTXO(utxo)


Waiting for the updates from dcrdata here instead. This should help with some of the transient balance display issues.

The balance display being wrong for a moment at startup is because the balance is being stored as part of the encrypted wallet object, so when an update to the balance occurs while the wallet is locked, the new value is not properly stored. I have a plan, but it should wait until the database changes proposed in the budget proposal are complete.

buck54321 · 2019-11-22T13:40:55Z

pydecred/dcrdata.py

@@ -927,7 +927,7 @@ def broadcast(self, txHex):
            return True
        except Exception as e:
            log.error("broadcast error: %s" % e)
-        return False
+            raise e


Propagate broadcast exceptions.

buck54321 · 2019-11-22T13:41:59Z

ui/screens.py

+        if cfg.net.Name == "mainnet":
+            self.pools = [p for p in pools if tNow - p["LastUpdated"] < 86400 and self.scorePool(p) > 95]


Pools have a simple filter for inactivity or really poor performance. Unfortunately, 2 of the 3 listed testnet VSPs do not currently pass. Disabling filtering for testnet for now.

buck54321 · 2019-11-22T13:44:33Z

@JoeGruffins

I was able to improve some of the undesirable behavior you were seeing. Some others will get better once the database work is done and the storage is refactored.

I am unable to reproduce your broadcast issues, but the errors will propagate to the UI now. Let's chat on matrix and see if we can figure out how to reproduce.

JoeGruffins

The percent staked still doesn't change until after a restart.

But other than that it's working great! LGTM

Allow crappy pools, lol. Could add my testnet pool, it's not crappy...

buck54321 · 2019-11-23T12:13:32Z

ui/screens.py

@@ -461,8 +461,7 @@ def __init__(self, app):
        optsLyt.addWidget(self.spinner, 0, 1, Q.ALIGN_RIGHT)

        # Open staking window. Button is initally hidden until sync is complete.
-        self.stakeBttn = btn = app.getButton(SMALL, "Staking")
-        btn.setVisible(False)
+        btn = app.getButton(SMALL, "Staking")


No need to hide the staking button anymore with the changes in this PR. This and the gap address synchronization handling should make the sync much faster.

JoeGruffins

Doubly approved.

buck54321 force-pushed the staking branch from 01a68f2 to a25ce8b Compare October 6, 2019 18:06

buck54321 force-pushed the staking branch from a25ce8b to be41115 Compare November 13, 2019 22:09

buck54321 commented Nov 13, 2019

View reviewed changes

buck54321 marked this pull request as ready for review November 13, 2019 23:32

JoeGruffins reviewed Nov 15, 2019

View reviewed changes

JoeGruffins reviewed Nov 16, 2019

View reviewed changes

This was referenced Nov 18, 2019

Add accountless ticket purchasing. #16

Open

Add accountless ticket verification scheme #17

Open

multi: Separate unit tests #18

Merged

teknico mentioned this pull request Nov 21, 2019

Multi: remove indentation tabs #20

Merged

buck54321 added 11 commits November 22, 2019 07:33

add full script support. add stakepool client

43231b3

Adds support for all script types other that alt-sig types. Implementation is a mirror of parts of txscript and dcrutil modules. Also adds basic stake pool client. Adds new http module in util, used in both dcrdata and stakepool.

add vsp and ticket purchase support. add more txscript tests.

2af704b

Also moves pydecred tests to dedicated test file so they aren't evaluated on import.

vsp and ticket screens for gui. pay-to-stake-pubkey-hash script signing

5f12e35

Updated interface to support ticket purchases via VSP. No solo voting and still needs revocation support, but signing stake-P2PKH outputs of all types is now supported.

fix syncing and gap address handling

8e456e6

paginate vsp accounts screen

cab42a6

add missing docs. remove unused code

426d4cd

wallets directory for wallet-related modules

040fbbb

improved wallet loading sequence. rename http to tinyhttp to avoid Py…

ed2bb65

…thon conflict

missing docs. stakepool -> vsp. other fixes

f46c27d

fix sync callback balance = 0 case

295c11e

propagate broadcast exception. allow crappy pools on testnet

7e7a203

buck54321 force-pushed the staking branch from f27d5a7 to 7e7a203 Compare November 22, 2019 13:33

buck54321 commented Nov 22, 2019

View reviewed changes

JoeGruffins approved these changes Nov 23, 2019

View reviewed changes

buck54321 commented Nov 23, 2019

View reviewed changes

calc stake stats internally. more sync improvements

de0a97a

buck54321 force-pushed the staking branch from b5c3731 to de0a97a Compare November 23, 2019 12:57

JoeGruffins approved these changes Nov 24, 2019

View reviewed changes

JoeGruffins mentioned this pull request Nov 26, 2019

multi: Add accountless ticket purchasing #22

Draft

dajohi approved these changes Nov 26, 2019

View reviewed changes

buck54321 merged commit a7d871c into decred:master Nov 26, 2019

buck54321 mentioned this pull request Dec 21, 2019

examples: add ticket purchase example script #33

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add staking #7

add staking #7

buck54321 commented Sep 17, 2019

buck54321 Nov 13, 2019

buck54321 Nov 13, 2019 •

edited

Loading

buck54321 Nov 13, 2019

buck54321 Nov 13, 2019

JoeGruffins Nov 14, 2019

buck54321 Nov 13, 2019

buck54321 Nov 13, 2019

buck54321 Nov 13, 2019

buck54321 Nov 13, 2019

buck54321 Nov 13, 2019 •

edited

Loading

buck54321 Nov 13, 2019

JoeGruffins commented Nov 14, 2019

JoeGruffins left a comment •

edited

Loading

JoeGruffins Nov 14, 2019

JoeGruffins Nov 14, 2019

xaur commented Nov 15, 2019

JoeGruffins Nov 16, 2019 •

edited

Loading

buck54321 Nov 18, 2019

buck54321 Nov 18, 2019

buck54321 commented Nov 18, 2019

buck54321 commented Nov 18, 2019 •

edited

Loading

JoeGruffins commented Nov 20, 2019 •

edited

Loading

buck54321 Nov 22, 2019

buck54321 Nov 22, 2019

buck54321 Nov 22, 2019

buck54321 commented Nov 22, 2019

JoeGruffins left a comment

buck54321 Nov 23, 2019

JoeGruffins left a comment

		if cfg.net.Name == "mainnet":
		self.pools = [p for p in pools if tNow - p["LastUpdated"] < 86400 and self.scorePool(p) > 95]

add staking #7

add staking #7

Conversation

buck54321 commented Sep 17, 2019

Choose a reason for hiding this comment

buck54321 Nov 13, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

buck54321 Nov 13, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JoeGruffins commented Nov 14, 2019

JoeGruffins left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xaur commented Nov 15, 2019

JoeGruffins Nov 16, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

buck54321 commented Nov 18, 2019

buck54321 commented Nov 18, 2019 • edited Loading

JoeGruffins commented Nov 20, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

buck54321 commented Nov 22, 2019

JoeGruffins left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JoeGruffins left a comment

Choose a reason for hiding this comment

buck54321 Nov 13, 2019 •

edited

Loading

buck54321 Nov 13, 2019 •

edited

Loading

JoeGruffins left a comment •

edited

Loading

JoeGruffins Nov 16, 2019 •

edited

Loading

buck54321 commented Nov 18, 2019 •

edited

Loading

JoeGruffins commented Nov 20, 2019 •

edited

Loading