server: app config and asset drivers system #81

chappjc · 2019-11-18T22:00:29Z

The primary changes in this PR are:

Create a DEX type in the new decred.org/dcrdex/server/dex package.
Create a driver system for the asset package and the different backends (drivers). This allows the main app (or a future dex package) to enable backends by name. This is similar to how sql drivers are implemented in and for the "database/sql" standard lib.
Create a similar driver system for the db package.
Define the structure of markets.json and create a parser (loadMarkets).
Introduce the dcrdex CLI app.
Introduce the dex.Runner interface that all DEX subsystems must implement with a Run(context.Context) method
Introduce the dex.StartStopWaiter type, which wraps a dex.Runner to provide the familiar Start/Stop/WaitForShutdown methods.
Change all subsystems to dex.Runners.
Add db/driver/pg.ActiveMatches to get all active swaps for a given user. This is used by the AuthManager for the client connect message.
Add comms.Link as an input argument to auth.respHandler.f to match the handler in other Request functions.
Add a CLI app server/cmd/genkey for generating a public and private key for dex message signing, dumping them to unstructured binary files. The private key is used by DEX. NOTE: The key storage scheme still needs to be implemented properly with encryption and/or PEM encoding.
Add the MarketBuyBuffer setting to each MarketInfo as this can be different for each market.
Similarly allow epoch duration to be different for each Market.
Implement server/asset/dcr.UnspentCoinDetails for FeeChecker.
Several fixes for the outpoint=>coinID changes, and other interface-implementation mismatches such as swap.AuthManager.Penalize and MarketTunnel.CoinLocked.
Add Close() to DEXArchivist interface.
Add epochIdx to bookUpdateSignal so that the BookRouter does not need to compute the epoch index from server time and an epoch duration.
Remove the Time field from order.UserMatch and msgjson.Match.

server/cmd/dcrdex/main.go

buck54321 · 2019-12-06T13:16:26Z

Related to this work: Plugin-style asset backends are still a goal. From the little bit I've played with them, I think it'll be easier to start that work once we have a tagged /server/asset/ module for the plugins to import, and probably a couple of dcrd dependency versions will have to be locked down too.

server/dex/dex.go

server/db/driver/pg/matches.go

chappjc

There is a good deal of unfinished work here, primarily around the DEX manager and the handling of the subsystems it starts and is supposed to be managing, but there is a lot that is ready for review and should be merged.

chappjc · 2020-01-02T16:16:53Z

server/cmd/dcrdex/settings.go

+	"decred.org/dcrdex/dex"
+	dexsrv "decred.org/dcrdex/server/dex"


Renaming "decred.org/dcrdex/server/dex" might be desirable.

chappjc · 2020-01-02T16:19:47Z

server/cmd/genkey/main.go

+	// privkeyFile, err := os.OpenFile("dexprivkey.pem", os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0600)
+	// if err != nil {
+	// 	return err
+	// }
+	// defer privkeyFile.Close()
+
+	// err = pem.Encode(privkeyFile, &pem.Block{
+	// 	Type:  "EC PRIVATE KEY",
+	// 	Bytes: secPrivKey.Serialize(),
+	// })
+	// if err != nil {
+	// 	return err
+	// }
+
+	err = ioutil.WriteFile("dexprivkey", secPrivKey.Serialize(), 0600)
+	if err != nil {
+		return err
+	}


It would be nice if the openssl command could be used to generate the secp256k1 keys in a format the dcrdex can read. We can read PEM, but the way openssl encodes the curve parameters and the public key in the same element is not clear to me yet. We would also want to write in a similar way so openssl could inspect the keys. This will take more work and it's not a show stopper, so I've just fallen back to unstructured binary files.

cspp does this so I will look into the approach there and follow up on this.

chappjc · 2020-01-02T16:34:54Z

server/market/bookrouter.go

 				epochNote.Seq = seq
 				epochNote.MarketID = book.name
-				t := uint64(u.order.Time())
-				epochNote.Epoch = t - t%r.epochLen
+				epochNote.Epoch = uint64(u.epochIdx)


The source of the order feed knows this info, so now we just send it so the BookRouter does not have to know epoch durations or inspect order time stamps.

chappjc · 2020-01-02T16:36:08Z

server/market/orderrouter.go

-	auth     AuthManager
-	assets   map[uint32]*asset.BackedAsset
-	tunnels  map[string]MarketTunnel
-	mbBuffer float64


Provided by the market's MarketTunnel now.

chappjc · 2020-01-02T16:37:19Z

server/swap/swap.go

 	Send(account.AccountID, *msgjson.Message)
 	Request(account.AccountID, *msgjson.Message, func(comms.Link, *msgjson.Message))
-	Penalize(account.AccountID, *order.Match, order.MatchStatus)


The reason for providing order.Match and order.MatchStatus isn't clear, but the implementation in the auth package just requires the account.Rule.

It's not clear to me what will be needed here either. Good for now.

server/dex/dex.go

buck54321

It's Alive!!!

This review is just a first pass, but it's looking really good.

server/asset/btc/btc.go

dex/market.go

server/asset/btc/btc.go

buck54321 · 2020-01-07T11:46:18Z

server/asset/btc/btc_test.go

+	var wg sync.WaitGroup
+	shutdown := func() {
+		cancel()
+		wg.Wait()
+	}
+	wg.Add(1)
+	go func() {
+		btc.Run(ctx)
+		wg.Done()
+	}()


Does it make sense to use a StartStopWaiter here and in tests in general?

I was sort of using the test as an example of direct usage without StartStopWaiter, but I suppose the type itself is the example.

buck54321 · 2020-01-07T11:56:28Z

server/asset/dcr/dcr.go

@@ -155,6 +165,15 @@ func (dcr *DCRBackend) CheckAddress(addr string) bool {
 	return err == nil
 }

+func (dcr *DCRBackend) UnspentCoinDetails(coinID []byte) (addr string, val uint64, confs int64, err error) {


UnspentDetails was just a guess, and it's now unused other than here and a test. You can just change the method signature instead of wrapping it.

Unless you object, I think it would be good to keep the UTXO version available although obv not part of the DEXAsset interface. ba76c77 docs UnspentCoinDetails, tests it directly, and renames UnspentDetails to UTXODetails.

server/dex/dex.go

server/market/market.go

server/cmd/dcrdex/main.go

server/market/market_test.go

server/cmd/dcrdex/config.go

chappjc · 2020-01-08T17:45:59Z

I just restored several commits that I accidentally nuked when rebasing.

buck54321

Second pass. Looking good.

server/asset/btc/btc.go

server/auth/auth.go

.gitignore

server/cmd/dcrdex/config.go

server/market/market.go

buck54321 · 2020-01-13T23:51:47Z

server/market/market.go

+// WaitForEpochOpen waits until the start of epoch processing.
+func (m *Market) WaitForEpochOpen() {
+	<-m.running
+}


This appears to be used for testing only. Looks like it can be unexported at least.

My initial thought was it would be helpful to delay the startup of other subsystems such as the comms.Server or perhaps the BookRouter, but really the OrderRouter will get error messages back from the Market's epoch processing loop that would be just as good as what the OrderRouter would provide to the client if it had known the market(s) were not yet running before submitting the orders for processing.
Delaying the start of the Server might be helpful eventually but I also don't think that's critical.

unexporting as suggested.

server/market/market.go

dex/runner.go

buck54321

Just the pprof issue and it's a go for me.

server/dex/dex.go

buck54321 · 2020-01-15T13:37:50Z

server/dex/dex.go

+	bookSources := make(map[string]market.BookSource, len(cfg.Markets))
+	marketTunnels := make(map[string]market.MarketTunnel, len(cfg.Markets))


I'm considering changing the OrderRouterConfig.Markets field and NewBookRouter argument to []*Market and just converting to the MarketTunnel and BookSource interfaces internally. I can follow up.

buck54321 · 2020-01-15T13:53:54Z

server/cmd/genkey/main.go

+	// 	return err
+	// }
+
+	err = ioutil.WriteFile("dexprivkey", secPrivKey.Serialize(), 0600)


What do you think about adding a network identifier to the filename?

Possibly, although all of this key management is going to change. But my thoughts about genkey are that you'd run it and them move/rename it to wherever depending on your needs. I'd like to revisit this idea when we code up encrypted keys properly.

I guess it's actually the regfeexpub that is network-versioned. The private key is fine.

buck54321 · 2020-01-15T14:08:53Z

server/cmd/dcrdex/config.go

+	HTTPProfile  bool   `long:"httpprof" short:"p" description:"Start HTTP profiler."`
+	HTTPProfPath string `long:"httpprofprefix" description:"URL path prefix for the HTTP profiler."`
+	CPUProfile   string `long:"cpuprofile" description:"File for CPU profiling."`


Description of default behavior would be helpful, e.g. port 9232

Also, when I try to set httpprofprefix, I get some kind of recursion issue.

Ah, this doesn't work with http.DefaultServeMux. I'm removing HTTPProfPath since it was of limited use.

buck54321 · 2020-01-15T14:17:43Z

server/cmd/dcrdex/main.go

+		profPath := opts.HTTPProfPath
+		log.Warnf("Starting the HTTP profiler on path %s.", profPath)
+		// http pprof uses http.DefaultServeMux
+		http.Handle("/", http.RedirectHandler(profPath+"/debug/pprof/", http.StatusSeeOther))


Does profPath need to be applied to the first argument here too?

server/dex package with the DEX (manager) type that starts and manages all of the server subsystems. Notable changes: - Create a DEX type in the new decred.org/dcrdex/server/dex package. - Create a driver system for the asset package and the different backends (drivers). This allows the main app (or a future dex package) to enable backends by name. This is similar to how sql drivers are implemented in and for the "database/sql" standard lib. - Create a similar driver system for the db package. - Define the structure of markets.json and create a parser, loadMarkets. - Introduce the dcrdex CLI app. - Introduce the dex.Runner interface that all DEX subsystems must implement with a Run(context.Context) method - Introduce the dex.StartStopWaiter type, which wraps a dex.Runner to provide the familiar Start/Stop/WaitForShutdown methods. - Change all server subsystems to dex.Runners. - Add db/driver/pg.ActiveMatches to get all active swaps for a given user. This is used by the AuthManager for the client connect message. - Add the MarketBuyBuffer setting to each MarketInfo as this can be different for each market. - Similarly allow epoch duration to be different for each Market. - Implement server/asset/dcr.UnspentCoinDetails for FeeChecker. - Several fixes for the outpoint=>coinID changes, and other interface-implementation mismatches such as swap.AuthManager.Penalize and MarketTunnel.CoinLocked. - Add Close() to DEXArchivist interface. - Add epochIdx to bookUpdateSignal so that the BookRouter does not need to compute the epoch index from server time and an epoch duration. - dex package: add Network.String and NetFromString - Remove dex.Asset.SwapSize - coinlock: Add (*DEXCoinLocker).AssetLocker - Remove the Time field from msgjson.Match and order.UserMatch - market: Remove EpochDuration from BookRouterConfig. Epoch duration may be set per-market. Add epochIdx to bookUpdateSignal, and set bookUpdateSignal.epochIdx in Market.notifyNewOrder. The epochIdx is only used in BookRouter for epochAction (not book and unbookAction). Include mbBuffer and epoch dur in market config. - market: Add EpochDuration() to MarketTunnel. - Remove startEpoch from connect response and AuthManager config. The start epoch for a market must be provided in the 'config' route response (handler not yet implemented). - market: Start at the next epoch by default. - matcher: Avoid seeding a PRNG if len(queue)<2. - btc: Add a note about reorg detection and move a log line. - Reduce reg fee confirm requirement from 6 to 4. Note about the Run(ctx) pattern and dex.StartStopWaiter: Instead of NewMarket starting the main loop in a goroutine automatically this creates Run, SetStartEpochIdx, and Start (wrapping the former). Run and Start are synchronous, so they should be launched inside goroutines (not as goroutines) defined by the caller. A context is passed to Run/Start rather than the constructor. The Market is stopped by cancelling the context.Context. A context.Context and context.CancelFun are no longer Market fields; they are passed around in a somewhat tedious but idiomatic fasion that I fought against initially and now reluctantly accept. Most subsystems are changed into dex.Runners.

NOTE: The key storage scheme still needs to be implemented properly with encryption and/or PEM encoding! Add a CLI app server/cmd/genkey for generating a public and private key for dex message signing, dumping them to unstructured binary files. The private key is used by DEX.

chappjc changed the title ~~Server app~~ server: app config and asset drivers system Nov 18, 2019

chappjc force-pushed the server-app branch from 9a69b97 to 439fc66 Compare November 18, 2019 22:05

chappjc commented Nov 18, 2019

View reviewed changes

server/cmd/dcrdex/main.go Outdated Show resolved Hide resolved

buck54321 mentioned this pull request Nov 18, 2019

market: epoch queue processing #66

Merged

chappjc force-pushed the server-app branch 3 times, most recently from b8a2441 to 5617e7c Compare December 20, 2019 20:41

chappjc commented Dec 30, 2019

View reviewed changes

server/dex/dex.go Outdated Show resolved Hide resolved

chappjc commented Dec 30, 2019

View reviewed changes

server/dex/dex.go Outdated Show resolved Hide resolved

chappjc commented Dec 30, 2019

View reviewed changes

server/dex/dex.go Outdated Show resolved Hide resolved

chappjc force-pushed the server-app branch 2 times, most recently from d3250d0 to d3a84cb Compare January 1, 2020 15:08

chappjc commented Jan 1, 2020

View reviewed changes

server/db/driver/pg/matches.go Show resolved Hide resolved

chappjc commented Jan 2, 2020

View reviewed changes

chappjc marked this pull request as ready for review January 2, 2020 16:41

chappjc commented Jan 3, 2020

View reviewed changes

server/dex/dex.go Outdated Show resolved Hide resolved

chappjc force-pushed the server-app branch 2 times, most recently from 1ee15ae to 531b152 Compare January 6, 2020 17:36

buck54321 requested changes Jan 7, 2020

View reviewed changes

chappjc commented Jan 7, 2020

View reviewed changes

server/market/market_test.go Outdated Show resolved Hide resolved

chappjc commented Jan 7, 2020

View reviewed changes

server/cmd/dcrdex/config.go Outdated Show resolved Hide resolved

chappjc commented Jan 7, 2020

View reviewed changes

server/cmd/dcrdex/config.go Outdated Show resolved Hide resolved

This was referenced Jan 8, 2020

refactor goroutine management #117

Closed

db: server should save all match-related data #112

Closed

chappjc force-pushed the server-app branch from 3a070cf to 5323fd8 Compare January 8, 2020 14:33

chappjc force-pushed the server-app branch 2 times, most recently from 0382458 to 25ef991 Compare January 13, 2020 16:14

chappjc requested a review from buck54321 January 13, 2020 16:17

chappjc force-pushed the server-app branch from 25ef991 to 5eef3e0 Compare January 13, 2020 20:16

chappjc mentioned this pull request Jan 13, 2020

client: application skeleton #100

Merged

buck54321 requested changes Jan 14, 2020

View reviewed changes

buck54321 requested changes Jan 15, 2020

View reviewed changes

buck54321 approved these changes Jan 15, 2020

View reviewed changes

chappjc force-pushed the server-app branch from cd8ddcc to 4484e9d Compare January 15, 2020 15:27

chappjc added 2 commits January 15, 2020 10:14

chappjc force-pushed the server-app branch from 4484e9d to a67bb61 Compare January 15, 2020 16:16

chappjc merged commit 5bf9082 into decred:master Jan 15, 2020

chappjc deleted the server-app branch January 15, 2020 16:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

server: app config and asset drivers system #81

server: app config and asset drivers system #81

chappjc commented Nov 18, 2019 •

edited

Loading

buck54321 commented Dec 6, 2019 •

edited

Loading

chappjc left a comment

chappjc Jan 2, 2020

chappjc Jan 2, 2020

chappjc Jan 9, 2020

chappjc Jan 2, 2020

chappjc Jan 2, 2020

chappjc Jan 2, 2020

buck54321 Jan 7, 2020

buck54321 left a comment

buck54321 Jan 7, 2020

chappjc Jan 9, 2020

buck54321 Jan 7, 2020 •

edited

Loading

chappjc Jan 9, 2020

chappjc commented Jan 8, 2020

buck54321 left a comment

buck54321 Jan 13, 2020

chappjc Jan 14, 2020

buck54321 left a comment

buck54321 Jan 15, 2020

buck54321 Jan 15, 2020

chappjc Jan 15, 2020

buck54321 Jan 19, 2020

buck54321 Jan 15, 2020

chappjc Jan 15, 2020

buck54321 Jan 15, 2020

		"decred.org/dcrdex/dex"
		dexsrv "decred.org/dcrdex/server/dex"

		bookSources := make(map[string]market.BookSource, len(cfg.Markets))
		marketTunnels := make(map[string]market.MarketTunnel, len(cfg.Markets))

server: app config and asset drivers system #81

server: app config and asset drivers system #81

Conversation

chappjc commented Nov 18, 2019 • edited Loading

buck54321 commented Dec 6, 2019 • edited Loading

chappjc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

buck54321 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

buck54321 Jan 7, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chappjc commented Jan 8, 2020

buck54321 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

buck54321 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chappjc commented Nov 18, 2019 •

edited

Loading

buck54321 commented Dec 6, 2019 •

edited

Loading

buck54321 Jan 7, 2020 •

edited

Loading