From 83008eb4fce234f74ae333f0de24664d6f27bdcc Mon Sep 17 00:00:00 2001 From: Shawn-Huang-Tron <107823650+Shawn-Huang-Tron@users.noreply.github.com> Date: Tue, 18 Jun 2024 16:22:55 +0800 Subject: [PATCH] Release 3.0.0 (#442) * feat[cheque]: add command about batch cashing cheque * ci: add test command * feat: encrypt support symmetrical encryption * feat: implementation btip-73 * feat: use last connect peer * feat: refactor use last conn peer * feat: use last conn to boot * doc: update annotations * fix: need conn peer filer error * feat: update config dependency for updating bootstrap * feat: get peer addr from addrbook * feat: add try duration from last conn * feat: change go routine limit to peers size on do conn * fix: judge if from option is empty * fix: check if pass is empty * feat: give a more friendly hint * fix: get local data when from is exactly local node * feat: add a timeout when get an encrypted cid * fix: status report list should not return a nil * feat: update dashboard cid and version * feat: update readme * feat: update readme * feat: update readme * feat: udpate some expressions in readme * feat: update latest cid --------- Co-authored-by: cody Co-authored-by: cody Co-authored-by: Cody <133026515+mengcody@users.noreply.github.com> --- README.md | 39 +++++-- cmd/btfs/init.go | 28 +++++ core/commands/backup.go | 3 - core/commands/cheque/cheque.go | 53 ++++++++- core/commands/commands_test.go | 1 + core/commands/encrypt.go | 208 ++++++++++++++++++++++++++------- core/commands/statusonline.go | 14 ++- core/core.go | 2 +- core/corehttp/webui.go | 3 +- core/node/groups.go | 7 +- core/node/libp2p/peerstore.go | 16 +++ core/node/peering.go | 172 ++++++++++++++++++++++++++- go.mod | 2 +- go.sum | 4 +- version.go | 2 +- 15 files changed, 481 insertions(+), 73 deletions(-) diff --git a/README.md b/README.md index 226017b2f..b820706cb 100644 --- a/README.md +++ b/README.md @@ -1,22 +1,41 @@ # go-btfs -## What is BTFS 2.0? +## What is BTFS 3.0? -BitTorrent File System (BTFS) is a next-generation file sharing protocol in the BitTorrent ecosystem. Current mainstream public blockchains mostly focus on computational tasks but lack cost-effective, scalable, and high-performing file storage and sharing solutions. +BTFS, as an innovative force in the BitTorrent ecosystem, has not only accelerated the development of distributed file sharing technology, but also taken a leading position in the field of DePIN. DePIN - which stands for Decentralized Physical Infrastructure Network - encourages network participants to jointly invest resources to deploy and maintain a more stable and efficient network infrastructure through a token reward mechanism. Current mainstream public blockchains mostly focus on computational tasks but lack cost-effective, scalable, and high-performing file storage and sharing solutions. -These are exactly what BTFS aims to clear up. Besides, underpinned by BTTC, BTFS enables cross-chain connectivity and multi-channel payments, making itself a more convenient choice. The intgration of BTFS, BitTorrent, and the BTTC network will boost DApp developers' efficiency in serving a wider market. +These are exactly what BTFS aims to clear up. Additionally, underpinned by BTTC, BTFS enables cross-chain connectivity and multi-channel payments, making it a more convenient choice. The integration of BTFS, BitTorrent, and the BTTC network will boost DApp developers' efficiency in serving a wider market. -* The [documentation](https://docs.btfs.io/) walks developers through BTFS 2.0 setup, usage, and API references. -* Please join the BTFS community at [discord](https://discord.com/invite/tJ4fz6C6rw). +* The [documentation](https://docs.btfs.io/) walks developers through BTFS 3.0 setup, usage, and API references. +* Please join the BTFS community on [Discord](https://discord.com/invite/tJ4fz6C6rw). -## BTFS 2.0 Architecture Diagram +Decentralized Physical Infrastructure Networks (DePINs), catalyzed by advancements in blockchain technology, aim to revolutionize network architecture by shifting away from centralized infrastructure. This transformation is made possible by the evolution of blockchain infrastructure and advancements in cryptography, enabling a global collaborative effort to establish open and permissionless infrastructure. This approach disrupts traditional infrastructure models, which are typically dominated by major corporations. +The BitTorrent File System (BTFS), an open and distributed file storage system, aligns perfectly with the goals of DePIN. + +BTFS represents the next generation of decentralized file storage systems powered by blockchain technology and peer-to-peer transmission. It allows users to store files across multiple global nodes, enhancing file security and reliability while facilitating faster access and transfer. This setup provides an effortless file management and sharing experience. Additionally, it incorporates key features from the BitTorrent Chain (BTTC) network, such as cross-chain connectivity and multichannel payment options. + +As a stellar player in the DePIN space, BTFS v3.0 boasts the following advantages: + +* Decentralized Storage: BTFS has achieved global decentralized file storage through its distributed network. Unlike centralized storage on single servers, this method significantly increases data redundancy and reduces the risk of data loss. The BTFS v3.0 update includes enhancements such as symmetric encryption for secure data transmission and new features like Keystore files for managing BTFS nodes. +* Enhanced Security: The decentralized nature of data storage makes it difficult for attackers to target all nodes simultaneously, greatly improving the system's overall security. +* Censorship-Proof: The inherent nature of a decentralized storage system makes it extremely difficult for any single organization to censor or block content, ensuring the free flow of data. +* Seamless Integration Into the BitTorrent Ecosystem: The seamless integration of BTFS with the BitTorrent ecosystem allows users to freely access and share files, as well as facilitates cross-chain transactions and payments via BTTC. +* User-friendly API and Visual Interface: A newly updated BTFS website offers a user-friendly API and visual interface, featuring detailed road maps on Storage Reward halving and various other modules. +* Transparent Market Mechanism: The market mechanism within BTFS ensures complete transparency in storage and payment processes, providing users with maximum visibility into their data. +* Encryption Technology: By leveraging advanced encryption technologies, BTFS enhances data privacy and security, ensuring data integrity and preventing unauthorized access during transmission and storage. + +In addition to these features, the upgraded BTFS v3.0 introduces a crucial model update—the Reward Halving. As the BTFS network and its user base grow, the reward halving is likely to contribute to the reliable and sustainable growth of the entire ecosystem. + +BTFS, a leading project in the space of DePIN, is committed to advancing and enhancing the decentralized physical infrastructure network. The improved token economics of BTFS not only align with the vision of DePIN but also lay a solid foundation for the growth of decentralized networks. As technology and community evolve, BTFS is poised to play an increasingly significant role in the decentralized world. + +## BTFS 3.0 Architecture Diagram ![Architecture Diagram](https://files.readme.io/a21e9fb--min.png) ## Table of Contents - [go-btfs](#go-btfs) - - [What is BTFS 2.0?](#what-is-btfs-20) - - [BTFS 2.0 Architecture Diagram](#btfs-20-architecture-diagram) + - [What is BTFS 3.0?](#what-is-btfs-30) + - [BTFS 3.0 Architecture Diagram](#btfs-30-architecture-diagram) - [Table of Contents](#table-of-contents) - [Faucet](#faucet) - [Install BTFS](#install-btfs) @@ -39,7 +58,7 @@ These are exactly what BTFS aims to clear up. Besides, underpinned by BTTC, BTFS ## Faucet -In order to ensure the normal use of btfs 2.0 testnet, you need to apply for BTT at BTTC testnet, which is obtained [**here**](https://testfaucet.bt.io/#/). +In order to ensure the normal use of btfs 3.0 testnet, you need to apply for BTT at BTTC testnet, which is obtained [**here**](https://testfaucet.bt.io/#/). ## Install BTFS @@ -82,7 +101,7 @@ Specify the chain for btfs to run by `--chain-id`, the chainid of the test netwo ``` $ btfs daemon --chain-id 1029 Initializing daemon... -go-btfs version: 2.0 +go-btfs version: 3.0 Repo version: 10 System version: amd64/darwin Golang version: go1.16.5 diff --git a/cmd/btfs/init.go b/cmd/btfs/init.go index 63950b5c9..178fd03a7 100644 --- a/cmd/btfs/init.go +++ b/cmd/btfs/init.go @@ -2,6 +2,7 @@ package main import ( "context" + "encoding/hex" "encoding/json" "errors" "fmt" @@ -21,6 +22,8 @@ import ( "github.com/bittorrent/go-btfs/core/commands" "github.com/bittorrent/go-btfs/namesys" fsrepo "github.com/bittorrent/go-btfs/repo/fsrepo" + "github.com/ethereum/go-ethereum/accounts/keystore" + ethCrypto "github.com/ethereum/go-ethereum/crypto" cmds "github.com/bittorrent/go-btfs-cmds" config "github.com/bittorrent/go-btfs-config" @@ -33,6 +36,8 @@ const ( profileOptionName = "profile" keyTypeDefault = "BIP39" keyTypeOptionName = "key" + keyPathOptionName = "keypath" + passOptionName = "pass" importKeyOptionName = "import" rmOnUnpinOptionName = "rm-on-unpin" seedOptionName = "seed" @@ -70,6 +75,8 @@ environment variable: cmds.BoolOption(emptyRepoOptionName, "e", "Don't add and pin help files to the local storage."), cmds.StringOption(profileOptionName, "p", "Apply profile settings to config. Multiple profiles can be separated by ','"), cmds.StringOption(keyTypeOptionName, "k", "Key generation algorithm, e.g. RSA, Ed25519, Secp256k1, ECDSA, BIP39. By default is BIP39"), + cmds.StringOption(keyPathOptionName, "kp", "Keystore file path when key type is keystore"), + cmds.StringOption(passOptionName, "Keystore file password when key type is keystore"), cmds.StringOption(importKeyOptionName, "i", "Import TRON private key to generate btfs PeerID."), cmds.BoolOption(rmOnUnpinOptionName, "r", "Remove unpinned files.").WithDefault(false), cmds.StringOption(seedOptionName, "s", "Import seed phrase"), @@ -139,10 +146,31 @@ environment variable: keyType, _ := req.Options[keyTypeOptionName].(string) seedPhrase, _ := req.Options[seedOptionName].(string) simpleModeIn, _ := req.Options[simpleMode].(bool) + keyPath, _ := req.Options[keyPathOptionName].(string) + keyPass, _ := req.Options[passOptionName].(string) /* password, _ := req.Options[passWordOptionName].(string) passwordFile, _ := req.Options[passwordFileoptionName].(string) */ + if keyType == "keystore" { + if keyPath == "" || keyPass == "" { + return fmt.Errorf("keypath or pass option should not be empty when key type is keystore") + } + file, err := os.Open(keyPath) + if err != nil { + return err + } + jsonBytes, err := io.ReadAll(file) + if err != nil { + return err + } + key, err := keystore.DecryptKey(jsonBytes, keyPass) + if err != nil { + return err + } + importKey = hex.EncodeToString(ethCrypto.FromECDSA(key.PrivateKey)) + keyType = "Secp256k1" + } backupPath, ok := req.Options[recoveryOptionName].(string) if ok { btfsPath := env.(*oldcmds.Context).ConfigRoot diff --git a/core/commands/backup.go b/core/commands/backup.go index 340a0f5e1..e88862d39 100644 --- a/core/commands/backup.go +++ b/core/commands/backup.go @@ -170,7 +170,6 @@ func Tar(src, dst string, excludePath []string) (err error) { } hdr.Name = rel - // 写入文件信息 if err = tw.WriteHeader(hdr); err != nil { return err } @@ -185,7 +184,6 @@ func Tar(src, dst string, excludePath []string) (err error) { } defer fr.Close() - // copy 文件数据到 tw _, err = io.Copy(tw, fr) if err != nil { return err @@ -232,7 +230,6 @@ func UnTar(src, dst string) (err error) { return err } defer fw.Close() - // 写文件 _, err = io.Copy(fw, tr) if err != nil { return err diff --git a/core/commands/cheque/cheque.go b/core/commands/cheque/cheque.go index 7e578acc8..5af9c0746 100644 --- a/core/commands/cheque/cheque.go +++ b/core/commands/cheque/cheque.go @@ -3,12 +3,13 @@ package cheque import ( "errors" "fmt" - "github.com/bittorrent/go-btfs/chain/tokencfg" - "github.com/bittorrent/go-btfs/utils" "io" "math/big" "time" + "github.com/bittorrent/go-btfs/chain/tokencfg" + "github.com/bittorrent/go-btfs/utils" + cmds "github.com/bittorrent/go-btfs-cmds" "github.com/bittorrent/go-btfs/chain" "github.com/ethereum/go-ethereum/common" @@ -80,6 +81,7 @@ Vault services include issue cheque to peer, receive cheque and store operations }, Subcommands: map[string]*cmds.Command{ "cash": CashChequeCmd, + "batch-cash": BatchCashChequeCmd, "cashstatus": ChequeCashStatusCmd, "cashlist": ChequeCashListCmd, "price": StorePriceCmd, @@ -257,3 +259,50 @@ var CashChequeCmd = &cmds.Command{ }), }, } + +var BatchCashChequeCmd = &cmds.Command{ + Helptext: cmds.HelpText{ + Tagline: "Batch cash the cheques by peerIDs.", + }, + Arguments: []cmds.Argument{ + cmds.StringArg("peer-ids", true, true, "Peer id tobe cashed."), + }, + Options: []cmds.Option{ + cmds.StringOption(tokencfg.TokenTypeName, "tk", "file storage with token type,default WBTT, other TRX/USDD/USDT.").WithDefault("WBTT"), + }, + RunTimeout: 5 * time.Minute, + Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) error { + err := utils.CheckSimpleMode(env) + if err != nil { + return err + } + + peerIDs := req.Arguments + tokenStr := req.Options[tokencfg.TokenTypeName].(string) + token, bl := tokencfg.MpTokenAddr[tokenStr] + if !bl { + return errors.New("your input token is none. ") + } + + for _, peerID := range peerIDs { + tx_hash, err := chain.SettleObject.SwapService.CashCheque(req.Context, peerID, token) + if err != nil { + return err + } + err = res.Emit(&CashChequeRet{ + TxHash: tx_hash.String(), + }) + if err != nil { + return err + } + } + return nil + }, + Type: CashChequeRet{}, + Encoders: cmds.EncoderMap{ + cmds.Text: cmds.MakeTypedEncoder(func(req *cmds.Request, w io.Writer, out *CashChequeRet) error { + _, err := fmt.Fprintf(w, "the hash of transaction: %s", out.TxHash) + return err + }), + }, +} diff --git a/core/commands/commands_test.go b/core/commands/commands_test.go index bcc0877a9..ed804deb5 100644 --- a/core/commands/commands_test.go +++ b/core/commands/commands_test.go @@ -284,6 +284,7 @@ func TestCommands(t *testing.T) { "/cheque/token_balance", "/cheque/all_token_balance", "/cheque/cash", + "/cheque/batch-cash", "/cheque/cashstatus", "/cheque/chaininfo", "/cheque/price", diff --git a/core/commands/encrypt.go b/core/commands/encrypt.go index 864881591..e3ef23002 100644 --- a/core/commands/encrypt.go +++ b/core/commands/encrypt.go @@ -2,10 +2,20 @@ package commands import ( "bytes" + "context" + "crypto/aes" + "crypto/cipher" + "crypto/md5" "crypto/rand" "encoding/base64" "errors" + "fmt" "io" + "net/http" + "os" + "path" + "strings" + "time" shell "github.com/bittorrent/go-btfs-api" cmds "github.com/bittorrent/go-btfs-cmds" @@ -15,9 +25,11 @@ import ( ethCrypto "github.com/ethereum/go-ethereum/crypto" "github.com/ethereum/go-ethereum/crypto/ecies" peer "github.com/libp2p/go-libp2p/core/peer" + "github.com/mitchellh/go-homedir" ) const toOption = "to" +const passOption = "pass" const fromOption = "from" var encryptCmd = &cmds.Command{ @@ -29,35 +41,13 @@ var encryptCmd = &cmds.Command{ }, Options: []cmds.Option{ cmds.StringOption(toOption, "the peerID of the node which you want to share with"), + cmds.StringOption(passOption, "p", "the password that you want to encrypt the file by AES"), }, Run: func(r *cmds.Request, re cmds.ResponseEmitter, e cmds.Environment) error { n, err := cmdenv.GetNode(e) if err != nil { return err } - to, ok := r.Options[toOption].(string) - if !ok { - to = n.Identity.String() - } - id, err := peer.Decode(to) - if err != nil { - return errors.New("the to option must be a valid peerID") - } - p2pPk, err := id.ExtractPublicKey() - if err != nil { - return errors.New("can't extract public key from peerID") - } - pkBytes, err := cp.Secp256k1PublicKeyRaw(p2pPk) - if err != nil { - return errors.New("can't change from p2p public key to secp256k1 public key from peerID") - } - - ethPk, err := ethCrypto.UnmarshalPubkey(pkBytes) - if err != nil { - return errors.New("can't unmarshall public key from peerID") - } - - eciesPk := ecies.ImportECDSAPublic(ethPk) it := r.Files.Entries() file, err := cmdenv.GetFileArg(it) if err != nil { @@ -67,10 +57,48 @@ var encryptCmd = &cmds.Command{ if err != nil { return err } - encryptedBytes, err := ECCEncrypt(originalBytes, *eciesPk) - if err != nil { - return err + var encryptedBytes []byte + pass, ok := r.Options[passOption].(string) + if ok { + // That means it's symmetrical encryption + hasher := md5.New() + hasher.Write([]byte(pass)) + secretBytes := hasher.Sum(nil) + encryptedBytes, err = AesEncrypt(originalBytes, secretBytes) + if err != nil { + return err + } + } else { + // That means it's asymmetric encryption + to, ok := r.Options[toOption].(string) + if !ok { + to = n.Identity.String() + } + id, err := peer.Decode(to) + if err != nil { + return errors.New("the to option must be a valid peerID") + } + p2pPk, err := id.ExtractPublicKey() + if err != nil { + return errors.New("can't extract public key from peerID") + } + pkBytes, err := cp.Secp256k1PublicKeyRaw(p2pPk) + if err != nil { + return errors.New("can't change from p2p public key to secp256k1 public key from peerID") + } + + ethPk, err := ethCrypto.UnmarshalPubkey(pkBytes) + if err != nil { + return errors.New("can't unmarshall public key from peerID") + } + + eciesPk := ecies.ImportECDSAPublic(ethPk) + encryptedBytes, err = ECCEncrypt(originalBytes, *eciesPk) + if err != nil { + return err + } } + btfsClient := shell.NewLocalShell() cid, err := btfsClient.Add(bytes.NewReader(encryptedBytes), shell.Pin(true)) if err != nil { @@ -89,6 +117,7 @@ var decryptCmd = &cmds.Command{ }, Options: []cmds.Option{ cmds.StringOption(fromOption, "specify the source peerID of CID"), + cmds.StringOption(passOption, "p", "the password that you want to decrypt the file by AES"), }, Run: func(r *cmds.Request, re cmds.ResponseEmitter, e cmds.Environment) error { conf, err := cmdenv.GetConfig(e) @@ -107,42 +136,93 @@ var decryptCmd = &cmds.Command{ var readClose io.ReadCloser cid := r.Arguments[0] from, ok := r.Options[fromOption].(string) - if ok { + timeout := 1 * time.Minute + if ok && strings.TrimSpace(from) != "" && strings.TrimSpace(from) != n.Identity.String() { peerID, err := peer.Decode(from) if err != nil { return err } - b, err := remote.P2PCallStrings(r.Context, n, api, peerID, "/decryption", cid) + ctx, cancel := context.WithTimeout(r.Context, timeout) + defer cancel() + b, err := remote.P2PCallStrings(ctx, n, api, peerID, "/decryption", cid) + if err != nil && strings.Contains(err.Error(), "unsupported path namespace") { + return errors.New("cid not found") + } if err != nil { return err } readClose = io.NopCloser(bytes.NewReader(b)) } else { - readClose, err = shell.NewLocalShell().Cat(cid) + c := &http.Client{ + Transport: &http.Transport{ + Proxy: http.ProxyFromEnvironment, + DisableKeepAlives: true, + }, + Timeout: timeout, + } + baseDir := os.Getenv("BTFS_PATH") + if baseDir == "" { + baseDir = "~/.btfs" + } + + baseDir, err := homedir.Expand(baseDir) if err != nil { - return err + return nil } - } - pkbytesOri, err := base64.StdEncoding.DecodeString(conf.Identity.PrivKey) - if err != nil { - return err - } - ecdsaPrivateKey, err := ethCrypto.ToECDSA(pkbytesOri[4:]) - if err != nil { - return err + apiFile := path.Join(baseDir, "api") + if _, err := os.Stat(apiFile); err != nil { + return nil + } + api, err := os.ReadFile(apiFile) + if err != nil { + return nil + } + readClose, err = shell.NewShellWithClient(strings.TrimSpace(string(api)), c).Cat(cid) + if err != nil && strings.Contains(err.Error(), "unsupported path namespace") { + return errors.New("cid not found") + } + if err != nil && strings.Contains(err.Error(), "Timeout") { + return fmt.Errorf("timeout when try to get cid: %s", cid) + } } - eciesPrivateKey := ecies.ImportECDSA(ecdsaPrivateKey) - endata, err := io.ReadAll(readClose) + encryptedData, err := io.ReadAll(readClose) if err != nil { return err } defer readClose.Close() - dedata, err := ECCDecrypt(endata, *eciesPrivateKey) - if err != nil { - return errors.New("decryption is failed, maybe the content of encryption is not encrypted by your public key") + + var decryptedData []byte + pass, ok := r.Options[passOption].(string) + if ok && strings.TrimSpace(pass) != "" { + // That means it's symmetrical encryption + hasher := md5.New() + hasher.Write([]byte(pass)) + secretBytes := hasher.Sum(nil) + decryptedData, err = AesDecrypt(encryptedData, secretBytes) + if err != nil { + log.Error(err) + return errors.New("error happens during decryption by AES, may be you have a wrong password or it is not encrypted by AES") + } + } else { + // That means it's asymmetric encryption + pkbytesOri, err := base64.StdEncoding.DecodeString(conf.Identity.PrivKey) + if err != nil { + return err + } + ecdsaPrivateKey, err := ethCrypto.ToECDSA(pkbytesOri[4:]) + if err != nil { + return err + } + eciesPrivateKey := ecies.ImportECDSA(ecdsaPrivateKey) + + decryptedData, err = ECCDecrypt(encryptedData, *eciesPrivateKey) + if err != nil { + log.Error(err) + return errors.New("decryption is failed, maybe the content of encryption is not encrypted by your public key") + } } - return re.Emit(bytes.NewReader(dedata)) + return re.Emit(bytes.NewReader(decryptedData)) }, } @@ -155,3 +235,43 @@ func ECCDecrypt(ct []byte, prk ecies.PrivateKey) ([]byte, error) { pt, err := prk.Decrypt(ct, nil, nil) return pt, err } + +func PKCS7Padding(ciphertext []byte, blockSize int) []byte { + padding := blockSize - len(ciphertext)%blockSize + padtext := bytes.Repeat([]byte{byte(padding)}, padding) + return append(ciphertext, padtext...) +} + +func PKCS7UnPadding(origData []byte) []byte { + length := len(origData) + unpadding := int(origData[length-1]) + return origData[:(length - unpadding)] +} + +// AES decryption,CBC +func AesEncrypt(origData, key []byte) ([]byte, error) { + block, err := aes.NewCipher(key) + if err != nil { + return nil, err + } + blockSize := block.BlockSize() + origData = PKCS7Padding(origData, blockSize) + blockMode := cipher.NewCBCEncrypter(block, key[:blockSize]) + crypted := make([]byte, len(origData)) + blockMode.CryptBlocks(crypted, origData) + return crypted, nil +} + +// AES decryption +func AesDecrypt(crypted, key []byte) ([]byte, error) { + block, err := aes.NewCipher(key) + if err != nil { + return nil, err + } + blockSize := block.BlockSize() + blockMode := cipher.NewCBCDecrypter(block, key[:blockSize]) + origData := make([]byte, len(crypted)) + blockMode.CryptBlocks(origData, crypted) + origData = PKCS7UnPadding(origData) + return origData, nil +} diff --git a/core/commands/statusonline.go b/core/commands/statusonline.go index 4948bc344..89b677144 100644 --- a/core/commands/statusonline.go +++ b/core/commands/statusonline.go @@ -3,15 +3,16 @@ package commands import ( "encoding/json" "fmt" + "io" + "strconv" + "time" + cmds "github.com/bittorrent/go-btfs-cmds" onlinePb "github.com/bittorrent/go-btfs-common/protos/online" "github.com/bittorrent/go-btfs/chain" "github.com/bittorrent/go-btfs/core/commands/cmdenv" "github.com/bittorrent/go-btfs/spin" "github.com/bittorrent/go-btfs/utils" - "io" - "strconv" - "time" ) // ReportOnlineDailyCmd (report online daily) @@ -100,7 +101,12 @@ var ReportListDailyCmd = &cmds.Command{ return err } if list == nil { - return nil + return cmds.EmitOnce(res, &RetReportOnlineListDaily{ + Records: nil, + Total: 0, + PeerId: peerId, + BttcAddr: bttcAddr, + }) } total := len(list) // order by time desc diff --git a/core/core.go b/core/core.go index c26256236..fce6c68c3 100644 --- a/core/core.go +++ b/core/core.go @@ -82,7 +82,7 @@ type IpfsNode struct { Discovery discovery.Service `optional:"true"` FilesRoot *mfs.Root RecordValidator record.Validator - //Statestore storage.StateStorer + // Statestore storage.StateStorer // Online PeerHost p2phost.Host `optional:"true"` // the network host (server+client) diff --git a/core/corehttp/webui.go b/core/corehttp/webui.go index 4688dda44..148177b76 100644 --- a/core/corehttp/webui.go +++ b/core/corehttp/webui.go @@ -1,10 +1,11 @@ package corehttp -const WebUIPath = "/btfs/QmSTcj1pWk972bdtStQtJeu3yCYo1SDShrbEHaEybciTLR" // v2.3.5 +const WebUIPath = "/btfs/QmRwt94fr6b3phJThuDMjZUXr6nHxxG61znN3PCukNAxyi" // v3.0.0 // this is a list of all past webUI paths. var WebUIPaths = []string{ WebUIPath, + "/btfs/QmSTcj1pWk972bdtStQtJeu3yCYo1SDShrbEHaEybciTLR", // v2.3.5 "/btfs/QmPFT7PscyJ1FZ4FeFPFgikszBugQSBFNycPpy5zpK2pZe", // v2.3.3 "/btfs/QmUKCyDc4h9KN93AdZ7ZVqgPProsKs8NAbVJkK3ux9788d", // v2.3.2 "/btfs/QmRdt8SzRBz5px7KfU4hFveJSKzBMFqv73YE4xXJBsVdDJ", // v2.3.1 diff --git a/core/node/groups.go b/core/node/groups.go index 45cf0309d..a2ad80556 100644 --- a/core/node/groups.go +++ b/core/node/groups.go @@ -214,7 +214,8 @@ func Identity(cfg *config.Config) fx.Option { if cfg.Identity.PrivKey == "" { return fx.Options( // No PK (usually in tests) fx.Provide(PeerID(id)), - fx.Provide(libp2p.Peerstore), + // fx.Provide(libp2p.Peerstore), + fx.Provide(libp2p.PeerstoreDs), ) } @@ -236,7 +237,8 @@ func Identity(cfg *config.Config) fx.Option { return fx.Options( // Full identity fx.Provide(PeerID(id)), fx.Provide(PrivateKey(sk)), - fx.Provide(libp2p.Peerstore), + // fx.Provide(libp2p.Peerstore), + fx.Provide(libp2p.PeerstoreDs), fx.Invoke(libp2p.PstoreAddSelfKeys), ) @@ -296,6 +298,7 @@ func Online(bcfg *BuildCfg, cfg *config.Config) fx.Option { fx.Provide(Namesys(ipnsCacheSize)), fx.Provide(Peering), PeerWith(cfg.Peering.Peers...), + PeerWithLastConn(), fx.Invoke(IpnsRepublisher(repubPeriod, recordLifetime)), diff --git a/core/node/libp2p/peerstore.go b/core/node/libp2p/peerstore.go index 198fb9ac2..261ac52f1 100644 --- a/core/node/libp2p/peerstore.go +++ b/core/node/libp2p/peerstore.go @@ -2,6 +2,8 @@ package libp2p import ( "context" + "github.com/bittorrent/go-btfs/repo" + "github.com/libp2p/go-libp2p/p2p/host/peerstore/pstoreds" "github.com/libp2p/go-libp2p/core/peerstore" "github.com/libp2p/go-libp2p/p2p/host/peerstore/pstoremem" @@ -22,3 +24,17 @@ func Peerstore(lc fx.Lifecycle) peerstore.Peerstore { return pstore } + +func PeerstoreDs(lc fx.Lifecycle, repo repo.Repo) peerstore.Peerstore { + pstore, err := pstoreds.NewPeerstore(context.Background(), repo.Datastore(), pstoreds.DefaultOpts()) + if err != nil { + log.Errorln(err) + return nil + } + lc.Append(fx.Hook{ + OnStop: func(ctx context.Context) error { + return pstore.Close() + }, + }) + return pstore +} diff --git a/core/node/peering.go b/core/node/peering.go index b9ff39240..ce613bb9a 100644 --- a/core/node/peering.go +++ b/core/node/peering.go @@ -2,12 +2,16 @@ package node import ( "context" - + config "github.com/bittorrent/go-btfs-config" "github.com/bittorrent/go-btfs/peering" - "github.com/libp2p/go-libp2p/core/host" + "github.com/libp2p/go-libp2p/core/network" "github.com/libp2p/go-libp2p/core/peer" "go.uber.org/fx" + "golang.org/x/sync/errgroup" + "math/rand" + "sync/atomic" + "time" ) // Peering constructs the peering service and hooks it into fx's lifetime @@ -33,3 +37,167 @@ func PeerWith(peers ...peer.AddrInfo) fx.Option { } }) } + +const ( + maxNLastConn = 10 + maxTryLimit = 100 + maxTimeDuration = 20 * time.Second + connTimeout = 3 * time.Second +) + +func loadConnPeers(host host.Host, cfg *config.Config) map[peer.ID]bool { + peerIds := host.Peerstore().PeersWithAddrs() + + bootstrap, err := cfg.BootstrapPeers() + if err != nil { + logger.Warn("failed to parse bootstrap peers from config") + } + + filter := make(map[peer.ID]bool, len(bootstrap)) + for _, id := range bootstrap { + filter[id.ID] = true + } + + canConnect := make(map[peer.ID]bool) + for _, id := range peerIds { + if host.Network().Connectedness(id) == network.Connected || id == host.ID() || filter[id] { + continue + } + canConnect[id] = true + } + return canConnect +} + +func randomSubsetOfPeers(in map[peer.ID]bool, max int) map[peer.ID]bool { + c := 0 + for _, v := range in { + if v { + c++ + } + } + + if max > c { + max = c + } + + out := make(map[peer.ID]bool, max) + + tem := make([]peer.ID, 0) + for k, v := range in { + if v { + tem = append(tem, k) + } + } + + for _, val := range rand.Perm(max) { + out[tem[val]] = true + } + + return out +} + +func clearTriedPeer(peers map[peer.ID]bool, triedPeer map[peer.ID]bool) map[peer.ID]bool { + for k := range triedPeer { + peers[k] = false + } + return peers +} + +func doConcurrentConn(ctx context.Context, host host.Host, peers map[peer.ID]bool) int32 { + if len(peers) < 1 { + return 0 + } + + g := errgroup.Group{} + g.SetLimit(len(peers)) + + connected := int32(0) + + for id := range peers { + peerId := id + g.Go(func() error { + if err := host.Connect(ctx, host.Peerstore().PeerInfo(peerId)); err != nil { + logger.Debugf("connect to last connection peer %s, error %v", peerId, err) + return nil + } + atomic.AddInt32(&connected, 1) + return nil + }) + } + _ = g.Wait() + + return connected +} + +func tryConn(host host.Host, peers map[peer.ID]bool) { + + success := make(chan struct{}) + useOut := make(chan struct{}) + maxTry := make(chan struct{}) + timeout := make(chan struct{}) + + timer := time.NewTimer(maxTimeDuration) + + needPeerCount := maxNLastConn + canTryPeerCount := maxTryLimit + + ctx := context.Background() + ctx, cancel := context.WithTimeout(ctx, connTimeout) + defer cancel() + + go func() { + for { + select { + case <-timer.C: + timeout <- struct{}{} + return + default: + conns := randomSubsetOfPeers(peers, needPeerCount) + peers = clearTriedPeer(peers, conns) + + connCount := doConcurrentConn(ctx, host, conns) + + needPeerCount -= int(connCount) + canTryPeerCount -= len(conns) + + if len(conns) <= 0 { + useOut <- struct{}{} + return + } + + if needPeerCount <= 0 { + success <- struct{}{} + return + } + + if canTryPeerCount <= 0 { + maxTry <- struct{}{} + return + } + } + } + }() + + select { + case <-timeout: + logger.Debugf("connect to last connection timeout") + return + case <-success: + logger.Debugf("connect to last connection success") + return + case <-useOut: + logger.Debugf("connect to last connection use out") + return + case <-maxTry: + logger.Debugf("connect to last connection try limited") + return + } +} + +// PeerWithLastConn tryConn to connect to last peers +func PeerWithLastConn() fx.Option { + return fx.Invoke(func(host host.Host, cfg *config.Config) { + peers := loadConnPeers(host, cfg) + tryConn(host, peers) + }) +} diff --git a/go.mod b/go.mod index 26cf9bcde..ee21874d6 100644 --- a/go.mod +++ b/go.mod @@ -10,7 +10,7 @@ require ( github.com/bittorrent/go-btfs-chunker v0.4.0 github.com/bittorrent/go-btfs-cmds v0.3.0 github.com/bittorrent/go-btfs-common v0.9.0 - github.com/bittorrent/go-btfs-config v0.13.0-pre2 + github.com/bittorrent/go-btfs-config v0.13.2 github.com/bittorrent/go-btfs-files v0.3.1 github.com/bittorrent/go-btns v0.2.0 github.com/bittorrent/go-common/v2 v2.4.0 diff --git a/go.sum b/go.sum index 17535d97b..683346ce1 100644 --- a/go.sum +++ b/go.sum @@ -208,8 +208,8 @@ github.com/bittorrent/go-btfs-cmds v0.3.0 h1:xpCBgk3zIm84Ne6EjeJgi8WLB5YJJUIFMjK github.com/bittorrent/go-btfs-cmds v0.3.0/go.mod h1:Fbac/Rou32G0jpoa6wLrNNDxcGOZbGfk+GiG0r3uEIU= github.com/bittorrent/go-btfs-common v0.9.0 h1:jHcFvYQmvmA4IdvVtkI5d/S/HW65Qz21C6oxeyK812w= github.com/bittorrent/go-btfs-common v0.9.0/go.mod h1:OG1n3DfcTxQYfLd5zco54LfL3IiDDaw3s7Igahu0Rj0= -github.com/bittorrent/go-btfs-config v0.13.0-pre2 h1:sneJ4a5bA15ST9WRUR4G+1FuGUVmszSEuihCeKzeyNk= -github.com/bittorrent/go-btfs-config v0.13.0-pre2/go.mod h1:DNaHVC9wU84KLKoC4HkvdoFJKVZ7TF530qzfYu30fCI= +github.com/bittorrent/go-btfs-config v0.13.2 h1:wTS5kz3w4xRLfggubPq+uvveKA9WFBYFj8mCIba0oHQ= +github.com/bittorrent/go-btfs-config v0.13.2/go.mod h1:DNaHVC9wU84KLKoC4HkvdoFJKVZ7TF530qzfYu30fCI= github.com/bittorrent/go-btfs-files v0.3.0/go.mod h1:ylMf73m6oK94hL7VPblY1ZZpePsr6XbPV4BaNUwGZR0= github.com/bittorrent/go-btfs-files v0.3.1 h1:esq3j+6FtZ+SlaxKjVtiYgvXk/SWUiTcv0Q1MeJoPnQ= github.com/bittorrent/go-btfs-files v0.3.1/go.mod h1:ylMf73m6oK94hL7VPblY1ZZpePsr6XbPV4BaNUwGZR0= diff --git a/version.go b/version.go index 5c6fdcdbd..f4ce182ce 100644 --- a/version.go +++ b/version.go @@ -4,7 +4,7 @@ package btfs var CurrentCommit string // CurrentVersionNumber is the current application's version literal -const CurrentVersionNumber = "2.3.5" +const CurrentVersionNumber = "3.0.0" const ApiVersion = "/go-btfs/" + CurrentVersionNumber + "/"