Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

split out processing into a separate thread #1

Closed
wants to merge 69 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
13206e4
Let clients subscribe to the ledger over TCP
garious Mar 27, 2018
7c96810
Drop support for random access to the ledger
garious Mar 27, 2018
c8ddc68
Rename project: silk -> solana
garious Mar 27, 2018
26b19dd
Rename project: silk -> solana
garious Mar 27, 2018
116166f
Rename project: silk -> solana
garious Mar 27, 2018
15c093c
typo
garious Mar 27, 2018
7d3d4b9
nit
garious Mar 27, 2018
7656b55
nit
garious Mar 27, 2018
bf902ef
Ignore accountant_stub test
garious Mar 28, 2018
8642a41
See if Travis will tolerate executing some of the test
garious Mar 28, 2018
27f2901
Merge pull request #83 from garious/tcp-client
garious Mar 28, 2018
849bced
Fix up client demo
garious Mar 28, 2018
9db42c1
Merge pull request #86 from garious/tcp-client
garious Mar 28, 2018
98c0a2a
tx confirmed/sec ---> tx processed/sec
garious Mar 28, 2018
04f3071
Merge pull request #87 from garious/tcp-client
garious Mar 28, 2018
0bec360
Revert TCP sync of ledger
garious Mar 29, 2018
f6ca176
Merge pull request #88 from garious/revert-tcp-client
garious Mar 29, 2018
4bc41d8
Fix compiler warning
garious Mar 29, 2018
878ca8c
Add microbenchmark for signature verification
garious Mar 29, 2018
cc2bb29
Merge pull request #89 from garious/sig-verify-bench
garious Mar 29, 2018
3bf225e
Don't require install to run demo
garious Mar 29, 2018
a03d7bf
Missed a couple
garious Mar 29, 2018
132495b
A simple consensus diagram to guide rollback/coalescing
garious Mar 29, 2018
5517910
Add more documentation
garious Mar 29, 2018
89f5b80
Merge pull request #91 from garious/more-docs
garious Mar 29, 2018
1fbb346
Fix compiler warning
garious Mar 29, 2018
232e1bb
Colocate packet dependencies
garious Mar 29, 2018
c59c38e
Refactor for batch verification
garious Mar 29, 2018
22f5985
Do request verification in parallel, and then process the verified re…
garious Mar 29, 2018
f9309b4
Merge pull request #92 from garious/par-req-processing
garious Mar 29, 2018
2f1e585
Better benchmark
garious Mar 29, 2018
b8cf5f9
Fix transaction logging
garious Mar 29, 2018
6a87d89
Merge pull request #93 from garious/par-req-processing
garious Mar 29, 2018
eaec25f
Version bump
garious Mar 29, 2018
ef169a6
94: source doc review
jackson-sandland Mar 30, 2018
9f256f0
94 - snakecase mod names
jackson-sandland Mar 30, 2018
54e4f75
Merge pull request #95 from jackson-sandland/source-documentation-review
garious Mar 30, 2018
e683c34
Version bump
garious Mar 31, 2018
46e8c09
Revoke API access to first_id
garious Apr 2, 2018
3abe305
Move reserve_signatures into accountant
garious Apr 2, 2018
da2b496
Move verify_slice() into a trait
garious Apr 2, 2018
fc54039
Update docs
garious Apr 2, 2018
17de687
Add simpler accountant constructor
garious Apr 2, 2018
d63506f
No longer allow deposits outside the constructor
garious Apr 2, 2018
90cd9bd
Move balance check so that log_* methods are only used to add logging
garious Apr 2, 2018
2b788d0
Move the historian up to accountant_skel
garious Apr 2, 2018
daadae7
Move replaying ledger out of accountant
garious Apr 2, 2018
49708e9
Use last_id instead of seed
garious Apr 2, 2018
84df487
Merge pull request #97 from garious/rollback
garious Apr 2, 2018
6fec8fa
Adding from to the signature is redundant
garious Apr 3, 2018
07aa2e1
Add witness data to entry hash
garious Apr 3, 2018
fe32159
Add a test to ensure witness data continues to be hashed
garious Apr 3, 2018
94eea3a
fmt
garious Apr 3, 2018
95749ed
Merge pull request #100 from garious/rollback
garious Apr 3, 2018
2f80747
Move tests
garious Apr 3, 2018
5c672ad
Merge pull request #101 from garious/rollback
garious Apr 3, 2018
c323bd3
Fix clippy warnings
garious Apr 3, 2018
bc0dde6
Merge pull request #102 from garious/rollback
garious Apr 3, 2018
5ac7df1
Implement window service
aeyakovenko Apr 3, 2018
123d7c6
Merge pull request #99 from aeyakovenko/subscribers
garious Apr 3, 2018
4fd1ebc
In-progress change for cuda verify integration
sakridge Mar 27, 2018
8e83ba8
we have logs
aeyakovenko Apr 5, 2018
9cf7228
enable trace
aeyakovenko Apr 5, 2018
5553f31
trace msgs
aeyakovenko Apr 5, 2018
35c8685
cleanup
aeyakovenko Apr 5, 2018
f10a24e
100m test
aeyakovenko Apr 5, 2018
89d2f22
easy peasy
aeyakovenko Apr 5, 2018
fe49519
update
aeyakovenko Apr 5, 2018
b49b9f3
stash
aeyakovenko Apr 8, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 16 additions & 13 deletions Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,42 +1,42 @@
[package]
name = "silk"
description = "A silky smooth implementation of the Loom architecture"
name = "solana"
description = "High Performance Blockchain"
version = "0.4.0"
documentation = "https://docs.rs/silk"
documentation = "https://docs.rs/solana"
homepage = "http://loomprotocol.com/"
repository = "https://github.com/loomprotocol/silk"
repository = "https://github.com/solana-labs/solana"
authors = [
"Anatoly Yakovenko <aeyakovenko@gmail.com>",
"Greg Fitzgerald <garious@gmail.com>",
"Anatoly Yakovenko <anatoly@solana.co>",
"Greg Fitzgerald <greg@solana.co>",
]
license = "Apache-2.0"

[[bin]]
name = "silk-historian-demo"
name = "solana-historian-demo"
path = "src/bin/historian-demo.rs"

[[bin]]
name = "silk-client-demo"
name = "solana-client-demo"
path = "src/bin/client-demo.rs"

[[bin]]
name = "silk-testnode"
name = "solana-testnode"
path = "src/bin/testnode.rs"

[[bin]]
name = "silk-genesis"
name = "solana-genesis"
path = "src/bin/genesis.rs"

[[bin]]
name = "silk-genesis-demo"
name = "solana-genesis-demo"
path = "src/bin/genesis-demo.rs"

[[bin]]
name = "silk-mint"
name = "solana-mint"
path = "src/bin/mint.rs"

[badges]
codecov = { repository = "loomprotocol/silk", branch = "master", service = "github" }
codecov = { repository = "solana-labs/solana", branch = "master", service = "github" }

[features]
unstable = []
Expand All @@ -54,4 +54,7 @@ untrusted = "0.5.1"
bincode = "1.0.0"
chrono = { version = "0.4.0", features = ["serde"] }
log = "^0.4.1"
env_logger = "^0.4.1"
matches = "^0.1.6"
byteorder = "^1.2.1"
libc = "^0.2.1"
46 changes: 46 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
export RUST_LOG=solana::gpu=TRACE
#export RUST_BACKTRACE=1

all: htest

htest:wfmt
#cargo test accountant_skel::tests::test_layout -- --nocapture 2>&1 | head -n 30
#cargo test accountant_skel::tests::test_layout -- --nocapture
cargo test accountant_stub -- --nocapture 2>&1 | head -n 30

ci: test bench release clippy ipv6

build:
cargo build 2>&1 | head -n 30

loop:
while true; do fswatch -1 -r src; make; done

test:
cargo test

clippy:
cargo +nightly clippy --features="unstable"

cov:
docker run -it --rm --security-opt seccomp=unconfined --volume "$$PWD:/volume" elmtai/docker-rust-kcov

wfmt:
cargo fmt -- --write-mode=overwrite

release:
cargo build --all-targets --release

node:
cat genesis.log | cargo run --bin silk-testnode > transactions0.log

bench:
cargo +nightly bench --features="unstable" -- --nocapture

ipv6:
cargo test ipv6 --features="ipv6" -- --nocapture

lib:libcuda_verify_ed25519.a
libcuda_verify_ed25519.a:dummy.c
cc -o dummy.o -c dummy.c
ar -cvq libcuda_verify_ed25519.a dummy.o
43 changes: 15 additions & 28 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,25 +1,18 @@
[![Silk crate](https://img.shields.io/crates/v/silk.svg)](https://crates.io/crates/silk)
[![Silk documentation](https://docs.rs/silk/badge.svg)](https://docs.rs/silk)
[![Build Status](https://travis-ci.org/loomprotocol/silk.svg?branch=master)](https://travis-ci.org/loomprotocol/silk)
[![codecov](https://codecov.io/gh/loomprotocol/silk/branch/master/graph/badge.svg)](https://codecov.io/gh/loomprotocol/silk)
[![Solana crate](https://img.shields.io/crates/v/solana.svg)](https://crates.io/crates/solana)
[![Solana documentation](https://docs.rs/solana/badge.svg)](https://docs.rs/solana)
[![Build Status](https://travis-ci.org/solana-labs/solana.svg?branch=master)](https://travis-ci.org/solana-labs/solana)
[![codecov](https://codecov.io/gh/solana-labs/solana/branch/master/graph/badge.svg)](https://codecov.io/gh/solana-labs/solana)

Disclaimer
===

All claims, content, designs, algorithms, estimates, roadmaps, specifications, and performance measurements described in this project are done with the author's best effort. It is up to the reader to check and validate their accuracy and truthfulness. Furthermore nothing in this project constitutes a solicitation for investment.

Silk, a silky smooth implementation of the Loom specification
Solana: High Performance Blockchain
===

Loom&trade; is a new architecture for a high performance blockchain. Its white paper boasts a theoretical
throughput of 710k transactions per second on a 1 gbps network. The specification is implemented
in two git repositories. Research is performed in the loom repository. That work drives the
Loom specification forward. This repository, on the other hand, aims to implement the specification
as-is. We care a great deal about quality, clarity and short learning curve. We avoid the use
of `unsafe` Rust and write tests for *everything*. Optimizations are only added when
corresponding benchmarks are also added that demonstrate real performance boosts. We expect the
feature set here will always be a ways behind the loom repo, but that this is an implementation
you can take to the bank, literally.
Solana&trade; is a new architecture for a high performance blockchain. It aims to support
over 700 thousand transactions per second on a gigabit network.

Running the demo
===
Expand All @@ -31,34 +24,28 @@ $ curl https://sh.rustup.rs -sSf | sh
$ source $HOME/.cargo/env
```

Install the silk executables:

```bash
$ cargo install silk
```

The testnode server is initialized with a ledger from stdin and
generates new ledger entries on stdout. To create the input ledger, we'll need
to create *the mint* and use it to generate a *genesis ledger*. It's done in
two steps because the mint.json file contains a private key that will be
used later in this demo.

```bash
$ echo 500 | silk-mint > mint.json
$ cat mint.json | silk-genesis > genesis.log
$ echo 1000000000 | cargo run --release --bin solana-mint | tee mint.json
$ cat mint.json | cargo run --release --bin solana-genesis | tee genesis.log
```

Now you can start the server:

```bash
$ cat genesis.log | silk-testnode > transactions0.log
$ cat genesis.log | cargo run --release --bin solana-testnode | tee transactions0.log
```

Then, in a separate shell, let's execute some transactions. Note we pass in
the JSON configuration file here, not the genesis ledger.

```bash
$ cat mint.json | silk-client-demo
$ cat mint.json | cargo run --release --bin solana-client-demo
```

Now kill the server with Ctrl-C, and take a look at the ledger. You should
Expand All @@ -74,14 +61,14 @@ Now restart the server from where we left off. Pass it both the genesis ledger,
the transaction ledger.

```bash
$ cat genesis.log transactions0.log | silk-testnode > transactions1.log
$ cat genesis.log transactions0.log | cargo run --release --bin solana-testnode | tee transactions1.log
```

Lastly, run the client demo again, and verify that all funds were spent in the
previous round, and so no additional transactions are added.

```bash
$ cat mint.json | silk-client-demo
$ cat mint.json | cargo run --release --bin solana-client-demo
```

Stop the server again, and verify there are only Tick entries, and no Transaction entries.
Expand All @@ -103,8 +90,8 @@ $ rustup component add rustfmt-preview
Download the source code:

```bash
$ git clone https://github.com/loomprotocol/silk.git
$ cd silk
$ git clone https://github.com/solana-labs/solana.git
$ cd solana
```

Testing
Expand Down
8 changes: 8 additions & 0 deletions build.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
fn main() {
println!("cargo:rustc-link-search=native=.");
println!("cargo:rustc-link-lib=static=cuda_verify_ed25519");
println!("cargo:rustc-link-search=native=/usr/local/cuda/lib64");
println!("cargo:rustc-link-lib=dylib=cudart");
println!("cargo:rustc-link-lib=dylib=cuda");
println!("cargo:rustc-link-lib=dylib=cudadevrt");
}
15 changes: 15 additions & 0 deletions doc/consensus.msc
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
msc {
client,leader,verifier_a,verifier_b,verifier_c;

client=>leader [ label = "SUBMIT" ] ;
leader=>client [ label = "CONFIRMED" ] ;
leader=>verifier_a [ label = "CONFIRMED" ] ;
leader=>verifier_b [ label = "CONFIRMED" ] ;
leader=>verifier_c [ label = "CONFIRMED" ] ;
verifier_a=>leader [ label = "VERIFIED" ] ;
verifier_b=>leader [ label = "VERIFIED" ] ;
leader=>client [ label = "FINALIZED" ] ;
leader=>verifier_a [ label = "FINALIZED" ] ;
leader=>verifier_b [ label = "FINALIZED" ] ;
leader=>verifier_c [ label = "FINALIZED" ] ;
}
12 changes: 6 additions & 6 deletions doc/historian.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@ with by verifying each entry's hash can be generated from the hash in the previo
![historian](https://user-images.githubusercontent.com/55449/36950845-459bdb58-1fb9-11e8-850e-894586f3729b.png)

```rust
extern crate silk;
extern crate solana;

use silk::historian::Historian;
use silk::ledger::{verify_slice, Entry, Hash};
use silk::event::{generate_keypair, get_pubkey, sign_claim_data, Event};
use solana::historian::Historian;
use solana::ledger::{Block, Entry, Hash};
use solana::event::{generate_keypair, get_pubkey, sign_claim_data, Event};
use std::thread::sleep;
use std::time::Duration;
use std::sync::mpsc::SendError;
Expand All @@ -38,7 +38,7 @@ fn main() {
}
// Proof-of-History: Verify the historian learned about the events
// in the same order they appear in the vector.
assert!(verify_slice(&entries, &seed));
assert!(entries[..].verify(&seed));
}
```

Expand All @@ -56,7 +56,7 @@ Proof-of-History
Take note of the last line:

```rust
assert!(verify_slice(&entries, &seed));
assert!(entries[..].verify(&seed));
```

[It's a proof!](https://en.wikipedia.org/wiki/Curry–Howard_correspondence) For each entry returned by the
Expand Down
2 changes: 1 addition & 1 deletion doc/historian.msc
Original file line number Diff line number Diff line change
Expand Up @@ -14,5 +14,5 @@ msc {
recorder=>historian [ label = "e2 = Entry{id: h6, n: 3, event: Tick}" ] ;
client=>historian [ label = "collect()" ] ;
historian=>client [ label = "entries = [e0, e1, e2]" ] ;
client=>client [ label = "verify_slice(entries, h0)" ] ;
client=>client [ label = "entries.verify(h0)" ] ;
}
51 changes: 51 additions & 0 deletions dummy.c
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
#include <stdint.h>
#include <assert.h>
#include <stdio.h>

#define PACKET_SIZE 288
#define PACKET_DATA_SIZE 256
union Packet {
char data[PACKET_DATA_SIZE];
char total[PACKET_SIZE];
};

struct Elems {
union Packet *packet;
uint32_t len;
};

int ed25519_verify_many(
const struct Elems *vecs,
uint32_t num,
uint32_t message_size,
uint32_t public_key_offset,
uint32_t signature_offset,
uint32_t signed_message_offset,
uint32_t signed_message_len_offset,
uint8_t *out
) {
int i, p = 0;
assert(num > 0);
for(i = 0; i < num; ++i) {
int j;
assert(vecs[i].len > 0);
assert(message_size == PACKET_SIZE);
assert(signed_message_len_offset == PACKET_DATA_SIZE);
for(j = 0; j < vecs[i].len; ++j) {
uint32_t *len = (uint32_t*)&vecs[i].packet[j].total[signed_message_len_offset];
assert(*len <= PACKET_DATA_SIZE);
p += 1;
if(public_key_offset > *len - 32) {
continue;
}
if(signature_offset > *len - 64) {
continue;
}
if(signed_message_offset > *len) {
continue;
}
out[p - 1] = 1;
}
}
return 0;
}
Loading