Slow Sign #42

pascaldekloe · 2020-05-06T18:57:55Z

No benchmarks so I did a small comparison.

goos: darwin
goarch: amd64
pkg: github.com/pascaldekloe/jwt
BenchmarkCristalHQ/sign-HS256-4         	  543394	      2212 ns/op
BenchmarkCristalHQ/check-HS256-4        	  271428	      4300 ns/op
BenchmarkHMAC/sign-HS256-4              	  589506	      2034 ns/op	       123 B/token
BenchmarkHMAC/check-HS256-4             	  274357	      4277 ns/op
PASS
ok  	github.com/pascaldekloe/jwt	5.242s

func BenchmarkCristalHQ(b *testing.B) {
        // 512-bit key
        secret := make([]byte, 64)

        signer, err := cjwt.NewHS256(secret)
        if err != nil {
                b.Fatal(err)
        }
        builder := cjwt.NewTokenBuilder(signer)

        claims := &cjwt.StandardClaims{
                Issuer: benchClaims.Issuer,
                IssuedAt: cjwt.Timestamp(*benchClaims.Issued),
        }

        var token []byte
        b.Run("sign-HS256", func(b *testing.B) {
                for i := 0; i < b.N; i++ {
                        obj, err := builder.Build(claims)
                        if err != nil {
                                b.Fatal(err)
                        }
                        token = obj.Raw()
                }
        })
        b.Run("check-HS256", func(b *testing.B) {
                for i := 0; i < b.N; i++ {
                        obj, err := cjwt.ParseAndVerify(token, signer)
                        if err != nil {
                                b.Fatal(err)
                        }
                        err = json.Unmarshal(obj.RawClaims(), new(map[string]interface{}))
                        if err != nil {
                                b.Fatal(err)
                        }
                }
        })
}

The text was updated successfully, but these errors were encountered:

cristaloleg · 2020-05-06T19:13:16Z

Hi @pascaldekloe thanks for the benchs, I haven't added them to the repo yet. I've a question, am I get it right: 2212 ns/op vs 2034 ns/op ?

pascaldekloe · 2020-05-06T20:09:04Z

Yes that is correct. A 10 % slowdown in comparison.

quasilyte · 2020-05-06T20:56:00Z

I suggest using the benchstat when doing benchmarks. This way, you can share benchmark results in a more intuitive way. Plus it tries to determine whether results are stable or it's a noise.

cristaloleg · 2020-07-30T15:05:05Z

Fixed in #46 #47 #48 #53

pascaldekloe · 2020-07-30T21:16:43Z

The benchmarks are a start. Sign is still slow in comparison. Maybe remove the fast claim from the README otherwise?

cristaloleg · 2020-07-30T21:19:13Z

Will revisit benches during the weekend 👍

cristaloleg · 2020-07-31T11:48:37Z

@pascaldekloe have run benchmarks, please tell me where it's 'still slow' ?

https://github.com/cristaloleg/benches/tree/master/jwt

pascaldekloe · 2020-08-02T07:17:49Z

Didn't look. Just shared the observation here. Please reopen the issue.

cristaloleg · 2020-08-02T07:21:18Z

Please provide where it's slow first.

pascaldekloe · 2020-08-04T16:43:23Z

% go test -bench ./HS256 -count 10 -run none > bench.txt
% benchstat bench.txt
name                     time/op
HMAC/sign-HS256-4        2.04µs ± 1%
HMAC/check-HS256-4       4.24µs ± 0%
CristalHQ/sign-HS256-4   2.20µs ± 1%
CristalHQ/check-HS256-4  4.27µs ± 1%

No improvement: v3.0.2 is still about 10% slower.

cristaloleg · 2020-08-04T16:55:59Z

Nice. Please share a benchmark code/repo, your OS and environment, machine setup and Go version.

pascaldekloe · 2020-08-04T16:59:58Z

The code & repo is in the description. I'm using a Mac Pro from 2013 and Go version 1.14.

FZambia · 2020-08-04T17:22:17Z

Just run code from https://github.com/cristaloleg/benches/tree/master/jwt and I can confirm that cristalhq/jwt/v3 performs a bit better on my machine (MacBook Pro (13-inch, Late 2016)):

$ time ./jwt-bench -test.v -test.benchmem -test.bench HMAC/sign-HS256 -test.count 5 -test.run ^$
goos: darwin
goarch: amd64
pkg: github.com/cristaloleg/benches/jwt
Benchmark_One_HMAC
Benchmark_One_HMAC/sign-HS256
Benchmark_One_HMAC/sign-HS256-4         	  481699	      2447 ns/op	       123 B/token	     656 B/op	       7 allocs/op
Benchmark_One_HMAC/sign-HS256-4         	  504902	      2813 ns/op	       123 B/token	     656 B/op	       7 allocs/op
Benchmark_One_HMAC/sign-HS256-4         	  498763	      2438 ns/op	       123 B/token	     656 B/op	       7 allocs/op
Benchmark_One_HMAC/sign-HS256-4         	  514028	      2435 ns/op	       123 B/token	     656 B/op	       7 allocs/op
Benchmark_One_HMAC/sign-HS256-4         	  491911	      2426 ns/op	       123 B/token	     656 B/op	       7 allocs/op
Benchmark_Two_HMAC
Benchmark_Two_HMAC/sign-HS256
Benchmark_Two_HMAC/sign-HS256-4         	  529950	      2314 ns/op	       129 B/token	     400 B/op	       6 allocs/op
Benchmark_Two_HMAC/sign-HS256-4         	  525048	      2327 ns/op	       129 B/token	     400 B/op	       6 allocs/op
Benchmark_Two_HMAC/sign-HS256-4         	  532951	      2320 ns/op	       129 B/token	     400 B/op	       6 allocs/op
Benchmark_Two_HMAC/sign-HS256-4         	  517843	      2301 ns/op	       129 B/token	     400 B/op	       6 allocs/op
Benchmark_Two_HMAC/sign-HS256-4         	  517129	      2315 ns/op	       129 B/token	     400 B/op	       6 allocs/op
PASS

real	0m13.551s
user	0m13.992s
sys	0m0.363s

Similar results on 40$ per month Linux droplet on Digitalocean (Linux ubuntu-s-4vcpu-8gb-ams3-01 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux, go 1.14.6):

root@ubuntu-s-4vcpu-8gb-ams3-01:~/benches/jwt# time ./jwt-bench -test.v -test.benchmem -test.bench HMAC/sign-HS256 -test.count 5 -test.run ^$
goos: linux
goarch: amd64
pkg: github.com/cristaloleg/benches/jwt
Benchmark_One_HMAC
Benchmark_One_HMAC/sign-HS256
Benchmark_One_HMAC/sign-HS256-4         	  486711	      2623 ns/op	       123 B/token	     656 B/op	       7 allocs/op
Benchmark_One_HMAC/sign-HS256-4         	  452798	      2553 ns/op	       123 B/token	     656 B/op	       7 allocs/op
Benchmark_One_HMAC/sign-HS256-4         	  491455	      2544 ns/op	       123 B/token	     656 B/op	       7 allocs/op
Benchmark_One_HMAC/sign-HS256-4         	  501620	      2541 ns/op	       123 B/token	     656 B/op	       7 allocs/op
Benchmark_One_HMAC/sign-HS256-4         	  482263	      2590 ns/op	       123 B/token	     656 B/op	       7 allocs/op
Benchmark_Two_HMAC
Benchmark_Two_HMAC/sign-HS256
Benchmark_Two_HMAC/sign-HS256-4         	  517096	      2429 ns/op	       129 B/token	     400 B/op	       6 allocs/op
Benchmark_Two_HMAC/sign-HS256-4         	  499501	      2372 ns/op	       129 B/token	     400 B/op	       6 allocs/op
Benchmark_Two_HMAC/sign-HS256-4         	  513686	      2456 ns/op	       129 B/token	     400 B/op	       6 allocs/op
Benchmark_Two_HMAC/sign-HS256-4         	  514288	      2589 ns/op	       129 B/token	     400 B/op	       6 allocs/op
Benchmark_Two_HMAC/sign-HS256-4         	  512469	      2429 ns/op	       129 B/token	     400 B/op	       6 allocs/op
PASS

real	0m12.796s
user	0m13.277s
sys	0m0.290s

pascaldekloe · 2020-08-05T00:55:43Z

Check Errors & Parse JSON

diff --git a/jwt/bench_test.go b/jwt/bench_test.go
index ef44b9f..1d90582 100644
--- a/jwt/bench_test.go
+++ b/jwt/bench_test.go
@@ -5,6 +5,7 @@ import (
        "crypto/ed25519"
        "crypto/rsa"
        "crypto/x509"
+       "encoding/json"
        "encoding/pem"
        "fmt"
        "testing"
@@ -181,7 +182,10 @@ func Benchmark_Two_ECDSA(b *testing.B) {
        }
 
        for _, test := range tests {
-               signer, _ := jwt2.NewSignerES(test.alg, test.key)
+               signer, err := jwt2.NewSignerES(test.alg, test.key)
+               if err != nil {
+                       b.Fatal(err)
+               }
                bui := jwt2.NewBuilder(signer)
                b.Run("sign-"+test.alg.String(), func(b *testing.B) {
                        var tokenLen int
@@ -197,17 +201,27 @@ func Benchmark_Two_ECDSA(b *testing.B) {
        }
 
        for _, test := range tests {
-               signer, _ := jwt2.NewSignerES(test.alg, test.key)
+               signer, err := jwt2.NewSignerES(test.alg, test.key)
+               if err != nil {
+                       b.Fatal(err)
+               }
                bui := jwt2.NewBuilder(signer)
-               token, err := bui.Build(mybenchClaims)
+               token, err := bui.BuildBytes(mybenchClaims)
                if err != nil {
                        b.Fatal(err)
                }
 
-               verifier, _ := jwt2.NewVerifierES(test.alg, &test.key.PublicKey)
+               verifier, err := jwt2.NewVerifierES(test.alg, &test.key.PublicKey)
+               if err != nil {
+                       b.Fatal(err)
+               }
                b.Run("check-"+test.alg.String(), func(b *testing.B) {
                        for i := 0; i < b.N; i++ {
-                               err := verifier.Verify(token.Payload(), token.Signature())
+                               obj, err := jwt2.ParseAndVerify(token, verifier)
+                               if err != nil {
+                                       b.Fatal(err)
+                               }
+                               err = json.Unmarshal(obj.RawClaims(), new(map[string]interface{}))
                                if err != nil {
                                        b.Fatal(err)
                                }
@@ -217,7 +231,10 @@ func Benchmark_Two_ECDSA(b *testing.B) {
 }
 
 func Benchmark_Two_EdDSA(b *testing.B) {
-       signer, _ := jwt2.NewSignerEdDSA(testKeyEd25519Private)
+       signer, err := jwt2.NewSignerEdDSA(testKeyEd25519Private)
+       if err != nil {
+               b.Fatal(err)
+       }
        bui := jwt2.NewBuilder(signer)
        b.Run("sign-"+jwt2.EdDSA.String(), func(b *testing.B) {
                var tokenLen int
@@ -231,15 +248,22 @@ func Benchmark_Two_EdDSA(b *testing.B) {
                b.ReportMetric(float64(tokenLen)/float64(b.N), "B/token")
        })
 
-       token, err := jwt2.NewBuilder(signer).Build(mybenchClaims)
+       token, err := bui.BuildBytes(mybenchClaims)
        if err != nil {
                b.Fatal(err)
        }
-       verifier, _ := jwt2.NewVerifierEdDSA(testKeyEd25519Public)
 
+       verifier, err := jwt2.NewVerifierEdDSA(testKeyEd25519Public)
+       if err != nil {
+               b.Fatal(err)
+       }
        b.Run("check-"+jwt2.EdDSA.String(), func(b *testing.B) {
                for i := 0; i < b.N; i++ {
-                       err := verifier.Verify(token.Payload(), token.Signature())
+                       obj, err := jwt2.ParseAndVerify(token, verifier)
+                       if err != nil {
+                               b.Fatal(err)
+                       }
+                       err = json.Unmarshal(obj.RawClaims(), new(map[string]interface{}))
                        if err != nil {
                                b.Fatal(err)
                        }
@@ -253,7 +277,10 @@ func Benchmark_Two_HMAC(b *testing.B) {
        algs := []jwt2.Algorithm{jwt2.HS256, jwt2.HS384, jwt2.HS512}
 
        for _, alg := range algs {
-               signer, _ := jwt2.NewSignerHS(alg, secret)
+               signer, err := jwt2.NewSignerHS(alg, secret)
+               if err != nil {
+                       b.Fatal(err)
+               }
                bui := jwt2.NewBuilder(signer)
                b.Run("sign-"+alg.String(), func(b *testing.B) {
                        var tokenLen int
@@ -269,16 +296,26 @@ func Benchmark_Two_HMAC(b *testing.B) {
        }
 
        for _, alg := range algs {
-               signer, _ := jwt2.NewSignerHS(alg, secret)
-               token, err := jwt2.NewBuilder(signer).Build(mybenchClaims)
+               signer, err := jwt2.NewSignerHS(alg, secret)
+               if err != nil {
+                       b.Fatal(err)
+               }
+               token, err := jwt2.NewBuilder(signer).BuildBytes(mybenchClaims)
                if err != nil {
                        b.Fatal(err)
                }
 
-               verifier, _ := jwt2.NewVerifierHS(alg, secret)
+               verifier, err := jwt2.NewVerifierHS(alg, secret)
+               if err != nil {
+                       b.Fatal(err)
+               }
                b.Run("check-"+alg.String(), func(b *testing.B) {
                        for i := 0; i < b.N; i++ {
-                               err := verifier.Verify(token.Payload(), token.Signature())
+                               obj, err := jwt2.ParseAndVerify(token, verifier)
+                               if err != nil {
+                                       b.Fatal(err)
+                               }
+                               err = json.Unmarshal(obj.RawClaims(), new(map[string]interface{}))
                                if err != nil {
                                        b.Fatal(err)
                                }
@@ -307,16 +344,26 @@ func Benchmark_Two_RSA(b *testing.B) {
        }
 
        for _, key := range keys {
-               signer, _ := jwt2.NewSignerRS(jwt2.RS384, key)
-               token, err := jwt2.NewBuilder(signer).Build(mybenchClaims)
+               signer, err := jwt2.NewSignerRS(jwt2.RS384, key)
+               if err != nil {
+                       b.Fatal(err)
+               }
+               token, err := jwt2.NewBuilder(signer).BuildBytes(mybenchClaims)
                if err != nil {
                        b.Fatal(err)
                }
 
-               verifier, _ := jwt2.NewVerifierRS(jwt2.RS384, &key.PublicKey)
+               verifier, err := jwt2.NewVerifierRS(jwt2.RS384, &key.PublicKey)
+               if err != nil {
+                       b.Fatal(err)
+               }
                b.Run(fmt.Sprintf("check-%d-bit", key.Size()*8), func(b *testing.B) {
                        for i := 0; i < b.N; i++ {
-                               err := verifier.Verify(token.Payload(), token.Signature())
+                               obj, err := jwt2.ParseAndVerify(token, verifier)
+                               if err != nil {
+                                       b.Fatal(err)
+                               }
+                               err = json.Unmarshal(obj.RawClaims(), new(map[string]interface{}))
                                if err != nil {
                                        b.Fatal(err)
                                }

ECDSA is still broken on v3.0.2:

--- FAIL: Benchmark_Two_ECDSA/check-ES256
    bench_test.go:222: jwt: token format is not valid
--- FAIL: Benchmark_Two_ECDSA/check-ES384
    bench_test.go:222: jwt: token format is not valid

The HMAC sync pool really pays of here. I'll try to borrow that idea. 😏

cristaloleg · 2020-08-05T04:53:38Z

lol

pascaldekloe · 2020-08-06T21:36:26Z

Done with pascaldekloe/jwt@9f0fa18.

Do you need a bug report for the broken ECDSA @cristaloleg?

pascaldekloe · 2020-08-07T21:20:23Z

Created a pull request cristaloleg/benches#1 @cristaloleg. This time with a regular intel [not Xeon] and it's still significantly slower.

cristaloleg · 2020-08-19T13:26:22Z

Forgot to mention: problem with sign in ES is fixed in v3.0.3 https://github.com/cristalhq/jwt/releases/tag/v3.0.3

Thank you for the info.

cristaloleg self-assigned this May 6, 2020

cristaloleg closed this as completed Jul 30, 2020

cristaloleg mentioned this issue Aug 12, 2020

Check errors, parse claims consistently and compare reuse cristaloleg/benches#1

Closed

cristaloleg added the feature New feature or request label Feb 4, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Slow Sign #42

Slow Sign #42

pascaldekloe commented May 6, 2020

cristaloleg commented May 6, 2020

pascaldekloe commented May 6, 2020

quasilyte commented May 6, 2020

cristaloleg commented Jul 30, 2020

pascaldekloe commented Jul 30, 2020

cristaloleg commented Jul 30, 2020 •

edited

Loading

cristaloleg commented Jul 31, 2020

pascaldekloe commented Aug 2, 2020

cristaloleg commented Aug 2, 2020

pascaldekloe commented Aug 4, 2020

cristaloleg commented Aug 4, 2020

pascaldekloe commented Aug 4, 2020

FZambia commented Aug 4, 2020 •

edited

Loading

pascaldekloe commented Aug 5, 2020

cristaloleg commented Aug 5, 2020

pascaldekloe commented Aug 6, 2020

pascaldekloe commented Aug 7, 2020

cristaloleg commented Aug 19, 2020 •

edited

Loading

Slow Sign #42

Slow Sign #42

Comments

pascaldekloe commented May 6, 2020

cristaloleg commented May 6, 2020

pascaldekloe commented May 6, 2020

quasilyte commented May 6, 2020

cristaloleg commented Jul 30, 2020

pascaldekloe commented Jul 30, 2020

cristaloleg commented Jul 30, 2020 • edited Loading

cristaloleg commented Jul 31, 2020

pascaldekloe commented Aug 2, 2020

cristaloleg commented Aug 2, 2020

pascaldekloe commented Aug 4, 2020

cristaloleg commented Aug 4, 2020

pascaldekloe commented Aug 4, 2020

FZambia commented Aug 4, 2020 • edited Loading

pascaldekloe commented Aug 5, 2020

cristaloleg commented Aug 5, 2020

pascaldekloe commented Aug 6, 2020

pascaldekloe commented Aug 7, 2020

cristaloleg commented Aug 19, 2020 • edited Loading

cristaloleg commented Jul 30, 2020 •

edited

Loading

FZambia commented Aug 4, 2020 •

edited

Loading

cristaloleg commented Aug 19, 2020 •

edited

Loading