Skip to content
This repository has been archived by the owner on Oct 3, 2024. It is now read-only.

Commit

Permalink
update perf result
Browse files Browse the repository at this point in the history
  • Loading branch information
@nkrusch
nkrusch committed Dec 21, 2023
1 parent bc3240d commit 8e49bfd
Show file tree
Hide file tree
Showing 26 changed files with 359 additions and 341 deletions.
Binary file modified ref_result/__bar_acc_ref_result_perf_ubuntu.pdf
View file
Binary file not shown.
12 changes: 6 additions & 6 deletions ref_result/__cls_acc_ref_result_perf_ubuntu.txt
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
classifier exp-name accuracy±σ
Neural Network PT-1-03 91.5 ± 0.4
Neural Network PT-1-06 91.6 ± 0.0
Neural Network PT-1-12 91.6 ± 0.2
Neural Network PT-2-03 91.8 ± 0.2
Neural Network PT-2-06 91.7 ± 0.1
Neural Network PT-2-12 91.6 ± 0.1
Neural Network PT-1-03 91.6 ± 0.3
Neural Network PT-1-06 91.9 ± 0.3
Neural Network PT-1-12 91.8 ± 0.2
Neural Network PT-2-03 91.4 ± 0.1
Neural Network PT-2-06 91.8 ± 0.2
Neural Network PT-2-12 91.7 ± 0.2
10 changes: 5 additions & 5 deletions ref_result/__config_ref_result_attacks.txt
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
exp-name records cls ft *val #C ⊥ P ft/P
IoT-23 10000 2 18 4 18 10 8 3.9
LCLD 20000 2 47 20 29 19 10 3.2
UNSW-NB15 10000 2 32 16 6 0 6 4.2
URL 11430 2 63 35 14 0 14 3.3
exp-name records cls #ft *-val #C #⊥ #P ft/P
IoT-23 10000 2 18 4 18 10 8 3.9
LCLD 20000 2 47 20 29 19 10 3.2
UNSW-NB15 10000 2 32 16 6 0 6 4.2
URL 11430 2 63 35 14 0 14 3.3
10 changes: 5 additions & 5 deletions ref_result/__config_ref_result_original.txt
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
exp-name records cls ft *val #C ⊥ P ft/P
IoT-23 10000 2 18 4 18 10 8 3.9
LCLD 20000 2 47 20 29 19 10 3.2
UNSW-NB15 10000 2 32 16 6 0 6 4.2
URL 11430 2 63 35 14 0 14 3.3
exp-name records cls #ft *-val #C #⊥ #P ft/P
IoT-23 10000 2 18 4 18 10 8 3.9
LCLD 20000 2 47 20 29 19 10 3.2
UNSW-NB15 10000 2 32 16 6 0 6 4.2
URL 11430 2 63 35 14 0 14 3.3
14 changes: 7 additions & 7 deletions ref_result/__config_ref_result_perf_ubuntu.txt
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
exp-name records cls ft *val #C ⊥ P ft/P
PT-1-03 10000 2 32 27 3 0 3 2.0
PT-1-06 10000 2 32 19 6 0 6 2.3
PT-1-12 10000 2 32 15 12 0 12 2.5
PT-2-03 10000 2 32 29 3 0 3 1.0
PT-2-06 10000 2 32 26 6 0 6 1.0
PT-2-12 10000 2 32 20 12 0 12 1.0
exp-name records cls #ft *-val #C #⊥ #P ft/P
PT-1-03 10000 2 32 27 3 0 3 2.0
PT-1-06 10000 2 32 19 6 0 6 2.3
PT-1-12 10000 2 32 15 12 0 12 2.5
PT-2-03 10000 2 32 29 3 0 3 1.0
PT-2-06 10000 2 32 26 6 0 6 1.0
PT-2-12 10000 2 32 20 12 0 12 1.0
10 changes: 5 additions & 5 deletions ref_result/__config_ref_result_reset.txt
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
exp-name records cls ft *val #C ⊥ P ft/P
IoT-23 10000 2 18 4 18 10 8 3.9
LCLD 20000 2 47 20 29 19 10 3.2
UNSW-NB15 10000 2 32 16 6 0 6 4.2
URL 11430 2 63 35 14 0 14 3.3
exp-name records cls #ft *-val #C #⊥ #P ft/P
IoT-23 10000 2 18 4 18 10 8 3.9
LCLD 20000 2 47 20 29 19 10 3.2
UNSW-NB15 10000 2 32 16 6 0 6 4.2
URL 11430 2 63 35 14 0 14 3.3
32 changes: 16 additions & 16 deletions ref_result/__evasion_ref_result_perf_ubuntu.txt
View file
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
classifier exp-name attack accuracy evades valid
Neural Network PT-1-03 PGD 91.0 90.4 80.7
Neural Network PT-1-03 VPGD 91.7 91.7 91.7
Neural Network PT-1-03 CPGD 91.7 71.2 43.7
Neural Network PT-1-06 PGD 91.7 90.8 71.4
Neural Network PT-1-06 VPGD 91.6 91.6 91.6
Neural Network PT-1-06 CPGD 91.6 71.3 29.6
Neural Network PT-1-12 PGD 91.3 90.5 74.5
Neural Network PT-1-12 CPGD 91.7 61.8 3.9
Neural Network PT-1-03 PGD 91.9 91.5 80.4
Neural Network PT-1-03 CPGD 91.3 66.5 36.4
Neural Network PT-1-03 VPGD 91.5 91.5 91.5
Neural Network PT-1-06 PGD 91.5 90.3 70.9
Neural Network PT-1-06 VPGD 92.0 92.0 92.0
Neural Network PT-1-06 CPGD 92.1 62.6 31.1
Neural Network PT-1-12 PGD 92.0 90.9 67.9
Neural Network PT-1-12 VPGD 91.7 91.7 91.7
Neural Network PT-2-03 PGD 91.6 89.2 89.2
Neural Network PT-2-03 CPGD 91.9 70.9 0.0
Neural Network PT-2-03 VPGD 91.8 91.8 91.8
Neural Network PT-2-06 PGD 91.5 90.2 90.2
Neural Network PT-2-06 CPGD 91.8 69.3 0.0
Neural Network PT-2-06 VPGD 91.7 91.7 91.7
Neural Network PT-2-12 PGD 91.6 91.2 91.2
Neural Network PT-1-12 CPGD 91.6 62.3 1.9
Neural Network PT-2-03 PGD 91.3 90.1 90.1
Neural Network PT-2-03 CPGD 91.4 63.0 0.1
Neural Network PT-2-03 VPGD 91.4 91.4 91.4
Neural Network PT-2-06 PGD 91.6 90.9 90.9
Neural Network PT-2-06 CPGD 91.6 65.3 0.0
Neural Network PT-2-06 VPGD 92.0 92.0 92.0
Neural Network PT-2-12 PGD 91.8 90.9 90.9
Neural Network PT-2-12 CPGD 91.8 64.7 1.3
Neural Network PT-2-12 VPGD 91.5 91.5 91.5
Neural Network PT-2-12 CPGD 91.7 78.8 1.2
36 changes: 18 additions & 18 deletions ref_result/__time_ref_result_perf_ubuntu.txt
View file
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
classifier exp-name attack t0 t(s) t-ratio
Neural Network PT-1-03 PGD 57 57 1.00
Neural Network PT-1-03 VPGD 57 86 1.51
Neural Network PT-1-03 CPGD 57 113 1.99
Neural Network PT-1-06 PGD 55 55 1.00
Neural Network PT-1-06 VPGD 55 96 1.73
Neural Network PT-1-06 CPGD 55 153 2.77
Neural Network PT-1-12 PGD 49 49 1.00
Neural Network PT-1-12 CPGD 49 196 4.02
Neural Network PT-1-12 VPGD 49 130 2.67
Neural Network PT-2-03 PGD 52 52 1.00
Neural Network PT-2-03 CPGD 52 125 2.40
Neural Network PT-2-03 VPGD 52 82 1.57
Neural Network PT-2-06 PGD 57 57 1.00
Neural Network PT-2-06 CPGD 57 145 2.52
Neural Network PT-2-06 VPGD 57 98 1.71
Neural Network PT-2-12 PGD 53 53 1.00
Neural Network PT-2-12 VPGD 53 131 2.46
Neural Network PT-2-12 CPGD 53 190 3.56
Neural Network PT-1-03 PGD 29 29 1.00
Neural Network PT-1-03 CPGD 29 65 2.21
Neural Network PT-1-03 VPGD 29 42 1.41
Neural Network PT-1-06 PGD 30 30 1.00
Neural Network PT-1-06 VPGD 30 54 1.81
Neural Network PT-1-06 CPGD 30 87 2.92
Neural Network PT-1-12 PGD 30 30 1.00
Neural Network PT-1-12 VPGD 30 80 2.71
Neural Network PT-1-12 CPGD 30 128 4.32
Neural Network PT-2-03 PGD 25 25 1.00
Neural Network PT-2-03 CPGD 25 70 2.79
Neural Network PT-2-03 VPGD 25 43 1.70
Neural Network PT-2-06 PGD 26 26 1.00
Neural Network PT-2-06 CPGD 26 100 3.87
Neural Network PT-2-06 VPGD 26 54 2.10
Neural Network PT-2-12 PGD 26 26 1.00
Neural Network PT-2-12 CPGD 26 129 5.06
Neural Network PT-2-12 VPGD 26 82 3.20
31 changes: 16 additions & 15 deletions ...erf/ubuntu/PT-1-03_dnn_cpgd_auto_668.yaml → ...erf/ubuntu/PT-1-03_dnn_cpgd_auto_258.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -50,21 +50,21 @@ experiment:
stcpb: [0, 4294729542]
swin: [0, 255]
trans_depth: [0, 2]
capture_utc: 1696821859673103993
capture_utc: 1703120475262057818
class_distribution: {0: 5000, 1: 5000}
config: config/prf1_1.yaml
dataset: data/unsw_nb15.csv
description: Constraint performance test with 3 correlated constraints
duration_sec: 566.1
end: 1696821858668222638
duration_sec: 467.6
end: 1703120474258052186
k_folds: 5
n_attributes: 33
n_classes: 2
n_records: 10000
name: PT-1-03
start: 1696821292555850483
start: 1703120006691911009
system:
cpu_current_frequency: 2200.196
cpu_current_frequency: 2249.998
cpu_max_frequency: 0.0
cpu_min_frequency: 0.0
cpu_physical_cores: 4
Expand All @@ -77,19 +77,20 @@ experiment:
release: 5.15.0-1042-gcp
system: Linux
version: '#50~20.04.1-Ubuntu SMP Mon Sep 11 03:30:57 UTC 2023'
virtual_mem_available: 28.66GB
virtual_mem_percentage: 8.6
virtual_mem_available: 30.41GB
virtual_mem_percentage: 3.0
virtual_mem_total_size: 31.34GB
virtual_mem_used: 2.29GB
virtual_mem_used: 555.92MB
folds:
accuracy: [0.916, 0.9135, 0.914, 0.92, 0.921]
f_score: [0.917728, 0.917501, 0.908511, 0.919759, 0.919797]
n_evasions: [1667, 865, 1059, 1689, 1841]
accuracy: [0.9175, 0.924, 0.914, 0.907, 0.905]
dur: [65095165314, 64968863300, 64771840147, 64987943737, 65250702167]
f_score: [0.918114, 0.924453, 0.909759, 0.908913, 0.898721]
n_evasions: [1053, 1225, 1827, 1080, 1463]
n_records: [2000, 2000, 2000, 2000, 2000]
n_valid: [1113, 1116, 1132, 1230, 1109]
n_valid_evades: [866, 672, 820, 1013, 1003]
precision: [0.910593, 0.91358, 0.927253, 0.921608, 0.919797]
recall: [0.924975, 0.921456, 0.890511, 0.917918, 0.919797]
n_valid: [1090, 1104, 1136, 1109, 1154]
n_valid_evades: [816, 703, 1011, 223, 889]
precision: [0.930584, 0.944162, 0.931257, 0.905366, 0.920306]
recall: [0.905975, 0.90555, 0.889231, 0.912488, 0.878125]
invalid_rows: []
validation:
dependencies:
Expand Down
31 changes: 16 additions & 15 deletions ...t/perf/ubuntu/PT-1-03_dnn_pgd_40_153.yaml → ...t/perf/ubuntu/PT-1-03_dnn_pgd_40_556.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -48,46 +48,47 @@ experiment:
stcpb: [0, 4294729542]
swin: [0, 255]
trans_depth: [0, 2]
capture_utc: 1696820997157625457
capture_utc: 1703119998559698513
class_distribution: {0: 5000, 1: 5000}
config: config/prf1_1.yaml
dataset: data/unsw_nb15.csv
description: Constraint performance test with 3 correlated constraints
duration_sec: 427.9
end: 1696820996152503755
duration_sec: 343.5
end: 1703119997555427826
k_folds: 5
n_attributes: 33
n_classes: 2
n_records: 10000
name: PT-1-03
start: 1696820568256859934
start: 1703119654056152231
system:
cpu_current_frequency: 2200.196
cpu_current_frequency: 2249.998
cpu_max_frequency: 0.0
cpu_min_frequency: 0.0
cpu_physical_cores: 4
cpu_total_cores: 8
cpu_total_usage: 14.6
cpu_total_usage: 14.7
cpu_usage_per_core: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
machine: x86_64
processor: x86_64
python_runtime: 3.10.13 (main, Aug 25 2023, 13:20:03) [GCC 9.4.0]
release: 5.15.0-1042-gcp
system: Linux
version: '#50~20.04.1-Ubuntu SMP Mon Sep 11 03:30:57 UTC 2023'
virtual_mem_available: 28.64GB
virtual_mem_percentage: 8.6
virtual_mem_available: 30.39GB
virtual_mem_percentage: 3.0
virtual_mem_total_size: 31.34GB
virtual_mem_used: 2.30GB
virtual_mem_used: 570.89MB
folds:
accuracy: [0.925, 0.9105, 0.9245, 0.9105, 0.9125]
f_score: [0.923077, 0.907874, 0.924387, 0.906819, 0.91021]
n_evasions: [1850, 1821, 1849, 1821, 1825]
accuracy: [0.9215, 0.909, 0.9065, 0.9185, 0.92]
dur: [41763356350, 40947612462, 41836662782, 41477634922, 41491319222]
f_score: [0.918272, 0.910784, 0.904053, 0.918784, 0.91684]
n_evasions: [1843, 1818, 1813, 1837, 1840]
n_records: [2000, 2000, 2000, 2000, 2000]
n_valid: [2000, 2000, 2000, 2000, 2000]
n_valid_evades: [1850, 1821, 1849, 1821, 1825]
precision: [0.948367, 0.925498, 0.945697, 0.946739, 0.922037]
recall: [0.899101, 0.890909, 0.904016, 0.87013, 0.898683]
n_valid_evades: [1843, 1818, 1813, 1837, 1840]
precision: [0.944325, 0.88645, 0.935244, 0.934144, 0.948387]
recall: [0.893617, 0.936492, 0.874876, 0.903922, 0.887324]
invalid_rows: []
validation:
dependencies:
Expand Down
33 changes: 17 additions & 16 deletions ...erf/ubuntu/PT-1-06_dnn_cpgd_auto_349.yaml → ...erf/ubuntu/PT-1-06_dnn_cpgd_auto_138.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -50,46 +50,47 @@ experiment:
stcpb: [0, 4294729542]
swin: [0, 255]
trans_depth: [0, 2]
capture_utc: 1696823877353328814
capture_utc: 1703121744142365739
class_distribution: {0: 5000, 1: 5000}
config: config/prf1_2.yaml
dataset: data/unsw_nb15.csv
description: Constraint performance test with 6 correlated constraints
duration_sec: 765.5
end: 1696823876348441428
duration_sec: 569.1
end: 1703121743138263783
k_folds: 5
n_attributes: 33
n_classes: 2
n_records: 10000
name: PT-1-06
start: 1696823110860587125
start: 1703121174042185857
system:
cpu_current_frequency: 2200.196
cpu_current_frequency: 2249.998
cpu_max_frequency: 0.0
cpu_min_frequency: 0.0
cpu_physical_cores: 4
cpu_total_cores: 8
cpu_total_usage: 12.4
cpu_total_usage: 13.3
cpu_usage_per_core: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
machine: x86_64
processor: x86_64
python_runtime: 3.10.13 (main, Aug 25 2023, 13:20:03) [GCC 9.4.0]
release: 5.15.0-1042-gcp
system: Linux
version: '#50~20.04.1-Ubuntu SMP Mon Sep 11 03:30:57 UTC 2023'
virtual_mem_available: 28.65GB
virtual_mem_percentage: 8.6
virtual_mem_available: 30.41GB
virtual_mem_percentage: 3.0
virtual_mem_total_size: 31.34GB
virtual_mem_used: 2.29GB
virtual_mem_used: 558.56MB
folds:
accuracy: [0.904, 0.9205, 0.9115, 0.929, 0.9155]
f_score: [0.901639, 0.918503, 0.909648, 0.926272, 0.915879]
n_evasions: [1413, 1636, 892, 1471, 1720]
accuracy: [0.92, 0.919, 0.918, 0.927, 0.9215]
dur: [87213282315, 87265878121, 87076758053, 86463668865, 86718179208]
f_score: [0.920477, 0.915537, 0.917505, 0.926039, 0.919363]
n_evasions: [801, 1781, 1700, 1032, 943]
n_records: [2000, 2000, 2000, 2000, 2000]
n_valid: [472, 1001, 965, 1134, 1110]
n_valid_evades: [279, 777, 323, 696, 882]
precision: [0.943194, 0.92562, 0.939873, 0.93501, 0.925553]
recall: [0.863592, 0.911495, 0.881306, 0.917695, 0.906404]
n_valid: [1107, 721, 843, 819, 917]
n_valid_evades: [636, 643, 710, 262, 854]
precision: [0.91774, 0.938034, 0.953975, 0.938398, 0.928423]
recall: [0.92323, 0.894094, 0.883721, 0.914, 0.910478]
invalid_rows: []
validation:
dependencies:
Expand Down
33 changes: 17 additions & 16 deletions ...t/perf/ubuntu/PT-1-06_dnn_pgd_40_229.yaml → ...t/perf/ubuntu/PT-1-06_dnn_pgd_40_862.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -48,46 +48,47 @@ experiment:
stcpb: [0, 4294729542]
swin: [0, 255]
trans_depth: [0, 2]
capture_utc: 1696822823233807685
capture_utc: 1703121165866454196
class_distribution: {0: 5000, 1: 5000}
config: config/prf1_2.yaml
dataset: data/unsw_nb15.csv
description: Constraint performance test with 6 correlated constraints
duration_sec: 479.0
end: 1696822822228740733
duration_sec: 387.0
end: 1703121164862257060
k_folds: 5
n_attributes: 33
n_classes: 2
n_records: 10000
name: PT-1-06
start: 1696822343221815146
start: 1703120777870250291
system:
cpu_current_frequency: 2200.196
cpu_current_frequency: 2249.998
cpu_max_frequency: 0.0
cpu_min_frequency: 0.0
cpu_physical_cores: 4
cpu_total_cores: 8
cpu_total_usage: 14.4
cpu_usage_per_core: [0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0]
cpu_total_usage: 15.2
cpu_usage_per_core: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
machine: x86_64
processor: x86_64
python_runtime: 3.10.13 (main, Aug 25 2023, 13:20:03) [GCC 9.4.0]
release: 5.15.0-1042-gcp
system: Linux
version: '#50~20.04.1-Ubuntu SMP Mon Sep 11 03:30:57 UTC 2023'
virtual_mem_available: 28.64GB
virtual_mem_percentage: 8.6
virtual_mem_available: 30.39GB
virtual_mem_percentage: 3.0
virtual_mem_total_size: 31.34GB
virtual_mem_used: 2.30GB
virtual_mem_used: 571.74MB
folds:
accuracy: [0.9255, 0.907, 0.9145, 0.92, 0.9155]
f_score: [0.926997, 0.906627, 0.914968, 0.921722, 0.912116]
n_evasions: [1851, 1814, 1829, 1840, 1831]
accuracy: [0.9215, 0.925, 0.911, 0.9195, 0.923]
dur: [54033907588, 53649467840, 53693591367, 54333253373, 54030753272]
f_score: [0.919528, 0.924925, 0.907292, 0.917394, 0.925388]
n_evasions: [1843, 1850, 1822, 1839, 1846]
n_records: [2000, 2000, 2000, 2000, 2000]
n_valid: [2000, 2000, 2000, 2000, 2000]
n_valid_evades: [1851, 1814, 1829, 1840, 1831]
precision: [0.913127, 0.921429, 0.919081, 0.916342, 0.907867]
recall: [0.941294, 0.892292, 0.910891, 0.927165, 0.916405]
n_valid_evades: [1843, 1850, 1822, 1839, 1846]
precision: [0.929534, 0.939024, 0.904465, 0.947034, 0.930799]
recall: [0.909736, 0.911243, 0.910136, 0.889552, 0.920039]
invalid_rows: []
validation:
dependencies:
Expand Down
Loading

0 comments on commit 8e49bfd

Please sign in to comment.