Fix numerical overflow errors (simplified version of #34). #41

c-bata · 2020-06-22T03:29:34Z

Simplify #34

github-actions · 2020-06-22T03:32:36Z

Benchmark of Six-Hump Camel function

Report ID: e62895a27fa33bfb164866342cc7e5c28d4c3f9b6369dea1f6d90dd7db32cb62
Kurobako Version: 0.1.11
Number of Solvers: 3
Number of Problems: 1
Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Overall Results

Solver	Borda	Firsts
Random	0	0
cmaes	1	1
pycma	1	1

Individual Results

(1) Problem: Six-Hump Camel Function

Ranking	Solver	Best (avg +- sd)	AUC (avg +- sd)	Elapsed (avg +- sd)
1	cmaes (study)	-1.031628 +- 0.000000	50105.831 +- 49399.645	1.069 +- 0.357
1	pycma (study)	-1.031628 +- 0.000000	53224.211 +- 51419.988	46.263 +- 1.152
3	Random (study)	-0.413809 +- 0.607674	57232.028 +- 63026.273	0.000 +- 0.000

Solvers

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: 2d3de16774795cafa1c55d9a3930429d0feeccdff85f939e461dc9524da86710

recipe:

{
  "name": "cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "cmaes"
    ]
  }
}

specification:

{
  "name": "cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.5.0, kurobako-py=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 481138bcec61d254d11b91674720136f8e38bdb5928a8a149da588278c4d6ca0

recipe:

{
  "name": "pycma",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "pycma"
    ]
  }
}

specification:

{
  "name": "pycma",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.5.0, kurobako-py=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

Problems

ID: 6467a2061b1fa5c028c60874a64fabe9c1b0ab7d776af0da0df6cfb4dfd02ba8

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/problem_six_hump_camel.py"
    ]
  }
}

specification:

{
  "name": "Six-Hump Camel Function",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Six-Hump Camel",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: f31b73c660adb72fed697d11c15c80ce13258758a11825018770b9839389d350

problem: Six-Hump Camel Function
solver: Random
budget: 300
repeats: 10
concurrency: 1

ID: 7eccca2d9d5d2604f5018993bf7e71a5d084984a15838857191d18063ab98b04

problem: Six-Hump Camel Function
solver: cmaes
budget: 300
repeats: 10
concurrency: 1

ID: 0f3e974a00f395262cca5c61e98d0b7583bf26605406749ecec81f123eb17e60

problem: Six-Hump Camel Function
solver: pycma
budget: 300
repeats: 10
concurrency: 1

github-actions · 2020-06-22T03:32:40Z

Benchmark of Himmelblau function

Report ID: e1226441d44fafa6ee71a77950ff8db5dbf2eb6386aded92ee947f3e5ec98b00
Kurobako Version: 0.1.11
Number of Solvers: 3
Number of Problems: 1
Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Overall Results

Solver	Borda	Firsts
Random	0	0
cmaes	1	1
pycma	1	1

Individual Results

(1) Problem: Himmelblau Function

Ranking	Solver	Best (avg +- sd)	AUC (avg +- sd)	Elapsed (avg +- sd)
1	cmaes (study)	0.000000 +- 0.000001	895.533 +- 396.659	1.369 +- 0.123
1	pycma (study)	0.000009 +- 0.000025	1015.298 +- 449.553	45.652 +- 1.145
3	Random (study)	0.397676 +- 0.302454	875.169 +- 418.477	0.000 +- 0.000

Solvers

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: 2d3de16774795cafa1c55d9a3930429d0feeccdff85f939e461dc9524da86710

recipe:

{
  "name": "cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "cmaes"
    ]
  }
}

specification:

{
  "name": "cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.5.0, kurobako-py=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 481138bcec61d254d11b91674720136f8e38bdb5928a8a149da588278c4d6ca0

recipe:

{
  "name": "pycma",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "pycma"
    ]
  }
}

specification:

{
  "name": "pycma",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.5.0, kurobako-py=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

Problems

ID: c7b76605f4f5c667f11ced8a18554e8e83c157ccba2dfbb1ebfe705c9dcaf1d4

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/problem_himmelblau.py"
    ]
  }
}

specification:

{
  "name": "Himmelblau Function",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -4.0,
        "high": 4.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -4.0,
        "high": 4.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Himmelblau",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: f217a39412333ce79dc820f3f52c0a8fce663cb8c7fe46337ad40c552d03adc4

problem: Himmelblau Function
solver: Random
budget: 300
repeats: 10
concurrency: 1

ID: 398e5c43f8a73c7bf520a798969a4d51ecb29dfb756f42958c04f528e693fbaa

problem: Himmelblau Function
solver: cmaes
budget: 300
repeats: 10
concurrency: 1

ID: 6bdf1f82a81f12f441ce55e3ac8035846f93fcfc14602ca2886dab202d60c54b

problem: Himmelblau Function
solver: pycma
budget: 300
repeats: 10
concurrency: 1

github-actions · 2020-06-22T03:33:07Z

Benchmark of Rosenbrock function

Report ID: 6e8d1a319e3048cd75756424c417929437900014d42b565bb5eb2d3ff5a43efd
Kurobako Version: 0.1.11
Number of Solvers: 3
Number of Problems: 1
Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Overall Results

Solver	Borda	Firsts
Random	0	0
cmaes	1	1
pycma	1	1

Individual Results

(1) Problem: Rosenbrock Function

Ranking	Solver	Best (avg +- sd)	AUC (avg +- sd)	Elapsed (avg +- sd)
1	cmaes (study)	0.152145 +- 0.255840	191588.258 +- 201508.713	0.913 +- 0.285
1	pycma (study)	0.117136 +- 0.128045	209811.767 +- 234431.051	36.460 +- 0.998
3	Random (study)	1.492788 +- 1.094340	239394.384 +- 313827.719	0.000 +- 0.000

Solvers

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: 2d3de16774795cafa1c55d9a3930429d0feeccdff85f939e461dc9524da86710

recipe:

{
  "name": "cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "cmaes"
    ]
  }
}

specification:

{
  "name": "cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.5.0, kurobako-py=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 481138bcec61d254d11b91674720136f8e38bdb5928a8a149da588278c4d6ca0

recipe:

{
  "name": "pycma",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "pycma"
    ]
  }
}

specification:

{
  "name": "pycma",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.5.0, kurobako-py=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

Problems

ID: fa3c593805d48da4e3e5c06929acd0a92657b18a36c0e5529fc6108268008949

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/problem_rosenbrock.py"
    ]
  }
}

specification:

{
  "name": "Rosenbrock Function",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Rosenbrock",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: bb45956ae6a92ec6165f31c9bc7cbdbf12dea94976fe60de037d520f5657fe2a

problem: Rosenbrock Function
solver: Random
budget: 300
repeats: 10
concurrency: 1

ID: 2ec23b47296c806a7b9ff1c9b2052543c73c048ef81a30ba8fdb0d6e462d518d

problem: Rosenbrock Function
solver: cmaes
budget: 300
repeats: 10
concurrency: 1

ID: f9ba6ad7a77b7aa64f50562e2ee93f5e14701573ff75525949df4a55327ca2e3

problem: Rosenbrock Function
solver: pycma
budget: 300
repeats: 10
concurrency: 1

c-bata · 2020-06-22T03:41:49Z

@HideakiImamura I tested the kurobako benchmark script and the error wasn't raised on this branch.

Could you review this?

HideakiImamura · 2020-06-22T05:07:46Z

Thanks for the simplified PR! I'll review this PR now.

cmaes/cma.py

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

HideakiImamura

Thanks! LGTM!

c-bata · 2020-06-22T06:13:12Z

I released v0.5.1 to PyPI.
https://github.com/CyberAgent/cmaes/releases/tag/v0.5.1

HideakiImamura and others added 4 commits June 3, 2020 11:26

Avoid numerical errors and overflow

58e67ec

Fix by review comments

0ccc62b

Simplify HideakiImamura's patch

96be8d5

Set epsilon if eigenvalue is negative

6cd22d0

c-bata mentioned this pull request Jun 22, 2020

Avoid numerical errors and overflow. #34

Merged

c-bata marked this pull request as ready for review June 22, 2020 03:39

HideakiImamura reviewed Jun 22, 2020

View reviewed changes

cmaes/cma.py Outdated Show resolved Hide resolved

Apply suggestions from code review

e8293fa

Co-authored-by: Hideaki Imamura <38826298+HideakiImamura@users.noreply.github.com>

c-bata changed the title ~~Simplify HideakiImamura's patch~~ Fix numerical overflow errors (simplified version of #34). Jun 22, 2020

HideakiImamura approved these changes Jun 22, 2020

View reviewed changes

c-bata merged commit 156bb47 into master Jun 22, 2020

c-bata deleted the simplify-hideakiimamura-patch branch June 22, 2020 05:43

Fix numerical overflow errors (simplified version of #34). #41

Fix numerical overflow errors (simplified version of #34). #41

Conversation

c-bata commented Jun 22, 2020 • edited Loading

github-actions bot commented Jun 22, 2020 • edited Loading

Benchmark of Six-Hump Camel function

Table of Contents

Overall Results

Individual Results

(1) Problem: Six-Hump Camel Function

Solvers

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

ID: 2d3de16774795cafa1c55d9a3930429d0feeccdff85f939e461dc9524da86710

ID: 481138bcec61d254d11b91674720136f8e38bdb5928a8a149da588278c4d6ca0

Problems

ID: 6467a2061b1fa5c028c60874a64fabe9c1b0ab7d776af0da0df6cfb4dfd02ba8

Studies

ID: f31b73c660adb72fed697d11c15c80ce13258758a11825018770b9839389d350

ID: 7eccca2d9d5d2604f5018993bf7e71a5d084984a15838857191d18063ab98b04

ID: 0f3e974a00f395262cca5c61e98d0b7583bf26605406749ecec81f123eb17e60

github-actions bot commented Jun 22, 2020 • edited Loading

Benchmark of Himmelblau function

Table of Contents

Overall Results

Individual Results

(1) Problem: Himmelblau Function

Solvers

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

ID: 2d3de16774795cafa1c55d9a3930429d0feeccdff85f939e461dc9524da86710

ID: 481138bcec61d254d11b91674720136f8e38bdb5928a8a149da588278c4d6ca0

Problems

ID: c7b76605f4f5c667f11ced8a18554e8e83c157ccba2dfbb1ebfe705c9dcaf1d4

Studies

ID: f217a39412333ce79dc820f3f52c0a8fce663cb8c7fe46337ad40c552d03adc4

ID: 398e5c43f8a73c7bf520a798969a4d51ecb29dfb756f42958c04f528e693fbaa

ID: 6bdf1f82a81f12f441ce55e3ac8035846f93fcfc14602ca2886dab202d60c54b

github-actions bot commented Jun 22, 2020 • edited Loading

Benchmark of Rosenbrock function

Table of Contents

Overall Results

Individual Results

(1) Problem: Rosenbrock Function

Solvers

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

ID: 2d3de16774795cafa1c55d9a3930429d0feeccdff85f939e461dc9524da86710

ID: 481138bcec61d254d11b91674720136f8e38bdb5928a8a149da588278c4d6ca0

Problems

ID: fa3c593805d48da4e3e5c06929acd0a92657b18a36c0e5529fc6108268008949

Studies

ID: bb45956ae6a92ec6165f31c9bc7cbdbf12dea94976fe60de037d520f5657fe2a

ID: 2ec23b47296c806a7b9ff1c9b2052543c73c048ef81a30ba8fdb0d6e462d518d

ID: f9ba6ad7a77b7aa64f50562e2ee93f5e14701573ff75525949df4a55327ca2e3

c-bata commented Jun 22, 2020

HideakiImamura commented Jun 22, 2020

HideakiImamura left a comment

Choose a reason for hiding this comment

c-bata commented Jun 22, 2020

c-bata commented Jun 22, 2020 •

edited

Loading

github-actions bot commented Jun 22, 2020 •

edited

Loading

github-actions bot commented Jun 22, 2020 •

edited

Loading

github-actions bot commented Jun 22, 2020 •

edited

Loading