Make 'cmaes.cma' module private with deprecation message #46

c-bata · 2020-07-23T02:09:18Z

Currently, we can import "CMA" class in 2 ways.

from cmaes import CMA

from cmaes.cma import CMA

The latter one is a bit redundant than the former one.
In this PR, I rename cma.py into _cma.py and showing the warning message when importing cmaes.cma module to keep backward compatibility.

(venv) $ python
Python 3.8.2 (default, Apr  6 2020, 10:38:56) 
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from cmaes.cma import CMA
/Users/a14737/src/github.com/CyberAgent/cmaes/cmaes/cma.py:5: UserWarning: This module is deprecated. Please import CMA class from the package root (ex: from cmaes import CMA).
  warnings.warn(
>>>

github-actions · 2020-07-23T02:12:25Z

Benchmark of Rosenbrock function

Report ID: 41a85c15ddf351a0e377717ad5dcfc7e0a3ea722be5ed5b8bcf91ead4509846b
Kurobako Version: 0.1.11
Number of Solvers: 3
Number of Problems: 1
Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Overall Results

Solver	Borda	Firsts
Random	0	0
cmaes	1	1
pycma	1	1

Individual Results

(1) Problem: Rosenbrock Function

Ranking	Solver	Best (avg +- sd)	AUC (avg +- sd)	Elapsed (avg +- sd)
1	cmaes (study)	0.152145 +- 0.255840	191588.258 +- 201508.713	1.012 +- 0.233
1	pycma (study)	0.117136 +- 0.128045	209811.767 +- 234431.051	38.447 +- 1.352
3	Random (study)	1.492788 +- 1.094340	239394.384 +- 313827.719	0.000 +- 0.000

Solvers

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: 2d3de16774795cafa1c55d9a3930429d0feeccdff85f939e461dc9524da86710

recipe:

{
  "name": "cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "cmaes"
    ]
  }
}

specification:

{
  "name": "cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.5.0, kurobako-py=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 481138bcec61d254d11b91674720136f8e38bdb5928a8a149da588278c4d6ca0

recipe:

{
  "name": "pycma",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "pycma"
    ]
  }
}

specification:

{
  "name": "pycma",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.5.0, kurobako-py=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

Problems

ID: fa3c593805d48da4e3e5c06929acd0a92657b18a36c0e5529fc6108268008949

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/problem_rosenbrock.py"
    ]
  }
}

specification:

{
  "name": "Rosenbrock Function",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Rosenbrock",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: bb45956ae6a92ec6165f31c9bc7cbdbf12dea94976fe60de037d520f5657fe2a

problem: Rosenbrock Function
solver: Random
budget: 300
repeats: 10
concurrency: 1

ID: 2ec23b47296c806a7b9ff1c9b2052543c73c048ef81a30ba8fdb0d6e462d518d

problem: Rosenbrock Function
solver: cmaes
budget: 300
repeats: 10
concurrency: 1

ID: f9ba6ad7a77b7aa64f50562e2ee93f5e14701573ff75525949df4a55327ca2e3

problem: Rosenbrock Function
solver: pycma
budget: 300
repeats: 10
concurrency: 1

github-actions · 2020-07-23T02:12:30Z

Benchmark of Himmelblau function

Report ID: f2123537576d20614969f25b20aefa401bf2069ec684ea7094c51206ba008060
Kurobako Version: 0.1.11
Number of Solvers: 3
Number of Problems: 1
Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Overall Results

Solver	Borda	Firsts
Random	0	0
cmaes	1	1
pycma	1	1

Individual Results

(1) Problem: Himmelblau Function

Ranking	Solver	Best (avg +- sd)	AUC (avg +- sd)	Elapsed (avg +- sd)
1	cmaes (study)	0.000000 +- 0.000001	895.533 +- 396.659	1.353 +- 0.080
1	pycma (study)	0.000009 +- 0.000025	1015.298 +- 449.553	47.012 +- 1.070
3	Random (study)	0.397676 +- 0.302454	875.169 +- 418.477	0.000 +- 0.000

Solvers

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: 2d3de16774795cafa1c55d9a3930429d0feeccdff85f939e461dc9524da86710

recipe:

{
  "name": "cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "cmaes"
    ]
  }
}

specification:

{
  "name": "cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.5.0, kurobako-py=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 481138bcec61d254d11b91674720136f8e38bdb5928a8a149da588278c4d6ca0

recipe:

{
  "name": "pycma",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "pycma"
    ]
  }
}

specification:

{
  "name": "pycma",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.5.0, kurobako-py=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

Problems

ID: c7b76605f4f5c667f11ced8a18554e8e83c157ccba2dfbb1ebfe705c9dcaf1d4

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/problem_himmelblau.py"
    ]
  }
}

specification:

{
  "name": "Himmelblau Function",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -4.0,
        "high": 4.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -4.0,
        "high": 4.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Himmelblau",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: f217a39412333ce79dc820f3f52c0a8fce663cb8c7fe46337ad40c552d03adc4

problem: Himmelblau Function
solver: Random
budget: 300
repeats: 10
concurrency: 1

ID: 398e5c43f8a73c7bf520a798969a4d51ecb29dfb756f42958c04f528e693fbaa

problem: Himmelblau Function
solver: cmaes
budget: 300
repeats: 10
concurrency: 1

ID: 6bdf1f82a81f12f441ce55e3ac8035846f93fcfc14602ca2886dab202d60c54b

problem: Himmelblau Function
solver: pycma
budget: 300
repeats: 10
concurrency: 1

github-actions · 2020-07-23T02:14:21Z

Benchmark of Six-Hump Camel function

Report ID: 86c27740c9d7d2d673eb0e5303eb4ffd477e57e62bb98827932b423b715ca320
Kurobako Version: 0.1.11
Number of Solvers: 3
Number of Problems: 1
Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Overall Results

Solver	Borda	Firsts
Random	0	0
cmaes	1	1
pycma	1	1

Individual Results

(1) Problem: Six-Hump Camel Function

Ranking	Solver	Best (avg +- sd)	AUC (avg +- sd)	Elapsed (avg +- sd)
1	cmaes (study)	-1.031628 +- 0.000000	50105.831 +- 49399.645	1.153 +- 0.078
1	pycma (study)	-1.031628 +- 0.000000	53224.211 +- 51419.988	40.543 +- 0.947
3	Random (study)	-0.413809 +- 0.607674	57232.028 +- 63026.273	0.000 +- 0.000

Solvers

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: 2d3de16774795cafa1c55d9a3930429d0feeccdff85f939e461dc9524da86710

recipe:

{
  "name": "cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "cmaes"
    ]
  }
}

specification:

{
  "name": "cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.5.0, kurobako-py=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 481138bcec61d254d11b91674720136f8e38bdb5928a8a149da588278c4d6ca0

recipe:

{
  "name": "pycma",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "pycma"
    ]
  }
}

specification:

{
  "name": "pycma",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.5.0, kurobako-py=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

Problems

ID: 6467a2061b1fa5c028c60874a64fabe9c1b0ab7d776af0da0df6cfb4dfd02ba8

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/problem_six_hump_camel.py"
    ]
  }
}

specification:

{
  "name": "Six-Hump Camel Function",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Six-Hump Camel",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: f31b73c660adb72fed697d11c15c80ce13258758a11825018770b9839389d350

problem: Six-Hump Camel Function
solver: Random
budget: 300
repeats: 10
concurrency: 1

ID: 7eccca2d9d5d2604f5018993bf7e71a5d084984a15838857191d18063ab98b04

problem: Six-Hump Camel Function
solver: cmaes
budget: 300
repeats: 10
concurrency: 1

ID: 0f3e974a00f395262cca5c61e98d0b7583bf26605406749ecec81f123eb17e60

problem: Six-Hump Camel Function
solver: pycma
budget: 300
repeats: 10
concurrency: 1

c-bata added 3 commits July 23, 2020 10:57

Make 'cmaes.cma' module private

4488f77

Fix lint errors

9c0c8f3

Set FutureWarning category

50d7caf

c-bata mentioned this pull request Jul 23, 2020

Roadmap to v1.0.0 #35

Closed

3 tasks

c-bata changed the title ~~Make 'cmaes.cma' module private~~ Make 'cmaes.cma' module private with deprecation message Jul 23, 2020

c-bata merged commit 3eb7423 into master Jul 23, 2020

c-bata deleted the make-cma-module-private branch July 23, 2020 02:17

Make 'cmaes.cma' module private with deprecation message #46

Make 'cmaes.cma' module private with deprecation message #46

Conversation

c-bata commented Jul 23, 2020

github-actions bot commented Jul 23, 2020 • edited Loading

Benchmark of Rosenbrock function

Table of Contents

Overall Results

Individual Results

(1) Problem: Rosenbrock Function

Solvers

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

ID: 2d3de16774795cafa1c55d9a3930429d0feeccdff85f939e461dc9524da86710

ID: 481138bcec61d254d11b91674720136f8e38bdb5928a8a149da588278c4d6ca0

Problems

ID: fa3c593805d48da4e3e5c06929acd0a92657b18a36c0e5529fc6108268008949

Studies

ID: bb45956ae6a92ec6165f31c9bc7cbdbf12dea94976fe60de037d520f5657fe2a

ID: 2ec23b47296c806a7b9ff1c9b2052543c73c048ef81a30ba8fdb0d6e462d518d

ID: f9ba6ad7a77b7aa64f50562e2ee93f5e14701573ff75525949df4a55327ca2e3

github-actions bot commented Jul 23, 2020 • edited Loading

Benchmark of Himmelblau function

Table of Contents

Overall Results

Individual Results

(1) Problem: Himmelblau Function

Solvers

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

ID: 2d3de16774795cafa1c55d9a3930429d0feeccdff85f939e461dc9524da86710

ID: 481138bcec61d254d11b91674720136f8e38bdb5928a8a149da588278c4d6ca0

Problems

ID: c7b76605f4f5c667f11ced8a18554e8e83c157ccba2dfbb1ebfe705c9dcaf1d4

Studies

ID: f217a39412333ce79dc820f3f52c0a8fce663cb8c7fe46337ad40c552d03adc4

ID: 398e5c43f8a73c7bf520a798969a4d51ecb29dfb756f42958c04f528e693fbaa

ID: 6bdf1f82a81f12f441ce55e3ac8035846f93fcfc14602ca2886dab202d60c54b

github-actions bot commented Jul 23, 2020 • edited Loading

Benchmark of Six-Hump Camel function

Table of Contents

Overall Results

Individual Results

(1) Problem: Six-Hump Camel Function

Solvers

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

ID: 2d3de16774795cafa1c55d9a3930429d0feeccdff85f939e461dc9524da86710

ID: 481138bcec61d254d11b91674720136f8e38bdb5928a8a149da588278c4d6ca0

Problems

ID: 6467a2061b1fa5c028c60874a64fabe9c1b0ab7d776af0da0df6cfb4dfd02ba8

Studies

ID: f31b73c660adb72fed697d11c15c80ce13258758a11825018770b9839389d350

ID: 7eccca2d9d5d2604f5018993bf7e71a5d084984a15838857191d18063ab98b04

ID: 0f3e974a00f395262cca5c61e98d0b7583bf26605406749ecec81f123eb17e60

github-actions bot commented Jul 23, 2020 •

edited

Loading

github-actions bot commented Jul 23, 2020 •

edited

Loading

github-actions bot commented Jul 23, 2020 •

edited

Loading