Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deprecate patch_fast_itersection_search_space #28

Merged
merged 1 commit into from
Apr 22, 2020
Merged

Deprecate patch_fast_itersection_search_space #28

merged 1 commit into from
Apr 22, 2020

Conversation

c-bata
Copy link
Collaborator

@c-bata c-bata commented Apr 22, 2020

No description provided.

c-bata
c-bata commented Apr 22, 2020
View reviewed changes
cmaes/monkeypatch.py
# https://github.com/optuna/optuna/pull/885
if get_optuna_version() < (1, 3):
# CMA-ES sampler is added at v1.3.0.
# https://github.com/optuna/optuna/pull/1142 is added at v1.4.0.
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

optuna/optuna#1142

@github-actions
Copy link

Benchmark of Six-Hump Camel function

plot curve image

  • Report ID: 578ea062c89fcf610c57b3f9519b9aaf6dd322e0f47fc817c5b7bccdfc043ea4
  • Kurobako Version: 0.1.4
  • Number of Solvers: 3
  • Number of Problems: 1
  • Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Table of Contents

  1. Overall Results
  2. Individual Results
  3. Solvers
  4. Problems
  5. Studies

Overall Results

Solver Borda Firsts
Random 0 0
cmaes 1 1
pycma 1 1

Individual Results

(1) Problem: Six-Hump Camel Function

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 pycma (study) -1.031628 +- 0.000000 53224.211 +- 51419.988 52.753 +- 1.661
1 cmaes (study) -1.031628 +- 0.000000 50105.831 +- 49399.645 1.598 +- 0.096
3 Random (study) -0.638129 +- 0.306493 49519.918 +- 67891.710 0.000 +- 0.000

Solvers

ID: 29b0b413e0e229de860c855bd7258ff2335c36821b0877f4c28fe96fc82afb3c

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.1.4"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: c68f7e3ddfa8d05b82f4a692bd53a2deeacbf96ce9359399548fed3549065c0a

recipe:

{
  "name": "cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/solver_cmaes.py"
    ]
  }
}

specification:

{
  "name": "cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.3.0, kurobako-py=0.1.5"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 40c000519d4870c8307296d20258c148edf4b09499b6cc18d348e4163d4feee0

recipe:

{
  "name": "pycma",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/solver_pycma.py"
    ]
  }
}

specification:

{
  "name": "pycma",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.3.0, kurobako-py=0.1.5"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

Problems

ID: 6467a2061b1fa5c028c60874a64fabe9c1b0ab7d776af0da0df6cfb4dfd02ba8

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/problem_six_hump_camel.py"
    ]
  }
}

specification:

{
  "name": "Six-Hump Camel Function",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Six-Hump Camel",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: 469a87b7e952a37da5c21e58f936acf5371e438f48236c91a18cd7633c167f74

ID: ec87f9d75b4551d7c4f1276680bc0a60a64506587fa4de30e0aa97e5eb2ead07

ID: d4959c9229e36f6dd07748b067776a90367bc4fbbca3e5c71c6f1f0c992383c6

@github-actions
Copy link

Benchmark of Himmelblau function

plot curve image

  • Report ID: 09410d2e28f12ea84ca387f743d8e4fdaff80bda4c3463713212dc0337316642
  • Kurobako Version: 0.1.4
  • Number of Solvers: 3
  • Number of Problems: 1
  • Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Table of Contents

  1. Overall Results
  2. Individual Results
  3. Solvers
  4. Problems
  5. Studies

Overall Results

Solver Borda Firsts
Random 0 0
cmaes 1 1
pycma 1 1

Individual Results

(1) Problem: Himmelblau Function

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 pycma (study) 0.000009 +- 0.000025 1015.298 +- 449.553 55.555 +- 1.510
1 cmaes (study) 0.000000 +- 0.000001 895.533 +- 396.659 1.633 +- 0.158
3 Random (study) 0.330382 +- 0.260764 795.051 +- 405.016 0.000 +- 0.000

Solvers

ID: 29b0b413e0e229de860c855bd7258ff2335c36821b0877f4c28fe96fc82afb3c

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.1.4"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: c68f7e3ddfa8d05b82f4a692bd53a2deeacbf96ce9359399548fed3549065c0a

recipe:

{
  "name": "cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/solver_cmaes.py"
    ]
  }
}

specification:

{
  "name": "cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.3.0, kurobako-py=0.1.5"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 40c000519d4870c8307296d20258c148edf4b09499b6cc18d348e4163d4feee0

recipe:

{
  "name": "pycma",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/solver_pycma.py"
    ]
  }
}

specification:

{
  "name": "pycma",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.3.0, kurobako-py=0.1.5"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

Problems

ID: c7b76605f4f5c667f11ced8a18554e8e83c157ccba2dfbb1ebfe705c9dcaf1d4

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/problem_himmelblau.py"
    ]
  }
}

specification:

{
  "name": "Himmelblau Function",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -4.0,
        "high": 4.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -4.0,
        "high": 4.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Himmelblau",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: 0410569911381c38a4dfc80ffe1f34c3f63df093167bfba6bbaabb769ccd411e

ID: 76c58f16fcc180ac7157c75efcbde679bca22909ba3b6180a8972f84ec195ec0

ID: ef1fd063a7e0b6d2c0e5b3aa01f3a6c179ae712a8dfc47cea7f7d0e7694fb32b

@github-actions
Copy link

Benchmark of Rosenbrock function

plot curve image

  • Report ID: 676606de48ed96671ba7703dd5e9b935c4646784d9fc75b0ecd984dd435e003e
  • Kurobako Version: 0.1.4
  • Number of Solvers: 3
  • Number of Problems: 1
  • Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Table of Contents

  1. Overall Results
  2. Individual Results
  3. Solvers
  4. Problems
  5. Studies

Overall Results

Solver Borda Firsts
Random 0 0
cmaes 1 1
pycma 1 1

Individual Results

(1) Problem: Rosenbrock Function

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 pycma (study) 0.117136 +- 0.128045 209811.767 +- 234431.051 51.173 +- 2.178
1 cmaes (study) 0.152145 +- 0.255840 191588.258 +- 201508.713 1.537 +- 0.093
3 Random (study) 1.620025 +- 1.199580 231925.306 +- 326653.098 0.000 +- 0.000

Solvers

ID: 29b0b413e0e229de860c855bd7258ff2335c36821b0877f4c28fe96fc82afb3c

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.1.4"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: c68f7e3ddfa8d05b82f4a692bd53a2deeacbf96ce9359399548fed3549065c0a

recipe:

{
  "name": "cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/solver_cmaes.py"
    ]
  }
}

specification:

{
  "name": "cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.3.0, kurobako-py=0.1.5"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 40c000519d4870c8307296d20258c148edf4b09499b6cc18d348e4163d4feee0

recipe:

{
  "name": "pycma",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/solver_pycma.py"
    ]
  }
}

specification:

{
  "name": "pycma",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.3.0, kurobako-py=0.1.5"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

Problems

ID: fa3c593805d48da4e3e5c06929acd0a92657b18a36c0e5529fc6108268008949

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/problem_rosenbrock.py"
    ]
  }
}

specification:

{
  "name": "Rosenbrock Function",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Rosenbrock",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: 988b3ee2fc93f867e9f50a8e14729e90a5ff829cc4d3f741663efeeefa1bf288

ID: 9ca9592a3433ddcf1fa317be0eee41b517b701a8f54b0ece0bcb22c2b8155241

ID: 733e2e2cb83d33069bf588d2a335c53d80e7c65051e677999aa541738a4494a4

@c-bata c-bata merged commit 6b5d806 into master Apr 22, 2020
@c-bata c-bata deleted the deprecate-faster-intersection-search-space branch April 22, 2020 09:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@c-bata