Code

Measure inference time#532

Code

Merged

PGijsbers merged 39 commits intomasteropenml/automlbenchmark:masterfrom measure_inference_timeopenml/automlbenchmark:measure_inference_timeCopy head branch name to clipboardJun 17, 2023

+379-80

Commits

Commits on Jun 11, 2023

• Add method to split off a subsample of the test set to file

  PGijsbers

  committed
  71e7a8b

  Copy full SHA for 71e7a8b

Commits on Jun 12, 2023

• Add first draft for improving inference time measurements

  PGijsbers

  committed
  87f80aa

  Copy full SHA for 87f80aa

Commits on Jun 13, 2023

• Store all measured inference times to disk

  PGijsbers

  committed
  94ba496

  Copy full SHA for 94ba496

• Also accept a dataframe to allow to infer without disk load

  PGijsbers

  committed
  0c7bf7d

  Copy full SHA for 0c7bf7d

• Make repeats and batch sizes configurable

  PGijsbers

  committed
  df46e1b

  Copy full SHA for df46e1b

• Forward inference measurement configuration through task config

  PGijsbers

  committed
  0621ee2

  Copy full SHA for 0621ee2

• Randomize samples within the same batch size

  PGijsbers

  committed
  1a72ae1

  Copy full SHA for 1a72ae1

• Rename inference_X_rows column to infer_batch_size_X

  PGijsbers

  committed
  7100391

  Copy full SHA for 7100391

• Add docstring and move value checking of fmt

  Show description for f76e755
  PGijsbers

  committed
  f76e755

  Copy full SHA for f76e755

• Add inference time measurements for flaml

  PGijsbers

  committed
  bf8cbc7

  Copy full SHA for bf8cbc7

• Add inference time measurements

  PGijsbers

  committed
  cb04989

  Copy full SHA for cb04989

• Add inference time measurements

  PGijsbers

  committed
  8ea9033

  Copy full SHA for 8ea9033

• Add inference time measurements

  PGijsbers

  committed
  338fc94

  Copy full SHA for 338fc94

• Document shortcoming of measuring inference time for tpot

  PGijsbers

  committed
  9b024e2

  Copy full SHA for 9b024e2

• Add inference time measurement

  PGijsbers

  committed
  7b3455f

  Copy full SHA for 7b3455f

• Bump ubuntu base to 22.04

  PGijsbers

  committed
  5733223

  Copy full SHA for 5733223

• Add inference measurement for dataframe

  PGijsbers

  committed
  25a1bd4

  Copy full SHA for 25a1bd4

• Defaults to display median inference time

  Show description for a6f7a52
  PGijsbers

  committed
  a6f7a52

  Copy full SHA for a6f7a52

• Add seed to filename

  Show description for 94f11f6
  PGijsbers

  committed
  94f11f6

  Copy full SHA for 94f11f6

• Start with inference measurements (broken)

  PGijsbers

  committed
  4119d56

  Copy full SHA for 4119d56

Commits on Jun 14, 2023

• Add inference measurement on dataframe

  PGijsbers

  committed
  0000c63

  Copy full SHA for 0000c63

• Add inference time measurement with dataframes

  PGijsbers

  committed
  1d2054a

  Copy full SHA for 1d2054a

• Add dataframe inference time measurement

  PGijsbers

  committed
  357a63f

  Copy full SHA for 357a63f

• Add dataframe inference time measurement

  PGijsbers

  committed
  2dd1ffc

  Copy full SHA for 2dd1ffc

• Add dataframe inference measurement (ignores encoding)

  PGijsbers

  committed
  1e5a61d

  Copy full SHA for 1e5a61d

• Add dataframe inference measurement

  PGijsbers

  committed
  b48c5ed

  Copy full SHA for b48c5ed

• Add inference time measurements

  Show description for 42d38ac
  PGijsbers

  committed
  42d38ac

  Copy full SHA for 42d38ac

• Add inference time measurements

  PGijsbers

  committed
  4a38f6c

  Copy full SHA for 4a38f6c

Commits on Jun 16, 2023

• Allow newer autosklearn versions to use the pandas data instead (#534)

  PGijsbers

  authored
  6c9ccb3

  Copy full SHA for 6c9ccb3

• Add single row file inference for H2O

  PGijsbers

  committed
  b24c12c

  Copy full SHA for b24c12c

• Update inference measurement to record its from file

  PGijsbers

  committed
  d5da2bd

  Copy full SHA for d5da2bd

• Dynamically set type depending on presence of pandas

  PGijsbers

  committed
  cdb0828

  Copy full SHA for cdb0828

• Add time to job timeout if inference measurements enabled

  PGijsbers

  committed
  4c14dbe

  Copy full SHA for 4c14dbe

• Disable inference time measurements for CI

  PGijsbers

  committed
  acae403

  Copy full SHA for acae403

Commits on Jun 17, 2023

• Remove one dash

  PGijsbers

  committed
  8caf629

  Copy full SHA for 8caf629

• Disable inference time measurement by default

  PGijsbers

  committed
  f0cbfc0

  Copy full SHA for f0cbfc0

• Make measuring inference time optional, also measure single row df

  PGijsbers

  committed
  b2c9b38

  Copy full SHA for b2c9b38

• Add inference time measurement to (T)RF baselines

  PGijsbers

  committed
  7a3f433

  Copy full SHA for 7a3f433

• Remove skip inference measurement override since its default now

  PGijsbers

  committed
  54a2dd6

  Copy full SHA for 54a2dd6