Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cloning over _time_created when cloning Experiment #2307

Closed
wants to merge 1 commit into from

Conversation

mgrange1998
Copy link
Contributor

Summary:
Quick RFC for everyone:

This equality check is failing sometimes in Python 3.9 on open source ( T184131441 ):

def test_clone_with(self) -> None:
        experiment = get_experiment()
        cloned_experiment = experiment.clone_with()
        self.assertEqual(cloned_experiment.name, "cloned_experiment_" + experiment.name)
        cloned_experiment._name = experiment.name
        self.assertEqual(cloned_experiment, experiment)

The error is

Experiment(test) (type <class 'ax.core.experiment.Experiment'>) != Experiment(test) (type <class 'ax.core.experiment.Experiment'>).

Fields with different values:

1) _time_created: 
2024-04-01 05:05:13.000613 (type <class 'datetime.datetime'>) 
!= 2024-04-01 05:05:12.998793 (type <class 'datetime.datetime'>).

This is a race condition, where if "get_experiment" and "clone_with" execute quickly the _time_created field will be equal, but if there is delay they'll be unequal.

Proposed fixes:

  1. In experiment equality check, ignore "_time_created" field entirely.
  2. In experiment equality check, check if the "_time_created" fields are "nearly equal"
  3. Set the clone's _time_created field manually to experiment._time_created

I've taken the first approach for this diff, but want to get team feedback on what the preferred approach would be.

Differential Revision: D55596348

Summary:
Quick RFC for everyone:

This equality check is failing sometimes in Python 3.9 on open source ( T184131441 ):
```
def test_clone_with(self) -> None:
        experiment = get_experiment()
        cloned_experiment = experiment.clone_with()
        self.assertEqual(cloned_experiment.name, "cloned_experiment_" + experiment.name)
        cloned_experiment._name = experiment.name
        self.assertEqual(cloned_experiment, experiment)
```
The error is 
```
Experiment(test) (type <class 'ax.core.experiment.Experiment'>) != Experiment(test) (type <class 'ax.core.experiment.Experiment'>).

Fields with different values:

1) _time_created: 
2024-04-01 05:05:13.000613 (type <class 'datetime.datetime'>) 
!= 2024-04-01 05:05:12.998793 (type <class 'datetime.datetime'>).
```

This is a race condition, where if "get_experiment" and "clone_with" execute quickly the _time_created field will be equal, but if there is delay they'll be unequal. 

Proposed fixes:
1. In experiment equality check, ignore "_time_created" field entirely. 
2. In experiment equality check, check if the "_time_created" fields are "nearly equal"
3. Set the clone's _time_created field manually to experiment._time_created

I've taken the first approach for this diff, but want to get team feedback on what the preferred approach would be.

Differential Revision: D55596348
@facebook-github-bot facebook-github-bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Apr 1, 2024
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D55596348

@codecov-commenter
Copy link

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.91%. Comparing base (86a4c8d) to head (e59dcf1).

❗ Current head e59dcf1 differs from pull request most recent head 38772b8. Consider uploading reports for the commit 38772b8 to get more accurate results

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2307   +/-   ##
=======================================
  Coverage   94.91%   94.91%           
=======================================
  Files         489      489           
  Lines       47710    47712    +2     
=======================================
+ Hits        45282    45284    +2     
  Misses       2428     2428           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@facebook-github-bot
Copy link
Contributor

This pull request has been merged in 95bafb0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed Do not delete this pull request or issue due to inactivity. fb-exported Merged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants