Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timestamp computation for the HEAR benchmark #2

Closed
alexanderwerning opened this issue Jan 24, 2023 · 1 comment
Closed

Timestamp computation for the HEAR benchmark #2

alexanderwerning opened this issue Jan 24, 2023 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@alexanderwerning
Copy link

Hello,
whenever I try to evaluate on any of the two HEAR tasks that require timestamp embeddings (DCASE2016 Task 2, Maestro), I get a score of NaN. I believe the timestamp computation in the RuntimeMAE ist incorrect. In the HEAR API, the timestamps are defined in milliseconds, not seconds.
See for example in hearbaseline/wav2vec2, the timestamp values need to be multiplied by 1000.

With this modification, I get reasonable results.

@daisukelab daisukelab self-assigned this Jan 25, 2023
@daisukelab daisukelab added the bug Something isn't working label Jan 25, 2023
@daisukelab
Copy link
Collaborator

daisukelab commented Jan 25, 2023

Hi @alexanderwerning,

Thank you for spotting the bug! We appreciate it. After fixing the bug, I confirmed the onset FMS result of 0.896 on DCASE2016 Task 2 using 80x512p16x16_paper/checkpoint-100.pth, a good score for a single model.

predict - dcase2016_task2 - 2023-01-25 10:10:36,356 - 356 - Final Test Results: {"test": {"test_loss": 0.0164401363581419, "test_score": 0.8962472677230835, "test_event_onset_200ms_fms": 0.8962472677230835, "test_event_onset_200ms_fms_f_measure": 0.8962472677230835, "test_event
_onset_200ms_fms_precision": 0.937644362449646, "test_event_onset_200ms_fms_recall": 0.858350932598114, "test_segment_1s_er": 0.14847160875797272, "test_segment_1s_er_error_rate": 0.14847160875797272, "validation_score": 0.9464285969734192, "epoch": 100, "time_in_min": 9.736017
545064291},

I will commit my fix soon. Thanks again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants