-
Notifications
You must be signed in to change notification settings - Fork 190
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Error in AveragedTimeInterval Output #3670
Comments
The issue not only happens after picking up checkpoint, it also occurs before that. For example, with the same MWE, if I set: Δt = .01 # timestep (s)
T1 = 6 # first simulation stop time (s)
T2 = 2T1 # second simulation stop time (s)
δt = .03 # progress message interval and output saving interval |
Hmm ok. Can you identify the difference between the cases where it works and where it fails? I'd like some other devs to help out with this @navidcy @simone-silvestri |
Sure, I ran a simple test using the MWE above without picking up a checkpoint and set: Δt = .01 # timestep (s)
T1 = 5 # first simulation stop time (s)
δt = .03 # progress message interval and output saving interval
# Run a simulation
simulation = test_simulation(T1, Δt, δt, true)
run!(simulation) I've noticed that this strange behavior occurs when δt is smaller than or equal to 0.03, but it disappears when δt is greater than 0.03 (even at 0.031). It seems there is a cutoff value of δt below which the error emerges. |
By the way, we originally thought we might be able to solve this issue by setting the checkpointing interval to be an integer multiple of the averaging interval (and window) but that is evidently not the case since the bug shows up in the MWEs above that satisfy this condition (where the checkpoint time is |
Let me see if I understand this. We want to run a simulation in which the output interval and the averaging window is between 2-3 time-steps, right? Maybe we need to write out the sequence of steps that we expect to occur and then we can look at the source code and see why that is not happening. Does the problem also happen when |
Yes, exactly. The error occurs in this range.
Yes, I agree, the problem still happens when δt = 0.02. |
It seems like the |
@simone-silvestri can you spell out what you're saying more explicitly by referencing / linking into the source code? I can't figure out what you're saying (it seems I would have to dive into the source code to figure it out). |
It looks like the time step calculated here Oceananigans.jl/src/OutputWriters/windowed_time_average.jl Lines 211 to 213 in 406eb9c
does not agree with the time step used in the simulation because of finite precision. I don't think we need to recompute it, we can use the simulation time step size. Probably this will not solve this issue but it will help to make the averaging more consistent |
But Moreover, do you think that a small error in PS I updated your comment to show more of the code so that we could see explicitly the part of the code you were referring to. EDIT: and just to clarify the time increment between collections can be more than one time-step because we support strided averages (eg computing the averaged quantity less frequently than every time step). |
I'm also doubtful small round off errors in the timestep would be related, unless there a place in the code where the scheduled times for outputting / checkpointing need to exactly match some value. In the MWE, the problematic values look like they're all zeros. In our more complicated example, where the averaging interval is a large (decimal) multiple of the timestep, it's not clear if the values are underestimated because the velocities are of both signs so a biased average could go either way. |
I am not sure if this helps, but this might shed some light. function progress_message(sim)
wta = sim.output_writers[:timeavg2].outputs[1]
wd = wta.window_start_time
ws = wta.window_start_iteration
pc = wta.previous_collection_time
fo = wta.fetch_operand
res = sum(wta.result)
interval = wta.schedule.interval
pis = wta.schedule.previous_interval_stop_time
cll = wta.schedule.collecting
@info string("Iter: ", iteration(sim), ", time: ", prettytime(sim), " , u-avg: ", res, ", window_start_time: ", wd, ", window_start_iteration: ", ws, ", previous_collection_time: ", pc, ", fetch_operand: ", fo, ", interval: ", interval, ", previous_interval_stop_time: ", pis, ", collecting: ", cll)
end julia> include("test.jl")
[ Info: Initializing simulation...
[ Info: Iter: 0, time: 0 seconds , u-avg: 0.0, window_start_time: 0.0, window_start_iteration: 0, previous_collection_time: 0.0, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.0, collecting: true
[ Info: ... simulation initialization complete (257.729 ms)
[ Info: Executing initial time step...
[ Info: ... initial time step complete (1.212 seconds).
[ Info: Iter: 1, time: 10 ms , u-avg: 0.0031413629825035438, window_start_time: 0.0, window_start_iteration: 0, previous_collection_time: 0.01, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.0, collecting: true
[ Info: Iter: 2, time: 20 ms , u-avg: 0.007851828677677537, window_start_time: 0.0, window_start_iteration: 0, previous_collection_time: 0.02, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.0, collecting: true
[ Info: Iter: 3, time: 30 ms , u-avg: 0.014652384734839867, window_start_time: 0.0, window_start_iteration: 0, previous_collection_time: 0.03, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.03, collecting: false
[ Info: Iter: 4, time: 40 ms , u-avg: 0.0, window_start_time: 0.04, window_start_iteration: 4, previous_collection_time: 0.04, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.03, collecting: true
[ Info: Iter: 5, time: 50 ms , u-avg: 0.07837860105159718, window_start_time: 0.04, window_start_iteration: 4, previous_collection_time: 0.05, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.03, collecting: true
[ Info: Iter: 6, time: 60 ms , u-avg: 0.09557081820157169, window_start_time: 0.04, window_start_iteration: 4, previous_collection_time: 0.06, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.06, collecting: false
[ Info: Iter: 7, time: 70.000 ms , u-avg: 0.0, window_start_time: 0.06999999999999999, window_start_iteration: 7, previous_collection_time: 0.06999999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.06, collecting: true
[ Info: Iter: 8, time: 80.000 ms , u-avg: 0.20000603541011058, window_start_time: 0.06999999999999999, window_start_iteration: 7, previous_collection_time: 0.07999999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.06, collecting: true
[ Info: Iter: 9, time: 90.000 ms , u-avg: 0.22639226913537686, window_start_time: 0.06999999999999999, window_start_iteration: 7, previous_collection_time: 0.08999999999999998, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.06, collecting: true
[ Info: Iter: 10, time: 90 ms , u-avg: 0.22639226913537688, window_start_time: 0.06999999999999999, window_start_iteration: 7, previous_collection_time: 0.09, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.09, collecting: false
[ Info: Iter: 11, time: 100.000 ms , u-avg: 0.0, window_start_time: 0.09999999999999999, window_start_iteration: 11, previous_collection_time: 0.09999999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.09, collecting: true
[ Info: Iter: 12, time: 110.000 ms , u-avg: 0.3763650237305524, window_start_time: 0.09999999999999999, window_start_iteration: 11, previous_collection_time: 0.10999999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.09, collecting: true
[ Info: Iter: 13, time: 120.000 ms , u-avg: 0.41171106795258217, window_start_time: 0.09999999999999999, window_start_iteration: 11, previous_collection_time: 0.11999999999999998, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.09, collecting: true
[ Info: Iter: 14, time: 120 ms , u-avg: 0.4117110679525822, window_start_time: 0.09999999999999999, window_start_iteration: 11, previous_collection_time: 0.12, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.12, collecting: false
[ Info: Iter: 15, time: 130 ms , u-avg: 0.0, window_start_time: 0.13, window_start_iteration: 15, previous_collection_time: 0.13, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.12, collecting: true
[ Info: Iter: 16, time: 140 ms , u-avg: 0.6058901911474607, window_start_time: 0.13, window_start_iteration: 15, previous_collection_time: 0.14, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.12, collecting: true
[ Info: Iter: 17, time: 150 ms , u-avg: 0.649882311875457, window_start_time: 0.13, window_start_iteration: 15, previous_collection_time: 0.15, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.15, collecting: false
[ Info: Iter: 18, time: 160 ms , u-avg: 0.0, window_start_time: 0.16, window_start_iteration: 18, previous_collection_time: 0.16, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.15, collecting: true
[ Info: Iter: 19, time: 170 ms , u-avg: 0.8865442560258744, window_start_time: 0.16, window_start_iteration: 18, previous_collection_time: 0.17, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.15, collecting: true
[ Info: Iter: 20, time: 180 ms , u-avg: 0.9387919760818845, window_start_time: 0.16, window_start_iteration: 18, previous_collection_time: 0.18, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.18, collecting: false
[ Info: Iter: 21, time: 190 ms , u-avg: 0.0, window_start_time: 0.19, window_start_iteration: 21, previous_collection_time: 0.19, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.18, collecting: true
[ Info: Iter: 22, time: 200 ms , u-avg: 1.2158361130173538, window_start_time: 0.19, window_start_iteration: 21, previous_collection_time: 0.2, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.18, collecting: true
[ Info: Iter: 23, time: 210 ms , u-avg: 1.2758756779393434, window_start_time: 0.19, window_start_iteration: 21, previous_collection_time: 0.21, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.21, collecting: false
[ Info: Iter: 24, time: 220 ms , u-avg: 0.0, window_start_time: 0.22, window_start_iteration: 24, previous_collection_time: 0.22, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.21, collecting: true
[ Info: Iter: 25, time: 230 ms , u-avg: 1.5908429442874084, window_start_time: 0.22, window_start_iteration: 24, previous_collection_time: 0.23, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.21, collecting: true
[ Info: Iter: 26, time: 240 ms , u-avg: 1.6581414386466988, window_start_time: 0.22, window_start_iteration: 24, previous_collection_time: 0.24, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.24, collecting: false
[ Info: Iter: 27, time: 250 ms , u-avg: 0.0, window_start_time: 0.25, window_start_iteration: 27, previous_collection_time: 0.25, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.24, collecting: true
[ Info: Iter: 28, time: 260 ms , u-avg: 2.008236162653511, window_start_time: 0.25, window_start_iteration: 27, previous_collection_time: 0.26, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.24, collecting: true
[ Info: Iter: 29, time: 270 ms , u-avg: 2.082196240249851, window_start_time: 0.25, window_start_iteration: 27, previous_collection_time: 0.27, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.27, collecting: false
[ Info: Iter: 30, time: 280 ms , u-avg: 0.0, window_start_time: 0.28, window_start_iteration: 30, previous_collection_time: 0.28, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.27, collecting: true
[ Info: Iter: 31, time: 290.000 ms , u-avg: 2.464310956360575, window_start_time: 0.28, window_start_iteration: 30, previous_collection_time: 0.29000000000000004, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.27, collecting: true
[ Info: Iter: 32, time: 300 ms , u-avg: 2.5442761423093008, window_start_time: 0.28, window_start_iteration: 30, previous_collection_time: 0.3, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.27, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 33, time: 310 ms , u-avg: 2.6254722813218105, window_start_time: 0.28, window_start_iteration: 30, previous_collection_time: 0.31, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.3, collecting: false
[ Info: Iter: 34, time: 320 ms , u-avg: 0.0, window_start_time: 0.32, window_start_iteration: 34, previous_collection_time: 0.32, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.3, collecting: true
[ Info: Iter: 35, time: 330.000 ms , u-avg: 3.1255402085515462, window_start_time: 0.32, window_start_iteration: 34, previous_collection_time: 0.32999999999999996, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.3, collecting: false
[ Info: Iter: 36, time: 340.000 ms , u-avg: 0.0, window_start_time: 0.33999999999999997, window_start_iteration: 36, previous_collection_time: 0.33999999999999997, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.3, collecting: true
[ Info: Iter: 37, time: 350 ms , u-avg: 3.476005252456648, window_start_time: 0.33999999999999997, window_start_iteration: 36, previous_collection_time: 0.35, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.32999999999999996, collecting: false
[ Info: Iter: 38, time: 360 ms , u-avg: 0.0, window_start_time: 0.36, window_start_iteration: 38, previous_collection_time: 0.36, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.32999999999999996, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 39, time: 370 ms , u-avg: 3.837876565646086, window_start_time: 0.36, window_start_iteration: 38, previous_collection_time: 0.37, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.36, collecting: false
[ Info: Iter: 40, time: 380 ms , u-avg: 0.0, window_start_time: 0.38, window_start_iteration: 40, previous_collection_time: 0.38, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.36, collecting: true
[ Info: Iter: 41, time: 390 ms , u-avg: 4.209726007369981, window_start_time: 0.38, window_start_iteration: 40, previous_collection_time: 0.39, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.39, collecting: false
[ Info: Iter: 42, time: 400 ms , u-avg: 0.0, window_start_time: 0.4, window_start_iteration: 42, previous_collection_time: 0.4, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.39, collecting: true
[ Info: Iter: 43, time: 410.000 ms , u-avg: 4.590086057763697, window_start_time: 0.4, window_start_iteration: 42, previous_collection_time: 0.41000000000000003, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.39, collecting: true
[ Info: Iter: 44, time: 420 ms , u-avg: 4.6865378368773, window_start_time: 0.4, window_start_iteration: 42, previous_collection_time: 0.42, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.39, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 45, time: 430 ms , u-avg: 4.783510427744303, window_start_time: 0.4, window_start_iteration: 42, previous_collection_time: 0.43, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.42, collecting: false
[ Info: Iter: 46, time: 440 ms , u-avg: 0.0, window_start_time: 0.44, window_start_iteration: 46, previous_collection_time: 0.44, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.42, collecting: true
[ Info: Iter: 47, time: 450.000 ms , u-avg: 5.370305891865513, window_start_time: 0.44, window_start_iteration: 46, previous_collection_time: 0.44999999999999996, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.42, collecting: false
[ Info: Iter: 48, time: 460.000 ms , u-avg: 0.0, window_start_time: 0.45999999999999996, window_start_iteration: 48, previous_collection_time: 0.45999999999999996, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.42, collecting: true
[ Info: Iter: 49, time: 470 ms , u-avg: 5.76708650433695, window_start_time: 0.45999999999999996, window_start_iteration: 48, previous_collection_time: 0.47, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.44999999999999996, collecting: false
[ Info: Iter: 50, time: 480 ms , u-avg: 0.0, window_start_time: 0.48, window_start_iteration: 50, previous_collection_time: 0.48, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.44999999999999996, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 51, time: 490 ms , u-avg: 6.166231535087016, window_start_time: 0.48, window_start_iteration: 50, previous_collection_time: 0.49, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.48, collecting: false
[ Info: Iter: 52, time: 500 ms , u-avg: 0.0, window_start_time: 0.5, window_start_iteration: 52, previous_collection_time: 0.5, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.48, collecting: true
[ Info: Iter: 53, time: 510 ms , u-avg: 6.566165741031355, window_start_time: 0.5, window_start_iteration: 52, previous_collection_time: 0.51, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.51, collecting: false
[ Info: Iter: 54, time: 520 ms , u-avg: 0.0, window_start_time: 0.52, window_start_iteration: 54, previous_collection_time: 0.52, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.51, collecting: true
[ Info: Iter: 55, time: 530 ms , u-avg: 6.965310764571667, window_start_time: 0.52, window_start_iteration: 54, previous_collection_time: 0.53, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.51, collecting: true
[ Info: Iter: 56, time: 540 ms , u-avg: 7.064702771106775, window_start_time: 0.52, window_start_iteration: 54, previous_collection_time: 0.54, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.54, collecting: false
[ Info: Iter: 57, time: 550 ms , u-avg: 0.0, window_start_time: 0.55, window_start_iteration: 57, previous_collection_time: 0.55, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.54, collecting: true
[ Info: Iter: 58, time: 560 ms , u-avg: 7.559105120877197, window_start_time: 0.55, window_start_iteration: 57, previous_collection_time: 0.56, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.54, collecting: true
[ Info: Iter: 59, time: 570 ms , u-avg: 7.657023372200447, window_start_time: 0.55, window_start_iteration: 57, previous_collection_time: 0.57, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.54, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 60, time: 580 ms , u-avg: 7.7544847826693255, window_start_time: 0.55, window_start_iteration: 57, previous_collection_time: 0.58, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.57, collecting: false
[ Info: Iter: 61, time: 590 ms , u-avg: 0.0, window_start_time: 0.59, window_start_iteration: 61, previous_collection_time: 0.59, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.57, collecting: true
[ Info: Iter: 62, time: 600 ms , u-avg: 8.333461880244451, window_start_time: 0.59, window_start_iteration: 61, previous_collection_time: 0.6, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.6, collecting: false
[ Info: Iter: 63, time: 610 ms , u-avg: 0.0, window_start_time: 0.61, window_start_iteration: 63, previous_collection_time: 0.61, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.6, collecting: true
[ Info: Iter: 64, time: 620 ms , u-avg: 8.709752264017125, window_start_time: 0.61, window_start_iteration: 63, previous_collection_time: 0.62, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.6, collecting: true
[ Info: Iter: 65, time: 630 ms , u-avg: 8.802136412533468, window_start_time: 0.61, window_start_iteration: 63, previous_collection_time: 0.63, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.63, collecting: false
[ Info: Iter: 66, time: 640 ms , u-avg: 0.0, window_start_time: 0.64, window_start_iteration: 66, previous_collection_time: 0.64, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.63, collecting: true
[ Info: Iter: 67, time: 650 ms , u-avg: 9.256391825350283, window_start_time: 0.64, window_start_iteration: 66, previous_collection_time: 0.65, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.63, collecting: true
[ Info: Iter: 68, time: 660.000 ms , u-avg: 9.344764747359864, window_start_time: 0.64, window_start_iteration: 66, previous_collection_time: 0.6599999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.63, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 69, time: 670.000 ms , u-avg: 9.432128769553014, window_start_time: 0.64, window_start_iteration: 66, previous_collection_time: 0.6699999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.6599999999999999, collecting: false
[ Info: Iter: 70, time: 680.000 ms , u-avg: 0.0, window_start_time: 0.6799999999999999, window_start_iteration: 70, previous_collection_time: 0.6799999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.6599999999999999, collecting: true
[ Info: Iter: 71, time: 690 ms , u-avg: 9.944532401619153, window_start_time: 0.6799999999999999, window_start_iteration: 70, previous_collection_time: 0.69, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.6599999999999999, collecting: false
[ Info: Iter: 72, time: 700 ms , u-avg: 0.0, window_start_time: 0.7, window_start_iteration: 72, previous_collection_time: 0.7, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.6599999999999999, collecting: true
[ Info: Iter: 73, time: 710 ms , u-avg: 10.268085937114538, window_start_time: 0.7, window_start_iteration: 72, previous_collection_time: 0.71, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.69, collecting: false
[ Info: Iter: 74, time: 720 ms , u-avg: 0.0, window_start_time: 0.72, window_start_iteration: 74, previous_collection_time: 0.72, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.69, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 75, time: 730 ms , u-avg: 10.576240502438422, window_start_time: 0.72, window_start_iteration: 74, previous_collection_time: 0.73, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.72, collecting: false
[ Info: Iter: 76, time: 740 ms , u-avg: 0.0, window_start_time: 0.74, window_start_iteration: 76, previous_collection_time: 0.74, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.72, collecting: true
[ Info: Iter: 77, time: 750 ms , u-avg: 10.867779952303898, window_start_time: 0.74, window_start_iteration: 76, previous_collection_time: 0.75, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.75, collecting: false
[ Info: Iter: 78, time: 760 ms , u-avg: 0.0, window_start_time: 0.76, window_start_iteration: 78, previous_collection_time: 0.76, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.75, collecting: true
[ Info: Iter: 79, time: 770 ms , u-avg: 11.141553713694051, window_start_time: 0.76, window_start_iteration: 78, previous_collection_time: 0.77, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.75, collecting: true
[ Info: Iter: 80, time: 780 ms , u-avg: 11.206495836906438, window_start_time: 0.76, window_start_iteration: 78, previous_collection_time: 0.78, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.78, collecting: false
[ Info: Iter: 81, time: 790 ms , u-avg: 0.0, window_start_time: 0.79, window_start_iteration: 81, previous_collection_time: 0.79, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.78, collecting: true
[ Info: Iter: 82, time: 800 ms , u-avg: 11.516560410575497, window_start_time: 0.79, window_start_iteration: 81, previous_collection_time: 0.8, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.78, collecting: true
[ Info: Iter: 83, time: 810.000 ms , u-avg: 11.574058559459141, window_start_time: 0.79, window_start_iteration: 81, previous_collection_time: 0.8099999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.78, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 84, time: 820 ms , u-avg: 11.62982461713531, window_start_time: 0.79, window_start_iteration: 81, previous_collection_time: 0.82, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.8099999999999999, collecting: false
[ Info: Iter: 85, time: 830 ms , u-avg: 0.0, window_start_time: 0.83, window_start_iteration: 85, previous_collection_time: 0.83, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.8099999999999999, collecting: true
[ Info: Iter: 86, time: 840 ms , u-avg: 11.944939757472355, window_start_time: 0.83, window_start_iteration: 85, previous_collection_time: 0.84, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.84, collecting: false
[ Info: Iter: 87, time: 850 ms , u-avg: 0.0, window_start_time: 0.85, window_start_iteration: 87, previous_collection_time: 0.85, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.84, collecting: true
[ Info: Iter: 88, time: 860 ms , u-avg: 12.126506036338247, window_start_time: 0.85, window_start_iteration: 87, previous_collection_time: 0.86, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.84, collecting: true
[ Info: Iter: 89, time: 870 ms , u-avg: 12.167655766858058, window_start_time: 0.85, window_start_iteration: 87, previous_collection_time: 0.87, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.87, collecting: false
[ Info: Iter: 90, time: 880 ms , u-avg: 0.0, window_start_time: 0.88, window_start_iteration: 90, previous_collection_time: 0.88, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.87, collecting: true
[ Info: Iter: 91, time: 890 ms , u-avg: 12.356031044664883, window_start_time: 0.88, window_start_iteration: 90, previous_collection_time: 0.89, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.87, collecting: true
[ Info: Iter: 92, time: 900.000 ms , u-avg: 12.388421440868372, window_start_time: 0.88, window_start_iteration: 90, previous_collection_time: 0.8999999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.87, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 93, time: 910.000 ms , u-avg: 12.418820112689675, window_start_time: 0.88, window_start_iteration: 90, previous_collection_time: 0.9099999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.8999999999999999, collecting: false
[ Info: Iter: 94, time: 920.000 ms , u-avg: 0.0, window_start_time: 0.9199999999999999, window_start_iteration: 94, previous_collection_time: 0.9199999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.8999999999999999, collecting: true
[ Info: Iter: 95, time: 930.000 ms , u-avg: 12.579076993274457, window_start_time: 0.9199999999999999, window_start_iteration: 94, previous_collection_time: 0.9299999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.8999999999999999, collecting: false
[ Info: Iter: 96, time: 940 ms , u-avg: 0.0, window_start_time: 0.94, window_start_iteration: 96, previous_collection_time: 0.94, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.8999999999999999, collecting: true
[ Info: Iter: 97, time: 950 ms , u-avg: 12.654017134137858, window_start_time: 0.94, window_start_iteration: 96, previous_collection_time: 0.95, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.9299999999999999, collecting: false
[ Info: Iter: 98, time: 960 ms , u-avg: 0.0, window_start_time: 0.96, window_start_iteration: 98, previous_collection_time: 0.96, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.9299999999999999, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 99, time: 970 ms , u-avg: 12.704142124423381, window_start_time: 0.96, window_start_iteration: 98, previous_collection_time: 0.97, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.96, collecting: false
[ Info: Iter: 100, time: 980 ms , u-avg: 0.0, window_start_time: 0.98, window_start_iteration: 100, previous_collection_time: 0.98, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.96, collecting: true
[ Info: Iter: 101, time: 990 ms , u-avg: 12.729254143694298, window_start_time: 0.98, window_start_iteration: 100, previous_collection_time: 0.99, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.99, collecting: false
[ Info: Iter: 102, time: 1 second , u-avg: 0.0, window_start_time: 1.0, window_start_iteration: 102, previous_collection_time: 1.0, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.99, collecting: true
[ Info: Iter: 103, time: 1.010 seconds , u-avg: 12.729254086283136, window_start_time: 1.0, window_start_iteration: 102, previous_collection_time: 1.01, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.99, collecting: true
[ Info: Iter: 104, time: 1.020 seconds , u-avg: 12.724543620587964, window_start_time: 1.0, window_start_iteration: 102, previous_collection_time: 1.02, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.02, collecting: false
[ Info: Iter: 105, time: 1.030 seconds , u-avg: 0.0, window_start_time: 1.03, window_start_iteration: 105, previous_collection_time: 1.03, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.02, collecting: true
[ Info: Iter: 106, time: 1.040 seconds , u-avg: 12.682195964176623, window_start_time: 1.03, window_start_iteration: 105, previous_collection_time: 1.04, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.02, collecting: true
[ Info: Iter: 107, time: 1.050 seconds , u-avg: 12.668106406195333, window_start_time: 1.03, window_start_iteration: 105, previous_collection_time: 1.05, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.05, collecting: false
[ Info: Iter: 108, time: 1.060 seconds , u-avg: 0.0, window_start_time: 1.06, window_start_iteration: 108, previous_collection_time: 1.06, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.05, collecting: true
[ Info: Iter: 109, time: 1.070 seconds , u-avg: 12.579076594562144, window_start_time: 1.06, window_start_iteration: 108, previous_collection_time: 1.07, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.05, collecting: true
[ Info: Iter: 110, time: 1.080 seconds , u-avg: 12.555733004208836, window_start_time: 1.06, window_start_iteration: 108, previous_collection_time: 1.08, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.08, collecting: false
[ Info: Iter: 111, time: 1.090 seconds , u-avg: 0.0, window_start_time: 1.09, window_start_iteration: 111, previous_collection_time: 1.09, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.08, collecting: true
[ Info: Iter: 112, time: 1.100 seconds , u-avg: 12.420811272264622, window_start_time: 1.09, window_start_iteration: 111, previous_collection_time: 1.1, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.08, collecting: true
[ Info: Iter: 113, time: 1.110 seconds , u-avg: 12.388420848899857, window_start_time: 1.09, window_start_iteration: 111, previous_collection_time: 1.1099999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.08, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 114, time: 1.120 seconds , u-avg: 12.354060011630247, window_start_time: 1.09, window_start_iteration: 111, previous_collection_time: 1.1199999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.1099999999999999, collecting: false
[ Info: Iter: 115, time: 1.130 seconds , u-avg: 0.0, window_start_time: 1.13, window_start_iteration: 115, previous_collection_time: 1.13, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.1099999999999999, collecting: true
[ Info: Iter: 116, time: 1.140 seconds , u-avg: 12.126505258118181, window_start_time: 1.13, window_start_iteration: 115, previous_collection_time: 1.14, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.1099999999999999, collecting: false
[ Info: Iter: 117, time: 1.150 seconds , u-avg: 0.0, window_start_time: 1.15, window_start_iteration: 117, previous_collection_time: 1.15, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.1099999999999999, collecting: true
[ Info: Iter: 118, time: 1.160 seconds , u-avg: 11.944938876944853, window_start_time: 1.15, window_start_iteration: 117, previous_collection_time: 1.16, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.14, collecting: false
[ Info: Iter: 119, time: 1.170 seconds , u-avg: 0.0, window_start_time: 1.17, window_start_iteration: 119, previous_collection_time: 1.17, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.14, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 120, time: 1.180 seconds , u-avg: 11.741355753127747, window_start_time: 1.17, window_start_iteration: 119, previous_collection_time: 1.18, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.17, collecting: false
[ Info: Iter: 121, time: 1.190 seconds , u-avg: 0.0, window_start_time: 1.19, window_start_iteration: 121, previous_collection_time: 1.19, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.17, collecting: true
[ Info: Iter: 122, time: 1.200 seconds , u-avg: 11.51655933624829, window_start_time: 1.19, window_start_iteration: 121, previous_collection_time: 1.2, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.2, collecting: false
[ Info: Iter: 123, time: 1.210 seconds , u-avg: 0.0, window_start_time: 1.21, window_start_iteration: 123, previous_collection_time: 1.21, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.2, collecting: true
[ Info: Iter: 124, time: 1.220 seconds , u-avg: 11.271436795064185, window_start_time: 1.21, window_start_iteration: 123, previous_collection_time: 1.22, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.2, collecting: true
[ Info: Iter: 125, time: 1.230 seconds , u-avg: 11.206494650021211, window_start_time: 1.21, window_start_iteration: 123, previous_collection_time: 1.23, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.23, collecting: false
[ Info: Iter: 126, time: 1.240 seconds , u-avg: 0.0, window_start_time: 1.24, window_start_iteration: 126, previous_collection_time: 1.24, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.23, collecting: true
[ Info: Iter: 127, time: 1.250 seconds , u-avg: 10.867778659886214, window_start_time: 1.24, window_start_iteration: 126, previous_collection_time: 1.25, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.23, collecting: true
[ Info: Iter: 128, time: 1.260 seconds , u-avg: 10.795968973249174, window_start_time: 1.24, window_start_iteration: 126, previous_collection_time: 1.26, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.26, collecting: false
[ Info: Iter: 129, time: 1.270 seconds , u-avg: 0.0, window_start_time: 1.27, window_start_iteration: 129, previous_collection_time: 1.27, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.26, collecting: true
[ Info: Iter: 130, time: 1.280 seconds , u-avg: 10.424164173642733, window_start_time: 1.27, window_start_iteration: 129, previous_collection_time: 1.28, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.26, collecting: true
[ Info: Iter: 131, time: 1.290 seconds , u-avg: 10.3461243332739, window_start_time: 1.27, window_start_iteration: 129, previous_collection_time: 1.29, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.29, collecting: false
[ Info: Iter: 132, time: 1.300 seconds , u-avg: 0.0, window_start_time: 1.3, window_start_iteration: 132, previous_collection_time: 1.3, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.29, collecting: true
[ Info: Iter: 133, time: 1.310 seconds , u-avg: 9.944530889918811, window_start_time: 1.3, window_start_iteration: 132, previous_collection_time: 1.31, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.29, collecting: true
[ Info: Iter: 134, time: 1.320 seconds , u-avg: 9.860953582966044, window_start_time: 1.3, window_start_iteration: 132, previous_collection_time: 1.3199999999999998, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.29, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 135, time: 1.330 seconds , u-avg: 9.776254135548728, window_start_time: 1.3, window_start_iteration: 132, previous_collection_time: 1.3299999999999998, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.3199999999999998, collecting: false
[ Info: Iter: 136, time: 1.340 seconds , u-avg: 0.0, window_start_time: 1.3399999999999999, window_start_iteration: 136, previous_collection_time: 1.3399999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.3199999999999998, collecting: true
[ Info: Iter: 137, time: 1.350 seconds , u-avg: 9.256390196808994, window_start_time: 1.3399999999999999, window_start_iteration: 136, previous_collection_time: 1.3499999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.3199999999999998, collecting: false
[ Info: Iter: 138, time: 1.360 seconds , u-avg: 0.0, window_start_time: 1.3599999999999999, window_start_iteration: 138, previous_collection_time: 1.3599999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.3199999999999998, collecting: true
[ Info: Iter: 139, time: 1.370 seconds , u-avg: 8.894518883619556, window_start_time: 1.3599999999999999, window_start_iteration: 138, previous_collection_time: 1.3699999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.3499999999999999, collecting: false
[ Info: Iter: 140, time: 1.380 seconds , u-avg: 0.0, window_start_time: 1.38, window_start_iteration: 140, previous_collection_time: 1.38, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.3499999999999999, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 141, time: 1.390 seconds , u-avg: 8.522669441895662, window_start_time: 1.38, window_start_iteration: 140, previous_collection_time: 1.39, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.38, collecting: false
[ Info: Iter: 142, time: 1.400 seconds , u-avg: 0.0, window_start_time: 1.4, window_start_iteration: 142, previous_collection_time: 1.4, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.38, collecting: true
[ Info: Iter: 143, time: 1.410 seconds , u-avg: 8.142309391501945, window_start_time: 1.4, window_start_iteration: 142, previous_collection_time: 1.41, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.38, collecting: false
[ Info: Iter: 144, time: 1.420 seconds , u-avg: 0.0, window_start_time: 1.42, window_start_iteration: 144, previous_collection_time: 1.42, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.38, collecting: true
[ Info: Iter: 145, time: 1.430 seconds , u-avg: 7.754939839787331, window_start_time: 1.42, window_start_iteration: 144, previous_collection_time: 1.43, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.41, collecting: false
[ Info: Iter: 146, time: 1.440 seconds , u-avg: 0.0, window_start_time: 1.44, window_start_iteration: 146, previous_collection_time: 1.44, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.41, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 147, time: 1.450 seconds , u-avg: 7.362089557400126, window_start_time: 1.44, window_start_iteration: 146, previous_collection_time: 1.45, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.44, collecting: false
[ Info: Iter: 148, time: 1.460 seconds , u-avg: 0.0, window_start_time: 1.46, window_start_iteration: 148, previous_collection_time: 1.46, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.44, collecting: true
[ Info: Iter: 149, time: 1.470 seconds , u-avg: 6.965308944928689, window_start_time: 1.46, window_start_iteration: 148, previous_collection_time: 1.47, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.47, collecting: false
[ Info: Iter: 150, time: 1.480 seconds , u-avg: 0.0, window_start_time: 1.48, window_start_iteration: 150, previous_collection_time: 1.48, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.47, collecting: true
[ Info: Iter: 151, time: 1.490 seconds , u-avg: 6.566163914178623, window_start_time: 1.48, window_start_iteration: 150, previous_collection_time: 1.49, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.47, collecting: true
[ Info: Iter: 152, time: 1.500 seconds , u-avg: 6.466180362467067, window_start_time: 1.48, window_start_iteration: 150, previous_collection_time: 1.5, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.5, collecting: false
[ Info: Iter: 153, time: 1.510 seconds , u-avg: 0.0, window_start_time: 1.51, window_start_iteration: 153, previous_collection_time: 1.51, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.5, collecting: true
[ Info: Iter: 154, time: 1.520 seconds , u-avg: 5.966459950002811, window_start_time: 1.51, window_start_iteration: 153, previous_collection_time: 1.52, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.5, collecting: true
[ Info: Iter: 155, time: 1.530 seconds , u-avg: 5.866772317348391, window_start_time: 1.51, window_start_iteration: 153, previous_collection_time: 1.53, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.53, collecting: false
[ Info: Iter: 156, time: 1.540 seconds , u-avg: 0.0, window_start_time: 1.54, window_start_iteration: 156, previous_collection_time: 1.54, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.53, collecting: true
[ Info: Iter: 157, time: 1.550 seconds , u-avg: 5.370304086613586, window_start_time: 1.54, window_start_iteration: 156, previous_collection_time: 1.55, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.53, collecting: true
[ Info: Iter: 158, time: 1.560 seconds , u-avg: 5.271797207501014, window_start_time: 1.54, window_start_iteration: 156, previous_collection_time: 1.56, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.56, collecting: false
[ Info: Iter: 159, time: 1.570 seconds , u-avg: 0.0, window_start_time: 1.57, window_start_iteration: 159, previous_collection_time: 1.57, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.56, collecting: true
[ Info: Iter: 160, time: 1.580 seconds , u-avg: 4.782987845658554, window_start_time: 1.57, window_start_iteration: 159, previous_collection_time: 1.58, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.56, collecting: true
[ Info: Iter: 161, time: 1.590 seconds , u-avg: 4.686536074120517, window_start_time: 1.57, window_start_iteration: 159, previous_collection_time: 1.5899999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.56, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 162, time: 1.600 seconds , u-avg: 4.59066857242074, window_start_time: 1.57, window_start_iteration: 159, previous_collection_time: 1.5999999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.5899999999999999, collecting: false
[ Info: Iter: 163, time: 1.610 seconds , u-avg: 0.0, window_start_time: 1.6099999999999999, window_start_iteration: 163, previous_collection_time: 1.6099999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.5899999999999999, collecting: true
[ Info: Iter: 164, time: 1.620 seconds , u-avg: 4.022643185248518, window_start_time: 1.6099999999999999, window_start_iteration: 163, previous_collection_time: 1.6199999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.5899999999999999, collecting: false
[ Info: Iter: 165, time: 1.630 seconds , u-avg: 0.0, window_start_time: 1.63, window_start_iteration: 165, previous_collection_time: 1.63, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.5899999999999999, collecting: true
[ Info: Iter: 166, time: 1.640 seconds , u-avg: 3.6556017405745704, window_start_time: 1.63, window_start_iteration: 165, previous_collection_time: 1.64, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.6199999999999999, collecting: false
[ Info: Iter: 167, time: 1.650 seconds , u-avg: 0.0, window_start_time: 1.65, window_start_iteration: 167, previous_collection_time: 1.65, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.6199999999999999, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 168, time: 1.660 seconds , u-avg: 3.2992577798961906, window_start_time: 1.65, window_start_iteration: 167, previous_collection_time: 1.66, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.65, collecting: false
[ Info: Iter: 169, time: 1.670 seconds , u-avg: 0.0, window_start_time: 1.67, window_start_iteration: 169, previous_collection_time: 1.67, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.65, collecting: true
[ Info: Iter: 170, time: 1.680 seconds , u-avg: 2.9550176300281055, window_start_time: 1.67, window_start_iteration: 169, previous_collection_time: 1.68, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.68, collecting: false
[ Info: Iter: 171, time: 1.690 seconds , u-avg: 0.0, window_start_time: 1.69, window_start_iteration: 171, previous_collection_time: 1.69, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.68, collecting: true
[ Info: Iter: 172, time: 1.700 seconds , u-avg: 2.6242398495734784, window_start_time: 1.69, window_start_iteration: 171, previous_collection_time: 1.7, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.68, collecting: true
[ Info: Iter: 173, time: 1.710 seconds , u-avg: 2.54427468086229, window_start_time: 1.69, window_start_iteration: 171, previous_collection_time: 1.71, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.71, collecting: false
[ Info: Iter: 174, time: 1.720 seconds , u-avg: 0.0, window_start_time: 1.72, window_start_iteration: 174, previous_collection_time: 1.72, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.71, collecting: true
[ Info: Iter: 175, time: 1.730 seconds , u-avg: 2.1561549468272188, window_start_time: 1.72, window_start_iteration: 174, previous_collection_time: 1.73, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.71, collecting: true
[ Info: Iter: 176, time: 1.740 seconds , u-avg: 2.0821948885524773, window_start_time: 1.72, window_start_iteration: 174, previous_collection_time: 1.74, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.74, collecting: false
[ Info: Iter: 177, time: 1.750 seconds , u-avg: 0.0, window_start_time: 1.75, window_start_iteration: 177, previous_collection_time: 1.75, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.74, collecting: true
[ Info: Iter: 178, time: 1.760 seconds , u-avg: 1.7254386818218563, window_start_time: 1.75, window_start_iteration: 177, previous_collection_time: 1.76, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.74, collecting: true
[ Info: Iter: 179, time: 1.770 seconds , u-avg: 1.6581402086967247, window_start_time: 1.75, window_start_iteration: 177, previous_collection_time: 1.77, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.77, collecting: false
[ Info: Iter: 180, time: 1.780 seconds , u-avg: 0.0, window_start_time: 1.78, window_start_iteration: 180, previous_collection_time: 1.78, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.77, collecting: true
[ Info: Iter: 181, time: 1.790 seconds , u-avg: 1.3359141226176374, window_start_time: 1.78, window_start_iteration: 180, previous_collection_time: 1.79, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.77, collecting: true
[ Info: Iter: 182, time: 1.800 seconds , u-avg: 1.2758745806538938, window_start_time: 1.78, window_start_iteration: 180, previous_collection_time: 1.7999999999999998, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.77, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 183, time: 1.810 seconds , u-avg: 1.217529300743549, window_start_time: 1.78, window_start_iteration: 180, previous_collection_time: 1.8099999999999998, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.7999999999999998, collecting: false
[ Info: Iter: 184, time: 1.820 seconds , u-avg: 0.0, window_start_time: 1.8199999999999998, window_start_iteration: 184, previous_collection_time: 1.8199999999999998, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.7999999999999998, collecting: true
[ Info: Iter: 185, time: 1.830 seconds , u-avg: 0.8865433256230806, window_start_time: 1.8199999999999998, window_start_iteration: 184, previous_collection_time: 1.8299999999999998, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.7999999999999998, collecting: false
[ Info: Iter: 186, time: 1.840 seconds , u-avg: 0.0, window_start_time: 1.8399999999999999, window_start_iteration: 186, previous_collection_time: 1.8399999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.7999999999999998, collecting: true
[ Info: Iter: 187, time: 1.850 seconds , u-avg: 0.6938736028202257, window_start_time: 1.8399999999999999, window_start_iteration: 186, previous_collection_time: 1.8499999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.8299999999999998, collecting: false
[ Info: Iter: 188, time: 1.860 seconds , u-avg: 0.0, window_start_time: 1.8599999999999999, window_start_iteration: 188, previous_collection_time: 1.8599999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.8299999999999998, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 189, time: 1.870 seconds , u-avg: 0.5235899518877757, window_start_time: 1.8599999999999999, window_start_iteration: 188, previous_collection_time: 1.8699999999999999, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.8599999999999999, collecting: false
[ Info: Iter: 190, time: 1.880 seconds , u-avg: 0.0, window_start_time: 1.88, window_start_iteration: 190, previous_collection_time: 1.88, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.8599999999999999, collecting: true
[ Info: Iter: 191, time: 1.890 seconds , u-avg: 0.3763644046007603, window_start_time: 1.88, window_start_iteration: 190, previous_collection_time: 1.89, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.8599999999999999, collecting: false
[ Info: Iter: 192, time: 1.900 seconds , u-avg: 0.0, window_start_time: 1.9, window_start_iteration: 192, previous_collection_time: 1.9, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.8599999999999999, collecting: true
[ Info: Iter: 193, time: 1.910 seconds , u-avg: 0.252777992933367, window_start_time: 1.9, window_start_iteration: 192, previous_collection_time: 1.91, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.89, collecting: false
[ Info: Iter: 194, time: 1.920 seconds , u-avg: 0.0, window_start_time: 1.92, window_start_iteration: 194, previous_collection_time: 1.92, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.89, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/development/Oceananigans.jl/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 195, time: 1.930 seconds , u-avg: 0.15331845599118626, window_start_time: 1.92, window_start_iteration: 194, previous_collection_time: 1.93, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.92, collecting: false
[ Info: Iter: 196, time: 1.940 seconds , u-avg: 0.0, window_start_time: 1.94, window_start_iteration: 196, previous_collection_time: 1.94, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.92, collecting: true
[ Info: Iter: 197, time: 1.950 seconds , u-avg: 0.07837831512778862, window_start_time: 1.94, window_start_iteration: 196, previous_collection_time: 1.95, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.95, collecting: false
[ Info: Iter: 198, time: 1.960 seconds , u-avg: 0.0, window_start_time: 1.96, window_start_iteration: 198, previous_collection_time: 1.96, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.95, collecting: true
[ Info: Iter: 199, time: 1.970 seconds , u-avg: 0.0282533248422682, window_start_time: 1.96, window_start_iteration: 198, previous_collection_time: 1.97, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.95, collecting: true
[ Info: Iter: 200, time: 1.980 seconds , u-avg: 0.020407752224732935, window_start_time: 1.96, window_start_iteration: 198, previous_collection_time: 1.98, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.98, collecting: false
[ Info: Iter: 201, time: 1.990 seconds , u-avg: 0.0, window_start_time: 1.99, window_start_iteration: 201, previous_collection_time: 1.99, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.98, collecting: true
[ Info: Simulation is stopping after running for 2.041 seconds.
[ Info: Simulation time 2 seconds equals or exceeds stop time 2 seconds.
[ Info: Iter: 202, time: 2 seconds , u-avg: 8.929976983568233e-15, window_start_time: 1.99, window_start_iteration: 201, previous_collection_time: 2.0, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 1.98, collecting: true |
The zeros are written down at iteration 38, 50, 74, 98, 119, 140, 146, 167, 188, 194, and 201 from what I can see. If we find a pattern regarding the parameters of the |
What is going on here? I don't understand it but it looks wrong...
Maybe the |
@simone-silvestri, I think the problems arise earlier than the iterations you highlight, where zeros are written. At first the averages are correctly being collected over three iteration, which makes sense because the interval is equal to three times the timestep.
But after the warnings it goes out of whack:
It looks like between |
Not sure if it helps but the result is zero'd out here: Oceananigans.jl/src/OutputWriters/windowed_time_average.jl Lines 255 to 268 in 406eb9c
so when |
@tomchor uses this or did at some point. Not sure who else has used it but would nice to tag them here to gather a good consensus on tests to write up. |
Yeah, I think the problem is that |
@liuchihl, you tested that this bug is reproduced in which versions of Oceananigans again? |
@hdrake I am using v0.91.5, but v0.90.11 already has this issue, so it doesn't seem like a new problem. |
The problem is this iteration: [ Info: Iter: 32, time: 300 ms , u-avg: 2.5442761423093008, window_start_time: 0.28, window_start_iteration: 30, previous_collection_time: 0.3, fetch_operand: true, interval: 0.03, previous_interval_stop_time: 0.27, collecting: true We should have That means
spuriously returns
So we want to look at Probably Oceananigans.jl/src/Utils/schedules.jl Lines 56 to 61 in 406eb9c
Would be helpful to have a test too. |
It would be great if you are able to take this up! I can help of course. It will also be useful to have fresh eyes on this algorithm. |
I will try to figure it out. |
It does seem like a roundoff error, below is the progress message. At iter:64, for example, [ Info: Initializing simulation...
[ Info: Iter: 0, time: 0 seconds, model clock time:0.0, previous_interval_stop_time + interval:0.1, u-avg: 0.0, window_start_time: 0.0, window_start_iteration: 0, previous_collection_time:0.0, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 0.0, collecting: true
[ Info: ... simulation initialization complete (231.038 ms)
[ Info: Executing initial time step...
[ Info: ... initial time step complete (546.837 ms).
[ Info: Iter: 11, time: 100 ms, model clock time:0.1, previous_interval_stop_time + interval:0.2, u-avg: 0.1202985845671951, window_start_time: 0.0, window_start_iteration: 0, previous_collection_time:0.1, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 0.1, collecting: false
[ Info: Iter: 22, time: 200 ms, model clock time:0.2, previous_interval_stop_time + interval:0.30000000000000004, u-avg: 0.8057920910530502, window_start_time: 0.11, window_start_iteration: 12, previous_collection_time:0.2, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 0.2, collecting: false
[ Info: Iter: 32, time: 300.000 ms, model clock time:0.30000000000000004, previous_interval_stop_time + interval:0.4, u-avg: 2.0225593735128355, window_start_time: 0.21, window_start_iteration: 23, previous_collection_time:0.30000000000000004, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 0.30000000000000004, collecting: false
[ Info: Iter: 42, time: 400 ms, model clock time:0.4, previous_interval_stop_time + interval:0.5, u-avg: 3.664512241688186, window_start_time: 0.3100000000000001, window_start_iteration: 33, previous_collection_time:0.4, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 0.4, collecting: false
[ Info: Iter: 52, time: 500 ms, model clock time:0.5, previous_interval_stop_time + interval:0.5, u-avg: 5.570924908683772, window_start_time: 0.4100000000000001, window_start_iteration: 43, previous_collection_time:0.5, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 0.4, collecting: false
[ Info: Iter: 64, time: 600.000 ms, model clock time:0.6000000000000001, previous_interval_stop_time + interval:0.6, u-avg: 0.0, window_start_time: 0.6000000000000001, window_start_iteration: 64, previous_collection_time:0.6000000000000001, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 0.5, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/.julia/packages/Oceananigans/OMBY0/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 75, time: 700.000 ms, model clock time:0.7000000000000001, previous_interval_stop_time + interval:0.8, u-avg: 9.512220799868613, window_start_time: 0.6199999999999999, window_start_iteration: 66, previous_collection_time:0.7000000000000001, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 0.7000000000000001, collecting: false
[ Info: Iter: 86, time: 800 ms, model clock time:0.8, previous_interval_stop_time + interval:0.9, u-avg: 10.99170409268473, window_start_time: 0.71, window_start_iteration: 76, previous_collection_time:0.8, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 0.8, collecting: false
[ Info: Iter: 97, time: 900 ms, model clock time:0.9, previous_interval_stop_time + interval:0.9, u-avg: 12.107573765048185, window_start_time: 0.8099999999999999, window_start_iteration: 87, previous_collection_time:0.9, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 0.8, collecting: false
[ Info: Iter: 108, time: 1 second, model clock time:1.0, previous_interval_stop_time + interval:1.0, u-avg: 12.691638724405168, window_start_time: 0.9299999999999997, window_start_iteration: 100, previous_collection_time:1.0, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 0.9, collecting: false
[ Info: Iter: 118, time: 1.100 seconds, model clock time:1.1, previous_interval_stop_time + interval:1.2000000000000002, u-avg: 12.56681992191321, window_start_time: 1.0300000000000007, window_start_iteration: 111, previous_collection_time:1.1, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 1.1, collecting: false
[ Info: Iter: 128, time: 1.200 seconds, model clock time:1.2000000000000002, previous_interval_stop_time + interval:1.3000000000000003, u-avg: 11.926603358212661, window_start_time: 1.1100000000000003, window_start_iteration: 119, previous_collection_time:1.2000000000000002, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 1.2000000000000002, collecting: false
[ Info: Iter: 139, time: 1.300 seconds, model clock time:1.3, previous_interval_stop_time + interval:1.3000000000000003, u-avg: 10.709836075752904, window_start_time: 1.2100000000000004, window_start_iteration: 129, previous_collection_time:1.3, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 1.2000000000000002, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/.julia/packages/Oceananigans/OMBY0/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 149, time: 1.400 seconds, model clock time:1.4000000000000001, previous_interval_stop_time + interval:1.5000000000000002, u-avg: 8.9791965740231, window_start_time: 1.3200000000000005, window_start_iteration: 141, previous_collection_time:1.4000000000000001, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 1.4000000000000001, collecting: false
[ Info: Iter: 160, time: 1.500 seconds, model clock time:1.5, previous_interval_stop_time + interval:1.5000000000000002, u-avg: 7.161470540582019, window_start_time: 1.4100000000000004, window_start_iteration: 150, previous_collection_time:1.5, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 1.4000000000000001, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/.julia/packages/Oceananigans/OMBY0/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 170, time: 1.600 seconds, model clock time:1.6, previous_interval_stop_time + interval:1.7000000000000002, u-avg: 5.078554914290692, window_start_time: 1.5200000000000005, window_start_iteration: 162, previous_collection_time:1.6, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 1.6, collecting: false
[ Info: Iter: 180, time: 1.700 seconds, model clock time:1.7000000000000002, previous_interval_stop_time + interval:1.8000000000000003, u-avg: 3.309337820047445, window_start_time: 1.6100000000000003, window_start_iteration: 171, previous_collection_time:1.7000000000000002, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 1.7000000000000002, collecting: false
[ Info: Iter: 191, time: 1.800 seconds, model clock time:1.8, previous_interval_stop_time + interval:1.8000000000000003, u-avg: 1.7406913565812152, window_start_time: 1.7100000000000004, window_start_iteration: 181, previous_collection_time:1.8, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 1.7000000000000002, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/.julia/packages/Oceananigans/OMBY0/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 201, time: 1.900 seconds, model clock time:1.9000000000000001, previous_interval_stop_time + interval:2.0, u-avg: 0.5790445551477348, window_start_time: 1.8200000000000005, window_start_iteration: 193, previous_collection_time:1.9000000000000001, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 1.9000000000000001, collecting: false
[ Info: Iter: 212, time: 2 seconds, model clock time:2.0, previous_interval_stop_time + interval:2.0, u-avg: 0.07095790120929331, window_start_time: 1.9100000000000004, window_start_iteration: 202, previous_collection_time:2.0, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 1.9000000000000001, collecting: false
[ Info: Iter: 223, time: 2.100 seconds, model clock time:2.1000000000000005, previous_interval_stop_time + interval:2.2, u-avg: 0.16557552735280148, window_start_time: 2.0299999999999994, window_start_iteration: 215, previous_collection_time:2.1000000000000005, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 2.1, collecting: false
[ Info: Iter: 234, time: 2.200 seconds, model clock time:2.2, previous_interval_stop_time + interval:2.3000000000000003, u-avg: 0.8057920910533722, window_start_time: 2.1100000000000003, window_start_iteration: 224, previous_collection_time:2.2, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 2.2, collecting: false
[ Info: Iter: 245, time: 2.300 seconds, model clock time:2.3000000000000007, previous_interval_stop_time + interval:2.4000000000000004, u-avg: 2.0225593735131833, window_start_time: 2.21, window_start_iteration: 235, previous_collection_time:2.3000000000000007, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 2.3000000000000003, collecting: false
[ Info: Iter: 256, time: 2.400 seconds, model clock time:2.4000000000000004, previous_interval_stop_time + interval:2.5000000000000004, u-avg: 3.6645122416885725, window_start_time: 2.3100000000000005, window_start_iteration: 246, previous_collection_time:2.4000000000000004, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 2.4000000000000004, collecting: false
[ Info: Iter: 267, time: 2.500 seconds, model clock time:2.5, previous_interval_stop_time + interval:2.5000000000000004, u-avg: 5.570924908684207, window_start_time: 2.41, window_start_iteration: 257, previous_collection_time:2.5, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 2.4000000000000004, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/.julia/packages/Oceananigans/OMBY0/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 278, time: 2.600 seconds, model clock time:2.6000000000000005, previous_interval_stop_time + interval:2.7, u-avg: 7.6538405349756085, window_start_time: 2.5199999999999996, window_start_iteration: 269, previous_collection_time:2.6000000000000005, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 2.6, collecting: false
[ Info: Iter: 289, time: 2.700 seconds, model clock time:2.7, previous_interval_stop_time + interval:2.8000000000000003, u-avg: 9.42305762921893, window_start_time: 2.6100000000000003, window_start_iteration: 279, previous_collection_time:2.7, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 2.7, collecting: false
[ Info: Iter: 300, time: 2.800 seconds, model clock time:2.8000000000000007, previous_interval_stop_time + interval:2.9000000000000004, u-avg: 10.991704092685225, window_start_time: 2.71, window_start_iteration: 290, previous_collection_time:2.8000000000000007, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 2.8000000000000003, collecting: false
[ Info: Iter: 311, time: 2.900 seconds, model clock time:2.9000000000000004, previous_interval_stop_time + interval:3.0000000000000004, u-avg: 12.107573765048683, window_start_time: 2.8100000000000005, window_start_iteration: 301, previous_collection_time:2.9000000000000004, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 2.9000000000000004, collecting: false
[ Info: Iter: 322, time: 3 seconds, model clock time:3.0, previous_interval_stop_time + interval:3.0000000000000004, u-avg: 12.66143754805721, window_start_time: 2.91, window_start_iteration: 312, previous_collection_time:3.0, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 2.9000000000000004, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/.julia/packages/Oceananigans/OMBY0/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 333, time: 3.100 seconds, model clock time:3.1000000000000005, previous_interval_stop_time + interval:3.2, u-avg: 12.583985175726626, window_start_time: 3.0199999999999996, window_start_iteration: 324, previous_collection_time:3.1000000000000005, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 3.1, collecting: false
[ Info: Iter: 344, time: 3.200 seconds, model clock time:3.2, previous_interval_stop_time + interval:3.3000000000000003, u-avg: 11.926603358213136, window_start_time: 3.1100000000000003, window_start_iteration: 334, previous_collection_time:3.2, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 3.2, collecting: false
[ Info: Iter: 355, time: 3.300 seconds, model clock time:3.3000000000000007, previous_interval_stop_time + interval:3.4000000000000004, u-avg: 10.709836075753326, window_start_time: 3.21, window_start_iteration: 345, previous_collection_time:3.3000000000000007, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 3.3000000000000003, collecting: false
[ Info: Iter: 366, time: 3.400 seconds, model clock time:3.4000000000000004, previous_interval_stop_time + interval:3.5000000000000004, u-avg: 9.067883207577935, window_start_time: 3.3100000000000005, window_start_iteration: 356, previous_collection_time:3.4000000000000004, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 3.4000000000000004, collecting: false
[ Info: Iter: 377, time: 3.500 seconds, model clock time:3.5, previous_interval_stop_time + interval:3.5000000000000004, u-avg: 7.161470540582303, window_start_time: 3.41, window_start_iteration: 367, previous_collection_time:3.5, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 3.4000000000000004, collecting: true
┌ Warning: Returning a WindowedTimeAverage before the collection period is complete.
└ @ Oceananigans.OutputWriters ~/.julia/packages/Oceananigans/OMBY0/src/OutputWriters/windowed_time_average.jl:201
[ Info: Iter: 388, time: 3.600 seconds, model clock time:3.6000000000000005, previous_interval_stop_time + interval:3.7, u-avg: 5.078554914290898, window_start_time: 3.5199999999999996, window_start_iteration: 379, previous_collection_time:3.6000000000000005, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 3.6, collecting: false
[ Info: Iter: 399, time: 3.700 seconds, model clock time:3.7, previous_interval_stop_time + interval:3.7, u-avg: 3.309337820047581, window_start_time: 3.6100000000000003, window_start_iteration: 389, previous_collection_time:3.7, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 3.6, collecting: false
[ Info: Iter: 410, time: 3.800 seconds, model clock time:3.8000000000000007, previous_interval_stop_time + interval:3.9000000000000004, u-avg: 1.6002624850128424, window_start_time: 3.7299999999999995, window_start_iteration: 402, previous_collection_time:3.8000000000000007, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 3.8000000000000003, collecting: false
[ Info: Iter: 421, time: 3.900 seconds, model clock time:3.9000000000000004, previous_interval_stop_time + interval:4.0, u-avg: 0.6248216842178228, window_start_time: 3.8100000000000005, window_start_iteration: 411, previous_collection_time:3.9000000000000004, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 3.9000000000000004, collecting: false
[ Info: Iter: 432, time: 4 seconds, model clock time:4.0, previous_interval_stop_time + interval:4.0, u-avg: 0.07095790120929706, window_start_time: 3.91, window_start_iteration: 422, previous_collection_time:4.0, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 3.9000000000000004, collecting: false
[ Info: Iter: 443, time: 4.100 seconds, model clock time:4.1000000000000005, previous_interval_stop_time + interval:4.2, u-avg: 0.16557552735280162, window_start_time: 4.029999999999999, window_start_iteration: 435, previous_collection_time:4.1000000000000005, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 4.1000000000000005, collecting: false
[ Info: Iter: 454, time: 4.200 seconds, model clock time:4.2, previous_interval_stop_time + interval:4.2, u-avg: 0.8057920910533768, window_start_time: 4.11, window_start_iteration: 444, previous_collection_time:4.2, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 4.1000000000000005, collecting: false
[ Info: Iter: 465, time: 4.300 seconds, model clock time:4.3, previous_interval_stop_time + interval:4.3, u-avg: 2.164461823304153, window_start_time: 4.2299999999999995, window_start_iteration: 457, previous_collection_time:4.3, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 4.2, collecting: false
[ Info: Iter: 476, time: 4.400 seconds, model clock time:4.4, previous_interval_stop_time + interval:4.5, u-avg: 3.8428643990560714, window_start_time: 4.329999999999999, window_start_iteration: 468, previous_collection_time:4.4, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 4.4, collecting: false
[ Info: Iter: 487, time: 4.500 seconds, model clock time:4.5, previous_interval_stop_time + interval:4.5, u-avg: 5.57092490868422, window_start_time: 4.41, window_start_iteration: 477, previous_collection_time:4.5, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 4.4, collecting: false
[ Info: Iter: 498, time: 4.600 seconds, model clock time:4.6000000000000005, previous_interval_stop_time + interval:4.7, u-avg: 7.7522019307475505, window_start_time: 4.529999999999999, window_start_iteration: 490, previous_collection_time:4.6000000000000005, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 4.6000000000000005, collecting: false
[ Info: Iter: 509, time: 4.700 seconds, model clock time:4.7, previous_interval_stop_time + interval:4.7, u-avg: 9.42305762921895, window_start_time: 4.61, window_start_iteration: 499, previous_collection_time:4.7, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 4.6000000000000005, collecting: false
[ Info: Iter: 520, time: 4.800 seconds, model clock time:4.800000000000001, previous_interval_stop_time + interval:4.9, u-avg: 11.132132964253687, window_start_time: 4.7299999999999995, window_start_iteration: 512, previous_collection_time:4.800000000000001, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 4.800000000000001, collecting: false
[ Info: Iter: 531, time: 4.900 seconds, model clock time:4.9, previous_interval_stop_time + interval:5.0, u-avg: 12.10757376504871, window_start_time: 4.8100000000000005, window_start_iteration: 521, previous_collection_time:4.9, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 4.9, collecting: false
[ Info: Iter: 542, time: 5 seconds, model clock time:5.0, previous_interval_stop_time + interval:5.0, u-avg: 12.661437548057236, window_start_time: 4.91, window_start_iteration: 532, previous_collection_time:5.0, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 4.9, collecting: false
[ Info: Iter: 553, time: 5.100 seconds, model clock time:5.1000000000000005, previous_interval_stop_time + interval:5.2, u-avg: 12.566819921913737, window_start_time: 5.029999999999999, window_start_iteration: 545, previous_collection_time:5.1000000000000005, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 5.1000000000000005, collecting: false
[ Info: Iter: 564, time: 5.200 seconds, model clock time:5.2, previous_interval_stop_time + interval:5.2, u-avg: 11.926603358213159, window_start_time: 5.11, window_start_iteration: 554, previous_collection_time:5.2, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 5.1000000000000005, collecting: false
[ Info: Iter: 575, time: 5.300 seconds, model clock time:5.300000000000001, previous_interval_stop_time + interval:5.4, u-avg: 10.567933625962374, window_start_time: 5.2299999999999995, window_start_iteration: 567, previous_collection_time:5.300000000000001, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 5.300000000000001, collecting: false
[ Info: Iter: 586, time: 5.400 seconds, model clock time:5.4, previous_interval_stop_time + interval:5.5, u-avg: 9.067883207577948, window_start_time: 5.3100000000000005, window_start_iteration: 576, previous_collection_time:5.4, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 5.4, collecting: false
[ Info: Iter: 597, time: 5.500 seconds, model clock time:5.5, previous_interval_stop_time + interval:5.5, u-avg: 7.1614705405823145, window_start_time: 5.41, window_start_iteration: 587, previous_collection_time:5.5, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 5.4, collecting: false
[ Info: Iter: 608, time: 5.600 seconds, model clock time:5.6000000000000005, previous_interval_stop_time + interval:5.7, u-avg: 4.980193518518983, window_start_time: 5.529999999999999, window_start_iteration: 600, previous_collection_time:5.6000000000000005, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 5.6000000000000005, collecting: false
[ Info: Iter: 619, time: 5.700 seconds, model clock time:5.7, previous_interval_stop_time + interval:5.7, u-avg: 3.309337820047582, window_start_time: 5.61, window_start_iteration: 609, previous_collection_time:5.7, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 5.6000000000000005, collecting: false
[ Info: Iter: 630, time: 5.800 seconds, model clock time:5.800000000000001, previous_interval_stop_time + interval:5.9, u-avg: 1.6002624850128442, window_start_time: 5.7299999999999995, window_start_iteration: 622, previous_collection_time:5.800000000000001, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 5.800000000000001, collecting: false
[ Info: Iter: 641, time: 5.900 seconds, model clock time:5.9, previous_interval_stop_time + interval:6.0, u-avg: 0.6248216842178208, window_start_time: 5.8100000000000005, window_start_iteration: 631, previous_collection_time:5.9, fetch_operand: true, interval: 0.1, previous_interval_stop_time: 5.9, collecting: false
[ Info: Simulation is stopping after running for 1.413 seconds. |
@glwagner Could you maybe provide some hints on where the source code in AveragedTimeInterval should be modified to align with the implementation in TimeInterval? So far I haven't been able to fix the bug. My guess is something along these lines, but I could be wrong: Oceananigans.jl/src/OutputWriters/windowed_time_average.jl Lines 130 to 142 in 406eb9c
|
Yes! Make sure to work from Oceananigans.jl/src/Utils/schedules.jl Lines 65 to 79 in dc5dc28
when Oceananigans.jl/src/Utils/schedules.jl Lines 58 to 63 in dc5dc28
The first actuation time and number of actuations have to be initialized: Oceananigans.jl/src/Utils/schedules.jl Lines 50 to 56 in dc5dc28
I think
and
which in turn is called here:
So there's something about using the previous stop time, rather than the number of actuations, which incurs a floating point error issue. I think |
Thanks a lot for your hint, I wanted to make sure what you mean by number of actuations as a criteria. Do you mean first_actuation_time::Number and do you think this will potentially prevent the floating point issue? |
Not quite.
Oceananigans.jl/src/Utils/schedules.jl Lines 58 to 63 in dc5dc28
To understand this consider an example.
Thus rather than storing the "previous actuation time" (a floating point number), we instead store the number of actuations (an integer, and exact), and compute the next actuation time with the formula above. Does this make sense? |
yes, it makes sense to me now! Thanks for the clear explanation. |
Hi all,
Thanks, @glwagner for creating the PR for solving the issue of
TimeInterval
. After merging it, I noticed a potential problem related toAveragedTimeInterval
. Specifically, when resuming from a checkpoint, discontinuity seems to occur. For instance, in the figure below, the tidally averaged velocity is saved for each tidal period. The first run covers days 0-20, and the second run covers days 20-40. A noticeable discontinuity appears at the checkpoint transition from the first run.Here is a MWE for this issue:
The text was updated successfully, but these errors were encountered: