LARMIP/AIS does not randomize samples

[bobkopp]

It appears that larmip/AIS goes through sequentially, applying model 1 to the first N, model 2 to the next N, etc. It should instead be applying the models in a random order, similar to ipccar5/glaciers, so it does not create structure in the sampling:

		# Randomly pulls time-series indices from the selection pool corresponding to samps_per_model for this method
		rnd_sample_idx = np.random.choice(time_series_idx, size=samps_per_model[method_idx], replace=False)

		# Removes the selected time-series indices from the selection pool for the next loop
		time_series_idx = np.setdiff1d(time_series_idx, rnd_sample_idx)

[AlexReedy]

we want the logic to be looking like this

NUMBER OF MODELS: 16
NSAMPS: 25
NUMBER OF SAMPLES PER MODEL: [1 2 2 2 2 1 1 2 2 2 1 1 2 1 1 2]
CURRENT MODEL: PISM_PIK
>> Number of Samples: 1
>> Indices of Time-Series Data used for this Model: [24]
>> For Sample Index [24]
>> Ocean Model Index: 17
CURRENT MODEL: PS3D_PSU
>> Number of Samples: 2
>> Indices of Time-Series Data used for this Model: [18 23]
>> For Sample Index [18]
>> Ocean Model Index: 4
>> For Sample Index [23]
>> Ocean Model Index: 13
CURRENT MODEL: CISM_NCA
>> Number of Samples: 2
>> Indices of Time-Series Data used for this Model: [4 3]
>> For Sample Index [4]
>> Ocean Model Index: 7
>> For Sample Index [3]
>> Ocean Model Index: 7
CURRENT MODEL: PISM_AWI
>> Number of Samples: 2
>> Indices of Time-Series Data used for this Model: [20 7]
>> For Sample Index [20]
>> Ocean Model Index: 15
>> For Sample Index [7]
>> Ocean Model Index: 17
CURRENT MODEL: SICO_ILTS
>> Number of Samples: 2
>> Indices of Time-Series Data used for this Model: [12 15]
>> For Sample Index [12]
>> Ocean Model Index: 5
>> For Sample Index [15]
>> Ocean Model Index: 15
CURRENT MODEL: AISM_VUB
>> Number of Samples: 1
>> Indices of Time-Series Data used for this Model: [0]
>> For Sample Index [0]
>> Ocean Model Index: 6
CURRENT MODEL: ISSM_JPL
>> Number of Samples: 1
>> Indices of Time-Series Data used for this Model: [2]
>> For Sample Index [2]
>> Ocean Model Index: 12
CURRENT MODEL: FETI_ULB
>> Number of Samples: 2
>> Indices of Time-Series Data used for this Model: [21 22]
>> For Sample Index [21]
>> Ocean Model Index: 6
>> For Sample Index [22]
>> Ocean Model Index: 8
CURRENT MODEL: ISSM_UCI
>> Number of Samples: 2
>> Indices of Time-Series Data used for this Model: [6 1]
>> For Sample Index [6]
>> Ocean Model Index: 8
>> For Sample Index [1]
>> Ocean Model Index: 11
CURRENT MODEL: IMAU_UU
>> Number of Samples: 2
>> Indices of Time-Series Data used for this Model: [ 5 17]
>> For Sample Index [5]
>> Ocean Model Index: 10
>> For Sample Index [17]
>> Ocean Model Index: 2
CURRENT MODEL: BISI_LBL
>> Number of Samples: 1
>> Indices of Time-Series Data used for this Model: [19]
>> For Sample Index [19]
>> Ocean Model Index: 0
CURRENT MODEL: UA_UNN
>> Number of Samples: 1
>> Indices of Time-Series Data used for this Model: [13]
>> For Sample Index [13]
>> Ocean Model Index: 4
CURRENT MODEL: MALI_DOE
>> Number of Samples: 2
>> Indices of Time-Series Data used for this Model: [ 8 10]
>> For Sample Index [8]
>> Ocean Model Index: 0
>> For Sample Index [10]
>> Ocean Model Index: 1
CURRENT MODEL: PISM_VUW
>> Number of Samples: 1
>> Indices of Time-Series Data used for this Model: [9]
>> For Sample Index [9]
>> Ocean Model Index: 10
CURRENT MODEL: GRIS_LSC
>> Number of Samples: 1
>> Indices of Time-Series Data used for this Model: [14]
>> For Sample Index [14]
>> Ocean Model Index: 15
CURRENT MODEL: PISM_DMI
>> Number of Samples: 2
>> Indices of Time-Series Data used for this Model: [11 16]
>> For Sample Index [11]
>> Ocean Model Index: 11
>> For Sample Index [16]
>> Ocean Model Index: 14

[bobkopp]

That looks write to me. Check is to make sure whatever changes leave the 17-83rd percentile range essentially unchanged but if you plot samples (eg in 2100) vs sample number it’ll look unstructured.

________________________________ From: AHReedy ***@***.***> Sent: Wednesday, December 20, 2023 2:05:03 PM To: radical-collaboration/facts ***@***.***> Cc: Bob Kopp ***@***.***>; Author ***@***.***> Subject: Re: [radical-collaboration/facts] LARMIP/AIS does not randomize samples (Issue #294) we want the logic to be looking like this NUMBER OF MODELS: 16 NSAMPS: 25 NUMBER OF SAMPLES PER MODEL: [1 2 2 2 2 1 1 2 2 2 1 1 2 1 1 2] CURRENT MODEL: PISM_PIK

> Number of Samples: 1 > Indices of Time-Series Data used for this Model: [24] > For Sample Index [24] > Ocean Model Index: 17

CURRENT MODEL: PS3D_PSU

> Number of Samples: 2 > Indices of Time-Series Data used for this Model: [18 23] > For Sample Index [18] > Ocean Model Index: 4 > For Sample Index [23] > Ocean Model Index: 13

CURRENT MODEL: CISM_NCA

> Number of Samples: 2 > Indices of Time-Series Data used for this Model: [4 3] > For Sample Index [4] > Ocean Model Index: 7 > For Sample Index [3] > Ocean Model Index: 7

CURRENT MODEL: PISM_AWI

> Number of Samples: 2 > Indices of Time-Series Data used for this Model: [20 7] > For Sample Index [20] > Ocean Model Index: 15 > For Sample Index [7] > Ocean Model Index: 17

CURRENT MODEL: SICO_ILTS

> Number of Samples: 2 > Indices of Time-Series Data used for this Model: [12 15] > For Sample Index [12] > Ocean Model Index: 5 > For Sample Index [15] > Ocean Model Index: 15

CURRENT MODEL: AISM_VUB

> Number of Samples: 1 > Indices of Time-Series Data used for this Model: [0] > For Sample Index [0] > Ocean Model Index: 6

CURRENT MODEL: ISSM_JPL

> Number of Samples: 1 > Indices of Time-Series Data used for this Model: [2] > For Sample Index [2] > Ocean Model Index: 12

CURRENT MODEL: FETI_ULB

> Number of Samples: 2 > Indices of Time-Series Data used for this Model: [21 22] > For Sample Index [21] > Ocean Model Index: 6 > For Sample Index [22] > Ocean Model Index: 8

CURRENT MODEL: ISSM_UCI

> Number of Samples: 2 > Indices of Time-Series Data used for this Model: [6 1] > For Sample Index [6] > Ocean Model Index: 8 > For Sample Index [1] > Ocean Model Index: 11

CURRENT MODEL: IMAU_UU

> Number of Samples: 2 > Indices of Time-Series Data used for this Model: [ 5 17] > For Sample Index [5] > Ocean Model Index: 10 > For Sample Index [17] > Ocean Model Index: 2

CURRENT MODEL: BISI_LBL

> Number of Samples: 1 > Indices of Time-Series Data used for this Model: [19] > For Sample Index [19] > Ocean Model Index: 0

CURRENT MODEL: UA_UNN

> Number of Samples: 1 > Indices of Time-Series Data used for this Model: [13] > For Sample Index [13] > Ocean Model Index: 4

CURRENT MODEL: MALI_DOE

> Number of Samples: 2 > Indices of Time-Series Data used for this Model: [ 8 10] > For Sample Index [8] > Ocean Model Index: 0 > For Sample Index [10] > Ocean Model Index: 1

CURRENT MODEL: PISM_VUW

> Number of Samples: 1 > Indices of Time-Series Data used for this Model: [9] > For Sample Index [9] > Ocean Model Index: 10

CURRENT MODEL: GRIS_LSC

> Number of Samples: 1 > Indices of Time-Series Data used for this Model: [14] > For Sample Index [14] > Ocean Model Index: 15

CURRENT MODEL: PISM_DMI

> Number of Samples: 2 > Indices of Time-Series Data used for this Model: [11 16] > For Sample Index [11] > Ocean Model Index: 11 > For Sample Index [16] > Ocean Model Index: 14

— Reply to this email directly, view it on GitHub<#294 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAJSFEAAMEGXVN4NUL4W543YKMZF7AVCNFSM6AAAAABATTEIF2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRUHE4TGNRZGE>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[AlexReedy]

ok excellent getting the plots and percentiles now

[AlexReedy]

For coupling.ssp245
values for the new run are on the left, and AR6 values on the right
FAIR Temp GSAT [LarmipNew/AR6] 2.52 (2.15-3.01) / 2.61 (2.19-3.12)
Larmip_AIS_New [LarmipNew/AR6] 0.13 (0.05-0.28) / 0.14 (0.05-0.29)

The plot doesn't look that much less structured to me than it did before

[AlexReedy]

For Coupling.SSP585
FAIR TEMP GSAT 4.24 (3.62-5.02)
Lamrip AIS 0.14 (0.06-0.34)

[AlexReedy]

Also noticing in the original script and by extension the new script that this loop doesn't actually assign i to anything. Are the ocean models being assigned to specific values int he time series data as well?

	for i in np.arange(samps_per_model[model_idx]):

		# Choose a random forcing from the temperature data
		temp_idx = tempcount
		tempcount = tempcount + 1
		Temp = np.array(SAT[:,temp_idx])

[bobkopp]

Check with @pkjr002 - he had plots that looked more structured, we should understand if he was doing this in a different way.

The loop over I is just incrementing the temperature index -- we don't currently guarantee a equal number of draws of ocean model, though we could and it'd be in the spirit of what we do with ice models.

[bobkopp]

Also -- X axis on the plot should be sample index number, not temperature, which I think is what you are showing.

[AlexReedy]

[AlexReedy]

New plot with changes updated in todays meeting:

[bobkopp]

Ok, looks like there is still structure there.

[pkjr002]

Attached links are outputs from a notebook (global , local) that computes presence of pattern table for FACTS 1.1.1 (data from Alex) and also includes plots.

It appears that there is a strong pattern for larmip(AIS,W/EAIS,PEN), similar to the plot above.
A weaker pattern appears to exists for bamber(EAIS), FittedISMIP(GIS) and local file of tlm(2070,2090).

[bobkopp]

Thanks @pkjr002! Could one of you rerun these modules with a different random seed parameter and evaluate whether these patterns persist?

[AlexReedy]

Sure Bob I can do this todaySent from my Verizon, Samsung Galaxy smartphone -------- Original message --------From: Bob Kopp ***@***.***> Date: 1/22/24 8:07 AM (GMT-05:00) To: radical-collaboration/facts ***@***.***> Cc: AHReedy ***@***.***>, Assign ***@***.***> Subject: Re: [radical-collaboration/facts] LARMIP/AIS does not randomize samples (Issue #294) Thanks @pkjr002! Could one of you rerun these modules with a different random seed parameter and evaluate whether these patterns persist? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were assigned.Message ID: ***@***.***>

[AlexReedy]

@pkjr002 do yo have the plot for tlm?

[pkjr002]

@AlexReedy in the (global) notebook, very last row (it is incorrectly labeled as SD)

[AlexReedy]

This is Larmip/AIS at n=2000 with seed=5, looks less periodic than before but the structure is still there it would seem.

[AlexReedy]

This is Larmip/AIS with the randomization fixes in (sorry the above didn't include the fixes we added). Same parameters.

[bobkopp]

@AlexReedy is all the work done on this to date in the development branch?