4D data from cdf in dataobj() should match its specified dimensions

[thomas-nilsson-irfu]

// NOTE: This issue is added here mainly as a placeholder until the issue is fully solved, as fixing it in 9440568 generated A LOT more dependent problems.

Step 1: Latest code?

• This issue is present in the most recent code (i.e. after running git pull this is still a problem).

Step 2: Describe your environment

• irfu-matlab branch (i.e. "master", "devel"): devel
• Matlab version used (i.e. "Matlab R2017a"): R2017b
• Operating system (ie "Windows 7", "Mac OS X 10.12.3", "Linux Ubuntu 16.04"): All of the above

• A 64-bit installation of Matlab was used.

Step 3: Describe the problem

Expected behavior

Objects and variables created using dataobj() SHALL match the dimensions of the variables inside the CDF files from which it was created. (Which in turn of course should match the appropriate instrument papers, data products guide, release notes or other documentation). That is the Depend_0 (being Epoch in most cases), and subsequent Depend_i for i=1:3. (I.e. 4D data variables) should be in the correct positions and data should be the appropriate sizes in each dimension.

Actual behavior

Presently dataobj()'s permute function change the data according to files created using the old Cluster method. But files created according to the most recent instructions from NASA SPDF should not have permute running in the same way.
These should only have the main time dependency put in the first column and not shifting the second to last column. A detailed overview can be viewed here: https://docs.google.com/spreadsheets/d/1L58nLwy1Mwl9y4OATjkgjb58p6dCiNFerEx2cVbsB0M/edit#gid=0
For MMS this impacts FPI data, and for future missions (SolarOrbiter?) it could impact any other files created according to the correct instructions (see "Test with SPDFCDFWRITE" in the spreadsheet).

Relevant code:

When all the dependent files are ready, then the following can be put into place again:

irfu-matlab/@dataobj/dataobj.m

Lines 357 to 385 in 9440568

	function fix_order_of_array_dimensions
	for iDimension=3:4
	% Check if dimensions match expected (ignoring Depend_0) and that
	% it is not a single vector (which could be "Nx1").
	indDatasets = find(cellfun(@(x) all(x(:) > 1) && numel(x)==iDimension-1, info.Variables(:,2)));
	for iDataset=1:numel(indDatasets)
	if iDimension==3
	% permutate it to N x m x n, where N corrsponds to record (ie Depend_0).
	data{indDatasets(iDataset)} = permute(data{indDatasets(iDataset)}, [3 1 2]);
	elseif iDimension==4
	% Check if it should be simple cyclic permutation (MMS), data
	% was read into "recSize x nRec". Also possibly a matrix of
	% size "recSize" if it only contained one single record.
	% Or if the data should be more complexly permuted (Cluster),
	% ie where the recSize does not match the data.
	recSize = info.Variables{indDatasets(iDataset), 2}; % Expected record size, (m x n x p)
	nRec = info.Variables{indDatasets(iDataset), 3}; % Numer of records, (N)
	dataSize = size(data{indDatasets(iDataset)}); % Size of data, as it was read
	if isequal([recSize, nRec], dataSize) || (nRec==1 && isequal(recSize, dataSize) )
	% Simple cyclic permutation to get N x m x n x p
	data{indDatasets(iDataset)} = permute(data{indDatasets(iDataset)}, [4 1 2 3]);
	else
	% Re-order Cluster data, from (n x p x m x N) to (N x m x n x p).
	data{indDatasets(iDataset)} = permute(data{indDatasets(iDataset)}, [4 3 1 2]);
	end
	end
	end
	end
	end

[ErikPGJ]

I believe myself to have seen the same kind of/similar phenomenon for 1+3 dimensional zVar (1 record dimension, 3 dimensions/record)

• 1ea1d71 thomas-nilsson-irfu (2019-09-06 13:15:26 +0200) (HEAD -> SOdevel, origin/SOdevel) Merge remote-tracking branch 'origin/devel' into SOdevel
  MATLAB R2016a
  Ubuntu 18.04.3 LTS

Found when Solar Orbiter/RPW calibration s/w "Bicas" reads a BIAS RCT (calibration file).
Different bad dimensions from dataobj depending on number of CDF zVar records.
(Note: TRANSFER_FUNCTION_COEFFS:DEPEND_0=Epoch_L)

Do = dataobj(...)

zVar/file with 1 CDF record :
cdfdump: "TRANSFER_FUNCTION_COEFFS CDF_DOUBLE/1 3:[2,8,4] F/TTT"
size(Do.data.TRANSFER_FUNCTION_COEFFS.data) == [ 4 2 8]

zVar/file with 2 CDF records:
cdfdump: "TRANSFER_FUNCTION_COEFFS CDF_DOUBLE/1 3:[2,8,4] T/TTT"
size(Do.data.TRANSFER_FUNCTION_COEFFS.data) == [2 4 2 8]

NOTE: Created separate issue #50 for the different behaviour depending on number of CDF records. Related?