Problem exporting to phy

[rtraghavan]

I was able to download the sample data and run a sort on it successfully. Opening it in the manual clustering suite also worked very well. However I cannot export to phy. I get the following error message when I execute it either form teh command line or from the manual GUI.

call_irc2_: Output argument "S_out" (and maybe others) not assigned during call to "irc>test_".
Error using rethrow
Input must be a structure.

Error in irc2phy>call_irc2_
(line 159)
rethrow ME;

Error in irc2phy>load_irc2_
(line 170)
function varargout =
load_irc2_(varargin), cell_out =
call_irc2_(dbstack(), varargin,
nargout); varargout = cell_out;
end

Error in irc2phy (line 24)
[S0, S_auto, trPc_spk,
trPc2_spk, P] =
load_irc2_(vcFile_prm);

Error in irc2 (line 195)
case {'export-phy', 'phy'},
open_phy_(irc2phy(vcFile_prm,
vcArg2)); return;

[rtraghavan]

Update and further details. I am sorting a neuropixel array file (Phase 3A). Sorting off a linux machine running Ubuntu 18.04, Matlab2018b, 64 GB ram, and an NVIDIA GTX 980 Ti graphics card.

I first execute irc2 'filename.imec.ap.bin' in a directory containing both the bin and meta file.

Sorting proceeds well and I can open the manual curation utility with ease. However when executing the following

irc2 export-phy 'path/raw_geom.prm'

matlab crashes, due to a memory error. Usually during the process of feature reading with the following output.
Loading feature 1...Killed

The output of dmesg is the following
[50582.989510] Out of memory: Killed process 14963 (MATLAB) total-vm:71172364kB, anon-rss:63398372kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:126668kB oom_score_adj:0

Is there a parameter I should be executing to turn on paged export that might help the exporter work?

[rtraghavan]

This is also the case for irc2 export-klusters

[TomBugnon]

I am having exactly the same issue with irc2 export-phy 'path/raw_geom.prm'

dmesg output:

[75492.107397] Out of memory: Killed process 9326 (MATLAB) total-vm:947076956kB, anon-rss:874562776kB, file-rss:0kB, shmem-rss:208kB, UID:1012 pgtables:1718684kB oom_score_adj:0

I'm on a Ubuntu machine with 1.5TB (!) RAM. Another try at running the command is currently using 1.2TB RAM (! again). The raw data used for sorting is 155GB (2h neuropixels recording) so there must be a problem somewhere.

Any help would be much appreciated,
Cheers

[TomBugnon]

@rtraghavan have you managed to find a way around the issue by chance?

[rtraghavan]

@TomBugnon no I haven't. I am trying to get it to work through the overall framework of spikeinterface on the off chance that those folk have ironed out the bugs?

[TomBugnon]

@rtraghavan I can't find your email, could you email me so we can have a little chat about this? Cheers

[WeissShahaf]

did you guys figure out the issue? how was it solved?

[rtraghavan]

Sadly no. But there is one potential solution which is to sort things in spikeinterface and let its wrappers take care of phy2 export.

[hpay]

I'm having the exact same problem as the original post. Did the solution of running ironclust through spikeinterface work? I have no idea how to access the sorting results otherwise (presumably they are in the firings.mda file but I'm not sure what format that is)

[rtraghavan]

Yes, I am able to call and run ironclust using spikeinterface. Exporting to phy is straightforward, with occasional bugs that the spikeinterface team is doing a great job addressing. Moreover, they bundle the ironclust sorter in docker, so they should be stable for some time. The .mda format was introduced as part of mountainsort, and spikeinterface can import those .mda files directly and export them to phy, so you do not need to run the sorting over again.

[hpay]

Thank you @rtraghavan! I'll pivot to spikeinterface as well