BUG: ValueError : Found array with 0 sample(s) (shape=(0, 41)) while a minimum of 1 is required by TruncatedSVD.

[j-moore94]

Describe the issue:

Hi,
(Sorry if this is just me not setting things up properly - I'm not very experienced with Kilosort.)

Every time I try to run kilosort (both from the GUI or using a notebook), I get this error. From doing some debugging, it seems like the whitening/drift correction steps results in the data becoming only zeros. I also seem to get no drift at all, which seems unlikely.

I first thought it was an issue with the way I was setting up the probe. I am using the same .mat file we used for Kilosort 2 but it seems like it is loading correctly so I'm not sure.

Any help would be very much appreciated! :)

Maybe it's helpful to add my probe variable that I get when I load my channel map file:

probe
{'xc': array([ 580., 580., 620., 420., 620., 420., 620., 180., 180.,
180., 380., 220., 380., 220., 380., 220., 620., 820.,
420., 580., 180., 180., 220., 380., 220., 380., 420.,
380., 420., 620., 580., 580., 820., 820., 780., 980.,
780., 1180., 1020., 1220., 980., 1180., 1020., 1220., 780.,
780., 980., 820., 1180., 1020., 1180., 1020., 1180., 1020.,
1220., 1220., 980., 1220., 980., 780., 980., 780., 580.,
820.], dtype=float32),
'yc': array([ -20., -180., -40., -160., -160., -40., -120., -20., -140.,
-60., -60., -200., -100., -120., -180., -80., -200., -80.,
-80., -140., -180., -100., -160., -20., -40., -140., -200.,
-220., -120., -80., -220., -100., -160., -200., -20., -60.,
-140., -140., -80., -200., -220., -60., -160., -80., -60.,
-220., -140., -40., -100., -40., -180., -200., -20., -120.,
-40., -120., -20., -160., -180., -100., -100., -180., -60.,
-120.], dtype=float32),
'kcoords': array([3., 3., 3., 2., 3., 2., 3., 1., 1., 1., 2., 1., 2., 1., 2., 1., 3.,
4., 2., 3., 1., 1., 1., 2., 1., 2., 2., 2., 2., 3., 3., 3., 4., 4.,
4., 5., 4., 6., 5., 6., 5., 6., 5., 6., 4., 4., 5., 4., 6., 5., 6.,
5., 6., 5., 6., 6., 5., 6., 5., 4., 5., 4., 3., 4., 8., 7., 7., 7.,
9.], dtype=float32),
'chanMap': array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,
17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,
34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50,
51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]),
'n_chan': 69}

Reproduce the bug:

Error message:

drift computed in  746.40s; total  748.88s


Traceback (most recent call last):

  File "C:\Users\jmoore\.conda\envs\kilosort\lib\site-packages\kilosort\gui\run_box.py", line 241, in add_plot_data

    
plot_drift_scatter(plot_window2, st0, settings)


  File "C:\Users\jmoore\.conda\envs\kilosort\lib\site-packages\kilosort\gui\sanity_plots.py", line 57, in plot_drift_scatter

    
p1.setTitle('Spike amplitude across time and depth', color='black')


  File "C:\Users\jmoore\.conda\envs\kilosort\lib\site-packages\pyqtgraph\graphicsItems\PlotItem\PlotItem.py", line 1155, in setTitle

    
self.titleLabel.setText(title, **args)


  File "C:\Users\jmoore\.conda\envs\kilosort\lib\site-packages\pyqtgraph\graphicsItems\LabelItem.py", line 58, in setText

    
color = fn.mkColor(color)


  File "C:\Users\jmoore\.conda\envs\kilosort\lib\site-packages\pyqtgraph\functions.py", line 297, in mkColor

    
args = [r,g,b,a]


UnboundLocalError
: 
local variable 'r' referenced before assignment



Extracting spikes using templates


Re-computing universal templates from data.


Traceback (most recent call last):

  File "C:\Users\jmoore\.conda\envs\kilosort\lib\site-packages\kilosort\gui\sorter.py", line 82, in run

    
st, tF, Wall0, clu0 = detect_spikes(ops, self.device, bfile, tic0=tic0,


  File "C:\Users\jmoore\.conda\envs\kilosort\lib\site-packages\kilosort\run_kilosort.py", line 332, in detect_spikes

    
st0, tF, ops = spikedetect.run(ops, bfile, device=device, progress_bar=progress_bar)


  File "C:\Users\jmoore\.conda\envs\kilosort\lib\site-packages\kilosort\spikedetect.py", line 193, in run

    
ops['wPCA'], ops['wTEMP'] = extract_wPCA_wTEMP(


  File "C:\Users\jmoore\.conda\envs\kilosort\lib\site-packages\kilosort\spikedetect.py", line 70, in extract_wPCA_wTEMP

    
model = TruncatedSVD(n_components=ops['settings']['n_pcs']).fit(clips)


  File "C:\Users\jmoore\.conda\envs\kilosort\lib\site-packages\sklearn\decomposition\_truncated_svd.py", line 209, in fit

    
self.fit_transform(X)


  File "C:\Users\jmoore\.conda\envs\kilosort\lib\site-packages\sklearn\utils\_set_output.py", line 157, in wrapped

    
data_to_wrap = f(self, X, *args, **kwargs)


  File "C:\Users\jmoore\.conda\envs\kilosort\lib\site-packages\sklearn\base.py", line 1152, in wrapper

    
return fit_method(estimator, *args, **kwargs)


  File "C:\Users\jmoore\.conda\envs\kilosort\lib\site-packages\sklearn\decomposition\_truncated_svd.py", line 229, in fit_transform

    
X = self._validate_data(X, accept_sparse=["csr", "csc"], ensure_min_features=2)


  File "C:\Users\jmoore\.conda\envs\kilosort\lib\site-packages\sklearn\base.py", line 605, in _validate_data

    
out = check_array(X, input_name="X", **check_params)


  File "C:\Users\jmoore\.conda\envs\kilosort\lib\site-packages\sklearn\utils\validation.py", line 967, in check_array

    
raise ValueError(


ValueError
: 
Found array with 0 sample(s) (shape=(0, 41)) while a minimum of 1 is required by TruncatedSVD.

Version information:

Windows 10, Python 3.9, CUDA 11.8

Context for the issue:

No response

Experiment information:

64-channel silicon probe, plus 5 non-ephys channels included in the binary file (total 69 channels). Recorded in Intan at 20000 samples/s.

[jacobpennington]

Hello,

The first part of that traceback, UnboundLocalError: local variable 'r' referenced before assignmentis related to an outdated dependency, I updated the setup file recently to fix that. You can run pip install --upgrade pyqtgraph in your kilosort environment to fix that without reinstalling, which will cause an additional drift correction plot to show up.

I'll have to do some more poking around to figure out where that other error is coming from. Are you able to post a screenshot of what the gui looks like after you've loaded your data & probe?

Separately, based on the probe configuration you've posted, it's not likely that drift correction will work well. It looks like there are only a handful of contacts on each shank, which is not going to give enough spread or sampling density along the depth of the probe to get a reliable drift estimate. So, you should probably just set nblocks = 0 to disable that step instead.

[j-moore94]

Hi Jacob, thanks for getting back to me. I've just run it through the GUI with nblocks = 0 and I now get an error saying that no spikes have been detected:

0 spikes extracted in  741.52s; total  744.00s


Traceback (most recent call last):

  File "C:\Users\jmoore\.conda\envs\kilosort\lib\site-packages\kilosort\gui\sorter.py", line 82, in run

    
st, tF, Wall0, clu0 = detect_spikes(ops, self.device, bfile, tic0=tic0,


  File "C:\Users\jmoore\.conda\envs\kilosort\lib\site-packages\kilosort\run_kilosort.py", line 337, in detect_spikes

    
raise ValueError('No spikes detected, cannot continue sorting.')


ValueError
: 
No spikes detected, cannot continue sorting.

(I've run this recording through Kilosort 2.5 previously and detected lots of spikes!)

Here's a screenshot of the GUI with the datafile loaded:

[jacobpennington]

Sorry, what does the data look like if you you click on 'whitened' at the bottom right of the plot area? I'm just trying to make sure it looks like the data loaded correctly, which is hard to tell from the raw view. If it does look like everything loaded in correctly, you could try reducing Th (universal) and Th (learned), maybe to something like 6 and 5, respectively. I believe the default values for those have changed between Kilosort versions.

[j-moore94]

No problem, here's the whitened data. :)

I got the same error again using 6 and 5 for the thresholds.

I assume this isn't relevant because I am using Windows not Linux, but I also get this warning earlier on:

Re-computing universal templates from data.


C:\Users\jmoore\.conda\envs\kilosort\lib\site-packages\threadpoolctl.py:1186: RuntimeWarning: 
Found Intel OpenMP ('libiomp') and LLVM OpenMP ('libomp') loaded at
the same time. Both libraries are known to be incompatible and this
can cause random crashes or deadlocks on Linux when loaded in the
same Python program.
Using threadpoolctl may cause crashes or deadlocks. For more
information and possible workarounds, please see
    https://github.com/joblib/threadpoolctl/blob/master/multiple_openmp.md

  warnings.warn(msg, RuntimeWarning)

[jacobpennington]

Yeah that's a known warning, we haven't seen any problems from it.
I tried poking around a bit more, but so far I'm not seeing anything that would cause this based on the probe information. Are you able to share some sample data so that I can debug this?

[rdarie]

I may have stumbled upon the same issue as well (and a solution). I'm working with a 30 channel probe and saw the same error about no spikes being detected. On line 150 of spikedetect.py, Aa[iC2] had entries that were all NaNs past the 30th index, which caused Amax to be all NaNs as well. I think this was happening because nearest_templates, by default, is 100 and exceeds the max number of channels. Setting it to less than that fixes the problem. Maybe there can be a check so that this happens automatically?

[jacobpennington]

That's helpful, thanks! I'll take a look at that. The software has worked fine on other datasets that 32 or 64 channels, so it's not just a channel count issue, but that narrows it down at least. I also can't reproduce the error by increasing nearest_templates to greater than the channel count on neuropixels data, so unfortunately it's not that simple.

[j-moore94]

Hi, thanks for the input Radu. I extracted a short section from my dat file to send, and I just tried running it with 100 nearest_templates and 50 nearest_templates. With 50, I do detect some spikes, but then run into this error:

Traceback (most recent call last):

  File "C:\Users\jmoore\.conda\envs\kilosort\lib\site-packages\kilosort\gui\sorter.py", line 82, in run

    
st, tF, Wall0, clu0 = detect_spikes(ops, self.device, bfile, tic0=tic0,


  File "C:\Users\jmoore\.conda\envs\kilosort\lib\site-packages\kilosort\run_kilosort.py", line 343, in detect_spikes

    
Wall3 = template_matching.postprocess_templates(Wall, ops, clu, st0, device=device)


  File "C:\Users\jmoore\.conda\envs\kilosort\lib\site-packages\kilosort\template_matching.py", line 83, in postprocess_templates

    
Wall2, _ = align_U(Wall, ops, device=device)


  File "C:\Users\jmoore\.conda\envs\kilosort\lib\site-packages\kilosort\template_matching.py", line 70, in align_U

    
X = conv1d(X.unsqueeze(1), ops['wTEMP'].unsqueeze(1), padding=ops['nt']//2)


RuntimeError
: 
Calculated padded input size per channel: (40). Kernel size: (41). Kernel size can't be greater than actual input size

With 100 templates, I get the same error as before. Here's a link to the 10 minute section of data if you want to try:
https://drive.google.com/drive/folders/1YyX9wzetNu4zcg6JTYvmY4-l21NuD1WD?usp=sharing

[rdarie]

That's interesting. It's worth noting that I never encountered the error with TruncatedSVD() in the first place, so there might be several differences between our datasets that are causing the issue. I've uploaded a segment of the data I'm working on at the link below:

https://drive.google.com/drive/folders/1JMHhmmHRH6OTHXtbrxgOXzBTpfIUS-WM?usp=sharing

By the way, @jacobpennington, should I expect sorting results to be comparable between KiloSort3 and KiloSort4? I'm finding that they agree in the case of really large units, but KS4 overall finds fewer units on the same dataset. I can start a separate issue with some examples if that's helpful.

[jacobpennington]

@rdarie In general no, they may not be comparable especially if there's a lot of drift. You can check out the bioarxiv preprint for an overview (Table 1 and Table 2 in particular).

[marius10p]

@jacobpennington @rdarie To be clear, Kilosort4 should usually work better and has the same pipeline backbone as Kilosort3 and the same drift correction, so the same thresholds should produce roughly the same number of spikes and units. What is different is the clustering algorithm, which should result in more well-isolated units / better refractory periods.

[ZaneMuir]

Hi, we are using a 32 channel probe and had the same issue of 0 samples for TruncatedSVD. By lowering the nearest_templates to 50 (or even smaller values for some recordings), there are spikes.

Is there a rule of thumb for picking different values, for nearest_templates, or in general?

[rdarie]

Thanks for the clarification! With that in mind, it looks to me that, at least on my dataset, everything else seems to work just fine after lowering n_nearest_templates.

[jacobpennington]

@j-moore94 It looks like the second issue you ran into after changing nearest_templates is related to other issues users are reporting with multi-shank probes. Basically, it's hard to determine a good value for dminx automatically when there are large (more than 50 microns) horizontal gaps between contacts since we aren't explicitly handling multiple shanks yet.

I got reasonable-looking results for your data by increasing dminx (under "Extra settings") to 400, you could play around with it some more from there. That controls the lateral spacing of the templates we use for spike detection, which tile the space of the probe.

If you're not satisfied with the results from that, you can try artificially rearranging your probe file to vertically stack contacts from different shanks, though it's not clear to me what the best way to do that is with your staggered probe arrangement, so I would try tweaking dminx first.

[j-moore94]

Great thanks, this seems to have worked for the full recording too! Any advice on how to choose the value of dminx and nearest_templates or should it not make much of a difference?

(EDIT: You can ignore the following, as I reinstalled again and got it working!):

Also, when I tried to check the results in phy, I get the issue mentioned in #601. I then tried reinstalling using the command pip install git+https://github.com/mouseland/kilosort.git as suggested there, but now I'm getting this error when I try to run the GUI:

QObject::moveToThread: Current thread (0x19dbd83b380) is not the object's thread (0x19dbd83ca00).
Cannot move to target thread (0x19dbd83b380)

qt.qpa.plugin: Could not load the Qt platform plugin "windows" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: minimal, offscreen, webgl, windows.