Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Request for model training #43

Closed
laura300 opened this issue Sep 13, 2022 · 7 comments
Closed

Request for model training #43

laura300 opened this issue Sep 13, 2022 · 7 comments
Assignees
Labels
enhancement New feature or request question Further information is requested

Comments

@laura300
Copy link

Hi Peter,
Thanks for developing this tool, it's really useful!
Would it be possible to have to train a model for Excitatory and inhibitory cells combined at 5Hz?
I am also wondering since we have been using GCaMP7f and this is not included in the ground truth, do you have any idea of the expected performance?
Thanks!

Laura

@PTRRupprecht
Copy link
Member

PTRRupprecht commented Sep 13, 2022

Hi @laura300,

Thanks for the feedback!

Training a model for excitatory and inhibitory cells combined would in theory be possible but I would not recommend it (but you can still convince me otherwise!). The model trained on both datasets would try to find a compromise and therefore result in sub-optimal predictions for excitatory neurons. I have done this upon a question of a reviewer of our paper, but we did not include this result in the paper.

Instead, I would rather recommend to train a model for excitatory and another model for inhibitory cells. If you think that this is a good idea, I can do this for you (the model for excitatory neurons is already available, I would only train the model for inhibitory neurons). The limitation of this approach is that there are only few training data for inhibitory neurons.

Alternatively, one could use the model trained for excitatory neurons for both excitatory and inhibitory neurons, and simply accept that the predictions will be still correlated with inhibitory spike rates (Figure 3a) but off by a large factor (Figure 3c,e).

I hope this helps you to decide what model to choose for your recordings! If not, let me know.

About GCaMP7f: Yes, it is not yet in included in the ground truth. I will include one available dataset in the near future (approx. end of 2022), but I want to do some analyses first. The expected performance is that a model trained on all available datasets (e.g., one of the global_EXC models) will perform equally well on GCaMP7f data. However, due to the shorter rise time of GCaMP7f, the inferred spike rate will be slightly earlier than the true spike rate (this systematic shift will occur for any spike inference algorithm optimized for GCaMP6). This effect is small for GCaMP7f (I think it is clearly less than 100 ms) but a bit stronger for GCaMP8. Together, this effect is not detrimental, but one needs to keep it in mind.

Let me know if you have any other questions!

@PTRRupprecht PTRRupprecht added enhancement New feature or request question Further information is requested labels Sep 13, 2022
@laura300
Copy link
Author

Hi Peter,

Thanks a lot for your quick answer and your input on the GCaMP7.
I'll follow your advice on using the available model for excitatory cells, I really think that this should work fine. I will eventually have some data with only inhibitory cells so I'll ask you then for the model trained with the inhibitory ground truth.
Many thanks!

All the best,
Laura

@PTRRupprecht
Copy link
Member

Ok, great!

P.S. @laura300: Since your data are sampled a relatively low temporal resolution (5 Hz), maybe check out whether the typical noise levels of your data is covered by the models (default coverage is noise levels between 2 and 9). There was recently an issue (issue #39), where we found that noise levels of the 4-Hz dataset were much higher than 9, and training a model with higher noise levels improved performance a lot. So, just in case you also work with low-signal recordings :-)

@nguyemi5
Copy link

nguyemi5 commented Feb 8, 2024

Hi Peter,
thank you so much for developing this pipeline! I would like to ask whether there's an update about the performance of your models on jGCaMP7f. I tried using the 'Global_EXC_30Hz_smoothing25ms_causalkernel' model on our data (acquired at 30Hz) and it seems to perform very nicely. However, I guess the absolute spiking rate might be overestimated since the dynamics of jGCaMP7f is faster, is that right? Unfortunately, I don't have any ground truth dataset available to confirm this thought. Do you think it would make sense to rescale the output based just on the ratio of published GCaMP6f to GCaMP7f parameters (time constants, dF/F per AP)?
Thank you,
Jana

@PTRRupprecht PTRRupprecht reopened this Feb 9, 2024
@PTRRupprecht
Copy link
Member

Hi Jana @nguyemi5,

That's a good question.

I think your intuition is correct. The overall shape of the inferred spiking should be correct but the absolute spike rate might be overestimated.

Luckily, the GCaMP8 paper includes also a ground truth dataset with jGCaMP7f. I have extracted this dataset and used it to check your question. Below, I plot the average spike rate inferred using the "Global_EXC_30Hz_smoothing25ms_causalkernel" model, and the ground truth spike rate. The spike rate is actually overestimated by a factor of 3-4 (median overestimate of 3.6x):
Comparison_GC7f

Of course, take this with a grain of salt, since it is based on only one single jGCaMP7f ground truth dataset with 21 neurons.
If you have any follow-up questions, just let me know.

I'm still in the process of analyzing how well the existing models generalize to GCaMP8f/m/s and GCaMP7f. This requires some attention to the detail and therefore took me longer than expected (and I was also busy with transitioning from postdoc to PI ...). The brief analyses like this one can be done quickly, but to bring things together and understand the effects will still take me a few more months. But if you have specific questions that you would find interesting to have analyzed, just let me know.

Best,
Peter

@PTRRupprecht PTRRupprecht self-assigned this Feb 9, 2024
@nguyemi5
Copy link

nguyemi5 commented Feb 9, 2024

Hi Peter,
thanks for such a fast and helpful analysis. It's good to know about these systematic effects and we'll keep these in mind when analyzing our data. Also, congratulations on you transition!
Thank you and all the best,
Jana

@PTRRupprecht
Copy link
Member

Hi @nguyemi5,

There is an update about using spike inference with Cascade on data acquired with GCaMP7f.

We have now tested models that were trained on either GCaMP6 or GCaMP8 and applied them to GCaMP7f, with the result that GCaMP8-trained models (which are already available for Cascade) performed better. However, GCaMP6-based models ("Global_EXC" models) also performed quite well. It is described in this new preprint on spike inference with GCaMP8 in Figure 1 Supplement 5.

If somebody is using MLSpike or OASIS (Suite2p) for spike inference with GCaMP7f, we also provide the best-fit values for GCaMP7f in the Methods section :-)

Best,
Peter

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request question Further information is requested
Projects
None yet
Development

No branches or pull requests

3 participants