Invasive neurophysiology toolbox #516

mpompolas · 2022-02-25T06:27:45Z

Adding here the repo I've been working on for visibility.

WIP

Automated copying of default config file during installation

Values will be in Volts instead of mV

Specifically for the Kilosort converter: the conversion from double to int16 has been achieved by F/max(max(abs(F))) * 15000. 15000 was selected as the middle of the maximum value that an int16 value can be. Tried 32678 but Kilosort was giving weird outputs. So far all the datasets I spike-sorted show reasonable results.

Added .xml converter straight from Brainstorm environment. Removed .xslx dependency - Kilosort can be used in non-Windows systems now. Enabled event-dot-visualization on spiking events Moved Kilosort installation to the plugin manager

Redesigned behavior when manual spike-sorting is performed in Klusters. All the spiking events that belong to electrodes that belong to the shank/Montage will be removed, and then the new manual spiking selections will be added to the link-to-raw file

Added ultramegasort2000 and waveclus spikesorters

Accommodate different starting timepoint than 0

Moved installation to the plugin manager

This allows dots-visualizations along the raw file exploration

Design change - the spikesorters when they finish their unsupervised part automatically attach the events to the link to raw file. The reason for this design change is that the previous commit allows deletion of the relevant events that are manually spike sorted

In case of .plx, the entire recording is converted to a .bst to avoid the inefficient loading that .plx files have

linspace created precision error on the computation of Fs. Fixed bug

CONVERT TO LFP: Added option to select the sampling rate of the LFPs. Substituted the resampling function from Matlab's 'downsample', to BST 'resample' Bug Fix on demultiplexers - In_spikesorting_rawelectrodes, in_spikesorting_convertforkilosort

ftadel · 2022-03-14T10:31:03Z

fixes #489

Spiking phase locking: display of total number of neurons, don't display omnibus test Tuning curves: took into account case where a single spike occured.

Fixed display when one of the streams is disabled during acquisition

Intan has inefficient loader (read_Intan_RHD2000). The entire recording is loaded every time by default. The previous importer (2018) modified Intan's importer to provide efficient importing. The new importer uses Intan's 2022 importer, but converts the recording to a .bst for efficient loading

update from master and upgrade intan

Plexon fopen cases were deleted by mistake while merging

Added check for GPU training. Updated bst_progress calls.

ftadel · 2022-04-22T18:48:33Z

@mpompolas
I uploaded the tutorial dataset to the web server (available from the download page)
I've been editing quite a lot all the processes and the tutorials.

The things I still need to do:

Edit process_spikesorting_supervised.m
Review the last tutorial: https://neuroimage.usc.edu/brainstorm/e-phys/functions
Edit the processes introduced in these tutorials

Things that do not work:

Display of UltraMegaSort2000 output
DespikeLFP option: Requires Optimization toolbox (fminunc) => I could not test it
The coordinates of the electrodes are probably a bit random (I took the old channel file you distributed, but this is related to old SCS coordinates...). If you have a solution to export these coordinates in the "world" coordinates of the MRI, that would be great

Things that you can already start working on:

Proof-reading the edits I made of the tutorial
Make sure that all the spike-sorting functions and conversion to LFP still work on your end, for various test files

I'll keep on working on this in the next few days, please do not push directly anything here.

This was not a "process", but a collection of interactive and shared functions. It was all moved to panel_spikes.m

ftadel · 2022-04-26T15:44:21Z

I'm done for now!

You can now:

test all the code,
test again all the supported file formats
proof-read the documentation.

There are the following remaining issues:

The processes "PSTH per neuron" and "PSTH per electrode" are not documented in the tutorials.
The tutorial section "Raster plot per electrode" should be removed, as it refers to a process that does not exist: https://neuroimage.usc.edu/brainstorm/e-phys/functions#Raster_plots
process_spike_triggered_average.m: lines 163 and 178: (divideBy - size(all_labels,2)) can be negative and lead to complex std values, which screw up completely the displays. Please fix.
I cannot reproduce your figures in the last tutorial: Can you please double-check the results you get with the current version of the code?
- Noise correlation:
- Spike field coherence:
- Spike triggered average: I get 70 files and not 53 as documented initially. And I don't see the nice spike illustrated in your figure. Is it because you used the initial recordings and not the LFP epochs, as everywhere else in the tutorial? If you did this and you think it makes sense to illustrate the function, then it needs to be explicit that the user should import both the original recordings and the LFP, for different purposes.

It would be great to a have processing script to reproduce all this, but I guess you won't have time to work on this before the end of the week :)

I won't be working on the code again before you're done on your side.

mpompolas · 2022-04-26T16:29:51Z

Thanks for checking everything out. Working on this now

sbaillet · 2022-04-26T16:32:38Z

Thank you indeed, François! On Apr 26, 2022, at 12:30 PM, Konstantinos ***@***.******@***.***>> wrote: Thanks for checking everything out. Working on this now — Reply to this email directly, view it on GitHub<#516 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AEOWKPNH2XU3B7YVGDVNO2LVHAKYXANCNFSM5PJNAKKQ>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

mpompolas · 2022-04-26T22:00:29Z

Pull request has been opened at: mpompolas#13

Tested spikesorters and Derive to LFP with/without Bayesian Denoising.
All good.

Plexon works.
Intan works.
NWB works.
TDT works.

There are the following remaining issues:

The processes "PSTH per neuron" and "PSTH per electrode" are not documented in the tutorials.

My bad. The initial version of the paper had them as Rasterplot per neuron and rasterplot per electrode. After the first revision we changed that to PSTH and in my head these were interchangeable. The new and correct raster plot function was introduced after the revision.
Updated the website to reflect the correct functions.
Also renamed the function PSTH per electrode to PSTH per channel.

The tutorial section "Raster plot per electrode" should be removed, as it refers to a process that does not exist: https://neuroimage.usc.edu/brainstorm/e-phys/functions#Raster_plots

Addressed above.

process_spike_triggered_average.m: lines 163 and 178: (divideBy - size(all_labels,2)) can be negative and lead to complex std values, which screw up completely the displays. Please fix.

Something is off in this function. I can't reproduce the figures I uploaded on the tutorial after your changes. I'll have to check the STD's when I revisit the function again.

Figures reproduction

I cannot reproduce your figures in the last tutorial: Can you please double-check the results you get with the current version of the code?

This is probably due to the fact you used the functions after you did spike-sorting within Brainstorm. I added a wiki-note on the webpage to alert users of this.

Noise correlation:

It's the same.

Spike field coherence:

It's the same.

Spike triggered average: I get 70 files and not 53 as documented initially.

This is due to the fact that after the new spike-sorting, you now have 70 neurons instead of the initial 53 that were assigned to the raw file.
However, as I mentioned before, I still can't reproduce the figure. Something has changed.

TODO

I have to read thoroughly the Tutorial again, but as far as coding goes, the only remaining issue that I see is the function process_spike_triggered_average.m to check for the STDs visualization and confirm that nothing is off after the changes.

mpompolas · 2022-04-27T22:13:03Z

Update - I run the old process_spike_triggered_average.m on your branch after converting to LFP and I can confirm that the produced figure is the same as what I had on the tutorial, with correct display of the STDs as well, therefore the changes introduced on mpompolas@ae9cd96#diff-a6c4c697a59feda12f423ab7bcc2b2b6aef1465cb6375e90c778f3422d4c5a2b have led to the deviation.

We just need to focus on the function

#13

ftadel · 2022-04-29T15:17:30Z

Pull request has been opened at: mpompolas#13

I was not sure how you were planning on cascading the merging of the PR, so I integrated these changes directly in here: d55a3b8

I added a script for reproducing the tutorial pipeline and figures automatically: tutorial_ephys.m

I fixed the bugs I introduced in process_spike_triggered_average.m.

I think this is ready for merging!
Please confirm.

ftadel · 2022-04-30T08:36:48Z

Process report obtained from the script tutorial_ephys.m:
Brainstorm process report.pdf

mpompolas · 2022-05-01T22:40:43Z

Thank you so much for all your help Francois.
Sorry for the delay this week.
I will make the last check tomorrow.

mpompolas · 2022-05-02T23:41:56Z

Updated STA tutorial page.

You can now merge.

Thank you for all your help @ftadel and @sbaillet

SCRIPT

Last check to do once you merge, is to make sure that the script you created is loaded on the website (note: spiking phase locking and PSTH are not called in the script): https://neuroimage.usc.edu/brainstorm/e-phys/functions#Scripting

FUTURE IMPROVEMENT

One thing that I would suggest to be done at some point in the future, would be to distribute a converted .pl2 file (https://plexon.com/plexutil-4-0-enables-conversion-from-plx-files-to-pl2-files/), instead of a .plx, so users don't get discouraged with the initial waiting time that it takes for converting from .plx to .bst

ftadel · 2022-05-03T12:43:56Z

Done!
What a amazing piece of work!

@mpompolas
Thank you so much for the continued efforts on this project over the past years. I'm sure your work will be extremely useful for lots of people around the world.
I wish you the best of luck and success with your new position.
Hopefully some day you'll miss academia and come back to work with us :-)

sbaillet · 2022-05-03T13:19:09Z

Just re-iterating Francois’ words of appreciation and encouragement (to come back and work with Brainstorm!). Cheers, Sylvain. On May 3, 2022, at 8:44 AM, Francois ***@***.******@***.***>> wrote: Done! What a amazing piece of work! @mpompolas<https://github.com/mpompolas> Thank you so much for the continued efforts on this project over the past years. I'm sure your work will be extremely useful for lots of people around the world. I wish you the best of luck and success with your new position. Hopefully some day you'll miss academia and come back to work with us :-) — Reply to this email directly, view it on GitHub<#516 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AEOWKPMOJCVNGHEYCHT2BGTVIENRPANCNFSM5PJNAKKQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

mpompolas added 16 commits February 21, 2022 11:42

Update bst_plugin.m to accommodate Invasive Neurophysiology toolboxes

6c5cdb5

Updated plexon importers

697327d

Updated kilosort-plugin installation

70b1aae

Automated copying of default config file during installation

Update in_fread_plexon.m

1bfa762

Values will be in Volts instead of mV

Update process_spikesorting_kilosort.m

b756d37

Added .xml converter straight from Brainstorm environment. Removed .xslx dependency - Kilosort can be used in non-Windows systems now. Enabled event-dot-visualization on spiking events Moved Kilosort installation to the plugin manager

Update bst_plugin.m

4ff8c40

Added ultramegasort2000 and waveclus spikesorters

Update process_spikesorting_supervised.m

9b18f7f

Accommodate different starting timepoint than 0

Updated spikesorters

b986abf

Moved installation to the plugin manager

Assigned channels on each spiking event

21552d1

This allows dots-visualizations along the raw file exploration

Merge branch 'master' into invasive_plugins

3822aa9

Separated .plx and .pl2 importers

3cc5052

In case of .plx, the entire recording is converted to a .bst to avoid the inefficient loading that .plx files have

Corrected precision error in timevector in_data_plx

eef0c0f

linspace created precision error on the computation of Fs. Fixed bug

Updated convert to LFP

7f6f5cf

CONVERT TO LFP: Added option to select the sampling rate of the LFPs. Substituted the resampling function from Matlab's 'downsample', to BST 'resample' Bug Fix on demultiplexers - In_spikesorting_rawelectrodes, in_spikesorting_convertforkilosort

mpompolas added 5 commits March 24, 2022 18:21

fixed rounding error in Kilosort

a99d65b

function changes

8576362

Spiking phase locking: display of total number of neurons, don't display omnibus test Tuning curves: took into account case where a single spike occured.

TDT loader fix

42ac8e3

Fixed display when one of the streams is disabled during acquisition

Merge pull request #12 from mpompolas/improve_intan

654a844

update from master and upgrade intan

mpompolas marked this pull request as ready for review March 26, 2022 08:09

mpompolas added 4 commits March 29, 2022 17:55

renamed Intan importer files to match master

c6af024

Typos on extensions

0c3327d

Merge branch 'master' into invasive_plugins

4eb7f32

Update in_fopen.m

478346e

Plexon fopen cases were deleted by mistake while merging

mpompolas mentioned this pull request Apr 5, 2022

Inefficient importers that convert to .bst files might be incompatible with Import MEG/EEG function #532

Closed

mpompolas added 2 commits April 7, 2022 19:38

Update process_spikesorting_kilosort.m

52bc2ae

Added check for GPU training. Updated bst_progress calls.

Update progress-bar popups

8520e1e

ftadel added 7 commits April 21, 2022 16:36

Editing spike sorter options: use plugin manager

424ea5a

Created spike file type (fix)

6811cb9

Reformatting spike sorting processes

c151fbc

Reformatted spike sorting processes

e1db819

Convert processes category from Custom to File

ea76641

Fix ASCII channel file import

b02cd89

Convert to lfp: Re-coding

35fd2cd

Removed process_spikesorting_supervised

fd37018

This was not a "process", but a collection of interactive and shared functions. It was all moved to panel_spikes.m

This was referenced Apr 25, 2022

E-phys: Integrate plugin manager and fix code online tutorials #387

Closed

.plx files show linear reading delay based on time-segment selection #489

Closed

Kilosort - Change the conversion to Brainstorm events through the bz_GetSpikes #156

Closed

Reformatted all the advanced functions

ae9cd96

This comment was marked as outdated.

Sign in to view

ftadel added 3 commits April 29, 2022 16:29

Reporting last changes from @pompolas PR #13

d55a3b8

#13

Added tutorial script

f43d81e

Bugfix: Errors in reformatting process_spike_triggered_average

e1637e1

Removed double registration display

9779e6b

ftadel merged commit f402a42 into brainstorm-tools:master May 3, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Invasive neurophysiology toolbox #516

Invasive neurophysiology toolbox #516

mpompolas commented Feb 25, 2022

ftadel commented Mar 14, 2022

ftadel commented Apr 22, 2022

ftadel commented Apr 26, 2022 •

edited

Loading

mpompolas commented Apr 26, 2022

sbaillet commented Apr 26, 2022 via email

mpompolas commented Apr 26, 2022 •

edited

Loading

mpompolas commented Apr 27, 2022

This comment was marked as outdated.

ftadel commented Apr 29, 2022

ftadel commented Apr 30, 2022

mpompolas commented May 1, 2022

mpompolas commented May 2, 2022 •

edited

Loading

ftadel commented May 3, 2022

sbaillet commented May 3, 2022 via email

Invasive neurophysiology toolbox #516

Invasive neurophysiology toolbox #516

Conversation

mpompolas commented Feb 25, 2022

ftadel commented Mar 14, 2022

ftadel commented Apr 22, 2022

ftadel commented Apr 26, 2022 • edited Loading

mpompolas commented Apr 26, 2022

sbaillet commented Apr 26, 2022 via email

mpompolas commented Apr 26, 2022 • edited Loading

Figures reproduction

TODO

mpompolas commented Apr 27, 2022

This comment was marked as outdated.

ftadel commented Apr 29, 2022

ftadel commented Apr 30, 2022

mpompolas commented May 1, 2022

mpompolas commented May 2, 2022 • edited Loading

SCRIPT

FUTURE IMPROVEMENT

ftadel commented May 3, 2022

sbaillet commented May 3, 2022 via email

ftadel commented Apr 26, 2022 •

edited

Loading

mpompolas commented Apr 26, 2022 •

edited

Loading

mpompolas commented May 2, 2022 •

edited

Loading