Error while running extract_data

[jel0624]

I encountered this error while running extract_data:

I was wondering if this is a problem???

Best,
Jay

[jel0624]

I tried nevertheless to run the IBL GUI. I get an error like the following:

[mayofaulkner]

Hi Jay,

For the first issue could you run the following code and paste the output here

import numpy as np
from pathlib import Path
out_path = Path('') # This should be the same out_path used above in the extraction

attributes = ['clusters', 'amps', 'times', 'depths']
for att in attributes:
    data = np.load(out_path.joinpath(f'spikes.{att}.npy'))
    print(f'spikes.{att}: {data.shape}, {np.nanmin(data)}, {np.nanmin(data)}')

data = np.load(out_path.joinpath('clusters.channels.npy'))
print(f'clusters.channels: {data.shape}')

For the second issue can you expand on how you got your xyz_picks.json. Did you make sure to transform it to the IBL coordinates?

[jel0624]

for the second issue, I obtained probe_ccf.npy from AP_histology
and used the following code to transform it (matlab->python)

% Pick filename
filename = uiputfile('*.npy');
% Write probe_ccf coordinates as NPY file
writeNPY(probe_ccf.trajectory_coords,filename)

import numpy as np
from pathlib import Path
import json
from ibllib.atlas import AllenAtlas

atlas = AllenAtlas(25)

brainreg_path = Path(r'C:\Neuropixels_sorting_folder\JL101_20240914_g0\JL101_1_processed\probe_ccf.npy')
xyz_apdvml = np.load(brainreg_path)
xyz_mlapdv = atlas.ccf2xyz(xyz_apdvml, ccf_order='apdvml') * 1e6
xyz_picks = {'xyz_picks': xyz_mlapdv.tolist()}

output_path = Path(r'C:\Neuropixels_sorting_folder\JL101_20240914_g0\alf')
with open(Path(output_path, 'xyz_picks.json'), "w") as f:
    json.dump(xyz_picks, f, indent=2)`

[mayofaulkner]

Hi,

Thanks for the response. Is there any chance you could share your ks output and also the proce_ccf.npy file that you have. On dropbox or gdrive? I can then take a look.

Many thanks!

[jel0624]

here is the link
https://www.dropbox.com/scl/fo/tvhzysf536usxcf8ocnpx/AODPA7RBbbH76hQHtgv2ZnA?rlkey=xxzaxfc4bc6whtpkb2kf3jd8h&st=fmbu84rn&dl=0

[mayofaulkner]

Hi Jay,

Thanks for the data. For the first issue please can you do pip install ibllib --upgrade and run the extraction code again.

For the second issue about the location of the histology. There are two things, the first is that the data you have is in pixel coordinates so you need to multiply by the resolution of the atlas used by AP-histology (10 um) to convert to um and the second is that it seems the ordering is mldvap

The following code should be get you the correct xyz_picks

import numpy as np
from pathlib import Path
import json
from iblatlas.atlas import AllenAtlas

atlas = AllenAtlas(25)

brainreg_path = Path(r'C:\Neuropixels_sorting_folder\JL101_20240914_g0\JL101_1_processed\probe_ccf.npy')
picks = np.load(brainreg_path)
# Convert picks to um by multiplying by resolution of atlas
picks = picks * 10
# Swap axis to picks has order apdvml (current oder is a mldvap)
picks[:, [2, 0]] = picks[:, [0, 2]]

# Convert to mlapdv relative to bregma
xyz_mlapdv = atlas.ccf2xyz(picks, ccf_order='apdvml') * 1e6
xyz_picks = {'xyz_picks': xyz_mlapdv.tolist()}

output_path = Path(r'C:\Neuropixels_sorting_folder\JL101_20240914_g0\alf')
with open(Path(output_path, 'xyz_picks.json'), "w") as f:
    json.dump(xyz_picks, f, indent=2)

When I load it in the GUI i get the probe in this location, does it look reasonable?

[jel0624]

Thanks for your help!
The coordinates should be around OFC, and that seems to be around GPe(?)
maybe some axis got flipped?

[mayofaulkner]

Does this look more reasonable?

In that case you were correct with the apdvml order. All you need to do is multiply by 10 to ensure the coordinates are in um.

import numpy as np
from pathlib import Path
import json
from iblatlas.atlas import AllenAtlas

atlas = AllenAtlas(25)

brainreg_path = Path(r'C:\Neuropixels_sorting_folder\JL101_20240914_g0\JL101_1_processed\probe_ccf.npy')
picks = np.load(brainreg_path)
# Convert picks to um by multiplying by resolution of atlas
picks = picks * 10

# Convert to mlapdv relative to bregma
xyz_mlapdv = atlas.ccf2xyz(picks, ccf_order='apdvml') * 1e6
xyz_picks = {'xyz_picks': xyz_mlapdv.tolist()}

output_path = Path(r'C:\Neuropixels_sorting_folder\JL101_20240914_g0\alf')
with open(Path(output_path, 'xyz_picks.json'), "w") as f:
    json.dump(xyz_picks, f, indent=2)

[jel0624]

Yes, this is the coordinates!! Thanks for your help, and will give this a try now

[jel0624]

Unfortunately, I still got this error while trying to extract the data.

[jel0624]

I fixed it by changing the _phy_spikes_subset.waveforms.npy to _phy_spikes_subset_waveforms.npy

For some reason, python didn't like the double dot notation.
Thanks!