SeqChunker error

[cmorganl]

Hi @thackl,

I'm encountering an issue very similar to #5 where proovread is unable to locate SeqChunker-perl.

However, after recreating the symlink as well as using the quick and dirty workaround of directly copying SeqChunker-perl to proovread/bin/ it still isn't able to locate it and I don't know what working directory its attempting to call it from. Maybe this stems from the readlink command instead? Here's the repeated error message:

readlink: illegal option -- f
usage: readlink [-n] [file ...]
Can't open perl script "./SeqChunker-perl": No such file or directory

Finally resulting in:

12:30:13] No sequences found matching given filter criteria
Use of uninitialized value $c in ord at /Users/Connor/bin/proovread/bin/../lib//Fastq/Parser.pm line 355.
Use of uninitialized value $c in string eq at /Users/Connor/bin/proovread/bin/../lib//Fastq/Parser.pm line 359.
[16-02-26 12:30:13] [siamaera] Input neither FASTA nor FASTQ

This is a different computer than what I was running in a previous issue, running Mac10.6.8 and perl version 5.18. Let me know if you need any more information.

Thanks for your help.

[thackl]

Hi,

you are probably right with the readlink problem. I am not that familiar with MacOS, I assigned the issue to my colleague Frank - he fixed the Mac related issue last time.

Cheers
Thomas

[greatfireball]

I am sorry, not to response earlier. Indeed it is a problem with readlink. The Linux version of readlink supports the parameter -f while the MacOS/BSD readlink does not. I created and preapred a fix for that issue, which should be integrated into the next proovread release.

[greatfireball]

The new version 0.4 of SeqChunker is supporting MacOSX, therefore this should resolve that issue. @thackl needs to approve the pull request, afterwards version 2.12.13 of proovread should work on Macs. @cmorganl are you able to run a test on your sequence data after the merge into our master branch? Just make sure you are using at least version 2.12.13 of proovread.

Let me know, if that solves your problem.

Best,
Frank

[cmorganl]

Hi @greatfireball, I'm still experiencing an issue running Mac 10.6.8. The initial error has been fixed but another has cropped up. Here is the stdout:

/Users/Connor/bin/proovread/bin/SeqChunker --chunk-number 100000 --chunk-step 2\
0 --chunks-per-step 8 --first-chunk 1 /Volumes/CAPTAIN/Hallam_projects/MAP_proj\
ects/RawData/Sakinaw_Lake/Sakinaw_Lake_Illumina_120m.fastq | /Users/Connor/bin/\
proovread/bin/../util/bwa/bwa-proovread mem -b 50 -l 250 -Y  -L 30,30 -a  -E 4,\
3 -r 1 -A 5 -T 2.5 -O 2,1 -t 8 -w 40 -W 20 -y 20 -B 10 -D 0 -k 12 SakinawLake_1\
20mMay2011_allD/read-long/SakinawLake_120mMay2011_allD.masked.fa /dev/fd/0 2>Sa\
kinawLake_120mMay2011_allD/bwa-sr-1/SakinawLake_120mMay2011_allD_bwa.log | /Use\
rs/Connor/bin/samtools-1.1/samtools view -@ 8 -bS /dev/fd/0 >SakinawLake_120mMa\
y2011_allD/bwa-sr-1/SakinawLake_120mMay2011_allD_tmp.bam
#------------------------------------------------------------------------------#
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix 
/Users/Connor/bin/proovread/bin/SeqChunker: line 17: : No such file or directory
/Users/Connor/bin/proovread/bin/SeqChunker: line 23: : No such file or directory
cat: : No such file or directory
/Users/Connor/bin/proovread/bin/../util/SeqChunker/bin/SeqChunker
rm: : No such file or directory
FILE should be located at '/Users/Connor/bin/proovread/bin/../util/SeqChunker/bin/SeqChunker'

SakinawLake_120mMay2011_allD/bwa-sr-1/SakinawLake_120mMay2011_allD_bwa.log contains the two lines:

[M::bwa_idx_load_from_disk] read 0 ALT contigs
[M::main_mem] read 533334 sequences (80000100 bp)...

Let me know if you need futher information.

[ashaneev]

hiii
when i'm trying to split my data(1.5gb) and it is showing that,
SeqChunker -s 1G -o mapped.reds.pac.fa mapped.reds.pac-subreads.fq
SeqChunker: command not found
Need help. i'm using ubuntu 14.04

[thackl]

Try to call SeqChunker with its complete path /path/to/SeqChunker -s 1G .... Does that work?

[ashaneev]

No...!! Showing like

FILE should be located at '/home/Documents/Tools_NGS/proovread/bin/../util/SeqChunker/bin/SeqChunker'

[thackl]

Does the SeqChunker file exist at the location it says above?

[ashaneev]

yes. It is there. but the folder shows a lock image. When i try to call with its complete path, showing like "FILE should be located at '/home/Documents/Tools_NGS/proovread/bin/../util/SeqChunker/bin/SeqChunker'
"

[thackl]

Hm, no idea what the lock means. The "File should ..." message itself is not an error, just a log message. Here is how things look like when I run SeqChunker

~/software/proovread/util/SeqChunker/bin/SeqChunker -n 2 foo.fa
FILE should be located at '/home/thackl/software/proovread/util/SeqChunker/bin/SeqChunker'
Reading foo.fa
>seq-shit
ATTAGTCTCACAATATTCGATAGTTTAAGTAGGACAGTAGCTGCAGAGAGGACCCAGCTATCGATTTTGCGACGAGAGCC
TCGCCGTGGTATTGCTTCTGAAAGGGGGTAGGGTTCCCATATGTTAACTGCAAAGGCAACTAACCTTGGGTCAATCCTCT
CTCTGAATCCGCATATAATAGAACCCCAAATTGTTTGAATTTTAGGTGGTCGGCAACCCCGCTACACTAATATGACATCA
...

Have you tried it with a fasta/fastq file?

[ashaneev]

I tried with fastq file about size of 1.7 gb(mitochondrial sequence). Then i skipped the seqchunker step and run proovread directly.
But, it also stopped after few hours and showing like,

[Tue Apr 2 04:16:36 2019] bam2cns unexpectedly returned
Exited at 'main'
'/home/Documents/Tools_NGS/proovread/bin/proovread', line 1636
Last call 'main::correct_sr_mt'
'/home/Documents/Tools_NGS/proovread/bin/proovread', line 868

Command used is given below..

**> /home/Documents/Tools_NGS/proovread/bin/proovread

-l pacbio.fa -s /media/TOSHIBAEXT/illumina/R1.fastq -s /media/TOSHIBAEXT/illumina/R2.fastq -t 20 --overwrite -p pacbio_corrected

**
Could you please have look into this issue.Thank you.

[thackl]

OK, this won't work because proovread also uses SeqChunker internally. However, I don't think that the error you got from bam2cns is even related to this. To really be able to help you I need some more information. How did you install proovread in the first place, did you just copy and paste from the install instructions? Did you move the folder with the files afterwards? And could you attach the complete log output of the crashed run? And lastly, have you tried to run proovreads example dataset cd proovread; make sample?

[ashaneev]

I've installed from the root by using 'sudo' command and didn't move any files from the proovread folder.Also used 'make sample' command.but,it ran smoothly without showing any error messages.Please find the attached log file.Thank you.

home@home-Lenovo-H30-50:~/Documents/Tools_NGS/proovread$ /home/Documents/Tools_NGS/proovread/bin/proovread -l chloro.pacbio.fa -s /media/TOSHIBAEXT/chloroplast_reads/illumina/R1.fastq -s /media/TOSHIBAEXT/chloroplast_reads/illumina/R2.fastq -t 40 --overwrite -p pacbio_proovread
[Tue Apr 2 15:01:23 2019] Running proovread-2.14.1 under Perl 5.18.2
[Tue Apr 2 15:01:23 2019] Reading core config
[Tue Apr 2 15:01:23 2019] Reading command line options
[Tue Apr 2 15:01:23 2019] Logging parameter to pacbio_proovread/pacbio_proovre ad.parameter.log
[Tue Apr 2 15:01:23 2019] Checking short read files

R1.fastq
Detected FASTQ format
Estimated short read length: 150 +-0
Etimated short read quality offset: 33
Estimating approximate number of short reads: 5.09M

R2.fastq
Detected FASTQ format
Estimated short read length: 150 +-0
Etimated short read quality offset: 33
Estimating approximate number of short reads: 5.09M

[Tue Apr 2 15:02:29 2019] Checking required binaries

[ok] samtools-1.7 /usr/local/bin
[ok] bwa-proovread /home/Documents/Tools_NGS/proovread/bin/../util
/bwa
[ok] blastn-2.7.0 /usr/local/bin

[Tue Apr 2 15:02:29 2019] Running mode: sr
[Tue Apr 2 15:02:29 2019] Preparing long reads
Skipped 2454 stubby reads with length <300
#------------------------------------------------------------------------------#
perl -I/home/Documents/Tools_NGS/proovread/bin/../lib/ /home/Doc
uments/Tools_NGS/proovread/bin/SeqFilter --fasta --in pacbio_proovread/read-
long/pacbio_proovread.fq --line-width 80 --lower-case --out pacbio_proovread/re
ad-long/pacbio_proovread.masked.fa --phred-offset 33
[15:03:19] /home/Documents/Tools_NGS/proovread/bin/SeqFilter-1.06
[15:03:19] --in: pacbio_proovread/read-long/pacbio_proovread.fq
[15:03:19] Detected FASTQ format, phred-offset 33
2.27G [=========================================================>] TTS 00:00:15
#------------------------------------------------------------------------------#
[15:03:34] Input
[15:03:34] Sequences 132092 #
[15:03:34] Total 1129581834 bp
[15:03:34] Longest 47305 bp
[15:03:34] Shortest 300 bp
[15:03:34] N50 12828 bp
[15:03:34] N90 4862 bp
#------------------------------------------------------------------------------#
[15:03:34] Filtered
[15:03:34] Sequences 132092 #
[15:03:34] Total 1129581834 bp
[15:03:34] Longest 47305 bp
[15:03:34] Shortest 300 bp
[15:03:34] N50 12828 bp
[15:03:34] N90 4862 bp
#------------------------------------------------------------------------------#

#------------------------------------------------------------------------------#
#------------------------------------------------------------------------------#
perl -I/home/Documents/Tools_NGS/proovread/bin/../lib/ /home/Doc
uments/Tools_NGS/proovread/bin/ccseq --bwa-path /home/Documents/Too
ls_NGS/proovread/bin/../util/bwa/ --threads 40 --pre pacbio_proovread/ccs-1 <pa
cbio_proovread/read-long/pacbio_proovread.fq | tee pacbio_proovread/ccs-1/pacbi
o_proovread.fq | perl -I/home/Documents/Tools_NGS/proovread/bin/../lib
/ /home/Documents/Tools_NGS/proovread/bin/SeqFilter --line-width 80 --
phred-offset 33 --phred-mask 20,41,120,195,60,0.7 --fasta --out pacbio_proovrea
d/ccs-1/pacbio_proovread.masked.fa --base-content N --tsv - --quiet
[19-04-02 15:03:35] [ccseq] Reading STDIN
[19-04-02 22:21:36] [ccseq] Processed 132101 subreads from 109127 reads
[19-04-02 22:21:36] [ccseq] 18678 consensus + 90449 bypassed single subreads
#------------------------------------------------------------------------------#

[Tue Apr 2 22:21:53 2019] Running task bwa-sr-1
[Tue Apr 2 22:21:53 2019] Indexing Long reads
#------------------------------------------------------------------------------#
/home/Documents/Tools_NGS/proovread/bin/../util/bwa/bwa-proovread inde
x pacbio_proovread/ccs-1/pacbio_proovread.masked.fa pacbio_proovread/ccs-1/pacb
io_proovread.masked.fa 2>pacbio_proovread/bwa-sr-1/pacbio_proovread_bwa.log
#------------------------------------------------------------------------------#
[Tue Apr 2 22:35:09 2019] Mapping reads /media/TOSHIBAEXT/chloroplast_re ads/illumina/R1.fastq /media/TOSHIBAEXT/chloroplast_reads/illum ina/R2.fastq
#------------------------------------------------------------------------------#
/home/Documents/Tools_NGS/proovread/bin/SeqChunker --chunk-number 1000
--chunk-step 20 --chunks-per-step 6 --first-chunk 1 /media/TOSHIBAEXT/ch
loroplast_reads/illumina/R1.fastq /media/TOSHIBAEXT/chloroplas
t_reads/illumina/R2.fastq | /home/Documents/Tools_NGS/proov
read/bin/../util/bwa/bwa-proovread mem -b 20 -l 300 -W 20 -L 30,30 -O 2,1 -T 2.
5 -k 12 -D 0 -Y -a -E 4,3 -A 5 -B 11 -r 1 -t 40 -y 20 -w 40 pacbio_proovread/
ccs-1/pacbio_proovread.masked.fa /dev/fd/0 2>pacbio_proovread/bwa-sr-1/pacbio_p
roovread_bwa.log | samtools view -@ 40 -bS /dev/fd/0 >pacbio_proovread/bwa-sr-1
/pacbio_proovread_tmp.bam
#------------------------------------------------------------------------------#
/home/Documents/Tools_NGS/proovread/bin/../util/SeqChunker/bin/SeqChunker
FILE should be located at '/home/Documents/Tools_NGS/proovread/bin/../util/SeqChunker/bin/SeqChunker'
Reading /media/TOSHIBAEXT/chloroplast_reads/illumina/R1.fastq
Reading /media/TOSHIBAEXT/chloroplast_reads/illumina/R2.fastq
Killed

[Tue Apr 2 23:22:52 2019] Sorting BAM pacbio_proovread/bwa-sr-1/pacbio_proovre ad.bam
#------------------------------------------------------------------------------#
samtools sort -m 2G -@ 21 -T pacbio_proovread/bwa-sr-1/pacbio_proovread_tmp -o
pacbio_proovread/bwa-sr-1/pacbio_proovread.bam pacbio_proovread/bwa-sr-1/pacbio
_proovread_tmp.bam
#------------------------------------------------------------------------------#
samtools sort: couldn't allocate memory for bam_mem
#------------------------------------------------------------------------------#
samtools sort failed: 256

Exited at 'main'
'/home/Documents/Tools_NGS/proovread/bin/proovread', line 1343
Last call 'main::create_sorted_bam'
'/home/Documents/Tools_NGS/proovread/bin/proovread', line 865

[thackl]

Ah, ok, that helps. Your actual problem is not coming from proovread. Have a look at the last command in the log to samtools sort. It failed with a out of memory issue. It is trying to allocate 2GB. How much memory does you computer have?

The 2GB are currently hard coded, you could change them by opening the proovread file in an editor and change line 1338 samtools sort -m <max-ram>G. However, if you really have less than 2GB of RAM, I can't promise that proovread won't run out of memory eventually...

[ashaneev]

my system has 16 gb of ram size,but still facing the same issue.

[thackl]

OK, that is odd. Can you try and just run that samtools line directly?

 samtools sort -m 2G -@ 21 -T pacbio_proovread/bwa-sr-1/pacbio_proovread_tmp -o
pacbio_proovread/bwa-sr-1/pacbio_proovread.bam pacbio_proovread/bwa-sr-1/pacbio
_proovread_tmp.bam

[thackl]

Hm, thinking about it, it probably means that other things on your system (including proovread) are taking up most of the RAM. Can you monitor the RAM consumption while running proovread?