Download data error

[123chenshixin]

Hi,

Thank you for developing and sharing the Fullscope-seq pipeline.

I am trying to download and analyze the Fullscope-seq ONT sequencing datasets provided in your repository. However, I encountered an issue with several downloaded FASTQ files.

The problematic files are:

OED0132146_T44_ONT_fullscope.fq.gz
OED01324280_T44_cyclone_fullscope.fq.gz
OED01324279_T47_cyclone_fullscope.fastq.gz
T44_direct_ONT-1
T44_direct_ONT-5
T44_direct_ONT-6

After completing the download, I found that the MD5 checksums of these files do not match the provided MD5 values. Moreover, the files cannot be properly decompressed or read.

For example: gzip -d OED0132146_T44_ONT_fullscope.fq.gz, returns an error.

I also tried checking the files using sequencing file validation tools (e.g., seqkit), but they reported errors, suggesting that the FASTQ files may be incomplete or corrupted.
Interestingly, another smaller dataset:

T47_ONT_fullscope_20240104.fastq.gz

was downloaded and processed successfully without any issues.

Could you please help confirm whether:

1.These FASTQ files on the server are intact?
2.The download links or files need to be updated?
3.There is any recommended method for downloading these large datasets?

Thank you very much for your help.

Best regards

[Y0NEKO]

Sorry for the late reply. I checked the database and confirmed this issue — it seems the data was corrupted during upload. I've contacted the NODE admin to replace it. Please wait a few days, and I'll notify you as soon as the re-upload is complete. Thanks for your understanding.

[Y0NEKO]

Data restored. Please revisit the original link and let us know if any issues remain.

[camelest]

@Y0NEKO Hi, I have had the same problem and thank you for updating on them. I'm trying on the re-uploaded files now.

I also had errors on the listed 2 Revio bam files:
OED01322057_T44_Pacbio_fullscope2.bam (md5: f1f9ae945c2c283dc557b24b9bd400f1)
OED01322892_T47_pacbio_fullscope.bam (md5: d949143212ecee035a920bcd04e44b1a)
I couldn't fine the provided md5 values so I listed mine.

And in addition, these files do not have matched Reads/segments counts as shown in Supplementary Table 1.
OED01322888_T42_fullscope_pacbio.bam (163868744 instead of 167015342)
OED01324183_M2-VM1106-cortex-pacbio-fullscope.bam (51334850 instead of 40095075)
OED01324184_M3-ME01-cortex-pacbio-fullscope.bam (66773470 instead of 36932954)

Could these files also have been corrupted? Thank you very much for your help.

[Y0NEKO]

Hi,

Thank you for your detailed feedback.
I am currently looking into the two Revio BAM files you listed. If they are indeed corrupted, I will re-upload the corrected versions and let you know.
Regarding the read/segment count discrepancies you noted for the three additional files, please note that these filenames correspond to the following sample names in Supplementary Table 1. Please check the counts listed under these sample names instead:
OED01322888_T42_fullscope_pacbio.bam → Macaque1-T3
OED01324183_M2-VM1106-cortex-pacbio-fullscope.bam → Macaque2-T1
OED01324184_M3-ME01-cortex-pacbio-fullscope.bam → Macaque3-T1
Please let me know if the counts still do not match when referenced this way, or if you suspect these files are corrupted as well.

Thank you again for bringing this to our attention.

[camelest]

Thank you very much for your prompt responses.
I still seem to have unmatched numbers of reads according to the sample names you kindly provided.

(By the way, in NODE link, Fullscope-Pacbio-revio-M1-2 is labeled to OED01322888_T42_fullscope_pacbio.bam. Could you double check that this this sample does not correspond to Macaque1-T2 but T3?)

$ wc -l */Fqsegment/*_fragment.tsv.summary.tsv
  163868744 OED01322888_T42_fullscope_pacbio/Fqsegment/OED01322888_T42_fullscope_pacbio_fragment.tsv.summary.tsv
   51334850 OED01324183_M2-VM1106-cortex-pacbio-fullscope/Fqsegment/OED01324183_M2-VM1106-cortex-pacbio-fullscope_fragment.tsv.summary.tsv
   66773470 OED01324184_M3-ME01-cortex-pacbio-fullscope/Fqsegment/OED01324184_M3-ME01-cortex-pacbio-fullscope_fragment.tsv.summary.tsv

For references, these were the expected numbers from Supplementary Table 1 I was referring to.

Macaque1-T3 Fullscope-Pacbio 267,813,324
Macaque2-T1 Fullscope-Pacbio 40,095,075
Macaque3-T1 Fullscope-Pacbio 36,932,954

Thank you very much!

[camelest]

@Y0NEKO Hi, I successfully confirmed that updated fastq files did not give errors in pigz -dc. @123chenshixin

I still have problems matching #segmented reads from Supplementary Table 1.
I have attached the #segmented reads I obtained using fullscope v1.3.0 and would really appreciate your help to debug it.

T44_direct_ONT-{1..6}.fq.gz
468,361,833
T44_ONT_fullscope.fragment.fastq.summary.tsv
389,214,460
T44_cyclone_fullscope.fragment.fastq.summary.tsv
209,162,026
T44_Pacbio_fullscope{1,2}_fragment.tsv.summary.tsv (error)
199,058,878
T42_fullscope_pacbio_fragment.tsv.summary.tsv
163,868,744
T47_cyclone_fullscope.fragment.fastq.summary.tsv
233,411,650
T47_ONT_fullscope_20240104.fragment.fastq.summary.tsv
7,354,997
T47_pacbio_fullscope_fragment.tsv.summary.tsv (error)
24,900,956
M2-VM1106-cortex-pacbio-fullscope_fragment.tsv.summary.tsv
51,334,850
M3-ME01-cortex-pacbio-fullscope_fragment.tsv.summary.tsv
66,773,470

Thank you very much.