Skip to content

fix_vcf_ploidies: bgzip must stay a filter, not take a filename - #1989

Merged
glennhickey merged 1 commit into
masterfrom
fix-vcf-ploidy-bgzip
Aug 31, 2026
Merged

fix_vcf_ploidies: bgzip must stay a filter, not take a filename#1989
glennhickey merged 1 commit into
masterfrom
fix-vcf-ploidy-bgzip

Conversation

@glennhickey

Copy link
Copy Markdown
Collaborator

14d05895 ("log input vcf path", merged in #1982) added raw_out_path as a positional argument to the bgzip call in fix_vcf_ploidies, while leaving cactus_call's infile=/outfile= redirects in place. A filename argument switches bgzip from a stdin/stdout filter to compressing in place, and the two cannot be combined.

cactus_call stages stdout under outfile + '.part' and renames it into place only on clean exit, so out_vcf_path does not exist when bgzip runs, and bgzip takes the in-place path:

  1. bgzip writes the real compressed VCF to raw_out_path + '.gz' — which is out_vcf_path — and deletes raw_out_path. It exits 0.
  2. cactus_call sees success and os.replace()s its empty .part file over out_vcf_path, discarding what bgzip just wrote.
  3. os.remove(raw_out_path) raises FileNotFoundError, because bgzip took it.

Step 3 is the only reason this fails loudly. By then the output is already gone, so anything that swallowed that exception would ship an empty VCF with exit 0.

Reachability

Only when the VCF actually needs ploidy padding. The scan above returns early via shutil.copyfile when every sample has one ploidy throughout, so a uniformly-diploid run never reaches this line. Mixed ploidy — chrY alongside the autosomes — is the common case that does.

Fix

Reverts to the form d5218801 introduced, and that minigraph_gfa_to_pansn uses: no filename, infile/outfile only.

Verification

Against the bgzip in bin/:

  • with a filename and no pre-existing .gz: deletes the input, exits 0 — the failure above
  • as a filter: leaves the input alone, writes 183 bytes for a 3-line test VCF, and round-trips byte-identically through bgzip -dc

Observed in the wild as repeated FileNotFoundError: .../clip.{raw,bub,wave}.ploidy.vcf from vcf_cat on a 30-way HPRC run.

14d0589 ("log input vcf path") added raw_out_path as a positional argument to
the bgzip call while leaving cactus_call's infile=/outfile= redirects in place.
The intent was to get the path into the logged command line, but a filename
argument switches bgzip from a stdin/stdout filter to compressing in place, and
the two cannot be combined.

cactus_call stages stdout under outfile + '.part' and only renames it into place
once the command exits cleanly, so out_vcf_path does not exist when bgzip runs
and bgzip takes the in-place path:

  1. bgzip writes the real compressed VCF to raw_out_path + '.gz', which IS
     out_vcf_path, and deletes raw_out_path.  It exits 0.
  2. cactus_call sees success and os.replace()s its empty .part file over
     out_vcf_path, discarding what bgzip just wrote.
  3. os.remove(raw_out_path) raises FileNotFoundError, because bgzip took it.

Step 3 is the only reason this is loud.  By then the output is already gone, so
anything that swallowed that exception would ship an empty VCF with exit 0.

Only reachable when the VCF actually needs ploidy padding: the scan above returns
early via shutil.copyfile when every sample has one ploidy throughout, so a run
whose samples are uniformly diploid never gets here.  Mixed ploidy -- chrY
alongside the autosomes -- is the common case that does.

Reverts to the form d521880 introduced and that minigraph_gfa_to_pansn uses:
no filename, infile/outfile only.  Verified against the bgzip in bin/: with a
filename and no pre-existing .gz it deletes the input and exits 0 (the failure
above); as a filter it leaves the input alone, writes 183 bytes for a 3-line
test VCF, and round-trips byte-identically through bgzip -dc.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@glennhickey
glennhickey merged commit 40ff8a9 into master Aug 31, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant