Genomics Formats: From Reads to Variants
A research-grade tour of FASTQ, BAM/SAM, and VCF using 1000 Genomes micro-samples — theory, QC metrics, and interactive figures.
Modern short-read pipelines are a contract between file formats:
instruments emit FASTQ, aligners write BAM, and variant callers emit VCF.
Each format encodes both primary data and the QC language of the field (Phred, MAPQ, FILTER, INFO).
This dashboard re-parses tiny public subsets end-to-end and surfaces the same statistics a computational biologist would check before trusting a cohort-scale run.
1. End-to-end format pipeline
Where each format sits in an NGS secondary-analysis workflow (e.g. GATK Best Practices / nf-core/sarek).
Base calls + qualities. Adapter trim, contamination, and Phred decay live here.
Reference coordinates, CIGAR, MAPQ, duplicate marking, insert size.
Alleles, genotypes, INFO annotations, population AF, FILTER decisions.
Indexing is not optional at scale: .bai/.csi enable region queries on BAM;
.tbi does the same for bgzip VCF. The included indexed_tbi.vcf.gz demonstrates the compressed+indexed form used in production object stores.
2. FASTQ — base quality & composition
Phred scores quantify per-base error probability: Q = −10 log₁₀(p). Q30 ≈ 99.9% accuracy.
Phred quick reference
Encoding here: Sanger Phred+33 (Illumina 1.8+). Older Illumina Phred+64 is rare but still appears in archives.
Files
What to look for in real data: 3′ quality drop (polymerase fatigue), cycle-specific dips (fluidics), GC bias, and adapter-driven polyG/polyA tails. This micro-subset is fixed-length 36-mer historical Illumina — modern NovaSeq reads are longer (2×150) with different quality profiles.
3. BAM — alignment QC
SAM flags, MAPQ, CIGAR operations, insert-size distribution, and a coarse positional occupancy proxy.
MAPQ interpretation
CIGAR & flags in practice
M/EQ/X— alignment match (not necessarily sequence match forM)I/D— insertions/deletions relative to referenceS/H— soft/hard clips (adapters, structural variant breakpoints)- Flag 0x2 proper pair — both ends oriented and insert size within aligner expectations
- Duplicate flag — optical/PCR duplicates; mark before variant calling
This toy BAM is chr11-local and tiny; treat rates as illustrative, not cohort benchmarks.
4. VCF — variant landscape
SNP/indel mix, transition/transversion ratio, allele frequency, and multi-sample genotype occupancy.
Ti/Tv for whole-genome SNPs is typically ~2.0–2.1; exomes differ. Extreme deviations can flag systematic calling artifacts. Allele frequency spectra (many rare variants) are expected under population genetics; the multi-sample file exposes genotype matrices across 1,000+ individuals even when the variant count in this demo slice is small.
The three VCF panels compare: site-only basic.vcf, multi-sample genotypes, and the same sites as
bgzip + tabix (indexed_tbi.vcf.gz) — the form used for cloud-region queries.
5. Interactive 3D views
Rotate and zoom. Surfaces show per-read × per-cycle Phred structure; scatters expose multivariate QC.
Quality surface: X = cycle, Y = read index, Z = Phred — 3′ decay appears as a sloping face. Read cloud: mean Q vs GC vs end-of-read Q separates chemistry issues from composition. BAM cloud: genomic position × MAPQ × insert size highlights soft regions and improper pairs.
6. Format cheatsheet
Field-level reference for reading specs and debugging pipelines.
7. Discussion & next steps
These micro-files are intentionally tiny so the project stays git-friendly and CI-testable, but the metrics and failure modes scale unchanged to WGS/WES cohorts. In production you would add:
- FastQC / MultiQC aggregates; adapter and overrepresented k-mer reports
- Alignment summary metrics (Picard CollectAlignmentSummaryMetrics, CollectInsertSizeMetrics)
- Coverage uniformity (Mosdepth / GATK DepthOfCoverage) and fingerprinting
- Variant QC: ts/tv by functional class, het/hom ratios, Mendelian errors, VQSR / hard filters
- Joint calling / gVCF workflows and population VCF (bcftools, Hail, Glow)
Bridge to Project B: the same discipline of format literacy + quantitative QC applies when pathology
foundation-model embeddings are joined to spatial transcriptomics (h5ad) and WSI pyramids.
Generated by scripts/build_dashboard.py · analytics at outputs/phd_genomics_analytics.json ·
sample data from 1000 Genomes public resources · research use.