How DNA data travels from the sequencer to a list of mutations
We open three tiny public files that stand for the whole pipeline: raw reads (FASTQ), aligned reads (BAM), and called variants (VCF) — and show the QC checks scientists actually run.
A research-grade tour of FASTQ, BAM/SAM, and VCF using 1000 Genomes micro-samples — theory, QC metrics, and interactive figures.
For everyone: sequencing produces text files of DNA letters, then maps them to a reference genome,
then lists places that differ. This page opens tiny public examples of each step and shows the health checks labs run.
For specialists: end-to-end re-parse of FASTQ/BAM/VCF micro-samples with Phred surfaces, MAPQ/CIGAR/insert QC,
Ti/Tv + AF + genotypes, and format packaging (sites / multi-sample / bgzip+tabix). Hover or tap
i
for dual-register definitions.
What this sample shows — written twice: first for everyone, then for specialists.
-
The second read of each pair is noisier. R1 average quality is 27.0 vs R2 22.2 (drop of 4.7 Phred). That is common in older Illumina chemistry and is why QC plots show a 3′ slide. See chart →mean Phred R1=26.98, R2=22.23; fixed read length 36 bp; GC≈39.4%.
-
Almost every read found a place on the reference genome. 100% of reads mapped (mean MAPQ 55). About 97% of pairs look like well-behaved DNA fragments. See chart →mapped=79/79; mean MAPQ=55.03; insert mean±sd=376.4±29.7; CIGAR soft-clip bases=967.
-
Variants are mostly single-letter DNA changes. The multi-sample file has 25 variants across 1233 people; Ti/Tv=1.27 (whole-genome SNPs often sit near ~2.0–2.1 — this tiny slice is illustrative only). See chart →multisample: n_var=25, snps=25, ti=14, tv=11, mean AF=0.005976484841667116; site-only basic.vcf n_var=48.
-
The same variant sites appear in three packaging formats. One file is plain text sites, one adds genotypes for 1,000+ people, one is compressed+indexed for fast region queries — same science, different engineering contracts. See chart →basic.vcf (sites) vs basic_multisample.vcf (GT matrix) vs indexed_tbi.vcf.gz + .tbi (bgzip/tabix).
How many short DNA snippets are in the toy FASTQ files.
Sum of records across R1/R2 FASTQ subsets.
Average confidence in each letter of DNA (higher is better; 30 ≈ 99.9% accuracy).
Mean Phred across all bases; Sanger Phred+33 encoding.
Watch for 3′ decay on the quality-by-cycle plot.
Share of G/C letters — extreme GC can bias sequencing and alignment.
Mean GC fraction across reads.
How many reads successfully found a location on the reference genome.
mapped/total; mapping rate=1.000.
Average confidence that the chosen genomic location is correct (MAPQ).
Mean MAPQ; 60 is a common high-confidence BWA-MEM scale value.
How many individuals have genotype calls in the multi-sample VCF.
len(VCF.samples) for basic_multisample.vcf.
How many DNA differences are recorded in the multi-sample file (tiny demo region).
Variant records in basic_multisample.vcf.
Balance of two chemical types of single-letter DNA changes; odd values can flag bad variant calls.
Transitions/transversions among biallelic SNPs; WGS often ~2.0–2.1.
This micro-slice is not a cohort benchmark.
0. Why care about FASTQ/BAM/VCF — can’t Grok just “do the genomics”?
In plain termsChat models can explain genomics and write scripts. They do not replace the file formats instruments and pipelines actually exchange, or the QC math that catches broken runs.
Technical detailSecondary analysis is a contract between binary/text formats with indexes (BAI/TBI). LLMs assist authoring and interpretation; they are not substitutes for deterministic parsers (pysam, htslib) on production cohorts.
How to read thisUse LLMs to navigate; use format-native tools to measure.
Chat models can explain genomics and write scripts. They do not replace the file formats instruments and pipelines actually exchange, or the QC math that catches broken runs.
Secondary analysis is a contract between binary/text formats with indexes (BAI/TBI). LLMs assist authoring and interpretation; they are not substitutes for deterministic parsers (pysam, htslib) on production cohorts.
Use LLMs to navigate; use format-native tools to measure.
The “why” is interoperability and trust: sequencers, aligners, and callers only meet if everyone speaks FASTQ → BAM → VCF.
For everyone
DNA machines do not output a paragraph. They output standardized files that different software must open years later, on different clouds, with bit-exact coordinates. FASTQ, BAM, and VCF are those standards — like PDF for documents, but for letters of DNA, where they matched the genome, and how people differ.
A model like Grok 4.5 is excellent at teaching you what MAPQ means, writing a samtools one-liner,
or reviewing whether Ti/Tv looks suspicious. It should not be the system of record for 100 million reads.
Production genomics is still dominated by specialized engines (BWA, GATK, DeepVariant, bcftools) that stream these formats.
For specialists
Benchmarks that matter for this stack are not MMLU. They are: alignment accuracy / mapping rate, coverage uniformity, variant F1 on Genome in a Bottle (GIAB), Mendelian consistency, and runtime/cost on WGS. File-level QC (Phred decay, insert-size modes, FILTER distributions) is how you know a run is usable before any ML.
LLMs help with orchestration and explanation; they do not define reference-compressed CIGAR semantics
or tabix random access. Multimodal chat cannot replace a .bai over a 100× CRAM.
Capability map
| Job | Format-native tools | Frontier LLM (e.g. Grok 4.5) |
|---|---|---|
| Parse 10⁸ reads | Primary (htslib/pysam) | Wrong abstraction |
| QC a lane / library | Primary (FastQC, Picard, mosdepth) | Interpret reports |
| Call variants vs GIAB | Primary (GATK, DV, happy) | Discuss discordant sites |
| Explain Ti/Tv to a PI | Plots + stats | Primary |
| Write a pipeline | nf-core / WDL / Snakemake | Strong pair-programmer |
Pathology Project B is the vision-FM analogue: specialist models for tiles; LLMs for the scientific narrative.
1. The three stops on the DNA data journey
In plain termsDNA comes off the machine as letter strings with quality scores, is pasted onto a reference genome, then differences are listed as variants.
Technical detailFASTQ → BAM/SAM (+BAI) → VCF/BCF (+TBI); typical of GATK Best Practices / nf-core/sarek secondary analysis.
How to read thisUse this as a map; later sections zoom into QC at each stop.
DNA comes off the machine as letter strings with quality scores, is pasted onto a reference genome, then differences are listed as variants.
FASTQ → BAM/SAM (+BAI) → VCF/BCF (+TBI); typical of GATK Best Practices / nf-core/sarek secondary analysis.
Use this as a map; later sections zoom into QC at each stop.
Where FASTQ, BAM, and VCF sit in a standard short-read workflow.
Base calls + qualities. Adapter trim, contamination, and Phred decay live here.
Reference coordinates, CIGAR, MAPQ, duplicate marking, insert size.
Alleles, genotypes, INFO annotations, population AF, FILTER decisions.
Indexing is not optional at scale: .bai/.csi enable region queries on BAM;
.tbi does the same for bgzip VCF. The included indexed_tbi.vcf.gz demonstrates the compressed+indexed form used in production object stores.
2. Raw reads: how sure are we about each DNA letter?
In plain termsEach letter gets a confidence score. High scores mean the machine is sure; scores often fall toward the end of a read.
Technical detailPhred Q = −10 log10(p_error); Sanger Phred+33. Per-cycle means, GC, base composition, read cloud.
How to read thisCompare R1 vs R2 curves; look for sudden cycle dips (fluidics) or adapter-driven tails.
Each letter gets a confidence score. High scores mean the machine is sure; scores often fall toward the end of a read.
Phred Q = −10 log10(p_error); Sanger Phred+33. Per-cycle means, GC, base composition, read cloud.
Compare R1 vs R2 curves; look for sudden cycle dips (fluidics) or adapter-driven tails.
Phred quality, GC content, and base mix — the first QC gate after sequencing.
Phred quick reference
Encoding here: Sanger Phred+33 (Illumina 1.8+). Older Illumina Phred+64 is rare but still appears in archives.
Files
What to look for in real data: 3′ quality drop (polymerase fatigue), cycle-specific dips (fluidics), GC bias, and adapter-driven polyG/polyA tails. This micro-subset is fixed-length 36-mer historical Illumina — modern NovaSeq reads are longer (2×150) with different quality profiles.
3. Alignments: did reads land in the right place?
In plain termsSoftware sticks each short read onto a reference genome. We check confidence, pair orientation, and whether pieces were clipped off.
Technical detailMAPQ, SAM flags, CIGAR ops, insert-size, strand balance, start-position occupancy.
How to read thisHigh MAPQ mass + proper pairs + sensible insert sizes are healthy PE libraries.
Software sticks each short read onto a reference genome. We check confidence, pair orientation, and whether pieces were clipped off.
MAPQ, SAM flags, CIGAR ops, insert-size, strand balance, start-position occupancy.
High MAPQ mass + proper pairs + sensible insert sizes are healthy PE libraries.
Mapping quality, pair geometry, and CIGAR surgery on a tiny BAM.
MAPQ interpretation
CIGAR & flags in practice
M/EQ/X— alignment match (not necessarily sequence match forM)I/D— insertions/deletions relative to referenceS/H— soft/hard clips (adapters, structural variant breakpoints)- Flag 0x2 proper pair — both ends oriented and insert size within aligner expectations
- Duplicate flag — optical/PCR duplicates; mark before variant calling
This toy BAM is chr11-local and tiny; treat rates as illustrative, not cohort benchmarks.
4. Variants: which DNA letters differ?
In plain termsA VCF is a spreadsheet of DNA differences: which site, what change, how confident, and (optionally) who carries it.
Technical detailSNP/indel mix, Ti/Tv, AF spectrum, genotype counts, site-only vs multi-sample vs tabix packaging.
How to read thisTreat Ti/Tv on this micro-slice as pedagogy, not a production cohort metric.
A VCF is a spreadsheet of DNA differences: which site, what change, how confident, and (optionally) who carries it.
SNP/indel mix, Ti/Tv, AF spectrum, genotype counts, site-only vs multi-sample vs tabix packaging.
Treat Ti/Tv on this micro-slice as pedagogy, not a production cohort metric.
SNP spectrum, frequencies, and genotypes across three VCF packagings.
Ti/Tv for whole-genome SNPs is typically ~2.0–2.1; exomes differ. Extreme deviations can flag systematic calling artifacts. Allele frequency spectra (many rare variants) are expected under population genetics; the multi-sample file exposes genotype matrices across 1,000+ individuals even when the variant count in this demo slice is small.
The three VCF panels compare: site-only basic.vcf, multi-sample genotypes, and the same sites as
bgzip + tabix (indexed_tbi.vcf.gz) — the form used for cloud-region queries.
5. See quality in 3D
In plain termsSpin the quality surface: one axis is position along the read, one is which read, and height is confidence.
Technical detailPlotly surface (reads×cycles×Phred) and scatter3d clouds for FASTQ and BAM multivariate QC.
How to read thisDrag to rotate; look for sloping 3′ faces on R2.
Spin the quality surface: one axis is position along the read, one is which read, and height is confidence.
Plotly surface (reads×cycles×Phred) and scatter3d clouds for FASTQ and BAM multivariate QC.
Drag to rotate; look for sloping 3′ faces on R2.
Interactive surfaces and clouds — drag to rotate.
Quality surface: X = cycle, Y = read index, Z = Phred — 3′ decay appears as a sloping face. Read cloud: mean Q vs GC vs end-of-read Q separates chemistry issues from composition. BAM cloud: genomic position × MAPQ × insert size highlights soft regions and improper pairs.
6. Pocket reference
In plain termsA short cheat sheet of what each file type stores and which tools open it.
Technical detailRecord layouts, indexes (.bai/.tbi), and common CLI tools.
How to read thisUse when debugging pipeline hand-offs between steps.
A short cheat sheet of what each file type stores and which tools open it.
Record layouts, indexes (.bai/.tbi), and common CLI tools.
Use when debugging pipeline hand-offs between steps.
Fields, indexes, and common tools for FASTQ / BAM / VCF.
7. What this demo cannot claim (and good next steps)
These micro-files are intentionally tiny so the project stays git-friendly and CI-testable, but the metrics and failure modes scale unchanged to WGS/WES cohorts. In production you would add:
- FastQC / MultiQC aggregates; adapter and overrepresented k-mer reports
- Alignment summary metrics (Picard CollectAlignmentSummaryMetrics, CollectInsertSizeMetrics)
- Coverage uniformity (Mosdepth / GATK DepthOfCoverage) and fingerprinting
- Variant QC: ts/tv by functional class, het/hom ratios, Mendelian errors, VQSR / hard filters
- Joint calling / gVCF workflows and population VCF (bcftools, Hail, Glow)
Bridge to Project B: the same discipline of format literacy + quantitative QC applies when pathology
foundation-model embeddings are joined to spatial transcriptomics (h5ad) and WSI pyramids.
Generated by scripts/build_dashboard.py · analytics at outputs/phd_genomics_analytics.json ·
sample data from 1000 Genomes public resources · research use.