fq-compressor is a C++23 FASTQ compressor built around a small, sequential, bounded-memory
archive format. It preserves read IDs, comments, sequences, quality strings, and paired-read
adjacency. Plain FASTQ and .gz FASTQ inputs are supported.
FQC v2 is intentionally incompatible with the legacy v1 archive and CLI. There is no v1 reader, migration command, random access, lossy mode, or original-order map.
The previous ABC/SCM implementation combined an expensive global analysis path, dense per-thread quality models, and two separate serial/parallel engines. Its tracked throughput was roughly 0.1 MiB/s. V2 replaces that stack with independent sequential frames and one engine shared by compression, decompression, and verification.
The production fallback uses:
- varint-framed ID/comment streams compressed with Zstd level 1;
- 2-bit uppercase A/C/G/T sequence packing plus exact-position exceptions for every other IUPAC symbol and lowercase base, followed by Zstd level 1;
- varint-framed quality streams compressed with Zstd level 1;
- XXH64 checksums for the global header, each logical frame, and footer totals.
Dataset profiles (illumina, ont, pacbio-hifi, and pacbio-clr) are recorded explicitly and
can be detected or overridden. They currently share the validated fallback codec. A specialised
codec is not admitted unless it keeps the throughput floor and beats fallback size by at least 5%
on the benchmark corpus without a per-dataset regression above 1%.
Requirements: CMake 3.28+, Conan 2.x, GCC 14+ or Clang 18+.
conan profile detect --force
./scripts/build.sh clang-releaseThe executable is build/clang-release/src/fqc.
# Single-end; profile is auto-detected.
fqc compress -i reads.fastq -o reads.fqc
# Paired files. Each R1/R2 pair stays adjacent and each frame contains complete pairs.
fqc compress -i reads_R1.fastq -2 reads_R2.fastq -o paired.fqc
# Explicit long-read profile.
fqc compress -i ont.fastq.gz -o ont.fqc --profile ont
# Full integrity verification and decompression.
fqc verify reads.fqc
fqc decompress -i reads.fqc -o restored.fastq
# Process pipelines.
producer | fqc compress -i - -o - | consumer
fqc decompress -i reads.fqc -o - | downstream-toolGlobal options may appear before or after the subcommand:
--memory-limit MiB Hard operation budget; default 16384, minimum 64
-q, --quiet Suppress non-error status messages
Compression additionally accepts --profile, --frame-mib, and --force. Decompression accepts
--force. Paired archives decompress as one canonical interleaved FASTQ stream.
FQC v2 consists of a checked global header, zero or more independent frames, and an end footer. The reader rejects unknown versions/codecs, oversized frames, inconsistent paired counts, truncation, trailing logical-stream or post-footer data, and checksum mismatches. It does not probe or decode v1.
See ARCHITECTURE.md for the byte layout and memory model.
Release CLI measurements on an 8-core AMD Ryzen 7 5800H x86_64 host include FASTQ parsing, checksums, frame construction, file I/O, decompression, and exact comparison.
| Synthetic input | Compress | Decompress | Max RSS at 64 MiB budget |
|---|---|---|---|
| Randomised Illumina-like 150 bp | 53.15 MiB/s | 182.40 MiB/s | 31.4 MiB |
| Randomised ONT-like 20 kbp | 55.66 MiB/s | 215.22 MiB/s | 25.5 MiB |
These results establish the local x86_64 throughput floor, not biological compression-ratio claims. The repository does not currently contain enough real ONT/HiFi/CLR data to admit a specialised long-read codec, and ARM64 remains an explicit release-machine verification item. The table reports three-run medians after full logical-record validation and the maximum RSS across repetitions.
Run the local scaling harness with:
FQC_BIN=build/clang-release/src/fqc \
FQC_PERF_SIZES="64 256" \
FQC_PERF_DATA=random \
FQC_PERF_ENFORCE_SLA=1 \
bash tests/e2e/test_performance.shThe dataset-driven peer comparison remains available through ./scripts/benchmark_v2.sh.
Codec pass/fail decisions are recorded in benchmark_v2/CODEC_GATES.md.
The project is in closeout mode. Keep changes small, delete duplicate paths, and run:
./scripts/test.sh clang-debug
./scripts/lint.sh format-checkFocused tests cover the v2 wire format, corruption/truncation, memory admission, parser/I/O, single/paired round trips, stdin/stdout, and the real CLI process boundary.
MIT. See LICENSE.