Performance

sql-splitter is designed for high throughput with minimal memory usage.

Benchmarks

Tested on Apple M2 Max:

Metric	Value
End-to-end split	~1 GB/s (0.8–1.4 GB/s across dialects)
Split without a disk bottleneck	1.6–2.4 GB/s (Docker tmpfs)
Memory usage	~20–150 MB, independent of file size
Cold start	~5 ms

End-to-end split includes reading, parsing, and writing per-table output files (parsing and writing are pipelined across parallel writer threads). Real-world throughput is content-dependent — wide rows, large BLOBs, and heavy escaping run slower — and on gigabyte-scale dumps the disk’s write bandwidth usually becomes the limit before the parser does.

Split Benchmark

File Size	Time	Throughput
124 MB (MySQL)	0.12s	~1.0 GB/s
248 MB (MySQL)	0.23s	~1.1 GB/s
1 GB (MySQL, 100 tables)	1.2–2.5s	0.4–0.9 GB/s (disk-write bound)

Slow Output Devices (--io-strategy)

The write path is SSD-tuned by default; on spinning disks and cheap flash the same settings cause seek thrash. split --io-strategy auto (the default) detects slow output devices — an 8MB fsync probe picks the starting strategy, then runtime backpressure monitoring adapts — and switches to a seek-friendly write profile.

What we observed on one specific device (a 1TB Seagate USB 3.0 spinning drive, ExFAT, 124MB–3.8GB fixtures, reading and writing on the same spindle):

Profile	Observed throughput
ssd (SSD defaults)	22–43 MB/s
hdd (pinned)	55–77 MB/s
auto	matched or beat pinned hdd

These numbers are indicative for similar hardware, not a general claim — run-to-run variance of ±25% was observed on this drive alone, and different devices, filesystems, and workloads will land elsewhere. Verify on your own target with scripts/verify-io-strategys.sh.

What the profiles actually do. Each profile is three concrete settings — writer thread count, the size of each write operation, and how much data is staged in memory between writes. Output is byte-identical in every profile; only the scheduling of bytes onto the device changes.

Profile	Writer threads	Write size	Staging memory	Device class it’s built for
ssd	up to 4 (all cores when compressing)	256 KB	32 MB	SSD/NVMe: high IOPS and internal parallelism — many small concurrent writes keep the device’s queues full
hdd	1	8 MB chunks, coalesced up to 64 MB	256 MB	Spinning disks: parallel writers make the head seek between files, so one writer issues long sequential writes that amortize head movement
cheap	1	32 MB chunks, coalesced up to 64 MB	512 MB	Cheap USB flash and network filesystems: per-operation cost dominates regardless of access pattern, so issue the fewest, largest operations possible (alias: potato — we know what it is like out there)
auto (default)	starts at 1, grows to ssd’s count if the device proves fast	adapts	adapts	Unknown targets: an 8 MB fsync probe picks the opening profile, then runtime backpressure monitoring switches between the rows above

Pin hdd for spinning disks or cheap for slow-per-operation devices when you already know the target; ssd pins today’s defaults. Expert env overrides SQL_SPLITTER_WRITERS / SQL_SPLITTER_WRITE_BUF beat every profile.

vs. Shell Alternatives

Measured in the Docker benchmark suite (1 GB file):

Tool	Time (1 GB file)
sql-splitter	~0.42s
mysqldumpsplitter (Bash)	~9.4s
Python script	~13.8s

sql-splitter is several times faster than shell- and script-based alternatives. See Benchmarking for the full hyperfine comparison against other splitter tools.

Optimization Tips

Use Native CPU Optimizations

Build with CPU-specific optimizations:

RUSTFLAGS="-C target-cpu=native" cargo build --release

Compressed Input

Compressed files can be faster than uncompressed when I/O is the bottleneck:

# Often faster than reading uncompressed
sql-splitter split backup.sql.gz -o tables/

Streaming to Database

Avoid intermediate files when possible:

# Direct stream (omit -o to write to stdout)
sql-splitter convert mysql.sql.gz --to postgres | psql "$PG_CONN"

# vs. intermediate file (slower)
sql-splitter convert mysql.sql.gz --to postgres -o temp.sql
psql "$PG_CONN" < temp.sql
rm temp.sql

Parallel Operations

For many files, use parallel execution:

find dumps -name '*.sql.gz' -print0 | \
  xargs -0 -n1 -P4 sql-splitter validate --strict

Query Caching

Cache imported databases for repeated queries:

# First query (slow - imports data)
sql-splitter query dump.sql "SELECT COUNT(*) FROM users" --cache

# Second query (fast - uses cache)
sql-splitter query dump.sql "SELECT * FROM users WHERE active = 1" --cache

Disk Mode for Large Files

For very large dumps:

sql-splitter query huge.sql "SELECT ..." --disk

Memory Usage

sql-splitter’s memory use is bounded by table count and the largest single statement — not by file size (typically ~20–150 MB):

• Streaming: Reads in chunks, never loads the entire file
• Bounded staging: Per-table write buffers are capped at 32 MB total across all tables
• No full load: A 10 GB dump peaks at the same memory as a 100 MB dump with the same schema