Unix Piping

sql-splitter follows Unix philosophy and works well in pipelines.

Standard I/O

Omit -o to write to stdout, enabling pipes:

sql-splitter convert mysql.sql --to postgres | psql "$PG_CONN"

Input must be a file path — sql-splitter does not read from stdin. Compressed files (.gz, .bz2, .xz, .zst) are decompressed natively, so there’s no need to zcat into a pipe:

sql-splitter analyze dump.sql.gz

Note: Most commands default to stdout when -o is not specified. -o - is also accepted as an explicit alias for stdout (merge, sample, shard, convert, redact, order).

Pipeline Examples

Convert and Import

# MySQL to PostgreSQL, direct import
sql-splitter convert mysql.sql.gz --to postgres | psql "$PG_CONN"

# PostgreSQL to MySQL, direct import
sql-splitter convert pg_dump.sql --to mysql | mysql -u user -p db

Compress on the Fly

# Merge and compress (omit -o to write to stdout)
sql-splitter merge tables/ | gzip > merged.sql.gz

# Merge and compress with zstd
sql-splitter merge tables/ | zstd > merged.sql.zst

Filter and Process

# Sample, then redact (chained via intermediate file)
sql-splitter sample prod.sql --percent 10 -o sampled.sql
sql-splitter redact sampled.sql --hash "*.email" -o dev.sql
rm sampled.sql

Parallel Validation

# Validate many files in parallel
find dumps -name '*.sql.gz' -print0 | \
  xargs -0 -n1 -P4 sql-splitter validate --strict

Decompress on the Fly

# Compressed input is decompressed natively; recompress the output
sql-splitter convert backup.sql.gz --to postgres | \
  gzip > backup_pg.sql.gz

Combining Commands

Full Dev Dataset Pipeline

sql-splitter sample prod.sql.gz --percent 10 --preserve-relations -o sampled.sql && \
  sql-splitter redact sampled.sql --hash "*.email" --fake "*.name" -o dev.sql && \
  sql-splitter validate dev.sql --strict && \
  echo "Valid dev dataset created"

Migration with Validation

# Validate, convert, validate, import
sql-splitter validate source.sql --strict && \
  sql-splitter convert source.sql --to postgres -o converted.sql && \
  sql-splitter validate converted.sql --dialect postgres --strict && \
  psql "$PG_CONN" < converted.sql

Exit Code Handling

# Stop on first failure
set -e
sql-splitter validate dump.sql --strict
sql-splitter convert dump.sql --to postgres -o output.sql

# Or explicit checking
if sql-splitter validate dump.sql --strict; then
  sql-splitter convert dump.sql --to postgres -o output.sql
else
  echo "Validation failed"
  exit 1
fi

JSON Pipeline

# Extract specific info with jq
sql-splitter analyze dump.sql --json | jq '.tables[].name'

# Filter validation results
sql-splitter validate "*.sql" --json | \
  jq '.results[] | select(.passed == false)'

Tee for Logging

# Process and log
sql-splitter convert dump.sql --to postgres | \
  tee conversion.log | \
  psql "$PG_CONN"