This is the full developer documentation for sql-splitter # sql-splitter Documentation > Split, merge, convert, validate, sample, shard, diff, redact, and query SQL dump files. MySQL, PostgreSQL, SQLite, and MSSQL support. ## Why sql-splitter? [Section titled “Why sql-splitter?”](#why-sql-splitter) sql-splitter is a **dump-first, CLI-first** tool designed for automation and CI/CD pipelines. It operates on SQL dump files on disk (optionally compressed) rather than live database connections. Blazing Fast \~1 GB/s end-to-end split throughput on large dumps. Several times faster than shell- and script-based alternatives. Memory Efficient \~20–150 MB memory, independent of file size. Handles files larger than RAM with streaming architecture. Multi-Dialect MySQL, PostgreSQL, SQLite, and MSSQL support including `COPY FROM stdin` and `GO` batches. Compressed Files Automatic detection of gzip, bzip2, xz, and zstd compression. ## Commands Overview [Section titled “Commands Overview”](#commands-overview) | Command | Purpose | | -------------------------------- | ----------------------------------------------- | | [`split`](/commands/split) | Split dump into per-table SQL files | | [`merge`](/commands/merge) | Merge per-table files into single dump | | [`analyze`](/commands/analyze) | Read-only statistics about tables | | [`convert`](/commands/convert) | Transform between MySQL/PostgreSQL/SQLite/MSSQL | | [`validate`](/commands/validate) | Check syntax, encoding, PK/FK integrity | | [`sample`](/commands/sample) | Create reduced FK-consistent datasets | | [`shard`](/commands/shard) | Extract tenant-specific data | | [`diff`](/commands/diff) | Compare two SQL dumps for changes | | [`redact`](/commands/redact) | Anonymize PII in SQL dumps | | [`graph`](/commands/graph) | Generate ERD diagrams (HTML, DOT, Mermaid) | | [`order`](/commands/order) | Reorder dump in topological FK order | | [`query`](/commands/query) | Run SQL analytics on dumps with DuckDB | | [`generate`](/commands/generate) | Generate synthetic data modeled on a dump | # Introduction > Get started with sql-splitter, the high-performance SQL dump toolkit sql-splitter is a command-line tool for working with SQL dump files. It’s designed for: * **Automation**: JSON output, consistent exit codes, glob patterns * **CI/CD pipelines**: Validation, integrity checks, reproducible operations * **Large files**: Streaming architecture handles multi-GB dumps with constant memory * **Database migrations**: Convert between MySQL, PostgreSQL, SQLite, and MSSQL ## What Can You Do? [Section titled “What Can You Do?”](#what-can-you-do) ### Split & Merge [Section titled “Split & Merge”](#split--merge) Break large dumps into per-table files for easier editing, then reassemble them: ```bash sql-splitter split dump.sql -o tables/ # Edit individual table files... sql-splitter merge tables/ -o updated.sql ``` ### Convert Between Dialects [Section titled “Convert Between Dialects”](#convert-between-dialects) Migrate data between different databases: ```bash sql-splitter convert mysql.sql --to postgres -o postgres.sql ``` ### Create Dev Datasets [Section titled “Create Dev Datasets”](#create-dev-datasets) Sample production data while preserving foreign key relationships: ```bash sql-splitter sample prod.sql -o dev.sql --percent 10 --preserve-relations ``` ### Validate Dumps [Section titled “Validate Dumps”](#validate-dumps) Check integrity before restoring to production: ```bash sql-splitter validate dump.sql --strict ``` ### Anonymize Data [Section titled “Anonymize Data”](#anonymize-data) Redact PII for safe sharing: ```bash sql-splitter redact dump.sql -o safe.sql --hash "*.email" --fake "*.name" ``` ### Query Without Restoring [Section titled “Query Without Restoring”](#query-without-restoring) Run SQL queries directly on dump files using DuckDB: ```bash sql-splitter query dump.sql "SELECT COUNT(*) FROM users" ``` ## Next Steps [Section titled “Next Steps”](#next-steps) * [Installation](/getting-started/installation) - Install sql-splitter * [Quick Start](/getting-started/quick-start) - Your first split → edit → merge workflow * [Concepts](/getting-started/concepts) - Understand dialects, streaming, and compression # Core Concepts > Understanding dialects, streaming, and compression in sql-splitter This page covers the core concepts you need to understand when using sql-splitter. For detailed definitions of database terminology used throughout the documentation, see the [Glossary](/reference/glossary). ## SQL Dialects [Section titled “SQL Dialects”](#sql-dialects) sql-splitter supports four SQL dialects: | Dialect | Tool | Features | | -------------- | --------------- | ------------------------------------------------ | | **MySQL** | `mysqldump` | Default dialect, backtick quoting, `INSERT INTO` | | **PostgreSQL** | `pg_dump` | `COPY FROM stdin`, `$$` strings, `SET` commands | | **SQLite** | `.dump` | `INTEGER PRIMARY KEY`, `.schema` commands | | **MSSQL** | `sqlcmd` / SSMS | `GO` batch separators, `IDENTITY`, `nvarchar` | ### Auto-Detection [Section titled “Auto-Detection”](#auto-detection) sql-splitter automatically detects the dialect by analyzing the start of the file: * Tool banners: `MySQL dump`, `PostgreSQL database dump` / `pg_dump`, `SQLite` * Dialect markers: `LOCK TABLES` and `/*!40...*/` comments (MySQL), `COPY ... FROM stdin` and `search_path` (PostgreSQL), `PRAGMA` (SQLite), `SET ANSI_NULLS` and `GO` batches (MSSQL) * Quoting: Backticks (MySQL), `[brackets]` (MSSQL) Override with `--dialect`: ```bash sql-splitter split dump.sql --dialect=postgres ``` ## Streaming Architecture [Section titled “Streaming Architecture”](#streaming-architecture) sql-splitter uses a streaming architecture that: 1. **Reads in chunks**: Fixed-size buffer reads, not full file loads 2. **Processes line-by-line**: Statements are parsed as they’re read 3. **Writes immediately**: Output files are written in real-time This means: * **Bounded memory**: \~20–150 MB regardless of file size * **No file size limit**: Process 10GB+ dumps * **Fast startup**: No waiting to load the entire file ## Compression [Section titled “Compression”](#compression) sql-splitter automatically detects and decompresses: | Extension | Format | Notes | | --------- | ------- | --------------------- | | `.gz` | gzip | Most common | | `.bz2` | bzip2 | High compression | | `.xz` | xz/LZMA | Very high compression | | `.zst` | zstd | Fast compression | Pass compressed files directly: ```bash sql-splitter split backup.sql.gz -o tables/ sql-splitter analyze dump.sql.zst sql-splitter convert mysql.sql.bz2 --to postgres ``` ## Glob Patterns [Section titled “Glob Patterns”](#glob-patterns) Most commands accept glob patterns for batch operations: ```bash # All SQL files in current directory sql-splitter validate "*.sql" # Recursive search sql-splitter analyze "backups/**/*.sql" # Specific patterns sql-splitter split "dumps/*.sql.gz" -o output/ ``` Use `--fail-fast` to stop on first error: ```bash sql-splitter validate "*.sql" --fail-fast ``` ## JSON Output [Section titled “JSON Output”](#json-output) Most commands support `--json` for machine-readable output (`diff` uses `--format json`, and `order` has no JSON mode): ```bash sql-splitter analyze dump.sql --json | jq '.tables[].name' sql-splitter validate dump.sql --json | jq '.issues' ``` JSON output includes: * Input/output file paths * Processing statistics * Per-table breakdowns * Warnings and errors ## Exit Codes [Section titled “Exit Codes”](#exit-codes) | Code | Meaning | | ---- | --------------------------------------- | | 0 | Success | | 1 | Error (syntax, I/O, validation failure) | | 2 | Invalid arguments | Use in CI scripts: ```bash sql-splitter validate dump.sql --strict || exit 1 ``` ## Statement Types [Section titled “Statement Types”](#statement-types) sql-splitter recognizes these SQL statement types: **DDL (Schema)**: * `CREATE TABLE` * `CREATE INDEX` * `ALTER TABLE` * `DROP TABLE` **DML (Data)**: * `INSERT INTO` * `COPY ... FROM stdin` (PostgreSQL) **Control**: * Session settings (`SET`, `USE`) * Comments * Transaction markers # Installation > Install sql-splitter with the shell installer, Homebrew, Cargo, or pre-built binaries * Shell macOS and Linux: ```bash curl -fsSL https://sql-splitter.dev/install.sh | sh ``` Downloads the pre-built binary for your platform from the latest [GitHub Release](https://github.com/helgesverre/sql-splitter/releases) and installs it to `$CARGO_HOME/bin` (or `$HOME/.cargo/bin`). Override the location with `SQL_SPLITTER_INSTALL_DIR`. * Homebrew ```bash brew install helgesverre/tap/sql-splitter ``` * Cargo If you have Rust installed: ```bash cargo install sql-splitter ``` Don’t have Rust? Install it from [rustup.rs](https://rustup.rs/). * Windows PowerShell: ```powershell powershell -ExecutionPolicy Bypass -c "irm https://sql-splitter.dev/install.ps1 | iex" ``` * Binaries Download from [GitHub Releases](https://github.com/helgesverre/sql-splitter/releases): | Platform | Download | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Linux x86\_64 | [sql-splitter-x86\_64-unknown-linux-gnu.tar.xz](https://github.com/helgesverre/sql-splitter/releases/latest/download/sql-splitter-x86_64-unknown-linux-gnu.tar.xz) | | Linux ARM64 | [sql-splitter-aarch64-unknown-linux-gnu.tar.xz](https://github.com/helgesverre/sql-splitter/releases/latest/download/sql-splitter-aarch64-unknown-linux-gnu.tar.xz) | | macOS Intel | [sql-splitter-x86\_64-apple-darwin.tar.xz](https://github.com/helgesverre/sql-splitter/releases/latest/download/sql-splitter-x86_64-apple-darwin.tar.xz) | | macOS Apple Silicon | [sql-splitter-aarch64-apple-darwin.tar.xz](https://github.com/helgesverre/sql-splitter/releases/latest/download/sql-splitter-aarch64-apple-darwin.tar.xz) | | Windows x86\_64 | [sql-splitter-x86\_64-pc-windows-msvc.zip](https://github.com/helgesverre/sql-splitter/releases/latest/download/sql-splitter-x86_64-pc-windows-msvc.zip) | Each archive ships with a matching `.sha256` checksum file. ## Build from Source [Section titled “Build from Source”](#build-from-source) ```bash git clone https://github.com/helgesverre/sql-splitter cd sql-splitter just install # Installs binary + shell completions + man pages ``` Or straight from the repository without cloning: ```bash cargo install --git https://github.com/helgesverre/sql-splitter ``` ## Optimized Build [Section titled “Optimized Build”](#optimized-build) For maximum performance, build with CPU-specific optimizations: ```bash RUSTFLAGS="-C target-cpu=native" cargo build --release ``` ## Verify Installation [Section titled “Verify Installation”](#verify-installation) ```bash sql-splitter --version ``` This should output `sql-splitter` followed by the version number. ## Shell Completions [Section titled “Shell Completions”](#shell-completions) Shell completions are automatically installed with `just install`. For manual installation: ```bash # Bash (user) sql-splitter completions bash >> ~/.bashrc # Zsh (oh-my-zsh) sql-splitter completions zsh > ~/.oh-my-zsh/completions/_sql-splitter # Fish sql-splitter completions fish > ~/.config/fish/completions/sql-splitter.fish ``` See the [completions command](/commands/completions/) for more options. ## Man Pages [Section titled “Man Pages”](#man-pages) View documentation with `man sql-splitter` or `man sql-splitter-diff`. For `cargo install` users, install man pages manually: ```bash git clone https://github.com/helgesverre/sql-splitter cd sql-splitter just install-man ``` ## Use as a Rust Library [Section titled “Use as a Rust Library”](#use-as-a-rust-library) sql-splitter is also published as a crate. See [Library Usage](/reference/library-usage/) for the API. # Quick Start > Your first sql-splitter workflow - split, edit, merge This guide walks you through the most common sql-splitter workflow: splitting a dump into individual table files, editing them, and merging back. Tip sql-splitter uses streaming I/O, handling files of any size with bounded (\~20–150 MB) memory. 1. **Split a Dump** Split a MySQL dump into per-table files: ```bash sql-splitter split database.sql --output=tables/ ``` Output: ```plaintext tables/ ├── users.sql ├── orders.sql ├── products.sql └── order_items.sql ``` **Dialect Auto-Detection** sql-splitter automatically detects MySQL, PostgreSQL, SQLite, and MSSQL dumps. Override with `--dialect`: ```bash # PostgreSQL pg_dump sql-splitter split pg_dump.sql -o tables/ --dialect=postgres # SQLite dump sql-splitter split sqlite.sql -o tables/ --dialect=sqlite # MSSQL/T-SQL sql-splitter split mssql.sql -o tables/ --dialect=mssql ``` **Split Specific Tables** Only split the tables you need: ```bash sql-splitter split dump.sql -o tables/ --tables users,orders ``` **Schema vs Data** Split only schema (DDL) or only data (DML): ```bash # Schema only (CREATE TABLE, indexes) sql-splitter split dump.sql -o schema/ --schema-only # Data only (INSERT statements) sql-splitter split dump.sql -o data/ --data-only ``` 2. **Edit Individual Tables** Now you can: * Edit specific table files * Delete tables you don’t need * Modify INSERT statements * Add or remove indexes 3. **Merge Back** Combine the per-table files back into a single dump: ```bash sql-splitter merge tables/ -o restored.sql ``` **Wrap in Transaction** For atomic restores: ```bash sql-splitter merge tables/ -o restored.sql --transaction ``` **Merge Specific Tables** Only merge certain tables: ```bash sql-splitter merge tables/ -o partial.sql --tables users,orders ``` **Exclude Tables** Skip certain tables when merging: ```bash sql-splitter merge tables/ -o restored.sql --exclude logs,cache ``` 4. **Verify the Result** Analyze the merged file to verify: ```bash sql-splitter analyze restored.sql ``` Output: ```plaintext Found 4 tables: Table Name INSERTs Total Stmts Size (MB) ──────────────────────────────────────────────────────────────────────────────── users 150 152 0.01 orders 500 502 0.04 products 100 102 0.01 order_items 1200 1202 0.09 ──────────────────────────────────────────────────────────────────────────────── TOTAL 1950 - 0.15 ``` ## Next Steps [Section titled “Next Steps”](#next-steps) * [Concepts](/getting-started/concepts) - Learn about dialects and compression * [Commands](/commands/) - Full command reference * [Cookbook](/cookbook/docker-usage/) - Common workflow patterns # Overview > Advanced usage and internals Deep dives into advanced features. | Topic | Description | | ------------------------------------------------ | ------------------------------------ | | [Performance](/advanced/performance) | Benchmarks and optimization tips | | [Memory & Streaming](/advanced/memory-streaming) | How streaming architecture works | | [Glob Patterns](/advanced/glob-patterns) | Multi-file processing with wildcards | | [Piping](/advanced/piping) | Unix composition patterns | | [AI Integration](/advanced/ai-integration) | llms.txt and agent skills | # AI Integration > Using sql-splitter with AI agents and tools sql-splitter is optimized for AI-driven workflows. ## llms.txt [Section titled “llms.txt”](#llmstxt) sql-splitter includes an [llms.txt](https://sql-splitter.dev/llms.txt) file for AI agents: ```plaintext https://sql-splitter.dev/llms.txt ``` This provides: * Complete command reference * Usage patterns * Error handling guidance AI agents can fetch this file to understand sql-splitter’s capabilities. ## Agent Detection [Section titled “Agent Detection”](#agent-detection) `sql-splitter --help` detects when it is being run by a coding agent and appends a pointer to the files above, so an agent finds them without being told: ```plaintext Notes for AI agents: Documentation in LLM-readable form (read this before composing commands): https://sql-splitter.dev/llms.txt index of the doc set ... ``` Detection reads the environment variables agents set in the commands they run — `CLAUDECODE`, `CURSOR_AGENT`, `GEMINI_CLI`, `CODEX_SANDBOX`, `OPENCODE`, `AMP_CURRENT_THREAD_ID`, `CLINE_ACTIVE`, and the generic `AI_AGENT` / `AGENT`, among others. Nothing else in the output changes, and human invocations are unaffected. Override it with `SQL_SPLITTER_AGENT`: ```bash SQL_SPLITTER_AGENT=0 sql-splitter --help # never show the note SQL_SPLITTER_AGENT=1 sql-splitter --help # always show it ``` ## Agent Skills [Section titled “Agent Skills”](#agent-skills) sql-splitter provides an [Agent Skill](https://agentskills.io) for coding assistants. ### Install in Claude Code [Section titled “Install in Claude Code”](#install-in-claude-code) ```bash git clone https://github.com/helgesverre/sql-splitter.git /tmp/sql-splitter cp -r /tmp/sql-splitter/skills/sql-splitter ~/.claude/skills/ ``` ### Install in Amp [Section titled “Install in Amp”](#install-in-amp) ```bash amp skill add helgesverre/sql-splitter ``` ### Install in VS Code / GitHub Copilot [Section titled “Install in VS Code / GitHub Copilot”](#install-in-vs-code--github-copilot) ```bash git clone https://github.com/helgesverre/sql-splitter.git /tmp/sql-splitter cp -r /tmp/sql-splitter/skills/sql-splitter .github/skills/ ``` ### Install in Cursor [Section titled “Install in Cursor”](#install-in-cursor) ```bash git clone https://github.com/helgesverre/sql-splitter.git /tmp/sql-splitter cp -r /tmp/sql-splitter/skills/sql-splitter .cursor/skills/ ``` ## JSON Output for Agents [Section titled “JSON Output for Agents”](#json-output-for-agents) Most commands support `--json` for machine-readable output (`diff` and `query` use `--format json` instead): ```bash sql-splitter analyze dump.sql --json ``` Agents can parse this structured output programmatically. ## Automation-Friendly Design [Section titled “Automation-Friendly Design”](#automation-friendly-design) sql-splitter is designed for automation: * **Consistent exit codes**: 0 success, 1 error, 2 bad args * **JSON output**: Machine-readable output for scripting * **Glob patterns**: Process multiple files at once * **`--dry-run`**: Safe exploration * **`--fail-fast`**: Stop on first error * **Streaming I/O**: Composable with Unix tools ## Example Agent Prompts [Section titled “Example Agent Prompts”](#example-agent-prompts) ### ”Validate my SQL dumps" [Section titled “”Validate my SQL dumps"”](#validate-my-sql-dumps) ```plaintext Agent: I'll check your SQL dumps for issues. sql-splitter validate "dumps/*.sql" --strict --json Result: Found 3 issues in backup_2024.sql: - Line 42: Syntax error near 'CREAT' - Duplicate PK in users table - FK violation: orders.user_id references non-existent user ``` ### "Convert MySQL to PostgreSQL" [Section titled “"Convert MySQL to PostgreSQL"”](#convert-mysql-to-postgresql) ```plaintext Agent: I'll convert your MySQL dump to PostgreSQL. sql-splitter validate mysql_dump.sql --strict sql-splitter convert mysql_dump.sql --to postgres -o postgres_dump.sql --progress sql-splitter validate postgres_dump.sql --dialect postgres --strict Result: Conversion complete. 2 warnings about ENUM types that were converted to VARCHAR. ``` ### "Create a safe dev dataset” [Section titled “"Create a safe dev dataset””](#create-a-safe-dev-dataset) ```plaintext Agent: I'll create an anonymized sample for development. sql-splitter sample prod.sql --percent 10 --preserve-relations -o sampled.sql sql-splitter redact sampled.sql --hash "*.email" --fake "*.name" --null "*.ssn" -o dev.sql sql-splitter validate dev.sql --strict Result: Created dev.sql with 10% of data, all PII anonymized. ``` ## Best Practices for Agents [Section titled “Best Practices for Agents”](#best-practices-for-agents) 1. **Always validate** before and after operations 2. **Use `--json`** for parsing results 3. **Use `--dry-run`** for exploration 4. **Use `--progress`** for user feedback 5. **Handle warnings** - check JSON for warning arrays # Glob Patterns & Multi-File Mode > Processing multiple files with glob patterns Several commands support glob patterns for batch processing multiple files. ## Supported Commands [Section titled “Supported Commands”](#supported-commands) | Command | Glob Support | Output Behavior | | ---------- | ------------ | --------------------------------------- | | `split` | ✅ | Creates subdirectory per input file | | `analyze` | ✅ | Aggregates results across all files | | `convert` | ✅ | Requires output directory | | `validate` | ✅ | Validates each file, aggregates results | ## Basic Patterns [Section titled “Basic Patterns”](#basic-patterns) ```bash # All SQL files in current directory sql-splitter analyze "*.sql" # All SQL files in a directory sql-splitter validate "dumps/*.sql" # Recursive (all subdirectories) sql-splitter analyze "backups/**/*.sql" # Compressed files sql-splitter validate "dumps/*.sql.gz" ``` **Important:** Quote glob patterns to prevent shell expansion. ## Pattern Syntax [Section titled “Pattern Syntax”](#pattern-syntax) | Pattern | Matches | | ------- | ---------------------------------------- | | `*` | Any characters except `/` | | `**` | Any characters including `/` (recursive) | | `?` | Any single character | | `[abc]` | Any character in brackets | | `[a-z]` | Any character in range | ## Examples [Section titled “Examples”](#examples) ```bash # All .sql files in dumps/ sql-splitter analyze "dumps/*.sql" # All .sql and .sql.gz files recursively sql-splitter validate "backups/**/*.sql*" # Files starting with "prod_" sql-splitter analyze "prod_*.sql" # Files from 2024 sql-splitter validate "backup_2024*.sql" ``` ## —fail-fast Flag [Section titled “—fail-fast Flag”](#fail-fast-flag) By default, glob processing continues even if one file fails. Use `--fail-fast` to stop on first error: ```bash # Stop on first error sql-splitter validate "*.sql" --fail-fast # Continue despite errors (default) sql-splitter validate "*.sql" ``` ## Output Behavior [Section titled “Output Behavior”](#output-behavior) ### analyze [Section titled “analyze”](#analyze) Outputs aggregate statistics across all files: ```bash sql-splitter analyze "dumps/*.sql" --json ``` ```json { "total_files": 5, "succeeded": 5, "failed": 0, "results": [ { "file": "dumps/users.sql", "tables_count": 1, "total_inserts": 1000, "status": "success" }, { "file": "dumps/orders.sql", "tables_count": 1, "total_inserts": 50000, "status": "success" } ] } ``` ### validate [Section titled “validate”](#validate) Reports validation status for each file: ```bash sql-splitter validate "*.sql" --json ``` ```json { "passed": 2, "failed": 1, "results": [ { "file": "a.sql", "passed": true, "summary": { "summary": { "errors": 0, "warnings": 0 } } }, { "file": "b.sql", "passed": true, "summary": { "summary": { "errors": 0, "warnings": 1 } } }, { "file": "c.sql", "passed": false, "summary": { "summary": { "errors": 2, "warnings": 0 } } } ] } ``` ### split [Section titled “split”](#split) Creates a subdirectory for each input file: ```bash sql-splitter split "dumps/*.sql" -o output/ ``` ```plaintext output/ ├── dump1/ │ ├── users.sql │ └── orders.sql ├── dump2/ │ ├── users.sql │ └── products.sql ``` ### convert [Section titled “convert”](#convert) Requires an output directory (not a file): ```bash # Correct: output directory sql-splitter convert "*.sql" --to postgres -o converted/ # Creates: # converted/ # ├── file1.sql # ├── file2.sql ``` ## Progress with Globs [Section titled “Progress with Globs”](#progress-with-globs) The `--progress` flag shows per-file and overall progress: ```bash sql-splitter validate "backups/**/*.sql.gz" --progress ``` ```plaintext [1/5] backups/2024/jan.sql.gz ✓ [2/5] backups/2024/feb.sql.gz ✓ [3/5] backups/2024/mar.sql.gz ⚠ 2 warnings [4/5] backups/2024/apr.sql.gz ✓ [5/5] backups/2024/may.sql.gz ✗ 1 error Summary: 4 passed, 1 failed ``` ## Parallel Processing [Section titled “Parallel Processing”](#parallel-processing) For parallel processing of many files, use external tools: ```bash # Process 4 files in parallel find dumps -name '*.sql.gz' -print0 | \ xargs -0 -n1 -P4 sql-splitter validate --strict ``` ## CI/CD Usage [Section titled “CI/CD Usage”](#cicd-usage) ```yaml # GitHub Actions example - name: Validate all SQL dumps run: | sql-splitter validate "migrations/*.sql" --strict --json > validation.json if jq -e '.failed > 0' validation.json; then echo "Validation failed" exit 1 fi ``` ## See Also [Section titled “See Also”](#see-also) * [Exit Codes](/reference/exit-codes) - Understanding exit codes for scripting * [Unix Piping](/advanced/piping) - Composing commands # Memory & Streaming > How sql-splitter handles large files efficiently sql-splitter uses a streaming architecture to handle files of any size with constant memory. ## The Problem [Section titled “The Problem”](#the-problem) Traditional approaches load the entire file into memory: ```python # DON'T DO THIS with open("dump.sql") as f: content = f.read() # Loads entire file process(content) ``` For a 10 GB file, this uses 10+ GB of RAM. ## The Solution: Streaming [Section titled “The Solution: Streaming”](#the-solution-streaming) sql-splitter processes files incrementally: 1. Read a chunk (64 KB buffer; 256 KB for files over 1 GB) 2. Parse statements in the chunk 3. Write output immediately 4. Release memory 5. Repeat Memory stays bounded (\~20–150 MB) regardless of file size. ## How It Works [Section titled “How It Works”](#how-it-works) ### Input Streaming [Section titled “Input Streaming”](#input-streaming) ```plaintext File → [64 KB Buffer] → Parser → Statement ``` The parser maintains state across chunk boundaries, handling: * Multi-line strings * Escaped characters * Statement boundaries ### Output Streaming [Section titled “Output Streaming”](#output-streaming) ```plaintext Statement → [64 KB Buffer] → File ``` Output is written immediately, not buffered in memory. ### Compression Streaming [Section titled “Compression Streaming”](#compression-streaming) Compressed files are decompressed on-the-fly: ```plaintext compressed.sql.gz → [Decompressor] → [64 KB Buffer] → Parser ``` Only the decompression buffer exists in memory, not the full file. ## Memory Profile [Section titled “Memory Profile”](#memory-profile) Measured peak RSS for `split` (MySQL dumps, Apple M2 Max): | File Size | Peak Memory | | ------------------ | ----------- | | 25 MB (10 tables) | \~35 MB | | 124 MB (10 tables) | \~19 MB | | 260 MB (10 tables) | \~33 MB | | 1 GB (100 tables) | \~148 MB | Peak memory tracks the table count and the largest single statement (a huge PostgreSQL `COPY` block must fit in memory), not the file size. ## Commands That Stream [Section titled “Commands That Stream”](#commands-that-stream) All commands use streaming: | Command | Streaming I/O | | ---------- | --------------------------------- | | `split` | Read → Write multiple files | | `merge` | Read multiple → Write one | | `analyze` | Read → Statistics in memory | | `convert` | Read → Transform → Write | | `validate` | Read → Validate → Report | | `sample` | Read → Filter → Write | | `shard` | Read → Filter → Write | | `diff` | Read both → Compare → Report | | `redact` | Read → Transform → Write | | `order` | Read → Reorder (memory for graph) | | `query` | Read → Import to DuckDB | ## Exceptions [Section titled “Exceptions”](#exceptions) A few operations require more memory: ### FK Graph Building [Section titled “FK Graph Building”](#fk-graph-building) `order` and `graph` commands build an in-memory graph of table relationships. For schemas with thousands of tables, this can use more memory. ### Query Import [Section titled “Query Import”](#query-import) `query` imports data into DuckDB, which uses memory proportional to data size. Use `--disk` for large files: ```bash sql-splitter query huge.sql "SELECT ..." --disk ``` ### Diff Data Comparison [Section titled “Diff Data Comparison”](#diff-data-comparison) `diff` with data comparison tracks primary keys in memory. Use `--max-pk-entries` to limit: ```bash sql-splitter diff old.sql new.sql --max-pk-entries 1000000 ``` # Performance > Benchmarks and optimization tips sql-splitter is designed for high throughput with minimal memory usage. ## Benchmarks [Section titled “Benchmarks”](#benchmarks) Tested on Apple M2 Max: | Metric | Value | | ------------------------------- | --------------------------------------- | | End-to-end split | \~1 GB/s (0.8–1.4 GB/s across dialects) | | Split without a disk bottleneck | 1.6–2.4 GB/s (Docker tmpfs) | | Memory usage | \~20–150 MB, independent of file size | | Cold start | \~5 ms | End-to-end split includes reading, parsing, and writing per-table output files (parsing and writing are pipelined across parallel writer threads). Real-world throughput is content-dependent — wide rows, large BLOBs, and heavy escaping run slower — and on gigabyte-scale dumps the disk’s write bandwidth usually becomes the limit before the parser does. ### Split Benchmark [Section titled “Split Benchmark”](#split-benchmark) | File Size | Time | Throughput | | ------------------------ | -------- | ------------------------------- | | 124 MB (MySQL) | 0.12s | \~1.0 GB/s | | 248 MB (MySQL) | 0.23s | \~1.1 GB/s | | 1 GB (MySQL, 100 tables) | 1.2–2.5s | 0.4–0.9 GB/s (disk-write bound) | ### Slow Output Devices (`--io-strategy`) [Section titled “Slow Output Devices (--io-strategy)”](#slow-output-devices---io-strategy) The write path is SSD-tuned by default; on spinning disks and cheap flash the same settings cause seek thrash. `split --io-strategy auto` (the default) detects slow output devices — an 8MB fsync probe picks the starting strategy, then runtime backpressure monitoring adapts — and switches to a seek-friendly write profile. What we observed on one specific device (a 1TB Seagate USB 3.0 spinning drive, ExFAT, 124MB–3.8GB fixtures, reading and writing on the same spindle): | Profile | Observed throughput | | -------------------- | ---------------------------- | | `ssd` (SSD defaults) | 22–43 MB/s | | `hdd` (pinned) | 55–77 MB/s | | `auto` | matched or beat pinned `hdd` | These numbers are indicative for similar hardware, not a general claim — run-to-run variance of ±25% was observed on this drive alone, and different devices, filesystems, and workloads will land elsewhere. Verify on your own target with `scripts/verify-io-strategys.sh`. **What the profiles actually do.** Each profile is three concrete settings — writer thread count, the size of each write operation, and how much data is staged in memory between writes. Output is byte-identical in every profile; only the scheduling of bytes onto the device changes. | Profile | Writer threads | Write size | Staging memory | Device class it’s built for | | ---------------- | ------------------------------------------------------------- | ----------------------------------- | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ssd` | up to 4 (all cores when compressing) | 256 KB | 32 MB | SSD/NVMe: high IOPS and internal parallelism — many small concurrent writes keep the device’s queues full | | `hdd` | 1 | 8 MB chunks, coalesced up to 64 MB | 256 MB | Spinning disks: parallel writers make the head seek between files, so one writer issues long sequential writes that amortize head movement | | `cheap` | 1 | 32 MB chunks, coalesced up to 64 MB | 512 MB | Cheap USB flash and network filesystems: per-operation cost dominates regardless of access pattern, so issue the fewest, largest operations possible (alias: `potato` — we know what it is like out there) | | `auto` (default) | starts at 1, grows to `ssd`’s count if the device proves fast | adapts | adapts | Unknown targets: an 8 MB fsync probe picks the opening profile, then runtime backpressure monitoring switches between the rows above | Pin `hdd` for spinning disks or `cheap` for slow-per-operation devices when you already know the target; `ssd` pins today’s defaults. Expert env overrides `SQL_SPLITTER_WRITERS` / `SQL_SPLITTER_WRITE_BUF` beat every profile. ### vs. Shell Alternatives [Section titled “vs. Shell Alternatives”](#vs-shell-alternatives) Measured in the Docker benchmark suite (1 GB file): | Tool | Time (1 GB file) | | ------------------------ | ---------------- | | sql-splitter | \~0.42s | | mysqldumpsplitter (Bash) | \~9.4s | | Python script | \~13.8s | sql-splitter is **several times faster** than shell- and script-based alternatives. See [Benchmarking](/contributing/benchmarking/) for the full hyperfine comparison against other splitter tools. ## Optimization Tips [Section titled “Optimization Tips”](#optimization-tips) ### Use Native CPU Optimizations [Section titled “Use Native CPU Optimizations”](#use-native-cpu-optimizations) Build with CPU-specific optimizations: ```bash RUSTFLAGS="-C target-cpu=native" cargo build --release ``` ### Compressed Input [Section titled “Compressed Input”](#compressed-input) Compressed files can be faster than uncompressed when I/O is the bottleneck: ```bash # Often faster than reading uncompressed sql-splitter split backup.sql.gz -o tables/ ``` ### Streaming to Database [Section titled “Streaming to Database”](#streaming-to-database) Avoid intermediate files when possible: ```bash # Direct stream (omit -o to write to stdout) sql-splitter convert mysql.sql.gz --to postgres | psql "$PG_CONN" # vs. intermediate file (slower) sql-splitter convert mysql.sql.gz --to postgres -o temp.sql psql "$PG_CONN" < temp.sql rm temp.sql ``` ### Parallel Operations [Section titled “Parallel Operations”](#parallel-operations) For many files, use parallel execution: ```bash find dumps -name '*.sql.gz' -print0 | \ xargs -0 -n1 -P4 sql-splitter validate --strict ``` ### Query Caching [Section titled “Query Caching”](#query-caching) Cache imported databases for repeated queries: ```bash # First query (slow - imports data) sql-splitter query dump.sql "SELECT COUNT(*) FROM users" --cache # Second query (fast - uses cache) sql-splitter query dump.sql "SELECT * FROM users WHERE active = 1" --cache ``` ### Disk Mode for Large Files [Section titled “Disk Mode for Large Files”](#disk-mode-for-large-files) For very large dumps: ```bash sql-splitter query huge.sql "SELECT ..." --disk ``` ## Memory Usage [Section titled “Memory Usage”](#memory-usage) sql-splitter’s memory use is bounded by table count and the largest single statement — not by file size (typically \~20–150 MB): * **Streaming**: Reads in chunks, never loads the entire file * **Bounded staging**: Per-table write buffers are capped at 32 MB total across all tables * **No full load**: A 10 GB dump peaks at the same memory as a 100 MB dump with the same schema # Unix Piping > Compose sql-splitter with Unix tools sql-splitter follows Unix philosophy and works well in pipelines. ## Standard I/O [Section titled “Standard I/O”](#standard-io) Omit `-o` to write to stdout, enabling pipes: ```bash sql-splitter convert mysql.sql --to postgres | psql "$PG_CONN" ``` Input must be a file path — sql-splitter does not read from stdin. Compressed files (`.gz`, `.bz2`, `.xz`, `.zst`) are decompressed natively, so there’s no need to `zcat` into a pipe: ```bash sql-splitter analyze dump.sql.gz ``` **Note:** Most commands default to stdout when `-o` is not specified. `-o -` is also accepted as an explicit alias for stdout (`merge`, `sample`, `shard`, `convert`, `redact`, `order`). ## Pipeline Examples [Section titled “Pipeline Examples”](#pipeline-examples) ### Convert and Import [Section titled “Convert and Import”](#convert-and-import) ```bash # MySQL to PostgreSQL, direct import sql-splitter convert mysql.sql.gz --to postgres | psql "$PG_CONN" # PostgreSQL to MySQL, direct import sql-splitter convert pg_dump.sql --to mysql | mysql -u user -p db ``` ### Compress on the Fly [Section titled “Compress on the Fly”](#compress-on-the-fly) ```bash # Merge and compress (omit -o to write to stdout) sql-splitter merge tables/ | gzip > merged.sql.gz # Merge and compress with zstd sql-splitter merge tables/ | zstd > merged.sql.zst ``` ### Filter and Process [Section titled “Filter and Process”](#filter-and-process) ```bash # Sample, then redact (chained via intermediate file) sql-splitter sample prod.sql --percent 10 -o sampled.sql sql-splitter redact sampled.sql --hash "*.email" -o dev.sql rm sampled.sql ``` ### Parallel Validation [Section titled “Parallel Validation”](#parallel-validation) ```bash # Validate many files in parallel find dumps -name '*.sql.gz' -print0 | \ xargs -0 -n1 -P4 sql-splitter validate --strict ``` ### Decompress on the Fly [Section titled “Decompress on the Fly”](#decompress-on-the-fly) ```bash # Compressed input is decompressed natively; recompress the output sql-splitter convert backup.sql.gz --to postgres | \ gzip > backup_pg.sql.gz ``` ## Combining Commands [Section titled “Combining Commands”](#combining-commands) ### Full Dev Dataset Pipeline [Section titled “Full Dev Dataset Pipeline”](#full-dev-dataset-pipeline) ```bash sql-splitter sample prod.sql.gz --percent 10 --preserve-relations -o sampled.sql && \ sql-splitter redact sampled.sql --hash "*.email" --fake "*.name" -o dev.sql && \ sql-splitter validate dev.sql --strict && \ echo "Valid dev dataset created" ``` ### Migration with Validation [Section titled “Migration with Validation”](#migration-with-validation) ```bash # Validate, convert, validate, import sql-splitter validate source.sql --strict && \ sql-splitter convert source.sql --to postgres -o converted.sql && \ sql-splitter validate converted.sql --dialect postgres --strict && \ psql "$PG_CONN" < converted.sql ``` ## Exit Code Handling [Section titled “Exit Code Handling”](#exit-code-handling) ```bash # Stop on first failure set -e sql-splitter validate dump.sql --strict sql-splitter convert dump.sql --to postgres -o output.sql # Or explicit checking if sql-splitter validate dump.sql --strict; then sql-splitter convert dump.sql --to postgres -o output.sql else echo "Validation failed" exit 1 fi ``` ## JSON Pipeline [Section titled “JSON Pipeline”](#json-pipeline) ```bash # Extract specific info with jq sql-splitter analyze dump.sql --json | jq '.tables[].name' # Filter validation results sql-splitter validate "*.sql" --json | \ jq '.results[] | select(.passed == false)' ``` ## Tee for Logging [Section titled “Tee for Logging”](#tee-for-logging) ```bash # Process and log sql-splitter convert dump.sql --to postgres | \ tee conversion.log | \ psql "$PG_CONN" ``` # Troubleshooting > Common errors and solutions Solutions to common issues when using sql-splitter. ## Installation Issues [Section titled “Installation Issues”](#installation-issues) ### ”command not found: sql-splitter” [Section titled “”command not found: sql-splitter””](#command-not-found-sql-splitter) The binary isn’t in your PATH. **Cargo install location:** ```bash # Add to your shell profile (~/.bashrc, ~/.zshrc, etc.) export PATH="$HOME/.cargo/bin:$PATH" # Then reload source ~/.bashrc # or ~/.zshrc ``` **Verify installation:** /Users/you/.cargo/bin/sql-splitter ```bash which sql-splitter ``` ### Build fails with “linker cc not found” [Section titled “Build fails with “linker cc not found””](#build-fails-with-linker-cc-not-found) Missing C compiler on Linux. ```bash # Ubuntu/Debian sudo apt-get install build-essential # Fedora/RHEL sudo dnf install gcc # Then retry cargo install sql-splitter ``` ### Build fails on Apple Silicon [Section titled “Build fails on Apple Silicon”](#build-fails-on-apple-silicon) Ensure you have Xcode command line tools: ```bash xcode-select --install ``` ## Dialect Detection Issues [Section titled “Dialect Detection Issues”](#dialect-detection-issues) ### Dialect detection fails on generic SQL [Section titled “Dialect detection fails on generic SQL”](#dialect-detection-fails-on-generic-sql) sql-splitter analyzes the first 8 KB of the file to detect the dialect. If your dump starts with generic SQL, detection falls back to MySQL with low confidence. **Solution:** Explicitly specify the dialect: ```bash sql-splitter split dump.sql -o output/ --dialect mysql sql-splitter split dump.sql -o output/ --dialect postgres ``` ### Wrong dialect detected [Section titled “Wrong dialect detected”](#wrong-dialect-detected) If auto-detection picks the wrong dialect: ```bash # Force specific dialect sql-splitter analyze dump.sql --dialect postgres # Check what was detected sql-splitter analyze dump.sql --json | jq '.dialect' ``` ## Memory Issues [Section titled “Memory Issues”](#memory-issues) ### ”memory allocation failed” or OOM killed [Section titled “”memory allocation failed” or OOM killed”](#memory-allocation-failed-or-oom-killed) This shouldn’t happen with normal sql-splitter commands (they use \~20–150 MB, independent of file size). If it does: 1. **Check for memory-intensive commands:** `validate` and `diff` with FK checks can use more memory on very large files. 2. **Disable FK checks for validation:** ```bash sql-splitter validate huge.sql --no-fk-checks ``` 3. **Use disk mode for query:** ```bash sql-splitter query huge.sql "SELECT ..." --disk ``` 4. **Limit rows per table:** ```bash sql-splitter validate huge.sql --max-rows-per-table 100000 ``` ## File Issues [Section titled “File Issues”](#file-issues) ### ”No such file or directory" [Section titled “”No such file or directory"”](#no-such-file-or-directory) ```bash # Check file exists ls -la dump.sql # Use absolute path sql-splitter split /full/path/to/dump.sql -o output/ # Check permissions chmod +r dump.sql ``` ### "Permission denied” on output [Section titled “"Permission denied” on output”](#permission-denied-on-output) ```bash # Check output directory is writable mkdir -p output/ chmod +w output/ # Or write to a different location sql-splitter split dump.sql -o ~/output/ ``` ### Compressed file not recognized [Section titled “Compressed file not recognized”](#compressed-file-not-recognized) Ensure the file extension matches the compression format: | Format | Extension | | --------- | --------- | | Gzip | `.gz` | | Bzip2 | `.bz2` | | XZ | `.xz` | | Zstandard | `.zst` | ```bash # Rename if needed mv dump.sql.gzip dump.sql.gz # Then process sql-splitter analyze dump.sql.gz ``` ## Parsing Issues [Section titled “Parsing Issues”](#parsing-issues) ### ”0 tables found” [Section titled “”0 tables found””](#0-tables-found) The dump file might not contain recognizable SQL statements. **Check file contents:** ```bash head -100 dump.sql ``` **Common causes:** * Binary format (use `pg_dump -Fp` for plain text) * Non-SQL file * Empty file * Encoding issues (see below) ### Encoding errors or garbled output [Section titled “Encoding errors or garbled output”](#encoding-errors-or-garbled-output) sql-splitter expects UTF-8 encoding. **Convert encoding:** ```bash # Check current encoding file dump.sql # Convert from Latin-1 to UTF-8 iconv -f ISO-8859-1 -t UTF-8 dump.sql > dump-utf8.sql # Convert from Windows-1252 iconv -f CP1252 -t UTF-8 dump.sql > dump-utf8.sql ``` ### Statements split incorrectly [Section titled “Statements split incorrectly”](#statements-split-incorrectly) Multi-line strings or unusual quoting can cause issues. **Workaround:** Try a different dialect: ```bash # PostgreSQL uses different escaping than MySQL sql-splitter split dump.sql --dialect postgres ``` ## Validation Issues [Section titled “Validation Issues”](#validation-issues) ### ”Duplicate primary key” false positives [Section titled “”Duplicate primary key” false positives”](#duplicate-primary-key-false-positives) If you’re validating a dump with intentional duplicates (e.g., for testing): ```bash # Skip PK/FK checks sql-splitter validate dump.sql --no-fk-checks ``` ### Validation too slow [Section titled “Validation too slow”](#validation-too-slow) PK/FK validation reads all data. Speed it up: ```bash # Limit rows checked per table sql-splitter validate dump.sql --max-rows-per-table 10000 # Skip FK checks entirely sql-splitter validate dump.sql --no-fk-checks ``` ## Query Command Issues [Section titled “Query Command Issues”](#query-command-issues) ### ”DuckDB error” or query fails [Section titled “”DuckDB error” or query fails”](#duckdb-error-or-query-fails) **Common causes:** 1. **SQL syntax:** DuckDB uses standard SQL, not MySQL/PostgreSQL extensions ```sql -- MySQL LIMIT with offset SELECT * FROM users LIMIT 10, 5 -- Won't work -- Standard SQL SELECT * FROM users LIMIT 5 OFFSET 10 -- Works ``` 2. **Column name conflicts:** Use quotes for reserved words ```sql SELECT "order", "user" FROM orders ``` 3. **Large file:** Use disk mode ```bash sql-splitter query huge.sql "SELECT ..." --disk ``` ### Cache issues [Section titled “Cache issues”](#cache-issues) ```bash # Clear corrupted cache sql-splitter query --clear-cache # List cached databases sql-splitter query --list-cache ``` ## Convert Issues [Section titled “Convert Issues”](#convert-issues) ### ”Unsupported feature” warnings [Section titled “”Unsupported feature” warnings”](#unsupported-feature-warnings) Some SQL features don’t have direct equivalents across dialects: * **ENUM types** → Converted to VARCHAR (with a warning) * **AUTO\_INCREMENT** → Converted to SERIAL (PostgreSQL) or IDENTITY (MSSQL) * **Stored procedures** → Skipped (out of scope) Use `--strict` to fail on unsupported features instead of warning: ```bash sql-splitter convert dump.sql --to postgres --strict ``` ### COPY statements not converted [Section titled “COPY statements not converted”](#copy-statements-not-converted) PostgreSQL `COPY FROM stdin` is converted to INSERT statements: ```bash # This works sql-splitter convert pg_dump.sql --to mysql -o mysql.sql ``` If you see raw COPY blocks in output, ensure dialect was detected correctly: ```bash sql-splitter convert pg_dump.sql --from postgres --to mysql -o mysql.sql ``` ## Sample/Shard Issues [Section titled “Sample/Shard Issues”](#sampleshard-issues) ### ”No rows sampled” or empty output [Section titled “”No rows sampled” or empty output”](#no-rows-sampled-or-empty-output) **Check that tables exist:** ```bash sql-splitter analyze dump.sql --json | jq '.tables[].name' ``` **Ensure required flags are set:** ```bash # sample requires --percent OR --rows sql-splitter sample dump.sql --percent 10 -o sample.sql # shard requires --tenant-value OR --tenant-values sql-splitter shard dump.sql --tenant-value 123 -o tenant.sql ``` ### FK preservation issues [Section titled “FK preservation issues”](#fk-preservation-issues) If related rows are missing: ```bash # Enable FK preservation sql-splitter sample dump.sql --percent 10 --preserve-relations -o sample.sql # Use strict mode to catch issues sql-splitter sample dump.sql --percent 10 --preserve-relations --strict-fk -o sample.sql ``` ## Getting Help [Section titled “Getting Help”](#getting-help) ### Verbose output [Section titled “Verbose output”](#verbose-output) Most commands support `--progress` for visibility: ```bash sql-splitter split dump.sql -o output/ --progress ``` ### JSON for debugging [Section titled “JSON for debugging”](#json-for-debugging) Use `--json` for machine-readable output you can inspect: ```bash sql-splitter analyze dump.sql --json | jq '.' ``` ### Report bugs [Section titled “Report bugs”](#report-bugs) If you’ve found a bug: 1. Check [existing issues](https://github.com/helgesverre/sql-splitter/issues) 2. Create a [new issue](https://github.com/helgesverre/sql-splitter/issues/new) with: * sql-splitter version (`sql-splitter --version`) * OS and architecture * Minimal reproduction steps * Sample SQL (anonymized if needed) ## See Also [Section titled “See Also”](#see-also) * [Exit Codes](/reference/exit-codes) - Understanding return codes * [Compression](/reference/compression) - Supported formats * [Dialects](/reference/dialects) - Dialect-specific information # Overview > All sql-splitter commands at a glance ## Command Reference [Section titled “Command Reference”](#command-reference) | Command | Purpose | Key Use Cases | | -------------------------------------- | ----------------------------------------------- | ------------------------------------------------- | | [`split`](/commands/split) | Split dump into per-table SQL files | Work on individual tables, parallel processing | | [`merge`](/commands/merge) | Merge per-table files into single dump | Reassemble after editing, restore preparation | | [`analyze`](/commands/analyze) | Read-only statistics about tables | Planning, size estimation, table discovery | | [`convert`](/commands/convert) | Transform between MySQL/PostgreSQL/SQLite/MSSQL | Database migrations, environment portability | | [`validate`](/commands/validate) | Check syntax, encoding, PK/FK integrity | CI gates, backup verification, pre-restore checks | | [`sample`](/commands/sample) | Create reduced FK-consistent datasets | Dev/CI seeding, testing with realistic data | | [`shard`](/commands/shard) | Extract tenant-specific data | Multi-tenant isolation, tenant exports | | [`diff`](/commands/diff) | Compare two SQL dumps for changes | Schema migrations, change detection, auditing | | [`redact`](/commands/redact) | Anonymize PII in SQL dumps | GDPR compliance, dev data, safe sharing | | [`generate`](/commands/generate) | Generate synthetic SQL from a model or a dump | CI fixtures, load testing, dev seeding | | [`graph`](/commands/graph) | Generate ERD diagrams (HTML, DOT, Mermaid) | Schema visualization, documentation | | [`order`](/commands/order) | Reorder dump in topological FK order | Safe imports, cycle detection, DROP order | | [`query`](/commands/query) | Run SQL analytics on dumps with DuckDB | Ad-hoc queries, data exploration | | [`completions`](/commands/completions) | Generate shell completion scripts | CLI productivity | ## Common Flags [Section titled “Common Flags”](#common-flags) Most commands support these flags (availability varies): | Flag | Description | Available In | | ---------------- | --------------------------------------- | ------------------------------------------------------------------------------------------------ | | `-d, --dialect` | Override dialect auto-detection | Most commands (except `convert` which uses `--from/--to`) | | `-p, --progress` | Show progress bar | `split`, `analyze`, `merge`, `sample`, `shard`, `convert`, `validate`, `diff`, `redact`, `graph` | | `--dry-run` | Preview without writing output | `split`, `merge`, `sample`, `shard`, `convert`, `redact`, `order` | | `--json` | Machine-readable JSON output | `split`, `analyze`, `merge`, `sample`, `shard`, `convert`, `validate`, `redact`, `graph` | | `--fail-fast` | Stop on first error (for glob patterns) | `split`, `analyze`, `convert`, `validate` | ## Command Aliases [Section titled “Command Aliases”](#command-aliases) For faster typing, all commands have short aliases: | Command | Alias | Command | Alias | | ---------- | ----- | ---------- | ----- | | `split` | `sp` | `diff` | `df` | | `analyze` | `an` | `redact` | `rd` | | `merge` | `mg` | `generate` | `gen` | | `sample` | `sa` | `graph` | `gr` | | `shard` | `sh` | `order` | `ord` | | `convert` | `cv` | `query` | `qy` | | `validate` | `val` | | | ## Decision Guide [Section titled “Decision Guide”](#decision-guide) | Goal | Command(s) | | ---------------------------- | ----------------------------- | | Check if dump is valid | `validate --strict` | | Understand dump contents | `analyze` | | Edit specific tables | `split` → edit → `merge` | | Create dev dataset | `sample --preserve-relations` | | Export tenant data | `shard --tenant-value` | | Convert MySQL → PostgreSQL | `convert --to postgres` | | Validate in CI | `validate "*.sql" --json` | | Compare two dumps | `diff old.sql new.sql` | | Anonymize PII | `redact` | | Generate synthetic test data | `generate` | | Visualize schema | `graph` | | Safe import order | `order` | | Query without restore | `query` | # analyze > Analyze a SQL dump and display table statistics Read-only analysis of SQL dump files to show table statistics. **Alias:** `an` (e.g., `sql-splitter an dump.sql`) ## When to Use This [Section titled “When to Use This”](#when-to-use-this) * **Quick inspection before splitting** — understand what’s in a dump before running [`split`](/commands/split) * **Finding the largest tables** — identify which tables consume the most space in your dump * **Estimating backup sizes** — get byte-level breakdowns without extracting files * **Comparing dump sizes over time** — track table growth by analyzing dumps from different dates * **Scripted reporting** — use `--json` output for automated monitoring and dashboards ## How It Works [Section titled “How It Works”](#how-it-works) The `analyze` command performs a **single [streaming](/reference/glossary#streaming) pass** through the dump file, counting statements and rows without loading the entire file into memory. **What counts as an INSERT:** Each `INSERT` statement counts once. A multi-row insert like `INSERT INTO users VALUES (1, 'alice'), (2, 'bob')` counts as 1 INSERT, not 2—the report shows statement counts, not row counts. **What counts as a statement:** Any complete SQL statement associated with a table—`CREATE TABLE`, `INSERT`, `ALTER TABLE`, `CREATE INDEX`, `DROP TABLE`, and PostgreSQL `COPY` blocks. **What “Size” measures:** The total bytes of raw SQL statements attributed to each table in the dump file. This includes the statement text itself (e.g., the full `INSERT INTO...` line). **Memory usage:** \~7MB peak for typical dumps. The streaming architecture keeps memory constant regardless of file size. **Exit codes:** * `0` — Analysis completed successfully * `1` — Error reading file or parsing SQL ## Usage [Section titled “Usage”](#usage) ```bash sql-splitter analyze [OPTIONS] ``` ## Options [Section titled “Options”](#options) | Flag | Short | Description | Default | | ------------- | ----- | --------------------------------------------------- | ----------- | | `--dialect` | `-d` | SQL dialect: `mysql`, `postgres`, `sqlite`, `mssql` | auto-detect | | `--progress` | `-p` | Show progress bar | false | | `--fail-fast` | | Stop on first error (for glob patterns) | false | | `--json` | | Output results as JSON | false | ## Examples [Section titled “Examples”](#examples) ### Basic Analysis [Section titled “Basic Analysis”](#basic-analysis) ```bash sql-splitter analyze database.sql ``` ### Large Files with Progress [Section titled “Large Files with Progress”](#large-files-with-progress) For dumps over 100MB, enable the progress bar to track completion: ```bash sql-splitter analyze large-dump.sql --progress ``` ### Batch Analysis [Section titled “Batch Analysis”](#batch-analysis) Analyze multiple dumps using glob patterns: ```bash sql-splitter analyze "backups/**/*.sql" --fail-fast ``` ### JSON Output for Scripting [Section titled “JSON Output for Scripting”](#json-output-for-scripting) ```bash sql-splitter analyze dump.sql --json ``` ## Output [Section titled “Output”](#output) ```plaintext Analyzing SQL file: dump.sql (0.15 MB) [dialect: mysql] ✓ Analysis completed in 25.4ms Found 4 tables: Table Name INSERTs Total Stmts Size (MB) ──────────────────────────────────────────────────────────────────────────────── order_items 12 13 0.09 orders 5 6 0.04 products 1 2 0.01 users 2 3 0.01 ──────────────────────────────────────────────────────────────────────────────── TOTAL 20 - 0.15 ``` ## JSON Output [Section titled “JSON Output”](#json-output) ```json { "input_file": "dump.sql", "dialect": "mysql", "size_mb": 0.15, "elapsed_secs": 0.025, "summary": { "total_tables": 4, "total_inserts": 20, "total_bytes": 158720 }, "tables": [ { "name": "users", "inserts": 2, "creates": 1, "statements": 3, "bytes": 12595, "size_mb": 0.01 }, { "name": "orders", "inserts": 5, "creates": 1, "statements": 6, "bytes": 46284, "size_mb": 0.04 } ] } ``` ## Composing with Other Tools [Section titled “Composing with Other Tools”](#composing-with-other-tools) ### Find Tables Over a Certain Size [Section titled “Find Tables Over a Certain Size”](#find-tables-over-a-certain-size) Use `jq` to filter the JSON output: ```bash sql-splitter analyze dump.sql --json | jq '.tables[] | select(.bytes > 1000000)' ``` ### Get the Top 5 Largest Tables [Section titled “Get the Top 5 Largest Tables”](#get-the-top-5-largest-tables) ```bash sql-splitter analyze dump.sql --json | jq -r '.tables | sort_by(-.bytes) | .[0:5] | .[] | "\(.name): \(.bytes / 1024 | floor) KB"' ``` ### Monitoring Script [Section titled “Monitoring Script”](#monitoring-script) Track dump sizes over time: ```bash #!/bin/bash DATE=$(date +%Y-%m-%d) sql-splitter analyze /backups/daily.sql --json | jq --arg date "$DATE" '{date: $date, total_bytes: .summary.total_bytes}' >> /var/log/dump-sizes.jsonl ``` ### Compare Two Dumps [Section titled “Compare Two Dumps”](#compare-two-dumps) ```bash diff <(sql-splitter analyze old.sql --json | jq '.tables | sort_by(.name)') \ <(sql-splitter analyze new.sql --json | jq '.tables | sort_by(.name)') ``` ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) ### INSERT count seems wrong [Section titled “INSERT count seems wrong”](#insert-count-seems-wrong) The INSERTs column counts `INSERT` statements, not rows. A multi-row insert like `INSERT INTO t VALUES (1), (2), (3)` counts as 1 INSERT even though it inserts 3 rows. Use [`query`](/commands/query) with `SELECT COUNT(*)` if you need exact row counts. ### Size doesn’t match the file size [Section titled “Size doesn’t match the file size”](#size-doesnt-match-the-file-size) The “Size” column only counts bytes from statements **attributed to specific tables**. Global statements like `SET` commands, comments, and file headers are not included in any table’s size. The sum of table sizes will typically be less than the total file size. ### Statement count is always 2 per table [Section titled “Statement count is always 2 per table”](#statement-count-is-always-2-per-table) A typical `mysqldump` output includes one `CREATE TABLE` and one `INSERT` statement per table, resulting in 2 statements. If you see higher counts, your dump may include `ALTER TABLE`, `CREATE INDEX`, or multiple `INSERT` batches. ### Analysis is slow on large files [Section titled “Analysis is slow on large files”](#analysis-is-slow-on-large-files) Enable `--progress` to see if the file is being read. For files over 1GB, expect \~1-2 seconds per 100MB. If significantly slower, check disk I/O or try decompressing the file first if it’s compressed. ### ”Unknown dialect” errors [Section titled “”Unknown dialect” errors”](#unknown-dialect-errors) Use `--dialect` to explicitly specify the SQL dialect when auto-detection fails: ```bash sql-splitter analyze dump.sql --dialect postgres ``` ## See Also [Section titled “See Also”](#see-also) * [`split`](/commands/split) — Split a dump into per-table files * [`validate`](/commands/validate) — Check dump integrity * [`sample`](/commands/sample) — Extract a percentage of rows for testing * [JSON Output Schema](/reference/json-output/) — Schema for `--json` output * [Glossary](/reference/glossary) — Definitions for statement, row, and other terms # completions > Generate shell completion scripts Generate shell completion scripts for bash, zsh, fish, and PowerShell. ## When to Use This [Section titled “When to Use This”](#when-to-use-this) * **First-time setup** — After installing sql-splitter, generate completions to enable tab completion * **After upgrading** — Regenerate completions when new commands or flags are added * **Setting up a new machine** — Include completion setup in your dotfiles or provisioning scripts * **Switching shells** — Generate completions for your new shell ## How It Works [Section titled “How It Works”](#how-it-works) The `completions` command uses [clap](https://docs.rs/clap/latest/clap/)’s built-in completion generator to produce shell-specific scripts. When loaded by your shell, these scripts intercept `Tab` key presses and provide context-aware suggestions for commands, flags, and flag values. Completions are generated at runtime based on the current version of sql-splitter, so they always reflect the available commands and options. ## Usage [Section titled “Usage”](#usage) ```bash sql-splitter completions ``` ## Supported Shells [Section titled “Supported Shells”](#supported-shells) | Shell | Value | | ---------- | ------------ | | Bash | `bash` | | Zsh | `zsh` | | Fish | `fish` | | PowerShell | `powershell` | ## Installation [Section titled “Installation”](#installation) Once installed to the correct location, completions persist across shell restarts—no need to regenerate unless you upgrade sql-splitter. Note If you installed sql-splitter via Homebrew or a package manager, completions may already be included. Test with `sql-splitter ` before manually installing. ### Bash [Section titled “Bash”](#bash) Bash supports two installation methods: **System-wide** (all users, requires sudo): ```bash sudo sql-splitter completions bash > /etc/bash_completion.d/sql-splitter ``` **User-only** (recommended): ```bash # Create completions directory if needed mkdir -p ~/.bash_completion.d # Generate completions sql-splitter completions bash > ~/.bash_completion.d/sql-splitter # Add to .bashrc (one-time setup) echo 'source ~/.bash_completion.d/sql-splitter' >> ~/.bashrc # Reload current session source ~/.bashrc ``` ### Zsh [Section titled “Zsh”](#zsh) Zsh loads completions from directories in `$fpath`. Find your fpath locations: ```bash echo $fpath | tr ' ' '\n' ``` **Standard installation:** ```bash # Use the first writable fpath directory (usually ~/.zsh/completions or similar) sql-splitter completions zsh > "${fpath[1]}/_sql-splitter" # Rebuild completion cache and reload autoload -Uz compinit && compinit source ~/.zshrc ``` **Oh-My-Zsh:** ```bash # Oh-My-Zsh includes ~/.oh-my-zsh/completions in fpath mkdir -p ~/.oh-my-zsh/completions sql-splitter completions zsh > ~/.oh-my-zsh/completions/_sql-splitter # Reload source ~/.zshrc ``` **If you don’t have a user fpath directory:** ```bash # Create one and add to fpath in .zshrc mkdir -p ~/.zsh/completions echo 'fpath=(~/.zsh/completions $fpath)' >> ~/.zshrc sql-splitter completions zsh > ~/.zsh/completions/_sql-splitter source ~/.zshrc ``` ### Fish [Section titled “Fish”](#fish) Fish has a dedicated completions directory that’s automatically loaded: ```bash # Create directory if needed mkdir -p ~/.config/fish/completions # Generate completions (takes effect immediately in new shells) sql-splitter completions fish > ~/.config/fish/completions/sql-splitter.fish ``` No reload required—Fish automatically picks up new completion files. ### PowerShell [Section titled “PowerShell”](#powershell) PowerShell loads completions from your profile script. Find your profile location: ```powershell echo $PROFILE # Typically: ~/.config/powershell/Microsoft.PowerShell_profile.ps1 (Linux/macOS) # Or: ~/Documents/PowerShell/Microsoft.PowerShell_profile.ps1 (Windows) ``` **Install completions:** ```powershell # Create profile if it doesn't exist if (!(Test-Path -Path $PROFILE)) { New-Item -ItemType File -Path $PROFILE -Force } # Append completions to profile sql-splitter completions powershell >> $PROFILE # Reload profile . $PROFILE ``` ## What Completions Provide [Section titled “What Completions Provide”](#what-completions-provide) Once installed, pressing `Tab` will autocomplete: * Command names (`split`, `analyze`, `merge`, etc.) * Command aliases (`sp`, `an`, `mg`, etc.) * Flag names (`--output`, `--dialect`, etc.) * Flag values (e.g., dialect options: `mysql`, `postgres`, `sqlite`, `mssql`) ## Example [Section titled “Example”](#example) ```bash $ sql-splitter sp $ sql-splitter split $ sql-splitter split dump.sql --dia $ sql-splitter split dump.sql --dialect $ sql-splitter split dump.sql --dialect mysql postgres sqlite mssql ``` ## Verification [Section titled “Verification”](#verification) After installation, test completions: ```bash # Type and press Tab sql-splitter # Should show all commands: # analyze completions convert diff graph merge order query redact sample shard split validate ``` ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) ### Completions not working [Section titled “Completions not working”](#completions-not-working) **Shell needs reload:** After installing completions, you must reload your shell configuration or open a new terminal: ```bash # Bash source ~/.bashrc # Zsh source ~/.zshrc # Fish (usually automatic, but try) source ~/.config/fish/config.fish # PowerShell . $PROFILE ``` **Zsh fpath issues:** Ensure the completion file is in a directory listed in `$fpath` and the filename starts with `_`: ```bash # Check if the file is in fpath echo $fpath | tr ' ' '\n' | xargs -I {} ls {}_sql-splitter 2>/dev/null # Rebuild completion cache rm -f ~/.zcompdump && compinit ``` ### Completions outdated after upgrade [Section titled “Completions outdated after upgrade”](#completions-outdated-after-upgrade) When you upgrade sql-splitter, new commands or flags won’t appear until you regenerate completions: ```bash # Regenerate for your shell (example: zsh) sql-splitter completions zsh > ~/.oh-my-zsh/completions/_sql-splitter source ~/.zshrc ``` ### Permission denied [Section titled “Permission denied”](#permission-denied) For system-wide installation, use `sudo`: ```bash sudo sql-splitter completions bash > /etc/bash_completion.d/sql-splitter ``` Or use user-only installation (recommended) which doesn’t require elevated permissions. ### PowerShell profile doesn’t exist [Section titled “PowerShell profile doesn’t exist”](#powershell-profile-doesnt-exist) Create the profile file before appending: ```powershell if (!(Test-Path -Path $PROFILE)) { New-Item -ItemType File -Path $PROFILE -Force } sql-splitter completions powershell >> $PROFILE ``` ## See Also [Section titled “See Also”](#see-also) * [Installation](/getting-started/installation) - Initial setup # convert > Convert SQL dumps between MySQL, PostgreSQL, SQLite, and MSSQL Transform SQL dumps between different database dialects. **Alias:** `cv` (e.g., `sql-splitter cv mysql.sql --to postgres -o pg.sql`) ## When to Use This [Section titled “When to Use This”](#when-to-use-this) * **Database migration** - Moving from MySQL to PostgreSQL (or any other combination) * **Cross-platform development** - Convert production dumps to SQLite for local testing * **Cloud migration** - Transform on-premise SQL Server dumps for cloud PostgreSQL * **Tool compatibility** - Convert between dialects when tools only support specific formats Use [`validate`](/commands/validate) after conversion to verify the output is valid for your target database. ## How It Works [Section titled “How It Works”](#how-it-works) The converter parses the source dump, transforms DDL (CREATE TABLE, indexes, constraints) and DML (INSERT, COPY) statements to the target dialect. This includes: * **Identifier quoting** - Backticks (MySQL) ↔ double quotes (PostgreSQL/SQLite) ↔ brackets (MSSQL) * **Data types** - Maps 50+ types between dialects (e.g., `AUTO_INCREMENT` → `SERIAL`) * **Bulk import syntax** - Converts PostgreSQL `COPY ... FROM stdin` to INSERT statements for other dialects * **String escaping** - Adapts escape sequences between dialects ## Usage [Section titled “Usage”](#usage) ```bash sql-splitter convert --to [OPTIONS] ``` ## Examples [Section titled “Examples”](#examples) ### Basic Conversion [Section titled “Basic Conversion”](#basic-conversion) ```bash # MySQL to PostgreSQL sql-splitter convert mysql.sql --to postgres -o pg.sql # PostgreSQL to MySQL (COPY blocks become INSERT statements) sql-splitter convert pg_dump.sql --to mysql -o mysql.sql # Any dialect to SQLite (great for local development) sql-splitter convert dump.sql --to sqlite -o sqlite.sql ``` ### Explicit Source Dialect [Section titled “Explicit Source Dialect”](#explicit-source-dialect) Auto-detection works in most cases, but you can specify the source explicitly: ```bash # When auto-detection fails or you want to be explicit sql-splitter convert dump.sql --from mssql --to mysql -o mysql.sql ``` ### Strict Mode for CI/CD [Section titled “Strict Mode for CI/CD”](#strict-mode-for-cicd) Use `--strict` to fail on any unsupported features instead of emitting warnings: ```bash sql-splitter convert dump.sql --to postgres --strict ``` ### Streaming Directly to Database [Section titled “Streaming Directly to Database”](#streaming-directly-to-database) Pipe the converted output directly into your database client: ```bash # Convert and restore in one step (omit -o to write to stdout) sql-splitter convert mysql.sql.gz --to postgres | psql "$PG_CONN" # Convert and load into SQLite sql-splitter convert pg_dump.sql --to sqlite | sqlite3 dev.db ``` ## Options [Section titled “Options”](#options) | Flag | Short | Description | Default | | ------------- | ----- | ------------------------------------------------------ | ----------- | | `--output` | `-o` | Output SQL file or directory | stdout | | `--from` | | Source dialect: `mysql`, `postgres`, `sqlite`, `mssql` | auto-detect | | `--to` | | Target dialect (required) | - | | `--strict` | | Fail on any unsupported feature | false | | `--progress` | `-p` | Show progress bar | false | | `--dry-run` | | Preview without writing files | false | | `--fail-fast` | | Stop on first error (for glob patterns) | false | | `--json` | | Output results as JSON | false | ## Supported Conversions [Section titled “Supported Conversions”](#supported-conversions) All 12 conversion pairs are supported: * MySQL ↔ PostgreSQL (including COPY → INSERT) * MySQL ↔ SQLite * MySQL ↔ MSSQL * PostgreSQL ↔ SQLite * PostgreSQL ↔ MSSQL * SQLite ↔ MSSQL ## Type Mappings [Section titled “Type Mappings”](#type-mappings) 50+ data type mappings are handled automatically: | MySQL | PostgreSQL | SQLite | MSSQL | | -------------------- | ------------ | --------------------- | ---------------- | | `INT AUTO_INCREMENT` | `SERIAL` | `INTEGER PRIMARY KEY` | `INT IDENTITY` | | `TINYINT(1)` | `BOOLEAN` | `INTEGER` | `BIT` | | `VARCHAR(n)` | `VARCHAR(n)` | `TEXT` | `NVARCHAR(n)` | | `DATETIME` | `TIMESTAMP` | `TEXT` | `DATETIME2` | | `TEXT` | `TEXT` | `TEXT` | `NVARCHAR(MAX)` | | `BLOB` | `BYTEA` | `BLOB` | `VARBINARY(MAX)` | ## PostgreSQL COPY Conversion [Section titled “PostgreSQL COPY Conversion”](#postgresql-copy-conversion) PostgreSQL’s `COPY ... FROM stdin` format is converted to INSERT statements: ```sql -- Input (PostgreSQL) COPY users (id, name) FROM stdin; 1 Alice 2 Bob \. -- Output (MySQL) INSERT INTO users (id, name) VALUES (1, 'Alice'); INSERT INTO users (id, name) VALUES (2, 'Bob'); ``` ## Warnings and Unsupported Features [Section titled “Warnings and Unsupported Features”](#warnings-and-unsupported-features) The conversion **continues successfully** even when encountering unsupported features - they generate warnings but don’t stop the process. ### Features that are converted with warnings [Section titled “Features that are converted with warnings”](#features-that-are-converted-with-warnings) These types are converted to the closest equivalent, with a warning: | Feature | Converted To | Notes | | --------------------------- | -------------------------------------------------------- | ---------------------------------------------------------------- | | MySQL `ENUM('a','b','c')` | `VARCHAR(255)` (PostgreSQL/MSSQL) or `TEXT` (SQLite) | Enum values are lost; consider adding CHECK constraints manually | | PostgreSQL arrays (`INT[]`) | `JSON` (MySQL), `TEXT` (SQLite), `NVARCHAR(MAX)` (MSSQL) | Data format preserved as JSON string | ### Features that are skipped entirely [Section titled “Features that are skipped entirely”](#features-that-are-skipped-entirely) These statements are **omitted from the output** (with a warning): * `CREATE TRIGGER` - Trigger syntax is too dialect-specific to convert * `CREATE PROCEDURE` / `CREATE FUNCTION` - Stored procedure logic cannot be automatically translated * `CREATE TYPE` - Custom types have no equivalent in most dialects * `CREATE DOMAIN` - PostgreSQL-specific * `CREATE EXTENSION` - PostgreSQL-specific * `CREATE SEQUENCE` - Skipped because `SERIAL`/`BIGSERIAL` columns are converted to `AUTO_INCREMENT`/`IDENTITY`, which handle sequencing automatically * `COMMENT ON` - Not supported in all dialects The command outputs a count of skipped statements at the end. Use `--strict` to fail instead of skipping. See [Type Mappings Reference](/reference/type-mappings) for the complete list of 50+ type conversions. ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) ### Conversion fails on specific statements [Section titled “Conversion fails on specific statements”](#conversion-fails-on-specific-statements) If conversion fails completely (not just warnings), check: 1. **Syntax errors in source dump** - Validate first: `sql-splitter validate dump.sql` 2. **Wrong source dialect detected** - Specify explicitly: `--from mysql` 3. **Malformed statements** - The error message includes the problematic statement ### Output differs between runs [Section titled “Output differs between runs”](#output-differs-between-runs) The converter preserves statement order from the input. If your dump has non-deterministic ordering (e.g., `mysqldump` without `--order-by-primary`, or tables listed in different order each time): ```bash # Sort the dump first for stable, reproducible output sql-splitter order dump.sql -o ordered.sql sql-splitter convert ordered.sql --to postgres -o pg.sql ``` This is especially important for CI where you want to detect actual changes, not ordering noise. ### Wrong dialect auto-detected [Section titled “Wrong dialect auto-detected”](#wrong-dialect-auto-detected) Force the source dialect explicitly: ```bash sql-splitter convert ambiguous.sql --from mysql --to postgres -o pg.sql ``` ### Converted dump fails to import [Section titled “Converted dump fails to import”](#converted-dump-fails-to-import) 1. **Validate the output**: `sql-splitter validate pg.sql --dialect postgres` 2. **Check for unsupported features**: Re-run with `--strict` to see all warnings 3. **Check type mappings**: See [Type Mappings Reference](/reference/type-mappings) for dialect-specific details ## See Also [Section titled “See Also”](#see-also) * [`validate`](/commands/validate) - Validate converted output * [Dialect Reference](/reference/dialects) - Detailed dialect information * [Type Mappings Reference](/reference/type-mappings) - Data type conversion details * [JSON Output Schema](/reference/json-output/) - Schema for `--json` output # diff > Compare two SQL dumps for schema and data changes Compare two SQL dump files to detect schema and data differences. **Alias:** `df` (e.g., `sql-splitter df old.sql new.sql`) ## When to Use This [Section titled “When to Use This”](#when-to-use-this) * **Pre-deployment verification** - Compare production dump with staging to catch unexpected changes * **Migration validation** - Verify that a migration produces the expected schema changes * **Schema drift detection** - Detect when databases have diverged from each other * **Generate migration scripts** - Use `--format sql` to create migration DDL automatically * **Data auditing** - Track row additions, deletions, and modifications between snapshots Use [`validate`](/commands/validate) if you only need to check a single dump’s integrity without comparison. ## How It Works [Section titled “How It Works”](#how-it-works) The diff command loads both dumps, extracts schema metadata (tables, columns, indexes, foreign keys), and compares them structurally. For data comparison, it uses primary keys to match rows efficiently - this means tables without primary keys require special handling (`--allow-no-pk`). **Exit codes:** * `0` - Comparison completed (whether or not differences were found) * `1` - Error during comparison (missing file, parse error) * `2` - Invalid CLI arguments To gate CI on differences, use `--format json` and inspect the summary (see below)—the exit code alone does not signal differences. ## Usage [Section titled “Usage”](#usage) ```bash sql-splitter diff [OPTIONS] ``` ## Examples [Section titled “Examples”](#examples) ### Basic Comparison [Section titled “Basic Comparison”](#basic-comparison) ```bash # Compare two dumps (schema + data) sql-splitter diff old.sql new.sql # Schema-only comparison (faster, no data loading) sql-splitter diff old.sql new.sql --schema-only # Data-only comparison (assumes schema is identical) sql-splitter diff old.sql new.sql --data-only ``` ### Generating Migration Scripts [Section titled “Generating Migration Scripts”](#generating-migration-scripts) The `--format sql` option generates runnable DDL to transform the old schema into the new: ```bash sql-splitter diff old.sql new.sql --format sql -o migration.sql ``` This creates ALTER TABLE, CREATE TABLE, and DROP TABLE statements. Review before running in production. ### CI/CD Integration [Section titled “CI/CD Integration”](#cicd-integration) ```bash # Fail CI if any schema differences exist (exit code is 0 either way, # so check the JSON summary instead) sql-splitter diff production.sql staging.sql --schema-only --format json \ | jq -e '.summary.tables_added + .summary.tables_removed + .summary.tables_modified == 0' # Export diff as JSON for further processing sql-splitter diff old.sql new.sql --format json -o diff.json ``` ### Filtering and Noise Reduction [Section titled “Filtering and Noise Reduction”](#filtering-and-noise-reduction) ```bash # Compare specific tables only sql-splitter diff old.sql new.sql --tables users,orders # A qualified table name selects that schema only sql-splitter diff old.sql new.sql --tables tenant_a.users # Ignore auto-updated timestamp columns sql-splitter diff old.sql new.sql --ignore-columns "*.updated_at,*.created_at" # Ignore column order differences (focus on types/constraints) sql-splitter diff old.sql new.sql --ignore-order ``` ### Tables Without Primary Keys [Section titled “Tables Without Primary Keys”](#tables-without-primary-keys) By default, tables without PKs are skipped for data comparison. Override this behavior: ```bash # Use all columns as a composite key (slower but works) sql-splitter diff old.sql new.sql --allow-no-pk # Or specify a custom key for specific tables sql-splitter diff old.sql new.sql --primary-key logs:timestamp+message ``` ## Options [Section titled “Options”](#options) | Flag | Short | Description | Default | | ------------------ | ----- | ------------------------------------------- | ----------- | | `--output` | `-o` | Output file | stdout | | `--dialect` | `-d` | SQL dialect | auto-detect | | `--schema-only` | | Compare schema only | false | | `--data-only` | | Compare data only | false | | `--format` | `-f` | Output format: `text`, `json`, `sql` | `text` | | `--tables` | `-t` | Only compare these tables | all | | `--exclude` | `-e` | Exclude these tables | none | | `--max-pk-entries` | | Max PK entries to track | 10,000,000 | | `--verbose` | `-v` | Show sample PK values | false | | `--primary-key` | | Override PK (format: `table:col1+col2`) | auto | | `--ignore-order` | | Ignore column order differences | false | | `--ignore-columns` | | Ignore columns matching glob patterns | none | | `--allow-no-pk` | | Compare tables without PK using all columns | false | | `--progress` | `-p` | Show progress bar | false | ## What Diff Detects [Section titled “What Diff Detects”](#what-diff-detects) **Schema changes:** * Tables added/removed * Columns added/removed/modified (type, nullability) * Primary key changes * Foreign key changes * Index changes (CREATE INDEX, inline INDEX/KEY) **Data changes:** * Rows added/removed/modified * Uses PK-based comparison for efficiency ## Output Formats [Section titled “Output Formats”](#output-formats) ### Text (default) [Section titled “Text (default)”](#text-default) ```plaintext Schema Changes: + Table 'audit_logs' (new) + id INT NULL [PK] - Table 'temp_data' (removed) ~ Table 'users': + Column 'phone' TEXT NULL Data Changes: Table 'users': +5 rows, -2 rows, ~10 modified Table 'orders': +100 rows Summary: 1 tables added, 1 removed, 1 modified 105 rows added, 2 removed, 10 modified ``` ### JSON [Section titled “JSON”](#json) ```json { "schema": { "tables_added": ["audit_logs"], "tables_removed": ["temp_data"], "tables_modified": [ { "table_name": "users", "columns_added": [ { "name": "phone", "col_type": "TEXT", "is_nullable": true, "is_primary_key": false } ], "columns_removed": [], "columns_modified": [], "pk_changed": false, "fks_added": [], "fks_removed": [], "indexes_added": [], "indexes_removed": [] } ] }, "data": { "tables": { "users": { "old_row_count": 12, "new_row_count": 15, "added_count": 5, "removed_count": 2, "modified_count": 10, "truncated": false } } }, "summary": { "tables_added": 1, "tables_removed": 1, "tables_modified": 1, "rows_added": 5, "rows_removed": 2, "rows_modified": 10, "truncated": false } } ``` ### SQL Migration [Section titled “SQL Migration”](#sql-migration) ```sql -- Migration script generated by sql-splitter diff ALTER TABLE users ADD COLUMN phone VARCHAR(20); ALTER TABLE users MODIFY COLUMN email VARCHAR(255); CREATE TABLE audit_logs ( id INT PRIMARY KEY, ... ); DROP TABLE temp_data; ``` ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) ### Diff reports many changes but I only reformatted [Section titled “Diff reports many changes but I only reformatted”](#diff-reports-many-changes-but-i-only-reformatted) Statement ordering and formatting differences can cause noise. Normalize both dumps first: ```bash sql-splitter order old.sql -o old-sorted.sql sql-splitter order new.sql -o new-sorted.sql sql-splitter diff old-sorted.sql new-sorted.sql ``` See [`order`](/commands/order) for details on topological sorting and handling circular dependencies. ### Diff is slow on large dumps [Section titled “Diff is slow on large dumps”](#diff-is-slow-on-large-dumps) * Use `--schema-only` if you only need DDL comparison * Use `--tables` to limit scope to specific tables * Use `--max-pk-entries` to limit row tracking for data comparison ### Many “Table has no primary key” warnings [Section titled “Many “Table has no primary key” warnings”](#many-table-has-no-primary-key-warnings) Tables without primary keys can’t be efficiently compared. Options: 1. **Skip data comparison for those tables**: Default behavior, just note the warnings 2. **Use `--allow-no-pk`**: Compare using all columns (slower, may have false positives) 3. **Specify custom keys**: `--primary-key logs:timestamp+source_id` ### Summary shows changes but no detail [Section titled “Summary shows changes but no detail”](#summary-shows-changes-but-no-detail) The default output summarizes row changes per table. Use `--verbose` to see sample PK values for the affected rows: ```bash sql-splitter diff old.sql new.sql --verbose ``` ## See Also [Section titled “See Also”](#see-also) * [`validate`](/commands/validate) - Validate dump integrity * [`convert`](/commands/convert) - Convert between dialects * [`order`](/commands/order) - Normalize statement order for stable diffs # generate > Generate relational synthetic SQL from a model, a source dump, or both Generate relational synthetic SQL from a hand-authored YAML model, a model inferred from a SQL dump, or an inferred model plus explicit overrides. **Alias:** `gen` (e.g., `sql-splitter gen production.sql -o synthetic.sql`) Synthetic is not anonymized Generated output can replay literal choices, samples, defaults, and constraint values from a source dump or model. The `GEN-SOURCE-VALUES` advisory names every rule that can replay a literal (without printing the value). Use [`redact`](/commands/redact/) when you need irreversible anonymization of real data. ## When to Use This [Section titled “When to Use This”](#when-to-use-this) * **Seed relational fixtures** — Produce data that satisfies its own [primary keys](/reference/glossary#primary-key-pk) and [foreign keys](/reference/glossary#foreign-key-fk), so it loads cleanly into the same schema * **Share safe datasets** — Ship synthetic dumps that keep a real schema’s shape without exposing its rows * **Make CI fixtures repeatable** — Pin `--seed` so every run produces byte-identical output * **Scale a dataset up or down** — Multiply or cap row counts with `--scale`, `--rows`, and per-table overrides for load tests or small demos * **Derive a model from a real dump** — Profile and infer a starting model automatically, then edit and reuse it Use [`redact`](/commands/redact/) instead when you must anonymize the *real* rows rather than synthesize new ones. ## How It Works [Section titled “How It Works”](#how-it-works) `generate` turns a model into SQL. When you pass a source dump, it derives that model first, without ever loading the full dataset into memory: 1. **Profile** — A bounded `basic`/`full` [streaming](/reference/glossary#streaming) profiler samples the dump’s columns within a fixed memory budget 2. **Infer** — Structural [referential-integrity](/reference/glossary#referential-integrity) and column heuristics resolve an explicit per-column rule for each table 3. **Compile** — The model is validated and resolved into a plan (row counts, dependency order, key domains) 4. **Render** — The engine produces `CREATE TABLE` DDL and batched `INSERT`/`COPY` rows in the target dialect A hand-authored `--config` model skips the profile and infer steps. See [Profiling and inference](/commands/generate/inference/) for how rules are chosen and [Privacy and verification](/commands/generate/privacy-verification/) for what `--verify` audits. ## Usage [Section titled “Usage”](#usage) ```bash sql-splitter generate [OPTIONS] [INPUT] ``` Provide a source dump, a complete `--config` model, or both. A `kind: overrides` document requires a source dump or base model because it patches the inferred model rather than standing alone. ## Examples [Section titled “Examples”](#examples) ### Generate from a dump [Section titled “Generate from a dump”](#generate-from-a-dump) ```bash # Profile production.sql, infer a model, and generate synthetic SQL sql-splitter generate production.sql -o synthetic.sql ``` ### Repeatable fixtures from a model [Section titled “Repeatable fixtures from a model”](#repeatable-fixtures-from-a-model) ```bash # A pinned seed makes the output byte-identical across runs sql-splitter generate --config model.yaml --seed 42 -o synthetic.sql ``` ### Infer and save an editable model [Section titled “Infer and save an editable model”](#infer-and-save-an-editable-model) ```bash # Write the resolved model without generating SQL, then edit and reuse it sql-splitter generate production.sql --emit-config model.yaml sql-splitter generate --config model.yaml -o synthetic.sql ``` ### Apply overrides and scale in one run [Section titled “Apply overrides and scale in one run”](#apply-overrides-and-scale-in-one-run) ```bash # Patch the inferred model, save the resolved model, and emit 10% of the rows sql-splitter generate production.sql \ --config overrides.yaml \ --emit-config resolved.yaml \ --scale 0.1 \ -o synthetic.sql ``` ### Preflight (check, dry-run, verify) [Section titled “Preflight (check, dry-run, verify)”](#preflight-check-dry-run-verify) ```bash # Parse and compile a complete model; write nothing sql-splitter generate --config model.yaml --check # Resolve counts and dependencies without generating rows sql-splitter generate --config model.yaml --dry-run # Explain inferred column rules without printing observed values sql-splitter generate production.sql --dry-run --explain # Generate, audit exact constraints, then publish atomically sql-splitter generate --config model.yaml --verify -o synthetic.sql ``` `--strict` promotes warning-severity diagnostics to failures; informational diagnostics and advisories stay non-fatal. `--quiet` suppresses the ordinary human report, but safety advisories still reach stderr. ## Options [Section titled “Options”](#options) ### Input and model [Section titled “Input and model”](#input-and-model) | Flag | Short | Description | Default | | ------------------ | ----- | ---------------------------------- | ----------- | | `[INPUT]` | | Source SQL dump to profile | none | | `--config` | `-c` | Complete model or overrides YAML | none | | `--emit-config` | | Write the resolved complete model | none | | `--output` | `-o` | Generated SQL; `-` means stdout | stdout | | `--profile-depth` | | Profiling depth: `basic` or `full` | `basic` | | `--profile-sample` | | Retained profiling sample capacity | `1000` | | `--input-dialect` | | Force the source parsing dialect | auto-detect | ### Rendering [Section titled “Rendering”](#rendering) | Flag | Short | Description | Default | | -------------------------- | ----- | -------------------------------------------- | ------------------------- | | `--dialect` | | Output SQL dialect | model, source, then MySQL | | `--schema-only` | | Emit DDL only | false | | `--data-only` | | Emit row data only | false | | `--batch-size` | | Rows per `INSERT`/`COPY` batch (1–1,000,000) | `1000` | | `--no-copy` | | Use PostgreSQL `INSERT` instead of `COPY` | false | | `--mssql-production-style` | | Add production-style MSSQL DDL conventions | false | | `--mssql-go` | | Emit `GO` every N MSSQL insert batches | every batch | | `--compress` | | Reserved; currently unavailable | none | ### Volume and selection [Section titled “Volume and selection”](#volume-and-selection) | Flag | Short | Description | Default | | --------------- | ----- | ---------------------------------------------------------------- | ------- | | `--scale` | | Multiply every resolved row count | none | | `--rows` | | Set absolute [root-table](/reference/glossary#root-table) counts | none | | `--table-rows` | | Per-table absolute count (`table=count`); repeatable | none | | `--table-scale` | | Per-table scale (`table=factor`); repeatable | none | | `--max-rows` | | Cap every table, including family-planner children, last | none | | `--tables` | | Include matching tables and required dependencies | all | | `--exclude` | | Exclude matching tables | none | ### Randomness [Section titled “Randomness”](#randomness) | Flag | Short | Description | Default | | ------------- | ----- | ----------------------------------------- | ---------------------- | | `--seed` | | Stable root seed | model or fresh entropy | | `--randomize` | | Ignore the configured seed; use a new one | false | ### Preflight and reporting [Section titled “Preflight and reporting”](#preflight-and-reporting) | Flag | Short | Description | Default | | ------------ | ----- | ----------------------------------------- | ------- | | `--check` | | Compile a complete model only | false | | `--dry-run` | | Compile and report resolved counts | false | | `--verify` | | Generate, audit, and publish atomically | false | | `--explain` | | Include inference decisions | false | | `--strict` | | Fail on warning-severity diagnostics | false | | `--json` | | Print a machine-readable report on stdout | false | | `--quiet` | | Suppress ordinary human reporting | false | | `--progress` | | Accepted but not wired to generation | false | ## Constraints [Section titled “Constraints”](#constraints) * `--check`, `--dry-run`, and `--verify` are mutually exclusive. * `--schema-only` and `--data-only` are mutually exclusive. * `--seed` and `--randomize` are mutually exclusive. * `--scale` and `--rows` are mutually exclusive. * A table cannot receive both `--table-rows` and `--table-scale`. * `--json`, `--output -`, and `--emit-config -` each own stdout; select only one. * `--json` writes only the report to stdout, so it needs `-o ` for the generated SQL (or a report-only `--check`/`--dry-run` run). * `--check` requires a complete model and cannot profile `[INPUT]`. * `--verify` requires a real output path and cannot use compression. * MSSQL-specific flags require an explicit `--dialect mssql`. ## JSON Output [Section titled “JSON Output”](#json-output) `--json` writes a machine-readable report to stdout, so pair it with `-o ` for the generated SQL (or use `--check`/`--dry-run` for a report-only run): ```bash sql-splitter generate --config model.yaml --seed 42 -o synthetic.sql --json ``` ```json { "mode": "generate", "rows_written": 65, "effective_seed": 42, "diagnostics": [ { "code": "GEN-SOURCE-VALUES", "severity": "advisory", "path": "tables", "message": "2 rule(s) replay hand-authored literal values; the output is synthetic, not anonymized source data", "documentation_url": "https://sql-splitter.dev/commands/generate/diagnostics/#GEN-SOURCE-VALUES", "related": [ { "path": "tables.customers.columns.notes", "description": "constant" }, { "path": "tables.orders.columns.status", "description": "source_default" } ] } ] } ``` `mode` is `generate`, `dry_run`, or `check`. Every diagnostic carries a stable `code`, a `severity`, the config `path` it applies to, and a `documentation_url`. ## Exit Codes [Section titled “Exit Codes”](#exit-codes) | Code | Meaning | | ---- | ------------------------------------------------------------------ | | `0` | Generation or the requested preflight mode succeeded | | `1` | Model, warning-under-strict, runtime, I/O, or verification failure | | `2` | Invalid CLI usage or conflicting flags | See [Exit Codes](/reference/exit-codes) for scripting examples. ## Composing with Other Tools [Section titled “Composing with Other Tools”](#composing-with-other-tools) ### Infer → edit → reuse [Section titled “Infer → edit → reuse”](#infer--edit--reuse) ```bash # Save the inferred model, hand-tune it, then generate from the frozen model sql-splitter generate production.sql --emit-config model.yaml $EDITOR model.yaml sql-splitter generate --config model.yaml --seed 42 -o synthetic.sql ``` ### Generate → validate [Section titled “Generate → validate”](#generate--validate) ```bash # Confirm the synthetic dump parses and its foreign keys resolve sql-splitter generate --config model.yaml -o synthetic.sql sql-splitter validate synthetic.sql ``` (`--verify` runs an equivalent audit inline and only publishes on success.) ### Generate → split [Section titled “Generate → split”](#generate--split) ```bash # Generate one dump, then break it into per-table files sql-splitter generate --config model.yaml -o synthetic.sql sql-splitter split synthetic.sql -o tables/ ``` ### Sample → generate [Section titled “Sample → generate”](#sample--generate) ```bash # Reduce a huge production dump first, then synthesize from the smaller sample sql-splitter sample production.sql --rows 5000 -o sample.sql sql-splitter generate sample.sql -o synthetic.sql ``` ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) ### Conflicting flags rejected with exit code 2 [Section titled “Conflicting flags rejected with exit code 2”](#conflicting-flags-rejected-with-exit-code-2) **Symptom:** The command exits immediately with a message about mutually exclusive options. **Cause:** Two flags that own the same behavior were passed together — for example `--check` with `--verify`, `--scale` with `--rows`, or `--seed` with `--randomize`. **Fix:** Keep one flag from each conflicting pair. See [Constraints](#constraints) for the full list. ### `--json` prints a report but no SQL appears [Section titled “--json prints a report but no SQL appears”](#--json-prints-a-report-but-no-sql-appears) **Symptom:** With `--json` you get the report on stdout, but no generated SQL. **Cause:** In generate mode `--json` owns stdout and emits only the report, so the SQL has nowhere to go. **Fix:** Add `-o ` for the SQL, or use `--check`/`--dry-run` for a report-only run. ### `--check` fails asking for a complete model [Section titled “--check fails asking for a complete model”](#--check-fails-asking-for-a-complete-model) **Symptom:** `--check` errors when you pass a source dump. **Cause:** `--check` validates a complete `--config` model and does not profile `[INPUT]`. **Fix:** Run `--check` against a `--config` model, or use `--dry-run` to compile and report counts for a profiled dump. ### MSSQL flags are rejected [Section titled “MSSQL flags are rejected”](#mssql-flags-are-rejected) **Symptom:** `--mssql-production-style` or `--mssql-go` errors. **Cause:** These flags only apply to MSSQL output and require the dialect to be set explicitly. **Fix:** Add `--dialect mssql`. ### Output looks like real data [Section titled “Output looks like real data”](#output-looks-like-real-data) **Symptom:** Generated values match values from the source dump, and a `GEN-SOURCE-VALUES` advisory appears. **Cause:** Some rules (constants, replayed defaults, observed samples) can reproduce literal values. The output is synthetic, not anonymized. **Fix:** Review the rules the advisory names, or replace them with non-replaying generators. To anonymize *real* rows instead, use [`redact`](/commands/redact/). ## See Also [Section titled “See Also”](#see-also) Generate’s detail lives in dedicated reference pages: * [Model reference](/commands/generate/model-reference/) — the YAML model language * [Generators](/commands/generate/generators/) — per-column value producers * [Modifiers](/commands/generate/modifiers/) — post-generation value transforms * [Planners](/commands/generate/planners/) — coordinated multi-column and parent/child invariants * [Profiling and inference](/commands/generate/inference/) — how a model is derived from a dump * [Privacy and verification](/commands/generate/privacy-verification/) — safety advisories and exact checks * [Diagnostics](/commands/generate/diagnostics/) — stable `GEN-*` codes and anchors * [Rust API](/commands/generate/library-api/) — embedding generation in Rust Related commands and references: * [`redact`](/commands/redact/) — Anonymize real data instead of synthesizing new data * [`sample`](/commands/sample/) — Reduce a real dump before profiling it * [`validate`](/commands/validate/) — Check the generated dump’s integrity * [Dialects](/reference/dialects) — Dialect-specific output behavior * [JSON Output Schema](/reference/json-output/) — Schema for `--json` output # Diagnostic reference > Every built-in generate diagnostic, with stable case-sensitive anchors Every built-in generation diagnostic has a stable code, severity, path, message, and canonical documentation URL. Human output appends that URL; JSON output exposes it as `documentation_url`. Diagnostic codes identify the class of problem. The occurrence’s `path`, `message`, optional `help`, and `related` locations explain the specific context. Two occurrences of the same code can therefore have different reasons or related locations without minting new codes. ## Severity behavior [Section titled “Severity behavior”](#severity-behavior) | Severity | Exit behavior | | ---------- | -------------------------------------------------------- | | `info` | Never blocks generation; not promoted by `--strict` | | `advisory` | Non-fatal safety notice; remains visible under `--quiet` | | `warning` | Non-fatal normally; becomes a failure under `--strict` | | `error` | Blocks the current operation | A URL such as `https://sql-splitter.dev/commands/generate/diagnostics/#GEN-INFER-PLANNER-NOMINATE` jumps directly to the exact, case-preserving heading below. ## GEN-AFFIX-MISSING-VALUE — Affix value is missing [Section titled “GEN-AFFIX-MISSING-VALUE — Affix value is missing”](#GEN-AFFIX-MISSING-VALUE) Typical severity: **error** · Category: **modifier** A prefix or suffix modifier does not declare the value it should add. ## GEN-BOOLEAN-PROBABILITY — Boolean probability is invalid [Section titled “GEN-BOOLEAN-PROBABILITY — Boolean probability is invalid”](#GEN-BOOLEAN-PROBABILITY) Typical severity: **error** · Category: **generator** A boolean generator probability falls outside the supported zero-to-one range. ## GEN-BYTES-LENGTH-RANGE — Byte length range is invalid [Section titled “GEN-BYTES-LENGTH-RANGE — Byte length range is invalid”](#GEN-BYTES-LENGTH-RANGE) Typical severity: **error** · Category: **generator** A bytes generator has incompatible minimum and maximum lengths. ## GEN-CANCELLED — Generation was cancelled [Section titled “GEN-CANCELLED — Generation was cancelled”](#GEN-CANCELLED) Typical severity: **error** · Category: **runtime** The generation loop observed a cancellation request before it finished. ## GEN-CASE-INVALID-MODE — Case mode is invalid [Section titled “GEN-CASE-INVALID-MODE — Case mode is invalid”](#GEN-CASE-INVALID-MODE) Typical severity: **error** · Category: **modifier** A case modifier names a mode other than the supported lower, upper, or title transformations. ## GEN-CHILD-COUNT-IMPOSSIBLE — Child row count is impossible [Section titled “GEN-CHILD-COUNT-IMPOSSIBLE — Child row count is impossible”](#GEN-CHILD-COUNT-IMPOSSIBLE) Typical severity: **error** · Category: **selection** The resolved child count cannot satisfy the configured minimum fan-out for every parent. ## GEN-CHOICE-EMPTY — Choice list is empty [Section titled “GEN-CHOICE-EMPTY — Choice list is empty”](#GEN-CHOICE-EMPTY) Typical severity: **error** · Category: **generator** A choice generator has no values from which to draw. ## GEN-CHOICE-INVALID-VALUE — Choice value is incompatible [Section titled “GEN-CHOICE-INVALID-VALUE — Choice value is incompatible”](#GEN-CHOICE-INVALID-VALUE) Typical severity: **error** · Category: **generator** A choice value cannot be represented by the target column type. ## GEN-CHOICE-MISSING-VALUES — Choice values are missing [Section titled “GEN-CHOICE-MISSING-VALUES — Choice values are missing”](#GEN-CHOICE-MISSING-VALUES) Typical severity: **error** · Category: **generator** A choice generator omits its required values list. ## GEN-CLAMP-MISSING-BOUNDS — Clamp bounds are missing [Section titled “GEN-CLAMP-MISSING-BOUNDS — Clamp bounds are missing”](#GEN-CLAMP-MISSING-BOUNDS) Typical severity: **error** · Category: **modifier** A clamp modifier supplies neither a minimum nor a maximum bound. ## GEN-CLAMP-RANGE — Clamp range is invalid [Section titled “GEN-CLAMP-RANGE — Clamp range is invalid”](#GEN-CLAMP-RANGE) Typical severity: **error** · Category: **modifier** A clamp modifier’s maximum is below its minimum. ## GEN-COLUMN-CYCLE — Column dependency cycle [Section titled “GEN-COLUMN-CYCLE — Column dependency cycle”](#GEN-COLUMN-CYCLE) Typical severity: **error** · Category: **selection** Column and planner read/write dependencies contain a cycle that no single planner owns. ## GEN-COLUMN-OWNER-CONFLICT — Column has multiple owners [Section titled “GEN-COLUMN-OWNER-CONFLICT — Column has multiple owners”](#GEN-COLUMN-OWNER-CONFLICT) Typical severity: **error** · Category: **selection** More than one generator or planner claims responsibility for the same output column. ## GEN-COLUMN-OWNER-MISSING — Column has no owner [Section titled “GEN-COLUMN-OWNER-MISSING — Column has no owner”](#GEN-COLUMN-OWNER-MISSING) Typical severity: **error** · Category: **selection** A generated column has no generator, planner, or structural source that can produce its value. ## GEN-COMMERCE-MONEY-RANGE — Money range is invalid [Section titled “GEN-COMMERCE-MONEY-RANGE — Money range is invalid”](#GEN-COMMERCE-MONEY-RANGE) Typical severity: **error** · Category: **generator** A commerce money generator has incompatible amount bounds. ## GEN-COMMERCE-MONEY-SCALE — Money scale is invalid [Section titled “GEN-COMMERCE-MONEY-SCALE — Money scale is invalid”](#GEN-COMMERCE-MONEY-SCALE) Typical severity: **error** · Category: **generator** A commerce money generator requests an unsupported decimal scale. ## GEN-CONSTANT-INVALID-VALUE — Constant value is incompatible [Section titled “GEN-CONSTANT-INVALID-VALUE — Constant value is incompatible”](#GEN-CONSTANT-INVALID-VALUE) Typical severity: **error** · Category: **generator** A configured `constant` value cannot be represented by its target column type. ## GEN-CONFIG-COMPLETE-MODEL — Complete model overrides the profiled base [Section titled “GEN-CONFIG-COMPLETE-MODEL — Complete model overrides the profiled base”](#GEN-CONFIG-COMPLETE-MODEL) Typical severity: **warning** · Category: **config** A complete model was supplied with a source dump, so the complete model is authoritative. ## GEN-CONFIG-IO — Configuration file could not be read [Section titled “GEN-CONFIG-IO — Configuration file could not be read”](#GEN-CONFIG-IO) Typical severity: **error** · Category: **config** The root configuration or an imported configuration file could not be read. ## GEN-CONFIG-PARSE — Configuration YAML is invalid [Section titled “GEN-CONFIG-PARSE — Configuration YAML is invalid”](#GEN-CONFIG-PARSE) Typical severity: **error** · Category: **config** A configuration document is not valid YAML or contains a duplicate mapping key. ## GEN-CONFIG-ROLE — Configuration role is invalid [Section titled “GEN-CONFIG-ROLE — Configuration role is invalid”](#GEN-CONFIG-ROLE) Typical severity: **error** · Category: **config** A merged document cannot be parsed as its declared model or overrides role. ## GEN-COORDINATE-BOUNDS — Coordinate bounds are invalid [Section titled “GEN-COORDINATE-BOUNDS — Coordinate bounds are invalid”](#GEN-COORDINATE-BOUNDS) Typical severity: **error** · Category: **planner** A coordinate planner bounding box is inverted or outside valid latitude or longitude limits. ## GEN-COORDINATE-COLUMN-MISSING — Coordinate column is missing [Section titled “GEN-COORDINATE-COLUMN-MISSING — Coordinate column is missing”](#GEN-COORDINATE-COLUMN-MISSING) Typical severity: **error** · Category: **planner** A coordinate planner role names a column that does not exist. ## GEN-COORDINATE-PRECISION — Coordinate precision is invalid [Section titled “GEN-COORDINATE-PRECISION — Coordinate precision is invalid”](#GEN-COORDINATE-PRECISION) Typical severity: **error** · Category: **planner** A coordinate planner precision falls outside the supported range. ## GEN-COPY-MISSING-SOURCE — Copy source is missing [Section titled “GEN-COPY-MISSING-SOURCE — Copy source is missing”](#GEN-COPY-MISSING-SOURCE) Typical severity: **error** · Category: **generator** A copy generator omits the sibling column it should read. ## GEN-COPY-TYPE-MISMATCH — Copy source type is incompatible [Section titled “GEN-COPY-TYPE-MISMATCH — Copy source type is incompatible”](#GEN-COPY-TYPE-MISMATCH) Typical severity: **error** · Category: **generator** A copy generator’s source value cannot populate the target column type. ## GEN-COPY-UNKNOWN-FIELD — Copy source column is unknown [Section titled “GEN-COPY-UNKNOWN-FIELD — Copy source column is unknown”](#GEN-COPY-UNKNOWN-FIELD) Typical severity: **error** · Category: **generator** A copy generator names a sibling column that does not exist. ## GEN-COUNT-CONTROL-CONFLICT — Global count controls conflict [Section titled “GEN-COUNT-CONTROL-CONFLICT — Global count controls conflict”](#GEN-COUNT-CONTROL-CONFLICT) Typical severity: **error** · Category: **selection** Mutually exclusive global row-count controls were supplied together. ## GEN-DECIMAL-RANGE — Decimal range is invalid [Section titled “GEN-DECIMAL-RANGE — Decimal range is invalid”](#GEN-DECIMAL-RANGE) Typical severity: **error** · Category: **generator** A decimal generator’s maximum is below its minimum. ## GEN-DECIMAL-SCALE — Decimal scale is invalid [Section titled “GEN-DECIMAL-SCALE — Decimal scale is invalid”](#GEN-DECIMAL-SCALE) Typical severity: **error** · Category: **generator** A decimal generator’s scale cannot represent its configured range or target type. ## GEN-DETACHED-DEPENDENCY — Dependency was detached [Section titled “GEN-DETACHED-DEPENDENCY — Dependency was detached”](#GEN-DETACHED-DEPENDENCY) Typical severity: **warning** · Category: **selection** A foreign key points to a table unavailable to the generation run, so its rendered constraint was omitted. ## GEN-DIALECT-INVALID — SQL dialect is invalid [Section titled “GEN-DIALECT-INVALID — SQL dialect is invalid”](#GEN-DIALECT-INVALID) Typical severity: **error** · Category: **config** A source or output dialect is not one of `mysql`, `postgres`, `sqlite`, or `mssql`. ## GEN-DUPLICATE-COLUMN — Duplicate column name [Section titled “GEN-DUPLICATE-COLUMN — Duplicate column name”](#GEN-DUPLICATE-COLUMN) Typical severity: **error** · Category: **selection** A table’s schema declares the same column name more than once. ## GEN-EMIT-IO — Resolved model could not be written [Section titled “GEN-EMIT-IO — Resolved model could not be written”](#GEN-EMIT-IO) Typical severity: **error** · Category: **rendering** Writing the resolved model configuration failed. ## GEN-EMIT-SERIALIZE — Resolved model could not be serialized [Section titled “GEN-EMIT-SERIALIZE — Resolved model could not be serialized”](#GEN-EMIT-SERIALIZE) Typical severity: **error** · Category: **rendering** The resolved model could not be encoded as YAML. ## GEN-EXCLUDED-DEPENDENCY — Required dependency was excluded [Section titled “GEN-EXCLUDED-DEPENDENCY — Required dependency was excluded”](#GEN-EXCLUDED-DEPENDENCY) Typical severity: **error** · Category: **selection** A selected table depends on another table that filtering excluded. ## GEN-FAMILY-SPOOL — Family spool failed [Section titled “GEN-FAMILY-SPOOL — Family spool failed”](#GEN-FAMILY-SPOOL) Typical severity: **error** · Category: **runtime** A buffered cross-table planner could not write or replay its protected temporary data. ## GEN-FILE-COLUMN-MISSING — File metadata column is missing [Section titled “GEN-FILE-COLUMN-MISSING — File metadata column is missing”](#GEN-FILE-COLUMN-MISSING) Typical severity: **error** · Category: **planner** A file metadata role names a column that does not exist. ## GEN-FILE-EXTENSIONS — File extension list is invalid [Section titled “GEN-FILE-EXTENSIONS — File extension list is invalid”](#GEN-FILE-EXTENSIONS) Typical severity: **error** · Category: **planner** A file metadata planner has no usable recognized extension. ## GEN-FILE-HASH-KIND — File hash kind is invalid [Section titled “GEN-FILE-HASH-KIND — File hash kind is invalid”](#GEN-FILE-HASH-KIND) Typical severity: **error** · Category: **planner** A file metadata planner requests an unsupported digest shape. ## GEN-FILE-SIZE-RANGE — File size range is invalid [Section titled “GEN-FILE-SIZE-RANGE — File size range is invalid”](#GEN-FILE-SIZE-RANGE) Typical severity: **error** · Category: **planner** A file metadata size range is negative or inverted. ## GEN-FK-CYCLE — Nullable foreign-key cycle [Section titled “GEN-FK-CYCLE — Nullable foreign-key cycle”](#GEN-FK-CYCLE) Typical severity: **warning** · Category: **selection** Nullable foreign keys form a cycle; the cyclic tables cannot all be ordered parent-before-child in the output. ## GEN-FOREIGN-KEY-UNRESOLVED — Foreign key value cannot be resolved [Section titled “GEN-FOREIGN-KEY-UNRESOLVED — Foreign key value cannot be resolved”](#GEN-FOREIGN-KEY-UNRESOLVED) Typical severity: **error** · Category: **selection** A relationship-owned foreign-key column has no usable value source. ## GEN-FORMAT-MISSING-TEMPLATE — Format template is missing [Section titled “GEN-FORMAT-MISSING-TEMPLATE — Format template is missing”](#GEN-FORMAT-MISSING-TEMPLATE) Typical severity: **error** · Category: **modifier** A format modifier omits its required template. ## GEN-GAUSSIAN-MISSING-PARAMS — Gaussian parameters are missing [Section titled “GEN-GAUSSIAN-MISSING-PARAMS — Gaussian parameters are missing”](#GEN-GAUSSIAN-MISSING-PARAMS) Typical severity: **error** · Category: **generator** A gaussian generator omits required distribution parameters. ## GEN-GAUSSIAN-NON-FINITE — Gaussian parameter is not finite [Section titled “GEN-GAUSSIAN-NON-FINITE — Gaussian parameter is not finite”](#GEN-GAUSSIAN-NON-FINITE) Typical severity: **error** · Category: **generator** A gaussian generator contains a NaN or infinite parameter. ## GEN-GAUSSIAN-RANGE — Gaussian range is invalid [Section titled “GEN-GAUSSIAN-RANGE — Gaussian range is invalid”](#GEN-GAUSSIAN-RANGE) Typical severity: **error** · Category: **generator** A gaussian generator’s configured bounds are incompatible. ## GEN-GENERATOR-TYPE — Generator does not support the column type [Section titled “GEN-GENERATOR-TYPE — Generator does not support the column type”](#GEN-GENERATOR-TYPE) Typical severity: **error** · Category: **generator** The selected generator cannot produce values for the target SQL type family. ## GEN-GENERATOR-UNKNOWN — Generator kind is unknown [Section titled “GEN-GENERATOR-UNKNOWN — Generator kind is unknown”](#GEN-GENERATOR-UNKNOWN) Typical severity: **error** · Category: **generator** The model names a generator kind that is not registered. ## GEN-HISTOGRAM-ALL-ZERO — Histogram weights are all zero [Section titled “GEN-HISTOGRAM-ALL-ZERO — Histogram weights are all zero”](#GEN-HISTOGRAM-ALL-ZERO) Typical severity: **error** · Category: **generator** A histogram has no bin with positive probability. ## GEN-HISTOGRAM-EMPTY — Histogram is empty [Section titled “GEN-HISTOGRAM-EMPTY — Histogram is empty”](#GEN-HISTOGRAM-EMPTY) Typical severity: **error** · Category: **generator** A histogram generator has no bins from which to draw. ## GEN-HISTOGRAM-INVALID-BIN — Histogram bin is invalid [Section titled “GEN-HISTOGRAM-INVALID-BIN — Histogram bin is invalid”](#GEN-HISTOGRAM-INVALID-BIN) Typical severity: **error** · Category: **generator** A histogram bin is malformed or cannot be represented by the target column. ## GEN-HISTOGRAM-MISSING-BINS — Histogram bins are missing [Section titled “GEN-HISTOGRAM-MISSING-BINS — Histogram bins are missing”](#GEN-HISTOGRAM-MISSING-BINS) Typical severity: **error** · Category: **generator** A histogram generator omits its required bins. ## GEN-HISTOGRAM-NON-FINITE — Histogram value is not finite [Section titled “GEN-HISTOGRAM-NON-FINITE — Histogram value is not finite”](#GEN-HISTOGRAM-NON-FINITE) Typical severity: **error** · Category: **generator** A histogram bin contains a NaN or infinite bound or weight. ## GEN-HISTOGRAM-RANGE — Histogram range is invalid [Section titled “GEN-HISTOGRAM-RANGE — Histogram range is invalid”](#GEN-HISTOGRAM-RANGE) Typical severity: **error** · Category: **generator** A histogram bin or overall range has incompatible bounds. ## GEN-HISTOGRAM-UNSORTED — Histogram bins are unsorted [Section titled “GEN-HISTOGRAM-UNSORTED — Histogram bins are unsorted”](#GEN-HISTOGRAM-UNSORTED) Typical severity: **error** · Category: **generator** Histogram bin boundaries are not in ascending order. ## GEN-IMPORT-COLLISION — Imported settings collide [Section titled “GEN-IMPORT-COLLISION — Imported settings collide”](#GEN-IMPORT-COLLISION) Typical severity: **error** · Category: **config** Two imports define the same configuration path without an unambiguous owner. ## GEN-IMPORT-KIND — Imported document has the wrong kind [Section titled “GEN-IMPORT-KIND — Imported document has the wrong kind”](#GEN-IMPORT-KIND) Typical severity: **error** · Category: **config** An imported document is not an overrides document. ## GEN-IMPORT-NESTED — Nested imports are not allowed [Section titled “GEN-IMPORT-NESTED — Nested imports are not allowed”](#GEN-IMPORT-NESTED) Typical severity: **error** · Category: **config** An imported overrides document declares its own imports. ## GEN-IMPORT-REMOTE — Remote import is not allowed [Section titled “GEN-IMPORT-REMOTE — Remote import is not allowed”](#GEN-IMPORT-REMOTE) Typical severity: **error** · Category: **config** An import path is absolute or looks like a remote URL. ## GEN-INCOMPLETE-ROWS — Row-count override is incomplete [Section titled “GEN-INCOMPLETE-ROWS — Row-count override is incomplete”](#GEN-INCOMPLETE-ROWS) Typical severity: **error** · Category: **config** An override changes the row-count rule kind without supplying the fields that kind needs. ## GEN-INFER-FAILED — Model inference failed [Section titled “GEN-INFER-FAILED — Model inference failed”](#GEN-INFER-FAILED) Typical severity: **error** · Category: **inference** Profiling evidence could not be converted into a complete synthetic model. ## GEN-INFER-PLANNER-NOMINATE — Schema matches an optional planner [Section titled “GEN-INFER-PLANNER-NOMINATE — Schema matches an optional planner”](#GEN-INFER-PLANNER-NOMINATE) Typical severity: **info** · Category: **inference** The schema has the columns required by a planner that inference does not enable automatically. ## GEN-INFER-SOURCE-DERIVED — Inference retained source-derived literals [Section titled “GEN-INFER-SOURCE-DERIVED — Inference retained source-derived literals”](#GEN-INFER-SOURCE-DERIVED) Typical severity: **warning** · Category: **inference** An inferred rule embeds bounded literal values observed in the source dump. ## GEN-INFER-TABLE-UNKNOWN — Profiled table is absent from the schema [Section titled “GEN-INFER-TABLE-UNKNOWN — Profiled table is absent from the schema”](#GEN-INFER-TABLE-UNKNOWN) Typical severity: **warning** · Category: **inference** Profiling produced evidence for a table that the inferred schema does not contain. ## GEN-INTEGER-RANGE — Integer range is invalid [Section titled “GEN-INTEGER-RANGE — Integer range is invalid”](#GEN-INTEGER-RANGE) Typical severity: **error** · Category: **generator** An integer generator’s maximum is below its minimum. ## GEN-INTERVAL-COLUMN-MISSING — Interval column is missing [Section titled “GEN-INTERVAL-COLUMN-MISSING — Interval column is missing”](#GEN-INTERVAL-COLUMN-MISSING) Typical severity: **error** · Category: **planner** A temporal interval role names a column that does not exist. ## GEN-INTERVAL-DURATION — Interval duration is invalid [Section titled “GEN-INTERVAL-DURATION — Interval duration is invalid”](#GEN-INTERVAL-DURATION) Typical severity: **error** · Category: **planner** A temporal interval duration is negative, overflowing, or incompatible with inclusive ends. ## GEN-INTERVAL-OPEN-END — Open interval cannot represent a missing end [Section titled “GEN-INTERVAL-OPEN-END — Open interval cannot represent a missing end”](#GEN-INTERVAL-OPEN-END) Typical severity: **error** · Category: **planner** Open rows are possible but the configured end column is not nullable. ## GEN-INTERVAL-START — Interval start configuration is invalid [Section titled “GEN-INTERVAL-START — Interval start configuration is invalid”](#GEN-INTERVAL-START) Typical severity: **error** · Category: **planner** A temporal interval start range is missing, unparsable, or inverted. ## GEN-INTERVAL-TIMEZONE — Interval timezone is invalid [Section titled “GEN-INTERVAL-TIMEZONE — Interval timezone is invalid”](#GEN-INTERVAL-TIMEZONE) Typical severity: **error** · Category: **planner** A temporal interval names an unknown timezone. ## GEN-INVALID-GLOB — Table glob is invalid [Section titled “GEN-INVALID-GLOB — Table glob is invalid”](#GEN-INVALID-GLOB) Typical severity: **error** · Category: **selection** A table include or exclude pattern cannot be compiled as a glob. ## GEN-JSON-VALUE-INVALID — JSON value is invalid [Section titled “GEN-JSON-VALUE-INVALID — JSON value is invalid”](#GEN-JSON-VALUE-INVALID) Typical severity: **error** · Category: **generator** A JSON value generator contains text that is not valid JSON. ## GEN-JUNCTION-COLUMN-MISSING — Junction column is missing [Section titled “GEN-JUNCTION-COLUMN-MISSING — Junction column is missing”](#GEN-JUNCTION-COLUMN-MISSING) Typical severity: **error** · Category: **planner** A junction planner role names a column that does not exist. ## GEN-JUNCTION-EXHAUSTED — Junction pair space is exhausted [Section titled “GEN-JUNCTION-EXHAUSTED — Junction pair space is exhausted”](#GEN-JUNCTION-EXHAUSTED) Typical severity: **error** · Category: **planner** The requested junction rows exceed the number of distinct left/right pairs. ## GEN-JUNCTION-KEY-UNSUPPORTED — Junction key domain is unsupported [Section titled “GEN-JUNCTION-KEY-UNSUPPORTED — Junction key domain is unsupported”](#GEN-JUNCTION-KEY-UNSUPPORTED) Typical severity: **error** · Category: **planner** A junction relationship does not expose a dense key domain. ## GEN-JUNCTION-RELATIONSHIP — Junction relationship is invalid [Section titled “GEN-JUNCTION-RELATIONSHIP — Junction relationship is invalid”](#GEN-JUNCTION-RELATIONSHIP) Typical severity: **error** · Category: **planner** A junction planner omits or names an unknown left or right relationship. ## GEN-KEY-DOMAIN-UNSUPPORTED — Relationship key domain is unsupported [Section titled “GEN-KEY-DOMAIN-UNSUPPORTED — Relationship key domain is unsupported”](#GEN-KEY-DOMAIN-UNSUPPORTED) Typical severity: **error** · Category: **runtime** A relationship targets a parent key that cannot be enumerated as a supported dense domain. ## GEN-LIFECYCLE-COLUMN-MISSING — Lifecycle column is missing [Section titled “GEN-LIFECYCLE-COLUMN-MISSING — Lifecycle column is missing”](#GEN-LIFECYCLE-COLUMN-MISSING) Typical severity: **error** · Category: **planner** A lifecycle planner role names a column that does not exist. ## GEN-LIFECYCLE-NULLABILITY — Lifecycle column nullability is impossible [Section titled “GEN-LIFECYCLE-NULLABILITY — Lifecycle column nullability is impossible”](#GEN-LIFECYCLE-NULLABILITY) Typical severity: **error** · Category: **planner** A timestamp can be absent in a reachable state but its target column is not nullable. ## GEN-LIFECYCLE-RANGE — Lifecycle start range is invalid [Section titled “GEN-LIFECYCLE-RANGE — Lifecycle start range is invalid”](#GEN-LIFECYCLE-RANGE) Typical severity: **error** · Category: **planner** A lifecycle start range is missing, unparsable, or inverted. ## GEN-LIFECYCLE-STATES — Lifecycle states are invalid [Section titled “GEN-LIFECYCLE-STATES — Lifecycle states are invalid”](#GEN-LIFECYCLE-STATES) Typical severity: **error** · Category: **planner** A lifecycle planner has an empty or duplicate state chain. ## GEN-LIFECYCLE-STATUS-VOCABULARY — Lifecycle status vocabulary is inconsistent [Section titled “GEN-LIFECYCLE-STATUS-VOCABULARY — Lifecycle status vocabulary is inconsistent”](#GEN-LIFECYCLE-STATUS-VOCABULARY) Typical severity: **error** · Category: **planner** A lifecycle column mapping names a state outside the declared state chain. ## GEN-LIFECYCLE-STEP — Lifecycle step is invalid [Section titled “GEN-LIFECYCLE-STEP — Lifecycle step is invalid”](#GEN-LIFECYCLE-STEP) Typical severity: **error** · Category: **planner** A lifecycle timestamp step is negative, overflowing, or uses an unknown unit. ## GEN-LIFECYCLE-WEIGHTS — Lifecycle weights are invalid [Section titled “GEN-LIFECYCLE-WEIGHTS — Lifecycle weights are invalid”](#GEN-LIFECYCLE-WEIGHTS) Typical severity: **error** · Category: **planner** Lifecycle terminal-state weights are malformed, negative, or sum to zero. ## GEN-LOSSY-TYPE — Type conversion is lossy [Section titled “GEN-LOSSY-TYPE — Type conversion is lossy”](#GEN-LOSSY-TYPE) Typical severity: **warning** · Category: **rendering** Cross-dialect rendering cannot preserve a source column type exactly. ## GEN-MAX-ROWS-CAPPED — Maximum row cap reduced a table [Section titled “GEN-MAX-ROWS-CAPPED — Maximum row cap reduced a table”](#GEN-MAX-ROWS-CAPPED) Typical severity: **warning** · Category: **selection** The final maximum-row limit reduced a table’s resolved row count. ## GEN-MISSING-COLUMN — Override column does not exist [Section titled “GEN-MISSING-COLUMN — Override column does not exist”](#GEN-MISSING-COLUMN) Typical severity: **error** · Category: **config** An overrides document names a column absent from the base model. ## GEN-MISSING-TABLE — Override table does not exist [Section titled “GEN-MISSING-TABLE — Override table does not exist”](#GEN-MISSING-TABLE) Typical severity: **error** · Category: **config** An overrides document names a table absent from the base model. ## GEN-MODIFIER-TYPE — Modifier does not support the column type [Section titled “GEN-MODIFIER-TYPE — Modifier does not support the column type”](#GEN-MODIFIER-TYPE) Typical severity: **error** · Category: **modifier** The selected modifier cannot transform the target SQL type family. ## GEN-MODIFIER-UNKNOWN — Modifier kind is unknown [Section titled “GEN-MODIFIER-UNKNOWN — Modifier kind is unknown”](#GEN-MODIFIER-UNKNOWN) Typical severity: **error** · Category: **modifier** The model names a modifier kind that is not registered. ## GEN-MONOTONIC-STEP — Monotonic step is invalid [Section titled “GEN-MONOTONIC-STEP — Monotonic step is invalid”](#GEN-MONOTONIC-STEP) Typical severity: **error** · Category: **generator** A monotonic generator uses a zero or otherwise invalid step. ## GEN-NULL-ON-NON-NULLABLE — Null generator targets a non-nullable column [Section titled “GEN-NULL-ON-NON-NULLABLE — Null generator targets a non-nullable column”](#GEN-NULL-ON-NON-NULLABLE) Typical severity: **error** · Category: **generator** A rule can emit null for a column whose schema forbids null values. ## GEN-NULL-RATE-MISSING-RATE — Null-rate probability is missing [Section titled “GEN-NULL-RATE-MISSING-RATE — Null-rate probability is missing”](#GEN-NULL-RATE-MISSING-RATE) Typical severity: **error** · Category: **modifier** A null-rate modifier omits its required rate. ## GEN-NULL-RATE-ON-NON-NULLABLE — Null-rate modifier targets a non-nullable column [Section titled “GEN-NULL-RATE-ON-NON-NULLABLE — Null-rate modifier targets a non-nullable column”](#GEN-NULL-RATE-ON-NON-NULLABLE) Typical severity: **error** · Category: **modifier** A null-rate modifier can emit null for a column whose schema forbids it. ## GEN-NULL-RATE-RANGE — Null-rate probability is invalid [Section titled “GEN-NULL-RATE-RANGE — Null-rate probability is invalid”](#GEN-NULL-RATE-RANGE) Typical severity: **error** · Category: **modifier** A null-rate probability falls outside the supported zero-to-one range. ## GEN-OBSERVED-SAMPLE-ALL-ZERO — Observed-sample weights are all zero [Section titled “GEN-OBSERVED-SAMPLE-ALL-ZERO — Observed-sample weights are all zero”](#GEN-OBSERVED-SAMPLE-ALL-ZERO) Typical severity: **error** · Category: **generator** An observed sample has no value with positive probability. ## GEN-OBSERVED-SAMPLE-EMPTY — Observed sample is empty [Section titled “GEN-OBSERVED-SAMPLE-EMPTY — Observed sample is empty”](#GEN-OBSERVED-SAMPLE-EMPTY) Typical severity: **error** · Category: **generator** An observed-sample generator has no values from which to draw. ## GEN-OBSERVED-SAMPLE-INVALID-VALUE — Observed-sample value is incompatible [Section titled “GEN-OBSERVED-SAMPLE-INVALID-VALUE — Observed-sample value is incompatible”](#GEN-OBSERVED-SAMPLE-INVALID-VALUE) Typical severity: **error** · Category: **generator** An observed sample value cannot be represented by the target column type. ## GEN-OBSERVED-SAMPLE-INVALID-WEIGHT — Observed-sample weight is invalid [Section titled “GEN-OBSERVED-SAMPLE-INVALID-WEIGHT — Observed-sample weight is invalid”](#GEN-OBSERVED-SAMPLE-INVALID-WEIGHT) Typical severity: **error** · Category: **generator** An observed sample contains a negative, NaN, or infinite weight. ## GEN-OBSERVED-SAMPLE-MISSING-VALUES — Observed-sample values are missing [Section titled “GEN-OBSERVED-SAMPLE-MISSING-VALUES — Observed-sample values are missing”](#GEN-OBSERVED-SAMPLE-MISSING-VALUES) Typical severity: **error** · Category: **generator** An observed-sample generator omits its required values. ## GEN-ORDER-FAMILY-CHILD-UNKNOWN — Order-family child table is unknown [Section titled “GEN-ORDER-FAMILY-CHILD-UNKNOWN — Order-family child table is unknown”](#GEN-ORDER-FAMILY-CHILD-UNKNOWN) Typical severity: **error** · Category: **planner** An order-family planner names a child table absent from the model. ## GEN-ORDER-FAMILY-COLUMN-MISSING — Order-family column is missing [Section titled “GEN-ORDER-FAMILY-COLUMN-MISSING — Order-family column is missing”](#GEN-ORDER-FAMILY-COLUMN-MISSING) Typical severity: **error** · Category: **planner** A required parent or child money role is absent or names an unknown column. ## GEN-ORDER-FAMILY-CONFIG — Order-family configuration is invalid [Section titled “GEN-ORDER-FAMILY-CONFIG — Order-family configuration is invalid”](#GEN-ORDER-FAMILY-CONFIG) Typical severity: **error** · Category: **planner** An order-family planner omits or misconfigures a required top-level option. ## GEN-ORDER-FAMILY-OVERFLOW — Order-family arithmetic overflow [Section titled “GEN-ORDER-FAMILY-OVERFLOW — Order-family arithmetic overflow”](#GEN-ORDER-FAMILY-OVERFLOW) Typical severity: **error** · Category: **planner** Order-family minor-unit arithmetic exceeds the representable range. ## GEN-ORDER-FAMILY-RELATIONSHIP — Order-family relationship is invalid [Section titled “GEN-ORDER-FAMILY-RELATIONSHIP — Order-family relationship is invalid”](#GEN-ORDER-FAMILY-RELATIONSHIP) Typical severity: **error** · Category: **planner** The named child relationship does not reference the planner’s parent table. ## GEN-ORDER-FAMILY-SCALE — Order-family currency scale is inconsistent [Section titled “GEN-ORDER-FAMILY-SCALE — Order-family currency scale is inconsistent”](#GEN-ORDER-FAMILY-SCALE) Typical severity: **error** · Category: **planner** A configured money column does not match the planner’s currency scale. ## GEN-ORDER-FAMILY-UNKNOWN-FIELD — Order-family field is unknown [Section titled “GEN-ORDER-FAMILY-UNKNOWN-FIELD — Order-family field is unknown”](#GEN-ORDER-FAMILY-UNKNOWN-FIELD) Typical severity: **error** · Category: **planner** An order-family planner uses a removed flat field instead of the parent or child column maps. ## GEN-ORDER-FAMILY-ZERO-LINES — Order-family line count is impossible [Section titled “GEN-ORDER-FAMILY-ZERO-LINES — Order-family line count is impossible”](#GEN-ORDER-FAMILY-ZERO-LINES) Typical severity: **error** · Category: **planner** A configured line distribution can yield zero lines where the planner requires at least one. ## GEN-OUTPUT-BATCH-SIZE — Output batch size is invalid [Section titled “GEN-OUTPUT-BATCH-SIZE — Output batch size is invalid”](#GEN-OUTPUT-BATCH-SIZE) Typical severity: **error** · Category: **config** A model output batch size is zero or exceeds 1,000,000 rows. ## GEN-OUTPUT-IO — Generated output could not be written [Section titled “GEN-OUTPUT-IO — Generated output could not be written”](#GEN-OUTPUT-IO) Typical severity: **error** · Category: **runtime** Opening, writing, flushing, or publishing the generated SQL output failed. ## GEN-OPERATOR-UNKNOWN-ARGUMENT — Operator argument is unknown [Section titled “GEN-OPERATOR-UNKNOWN-ARGUMENT — Operator argument is unknown”](#GEN-OPERATOR-UNKNOWN-ARGUMENT) Typical severity: **error** · Category: **config** A generator, modifier, or planner contains an argument that its registered descriptor does not accept. ## GEN-OVERRIDES-NO-BASE — Overrides document has no base model [Section titled “GEN-OVERRIDES-NO-BASE — Overrides document has no base model”](#GEN-OVERRIDES-NO-BASE) Typical severity: **error** · Category: **config** An overrides document was supplied without a source dump or other base model. ## GEN-PATTERN-MISSING-MASK — Pattern mask is missing [Section titled “GEN-PATTERN-MISSING-MASK — Pattern mask is missing”](#GEN-PATTERN-MISSING-MASK) Typical severity: **error** · Category: **generator** A pattern generator omits the mask that describes its output. ## GEN-PLANNER-UNKNOWN — Planner kind is unknown [Section titled “GEN-PLANNER-UNKNOWN — Planner kind is unknown”](#GEN-PLANNER-UNKNOWN) Typical severity: **error** · Category: **planner** The model names a planner kind that is not registered. ## GEN-POLYMORPHIC-COLUMN-MISSING — Polymorphic column is missing [Section titled “GEN-POLYMORPHIC-COLUMN-MISSING — Polymorphic column is missing”](#GEN-POLYMORPHIC-COLUMN-MISSING) Typical severity: **error** · Category: **planner** A polymorphic planner role names a column that does not exist. ## GEN-POLYMORPHIC-KEY-UNSUPPORTED — Polymorphic key domain is unsupported [Section titled “GEN-POLYMORPHIC-KEY-UNSUPPORTED — Polymorphic key domain is unsupported”](#GEN-POLYMORPHIC-KEY-UNSUPPORTED) Typical severity: **error** · Category: **planner** A polymorphic target does not expose a supported dense key domain. ## GEN-POLYMORPHIC-TARGET-UNKNOWN — Polymorphic target is unknown [Section titled “GEN-POLYMORPHIC-TARGET-UNKNOWN — Polymorphic target is unknown”](#GEN-POLYMORPHIC-TARGET-UNKNOWN) Typical severity: **error** · Category: **planner** A polymorphic target table or identifier column does not exist. ## GEN-POLYMORPHIC-TARGETS — Polymorphic targets are invalid [Section titled “GEN-POLYMORPHIC-TARGETS — Polymorphic targets are invalid”](#GEN-POLYMORPHIC-TARGETS) Typical severity: **error** · Category: **planner** A polymorphic planner has no usable target with rows and a valid weight. ## GEN-PROFILE-DECODE-SKIPPED — Some source values could not be decoded [Section titled “GEN-PROFILE-DECODE-SKIPPED — Some source values could not be decoded”](#GEN-PROFILE-DECODE-SKIPPED) Typical severity: **warning** · Category: **profiling** The profiler counted rows whose values could not all be decoded into column evidence. ## GEN-PROFILE-SCHEMA-LATE — Source schema arrived after data [Section titled “GEN-PROFILE-SCHEMA-LATE — Source schema arrived after data”](#GEN-PROFILE-SCHEMA-LATE) Typical severity: **warning** · Category: **profiling** The dump declared a table after more data rows than the bounded late-schema buffer could retain. The table row count remains exact, but per-column counts, null rates, and value evidence apply only to the retained rows. ## GEN-PROGRESS-COLUMN-MISSING — Progress column is missing [Section titled “GEN-PROGRESS-COLUMN-MISSING — Progress column is missing”](#GEN-PROGRESS-COLUMN-MISSING) Typical severity: **error** · Category: **planner** A progress-counter role names a column that does not exist. ## GEN-PROGRESS-COMPLETION — Progress completion state is impossible [Section titled “GEN-PROGRESS-COMPLETION — Progress completion state is impossible”](#GEN-PROGRESS-COMPLETION) Typical severity: **error** · Category: **planner** Configured completion timestamps or nullability cannot represent every reachable progress state. ## GEN-PROGRESS-OBSERVED — Observed progress lacks evidence [Section titled “GEN-PROGRESS-OBSERVED — Observed progress lacks evidence”](#GEN-PROGRESS-OBSERVED) Typical severity: **error** · Category: **planner** Observed progress was requested where no planner evidence can form the required exact partition. ## GEN-PROGRESS-OVERFLOW — Progress counter can overflow [Section titled “GEN-PROGRESS-OVERFLOW — Progress counter can overflow”](#GEN-PROGRESS-OVERFLOW) Typical severity: **error** · Category: **planner** A possible total exceeds the capacity of one or more counter columns. ## GEN-PROGRESS-PARTITION — Progress partition mode is invalid [Section titled “GEN-PROGRESS-PARTITION — Progress partition mode is invalid”](#GEN-PROGRESS-PARTITION) Typical severity: **error** · Category: **planner** A progress planner names an unsupported partition mode. ## GEN-PROGRESS-STATUS-VOCABULARY — Progress status vocabulary is incomplete [Section titled “GEN-PROGRESS-STATUS-VOCABULARY — Progress status vocabulary is incomplete”](#GEN-PROGRESS-STATUS-VOCABULARY) Typical severity: **error** · Category: **planner** A reachable progress state has no status label available. ## GEN-PROGRESS-TOTAL — Progress total range is invalid [Section titled “GEN-PROGRESS-TOTAL — Progress total range is invalid”](#GEN-PROGRESS-TOTAL) Typical severity: **error** · Category: **planner** A progress total is negative or has an inverted range. ## GEN-PROGRESS-WEIGHTS — Progress weights are invalid [Section titled “GEN-PROGRESS-WEIGHTS — Progress weights are invalid”](#GEN-PROGRESS-WEIGHTS) Typical severity: **error** · Category: **planner** Progress-state weights are non-finite, negative, or sum to zero. ## GEN-RANDOM-STRING-INVALID-ALPHABET — Random-string alphabet is invalid [Section titled “GEN-RANDOM-STRING-INVALID-ALPHABET — Random-string alphabet is invalid”](#GEN-RANDOM-STRING-INVALID-ALPHABET) Typical severity: **error** · Category: **generator** A random-string generator has an empty or unsupported alphabet. ## GEN-RANGED-INTEGER-RANGE — Ranged integer bounds are invalid [Section titled “GEN-RANGED-INTEGER-RANGE — Ranged integer bounds are invalid”](#GEN-RANGED-INTEGER-RANGE) Typical severity: **error** · Category: **generator** A ranged-integer generator has incompatible bounds. ## GEN-REGISTRY-ALIAS-DUPLICATE — Registry alias is duplicated [Section titled “GEN-REGISTRY-ALIAS-DUPLICATE — Registry alias is duplicated”](#GEN-REGISTRY-ALIAS-DUPLICATE) Typical severity: **error** · Category: **registry** Two registered operators claim the same alias. ## GEN-REGISTRY-ALIAS-SHADOWS-KIND — Registry alias shadows a kind [Section titled “GEN-REGISTRY-ALIAS-SHADOWS-KIND — Registry alias shadows a kind”](#GEN-REGISTRY-ALIAS-SHADOWS-KIND) Typical severity: **error** · Category: **registry** An operator alias collides with a registered canonical kind. ## GEN-REGISTRY-DUPLICATE — Registry kind is duplicated [Section titled “GEN-REGISTRY-DUPLICATE — Registry kind is duplicated”](#GEN-REGISTRY-DUPLICATE) Typical severity: **error** · Category: **registry** The same canonical operator kind was registered more than once. ## GEN-RELATIONSHIP-UNKNOWN — Relationship is unknown [Section titled “GEN-RELATIONSHIP-UNKNOWN — Relationship is unknown”](#GEN-RELATIONSHIP-UNKNOWN) Typical severity: **error** · Category: **planner** A rule or planner names a relationship absent from the table schema. ## GEN-RELATIVE-MISSING-SOURCE — Relative source is missing [Section titled “GEN-RELATIVE-MISSING-SOURCE — Relative source is missing”](#GEN-RELATIVE-MISSING-SOURCE) Typical severity: **error** · Category: **generator** A relative generator omits the sibling column it should offset. ## GEN-RELATIVE-RANGE — Relative offset range is invalid [Section titled “GEN-RELATIVE-RANGE — Relative offset range is invalid”](#GEN-RELATIVE-RANGE) Typical severity: **error** · Category: **generator** A relative generator has negative, inverted, or overflowing offset bounds. ## GEN-RELATIVE-UNKNOWN-SOURCE — Relative source column is unknown [Section titled “GEN-RELATIVE-UNKNOWN-SOURCE — Relative source column is unknown”](#GEN-RELATIVE-UNKNOWN-SOURCE) Typical severity: **error** · Category: **generator** A relative generator names a sibling column that does not exist. ## GEN-RENDER-COPY-DEFAULT — PostgreSQL COPY cannot render DEFAULT [Section titled “GEN-RENDER-COPY-DEFAULT — PostgreSQL COPY cannot render DEFAULT”](#GEN-RENDER-COPY-DEFAULT) Typical severity: **error** · Category: **rendering** A COPY row requests DEFAULT for a column without a database default, identity, or generated expression. ## GEN-RENDER-IO — SQL rendering failed [Section titled “GEN-RENDER-IO — SQL rendering failed”](#GEN-RENDER-IO) Typical severity: **error** · Category: **rendering** The renderer could not write or format generated SQL. ## GEN-RENDER-WARNING — SQL renderer reported a warning [Section titled “GEN-RENDER-WARNING — SQL renderer reported a warning”](#GEN-RENDER-WARNING) Typical severity: **warning** · Category: **rendering** The renderer encountered a non-fatal condition without a more specific diagnostic code. ## GEN-REQUEST-OUTPUT — Generation output request is invalid [Section titled “GEN-REQUEST-OUTPUT — Generation output request is invalid”](#GEN-REQUEST-OUTPUT) Typical severity: **error** · Category: **runtime** The requested output or resolved-model destination is unavailable or conflicting. ## GEN-REQUEST-SOURCE — Generation source is missing [Section titled “GEN-REQUEST-SOURCE — Generation source is missing”](#GEN-REQUEST-SOURCE) Typical severity: **error** · Category: **runtime** Neither a source dump nor a complete model configuration was supplied. ## GEN-ROUND-MISSING-SCALE — Round scale is missing [Section titled “GEN-ROUND-MISSING-SCALE — Round scale is missing”](#GEN-ROUND-MISSING-SCALE) Typical severity: **error** · Category: **modifier** A round modifier omits the number of decimal places to retain. ## GEN-ROUND-SCALE-RANGE — Round scale is invalid [Section titled “GEN-ROUND-SCALE-RANGE — Round scale is invalid”](#GEN-ROUND-SCALE-RANGE) Typical severity: **error** · Category: **modifier** A round modifier’s requested scale falls outside the supported range. ## GEN-ROWS-CYCLE — Row-count dependency cycle [Section titled “GEN-ROWS-CYCLE — Row-count dependency cycle”](#GEN-ROWS-CYCLE) Typical severity: **error** · Category: **selection** Parent-derived table row-count rules contain a dependency cycle. ## GEN-ROWS-OBSERVED-MISSING — Observed row count is unavailable [Section titled “GEN-ROWS-OBSERVED-MISSING — Observed row count is unavailable”](#GEN-ROWS-OBSERVED-MISSING) Typical severity: **error** · Category: **selection** An observed row-count rule has no attached profile count to resolve. ## GEN-SCHEMA-MISMATCH — Override schema assertion does not match [Section titled “GEN-SCHEMA-MISMATCH — Override schema assertion does not match”](#GEN-SCHEMA-MISMATCH) Typical severity: **error** · Category: **config** An override’s schema name or create statement disagrees with the base model. ## GEN-SEQUENCE-OUT-OF-RANGE — Sequence start is out of range [Section titled “GEN-SEQUENCE-OUT-OF-RANGE — Sequence start is out of range”](#GEN-SEQUENCE-OUT-OF-RANGE) Typical severity: **error** · Category: **generator** A sequence generator’s start value does not fit the declared integer width of its column. ## GEN-SEQUENCE-ZERO-STEP — Sequence step is zero [Section titled “GEN-SEQUENCE-ZERO-STEP — Sequence step is zero”](#GEN-SEQUENCE-ZERO-STEP) Typical severity: **error** · Category: **generator** A sequence generator cannot advance because its configured step is zero. ## GEN-SLUG-MAX-LENGTH — Slug max length is invalid [Section titled “GEN-SLUG-MAX-LENGTH — Slug max length is invalid”](#GEN-SLUG-MAX-LENGTH) Typical severity: **error** · Category: **generator** A slug generator’s max\_length is zero or not a positive integer. ## GEN-SLUG-MISSING-SOURCE — Slug source is missing [Section titled “GEN-SLUG-MISSING-SOURCE — Slug source is missing”](#GEN-SLUG-MISSING-SOURCE) Typical severity: **error** · Category: **generator** A slug generator omits the sibling column it should derive from. ## GEN-SLUG-UNKNOWN-FIELD — Slug source column is unknown [Section titled “GEN-SLUG-UNKNOWN-FIELD — Slug source column is unknown”](#GEN-SLUG-UNKNOWN-FIELD) Typical severity: **error** · Category: **generator** A slug generator names a sibling column that does not exist. ## GEN-SOFT-DELETE-COLUMN-MISSING — Soft-delete column is missing [Section titled “GEN-SOFT-DELETE-COLUMN-MISSING — Soft-delete column is missing”](#GEN-SOFT-DELETE-COLUMN-MISSING) Typical severity: **error** · Category: **planner** A soft-delete role names a column that does not exist. ## GEN-SOFT-DELETE-NULLABILITY — Soft-delete nullability is impossible [Section titled “GEN-SOFT-DELETE-NULLABILITY — Soft-delete nullability is impossible”](#GEN-SOFT-DELETE-NULLABILITY) Typical severity: **error** · Category: **planner** Non-deleted rows are possible but the deleted timestamp column is not nullable. ## GEN-SOFT-DELETE-RANGE — Soft-delete timestamp range is invalid [Section titled “GEN-SOFT-DELETE-RANGE — Soft-delete timestamp range is invalid”](#GEN-SOFT-DELETE-RANGE) Typical severity: **error** · Category: **planner** A deletion timestamp range is missing, unparsable, or inverted. ## GEN-SOURCE-FINGERPRINT — Source fingerprint does not match [Section titled “GEN-SOURCE-FINGERPRINT — Source fingerprint does not match”](#GEN-SOURCE-FINGERPRINT) Typical severity: **variable** · Category: **config** An overrides document’s source fingerprint differs from the base model under the configured policy. ## GEN-SOURCE-IO — Source dump could not be profiled [Section titled “GEN-SOURCE-IO — Source dump could not be profiled”](#GEN-SOURCE-IO) Typical severity: **error** · Category: **runtime** Reading or profiling the source SQL dump failed. ## GEN-SOURCE-VALUES — Generated output replays literal values [Section titled “GEN-SOURCE-VALUES — Generated output replays literal values”](#GEN-SOURCE-VALUES) Typical severity: **advisory** · Category: **privacy** One or more rules replay literal values and should be reviewed before sharing generated output. ## GEN-STRING-LENGTH-RANGE — String length range is invalid [Section titled “GEN-STRING-LENGTH-RANGE — String length range is invalid”](#GEN-STRING-LENGTH-RANGE) Typical severity: **error** · Category: **generator** A string generator has incompatible minimum and maximum lengths. ## GEN-TABLE-COUNT-CONFLICT — Per-table count controls conflict [Section titled “GEN-TABLE-COUNT-CONFLICT — Per-table count controls conflict”](#GEN-TABLE-COUNT-CONFLICT) Typical severity: **error** · Category: **selection** The same table matches both an absolute and a scaled row-count override. ## GEN-TABLE-SCALE-INVALID — Per-table scale is invalid [Section titled “GEN-TABLE-SCALE-INVALID — Per-table scale is invalid”](#GEN-TABLE-SCALE-INVALID) Typical severity: **error** · Category: **selection** A table scale is negative, non-finite, or otherwise unsupported. ## GEN-TEMPLATE-INVALID-PART — Template part is invalid [Section titled “GEN-TEMPLATE-INVALID-PART — Template part is invalid”](#GEN-TEMPLATE-INVALID-PART) Typical severity: **error** · Category: **generator** A template part is neither a supported literal nor a valid sibling-column reference. ## GEN-TEMPLATE-MISSING-PARTS — Template parts are missing [Section titled “GEN-TEMPLATE-MISSING-PARTS — Template parts are missing”](#GEN-TEMPLATE-MISSING-PARTS) Typical severity: **error** · Category: **generator** A template generator omits its required parts list. ## GEN-TEMPLATE-UNKNOWN-FIELD — Template field is unknown [Section titled “GEN-TEMPLATE-UNKNOWN-FIELD — Template field is unknown”](#GEN-TEMPLATE-UNKNOWN-FIELD) Typical severity: **error** · Category: **generator** A template generator references a sibling column that does not exist. ## GEN-TENANT-COLUMN-MISSING — Tenant-family column is missing [Section titled “GEN-TENANT-COLUMN-MISSING — Tenant-family column is missing”](#GEN-TENANT-COLUMN-MISSING) Typical severity: **error** · Category: **planner** A tenant-family role names a column that does not exist. ## GEN-TENANT-KEY-UNSUPPORTED — Tenant-family key domain is unsupported [Section titled “GEN-TENANT-KEY-UNSUPPORTED — Tenant-family key domain is unsupported”](#GEN-TENANT-KEY-UNSUPPORTED) Typical severity: **error** · Category: **planner** The parent relationship does not expose a supported dense key domain. ## GEN-TENANT-PARTITION — Tenant partition is invalid [Section titled “GEN-TENANT-PARTITION — Tenant partition is invalid”](#GEN-TENANT-PARTITION) Typical severity: **error** · Category: **planner** The requested tenant partition count cannot be formed from the available parent rows. ## GEN-TENANT-RELATIONSHIP — Tenant-family relationship is invalid [Section titled “GEN-TENANT-RELATIONSHIP — Tenant-family relationship is invalid”](#GEN-TENANT-RELATIONSHIP) Typical severity: **error** · Category: **planner** A tenant-family planner omits or names an unknown parent relationship. ## GEN-TIMESTAMPS-COLUMN-MISSING — Timestamp column is missing [Section titled “GEN-TIMESTAMPS-COLUMN-MISSING — Timestamp column is missing”](#GEN-TIMESTAMPS-COLUMN-MISSING) Typical severity: **error** · Category: **planner** A temporal timestamps role names a column that does not exist. ## GEN-TIMESTAMPS-DELAY — Timestamp delay is invalid [Section titled “GEN-TIMESTAMPS-DELAY — Timestamp delay is invalid”](#GEN-TIMESTAMPS-DELAY) Typical severity: **error** · Category: **planner** A temporal delay is negative, overflowing, inverted, or uses an unknown unit. ## GEN-TIMESTAMPS-RANGE — Timestamp range is invalid [Section titled “GEN-TIMESTAMPS-RANGE — Timestamp range is invalid”](#GEN-TIMESTAMPS-RANGE) Typical severity: **error** · Category: **planner** A created timestamp range is missing, unparsable, or inverted. ## GEN-TREE-BRANCHING — Tree branching limit is invalid [Section titled “GEN-TREE-BRANCHING — Tree branching limit is invalid”](#GEN-TREE-BRANCHING) Typical severity: **error** · Category: **planner** A hierarchy tree branching limit is below one. ## GEN-TREE-COLUMN-MISSING — Tree parent column is missing [Section titled “GEN-TREE-COLUMN-MISSING — Tree parent column is missing”](#GEN-TREE-COLUMN-MISSING) Typical severity: **error** · Category: **planner** A hierarchy tree parent role names a column that does not exist. ## GEN-TREE-DEPTH — Tree depth is invalid [Section titled “GEN-TREE-DEPTH — Tree depth is invalid”](#GEN-TREE-DEPTH) Typical severity: **error** · Category: **planner** A hierarchy tree maximum depth is below one. ## GEN-TREE-REQUIRED-CYCLE — Tree root cannot be represented [Section titled “GEN-TREE-REQUIRED-CYCLE — Tree root cannot be represented”](#GEN-TREE-REQUIRED-CYCLE) Typical severity: **error** · Category: **planner** The self-reference column is non-nullable, so the hierarchy cannot create a root row. ## GEN-TREE-ROOT-RATIO — Tree root ratio is invalid [Section titled “GEN-TREE-ROOT-RATIO — Tree root ratio is invalid”](#GEN-TREE-ROOT-RATIO) Typical severity: **error** · Category: **planner** A hierarchy tree root ratio falls outside the supported zero-to-one range. ## GEN-TRUNCATE-MISSING-MAX-LENGTH — Truncate length is missing [Section titled “GEN-TRUNCATE-MISSING-MAX-LENGTH — Truncate length is missing”](#GEN-TRUNCATE-MISSING-MAX-LENGTH) Typical severity: **error** · Category: **modifier** A truncate modifier omits its required maximum length. ## GEN-UNIQUE-INVALID-ON-EXHAUSTION — Unique exhaustion policy is invalid [Section titled “GEN-UNIQUE-INVALID-ON-EXHAUSTION — Unique exhaustion policy is invalid”](#GEN-UNIQUE-INVALID-ON-EXHAUSTION) Typical severity: **error** · Category: **modifier** A unique modifier names an unsupported behavior for exhausted retries. ## GEN-UNIQUE-WIDEN-UNSUPPORTED — Unique widening is unsupported [Section titled “GEN-UNIQUE-WIDEN-UNSUPPORTED — Unique widening is unsupported”](#GEN-UNIQUE-WIDEN-UNSUPPORTED) Typical severity: **error** · Category: **modifier** The target type has no safe widening strategy for unique-value exhaustion. ## GEN-VERIFY-FAILED — Generated output failed verification [Section titled “GEN-VERIFY-FAILED — Generated output failed verification”](#GEN-VERIFY-FAILED) Typical severity: **error** · Category: **verification** One or more exact verification checks failed and the destination was not published. ## GEN-VERIFY-IO — Verification input could not be read [Section titled “GEN-VERIFY-IO — Verification input could not be read”](#GEN-VERIFY-IO) Typical severity: **error** · Category: **verification** The verifier could not read the staged generated output. ## GEN-VERIFY-MODE — Verification mode is invalid [Section titled “GEN-VERIFY-MODE — Verification mode is invalid”](#GEN-VERIFY-MODE) Typical severity: **error** · Category: **verification** Verification was requested with an incompatible generation mode. ## GEN-VERIFY-NO-FILE — Verification requires a file [Section titled “GEN-VERIFY-NO-FILE — Verification requires a file”](#GEN-VERIFY-NO-FILE) Typical severity: **error** · Category: **verification** Verification cannot audit output that has no real file destination. ## GEN-VERIFY-NOTCHECKED — Some capabilities were not checked [Section titled “GEN-VERIFY-NOTCHECKED — Some capabilities were not checked”](#GEN-VERIFY-NOTCHECKED) Typical severity: **warning** · Category: **verification** Verification passed, but one or more capabilities had no exact audit predicate. ## GEN-VERIFY-PARSE — Generated SQL could not be parsed for verification [Section titled “GEN-VERIFY-PARSE — Generated SQL could not be parsed for verification”](#GEN-VERIFY-PARSE) Typical severity: **error** · Category: **verification** The verifier could not parse the staged generated SQL. ## GEN-VERIFY-PARTIAL-PUBLISH — Verified outputs were only partially published [Section titled “GEN-VERIFY-PARTIAL-PUBLISH — Verified outputs were only partially published”](#GEN-VERIFY-PARTIAL-PUBLISH) Typical severity: **error** · Category: **verification** Publishing a verified output set failed after one destination had already changed. ## GEN-VERIFY-STAGE — Verification staging failed [Section titled “GEN-VERIFY-STAGE — Verification staging failed”](#GEN-VERIFY-STAGE) Typical severity: **error** · Category: **verification** Creating or writing protected temporary output for verification failed. ## GEN-WEIGHTED-CHOICE-ALL-ZERO — Weighted-choice weights are all zero [Section titled “GEN-WEIGHTED-CHOICE-ALL-ZERO — Weighted-choice weights are all zero”](#GEN-WEIGHTED-CHOICE-ALL-ZERO) Typical severity: **error** · Category: **generator** A weighted choice has no entry with positive probability. ## GEN-WEIGHTED-CHOICE-EMPTY — Weighted choice is empty [Section titled “GEN-WEIGHTED-CHOICE-EMPTY — Weighted choice is empty”](#GEN-WEIGHTED-CHOICE-EMPTY) Typical severity: **error** · Category: **generator** A weighted-choice generator has no choices from which to draw. ## GEN-WEIGHTED-CHOICE-INVALID-ENTRY — Weighted-choice entry is invalid [Section titled “GEN-WEIGHTED-CHOICE-INVALID-ENTRY — Weighted-choice entry is invalid”](#GEN-WEIGHTED-CHOICE-INVALID-ENTRY) Typical severity: **error** · Category: **generator** A weighted choice contains a malformed entry. ## GEN-WEIGHTED-CHOICE-INVALID-VALUE — Weighted-choice value is incompatible [Section titled “GEN-WEIGHTED-CHOICE-INVALID-VALUE — Weighted-choice value is incompatible”](#GEN-WEIGHTED-CHOICE-INVALID-VALUE) Typical severity: **error** · Category: **generator** A weighted choice value cannot be represented by the target column type. ## GEN-WEIGHTED-CHOICE-INVALID-WEIGHT — Weighted-choice weight is invalid [Section titled “GEN-WEIGHTED-CHOICE-INVALID-WEIGHT — Weighted-choice weight is invalid”](#GEN-WEIGHTED-CHOICE-INVALID-WEIGHT) Typical severity: **error** · Category: **generator** A weighted choice contains a negative, NaN, or infinite weight. ## GEN-WEIGHTED-CHOICE-MISSING-CHOICES — Weighted choices are missing [Section titled “GEN-WEIGHTED-CHOICE-MISSING-CHOICES — Weighted choices are missing”](#GEN-WEIGHTED-CHOICE-MISSING-CHOICES) Typical severity: **error** · Category: **generator** A weighted-choice generator omits its required choices. # Generator reference > Choose and configure base value producers for generated columns A generator owns one column and produces its base value. The compiler checks the generator’s accepted SQL families and validates its arguments. [Modifiers](/commands/generate/modifiers/) run afterward in the order declared. ## Column evaluation order [Section titled “Column evaluation order”](#column-evaluation-order) When a generator reads sibling columns on the same row — `copy`, `template`, `slug`, and `before`/`after` do — the engine evaluates each column after the columns it reads, following the read dependencies rather than the order columns are declared. A derived column may therefore appear anywhere in the schema, before or after its source. A read whose source has not been produced yet (or a generator reading its own column) sees no value; a set of columns that read each other in a cycle is rejected at compile time as [`GEN-COLUMN-CYCLE`](/commands/generate/diagnostics/#GEN-COLUMN-CYCLE). Columns that read nothing keep their declared schema order. ```yaml columns: email: generator: { kind: internet.email } modifiers: - { kind: case, mode: lower } - { kind: unique, max_attempts: 20 } ``` ## Choose by intent [Section titled “Choose by intent”](#choose-by-intent) | Need | Start with | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | IDs and deterministic counters | [`sequence`](/commands/generate/generators/core/#sequence), [`uuid`](/commands/generate/generators/core/#uuid), [`monotonic`](/commands/generate/generators/observed-statistical/#monotonic) | | Bounded random primitives | [`integer`](/commands/generate/generators/core/#integer), [`decimal`](/commands/generate/generators/core/#decimal), [`boolean`](/commands/generate/generators/core/#boolean), [`string`](/commands/generate/generators/core/#string) | | Realistic domain-shaped text | [Semantic generators](/commands/generate/generators/semantic/) | | Passwords, tokens, and secrets | [Credential generators](/commands/generate/generators/credentials/) | | Preserve an observed distribution | [Observed and statistical generators](/commands/generate/generators/observed-statistical/) | | Foreign-key values | [Relationship generators](/commands/generate/generators/relationships/) or a planner | | Derive from another column | [`copy`](/commands/generate/generators/core/#copy), [`template`](/commands/generate/generators/core/#template), [`before`/`after`](/commands/generate/generators/semantic/#relative-times) | | Let the database fill the value | [`database_default`](/commands/generate/generators/core/#database_default) | ## Common rules [Section titled “Common rules”](#common-rules) * `kind` is required and must name a registered generator or alias. * Unknown arguments are rejected. Required arguments are checked by both the published schema and the compiler. * A generator can attach only to a compatible `schema.columns[].family`. * Cross-column generators may read only columns that exist and are ordered before them without forming a dependency cycle. * Every generator is deterministic under a fixed seed unless its documentation explicitly says otherwise. All current built-ins are deterministic. * A generator that embeds literal choices can trigger the non-fatal `GEN-SOURCE-VALUES` advisory. See [Privacy](/commands/generate/privacy-verification/). ## Catalog [Section titled “Catalog”](#catalog) * [Core](/commands/generate/generators/core/) — primitives, categorical values, templates, IDs, and defaults * [Semantic](/commands/generate/generators/semantic/) — names, addresses, internet, commerce, temporal, and file shapes * [Credentials](/commands/generate/generators/credentials/) — synthetic-only secrets and placeholders * [Observed and statistical](/commands/generate/generators/observed-statistical/) — samples, histograms, distributions, and monotonic values * [Relationships](/commands/generate/generators/relationships/) — ordinary and composite foreign keys For correlated multi-column or cross-table invariants, use a [planner](/commands/generate/planners/) instead of independent generators. # Core generators > Primitive, categorical, structural, and cross-column generators Core generators cover common SQL value families without domain-specific semantics. All stream row by row and are reproducible under a fixed seed. ## `constant` [Section titled “constant”](#constant) Alias: `const`. Argument: optional `value`, default SQL `NULL`. Accepts every family. Use it for fixture flags or deliberate fixed labels. Literal values are reported by `GEN-SOURCE-VALUES`. ```yaml generator: { kind: constant, value: active } ``` Constants are coerced to the target column family during compilation. Invalid boolean, integer, decimal, and other typed values fail with `GEN-CONSTANT-INVALID-VALUE`; values for text columns render as text. ## `null` [Section titled “null”](#null) No arguments; accepts every family. The target column must be nullable. Prefer `null_rate` when only some rows should be null. ## `sequence` [Section titled “sequence”](#sequence) | Argument | Default | Constraint | | -------- | ------- | ---------------- | | `start` | `0` | Integer | | `step` | `1` | Non-zero integer | Accepts `integer` and `big_integer`. It is deterministic, unique when the step does not overflow, and usable as a random-access parent key for relationships. ```yaml generator: { kind: sequence, start: 1, step: 1 } ``` ## `copy` [Section titled “copy”](#copy) Required `source` names a sibling column. The source must exist and produce a family compatible with the target; it may be declared anywhere, since evaluation follows the [read dependency](/commands/generate/generators/#column-evaluation-order). ```yaml columns: normalized_email: generator: { kind: copy, source: email } modifiers: [{ kind: case, mode: lower }] ``` ## `template` [Section titled “template”](#template) Required `parts` is a list of literal strings and `{ field: column_name }` references. Accepts text. It concatenates values; it is not an expression or format language. ```yaml generator: kind: template parts: ["INV-", { field: id }, "-", { field: region }] ``` Every referenced field must exist; each is evaluated before this column automatically, wherever it is declared (see [column evaluation order](/commands/generate/generators/#column-evaluation-order)). ## `slug` [Section titled “slug”](#slug) Required `source` names a sibling column; accepts text. Derives a URL slug from that column’s value on the same row: lowercase, every run of non-ASCII-alphanumeric characters collapses to a single `-`, and leading and trailing dashes are trimmed. Optional `max_length` (a positive integer) truncates the result, trimming any dash left dangling at the cut. Non-ASCII-alphanumeric characters act as separators; the generator does not transliterate accents or other scripts. The source may be declared anywhere, as evaluation follows the [read dependency](/commands/generate/generators/#column-evaluation-order); add a `unique` modifier when the slug column is constrained unique. ```yaml columns: title: generator: { kind: string, min_length: 12, max_length: 40 } slug: generator: { kind: slug, source: title, max_length: 60 } modifiers: [{ kind: unique }] ``` ## `pattern` [Section titled “pattern”](#pattern) Required `mask`; accepts text. | Character | Output | | ------------- | ---------------------- | | `#` | decimal digit | | `?` | uppercase ASCII letter | | `@` | lowercase ASCII letter | | `*` | ASCII letter or digit | | anything else | literal character | ```yaml generator: { kind: pattern, mask: "???-####-**" } ``` ## `database_default` [Section titled “database\_default”](#database_default) No arguments; accepts every family. Emits SQL `DEFAULT` and lets the database fill the value. PostgreSQL COPY omits such a column, which requires an actual schema default, identity, or generated expression. A preserved source default can contain literals and therefore participates in `GEN-SOURCE-VALUES`. ## `json_value` [Section titled “json\_value”](#json_value) Optional `value`, default `{}`; accepts `json`. The value must serialize to valid JSON. ```yaml generator: { kind: json_value, value: { synthetic: true, tags: [] } } ``` ## `integer` [Section titled “integer”](#integer) `min` defaults to `0`; `max` defaults to `1000`. Both bounds are inclusive and `min` must not exceed `max`. Accepts `integer` and `big_integer`. ## `decimal` [Section titled “decimal”](#decimal) `min` defaults to `0`, `max` to `1000`, and `scale` to `2`. Scale is `0..18`, the bounds must be ordered, and the target decimal type must represent them. ```yaml generator: { kind: decimal, min: -25.00, max: 250.00, scale: 2 } ``` ## `boolean` [Section titled “boolean”](#boolean) `probability` is the chance of `true`, defaults to `0.5`, and must be within `0..1`. Accepts `boolean`. ## `string` [Section titled “string”](#string) `min_length` defaults to `8`; `max_length` defaults to the minimum. Lengths are non-negative and ordered. Output is ASCII alphanumeric text. ## `bytes` [Section titled “bytes”](#bytes) `min_length` defaults to `16`; `max_length` defaults to the minimum. Accepts `bytes` and emits random bytes rather than encoded text. ## `uuid` [Section titled “uuid”](#uuid) No arguments. Emits RFC 4122 version 4 values into `uuid` or `text` columns. UUID parent keys can be regenerated by row index for relationship selection. ## `choice` [Section titled “choice”](#choice) Required non-empty `values`; uniform selection. Accepts every family after family-aware coercion. ```yaml generator: { kind: choice, values: [draft, active, archived] } ``` Use it for a small uniform vocabulary. Values are literal and therefore trigger the source-values advisory. ## `weighted_choice` [Section titled “weighted\_choice”](#weighted_choice) Required non-empty `choices`, each containing `value` and a finite non-negative `weight`. At least one weight must be positive. ```yaml generator: kind: weighted_choice choices: - { value: free, weight: 0.75 } - { value: pro, weight: 0.20 } - { value: enterprise, weight: 0.05 } ``` Use it for explicit categorical distributions. It replays literal choices and is always included in `GEN-SOURCE-VALUES`. # Credential generators > Synthetic-only tokens, keys, secrets, and password-shaped placeholders Credential generators deliberately avoid observed source values. Dump inference uses these kinds for high-confidence password, token, API-key, secret, private-key, and salt columns. Note These values are test fixtures, not production credentials. They are generated for shape and recognizability, not cryptographic security or password hashing. ## Catalog [Section titled “Catalog”](#catalog) | Kind | Arguments and defaults | Output | | -------------------------- | ---------------------------------------------------- | ---------------------------------------------- | | `credential.token` | `length=32`, `alphabet=alphanumeric`, `prefix=""` | Random credential-shaped text | | `credential.api_key` | `length=32`, `alphabet=alphanumeric`, `prefix="sk_"` | API-key-shaped text | | `credential.secret` | `length=48`, `alphabet=alphanumeric`, `prefix=""` | Longer random secret-shaped text | | `credential.password_hash` | none | `$synthetic$` plus 64 lowercase hex characters | | `credential.placeholder` | none | `SYNTHETIC_PLACEHOLDER_NOT_A_REAL_CREDENTIAL` | The configurable alphabets are `alphanumeric`, `hex`, `numeric`, `alpha`, and `url_safe`. Length must be positive and the prefix is literal. ```yaml columns: api_key: generator: kind: credential.api_key prefix: test_sk_ length: 40 password_hash: generator: { kind: credential.password_hash } private_key: generator: { kind: credential.placeholder } ``` ## Inference guarantee and explicit overrides [Section titled “Inference guarantee and explicit overrides”](#inference-guarantee-and-explicit-overrides) Inference never chooses `constant`, `weighted_choice`, or `observed_sample` for a recognized credential column. It may preserve safe shape facts such as length, alphabet class, or prefix while discarding content. An explicit model is authoritative: it can deliberately assign a literal-based generator to a credential-shaped column. That choice is allowed and is reported through `GEN-SOURCE-VALUES`; sql-splitter does not silently replace it. # Observed and statistical generators > Replay bounded evidence through samples, histograms, distributions, and monotonic values These generators preserve a distribution or progression. Inferred models use them when observed evidence is stronger than a semantic match. ## `observed_sample` [Section titled “observed\_sample”](#observed_sample) Required non-empty `values`; accepts every family. Values may be bare literals or weighted entries. ```yaml generator: kind: observed_sample values: - { value: pending, weight: 73 } - { value: complete, weight: 27 } ``` Weights must be finite and non-negative, with at least one positive entry. Because this generator replays literals, every use participates in the `GEN-SOURCE-VALUES` advisory. Prefer a semantic or statistical generator when literal replay is unacceptable. ## `histogram` [Section titled “histogram”](#histogram) Required non-empty `bins`; optional `scale=0` (0..18). Accepts integer, big-integer, and decimal families. ```yaml generator: kind: histogram scale: 0 bins: - { min: 0, max: 9, count: 50 } - { min: 10, max: 49, count: 35 } - { min: 50, max: 100, count: 15 } ``` Bins must be sorted, non-overlapping, finite, and have ordered bounds. Counts are frequency weights and cannot all be zero. A bin is selected by count, then a value is drawn uniformly inside it. ## `normal` [Section titled “normal”](#normal) Required `mean` and `std`; optional `min`, `max`, and `scale=0` (0..18). `std` must be finite and non-negative. Optional bounds clamp the Gaussian draw and must be ordered. ```yaml generator: { kind: normal, mean: 72, std: 12, min: 18, max: 110 } ``` Accepts integer, big-integer, and decimal families. Integer output rounds the draw; decimal output uses the configured scale. ## `lognormal` [Section titled “lognormal”](#lognormal) Required `mu` and `sigma`; optional `min`, `max`, and `scale=0` (0..18). It draws `exp(Normal(mu, sigma))`, making it suitable for positive, right-skewed sizes, durations, and amounts. `sigma` must be finite and non-negative. ```yaml generator: { kind: lognormal, mu: 4.5, sigma: 0.8, min: 1, max: 10000 } ``` ## `monotonic` [Section titled “monotonic”](#monotonic) `start=0`, `step=1`. Accepts integer and big-integer families. The value is `start + row_index * step`, so it is stable, ordered, and independent of batch boundaries. Step must be non-negative; arithmetic overflow fails generation. Use `monotonic` when a nondecreasing metric is intended. Use `sequence` when a non-zero signed step and random-access relationship key recipe are required. ## Choosing among them [Section titled “Choosing among them”](#choosing-among-them) | Evidence or intent | Generator | | ------------------------------------------------------ | ----------------- | | Small categorical vocabulary whose exact labels matter | `observed_sample` | | Irregular numeric density represented by buckets | `histogram` | | Symmetric bell-shaped numeric data | `normal` | | Positive right-skewed data | `lognormal` | | Ordered counter tied to row position | `monotonic` | Statistical resemblance is approximate. Bounded profiling does not retain a complete population, and generation does not promise exact bucket counts unless a planner explicitly defines an exact invariant. # Relationship generators > Assign ordinary and composite foreign-key values from parent key domains Relationship generators are compiler markers. The generation engine assigns their values from a named table-level relationship; the generator object itself does not invent an independent value. ## `relation.foreign_key` [Section titled “relation.foreign\_key”](#relationforeign_key) Alias: `foreign_key`. Use for a one-column relationship. ## `relation.composite_key` [Section titled “relation.composite\_key”](#relationcomposite_key) Alias: `composite_key`. Use for all columns of a multi-column relationship. The relationship is selected atomically, so its components refer to the same parent row. Both accept integer, big-integer, text, UUID, and other key families. | Argument | Default | Meaning | | -------------- | ------------------------------- | -------------------------------------------------- | | `relationship` | inferred when unambiguous | Name from the table’s `relationships` list | | `distribution` | relationship/default policy | `uniform`, `sequential`, `weighted`, or `observed` | | `null_rate` | relationship/nullability policy | Probability of a null reference, within `0..1` | ```yaml tables: orders: relationships: - name: orders_customer columns: [customer_id] references: { table: customers, columns: [id] } distribution: uniform columns: customer_id: generator: kind: relation.foreign_key relationship: orders_customer ``` The local columns and referenced columns must exist and have compatible families. A non-nullable relationship cannot generate null references. ## Parent key constraints [Section titled “Parent key constraints”](#parent-key-constraints) The engine must reproduce a parent key by parent row index without retaining the whole parent table. Supported key domains are: * a bare integer identity/primary key; * `sequence`; * UUID. A parent key owned by another stateful or non-random-access generator emits `GEN-KEY-DOMAIN-UNSUPPORTED` at generation time. ## When a planner is required [Section titled “When a planner is required”](#when-a-planner-is-required) Use a planner when correctness depends on more than selecting one ordinary parent row: * `hierarchy.tree` for bounded self-referential trees; * `relation.junction_pair` for unique left/right pairs; * `relation.polymorphic_pair` for atomic type/id pairs; * `relation.tenant_family` for tenant-consistent parent selection; * `commerce.order_family` for parent/line-item accounting equations. The `observed` relationship distribution currently uses a deterministic fixed root. Do not rely on it for fresh run-to-run variation. # Semantic generators > Realistic names, addresses, commerce, network, file, and time values Semantic generators produce domain-shaped synthetic values without retaining a source value vocabulary. The built-in catalog currently uses the English locale. Apply `unique` explicitly where the schema requires uniqueness; most semantic domains are not unique by construction. ## No-argument text catalog [Section titled “No-argument text catalog”](#no-argument-text-catalog) These generators accept `text`, take no arguments, and stream deterministic catalog draws: | Domain | Kinds | | -------- | ------------------------------------------------------------------------------------------------------------ | | Person | `person.first_name`, `person.last_name`, `person.full_name`, `person.username`, `person.title` | | Internet | `internet.email`, `internet.domain`, `internet.url`, `internet.ipv4`, `internet.ipv6`, `internet.user_agent` | | Phone | `phone.number`, `phone.country_code` | | Company | `company.name`, `company.department`, `company.job_title` | | Address | `address.line1`, `address.line2`, `address.city`, `address.region`, `address.postcode`, `address.country` | | Commerce | `commerce.currency` | | Text | `text.word`, `text.sentence`, `text.paragraph`, `text.slug` | | File | `file.name`, `file.extension`, `file.mime_type` | | Network | `network.mac` | ```yaml columns: name: generator: { kind: person.full_name } email: generator: { kind: internet.email } modifiers: [{ kind: unique }] ``` Independent generators do not coordinate their semantics. For example, `file.name`, `file.extension`, and `file.mime_type` chosen independently need not agree; use the `file.metadata` planner for a coherent tuple. ## Coordinates [Section titled “Coordinates”](#coordinates) `address.latitude` and `address.longitude` accept `decimal` or `text`, take no arguments, and draw from the full global ranges at six decimal places. Use the `geo.coordinate_pair` planner when latitude and longitude need shared custom bounds or coordinated ownership. ## Commerce [Section titled “Commerce”](#commerce) | Kind | Arguments | Accepts | Notes | | ----------------------- | ------------------------------ | --------------------- | ------------------------------- | | `commerce.product_name` | none | text | Two capitalized synthetic words | | `commerce.sku` | none | text | `AAA-###-####` shape | | `commerce.money` | `min=0`, `max=1000`, `scale=2` | decimal, text | Ordered bounds; scale `0..18` | | `commerce.quantity` | `min=1`, `max=100` | integer, big\_integer | Inclusive non-negative range | Independent money columns do not preserve accounting equations. Use `commerce.order_family` for coordinated line items, subtotal, tax, discounts, shipping, and total. ## Files, network, and duration [Section titled “Files, network, and duration”](#files-network-and-duration) | Kind | Arguments | Accepts | Notes | | -------------- | ----------------------- | --------------------- | -------------------- | | `file.size` | `min=1`, `max=10000000` | integer, big\_integer | Size in bytes | | `network.port` | `min=1`, `max=65535` | integer, big\_integer | Inclusive port range | | `duration` | `min=0`, `max=86400` | integer, big\_integer | Duration in seconds | Every minimum must not exceed its maximum. These generators produce one value; they do not coordinate with file-name or temporal endpoint columns. ## Identifiers [Section titled “Identifiers”](#identifiers) ### `identifier.token` [Section titled “identifier.token”](#identifiertoken) `length=32`, `alphabet=alphanumeric`, `prefix=""`. Alphabets are `alphanumeric`, `hex`, `numeric`, `alpha`, and `url_safe`. ### `identifier.nanoid` [Section titled “identifier.nanoid”](#identifiernanoid) `length=21`, `alphabet=url_safe`, `prefix=""`. This is NanoID-shaped output; choose an explicit length if another system expects a fixed width. ### `identifier.ulid` [Section titled “identifier.ulid”](#identifierulid) No arguments. Emits a 26-character Crockford-base32 ULID-shaped identifier. ### `identifier.hash` [Section titled “identifier.hash”](#identifierhash) `length=64`. Emits lowercase hexadecimal text. This is random hash-shaped data, not a digest of another generated column. ## Dates and times [Section titled “Dates and times”](#dates-and-times) | Kind | Arguments | Range | Accepts | | ---------- | --------- | --------------------------------- | ---------------- | | `date` | none | `1970-01-01` through `2035-12-31` | date\_time, text | | `time` | none | `00:00:00` through `23:59:59` | date\_time, text | | `datetime` | none | `1970-01-01` through `2035-12-31` | date\_time, text | The bounds of these single-column generators are fixed. Use a temporal planner for configurable ranges and ordering across several columns. ## Relative times [Section titled “Relative times”](#relative-times) `before` and `after` read a required sibling `source` column. They accept `date_time` or `text` and use `min_seconds=1`, `max_seconds=86400` by default. ```yaml columns: created_at: generator: { kind: datetime } expires_at: generator: kind: after source: created_at min_seconds: 3600 max_seconds: 86400 ``` The source must exist and contain a supported timestamp format at runtime. It may be declared anywhere — evaluation follows the [read dependency](/commands/generate/generators/#column-evaluation-order), so the source is produced first — but the two columns must not read each other in a cycle. Bounds must be ordered and fit the supported timestamp arithmetic range. # Profiling and inference > How a source dump becomes an explicit synthetic model When a source dump is provided, sql-splitter reads it once to build neutral, bounded evidence, then resolves that evidence into an explicit model. Profiling records what was observed; inference decides how to generate it. ## Profiling depths [Section titled “Profiling depths”](#profiling-depths) The CLI exposes `basic` and `full` and defaults to `basic`. | Capability | Basic | Full | | ----------------------------------------------------------------------------- | ----- | ---- | | Schema and exact row counts | yes | yes | | Exact null counts when schema precedes data; otherwise bounded evidence | yes | yes | | Distinct estimate, numeric summary, string shape, top values, bounded samples | yes | yes | | Declared-FK coverage and same-table temporal pair evidence | no | yes | Both modes scan the complete input. “Bounded” means retained evidence has a fixed capacity; it does not mean the profiler stops reading after N rows. The default per-column budget retains at most 1,000 sample values, 32 heavy hitters, 32 histogram bins, fixed-size distinct-count registers, and bounded prefix/suffix and timestamp strings. A retained individual sample is truncated to a safety ceiling. `--profile-sample N` replaces only the sample-value capacity; the other bounds stay fixed. If data arrives before its table DDL, the profiler buffers a bounded replay sample and keeps the row count exact. `GEN-PROFILE-SCHEMA-LATE` reports when the retained values cannot cover every early row. In that case, per-column counts, null rates, and value evidence apply only to the retained rows. Rows that cannot be decoded still count toward the exact table total and emit `GEN-PROFILE-DECODE-SKIPPED` once per affected table. ## Inference precedence [Section titled “Inference precedence”](#inference-precedence) Each heuristic proposes a candidate with a precedence class and confidence. A higher precedence class always beats a lower one; confidence only breaks ties inside a class. 1. explicit YAML override; 2. schema constraints such as identity and declared defaults; 3. credential guard; 4. declared relationship or planner; 5. strong semantic name plus value shape; 6. observed distribution; 7. type fallback. The winning generator, confidence, and reason are stored in the inference decision. Rejected candidates are recorded without their literal values. Use `--explain` to inspect this information. ```bash sql-splitter generate source.sql --dry-run --explain ``` ## What gets inferred [Section titled “What gets inferred”](#what-gets-inferred) * Schema constraints can select `database_default`, an identity sequence, or a relationship owner. * Credential-shaped columns select synthetic-only `credential.*` generators. * Strong names and compatible shapes select semantic generators such as `internet.email` or `person.full_name`. * Low-cardinality evidence can become `weighted_choice`; high-cardinality samples can become `observed_sample`; numeric evidence can select a statistical generator. * Every table absent from value evidence still receives a type-based complete fallback model. * Observed row counts are frozen into the model, and automatic inference is disabled before an emitted model is written. ## Keys and referential integrity [Section titled “Keys and referential integrity”](#keys-and-referential-integrity) Inference makes a few structural choices so a profiled model generates data that satisfies its own keys and foreign keys. These are driven entirely by the shape of the primary keys and foreign keys, not by table or column names, so they apply to any schema: * **Junction pivots.** A table whose primary key contains two or more single-column foreign keys is a many-to-many junction, so a `relation.junction_pair` planner is inserted over two of those foreign keys. The generated `(left, right)` pairs are distinct, which keeps the whole composite key distinct even when it also carries a discriminator column (the polymorphic pivot shape). * **Other composite keys.** When a composite primary key has fewer than two foreign keys but includes a plain (non-foreign-key) integer column, that column is sequenced so every row’s key tuple is distinct. * **Referenced text keys.** A text primary key that another table’s foreign key points at must be reproducible at generation time, so it is given a `uuid` (a text key that regenerates per row) instead of a plain random string. * **Unique columns.** A single-column unique column that inference would otherwise fill with a bounded or categorical generator (a bounded integer, `observed_sample`, `choice`, `histogram`, …) is given a high-cardinality generator instead — a `sequence` for an integer, a `uuid` for text — so it can supply a distinct value for every row without exhausting its uniqueness budget. This also stops replaying source literals on those columns. ## Planner nomination [Section titled “Planner nomination”](#planner-nomination) Beyond the structural junction pivots above, inference can recognize that a table has the columns needed by a *name/shape* planner, but it does not silently insert that configuration. Instead it emits an informational `GEN-INFER-PLANNER-NOMINATE` diagnostic naming the exact planner. For example, compatible `created_at` and `updated_at` columns can produce: ```text info[GEN-INFER-PLANNER-NOMINATE] tables.events table `events` is a candidate for `temporal.timestamps` because it has compatible `created_at` and `updated_at` columns ``` ELI5: inference has noticed that two puzzle pieces look like they belong together. It points at the matching instruction (`temporal.timestamps`) but does not assemble them, because that would change who owns the columns and could override your intent. Review the nomination, then add the planner to the model if you want its invariant. Current nominations include `temporal.timestamps` for created/updated pairs and `geo.coordinate_pair` for latitude/longitude pairs. An ordinary foreign key is not nominated as `relation.children`: that name is a row-count rule, not a planner. ## Source-derived rules [Section titled “Source-derived rules”](#source-derived-rules) An inference decision can use a literal-bearing generator when reproducing an observed categorical distribution is the strongest match. The inference result marks that decision as source-derived; the complete generation report aggregates all literal-bearing rules into one non-fatal `GEN-SOURCE-VALUES` advisory. Use semantic generators or explicit overrides to replace those rules before sharing output outside the source data’s trust boundary. ## Limitations [Section titled “Limitations”](#limitations) * Schema-only inference applies structural facts plus name-and-type heuristics (semantic, credential, and type fallbacks), but without value evidence it cannot use observed distributions or corroborate an ambiguous name against sampled data. * Planner nominations are suggestions; no planner is inserted automatically. * Bounded samples and sketches approximate distributions and cannot guarantee a statistically exact replica. * Profiling does not currently compute a fresh source fingerprint for drift comparison. * Observed planner-specific distributions are not all threaded into planners; each planner documents its fallback behavior. # Rust library API > Drive generation through the convenience builder or staged extension API The Rust crate exposes the same pipeline as the CLI. Use `Generate::builder()` for ordinary file workflows and the staged registry/compiler/engine API for custom operators, plan inspection, or a custom row sink. ## Convenience builder [Section titled “Convenience builder”](#convenience-builder) ```rust use sql_splitter::generate::Generate; fn main() -> Result<(), Box> { let report = Generate::builder() .config("model.yaml") .output("synthetic.sql") .seed(42) .run()?; println!("generated {} rows", report.rows_written); println!("effective seed: {:?}", report.effective_seed); Ok(()) } ``` | Builder method | Purpose | | ------------------------------- | ------------------------------------------------ | | `.input(path)` | Profile a source dump | | `.config(path)` | Load a model or overrides document | | `.output(path)` | Select generated SQL destination | | `.emit(path)` | Write the resolved complete model | | `.input_dialect(dialect)` | Override source dialect detection | | `.profile_depth(depth)` | Select schema/basic/full library profiling depth | | `.profile_sample(n)` | Set retained sample capacity | | `.output_dialect(dialect)` | Select render dialect | | `.seed(u64)` | Override root seed | | `.table_scale(table, factor)` | Add one validated table-scale override | | `.compile(options)` | Replace grouped count/selection compile options | | `.explain(bool)` | Include inference decisions | | `.verify(bool)` | Enable verified atomic publication | | `.mode(mode)` | Generate, check, dry-run, or emit model | | `.mssql_production_style(bool)` | Enable MSSQL production-style DDL | | `.mssql_go(n)` | Set MSSQL GO interval | | `.run()` | Build the request and execute it | `GenerateReport` contains: * `rows_written`; * `effective_seed`, including entropy drawn for an unseeded run; * `diagnostics`, including info/advisory/warning entries on success; * `source_values`, containing path and rule kind but never literal values; * `explain`, when requested. Always inspect diagnostics even when generation succeeds. ## Compile options [Section titled “Compile options”](#compile-options) `CompileOptions` is the library form of seed, count, include/exclude, and per-table overrides. `TableCountOverride::absolute` and `TableCountOverride::scale` build repeatable per-table rules. ```rust use sql_splitter::generate::{CompileOptions, Generate, TableCountOverride}; let report = Generate::builder() .config("model.yaml") .output("synthetic.sql") .compile(CompileOptions { scale: Some(0.1), max_rows: Some(100_000), table_rows: vec![TableCountOverride::absolute("audit_events", 500)], ..CompileOptions::default() }) .run()?; # Ok::<(), Box>(()) ``` ## Staged API [Section titled “Staged API”](#staged-api) ```rust use sql_splitter::generate::{ CompileOptions, ExtensionRegistry, GenerationEngine, ModelCompiler, RenderOptions, SqlRenderer, }; use sql_splitter::synthetic::SyntheticFile; let yaml = std::fs::read_to_string("model.yaml")?; let model = SyntheticFile::parse_str(&yaml)? .into_model() .expect("a complete model"); let registry = ExtensionRegistry::standard(); let plan = ModelCompiler::new(registry) .compile(model, CompileOptions::default())?; let mut renderer = SqlRenderer::new(Vec::new(), RenderOptions::default()); let engine_report = GenerationEngine::new(plan).run(&mut renderer)?; let sql = renderer.finish()?; assert_eq!(engine_report.rows_written, 100); std::fs::write("synthetic.sql", sql)?; # Ok::<(), Box>(()) ``` ### `ExtensionRegistry` [Section titled “ExtensionRegistry”](#extensionregistry) `ExtensionRegistry::standard()` contains every built-in generator, modifier, and planner. `ExtensionRegistry::new()` starts empty. Register custom statically linked factories with `register_generator`, `register_modifier`, and `register_planner`. Each factory publishes a descriptor: * canonical kind and aliases; * argument names, requiredness, and summaries; * accepted SQL families for column operators; * read/write scope; * determinism, buffering, and verification capabilities; * cross-table capability for planners. Registration rejects duplicate kinds, duplicate aliases, and aliases that shadow canonical names. Custom operator kinds are runtime/compiler validated; the standard public JSON Schema cannot describe them. ### `ModelCompiler` [Section titled “ModelCompiler”](#modelcompiler) `ModelCompiler::compile` validates and resolves a `SyntheticModel` into an immutable `GenerationPlan`. It collects independent problems instead of stopping at the first one. Errors return `DiagnosticBag`; successful plans keep warning and informational diagnostics in `GenerationPlan::diagnostics`. The compiler resolves final counts, table dependency order, ownership, cross-column dependencies, planner phases, seeds, selection, and verification predicates. A plan should be treated as the authority for execution. ### `GenerationEngine` and `RowSink` [Section titled “GenerationEngine and RowSink”](#generationengine-and-rowsink) `GenerationEngine::run` drives the compiled plan against any `RowSink`: ```rust pub trait RowSink { fn begin_table(&mut self, table: &PlannedTable) -> Result<(), GenerateError>; fn write_row( &mut self, table: &PlannedTable, row: &GeneratedRow, ) -> Result<(), GenerateError>; fn end_table(&mut self, table: &PlannedTable) -> Result<(), GenerateError>; } ``` The engine executes table, cross-table family, and deferred-constraint phases. Use `SqlRenderer` for SQL output or implement a sink for another in-process destination. The sink receives typed `GeneratedValue` values rather than SQL literal strings. ### `SqlRenderer` [Section titled “SqlRenderer”](#sqlrenderer) `SqlRenderer` implements `RowSink`. `RenderOptions` controls dialect, schema/data mode, PostgreSQL COPY behavior, batch size, and MSSQL conventions. Call `finish()` to flush and recover the writer. Renderer warnings should be merged into the caller’s report when driving stages manually. ## Profiling and inference [Section titled “Profiling and inference”](#profiling-and-inference) ```rust use sql_splitter::profile::{DumpProfiler, ModelInference, ProfileDepth}; let profile = DumpProfiler::builder() .depth(ProfileDepth::Basic) .seed(42) .build() .profile_path(std::path::Path::new("source.sql"))?; let inferred = ModelInference::standard() .infer(&profile.schema, &profile)?; let model = inferred.model; # Ok::<(), Box>(()) ``` `DumpProfile` contains the portable schema, table/column evidence, and structured profiling diagnostics. `InferenceResult` contains the complete model, decisions, structured diagnostics, and source-literal metadata. `ModelMerger::merge(base, overrides)` returns the merged model plus a diagnostic bag on success. Preserve that bag: fingerprint and other warning diagnostics must not disappear simply because the merge produced a model. ## Errors and diagnostics [Section titled “Errors and diagnostics”](#errors-and-diagnostics) `GenerateError` distinguishes several layers: | Variant | Meaning | | ---------------------------- | ------------------------------------------------------------ | | `Diagnostics(DiagnosticBag)` | Multiple load, merge, or compile diagnostics | | `Diagnostic(Diagnostic)` | One structured runtime, rendering, or verification failure | | `TypeMismatch` | A runtime operator received the wrong generated value shape | | `Overflow` | Checked generator/planner arithmetic overflowed | | `Exhausted` | A bounded constraint such as `unique` could not be satisfied | | `InvalidInput` | Uncoded extension/internal runtime input failure | Built-in structured diagnostics use canonical definitions from `sql_splitter::diagnostic::codes`. A diagnostic includes code, severity, YAML or runtime path, message, optional help, related locations, and a documentation URL. Extension diagnostics may use their own namespaced string codes; they do not receive a built-in documentation link. # Model reference > Complete YAML grammar for generate models and overrides The model language describes table schemas, row counts, column generators, relationships, and planners. A `kind: model` document is self-contained; a `kind: overrides` document is a partial patch applied to an inferred base. ## Document envelope [Section titled “Document envelope”](#document-envelope) ```yaml # yaml-language-server: $schema=https://sql-splitter.dev/schemas/generate-config.schema.json version: 1 kind: model # model | overrides ``` Only version `1` is accepted. Duplicate keys and unknown fields are errors. `$schema` is optional editor metadata and is ignored by generation. | Kind | Contract | | ----------- | ------------------------------------------------------------------------------------ | | `model` | Complete schema, row rule, and ownership information. Can run without a source dump. | | `overrides` | Partial patch. Requires a source dump from which a base model can be inferred. | `--emit-config` always writes a complete model with resolved counts and `defaults.inference: disabled`. ## Top-level fields [Section titled “Top-level fields”](#top-level-fields) | Field | Model | Overrides | Meaning | | ---------- | -------- | --------- | ------------------------------------------------------- | | `$schema` | optional | optional | Editor-only JSON Schema URL | | `version` | required | required | Must be `1` | | `kind` | required | required | `model` or `overrides` | | `imports` | optional | optional | Local override files merged before the root | | `source` | optional | optional | Provenance, dialect, and fingerprint policy | | `output` | optional | optional | Default render settings | | `seed` | optional | optional | Root seed; `null` requests fresh entropy in an override | | `defaults` | optional | optional | Column inference policy | | `tables` | required | optional | Complete table map or table patches | | `profiles` | optional | optional | Removable bounded inference metadata | ### Source metadata [Section titled “Source metadata”](#source-metadata) ```yaml source: dialect: mysql fingerprint: sha256:0123456789abcdef fingerprint_policy: warn # ignore | warn | require ``` The fingerprint policy is evaluated when an overrides document containing a fingerprint is merged with a base model that also has one: * `ignore` accepts a mismatch silently. * `warn` emits `GEN-SOURCE-FINGERPRINT`. * `require` makes the mismatch an error. The source object from an override replaces the base object wholesale; its fields are not merged individually. A `generate dump.sql` run does not currently compute and compare a new dump fingerprint automatically. ### Output defaults [Section titled “Output defaults”](#output-defaults) ```yaml output: dialect: postgres mode: schema_and_data # schema_and_data | schema_only | data_only inserts: auto # auto | insert | copy batch_size: 1000 ``` All output fields are optional in both models and overrides. Explicit CLI options win; the output block fills in unset CLI choices. | Field | Effect | | ------------ | ---------------------------------------------------------------------------- | | `dialect` | Render target. Precedence is CLI, model, source dialect, then MySQL. | | `mode` | Emit schema and data, schema only, or data only. | | `inserts` | `insert` forces PostgreSQL multi-row inserts; `auto` and `copy` permit COPY. | | `batch_size` | Rows per `INSERT` or `COPY` batch. Must be between 1 and 1,000,000. | A deliberately selected cross-dialect target maps types and reports lossy conversions. Preserving the source dialect retains original DDL where possible. ### Seeds [Section titled “Seeds”](#seeds) ```yaml seed: 42 tables: inherited: # omitted: inherit 42 independent: seed: 9001 random_each_run: seed: null ``` An omitted root seed draws fresh entropy and reports the effective seed. An omitted table seed inherits; an integer is independent; `null` opts that table out of deterministic inheritance. CLI seed controls replace the root choice, not explicit per-table choices. ### Defaults and inference [Section titled “Defaults and inference”](#defaults-and-inference) ```yaml defaults: inference: disabled # disabled | schema ``` * `disabled` requires an explicit owner for every generated column; an unowned column is `GEN-COLUMN-OWNER-MISSING`. * `schema` infers an owner for every otherwise-unowned column. Structural facts win first (identity/serial keys, foreign-key markers, bound `DEFAULT`s, bare integer primary keys), then a column’s **name and type** choose a generator: semantic matches (`email` → `internet.email`, `first_name` → `person.first_name`, `city` → `address.city`, a `price`/`amount` decimal → `commerce.money`), credential names (`password`, `api_key`, …) → `credential.*`, and finally a type fallback that guarantees a value for every SQL family. A run under `schema` never leaves a column unowned. Schema inference still uses only the declared schema — no value evidence. It therefore cannot replay observed distributions, corroborate an ambiguous name against sampled values, or nominate planners; those require the profiling pipeline described in [Inference](/commands/generate/inference/). Prefer explicit generators when you need a specific shape. ### Imports [Section titled “Imports”](#imports) ```yaml version: 1 kind: model imports: - fragments/core.yaml - fragments/commerce.yaml tables: {} ``` Each imported document must be `kind: overrides`. Paths are relative to the root document, must not be absolute or remote, and may not import further documents. Imports merge in declaration order; the root then wins. Maps merge by key, while lists replace wholesale. Two imports defining the same path are reported as `GEN-IMPORT-COLLISION`. ## Tables [Section titled “Tables”](#tables) ```yaml tables: customers: rows: { kind: fixed, count: 5000 } schema: { name: customers, columns: [] } columns: {} relationships: [] planners: [] ``` | Field | Model default | Meaning | | --------------- | ------------- | --------------------------------------------- | | `seed` | inherit | Per-table seed policy | | `rows` | required | Exact or derived row-count rule | | `schema` | required | Portable SQL table schema | | `columns` | `{}` | Column generator and modifier rules | | `relationships` | `[]` | Generation-time foreign-key relationships | | `planners` | `[]` | Coordinated multi-column or cross-table rules | ### Row-count rules [Section titled “Row-count rules”](#row-count-rules) ```yaml rows: { kind: fixed, count: 5000 } rows: { kind: observed, count: 9876 } rows: { kind: scale, base: 1000, factor: 2.0, count: 2000 } rows: kind: relation.children parent: orders count: 7600 distribution: { kind: histogram, mean: 3.8, min: 1.0, max: 25.0 } ``` | Kind | Fields | Behavior | | ------------------- | --------------------------------- | ---------------------------------------------------------------------- | | `fixed` | `count` | Exact hand-chosen count | | `observed` | `count` | Count captured from the source profile | | `scale` | `base`, `factor`, `count` | Scaled base with stochastic rounding where needed | | `relation.children` | `parent`, `count`, `distribution` | Child count and per-parent fan-out derived from the final parent count | Complete and emitted models store `count` even for derived rules. This freezes the resolved result for source-free reuse. An emitted child produced by a cross-table family planner keeps its `relation.children` rule as well as the resolved `count`, because the planner still needs the distribution to reproduce the same child rows. A child distribution has `kind`, `mean`, `min`, and `max`. Supported kinds are `observed`, `fixed`, `uniform`, `poisson`, and `histogram`. The minimum must be feasible for the resolved parent and child counts; otherwise compilation emits `GEN-CHILD-COUNT-IMPOSSIBLE`. ### Portable schema [Section titled “Portable schema”](#portable-schema) ```yaml schema: name: orders primary_key: [id] columns: - { name: id, type: bigint, nullable: false, primary_key: true } - { name: customer_id, type: bigint, nullable: false } unique_constraints: [] check_constraints: [] indexes: [] relationships: - name: fk_orders_customer columns: [customer_id] referenced_table: customers referenced_columns: [id] ``` | Field | Default | Meaning | | -------------------- | -------- | ---------------------------------------------------------- | | `name` | required | Table name | | `columns` | required | Ordered column list | | `primary_key` | `[]` | Primary-key columns | | `unique_constraints` | `[]` | Named or anonymous unique column sets | | `check_constraints` | `[]` | Named or anonymous SQL expressions | | `indexes` | `[]` | Index names, columns, uniqueness, and optional source type | | `create_statement` | none | Original DDL, reusable only for compatible output | | `relationships` | `[]` | DDL foreign keys rendered into schema output | Schema-level relationships affect DDL. The table-level relationships described below affect generated values. Declare both when both behaviors are needed. ### Portable columns [Section titled “Portable columns”](#portable-columns) ```yaml - name: email source_type: varchar(255) family: text nullable: false primary_key: false unique: true generated: false identity: false ``` | Field | Default | Meaning | | ------------- | --------- | -------------------------------------------------------------------------------------------------------- | | `name` | required | Column name | | `source_type` | required | Original SQL type text | | `type` | shorthand | Input alias for `source_type`; emitted models use canonical fields | | `family` | derived | `integer`, `big_integer`, `decimal`, `boolean`, `text`, `bytes`, `uuid`, `date_time`, `json`, or `other` | | `nullable` | required | Whether SQL `NULL` is legal | | `primary_key` | `false` | Primary-key member | | `unique` | `false` | Single-column uniqueness | | `default_sql` | none | Original SQL default expression | | `generated` | `false` | Database-computed column | | `identity` | `false` | Identity/auto-increment column | | `collation` | none | Source collation | The family drives generator and modifier compatibility. `source_type` remains available for dialect rendering and scale/precision inspection. ## Column rules [Section titled “Column rules”](#column-rules) ```yaml columns: email: semantic: internet.email generator: { kind: internet.email } modifiers: - { kind: unique, max_attempts: 32, on_exhaustion: error } - { kind: case, mode: lower } ``` | Field | Default | Meaning | | ----------- | ------- | ------------------------------------------------ | | `semantic` | none | Informational inference annotation | | `generator` | none | One base value producer | | `modifiers` | `[]` | Ordered transformations applied after generation | Every generated column must have exactly one owner: a column generator, a planner, a relationship marker, or a structural inference rule. Conflicting owners emit `GEN-COLUMN-OWNER-CONFLICT`; missing owners emit `GEN-COLUMN-OWNER-MISSING`. Single-column primary and unique keys are protected automatically. Inherently unique `sequence`, `monotonic`, and UUID keys need no modifier; other producers receive a strict `unique` modifier unless one was explicitly configured. Composite uniqueness is not inferred from independent per-column uniqueness; model it by construction and use `--verify` to audit it. ## Generation relationships [Section titled “Generation relationships”](#generation-relationships) ```yaml relationships: - name: orders_customer columns: [customer_id] references: { table: customers, columns: [id] } distribution: uniform ``` | Field | Default | Meaning | | -------------- | -------- | ------------------------------------------------------------------- | | `name` | none | Required when a rule or planner names the relationship | | `columns` | required | Local foreign-key columns | | `references` | required | Parent table and parent columns | | `distribution` | none | `uniform`, `sequential`, `weighted`, or `observed` parent selection | Relationship columns use `relation.foreign_key` or `relation.composite_key`, either explicitly or through structural inference. The current random-access parent-key domains are bare integer identities, `sequence`, and UUID. Other target key recipes fail at generation with `GEN-KEY-DOMAIN-UNSUPPORTED`. Trees, polymorphic pairs, tenant-consistent references, and distinct junction pairs need planners because they coordinate more than an ordinary foreign key. ## Overrides [Section titled “Overrides”](#overrides) ```yaml version: 1 kind: overrides tables: audit_events: rows: { kind: observed, scale: 0.01 } users: columns: email: generator: { kind: internet.email } ``` All override fields are optional after `version` and `kind`, including `output`. Missing fields mean “leave the base unchanged.” | Field | Merge behavior | | ---------- | ------------------------------------------------- | | `source` | Replace the whole source object | | `defaults` | Replace the whole defaults object | | `seed` | Omitted inherits, `null` clears, integer replaces | | `output` | Replace only present output fields | | `tables` | Apply partial patches by table name | Within a table patch: * `rows` merges fields only when its `kind` matches the base kind. Switching kind requires every field needed by the new kind. * `schema` is an assertion over table name or original DDL, not a schema edit. * `columns` replaces each present `semantic`, `generator`, or complete `modifiers` list. * `relationships` and `planners` replace their complete lists when present. Missing tables and columns are reported independently, so one run can identify all stale override paths. ## JSON Schema and editor validation [Section titled “JSON Schema and editor validation”](#json-schema-and-editor-validation) Use the published [generate JSON Schema](/schemas/generate-config.schema.json) through a `$schema` key or YAML language-server modeline. It is derived from the model types and the standard operator registry. The schema: * restricts generator, modifier, and planner `kind` values to built-ins; * rejects unknown top-level operator arguments; * requires arguments marked required by the registry; * rejects unknown model fields and structurally invalid tagged objects. Argument value semantics remain compiler-owned. For example, JSON Schema can recognize a `unique.max_attempts` argument while the compiler validates numeric bounds and cross-field constraints. Custom statically linked operators are validated by their registry at runtime but cannot appear in the standard published schema. ## Profiles metadata [Section titled “Profiles metadata”](#profiles-metadata) ```yaml profiles: customers.status: rows: 182340 null_fraction: 0.0 distinct_estimate: 3 inference: selected: weighted_choice confidence: high reasons: [low_cardinality, stable_top_values, status_name] ``` Profiles are bounded explanations keyed by `table.column`. Deleting the entire map does not change generation because the selected rules and counts are stored elsewhere in the complete model. # Modifiers > Ordered post-generation transformations, constraints, and formatting Modifiers transform the base value produced by a generator. They run in list order, so changing their order can change nullability, uniqueness, length, and formatting. ```yaml generator: { kind: internet.email } modifiers: - { kind: case, mode: lower } - { kind: prefix, value: "test+" } - { kind: truncate, max_length: 255 } - { kind: unique, max_attempts: 20 } ``` All built-in modifiers operate only on their own column, stream row by row, and are deterministic under a fixed seed. ## `null_rate` [Section titled “null\_rate”](#null_rate) Required `rate` in `0..1`; accepts every family. The target schema column must be nullable. The modifier replaces the current value with SQL `NULL` at the configured probability. Place it after any modifier that cannot accept `NULL`. If it runs first, a later text or numeric modifier can fail on the null value. ## `unique` [Section titled “unique”](#unique) | Argument | Default | Meaning | | --------------- | --------- | ------------------------------------------- | | `max_attempts` | `10` | Candidate mutations tried after a collision | | `on_exhaustion` | `error` | `error`, `warn`, or `widen` | | `max_tracked` | `1000000` | Maximum distinct values retained in memory | `unique` tracks complete typed values up to `max_tracked`. On collision it can mutate integers/decimals, append to text, extend bytes, and retry. UUID values are normally unique by construction. Boolean and some other families cannot be widened. * `error` fails generation when attempts or tracking capacity are exhausted. * `warn` accepts the final duplicate candidate. Despite the policy name, the current runtime does not add a separate warning diagnostic. * `widen` tries ten times the configured attempt budget, then accepts the final candidate like `warn`. It is rejected for families with no mutation strategy. For large keys, prefer an inherently unique generator. The compiler recognizes `sequence`, `uuid`, `monotonic`, `identifier.ulid`, and any sufficiently high-entropy random string (a default-length `identifier.nanoid`, `identifier.token`, `identifier.hash`, or `credential.*`) as unique by construction, and will **not** auto-attach a tracking `unique` modifier to a single-column key that uses one — so those keys are never bounded by `max_tracked`. A short, low-entropy random string (for example a 4-character token) can still collide, so it is tracked. Global uniqueness enforced by this modifier necessarily retains state and is bounded by `max_tracked`. ## `prefix` and `suffix` [Section titled “prefix and suffix”](#prefix-and-suffix) Required string `value`; optional `max_length`. Both accept text. `prefix` prepends and `suffix` appends, then truncates to the first `max_length` Unicode scalar values if configured. ```yaml modifiers: - { kind: prefix, value: "DEV-", max_length: 32 } - { kind: suffix, value: "-synthetic", max_length: 64 } ``` ## `truncate` [Section titled “truncate”](#truncate) Required `max_length`; accepts text. Keeps the first N Unicode scalar values. This is character-oriented, not a byte-length or grapheme-cluster guarantee. ## `case` [Section titled “case”](#case) Required `mode`: `upper`, `lower`, or `title`; accepts text. Title mode uppercases the first scalar of each whitespace-delimited segment and otherwise preserves the remainder. ## `clamp` [Section titled “clamp”](#clamp) Required numeric `min` and `max`; accepts integer, big-integer, and decimal. Bounds are inclusive and `min` must not exceed `max`. ```yaml modifiers: [{ kind: clamp, min: 0, max: 100 }] ``` Decimal bounds are converted to the generated value’s scale before clamping. ## `round` [Section titled “round”](#round) Required non-negative `scale`; accepts decimal. Uses half-up rounding when the target scale is smaller and is a no-op when it is equal or larger. ## `format` [Section titled “format”](#format) Required string `template`; accepts text. Every literal `{value}` placeholder is replaced with the current text. ```yaml modifiers: [{ kind: format, template: "https://example.test/{value}" }] ``` It is not a general expression language and does not read sibling columns; use the `template` generator for cross-column composition. ## Type and ordering failures [Section titled “Type and ordering failures”](#type-and-ordering-failures) A modifier attached to an unsupported SQL family fails compilation with `GEN-MODIFIER-TYPE`. A runtime type mismatch can still occur when an earlier modifier changes the generated value into a shape the next modifier cannot consume. Keep null-producing steps last unless later steps explicitly tolerate nulls. # Planners > Coordinate multi-column and cross-table invariants during generation A planner owns several columns as one unit. Use one when independent generators could produce individually valid values that violate a shared equation, ordering, state transition, or relationship. ```yaml tables: sessions: planners: - kind: temporal.interval columns: { start: started_at, end: ended_at, duration: duration_seconds } start: { kind: range, min: 2025-01-01, max: 2025-12-31 } duration: { kind: uniform, unit: seconds, min: 60, max: 3600 } ``` Planner-owned columns must not also have column generators. The compiler resolves the planner’s configured reads/writes, rejects missing columns and ownership conflicts, and orders planners with other column dependencies. Row counts are separate `relation.children` is a row-count rule under `tables..rows`, not a planner. It decides how many child rows belong to each parent. A planner may consume that fan-out, but it does not replace the row rule. ## Planner capabilities [Section titled “Planner capabilities”](#planner-capabilities) | Planner | Scope | Primary invariant | | ---------------------------- | -------------- | ------------------------------------------------ | | `temporal.interval` | same table | `end = start + duration` and open-row coherence | | `workflow.progress_counters` | same table | counter partitions, status, and completion agree | | `commerce.order_family` | parent + child | line-item arithmetic sums to order totals | | `temporal.timestamps` | same table | created/updated/extra timestamps are causal | | `temporal.soft_delete` | same table | deleted timestamp and flag agree | | `temporal.lifecycle` | same table | ordered state-machine timestamps and status | | `hierarchy.tree` | same table | bounded acyclic self-parent tree | | `relation.junction_pair` | cross-table | distinct left/right parent pairs | | `relation.polymorphic_pair` | cross-table | atomic type/id target pair | | `relation.tenant_family` | cross-table | child and selected parent share a tenant | | `geo.coordinate_pair` | same table | latitude/longitude stay within one bounding box | | `file.metadata` | same table | filename, extension, MIME, size, and hash agree | ## `temporal.interval` [Section titled “temporal.interval”](#temporalinterval) ```yaml planners: - kind: temporal.interval columns: start: started_at end: ended_at duration: duration_seconds open: is_open start: kind: range # range | observed_range | monotonic min: 2025-01-01 max: 2025-12-31 duration: kind: normal # fixed | uniform | normal | histogram | observed unit: seconds mean: 900 stddev: 120 min: 60 max: 3600 open_probability: 0.05 end_inclusive: false timezone: preserve ``` Required roles are `columns.start`, `columns.end`, and `columns.duration`; `columns.open` is optional. Start kinds `range` and `observed_range` need parseable `min`/`max`. `monotonic` uses `step_seconds=1` by default. Duration kinds: * `fixed` uses `value`. * `uniform` uses `min` and `max`. * `normal` uses `mean`, `stddev`, and bounds. * `histogram`/`observed` currently use a bounded min-skewed placeholder within `min`/`max`; profiled bucket evidence is not threaded into the planner. Units include nanoseconds through days and their common abbreviations. `open_probability` is within `0..1`; when positive, `end` must be nullable. Closed rows satisfy the interval equation. `end_inclusive` requires a duration of at least one unit. `timezone` accepts `preserve`, `utc`, or an IANA zone; `preserve` currently renders UTC because per-row source zones are not retained. ## `workflow.progress_counters` [Section titled “workflow.progress\_counters”](#workflowprogress_counters) ```yaml planners: - kind: workflow.progress_counters columns: total: total_rows processed: processed_rows succeeded: succeeded_rows failed: failed_rows pending: pending_rows status: status completed_at: completed_at total: { kind: uniform, min: 10, max: 500 } progress: kind: mixture # mixture | complete | in_progress | not_started | observed complete_weight: 0.72 active_weight: 0.23 not_started_weight: 0.05 partition: exact # exact | allow_unclassified success_ratio: 0.9 completed_statuses: [completed, failed] active_statuses: [queued, running] ``` Only `columns.total` is required; configure the counters the table actually has. `total` supports fixed and inclusive uniform shapes. Exact partitioning guarantees `succeeded + failed = processed` and `pending = total - processed`. `allow_unclassified` reserves an unclassified share while keeping the whole processed count exact. `progress.kind` can force one state or choose a weighted mixture. `observed` has no planner profile input today: exact partitioning rejects it, while `allow_unclassified` falls back to a 0.70/0.25/0.05 mixture. If a status role is configured, reachable completed and active states need non-empty vocabularies. Completed rows use `processed = total` and a completion timestamp; incomplete rows keep it null. ## `commerce.order_family` [Section titled “commerce.order\_family”](#commerceorder_family) This cross-table planner coordinates one parent order with its child line items using integer minor-unit arithmetic. ```yaml planners: - kind: commerce.order_family children: order_items relationship: order_items_order columns: subtotal: subtotal discount: discount_total tax: tax_total shipping: shipping_total total: grand_total child_columns: quantity: quantity unit_price: unit_price discount: discount_amount tax: tax_amount line_total: line_total currency_scale: 2 rounding: largest_remainder # largest_remainder | last_line | bankers quantity: { min: 1, max: 5 } unit_price: { min_minor: 100, max_minor: 10000 } tax: kind: weighted_choice rates: [0.0, 0.08, 0.25] weights: [0.05, 0.15, 0.80] discount: { kind: fixed_rate, rate: 0.0 } shipping: { amount_minor: 0 } ``` `children`, `relationship`, parent `subtotal`/`total`, child `quantity`/`unit_price`/`line_total`, `currency_scale`, and `rounding` are required. Optional parent/child tax and discount roles are written only when mapped. Money columns must use the configured `currency_scale`, which must be between 0 and 18. The child table’s `rows.distribution` is the sole line-count source; the planner intentionally has no line-count argument. Tax and discount support a fixed rate or weighted rate choice. Shipping accepts major or minor units. Largest-remainder rounding distributes residual minor units so child sums match the parent exactly; `last_line` places the residual on the final line. Because the planner draws each order’s line count itself, the child table’s realized row total drifts from the compile-time `parents × mean` estimate under a stochastic fan-out (`poisson`, `uniform`, `histogram`, `observed`). `--verify` therefore reports the child’s row count as sampled rather than exact — it passes whenever the child is non-empty as expected. The exact family-sum, arity, non-null, and foreign-key checks still audit that every generated line is well-formed and sums to its parent, so a broken family is never a silent pass. The descriptor reports verification as unsupported. Runtime verification can check simple tax/discount sums, but cannot independently express all subtotal, product, shipping, and total equations. ## `temporal.timestamps` [Section titled “temporal.timestamps”](#temporaltimestamps) ```yaml planners: - kind: temporal.timestamps columns: created_at: created_at updated_at: updated_at archived_at: archived_at created: { kind: range, min: 2020-01-01, max: 2026-01-01 } update_delay: { kind: uniform, unit: hours, min: 0, max: 72 } other_delay: { kind: uniform, unit: days, min: 0, max: 30 } ``` `columns.created_at` and `columns.updated_at` are required. Any additional flat keys directly below `columns` are extra timestamp roles; do not nest them. `created` defines the initial range. `update_delay` controls updated time and `other_delay` controls extra roles, defaulting to the update block. Every later timestamp is at or after creation. ## `temporal.soft_delete` [Section titled “temporal.soft\_delete”](#temporalsoft_delete) ```yaml planners: - kind: temporal.soft_delete columns: { deleted_at: deleted_at, is_deleted: is_deleted } deletion_probability: 0.1 deleted_range: { kind: range, min: 2024-01-01, max: 2026-01-01 } ``` `columns.deleted_at` is required; `is_deleted` is optional. `deletion_probability` defaults to zero and is within `0..1`. If it is less than one, `deleted_at` must be nullable. With a flag role, flag and timestamp nullness are verified together; without it, only timestamp range is checked. ## `temporal.lifecycle` [Section titled “temporal.lifecycle”](#temporallifecycle) ```yaml planners: - kind: temporal.lifecycle columns: status: status draft: created_at active: activated_at archived: archived_at states: [draft, active, archived] weights: [0.1, 0.7, 0.2] start: { kind: range, min: 2024-01-01, max: 2026-01-01 } step: { kind: uniform, unit: days, min: 1, max: 14 } ``` `columns.status`, `states`, `weights`, `start`, and `step` define a finite state machine. State-to-column entries are flat siblings under `columns`. The planner chooses a terminal state, stamps each reached state in order, and leaves later state timestamps null. Weights must match the states and contain a positive total. Any state timestamp that can remain unreached must be nullable. ## `hierarchy.tree` [Section titled “hierarchy.tree”](#hierarchytree) ```yaml planners: - kind: hierarchy.tree columns: { parent: parent_id } root_ratio: 0.1 max_depth: 6 max_branching: 4 ``` The required parent role names a nullable self-reference; an optional `depth` role names a column that receives each row’s depth (roots = 0). `root_ratio` defaults to `0.1`; `max_depth` defaults to `6`; `max_branching` is unlimited when omitted. Roots are at depth 0, so `max_depth` is the deepest level a node may occupy — a tree with `max_depth: 6` spans depths 0 through 6. Rows choose only eligible earlier rows as parents, guaranteeing no cycle. The first row and any row without an eligible parent become roots. ## `relation.junction_pair` [Section titled “relation.junction\_pair”](#relationjunction_pair) ```yaml planners: - kind: relation.junction_pair columns: { left: user_id, right: role_id } left_relationship: user_roles_user right_relationship: user_roles_role ``` Both relationships and both output roles are required. Parent keys must use supported random-access domains. The planner maps row indices into the Cartesian pair space without repeats. A requested row count above `left_count * right_count` is `GEN-JUNCTION-EXHAUSTED`. ## `relation.polymorphic_pair` [Section titled “relation.polymorphic\_pair”](#relationpolymorphic_pair) ```yaml planners: - kind: relation.polymorphic_pair columns: { type: commentable_type, id: commentable_id } targets: - { table: posts, weight: 0.7 } - { table: photos, weight: 0.3, type: photo, id_column: id } ``` The type/id roles and a non-empty target list are required. Target `type` defaults to its table name and `id_column` to the target primary key. Weights are non-negative with a positive total. Zero-row targets are dropped. Every remaining target needs a supported parent-key domain. Type/id atomicity is guaranteed by construction but has no separate verification predicate today. ## `relation.tenant_family` [Section titled “relation.tenant\_family”](#relationtenant_family) ```yaml planners: - kind: relation.tenant_family columns: { tenant: tenant_id, parent: customer_id } relationship: orders_customer num_tenants: 8 tenant_start: 1 ``` Tenant and parent roles, a parent relationship, and a positive `num_tenants` are required. `tenant_start` defaults to `1`. Parent rows are split into contiguous balanced tenant blocks. Each child chooses a tenant, then a parent from the same block. Parent keys must use supported random-access domains. ## `geo.coordinate_pair` [Section titled “geo.coordinate\_pair”](#geocoordinate_pair) ```yaml planners: - kind: geo.coordinate_pair columns: { latitude: lat, longitude: lng } precision: 6 bounds: { min_lat: 35, max_lat: 71, min_lon: -25, max_lon: 45 } ``` Both roles are required. `precision` defaults to six decimal places. Bounds default to the global latitude/longitude ranges, must be ordered, and must stay within valid geographic limits. Generation guarantees bounds, but the current verification predicate set cannot independently audit decimal coordinate ranges. ## `file.metadata` [Section titled “file.metadata”](#filemetadata) ```yaml planners: - kind: file.metadata columns: name: file_name extension: extension mime_type: mime_type size: size_bytes hash: checksum extensions: [pdf, jpg, png, docx] size: { min: 1024, max: 5000000 } hash_kind: sha256 # md5 | sha1 | sha256 | sha512 ``` All five roles are required. Omit `extensions` to use the built-in catalog; a provided list must contain supported extensions. Size defaults to the planner’s built-in non-negative range and must be ordered. Hash kind controls the output hex length. The tuple is coherent by construction. Verification currently checks size non-negativity, not textual filename/extension/MIME agreement. ## Verification expectations [Section titled “Verification expectations”](#verification-expectations) Descriptor-level support means a planner can provide predicates, not that every capability is expressible. A passing verified run reports `GEN-VERIFY-NOTCHECKED` when a configured capability could not be checked exactly. Review [Privacy and verification](/commands/generate/privacy-verification/) before treating verified output as an independent proof of every planner invariant. # Privacy and verification > Literal-value safety, temporary files, atomic publication, and verification coverage ## Synthetic output can contain source literals [Section titled “Synthetic output can contain source literals”](#synthetic-output-can-contain-source-literals) `generate` is not a redaction tool. A model may intentionally carry: * `constant`, `weighted_choice`, or `observed_sample` values; * a source schema default used through `database_default`; * literal text inside a preserved check constraint. Those values can reappear in generated SQL. Use [`redact`](/commands/redact/) when values must be irreversibly transformed. ## `GEN-SOURCE-VALUES` [Section titled “GEN-SOURCE-VALUES”](#gen-source-values) The resolved model is scanned before execution. When literal-bearing rules are present, the report receives one advisory diagnostic: ```text advisory[GEN-SOURCE-VALUES] tables 2 rule(s) replay literal values that may have been derived from the source dump; review the listed rules before sharing the output - tables.orders.columns.status (weighted_choice) - tables.customers.columns.plan (observed_sample) docs: https://sql-splitter.dev/commands/generate/diagnostics/#GEN-SOURCE-VALUES ``` Properties: * Values are never included; only YAML paths and rule kinds appear. * It is a regular structured diagnostic in human and JSON output. * Its severity is `advisory`, so `--strict` never promotes it. * It remains visible under `--quiet`. * Wording distinguishes a run that profiled a dump from a model-only run whose literals are hand-authored. Library callers can also inspect `GenerateReport.source_values`, which contains the same path/rule-kind locations without values. ## Build a literal-free model [Section titled “Build a literal-free model”](#build-a-literal-free-model) 1. Replace `constant`, `weighted_choice`, and `observed_sample` with semantic, statistical, structural, or planner-owned rules. 2. Replace `database_default` when its preserved source default contains a sensitive literal. 3. Review preserved check constraints. 4. Generate and confirm there is no `GEN-SOURCE-VALUES` diagnostic, or verify that `GenerateReport.source_values` is empty in a library integration. Credential inference already avoids literal-bearing generators. An explicit model remains authoritative and may choose one; the advisory then reports it. ## Verified generation [Section titled “Verified generation”](#verified-generation) ```bash sql-splitter generate --config model.yaml --verify -o synthetic.sql ``` Verification requires a filesystem destination. The command: 1. creates protected temporary output beside the destination; 2. generates the complete SQL there; 3. audits the generated rows against the compiled plan; 4. renames the temporary file into place only after a full pass. A failed audit leaves an existing destination untouched. If a resolved model is also requested, SQL is published first and the model second; a failure during the second publication is reported precisely as a partial publication because two independent filesystem renames cannot be one atomic transaction. ## Exact checks [Section titled “Exact checks”](#exact-checks) Where supported, verification checks: * row counts and row arity; * non-null, character-length, integer-width, decimal precision/scale, primary-key, and unique constraints; * ordinary foreign-key membership; * planner predicates such as temporal ordering, null/flag coherence, counter equations, selected ranges, and simple cross-table sums. Exact uniqueness, foreign-key, and family checks remain bounded for large outputs. Once their in-memory key budget is exceeded, verification writes protected sorted runs, merges them on disk, and evaluates exact key bytes in a linear merge pass. Verification parses the generated SQL. PostgreSQL COPY rows have less exact coverage than ordinary INSERT rows. `--verify` also follows the selected output mode. Data-only output uses the compiled model schema to decode rows without requiring emitted DDL. Schema-only output requires that no row data was rendered. It also compares the rendered columns, types, nullability, primary keys, unique constraints, and foreign keys with the compiled model. ## Honest coverage gaps [Section titled “Honest coverage gaps”](#honest-coverage-gaps) Some invariants are guaranteed by construction but cannot be expressed by the current verifier. A passing run reports `GEN-VERIFY-NOTCHECKED` when configured capabilities were not checked exactly. Known examples include: * all product/offset equations in `commerce.order_family`; * atomic type/id agreement in `relation.polymorphic_pair`; * decimal bounds in `geo.coordinate_pair`; * filename/extension/MIME textual agreement in `file.metadata`. Such a run exits successfully because every check it could perform passed, but the warning makes the boundary explicit. `--strict` can promote that warning to a failure when complete independent coverage is required. ## Temporary-file guarantees and limits [Section titled “Temporary-file guarantees and limits”](#temporary-file-guarantees-and-limits) Protected verification and planner-spool files use unpredictable names, exclusive creation, and owner-only permissions on Unix. Normal completion, ordinary errors, and handled Ctrl-C interrupts clean them through object destruction. No process can clean up after `SIGKILL`, power loss, or an aborting runtime. A protected temporary file may remain in those cases. Final output files follow normal destination permissions and are not exposed until publication. Cross-table planners can spill bounded family state to protected temporary storage. Spilling changes performance, not deterministic output. # graph > Generate ERD diagrams from SQL dumps Generate Entity-Relationship Diagrams (ERD) in multiple formats. **Alias:** `gr` (e.g., `sql-splitter gr dump.sql -o schema.html`) ## When to Use This [Section titled “When to Use This”](#when-to-use-this) * **Schema documentation** - Generate visual diagrams for documentation or onboarding * **Dependency analysis** - Understand which tables depend on which before refactoring * **Safe DROP planning** - See what depends on a table before dropping it * **Circular dependency detection** - Find problematic FK cycles that complicate migrations * **Impact assessment** - Before modifying a table, see all tables that reference it Use [`order`](/commands/order) if you need to reorder a dump for safe imports based on the dependency graph. ## How It Works [Section titled “How It Works”](#how-it-works) The graph command parses CREATE TABLE statements and foreign key definitions to build a dependency graph: * **Nodes** = Tables (with columns, types, and constraints) * **Edges** = Foreign key relationships (A → B means “A references B”) The direction matters: if `orders.user_id` references `users.id`, the edge goes `orders → users`. This means you must create `users` before `orders`. ## Usage [Section titled “Usage”](#usage) ```bash sql-splitter graph [OPTIONS] ``` ## Examples [Section titled “Examples”](#examples) ### Basic Diagram Generation [Section titled “Basic Diagram Generation”](#basic-diagram-generation) ```bash # Interactive HTML with pan/zoom (great for exploration) sql-splitter graph dump.sql -o schema.html # Mermaid format (embeds in Markdown docs) sql-splitter graph dump.sql -o schema.mmd --format mermaid # Graphviz DOT (for custom rendering) sql-splitter graph dump.sql -o schema.dot ``` ### Focusing on Specific Tables [Section titled “Focusing on Specific Tables”](#focusing-on-specific-tables) For large schemas, focus on what matters: ```bash # Show only user-related and order-related tables sql-splitter graph dump.sql --tables "user*,order*" --exclude "log*" # Focus on one table and its dependencies (what it references) sql-splitter graph dump.sql --table orders --transitive # Focus on one table and its dependents (what references it) sql-splitter graph dump.sql --table users --reverse ``` ### Finding Circular Dependencies [Section titled “Finding Circular Dependencies”](#finding-circular-dependencies) Circular FKs complicate imports and migrations. Find them: ```bash # Show only tables involved in cycles sql-splitter graph dump.sql --cycles-only ``` ### Machine-Readable Output [Section titled “Machine-Readable Output”](#machine-readable-output) ```bash # Full schema as JSON for tooling sql-splitter graph dump.sql --json # Pipe to jq for analysis sql-splitter graph dump.sql --json | jq '.relationships | length' ``` ## Options [Section titled “Options”](#options) | Flag | Short | Description | Default | | --------------- | ----- | ---------------------------------------------------- | ----------- | | `--output` | `-o` | Output file (html, dot, mmd, json, png, svg, pdf) | stdout | | `--format` | `-f` | Output format: `html`, `dot`, `mermaid`, `json` | auto | | `--dialect` | `-d` | SQL dialect | auto-detect | | `--layout` | | Layout direction: `lr` (horizontal), `tb` (vertical) | `lr` | | `--tables` | `-t` | Include tables matching glob patterns | all | | `--exclude` | `-e` | Exclude tables matching glob patterns | none | | `--table` | | Focus on a specific table | - | | `--transitive` | | Show all dependencies of focused table | false | | `--reverse` | | Show all tables that depend on focused table | false | | `--max-depth` | | Limit traversal depth | unlimited | | `--cycles-only` | | Only show tables in circular dependencies | false | | `--render` | | Render DOT to PNG/SVG/PDF using Graphviz | false | | `--progress` | `-p` | Show progress bar | false | | `--json` | | Output as JSON | false | ## Output Formats [Section titled “Output Formats”](#output-formats) ### HTML (default) [Section titled “HTML (default)”](#html-default) Interactive diagram with: * Dark/light theme toggle * Pan and zoom * Click to highlight relationships * Copy Mermaid button ### Graphviz DOT [Section titled “Graphviz DOT”](#graphviz-dot) schema.dot ```text digraph schema { rankdir=LR; users [label="users|id INT PK\lname VARCHAR\lemail VARCHAR\l"]; orders [label="orders|id INT PK\luser_id INT FK\l"]; orders -> users [label="user_id"]; } ``` Render with Graphviz: ```bash sql-splitter graph dump.sql -o schema.dot dot -Tpng schema.dot -o schema.png ``` Or use `--render`: ```bash sql-splitter graph dump.sql -o schema.png --render ``` ### Mermaid [Section titled “Mermaid”](#mermaid) ```mermaid erDiagram users { INT id PK VARCHAR name VARCHAR email } orders { INT id PK INT user_id FK } orders }|--|| users : user_id ``` ### JSON [Section titled “JSON”](#json) Full schema details: ```json { "tables": [ { "name": "users", "columns": [...], "primary_key": ["id"], "foreign_keys": [] } ], "relationships": [...], "cycles": [] } ``` ## Focusing on Tables [Section titled “Focusing on Tables”](#focusing-on-tables) ### Show dependencies [Section titled “Show dependencies”](#show-dependencies) ```bash sql-splitter graph dump.sql --table orders --transitive ``` Shows `orders` and all tables it references (directly or indirectly). ### Show dependents [Section titled “Show dependents”](#show-dependents) ```bash sql-splitter graph dump.sql --table users --reverse ``` Shows `users` and all tables that reference it. ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) ### Graph is too large to read [Section titled “Graph is too large to read”](#graph-is-too-large-to-read) Large schemas produce cluttered diagrams. Focus your view: ```bash # Limit to specific tables sql-splitter graph dump.sql --tables "core_*" --exclude "*_log,*_audit" # Focus on a single table's neighborhood sql-splitter graph dump.sql --table orders --transitive --max-depth 2 # Use JSON and filter programmatically sql-splitter graph dump.sql --json | jq '.tables | map(select(.foreign_keys | length > 0))' ``` ### Missing edges (FK relationships not shown) [Section titled “Missing edges (FK relationships not shown)”](#missing-edges-fk-relationships-not-shown) Foreign keys must be explicitly defined in CREATE TABLE statements. Implicit FKs (application-level only) won’t appear. Check if FKs are defined: ```bash sql-splitter graph dump.sql --json | jq '.relationships' ``` ### Graphviz rendering fails [Section titled “Graphviz rendering fails”](#graphviz-rendering-fails) Install Graphviz, then render manually or use `--render`: ```bash # Install Graphviz brew install graphviz # macOS apt install graphviz # Ubuntu/Debian # Render DOT to PNG sql-splitter graph dump.sql -o schema.dot dot -Tpng schema.dot -o schema.png # Or use --render (requires Graphviz installed) sql-splitter graph dump.sql -o schema.png --render ``` ### Wrong edge direction [Section titled “Wrong edge direction”](#wrong-edge-direction) Remember: edges point from the **referencing** table to the **referenced** table. If `orders.user_id → users.id`, the edge goes `orders → users` (orders depends on users). ## See Also [Section titled “See Also”](#see-also) * [`order`](/commands/order) - Reorder dump for safe imports * [`analyze`](/commands/analyze) - View table statistics * [`validate`](/commands/validate) - Check FK integrity * [JSON Output Schema](/reference/json-output/) - Schema for `--json` output # merge > Merge split SQL files back into a single dump Combine per-table SQL files into a single dump file. **Alias:** `mg` (e.g., `sql-splitter mg tables/ -o restored.sql`) ## When to Use This [Section titled “When to Use This”](#when-to-use-this) * **Reassembling after edits** - After using [`split`](/commands/split) to edit individual tables (e.g., with [`redact`](/commands/redact) or manual fixes), merge them back into a single deployable file * **Creating subset restores** - Build a dump containing only specific tables for partial database restores or testing * **Building migration files** - Combine schema/data for specific tables into a migration script for CI/CD deployment * **Streaming to databases** - Pipe merged output directly to `mysql` or `psql` for one-step restore * **Compressing output** - Pipe to `gzip` or `zstd` for compressed backups without intermediate files Use [`order`](/commands/order) first if you need foreign key-safe ordering—`merge` sorts alphabetically by filename, not by FK dependencies. ## How It Works [Section titled “How It Works”](#how-it-works) The merge process: 1. **Discover files** - Scans the input directory for all `.sql` files 2. **Filter** - Applies `--tables` (include) and `--exclude` filters 3. **Sort** - Sorts files **alphabetically by table name** (e.g., `accounts.sql` → `users.sql` → `zones.sql`) 4. **Stream** - Writes each file’s contents sequentially with separator comments **What gets included:** * All `.sql` files in the directory (non-recursive) * Each file is prefixed with a separator comment: `-- Table: tablename` * Files are streamed line-by-line (memory-efficient for large dumps) **What does NOT happen:** * `_global.sql` is treated like any other table file (sorted alphabetically with `_` at the start) * No FK dependency ordering—files are merged alphabetically. Use [`order`](/commands/order) on the output if you need topological order. * No deduplication—if a table appears in multiple files, both are included **Dialect-specific behavior:** * MySQL: Adds `SET FOREIGN_KEY_CHECKS = 0;` header, `SET FOREIGN_KEY_CHECKS = 1;` footer * PostgreSQL: Adds `SET client_encoding = 'UTF8';` header * SQLite: Adds `PRAGMA foreign_keys = OFF;` header, `PRAGMA foreign_keys = ON;` footer * MS SQL: Adds `SET ANSI_NULLS ON;`, `SET QUOTED_IDENTIFIER ON;`, `SET NOCOUNT ON;` headers ## Usage [Section titled “Usage”](#usage) ```bash sql-splitter merge [OPTIONS] ``` ## Examples [Section titled “Examples”](#examples) ### Basic Merge [Section titled “Basic Merge”](#basic-merge) ```bash # Merge all tables from a directory sql-splitter merge tables/ -o restored.sql # Preview what would be merged (no files written) sql-splitter merge tables/ --dry-run ``` ### Creating Subset Restores [Section titled “Creating Subset Restores”](#creating-subset-restores) ```bash # Merge only specific tables sql-splitter merge tables/ -o users-data.sql --tables users,profiles,settings # Merge everything except large or sensitive tables sql-splitter merge tables/ -o partial.sql --exclude logs,audit_trail,sessions ``` ### Atomic Restores with Transactions [Section titled “Atomic Restores with Transactions”](#atomic-restores-with-transactions) ```bash # Wrap entire merge in a transaction sql-splitter merge tables/ -o restored.sql --transaction ``` This produces a dump that either fully succeeds or fully rolls back on error. ### Streaming to Database [Section titled “Streaming to Database”](#streaming-to-database) ```bash # MySQL: Pipe directly (omit -o to write to stdout) sql-splitter merge tables/ | mysql -u user -p database # PostgreSQL sql-splitter merge tables/ --dialect postgres | psql "$PG_CONN" # SQLite sql-splitter merge tables/ --dialect sqlite | sqlite3 mydb.sqlite ``` ### Compressed Output [Section titled “Compressed Output”](#compressed-output) ```bash # Compress on the fly sql-splitter merge tables/ | gzip > merged.sql.gz sql-splitter merge tables/ | zstd > merged.sql.zst # With progress bar (written to stderr, doesn't interfere with pipe) sql-splitter merge tables/ --progress | gzip > merged.sql.gz ``` ### Minimal Output (No Headers) [Section titled “Minimal Output (No Headers)”](#minimal-output-no-headers) ```bash # Skip the generated header comments and dialect-specific SET statements sql-splitter merge tables/ -o raw.sql --no-header ``` ## Options [Section titled “Options”](#options) | Flag | Short | Description | Default | | --------------- | ----- | ----------------------------------------- | ------- | | `--output` | `-o` | Output SQL file | stdout | | `--dialect` | `-d` | SQL dialect for headers/footers | `mysql` | | `--tables` | `-t` | Only merge these tables (comma-separated) | all | | `--exclude` | `-e` | Exclude these tables (comma-separated) | none | | `--transaction` | | Wrap in BEGIN/COMMIT transaction | false | | `--no-header` | | Skip header comments | false | | `--progress` | `-p` | Show progress bar | false | | `--dry-run` | | Preview without writing files | false | | `--json` | | Output results as JSON | false | ## Transaction Wrapping [Section titled “Transaction Wrapping”](#transaction-wrapping) Use `--transaction` for atomic restores: ```bash sql-splitter merge tables/ -o restored.sql --transaction ``` Output (dialect-aware): ```sql -- MySQL START TRANSACTION; -- ... table contents ... COMMIT; -- PostgreSQL BEGIN; -- ... table contents ... COMMIT; -- SQLite / MS SQL BEGIN TRANSACTION; -- ... table contents ... COMMIT; ``` ## JSON Output [Section titled “JSON Output”](#json-output) ```bash sql-splitter merge tables/ -o merged.sql --json ``` ```json { "input_dir": "tables/", "output_file": "merged.sql", "dialect": "mysql", "dry_run": false, "statistics": { "tables_merged": 4, "bytes_written": 125000, "elapsed_secs": 0.015, "throughput_kb_per_sec": 8138.02 }, "tables": ["orders", "products", "users", "zones"], "options": { "transaction": false, "header": true } } ``` When writing to stdout (no `-o`), JSON is printed to stderr to avoid mixing with SQL output. ## Composing with Other Tools [Section titled “Composing with Other Tools”](#composing-with-other-tools) ### FK-Safe Merge [Section titled “FK-Safe Merge”](#fk-safe-merge) Merge then reorder for foreign key safety: ```bash sql-splitter merge tables/ -o temp.sql sql-splitter order temp.sql -o ordered.sql ``` ### Split → Edit → Merge Workflow [Section titled “Split → Edit → Merge Workflow”](#split--edit--merge-workflow) ```bash # Split the dump sql-splitter split dump.sql -o tables/ # Redact sensitive data in specific tables sql-splitter redact tables/users.sql -o tables/users.sql -c redact.yaml # Merge back together sql-splitter merge tables/ -o sanitized.sql ``` ### Merge Subset and Validate [Section titled “Merge Subset and Validate”](#merge-subset-and-validate) ```bash # Create subset dump sql-splitter merge tables/ -o subset.sql --tables users,orders,products # Validate FK integrity of the subset (PK/FK checks run by default) sql-splitter validate subset.sql ``` ### CI/CD Pipeline Integration [Section titled “CI/CD Pipeline Integration”](#cicd-pipeline-integration) ```bash # Generate deterministic output for diffing sql-splitter merge tables/ -o merged.sql sql-splitter order merged.sql -o ordered.sql git diff ordered.sql ``` ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) ### ”No .sql files found in directory” [Section titled “”No .sql files found in directory””](#no-sql-files-found-in-directory) The input path must be a directory containing `.sql` files: ```bash # Wrong: pointing to a file sql-splitter merge backup.sql -o out.sql # Error: no .sql files found in directory: backup.sql # Correct: point to directory created by split sql-splitter merge tables/ -o out.sql ``` ### Tables in Wrong Order for FK Constraints [Section titled “Tables in Wrong Order for FK Constraints”](#tables-in-wrong-order-for-fk-constraints) `merge` sorts files alphabetically, not by foreign key dependencies. If you get FK constraint errors during import: ```bash # Option 1: Reorder after merging sql-splitter merge tables/ -o merged.sql sql-splitter order merged.sql -o ordered.sql # Option 2: Use dialect header which disables FK checks sql-splitter merge tables/ -o merged.sql --dialect mysql # Header includes: SET FOREIGN_KEY_CHECKS = 0; # Option 3: Disable FK checks manually during import mysql -e "SET FOREIGN_KEY_CHECKS=0; SOURCE merged.sql; SET FOREIGN_KEY_CHECKS=1;" ``` ### Missing Tables in Output [Section titled “Missing Tables in Output”](#missing-tables-in-output) Check if tables were filtered out: ```bash # Preview what will be merged sql-splitter merge tables/ --dry-run # Check if table files exist ls tables/*.sql ``` Tables might be excluded by `--exclude` or not matched by `--tables` filter. Filters are **case-insensitive**. ### `_global.sql` Not at Start of Output [Section titled “\_global.sql Not at Start of Output”](#_globalsql-not-at-start-of-output) `_global.sql` (created by [`split`](/commands/split) for header statements like `SET NAMES`) is sorted alphabetically with other files. Since `_` sorts before letters, it typically appears first. If you need specific ordering: ```bash # Exclude _global from merge, prepend manually sql-splitter merge tables/ -o body.sql --exclude _global cat tables/_global.sql body.sql > complete.sql ``` ### JSON Output Mixed with SQL [Section titled “JSON Output Mixed with SQL”](#json-output-mixed-with-sql) When using `--json` without `-o`, both SQL and JSON would go to stdout. The command writes JSON to **stderr** instead: ```bash # JSON goes to stderr, SQL goes to stdout sql-splitter merge tables/ --json > merged.sql 2> stats.json # Or capture just the SQL sql-splitter merge tables/ --json > merged.sql ``` ### Out of Disk Space During Merge [Section titled “Out of Disk Space During Merge”](#out-of-disk-space-during-merge) The merge streams data and uses 256KB write buffers. For very large merges: ```bash # Check available space df -h . # Stream directly to compressed file (smaller output) sql-splitter merge tables/ | gzip > merged.sql.gz # Or pipe directly to database (no disk needed) sql-splitter merge tables/ | mysql -u user database ``` ## Exit Codes [Section titled “Exit Codes”](#exit-codes) | Code | Meaning | | ---- | ------------------------------------------------------ | | `0` | Success | | `1` | Error (no files found, write error, invalid directory) | ## See Also [Section titled “See Also”](#see-also) * [`split`](/commands/split) - Split a dump into per-table files (inverse of merge) * [`order`](/commands/order) - Reorder merged output for FK-safe imports * [`validate`](/commands/validate) - Check FK integrity after merging * [`redact`](/commands/redact) - Sanitize table data before merging * [JSON Output Schema](/reference/json-output/) - Full schema for `--json` output * [Glossary: Streaming](/reference/glossary#streaming) - How sql-splitter handles large files # order > Reorder SQL dump in topological FK order Reorder SQL dump statements in topological order based on foreign key dependencies for safe imports. **Alias:** `ord` (e.g., `sql-splitter ord dump.sql -o ordered.sql`) ## When to Use This [Section titled “When to Use This”](#when-to-use-this) * **Safe database restore** - Ensure parent tables are created before child tables to avoid FK constraint errors * **Stable diffs** - Normalize statement order before comparing dumps with [`diff`](/commands/diff) to eliminate ordering noise * **CI/CD pipelines** - Produce deterministic output from dumps that may have non-deterministic ordering * **DROP script generation** - Use `--reverse` to generate safe table drop order (children before parents) * **Cycle detection** - Use `--check` to find circular FK dependencies before migration Use [`graph`](/commands/graph) if you want to visualize the dependencies rather than reorder the dump. ## How It Works [Section titled “How It Works”](#how-it-works) The command performs a two-pass operation: 1. **First pass** - Parse the dump to extract: * CREATE TABLE statements (with FK relationships) * ALTER TABLE statements (for deferred FK definitions) * CREATE INDEX statements * INSERT/COPY statements (data) 2. **Second pass** - Rebuild the dump with statements grouped by table in topological order: * Header statements (SET, comments) come first * Tables ordered so parents come before children * For each table: CREATE TABLE → ALTER TABLE/CREATE INDEX → INSERT statements * SQL Server output writes a `GO` batch separator after each parsed statement **What gets reordered:** * CREATE TABLE statements * INSERT/COPY statements (grouped with their table) * ALTER TABLE statements (grouped with their table) * CREATE INDEX statements (grouped with their table) **What stays at the top (unchanged order):** * SET statements, session configuration * Comments, headers * Statements that don’t reference a specific table ## Usage [Section titled “Usage”](#usage) ```bash sql-splitter order [OPTIONS] ``` ## Examples [Section titled “Examples”](#examples) ### Basic Reordering [Section titled “Basic Reordering”](#basic-reordering) ```bash # Reorder for safe import sql-splitter order dump.sql -o ordered.sql # Preview the order without writing sql-splitter order dump.sql --dry-run ``` ### Validating Order (CI/CD) [Section titled “Validating Order (CI/CD)”](#validating-order-cicd) Use `--check` to verify ordering is possible and see the result without modifying anything: ```bash sql-splitter order dump.sql --check ``` **Exit codes:** * `0` - All tables can be ordered (no cycles) * `1` - Circular dependencies detected (cannot fully order) ### Generating DROP Scripts [Section titled “Generating DROP Scripts”](#generating-drop-scripts) Reverse the order so child tables come first (safe for DROP operations): ```bash sql-splitter order dump.sql --reverse -o drop-order.sql ``` This ensures you can run `DROP TABLE` statements without FK constraint violations. ### Normalizing for Diff [Section titled “Normalizing for Diff”](#normalizing-for-diff) When comparing two dumps, ordering differences cause noise. Normalize both first: ```bash sql-splitter order old.sql -o old-ordered.sql sql-splitter order new.sql -o new-ordered.sql sql-splitter diff old-ordered.sql new-ordered.sql ``` ## Options [Section titled “Options”](#options) | Flag | Short | Description | Default | | ----------- | ----- | ------------------------------------------------------------------- | ----------- | | `--output` | `-o` | Output SQL file | stdout | | `--dialect` | `-d` | SQL dialect | auto-detect | | `--check` | | Verify ordering is possible, report order, exit (no output written) | false | | `--dry-run` | | Show topological order to stderr (no output written) | false | | `--reverse` | | Reverse order (children before parents, for DROP) | false | ## Circular Dependencies [Section titled “Circular Dependencies”](#circular-dependencies) When tables have circular FK relationships (e.g., `orders → payments → invoices → orders`), a perfect topological order is impossible. ### What happens with cycles [Section titled “What happens with cycles”](#what-happens-with-cycles) 1. **Detection** - Cycles are detected using Kahn’s algorithm during topological sort 2. **Partial ordering** - Non-cyclic tables are ordered correctly 3. **Cyclic tables appended** - Tables in cycles are appended at the end of the output 4. **Warning printed** - You’ll see which tables are involved Example output: ```plaintext Warning: Circular dependencies detected! The following tables are part of cycles: - orders - payments - invoices Processed 15 tables in topological order. ``` ### Handling cycles during import [Section titled “Handling cycles during import”](#handling-cycles-during-import) Cyclic tables at the end of the output may cause FK errors during import. Options: **Option 1: Disable FK checks during import** ```bash # MySQL mysql -e "SET FOREIGN_KEY_CHECKS=0; SOURCE ordered.sql; SET FOREIGN_KEY_CHECKS=1;" # PostgreSQL (must be in same transaction) psql -c "BEGIN; SET CONSTRAINTS ALL DEFERRED;" -f ordered.sql -c "COMMIT;" ``` **Option 2: Analyze and refactor** ```bash # See exactly which tables form cycles sql-splitter graph dump.sql --cycles-only # Visualize the cycle sql-splitter graph dump.sql --cycles-only -o cycles.html ``` **Option 3: Split the dump** Use [`split`](/commands/split) to extract cyclic tables, import them separately with FK checks disabled. ### Exit code behavior with —check [Section titled “Exit code behavior with —check”](#exit-code-behavior-with-check) When using `--check`: * **No cycles** → exit 0, prints suggested order * **Cycles found** → exit 1, prints cyclic tables and suggests using `graph --cycles-only` ## Output Structure [Section titled “Output Structure”](#output-structure) The ordered dump follows this structure: ```sql -- 1. Header statements (SET, comments, session config) SET NAMES utf8mb4; SET FOREIGN_KEY_CHECKS=0; -- 2. Tables in topological order (parents first) -- For each table: CREATE → ALTER/INDEX → INSERT CREATE TABLE users (...); INSERT INTO users VALUES (...); CREATE TABLE orders (...); -- References users ALTER TABLE orders ADD INDEX idx_user (user_id); INSERT INTO orders VALUES (...); -- 3. Cyclic tables (if any) appended at end CREATE TABLE payments (...); -- Part of cycle INSERT INTO payments VALUES (...); ``` ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) ### ”No tables found in the file” [Section titled “”No tables found in the file””](#no-tables-found-in-the-file) The dump may not contain CREATE TABLE statements, or dialect detection failed: ```bash # Specify dialect explicitly sql-splitter order dump.sql --dialect postgres -o ordered.sql ``` ### Different order on each run [Section titled “Different order on each run”](#different-order-on-each-run) The order should be deterministic for the same input. If you see different orders: * Check if the input file itself is changing * Tables with no dependencies between them may appear in different relative order (this is correct - they’re interchangeable) ### Some INSERT statements missing [Section titled “Some INSERT statements missing”](#some-insert-statements-missing) INSERT statements are grouped with their table. If a table has INSERTs but no CREATE TABLE in the dump: ```bash # Check if the table exists in the dump sql-splitter analyze dump.sql | grep table_name ``` Data-only dumps (no DDL) can’t be reordered since there’s no schema to analyze. ### Cycle detection differs from graph command [Section titled “Cycle detection differs from graph command”](#cycle-detection-differs-from-graph-command) Both use the same algorithm, but `order --check` lists tables in cycles, while `graph --cycles-only` shows the actual cycle paths (A → B → C → A). Use `graph` for detailed analysis. ## See Also [Section titled “See Also”](#see-also) * [`graph`](/commands/graph) - Visualize table relationships and analyze cycles * [`diff`](/commands/diff) - Compare dumps (use `order` first for stable comparison) * [`validate`](/commands/validate) - Check dump integrity * [`split`](/commands/split) - Split dump by table (useful for handling cyclic tables separately) # query > Run SQL analytics on dump files using DuckDB Execute SQL queries directly on dump files without restoring to a database, powered by DuckDB. **Alias:** `qy` (e.g., `sql-splitter qy dump.sql "SELECT COUNT(*) FROM users"`) ## When to Use This [Section titled “When to Use This”](#when-to-use-this) * **Quick data exploration** - Answer questions about a dump without spinning up a database * **Data validation** - Verify row counts, check for nulls, or validate business rules before restoring * **Extract subsets** - Pull specific records to CSV/JSON for analysis or reporting * **Schema investigation** - Explore table structures and relationships interactively * **CI pipelines** - Automate data quality checks on dump files Use [`analyze`](/commands/analyze) instead if you only need basic statistics (row counts, table sizes) - it’s faster and doesn’t require loading data. ## How It Works [Section titled “How It Works”](#how-it-works) The command parses your SQL dump, imports the schema and data into an in-memory DuckDB database, then executes your query. **Disk mode (`--disk`):** For dumps >2GB, use `--disk` to store data in a temporary file instead of RAM. The temp file is stored in your system’s temp directory and **automatically deleted** when the command exits: | OS | Temp file location | | ------- | ----------------------------------------------------------------------------------- | | macOS | `/var/folders/.../sql-splitter-.duckdb` (via `$TMPDIR`) | | Linux | `/tmp/sql-splitter-.duckdb` | | Windows | `%TEMP%\sql-splitter-.duckdb` (e.g., `C:\Users\\AppData\Local\Temp\...`) | **Caching (`--cache`):** Saves the imported database persistently for fast repeated queries. Cache files are stored in your platform’s cache directory and **persist until you run `--clear-cache`**: | OS | Cache directory | | ------- | -------------------------------------------------------------------------------------------------- | | macOS | `~/Library/Caches/sql-splitter/duckdb/` | | Linux | `~/.cache/sql-splitter/duckdb/` | | Windows | `%LOCALAPPDATA%\sql-splitter\duckdb\` (e.g., `C:\Users\\AppData\Local\sql-splitter\duckdb\`) | Set the `SQL_SPLITTER_CACHE_DIR` environment variable to override the cache location. Cache files are named `.duckdb` where the hash covers the dump file’s path, size, and modification time, plus the `--tables` selection and `--dialect`. Each table selection gets its own cache, so a cache built with `--tables users` and a full-dump cache coexist. If the source dump changes, a new cache is created automatically and caches of the old version are cleaned up. ## Usage [Section titled “Usage”](#usage) ```bash sql-splitter query [QUERY] [OPTIONS] ``` ## Examples [Section titled “Examples”](#examples) ### Basic Queries [Section titled “Basic Queries”](#basic-queries) ```bash # Count rows in a table sql-splitter query dump.sql "SELECT COUNT(*) FROM users" # Filter and inspect data sql-splitter query dump.sql "SELECT * FROM orders WHERE total > 100" # Join across tables sql-splitter query dump.sql "SELECT u.name, COUNT(o.id) FROM users u JOIN orders o ON u.id = o.user_id GROUP BY u.name" ``` ### Exporting Results [Section titled “Exporting Results”](#exporting-results) Use `-f` to choose the output format and `-o` to write to a file: ```bash # Export to CSV for spreadsheet analysis sql-splitter query dump.sql "SELECT * FROM users" -o users.csv -f csv # Export to JSON for downstream tooling sql-splitter query dump.sql "SELECT * FROM orders WHERE status = 'pending'" -f json # Stream JSON Lines for large result sets (one JSON object per line) sql-splitter query dump.sql "SELECT * FROM events" -f jsonl ``` ### Interactive Exploration [Section titled “Interactive Exploration”](#interactive-exploration) When you don’t know what you’re looking for yet, use the REPL: ```bash sql-splitter query dump.sql --interactive ``` ### Performance Optimization [Section titled “Performance Optimization”](#performance-optimization) ```bash # Large dumps: use disk storage instead of RAM sql-splitter query huge.sql "SELECT ..." --disk # Repeated queries: cache the imported database (400x faster on subsequent runs) sql-splitter query dump.sql "SELECT COUNT(*) FROM users" --cache sql-splitter query dump.sql "SELECT * FROM users WHERE active = 1" --cache # Uses cache # Import only tables you need (faster import, less memory) sql-splitter query dump.sql "SELECT * FROM orders" --tables orders,users ``` ## Options [Section titled “Options”](#options) | Flag | Short | Description | Default | | ---------------- | ----- | ------------------------------------------------------- | ---------------- | | `--format` | `-f` | Output format: `table`, `json`, `jsonl`, `csv`, `tsv` | `table` | | `--output` | `-o` | Write output to file | stdout | | `--dialect` | `-d` | SQL dialect | auto-detect | | `--interactive` | `-i` | Start interactive REPL | false | | `--disk` | | Force disk-based storage (auto-enabled for large dumps) | false | | `--cache` | | Cache imported database for faster repeated queries | false | | `--tables` | | Only import specific tables (comma-separated) | all | | `--memory-limit` | | DuckDB memory limit (e.g., “4GB”) | - | | `--timing` | | Show query execution time | false | | `--progress` | | Show import progress | on in a terminal | | `--list-cache` | | List cached databases | - | | `--clear-cache` | | Clear all cached databases | - | **Note:** Use `--format json` or `--format jsonl` for JSON output (not `--json`). ## Output Formats [Section titled “Output Formats”](#output-formats) ### Table (default) [Section titled “Table (default)”](#table-default) ```plaintext ┌────┬─────────┬─────────────────────┐ │ id │ name │ email │ ├────┼─────────┼─────────────────────┤ │ 1 │ Alice │ alice@example.com │ │ 2 │ Bob │ bob@example.com │ └────┴─────────┴─────────────────────┘ ``` ### JSON [Section titled “JSON”](#json) ```json [ { "id": 1, "name": "Alice", "email": "alice@example.com" }, { "id": 2, "name": "Bob", "email": "bob@example.com" } ] ``` ### CSV [Section titled “CSV”](#csv) ```csv id,name,email 1,Alice,alice@example.com 2,Bob,bob@example.com ``` ## Interactive REPL [Section titled “Interactive REPL”](#interactive-repl) ```bash sql-splitter query dump.sql --interactive ``` REPL commands: | Command | Description | | ------------------------ | ------------------------------ | | `.tables` | List all tables | | `.schema [table]` | Show schema | | `.describe ` | Describe table columns | | `.format ` | Set output format | | `.count ` | Count rows | | `.sample [n]` | Show sample rows (default: 10) | | `.export ` | Export query to file | | `.exit` | Exit REPL | ## Caching [Section titled “Caching”](#caching) Cache the imported database for faster repeated queries: ```bash # First query imports and caches (~30s for large dump) sql-splitter query dump.sql "SELECT COUNT(*) FROM users" --cache # Subsequent queries use cache (~0.1s - 400x faster) sql-splitter query dump.sql "SELECT * FROM users WHERE active = 1" --cache ``` Manage cache: ```bash # List cached databases (shows source file, contained tables, size) sql-splitter query --list-cache # Clear all caches (reclaims disk space) sql-splitter query --clear-cache ``` The cache key covers the dump file (path, size, modification time), the `--tables` selection, and `--dialect` — changing any of these triggers a re-import. When an existing cache of the same dump covers a different table set, the re-import says so: ```plaintext Note: existing cache covers {users}; requested {users, orders} - re-importing ``` Cache writes are atomic: an interrupted import never corrupts or removes previously cached databases. To manually clear space, run `--clear-cache` or delete files from the cache directory (see [How It Works](#how-it-works) for paths). ## Large Files [Section titled “Large Files”](#large-files) For dumps larger than available RAM (automatically enabled for files >2GB): ```bash sql-splitter query huge.sql "SELECT ..." --disk ``` Disk mode stores data in a temporary DuckDB file instead of memory. The temp file is automatically deleted when the command exits. If the command crashes or is killed, you may need to manually clean up orphaned files from your temp directory (see [How It Works](#how-it-works) for paths). ## Composing with Other Tools [Section titled “Composing with Other Tools”](#composing-with-other-tools) Combine `query` with shell tools for powerful pipelines: ```bash # Extract user emails and process with jq sql-splitter query dump.sql "SELECT email FROM users" -f json | jq -r '.[].email' # Find tables with orphaned foreign keys sql-splitter query dump.sql "SELECT * FROM orders WHERE user_id NOT IN (SELECT id FROM users)" -f csv # Quick per-table stats without a full import (analyze is faster) sql-splitter analyze dump.sql --json | jq -r '.tables[] | "\(.name): \(.inserts) inserts"' # Export and compress in one step sql-splitter query dump.sql "SELECT * FROM large_table" -f csv | gzip > export.csv.gz ``` ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) ### Query returns zero results [Section titled “Query returns zero results”](#query-returns-zero-results) * **Check table names** - Use `.tables` in interactive mode to list available tables * **Case sensitivity** - DuckDB is case-insensitive for identifiers, but string comparisons are case-sensitive * **Dialect mismatch** - If auto-detection fails, try `--dialect mysql` or `--dialect postgres` ### Import is slow or runs out of memory [Section titled “Import is slow or runs out of memory”](#import-is-slow-or-runs-out-of-memory) * Use `--disk` for dumps larger than available RAM * Use `--tables` to import only the tables you need * Consider `--cache` if you’ll run multiple queries ### ”Table not found” error [Section titled “”Table not found” error”](#table-not-found-error) The dump may use schema-qualified names. Try: ```bash sql-splitter query dump.sql --interactive # then run .tables to list all tables sql-splitter query dump.sql "SELECT * FROM public.users" # Use schema prefix ``` ### Cache issues [Section titled “Cache issues”](#cache-issues) ```bash # List cached databases to see what's stored sql-splitter query --list-cache # Clear cache if data has changed sql-splitter query --clear-cache ``` ## See Also [Section titled “See Also”](#see-also) * [`analyze`](/commands/analyze) - Quick statistics without full import * [DuckDB SQL Reference](https://duckdb.org/docs/sql/introduction) # redact > Anonymize sensitive data (PII) in SQL dumps Replace sensitive data with anonymized values for safe sharing, development, or compliance. **Alias:** `rd` (e.g., `sql-splitter rd dump.sql --hash "*.email" -o safe.sql`) ## When to Use This [Section titled “When to Use This”](#when-to-use-this) * **GDPR/CCPA compliance** - Remove or anonymize [PII](/reference/glossary#pii-personally-identifiable-information) before sharing dumps with external parties * **Vendor data sharing** - Provide realistic data to contractors or consultants without exposing customer information * **Development environment seeding** - Create anonymized copies of production data for local development * **Testing data pipelines** - Validate ETL processes with structurally identical but safe data * **Compliance audits** - Demonstrate PII handling practices with sanitized sample exports Use [`sample`](/commands/sample) first if you also need to reduce the dataset size. ## How It Works [Section titled “How It Works”](#how-it-works) Redaction runs as a [streaming](/reference/glossary#streaming) single-pass operation: 1. **Parse statements** - Reads each SQL statement (INSERT, COPY) from the input 2. **Match columns** - Compares column names against your patterns (e.g., `*.email` matches `users.email`, `orders.customer_email`) 3. **Apply strategy** - Transforms matching values using the specified strategy (null, hash, fake, mask, etc.) 4. **Write output** - Emits transformed statements immediately (no buffering of entire file) **Memory characteristics:** Constant \~10MB regardless of file size. Only the current statement is held in memory. **Hashing behavior:** The `hash` strategy uses SHA-256 and is [deterministic](/reference/glossary#deterministic)—the same input value always produces the same hash. This preserves [foreign key](/reference/glossary#foreign-key-fk) relationships: if `users.email` and `audit_log.user_email` both contain `alice@example.com`, they both hash to the same value. **Fake data with `--seed`:** When using `fake` strategy, the same `--seed` value produces identical fake data across runs. The fake value is derived from the original value’s hash, so `alice@example.com` always becomes the same fake email (e.g., `jessica.smith@example.com`) with a given seed. **Exit codes:** * `0` - Success * `1` - Error (validation errors, `--strict` mode warnings, file not found, parse failure) * `2` - Invalid CLI arguments ## Usage [Section titled “Usage”](#usage) ```bash sql-splitter redact [OPTIONS] ``` ## Examples [Section titled “Examples”](#examples) ### Basic Redaction with Inline Patterns [Section titled “Basic Redaction with Inline Patterns”](#basic-redaction-with-inline-patterns) ```bash # Hash emails, null SSNs, fake names sql-splitter redact dump.sql -o safe.sql \ --null "*.ssn" \ --hash "*.email" \ --fake "*.name" # Mask credit cards (show last 4 digits) sql-splitter redact dump.sql -o safe.sql \ --mask "****-****-****-XXXX=*.credit_card" # Replace passwords with a constant value sql-splitter redact dump.sql -o safe.sql \ --constant "*.password=REDACTED" ``` ### Using Configuration Files [Section titled “Using Configuration Files”](#using-configuration-files) ```bash # Use YAML config for complex rules sql-splitter redact dump.sql -o safe.sql --config redact.yaml # Auto-generate config by analyzing column names sql-splitter redact dump.sql --generate-config -o redact.yaml # Validate config without processing sql-splitter redact dump.sql --config redact.yaml --validate ``` ### Reproducible Redaction [Section titled “Reproducible Redaction”](#reproducible-redaction) ```bash # Same seed = identical fake data across runs sql-splitter redact dump.sql -o safe.sql --fake "*.name" --seed 42 # Useful for CI: consistent test fixtures sql-splitter redact prod.sql -o fixtures/test-data.sql \ --config redact.yaml --seed 12345 ``` ### Targeting Specific Tables [Section titled “Targeting Specific Tables”](#targeting-specific-tables) ```bash # Only redact the users and orders tables sql-splitter redact dump.sql -o safe.sql \ --hash "*.email" --tables users,orders # Redact everything except audit_logs sql-splitter redact dump.sql -o safe.sql \ --hash "*.email" --exclude audit_logs ``` For a schema-qualified dump, use the full table name in `--tables` or `--exclude` to avoid applying the filter to the same table name in another schema. A rule for one qualified table uses the same final-column form, for example `tenant_a.users.email`. ## Options [Section titled “Options”](#options) | Flag | Short | Description | Default | | ------------------- | ----- | ------------------------------------------ | ----------- | | `--output` | `-o` | Output SQL file | stdout | | `--dialect` | `-d` | SQL dialect | auto-detect | | `--config` | `-c` | YAML config file | - | | `--generate-config` | | Analyze input and generate YAML config | false | | `--null` | | Columns to set to NULL (glob patterns) | - | | `--hash` | | Columns to hash with SHA256 | - | | `--fake` | | Columns to replace with fake data | - | | `--mask` | | Columns to mask (format: `pattern=column`) | - | | `--constant` | | Column=value pairs | - | | `--seed` | | Random seed for reproducibility | random | | `--locale` | | Locale for fake data | `en` | | `--tables` | `-t` | Only redact specific tables | all | | `--exclude` | `-x` | Exclude specific tables | none | | `--strict` | | Fail on warnings | false | | `--progress` | `-p` | Show progress bar | false | | `--dry-run` | | Preview without writing | false | | `--json` | | Output as JSON | false | | `--validate` | | Validate config only | false | ## Redaction Strategies [Section titled “Redaction Strategies”](#redaction-strategies) | Strategy | Description | Use Case | | ---------- | ---------------------------------------------------------------- | --------------------------------------------------------------- | | `null` | Replace with NULL | Remove data entirely | | `hash` | SHA256 hash ([deterministic](/reference/glossary#deterministic)) | Preserve [FK](/reference/glossary#foreign-key-fk) relationships | | `fake` | Generate realistic fake data | Realistic test data | | `mask` | Partial masking | Credit cards, SSNs | | `constant` | Fixed value | Placeholder values | | `skip` | No redaction | Passthrough | `shuffle` rules are rejected. They require a two-pass rewrite and are not safe to apply in the current streaming implementation. ## Column Pattern Matching [Section titled “Column Pattern Matching”](#column-pattern-matching) Patterns use `table.column` format with `*` as a wildcard: | Pattern | Matches | | -------------------- | ------------------------------------------------------------------- | | `*.email` | `email` column in any table (`users.email`, `orders.contact_email`) | | `users.email` | Only the `email` column in the `users` table | | `users.*` | All columns in the `users` table | | `*.ssn` | Any column named `ssn` in any table | | `audit_*.created_by` | `created_by` in tables starting with `audit_` | Patterns are matched in order—first matching rule wins. ## Mask Patterns [Section titled “Mask Patterns”](#mask-patterns) ```bash # Keep last 4 digits of credit card sql-splitter redact dump.sql -o safe.sql --mask "****-****-****-XXXX=*.credit_card" ``` Pattern symbols: * `*` - Replace with asterisk * `X` - Keep original character * `#` - Random digit ## Fake Data Generators [Section titled “Fake Data Generators”](#fake-data-generators) 25+ generators available: | Generator | Example Output | | ------------- | --------------------------------- | | `email` | `jessica.smith@example.com` | | `name` | `Robert Johnson` | | `first_name` | `Sarah` | | `last_name` | `Williams` | | `phone` | `+1 (555) 234-5678` | | `address` | `123 Oak Street, Springfield, IL` | | `city` | `Portland` | | `company` | `Acme Corporation` | | `ip` | `192.168.1.100` | | `uuid` | `550e8400-e29b-41d4-...` | | `date` | `1985-07-23` | | `credit_card` | `4532015112830366` | | `ssn` | `123-45-6789` | ## Auto-Generating Configuration [Section titled “Auto-Generating Configuration”](#auto-generating-configuration) The `--generate-config` flag analyzes column names to detect common [PII](/reference/glossary#pii-personally-identifiable-information) patterns: ```bash sql-splitter redact dump.sql --generate-config -o redact.yaml ``` **Detected patterns include:** * Email: `*email*`, `*e_mail*` * Phone: `*phone*`, `*mobile*`, `*tel*` * Names: `*name*`, `*first_name*`, `*last_name*` * Financial: `*ssn*`, `*social_security*`, `*credit_card*`, `*card_number*` * Addresses: `*address*`, `*street*`, `*zip*`, `*postal*` * Auth: `*password*`, `*secret*`, `*token*` Review and adjust the generated config before use—automated detection may miss custom column names or flag non-sensitive columns. ## YAML Config [Section titled “YAML Config”](#yaml-config) ```yaml seed: 12345 locale: en rules: - column: "*.ssn" strategy: "null" # quote it—unquoted null is a YAML null value - column: "*.email" strategy: hash - column: "*.name" strategy: fake generator: name - column: "*.credit_card" strategy: mask pattern: "****-****-****-XXXX" skip_tables: - schema_migrations ``` ## Composing with Other Tools [Section titled “Composing with Other Tools”](#composing-with-other-tools) ### Sample Then Redact [Section titled “Sample Then Redact”](#sample-then-redact) Reduce dataset size before anonymizing for faster processing: ```bash sql-splitter sample prod.sql -o sampled.sql --percent 10 --preserve-relations sql-splitter redact sampled.sql -o dev.sql --config redact.yaml ``` ### Validate After Redaction [Section titled “Validate After Redaction”](#validate-after-redaction) Ensure redacted output is valid SQL: ```bash sql-splitter redact dump.sql -o safe.sql --config redact.yaml sql-splitter validate safe.sql --strict ``` ### Full Dev Environment Pipeline [Section titled “Full Dev Environment Pipeline”](#full-dev-environment-pipeline) Create a complete anonymized dev dataset: ```bash # 1. Sample 5% with FK preservation sql-splitter sample prod.sql -o sampled.sql \ --percent 5 --preserve-relations --seed 42 # 2. Redact PII sql-splitter redact sampled.sql -o dev.sql \ --config redact.yaml --seed 42 # 3. Validate and reorder for clean import sql-splitter validate dev.sql --strict && \ sql-splitter order dev.sql -o dev-ordered.sql ``` ### Convert and Redact for Different Environments [Section titled “Convert and Redact for Different Environments”](#convert-and-redact-for-different-environments) ```bash # Redact MySQL dump, convert to PostgreSQL for testing sql-splitter redact mysql.sql -o safe.sql --config redact.yaml sql-splitter convert safe.sql -o postgres.sql --to postgres ``` ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) ### Hashed values break foreign key relationships [Section titled “Hashed values break foreign key relationships”](#hashed-values-break-foreign-key-relationships) **Symptom:** FK errors when importing redacted dump; parent/child relationships are broken. **Cause:** Only one side of a relationship was hashed, or different patterns matched parent and child columns. **Solutions:** 1. Ensure both sides use the same pattern: `--hash "*.user_id"` matches in all tables 2. Use explicit patterns for both: `--hash "users.id" --hash "orders.user_id"` 3. Verify with: `sql-splitter validate redacted.sql` ### Fake data repeats or is inconsistent across runs [Section titled “Fake data repeats or is inconsistent across runs”](#fake-data-repeats-or-is-inconsistent-across-runs) **Symptom:** Different fake values each time you run redaction, or unexpected repetition. **Cause:** No `--seed` specified (random seed each run), or same original values produce same fake values (by design). **Solutions:** 1. Use `--seed` for reproducible output: `--seed 42` 2. Same original value → same fake value is intentional (preserves consistency) 3. For truly random fake data each run, omit `--seed` ### Column not being redacted [Section titled “Column not being redacted”](#column-not-being-redacted) **Symptom:** Sensitive data appears in output despite having a matching rule. **Cause:** Pattern doesn’t match the actual table.column name, or an earlier rule with `skip` strategy matched first. **Solutions:** 1. Check exact column names: `sql-splitter analyze dump.sql` shows all table/column names 2. Use `--dry-run` to preview what would be redacted 3. Remember patterns match in order—more specific patterns should come before `*.column` wildcards 4. Verify pattern syntax: `users.email` not `users->email` ### Config validation errors [Section titled “Config validation errors”](#config-validation-errors) **Symptom:** “Invalid config” or “Unknown strategy” errors. **Cause:** YAML syntax errors, typos in strategy names, or invalid pattern format. **Solutions:** 1. Validate before running: `--validate` flag 2. Check strategy spelling: `null`, `hash`, `fake`, `mask`, `constant`, `skip` 3. Ensure patterns are quoted in YAML: `column: "*.email"` not `column: *.email` 4. Use `--generate-config` as a starting template ### ”Unknown column” warnings [Section titled “”Unknown column” warnings”](#unknown-column-warnings) **Symptom:** Warnings about rules that didn’t match any columns. **Cause:** Pattern doesn’t match any columns in the dump (typo or columns don’t exist). **Solutions:** 1. Run `sql-splitter analyze dump.sql` to see actual column names 2. Use `--strict` to fail on unmatched rules (catches config errors early) 3. Check for schema changes since config was created ## See Also [Section titled “See Also”](#see-also) * [`sample`](/commands/sample) - Create reduced datasets before redacting * [`validate`](/commands/validate) - Verify redacted output integrity * [`analyze`](/commands/analyze) - List tables and columns for pattern planning * [`convert`](/commands/convert) - Convert redacted dumps to different dialects * [Redact Config Reference](/reference/redact-config) - Full YAML configuration options * [JSON Output Schema](/reference/json-output/) - Schema for `--json` output * [Memory & Streaming](/advanced/memory-streaming) - Detailed memory profiles per command * [Glossary](/reference/glossary) - PII, hashing, and streaming terminology # sample > Create reduced datasets with FK preservation Create smaller test datasets while preserving foreign key relationships. **Alias:** `sa` (e.g., `sql-splitter sa dump.sql --percent 10 -o dev.sql`) Note Requires one of: `--percent`, `--rows`, or `--config` ## When to Use This [Section titled “When to Use This”](#when-to-use-this) * **Creating dev/test datasets** - Generate smaller databases for local development from production dumps * **Reducing CI test times** - Smaller datasets mean faster test runs without losing coverage * **Reproducing production bugs** - Extract a consistent slice of data that maintains relationships between tables * **Size-limited backups** - Create smaller backups for quick restores or archival purposes * **Demo environments** - Build realistic demo data that maintains [referential integrity](/reference/glossary#referential-integrity) Use [`shard`](/commands/shard) instead if you need to extract data for a specific tenant from a multi-tenant database. ## How It Works [Section titled “How It Works”](#how-it-works) The sample command operates in one of two modes: ### Simple Random Sampling (default) [Section titled “Simple Random Sampling (default)”](#simple-random-sampling-default) Without `--preserve-relations`, each table is sampled independently using reservoir sampling. Rows are selected randomly based on `--percent` or `--rows`. This is fast but doesn’t guarantee [foreign key](/reference/glossary#foreign-key-fk) relationships remain valid. ### FK-Preserving Sampling (`--preserve-relations`) [Section titled “FK-Preserving Sampling (--preserve-relations)”](#fk-preserving-sampling---preserve-relations) When FK preservation is enabled, the algorithm: 1. **Splits the dump** into per-table files in a temp directory 2. **Builds a [dependency graph](/reference/glossary#dependency-graph)** from CREATE TABLE and ALTER TABLE statements 3. **Identifies [root tables](/reference/glossary#root-table)** - tables with no outgoing foreign keys (or explicitly specified via `--root-tables`) 4. **Processes tables in [topological order](/reference/glossary#topological-order--topological-sort)** - parents before children 5. **Samples root tables first** using the specified percentage or row count 6. **Follows FK chains** - for each sampled row, tracks its [primary key](/reference/glossary#primary-key-pk) hash. When processing child tables, only rows whose FK values match a sampled parent’s PK are eligible 7. **Outputs a consistent dataset** where all FK relationships are satisfied **Memory usage**: PK values are stored as 64-bit hashes in a `HashSet` (8 bytes per row). For 10 million sampled rows across all tables, expect \~80MB of memory for PK tracking. Use `--max-total-rows` to cap memory growth. **Fractional rows**: When a percentage results in a fractional row count (e.g., 10% of 15 rows = 1.5), reservoir sampling naturally handles this—you’ll get approximately the target percentage over many runs, or use `--seed` for deterministic results. ## Usage [Section titled “Usage”](#usage) ```bash sql-splitter sample [OPTIONS] ``` ## Examples [Section titled “Examples”](#examples) ### Basic Sampling [Section titled “Basic Sampling”](#basic-sampling) ```bash # Sample 10% of rows from each table sql-splitter sample dump.sql -o sampled.sql --percent 10 # Sample fixed number of rows per table sql-splitter sample dump.sql -o sampled.sql --rows 1000 ``` ### FK-Preserving Samples [Section titled “FK-Preserving Samples”](#fk-preserving-samples) ```bash # Preserve FK relationships - ensures if you sample an order, the user exists sql-splitter sample dump.sql -o dev.sql --percent 10 --preserve-relations # Fail if any FK integrity issues are detected sql-splitter sample dump.sql -o dev.sql --percent 10 --preserve-relations --strict-fk ``` ### Reproducible Sampling [Section titled “Reproducible Sampling”](#reproducible-sampling) ```bash # Same seed = identical sample every time (useful for CI/debugging) sql-splitter sample dump.sql -o dev.sql --percent 10 --seed 42 ``` ### Table Filtering [Section titled “Table Filtering”](#table-filtering) ```bash # Sample only specific tables sql-splitter sample dump.sql -o dev.sql --percent 10 --tables users,orders,products # Exclude large or irrelevant tables sql-splitter sample dump.sql -o dev.sql --percent 10 --exclude logs,events,audit_trail ``` ### Using YAML Config for Per-Table Settings [Section titled “Using YAML Config for Per-Table Settings”](#using-yaml-config-for-per-table-settings) ```bash # Use a config file for complex sampling requirements sql-splitter sample dump.sql -o dev.sql --config sample.yaml ``` ## Options [Section titled “Options”](#options) | Flag | Short | Description | Default | | ---------------------- | ----- | ----------------------------------------------- | ----------- | | `--output` | `-o` | Output SQL file | stdout | | `--dialect` | `-d` | SQL dialect | auto-detect | | `--percent` | | Sample percentage (1-100) | - | | `--rows` | | Sample fixed number of rows per table | - | | `--preserve-relations` | | Preserve FK relationships | false | | `--tables` | `-t` | Only sample these tables (comma-separated) | all | | `--exclude` | `-e` | Exclude these tables (comma-separated) | none | | `--root-tables` | | Explicit root tables for sampling | auto | | `--include-global` | | Global table handling: `none`, `lookups`, `all` | `lookups` | | `--seed` | | Random seed for reproducibility | random | | `--config` | `-c` | YAML config file for per-table settings | - | | `--max-total-rows` | | Maximum total rows to sample | unlimited | | `--no-limit` | | Disable row limit | false | | `--strict-fk` | | Fail if any FK integrity issues | false | | `--no-schema` | | Exclude CREATE TABLE statements | false | | `--progress` | `-p` | Show progress bar | false | | `--dry-run` | | Preview without writing | false | | `--json` | | Output as JSON | false | ## Root Tables and `--root-tables` [Section titled “Root Tables and --root-tables”](#root-tables-and---root-tables) [Root tables](/reference/glossary#root-table) are tables with no outgoing foreign keys—they don’t reference other tables. The sample command auto-detects these as starting points for FK-preserving sampling. Override auto-detection when: * Your FKs are implicit (not declared in the schema) * You want to start sampling from specific tables * Auto-detection picks the wrong tables ```bash # Explicitly specify which tables to start sampling from sql-splitter sample dump.sql -o dev.sql --percent 10 --preserve-relations --root-tables users,organizations ``` ## Global and Lookup Tables [Section titled “Global and Lookup Tables”](#global-and-lookup-tables) The `--include-global` option controls how lookup/reference tables are handled: | Value | Behavior | | --------- | --------------------------------------------- | | `none` | Skip global/lookup tables entirely | | `lookups` | Include small lookup tables in full (default) | | `all` | Include all global tables in full | **What counts as a lookup table?** Tables are auto-classified as lookups if they match common patterns: * `countries`, `states`, `provinces`, `cities` * `currencies`, `languages`, `timezones` * `permissions`, `roles`, `settings` You can override classification in a [YAML config file](#yaml-configuration-file). **System tables** (migrations, jobs, cache, sessions) are automatically skipped. ## YAML Configuration File [Section titled “YAML Configuration File”](#yaml-configuration-file) For complex sampling requirements, use a YAML config file: sample.yaml ```yaml default: percent: 10 classification: # Include these tables fully global: - permissions - roles # Skip these tables system: - migrations - failed_jobs - telescope_entries # Classify as lookup tables lookup: - currencies - countries # Explicitly mark as root tables root: - organizations tables: users: rows: 500 # Fixed row count for this table posts: percent: 5 # Different percentage for this table sessions: skip: true # Skip entirely audit_logs: classification: system # Override classification ``` Then run: ```bash sql-splitter sample dump.sql -o dev.sql --config sample.yaml --preserve-relations ``` ## Composing with Other Tools [Section titled “Composing with Other Tools”](#composing-with-other-tools) ### Create anonymized dev database [Section titled “Create anonymized dev database”](#create-anonymized-dev-database) ```bash # Sample 10% with FK preservation, then redact PII sql-splitter sample prod.sql --percent 10 --preserve-relations -o sampled.sql sql-splitter redact sampled.sql --config redact.yaml -o dev.sql ``` ### Validate sampled output [Section titled “Validate sampled output”](#validate-sampled-output) ```bash # Ensure the sample maintains FK integrity sql-splitter sample dump.sql --percent 10 --preserve-relations -o dev.sql sql-splitter validate dev.sql --strict ``` ### Visualize dependencies before sampling [Section titled “Visualize dependencies before sampling”](#visualize-dependencies-before-sampling) ```bash # Understand the FK graph before deciding on root tables sql-splitter graph dump.sql --format mermaid -o schema.md sql-splitter sample dump.sql --percent 10 --preserve-relations --root-tables users,orgs -o dev.sql ``` ### Sample then query [Section titled “Sample then query”](#sample-then-query) ```bash # Create sample, then run ad-hoc queries against it sql-splitter sample dump.sql --percent 5 --preserve-relations -o sample.sql sql-splitter query sample.sql "SELECT COUNT(*) FROM users" ``` ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) ### Sample is larger than expected [Section titled “Sample is larger than expected”](#sample-is-larger-than-expected) **Cause**: FK preservation pulls in all related rows. If you sample 10% of orders, you get 100% of the users who placed those orders. **Solutions**: * Use `--max-total-rows` to cap the total: `--max-total-rows 100000` * Sample from tables higher in the dependency chain (use `--root-tables`) * Use `--dry-run` to preview sizes before sampling ```bash # Preview what would be sampled sql-splitter sample dump.sql --percent 10 --preserve-relations --dry-run --json | jq '.tables' ``` ### Missing related rows (broken FKs) [Section titled “Missing related rows (broken FKs)”](#missing-related-rows-broken-fks) **Cause**: Foreign keys weren’t detected in the schema (implicit FKs, or defined in application code). **Solutions**: * Specify root tables explicitly: `--root-tables users,organizations` * Add FK constraints to your schema dump * Use `--strict-fk` to fail early when FK issues are detected ### Out of memory [Section titled “Out of memory”](#out-of-memory) **Cause**: `--preserve-relations` tracks all sampled PK values in memory (8 bytes per row). **Solutions**: * Reduce sample size: `--percent 1` instead of `--percent 10` * Use `--max-total-rows 500000` to cap memory usage * Disable FK preservation for very large dumps (accept potential broken FKs) * Sample fewer tables: `--tables critical_table1,critical_table2` ### Non-deterministic results [Section titled “Non-deterministic results”](#non-deterministic-results) **Cause**: Without `--seed`, sampling uses a random seed each run. **Solution**: Add a fixed seed for reproducible results: ```bash sql-splitter sample dump.sql --percent 10 --seed 42 -o dev.sql ``` ### “Warning: Reached max\_total\_rows limit” [Section titled ““Warning: Reached max\_total\_rows limit””](#warning-reached-max_total_rows-limit) **Cause**: The `--max-total-rows` guard stopped sampling to prevent runaway growth. **Solutions**: * Increase the limit if you have sufficient memory * Reduce `--percent` to sample fewer rows initially * Review which tables are pulling in the most related rows with `--dry-run --json` ## Exit Codes [Section titled “Exit Codes”](#exit-codes) | Code | Meaning | | ---- | ----------------------------------------------------------------- | | `0` | Success | | `1` | Error (I/O, parse error, FK integrity failure with `--strict-fk`) | | `2` | Invalid arguments | See [Exit Codes](/reference/exit-codes) for usage in scripts. ## See Also [Section titled “See Also”](#see-also) * [`shard`](/commands/shard) - Extract tenant-specific data * [`redact`](/commands/redact) - Anonymize sampled data * [`validate`](/commands/validate) - Verify FK integrity of sampled output * [`graph`](/commands/graph) - Visualize FK relationships before sampling * [JSON Output Schema](/reference/json-output/) - Schema for `--json` output * [Glossary](/reference/glossary) - Definitions of FK, PK, root table, etc. # shard > Extract tenant-specific data from multi-tenant dumps Extract data for specific tenants from multi-tenant SQL dumps. **Alias:** `sh` (e.g., `sql-splitter sh dump.sql --tenant-value 123 -o tenant.sql`) Note Requires one of: `--tenant-value` or `--tenant-values` ## When to Use This [Section titled “When to Use This”](#when-to-use-this) * **Tenant data export** - Extract a single customer’s data for migration or analysis * **Development environments** - Create small, focused dev databases from a multi-tenant production dump * **Data isolation** - Separate tenant data for compliance, backup, or legal requirements * **Debugging** - Reproduce issues with a specific tenant’s data without the full database Use [`sample`](/commands/sample) instead if you want a random subset of data across all tenants. ## How It Works [Section titled “How It Works”](#how-it-works) Sharding follows your database’s foreign key relationships to extract complete, consistent data: 1. **Identifies tenant column** - Auto-detects `tenant_id`, `company_id`, `org_id`, etc., or uses your explicit `--tenant-column` 2. **Starts from root tables** - Tables with the tenant column that have no incoming foreign keys 3. **Follows FK relationships** - Recursively includes all related rows that belong to the tenant 4. **Includes lookup tables** - Optionally includes global/shared tables (countries, currencies, etc.) The result is a self-contained dump that maintains referential integrity. ## Usage [Section titled “Usage”](#usage) ```bash sql-splitter shard --tenant-value [OPTIONS] ``` ## Examples [Section titled “Examples”](#examples) ### Single Tenant Extraction [Section titled “Single Tenant Extraction”](#single-tenant-extraction) ```bash # Extract all data for tenant 123 sql-splitter shard dump.sql --tenant-value 123 -o tenant_123.sql # Specify the tenant column explicitly sql-splitter shard dump.sql --tenant-column account_id --tenant-value 42 -o out.sql ``` ### Multiple Tenants [Section titled “Multiple Tenants”](#multiple-tenants) Extract multiple tenants to separate files in one pass: ```bash sql-splitter shard dump.sql --tenant-values "1,2,3" -o shards/ ``` This creates: ```plaintext shards/ ├── tenant_1.sql ├── tenant_2.sql └── tenant_3.sql ``` ### Controlling Global Tables [Section titled “Controlling Global Tables”](#controlling-global-tables) ```bash # Include all shared/lookup tables (countries, currencies, config) sql-splitter shard dump.sql --tenant-value 123 --include-global all -o tenant.sql # Skip global tables entirely sql-splitter shard dump.sql --tenant-value 123 --include-global none -o tenant.sql # Only include small lookup tables (default behavior) sql-splitter shard dump.sql --tenant-value 123 --include-global lookups -o tenant.sql ``` ### Preview Before Extracting [Section titled “Preview Before Extracting”](#preview-before-extracting) ```bash # See what would be extracted without writing files sql-splitter shard dump.sql --tenant-value 123 --dry-run ``` ## Options [Section titled “Options”](#options) | Flag | Short | Description | Default | | --------------------- | ----- | ----------------------------------------------- | ----------- | | `--output` | `-o` | Output SQL file or directory | stdout | | `--dialect` | `-d` | SQL dialect | auto-detect | | `--tenant-column` | | Column name for tenant identification | auto-detect | | `--tenant-value` | | Single tenant value to extract | - | | `--tenant-values` | | Multiple tenant values (comma-separated) | - | | `--root-tables` | | Explicit root tables with tenant column | auto | | `--include-global` | | Global table handling: `none`, `lookups`, `all` | `lookups` | | `--config` | `-c` | YAML config file for table classification | - | | `--max-selected-rows` | | Maximum rows to select | unlimited | | `--no-limit` | | Disable row limit | false | | `--strict-fk` | | Fail if any FK integrity issues | false | | `--no-schema` | | Exclude CREATE TABLE statements | false | | `--progress` | `-p` | Show progress bar | false | | `--dry-run` | | Preview without writing | false | | `--json` | | Output as JSON | false | ## Tenant Column Detection [Section titled “Tenant Column Detection”](#tenant-column-detection) sql-splitter auto-detects common tenant columns: * `company_id` * `tenant_id` * `organization_id` * `org_id` * `account_id` * `team_id` * `workspace_id` A candidate column must appear in at least two tables to be auto-detected. Override with `--tenant-column`: ```bash sql-splitter shard dump.sql --tenant-column customer_id --tenant-value 42 ``` ## Multiple Tenants [Section titled “Multiple Tenants”](#multiple-tenants-1) Extract multiple tenants to separate files: ```bash sql-splitter shard dump.sql --tenant-values "1,2,3" -o shards/ ``` Creates: ```plaintext shards/ ├── tenant_1.sql ├── tenant_2.sql └── tenant_3.sql ``` ## Global Tables [Section titled “Global Tables”](#global-tables) The `--include-global` option controls shared/lookup tables: * `none`: Skip global tables * `lookups`: Include small lookup tables (default) * `all`: Include all non-tenant tables ## FK Traversal [Section titled “FK Traversal”](#fk-traversal) Sharding follows foreign key relationships: 1. Finds rows matching tenant value in root tables 2. Follows FKs to include related rows 3. Ensures complete, consistent tenant slice ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) ### ”No tenant column found” error [Section titled “”No tenant column found” error”](#no-tenant-column-found-error) Auto-detection looks for common column names. Specify yours explicitly: ```bash sql-splitter shard dump.sql --tenant-column customer_id --tenant-value 123 -o tenant.sql ``` ### Missing related data in output [Section titled “Missing related data in output”](#missing-related-data-in-output) Foreign key relationships might not be detected if: * They’re not defined in the schema (implicit FKs) * The FK references a table that’s excluded Use `--root-tables` to explicitly specify which tables have the tenant column: ```bash sql-splitter shard dump.sql --tenant-value 123 --root-tables users,projects,teams -o tenant.sql ``` ### Shard is larger than expected [Section titled “Shard is larger than expected”](#shard-is-larger-than-expected) Check which global tables are being included: ```bash # See what would be extracted sql-splitter shard dump.sql --tenant-value 123 --dry-run --json | jq '.tables' # Exclude large lookup tables sql-splitter shard dump.sql --tenant-value 123 --include-global none -o tenant.sql ``` ### FK integrity warnings [Section titled “FK integrity warnings”](#fk-integrity-warnings) Some rows might reference data outside the tenant’s scope. Options: 1. **`--strict-fk`**: Fail immediately on any integrity issue 2. **Review warnings**: Some cross-tenant references might be intentional (shared resources) 3. **Include global tables**: `--include-global all` includes all referenced lookup data ## See Also [Section titled “See Also”](#see-also) * [`sample`](/commands/sample) - Create reduced datasets * [`split`](/commands/split) - Split by table * [`graph`](/commands/graph) - Visualize FK relationships before sharding * [JSON Output Schema](/reference/json-output/) - Schema for `--json` output # split > Split a SQL dump file into individual table files Split a large SQL dump into separate files, one per table. **Alias:** `sp` (e.g., `sql-splitter sp dump.sql -o tables/`) ## When to Use This [Section titled “When to Use This”](#when-to-use-this) * **Extract specific tables** — Pull out just the tables you need without loading the entire dump * **Enable parallel processing** — Split once, then process tables concurrently in separate jobs * **Debug specific tables** — Isolate a problematic table’s [DDL](/reference/glossary#ddl-data-definition-language) and data for investigation * **Create partial backups** — Extract only customer-facing tables while ignoring analytics/logging tables * **Separate schema from data** — Use `--schema-only` for migrations, `--data-only` for data refreshes ## How It Works [Section titled “How It Works”](#how-it-works) The `split` command uses [streaming](/reference/glossary#streaming) to process dumps in a single pass with constant memory (\~10MB). Here’s what happens: 1. **Read** — Statements are read using buffered I/O (64KB buffer for files under 1GB, 256KB for larger files) 2. **Parse** — Each statement is identified and its target table extracted 3. **Route** — Statements are written to the appropriate `.sql` file via 256KB write buffers 4. **Flush** — Buffers are flushed every 100 statements to balance memory and I/O ### What Goes Where [Section titled “What Goes Where”](#what-goes-where) | Statement Type | Destination | | ----------------------------------------- | ------------- | | `CREATE TABLE users` | `users.sql` | | `INSERT INTO users` | `users.sql` | | `CREATE INDEX ON users` | `users.sql` | | `ALTER TABLE users` | `users.sql` | | `COPY users FROM stdin` (PostgreSQL) | `users.sql` | | `SET`, `BEGIN`, `COMMIT`, header comments | `_global.sql` | | Unrecognized statements | `_global.sql` | Table names are sanitized for the filesystem: special characters become underscores, and names are lowercased. ## Usage [Section titled “Usage”](#usage) ```bash sql-splitter split [OPTIONS] ``` ## Examples [Section titled “Examples”](#examples) ### Basic Splitting [Section titled “Basic Splitting”](#basic-splitting) ```bash # Split all tables to output/ sql-splitter split database.sql --output=tables/ # Preview what would be created (no files written) sql-splitter split database.sql --dry-run --verbose ``` ### Working with Specific Dialects [Section titled “Working with Specific Dialects”](#working-with-specific-dialects) ```bash # PostgreSQL pg_dump output sql-splitter split pg_dump.sql -o tables/ --dialect=postgres # SQLite .dump output sql-splitter split sqlite_dump.sql -o tables/ --dialect=sqlite ``` ### Extracting Specific Tables [Section titled “Extracting Specific Tables”](#extracting-specific-tables) ```bash # Only extract users and orders sql-splitter split dump.sql --tables users,orders -o subset/ # Extract schema for all tables, data for just users sql-splitter split dump.sql -o schema/ --schema-only sql-splitter split dump.sql -o data/ --data-only --tables users ``` Schema-qualified table names remain distinct. For a dump with both `tenant_a.users` and `tenant_b.users`, use the full name in `--tables` to select one table. ```bash sql-splitter split postgres_dump.sql --dialect postgres --tables tenant_a.users -o subset/ ``` ### Compressed Input [Section titled “Compressed Input”](#compressed-input) ```bash # Automatically decompresses based on extension sql-splitter split backup.sql.gz -o tables/ sql-splitter split backup.sql.zst -o tables/ sql-splitter split backup.sql.xz -o tables/ ``` See [Compression](/reference/compression) for all supported formats. ### Batch Processing [Section titled “Batch Processing”](#batch-processing) ```bash # Split multiple dumps sql-splitter split "backups/*.sql" -o output/ --fail-fast # Process all dumps recursively sql-splitter split "dumps/**/*.sql.gz" -o tables/ ``` ## Options [Section titled “Options”](#options) | Flag | Short | Description | Default | | --------------- | ----- | ------------------------------------------------------------------------------------ | ----------- | | `--output` | `-o` | Output directory | `output` | | `--dialect` | `-d` | SQL dialect: `mysql`, `postgres`, `sqlite`, `mssql` | auto-detect | | `--tables` | `-t` | Only split these tables (comma-separated) | all | | `--schema-only` | | Only [DDL](/reference/glossary#ddl-data-definition-language) (CREATE TABLE, indexes) | false | | `--data-only` | | Only [DML](/reference/glossary#dml-data-manipulation-language) (INSERT, COPY) | false | | `--verbose` | `-v` | Show verbose output | false | | `--progress` | `-p` | Show progress bar | false | | `--dry-run` | | Preview without writing files | false | | `--fail-fast` | | Stop on first error (for glob patterns) | false | | `--json` | | Output results as JSON | false | ## Output Structure [Section titled “Output Structure”](#output-structure) ```plaintext tables/ ├── users.sql # CREATE TABLE + INSERT for users ├── orders.sql # CREATE TABLE + INSERT for orders ├── products.sql # CREATE TABLE + INSERT for products └── _global.sql # Header statements, SET commands ``` The `_global.sql` file contains: * Session settings (`SET NAMES utf8mb4`, `SET FOREIGN_KEY_CHECKS=0`) * Transaction markers (`BEGIN`, `COMMIT`) * Header comments from the dump tool * Any statements that couldn’t be associated with a specific table ## JSON Output [Section titled “JSON Output”](#json-output) ```bash sql-splitter split dump.sql -o tables/ --json ``` ```json { "input_file": "dump.sql", "output_dir": "tables/", "dialect": "mysql", "dry_run": false, "statistics": { "tables_found": 2, "statements_processed": 654, "bytes_processed": 58023, "elapsed_secs": 0.042, "throughput_mb_per_sec": 612.5 }, "tables": ["orders", "users"] } ``` ## Exit Codes [Section titled “Exit Codes”](#exit-codes) | Code | Meaning | | ---- | -------------------------------------------------- | | `0` | Success — all files written | | `1` | Error — I/O failure, parse error, or invalid input | | `2` | Invalid arguments — bad CLI flags | See [Exit Codes](/reference/exit-codes) for scripting examples. ## Composing with Other Tools [Section titled “Composing with Other Tools”](#composing-with-other-tools) ### Split → Compress Individual Tables [Section titled “Split → Compress Individual Tables”](#split--compress-individual-tables) ```bash # Split, then compress each table file sql-splitter split dump.sql -o tables/ for f in tables/*.sql; do gzip "$f"; done ``` ### Split → Query Specific Tables [Section titled “Split → Query Specific Tables”](#split--query-specific-tables) ```bash # Extract users table, then query it sql-splitter split dump.sql --tables users -o tmp/ sql-splitter query tmp/users.sql "SELECT * FROM users WHERE created_at > '2024-01-01'" ``` ### Split → Parallel Import [Section titled “Split → Parallel Import”](#split--parallel-import) ```bash # Split once, import tables in parallel sql-splitter split dump.sql -o tables/ ls tables/*.sql | xargs -P4 -I{} mysql mydb < {} ``` ### Analyze → Split Specific Tables [Section titled “Analyze → Split Specific Tables”](#analyze--split-specific-tables) ```bash # Find largest tables, then extract just those sql-splitter analyze dump.sql --json | jq -r '.tables | sort_by(-.bytes) | .[0:3] | .[].name' # Output: orders, logs, events sql-splitter split dump.sql --tables orders,logs,events -o large_tables/ ``` ### Split → Merge Subset [Section titled “Split → Merge Subset”](#split--merge-subset) ```bash # Extract all tables, merge back a subset sql-splitter split dump.sql -o all/ sql-splitter merge all/ --tables users,orders,order_items -o subset.sql ``` ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) ### Table not appearing in output [Section titled “Table not appearing in output”](#table-not-appearing-in-output) **Symptom:** You expect a table but it’s not in the output directory. **Causes:** * The dump only contains [DDL](/reference/glossary#ddl-data-definition-language) (schema) or only [DML](/reference/glossary#dml-data-manipulation-language) (data), and you used the opposite `--schema-only` or `--data-only` flag * The table name in the dump differs from what you expect (check case sensitivity) * The table has no statements at all in the dump **Fix:** Run `sql-splitter analyze dump.sql` to see exactly which tables exist and what statements they have. ### Empty output files [Section titled “Empty output files”](#empty-output-files) **Symptom:** Table files are created but contain no data. **Causes:** * Using `--data-only` on a dump that only has `CREATE TABLE` statements * Using `--schema-only` on a data-only dump **Fix:** Remove the `--schema-only` or `--data-only` flag, or check your dump contains what you expect with [`analyze`](/commands/analyze). ### Dialect detection failed [Section titled “Dialect detection failed”](#dialect-detection-failed) **Symptom:** Error message about unable to detect dialect, or wrong SQL syntax in output. **Causes:** * The dump lacks dialect-specific markers (backticks for MySQL, `COPY` for PostgreSQL) * Mixed-dialect dump (manually edited) **Fix:** Explicitly specify the dialect: `--dialect=postgres` or `--dialect=mysql`. ### How do I get compressed output? [Section titled “How do I get compressed output?”](#how-do-i-get-compressed-output) **Symptom:** You want `.sql.gz` files instead of `.sql`. sql-splitter writes uncompressed output. Pipe through compression after: ```bash sql-splitter split dump.sql -o tables/ gzip tables/*.sql ``` Or for a single compressed output, pipe [`merge`](/commands/merge) through gzip: ```bash sql-splitter split dump.sql -o tables/ sql-splitter merge tables/ | gzip > combined.sql.gz ``` ### Permission denied writing output [Section titled “Permission denied writing output”](#permission-denied-writing-output) **Symptom:** `Permission denied` error when writing to output directory. **Fix:** Ensure the output directory exists and is writable: ```bash mkdir -p tables/ sql-splitter split dump.sql -o tables/ ``` ## See Also [Section titled “See Also”](#see-also) * [`merge`](/commands/merge) — Combine split files back into one dump * [`analyze`](/commands/analyze) — View statistics without splitting * [`order`](/commands/order) — Reorder statements in [topological order](/reference/glossary#topological-order--topological-sort) * [Compression](/reference/compression) — Supported input compression formats * [Dialects](/reference/dialects) — Dialect-specific behavior * [JSON Output Schema](/reference/json-output/) — Schema for `--json` output # validate > Check SQL dump integrity with comprehensive validation Validate SQL dump files for syntax errors, encoding issues, and [referential integrity](/reference/glossary#referential-integrity). **Alias:** `val` (e.g., `sql-splitter val dump.sql --strict`) ## When to Use This [Section titled “When to Use This”](#when-to-use-this) * **Pre-restore verification** - Check dumps before importing to catch issues early * **CI/CD quality gates** - Block deployments if backups are malformed * **Backup integrity monitoring** - Verify automated backups haven’t been corrupted * **Encoding issue detection** - Find invalid UTF-8 sequences that cause import failures * **Referential integrity auditing** - Detect orphaned [foreign key](/reference/glossary#foreign-key-fk) references and duplicate [primary keys](/reference/glossary#primary-key-pk) Use [`diff`](/commands/diff) instead if you need to compare two dumps for differences. ## How It Works [Section titled “How It Works”](#how-it-works) Validation runs in two passes: 1. **Schema pass** - Parses the entire dump, validates SQL syntax, checks UTF-8 encoding, and builds a table registry. This pass is fast and [streams](/reference/glossary#streaming) through the file with constant memory (\~10MB). 2. **Data integrity pass** (optional) - If [PK](/reference/glossary#primary-key-pk)/[FK](/reference/glossary#foreign-key-fk) checks are enabled, a second pass tracks primary key values and verifies foreign key references. This uses a `PkHashSet` that stores 64-bit hashes of seen values—memory grows with row count (roughly 8 bytes per unique PK value). **Memory considerations:** For a table with 10 million rows, expect \~80MB memory for PK tracking. Use `--no-fk-checks` for large dumps to skip the data integrity pass entirely, or `--max-rows-per-table` to limit tracking per table. **Exit codes:** * `0` - Valid (no errors; warnings may exist) * `1` - Invalid (errors found, or warnings with `--strict`), or runtime error (file not found, parse failure) * `2` - Invalid CLI arguments ## Usage [Section titled “Usage”](#usage) ```bash sql-splitter validate [OPTIONS] ``` ## Examples [Section titled “Examples”](#examples) ### Basic Validation [Section titled “Basic Validation”](#basic-validation) ```bash # Validate a single dump sql-splitter validate dump.sql # Validate with progress bar (useful for large files) sql-splitter validate large-dump.sql --progress # Validate a PostgreSQL pg_dump file sql-splitter validate pg_dump.sql --dialect=postgres ``` ### CI/CD Integration [Section titled “CI/CD Integration”](#cicd-integration) ```bash # Strict mode - treat warnings as errors sql-splitter validate dump.sql --strict # JSON output for parsing in CI sql-splitter validate dump.sql --json # Fail fast on first error (for batch validation) sql-splitter validate "backups/*.sql" --fail-fast --strict ``` ### Memory-Constrained Validation [Section titled “Memory-Constrained Validation”](#memory-constrained-validation) ```bash # Skip PK/FK checks entirely (fastest, lowest memory) sql-splitter validate dump.sql --no-fk-checks # Limit rows checked per table (middle ground) sql-splitter validate dump.sql --max-rows-per-table 100000 # No limit on rows (full check, highest memory) sql-splitter validate dump.sql --no-limit ``` ## Options [Section titled “Options”](#options) | Flag | Short | Description | Default | | ---------------------- | ----- | --------------------------------------------------- | ----------- | | `--dialect` | `-d` | SQL dialect: `mysql`, `postgres`, `sqlite`, `mssql` | auto-detect | | `--strict` | | Treat warnings as errors (exit 1) | false | | `--json` | | Output results as JSON | false | | `--max-rows-per-table` | | Max rows per table for PK/FK checks (0 = unlimited) | 1,000,000 | | `--no-limit` | | Disable row limit for PK/FK checks | false | | `--no-fk-checks` | | Skip PK/FK data integrity checks | false | | `--progress` | `-p` | Show progress bar | false | | `--fail-fast` | | Stop on first error (for glob patterns) | false | ## Validation Checks [Section titled “Validation Checks”](#validation-checks) | Check | What It Detects | Memory Impact | | --------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | -------------------------- | | **SQL syntax** | Malformed statements, unclosed strings, invalid keywords | None (streaming) | | **Encoding** | Invalid UTF-8 byte sequences, mixed encodings | None (streaming) | | **[DDL](/reference/glossary#ddl-data-definition-language)/[DML](/reference/glossary#dml-data-manipulation-language) consistency** | INSERT/COPY into non-existent tables, column count mismatches | Minimal (table registry) | | **Duplicate [PK](/reference/glossary#primary-key-pk)** | Multiple rows with the same primary key value | \~8 bytes per row | | **[FK](/reference/glossary#foreign-key-fk) referential integrity** | Foreign key values referencing non-existent parent rows | \~8 bytes per FK reference | The first three checks run during the schema pass. PK/FK checks require the data integrity pass and can be disabled with `--no-fk-checks`. ## Output [Section titled “Output”](#output) ```plaintext ✓ dump.sql is valid Warnings: - Table 'logs' has no primary key - Column 'users.email' allows NULL but has unique constraint ``` ## JSON Output [Section titled “JSON Output”](#json-output) ```bash sql-splitter validate dump.sql --json ``` ```json { "dialect": "mysql", "issues": [ { "severity": "warning", "code": "W002", "message": "Table 'logs' has no primary key", "location": { "table": "logs", "statement_index": 3 } } ], "summary": { "errors": 0, "warnings": 1, "info": 0, "tables_scanned": 5, "statements_scanned": 150 }, "checks": { "syntax": "ok", "encoding": "ok", "ddl_dml_consistency": "ok", "pk_duplicates": "ok", "fk_integrity": "ok" } } ``` ## CI Usage [Section titled “CI Usage”](#ci-usage) ```bash # Simple CI gate sql-splitter validate dump.sql --strict || exit 1 # Parse JSON with jq (check for errors) sql-splitter validate dump.sql --json | jq 'select(.summary.errors > 0)' # Multi-file validation returns aggregate with results array # (each result nests the per-file report under .summary) sql-splitter validate "*.sql" --json | jq '.results[] | select(.passed | not)' ``` ## Composing with Other Tools [Section titled “Composing with Other Tools”](#composing-with-other-tools) ### Validate Before Splitting [Section titled “Validate Before Splitting”](#validate-before-splitting) Catch issues before investing time in splitting large dumps: ```bash sql-splitter validate dump.sql --strict && \ sql-splitter split dump.sql -o tables/ ``` ### Validate After Converting Dialects [Section titled “Validate After Converting Dialects”](#validate-after-converting-dialects) Ensure dialect conversion produces valid output: ```bash sql-splitter convert mysql.sql -o postgres.sql --to postgres sql-splitter validate postgres.sql --dialect=postgres --strict ``` ### Validate in a Pipeline with analyze [Section titled “Validate in a Pipeline with analyze”](#validate-in-a-pipeline-with-analyze) Check integrity and gather statistics in one pass: ```bash for dump in backups/*.sql; do echo "=== $dump ===" sql-splitter validate "$dump" --no-fk-checks && \ sql-splitter analyze "$dump" done ``` ### Pre-import Workflow [Section titled “Pre-import Workflow”](#pre-import-workflow) Full validation pipeline before database restore: ```bash sql-splitter validate dump.sql --strict --json > validation.json if [ $? -eq 0 ]; then sql-splitter order dump.sql -o ordered.sql mysql mydb < ordered.sql fi ``` ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) ### Out of memory during validation [Section titled “Out of memory during validation”](#out-of-memory-during-validation) **Symptom:** Process killed or “memory allocation failed” error on large dumps. **Cause:** PK/FK integrity checks track hashes of all primary key values, which grows with row count. **Solutions:** 1. Skip integrity checks entirely: `--no-fk-checks` 2. Limit rows checked per table: `--max-rows-per-table 500000` 3. Validate tables individually after splitting: `sql-splitter split` then validate each file ### False positive FK errors [Section titled “False positive FK errors”](#false-positive-fk-errors) **Symptom:** Validation reports missing FK references, but data imports successfully. **Cause:** The dump may define foreign keys before the referenced parent rows appear (common with [deferred constraints](/reference/glossary#deferred-constraints) or out-of-order dumps). **Solutions:** 1. Reorder the dump first: [`sql-splitter order dump.sql -o ordered.sql`](/commands/order) 2. Skip FK checks: `--no-fk-checks` 3. If using PostgreSQL with deferred FKs, this is expected behavior—the database handles it at commit time ### Encoding errors on valid files [Section titled “Encoding errors on valid files”](#encoding-errors-on-valid-files) **Symptom:** UTF-8 encoding errors on files that open fine in editors. **Cause:** The file may use Latin-1, Windows-1252, or mixed encodings. **Solutions:** 1. Convert to UTF-8: `iconv -f WINDOWS-1252 -t UTF-8 dump.sql > dump-utf8.sql` 2. Check the original `mysqldump` charset: ensure `--default-character-set=utf8mb4` ### Validation passes but import fails [Section titled “Validation passes but import fails”](#validation-passes-but-import-fails) **Symptom:** No validation errors, but `mysql` or `psql` reports syntax errors. **Cause:** sql-splitter validates common SQL patterns but may not catch all dialect-specific edge cases. **Solutions:** 1. Ensure correct dialect: `--dialect=postgres` for pg\_dump files 2. Check for dialect-specific syntax not yet supported 3. Report the issue with a minimal reproduction ### ”Table has no primary key” warnings [Section titled “”Table has no primary key” warnings”](#table-has-no-primary-key-warnings) **Symptom:** Many warnings about tables without PKs. **Cause:** PK duplicate detection requires knowing the primary key. Tables without PKs can only have basic syntax/encoding checks. **Solutions:** 1. This is informational—data integrity for these tables is skipped 2. Use `--strict` only if you want to enforce PKs on all tables 3. Consider adding PKs to these tables in your schema ## See Also [Section titled “See Also”](#see-also) * [`analyze`](/commands/analyze) - View table statistics without validation * [`diff`](/commands/diff) - Compare two dumps for differences * [`order`](/commands/order) - Reorder statements for FK-safe imports * [JSON Output Schema](/reference/json-output/) - Schema for `--json` output * [Memory & Streaming](/advanced/memory-streaming) - Detailed memory profiles per command * [Glossary](/reference/glossary) - Database terminology reference # Overview > How to contribute to sql-splitter Thanks for your interest in contributing! This guide covers everything you need to get started. ## Quick Links [Section titled “Quick Links”](#quick-links) | Guide | Description | | ---------------------------------------------------- | --------------------------- | | [Development](/contributing/development) | Set up your dev environment | | [Testing](/contributing/testing) | Run and write tests | | [Benchmarking](/contributing/benchmarking) | Run performance benchmarks | | [Release Process](/contributing/release-process) | How releases work | | [Website OG Images](/contributing/website-og-images) | How OG images are generated | ## Pull Request Process [Section titled “Pull Request Process”](#pull-request-process) 1. Fork the repository 2. Create a feature branch (`git checkout -b feature/my-feature`) 3. Make your changes 4. Run tests and linting: ```bash cargo fmt cargo clippy cargo nextest run ``` 5. Commit with a clear message 6. Push and open a PR ### PR Expectations [Section titled “PR Expectations”](#pr-expectations) * Describe what the PR does and why * Keep changes focused (one feature/fix per PR) * Update documentation if needed * Add tests for new functionality ## Reporting Issues [Section titled “Reporting Issues”](#reporting-issues) * Search existing issues first * Include reproduction steps for bugs * Include Rust version (`rustc --version`) and OS * Be specific about expected vs actual behavior ## AI-Assisted Development [Section titled “AI-Assisted Development”](#ai-assisted-development) AI coding assistants are encouraged for contributing to this project. Tools like [Amp Code](https://ampcode.com), [Claude Code](https://claude.ai/code), Cursor, and similar assistants can significantly accelerate development. ### Quality Requirements [Section titled “Quality Requirements”](#quality-requirements) AI-generated code must meet the same standards as human-written code: * **All tests must pass genuinely** - Tests verify actual behavior, not just compile * **No stubbed tests** - Never use `todo!()`, `unimplemented!()`, or hardcoded expected values to make tests pass artificially * **No skipped assertions** - Every test must assert meaningful behavior * **Review critically** - Understand and verify all generated code before committing ### Feature Design Workflow [Section titled “Feature Design Workflow”](#feature-design-workflow) For new features or significant changes: 1. **Check existing designs** - Review `docs/features/` for related design docs 2. **Write a design doc** - For new features, create `docs/features/FEATURE_NAME.md` 3. **Implement in phases** - Break large features into phases as shown in the design doc 4. **Test thoroughly** - Add tests that verify the design requirements 5. **Reference in PR** - Link to the design doc in your pull request ### Modes of Work [Section titled “Modes of Work”](#modes-of-work) When starting an AI session, consider which mode you’re in: | Mode | Purpose | Example | | ----------- | ---------------------------------- | ------------------------------------ | | **Build** | New functionality | ”Implement sample command (Phase 1)“ | | **Improve** | Refactor without changing behavior | ”Extract sampler module” | | **Sweep** | Fix related bugs | ”Fix all INSERT parsing edge cases” | This prevents scope creep and keeps AI contributions focused. ## Code of Conduct [Section titled “Code of Conduct”](#code-of-conduct) Be respectful and constructive. We’re all here to build something useful. * Be welcoming to newcomers * Accept constructive criticism gracefully * Focus on what’s best for the project * Show empathy towards others ## Questions? [Section titled “Questions?”](#questions) Open an issue or start a [discussion](https://github.com/helgesverre/sql-splitter/discussions). # Benchmarking > Running competitor benchmarks in Docker sql-splitter includes a Docker-based benchmarking suite to compare performance against other SQL dump splitting tools. ## Tools Tested [Section titled “Tools Tested”](#tools-tested) | Tool | Language | Stars | Notes | | ------------------- | ----------- | ----- | ---------------------------------------- | | **sql-splitter** | Rust | - | Multi-dialect, streaming I/O | | mysqldbsplit | PHP | 101 | Fastest competitor; selective extraction | | mysqldump-splitter | Rust | 1 | Hierarchical output, gzip support | | mysql-dump-splitter | Go | 0 | Include/exclude tables | | mysqldumpsplit | Go | \~40 | Buffers in memory, has deadlock bug\* | | mysqldumpsplitter | Bash/awk | 540+ | Most popular shell-based tool | | mysql\_splitdump | Bash/csplit | 93 | Uses GNU coreutils csplit | | mysqldumpsplit | Node.js | 55 | Requires Node 10 (gulp 3.x) | | mysql-dump-split | Ruby | 77 | Archived project | | extract-mysql-dump | Python | \~5 | Multi-database extraction, Python 3.3+ | *\*Original Go tool has a deadlock bug with non-interleaved dumps; benchmarks use a [patched fork](https://github.com/HelgeSverre/mysqldumpsplit/tree/fix/handle-non-interleaved-dumps).* ## Running Benchmarks [Section titled “Running Benchmarks”](#running-benchmarks) Benchmarks run inside Docker for reproducibility across different machines. ### Quick Start [Section titled “Quick Start”](#quick-start) ```bash # Build the Docker image (first time only) just docker-build # Run benchmark with generated 100MB test file just docker-bench # Run with custom size (e.g., 200MB) ./docker/run-benchmark.sh --generate 200 ``` ### Options [Section titled “Options”](#options) | Flag | Description | | ----------------- | -------------------------------------- | | `--generate SIZE` | Generate test data of SIZE MB | | `--runs N` | Number of benchmark runs (default: 3) | | `--warmup N` | Warmup runs before timing (default: 1) | | `--export FILE` | Export results to markdown file | | `--list` | Show installed tools | | `--test` | Test which tools work with file | ### Examples [Section titled “Examples”](#examples) ```bash # Generate 500MB test and run 5 iterations ./docker/run-benchmark.sh --generate 500 --runs 5 # Export results to markdown ./docker/run-benchmark.sh --generate 100 --export results.md # List available tools ./docker/run-benchmark.sh --list # Test which tools work with a specific file ./docker/run-benchmark.sh --test /data/dump.sql ``` ## Benchmark Results [Section titled “Benchmark Results”](#benchmark-results) > **Hardware:** Apple M2 Max, 32GB RAM, Docker Desktop (linux/arm64) ### 100MB Generated Test File [Section titled “100MB Generated Test File”](#100mb-generated-test-file) | Tool | Mean | σ | Throughput | Relative | | :-------------------------------- | ------: | ---: | ---------: | :----------------- | | **sql-splitter (Rust)** | 64 ms | ±5 | 1605 MB/s | **1.00 (fastest)** | | mysqldbsplit (PHP) | 79 ms | ±3 | 1304 MB/s | 1.23x slower | | mysqldumpsplit (Go)\* | 193 ms | ±46 | 536 MB/s | 3.00x slower | | mysql-dump-splitter (Go/Bekkema) | 196 ms | ±53 | 526 MB/s | 3.05x slower | | mysql\_splitdump (csplit) | 237 ms | ±16 | 435 MB/s | 3.69x slower | | mysqldump-splitter (Rust/Scoopit) | 271 ms | ±35 | 381 MB/s | 4.21x slower | | mysqldumpsplit (Node.js) | 427 ms | ±20 | 242 MB/s | 6.64x slower | | mysql-dump-split (Ruby) | 979 ms | ±16 | 105 MB/s | **15.2x slower** | | mysqldumpsplitter (Bash) | 1027 ms | ±25 | 100 MB/s | **16.0x slower** | | extract-mysql-dump (Python) | 1655 ms | ±122 | 62 MB/s | **25.7x slower** | ### 1GB Generated Test File [Section titled “1GB Generated Test File”](#1gb-generated-test-file) | Tool | Mean | σ | Throughput | Relative | | :-------------------------------- | -----: | ----: | ---------: | :----------------- | | **sql-splitter (Rust)** | 0.42s | ±0.01 | 2453 MB/s | **1.00 (fastest)** | | mysqldbsplit (PHP) | 0.80s | ±0.10 | 1289 MB/s | 1.90x slower | | mysql-dump-splitter (Go/Bekkema) | 0.94s | ±0.01 | 1097 MB/s | 2.24x slower | | mysqldump-splitter (Rust/Scoopit) | 1.17s | ±0.02 | 884 MB/s | 2.78x slower | | mysqldumpsplit (Go)\* | 1.50s | ±0.06 | 691 MB/s | 3.55x slower | | mysql\_splitdump (csplit) | 2.43s | ±0.13 | 425 MB/s | 5.77x slower | | mysqldumpsplit (Node.js) | 2.91s | ±0.03 | 356 MB/s | 6.89x slower | | mysqldumpsplitter (Bash) | 9.37s | ±0.36 | 110 MB/s | **22.2x slower** | | mysql-dump-split (Ruby) | 9.92s | ±0.63 | 104 MB/s | **23.5x slower** | | extract-mysql-dump (Python) | 13.75s | ±0.12 | 75 MB/s | **32.6x slower** | ### 5GB Stress Test (disk-backed /tmp, 3 runs) [Section titled “5GB Stress Test (disk-backed /tmp, 3 runs)”](#5gb-stress-test-disk-backed-tmp-3-runs) | Tool | Mean | σ | Throughput | Relative | | :-------------------------------- | ----: | ----: | ---------: | :----------------- | | **sql-splitter (Rust)** | 10.8s | ±8.1 | 480 MB/s | **1.00 (fastest)** | | mysql-dump-splitter (Go/Bekkema) | 12.1s | ±0.7 | 427 MB/s | 1.12x slower | | mysqldbsplit (PHP) | 12.8s | ±1.5 | 404 MB/s | 1.19x slower | | mysqldump-splitter (Rust/Scoopit) | 22.8s | ±9.3 | 227 MB/s | 2.11x slower | | mysqldumpsplit (Node.js) | 23.8s | ±5.1 | 218 MB/s | 2.20x slower | | mysqldumpsplit (Go)\* | 35.6s | ±5.0 | 146 MB/s | 3.29x slower | | mysqldumpsplitter (Bash) | 49.8s | ±1.1 | 104 MB/s | 4.60x slower | | mysql-dump-split (Ruby) | 68.3s | ±1.3 | 76 MB/s | 6.32x slower | | extract-mysql-dump (Python) | 90.6s | ±1.5 | 57 MB/s | 8.38x slower | | mysql\_splitdump (csplit) | 92.5s | ±15.8 | 56 MB/s | 8.55x slower | At 5GB the container’s disk write bandwidth dominates and compresses the margins — sql-splitter stays fastest, with the top three tools all effectively I/O-bound (\~400–480 MB/s). ## Key Findings [Section titled “Key Findings”](#key-findings) ### Speed vs Memory Tradeoffs [Section titled “Speed vs Memory Tradeoffs”](#speed-vs-memory-tradeoffs) * **sql-splitter (Rust) is the fastest at every size** — 1.6 GB/s on 100MB, 2.4 GB/s on 1GB — since the v1.14.1+ write-path rework (parallel pipelined writers) and allocation-lean parsing. * **mysqldbsplit (PHP)** is the fastest competitor at \~1.3 GB/s—surprisingly beating all other compiled tools on mysqldump format. * **sql-splitter** streams with bounded memory (\~20–150 MB depending on command), independent of file size. * **csplit** is surprisingly fast for a shell tool, but relies on GNU coreutils (not available on stock macOS). * **Node.js** is \~7x slower but still reasonable for JS-based workflows. * **Ruby/Bash/awk** are 15-24x slower—fine for one-off use but not for automation. * **Python (extract-mysql-dump)** is the slowest at 26-33x slower, designed specifically for multi-database extraction scenarios. ### Format Compatibility [Section titled “Format Compatibility”](#format-compatibility) All competitors **only work with standard mysqldump format** that includes comment markers like: ```sql -- Table structure for table `users` ``` **sql-splitter parses actual SQL statements** (`CREATE TABLE`, `INSERT INTO`, `COPY`), so it works with: * TablePlus exports * DBeaver exports * pg\_dump (PostgreSQL) * sqlite3 .dump * Any valid SQL file Competitors produce **0 tables** on non-mysqldump files. ### Known Limitations [Section titled “Known Limitations”](#known-limitations) | Tool | Issue | | ----------------------------- | ----------------------------------------------------------------------------------------------- | | mysqldbsplit (PHP) | Requires PHP CLI; mysqldump format only | | mysqldump-splitter (Scoopit) | Rust; mysqldump format only; hierarchical output structure | | mysql-dump-splitter (Bekkema) | Go; mysqldump format only | | Go (afrase) | Deadlocks on files where all INSERTs for one table come before the next table | | Node.js (vekexasia) | Requires Node 10 (gulp 3.x incompatible with Node 12+) | | Ruby | Project archived, unmaintained | | Bash/awk | Slow, Unix-only | | csplit | Requires GNU coreutils | | extract-mysql-dump | Designed for multi-database dumps; no absolute paths; slowest of all tools | | sql-splitter | Parses actual SQL rather than mysqldump comment markers, so it does strictly more work per byte | ## When to Use sql-splitter [Section titled “When to Use sql-splitter”](#when-to-use-sql-splitter) 1. Non-mysqldump formats (TablePlus, DBeaver, pg\_dump, sqlite) 2. Large files (>1GB) where memory matters 3. CI/CD pipelines needing consistent behavior 4. Multi-dialect projects (MySQL + PostgreSQL + SQLite + MSSQL) ## Docker Files [Section titled “Docker Files”](#docker-files) The benchmark infrastructure lives in the `docker/` directory: | File | Purpose | | ------------------------------ | ---------------------------------- | | `Dockerfile.benchmark` | Container with all tools installed | | `docker-compose.benchmark.yml` | Compose configuration | | `run-benchmark.sh` | Entry point script | | `benchmark-runner.sh` | Core benchmarking logic | | `test-competitors.sh` | Test tool compatibility | ## See Also [Section titled “See Also”](#see-also) * [Performance](/advanced/performance) - Optimization tips * [Development](/contributing/development) - Development setup # Development > Set up your development environment ## Prerequisites [Section titled “Prerequisites”](#prerequisites) * Rust 1.70+ ([rustup.rs](https://rustup.rs)) * Git ## Clone and Build [Section titled “Clone and Build”](#clone-and-build) ```bash git clone https://github.com/helgesverre/sql-splitter cd sql-splitter cargo build ``` ## Run Development Build [Section titled “Run Development Build”](#run-development-build) ```bash cargo run -- split test.sql -o output/ ``` ## Build Release [Section titled “Build Release”](#build-release) ```bash cargo build --release ``` The binary will be at `target/release/sql-splitter`. ## Optimized Build [Section titled “Optimized Build”](#optimized-build) ```bash RUSTFLAGS="-C target-cpu=native" cargo build --release ``` ## Project Structure [Section titled “Project Structure”](#project-structure) * sql-splitter/ * src/ * main.rs Entry point * cmd/ CLI commands * split.rs * merge.rs * analyze.rs * … * parser/ SQL parser * … * splitter/ Split logic * … * merger/ Merge logic * … * analyzer/ Analysis logic * … * convert/ Dialect conversion * … * validate/ Validation logic * … * sample/ Sampling logic * … * shard/ Sharding logic * … * differ/ Diff logic * … * redactor/ Redaction logic * … * graph/ Graph/ERD logic * … * duckdb/ DuckDB integration * … * writer/ Output writers * … * tests/ * fixtures/ Test SQL files * … * \*.rs Test modules * man/ Man pages * … * website/ Documentation site * … * Cargo.toml ## Code Style [Section titled “Code Style”](#code-style) * Use `cargo fmt` before committing * Run `cargo clippy` for lints * Follow Rust idioms ## Make Commands [Section titled “Make Commands”](#make-commands) ```bash just build # Debug build just release # Release build just test # Run tests just install # Install binary + completions + man pages just clean # Clean build artifacts ``` # Release Process > How sql-splitter releases work ## Versioning [Section titled “Versioning”](#versioning) sql-splitter uses [Semantic Versioning](https://semver.org/): * **MAJOR**: Breaking changes * **MINOR**: New features (backward compatible) * **PATCH**: Bug fixes ## Release Checklist [Section titled “Release Checklist”](#release-checklist) 1. Update version in `Cargo.toml` 2. Update `CHANGELOG.md` 3. Run tests: `cargo nextest run` 4. Build: `cargo build --release` 5. Create git tag: `git tag v1.2.3` 6. Push: `git push && git push --tags` 7. GitHub Actions builds and releases ## GitHub Actions [Section titled “GitHub Actions”](#github-actions) Releases are automated via GitHub Actions: 1. Tag push triggers release workflow 2. Builds for all platforms: * Linux x86\_64 * Linux ARM64 * macOS Intel * macOS Apple Silicon * Windows x86\_64 3. Creates GitHub Release with binaries 4. Publishes to crates.io ## Changelog Format [Section titled “Changelog Format”](#changelog-format) ```markdown ## [1.2.3] - 2024-01-15 ### Added - New `query` command for SQL analytics (#123) ### Changed - Improved COPY statement parsing (#124) ### Fixed - Fixed crash on empty input files (#125) ``` ## Pre-Release Testing [Section titled “Pre-Release Testing”](#pre-release-testing) Before tagging: ```bash # Run all tests cargo nextest run # Verify real-world dumps just verify-realworld # Build and test binary cargo build --release ./target/release/sql-splitter --version ./target/release/sql-splitter validate tests/fixtures/static/mssql/simple.sql ``` ## Crates.io Publishing [Section titled “Crates.io Publishing”](#cratesio-publishing) Publishing to crates.io is automated: pushing a version tag triggers the publish workflow, which runs `cargo publish` using the `CARGO_REGISTRY_TOKEN` repository secret. # Testing > Running and writing tests ## Run Tests [Section titled “Run Tests”](#run-tests) ```bash # All tests cargo nextest run # Specific test cargo nextest run test_splitter_basic # With output cargo nextest run --no-capture ``` ## Test Categories [Section titled “Test Categories”](#test-categories) ### Unit Tests [Section titled “Unit Tests”](#unit-tests) In-module tests for specific functions: ```rust #[cfg(test)] mod tests { use super::*; #[test] fn test_parse_create_table() { // ... } } ``` ### Integration Tests [Section titled “Integration Tests”](#integration-tests) End-to-end tests in `tests/`: tests/splitter\_integration\_test.rs ```rust #[test] fn test_split_mysql_dump() { let output = Command::new("./target/release/sql-splitter") .args(["split", "tests/fixtures/static/mssql/simple.sql", "-o", "output/"]) .output() .expect("failed to execute"); assert!(output.status.success()); } ``` ### Real-World Verification [Section titled “Real-World Verification”](#real-world-verification) Test against real SQL dumps: ```bash just verify-realworld ``` ### Model YAML Fuzzing [Section titled “Model YAML Fuzzing”](#model-yaml-fuzzing) The `model_yaml` target mutates complete `generate` model files. It exercises UTF-8 and YAML rejection, error formatting, model compilation, bounded row generation, and SQL rendering to an I/O sink. The target runs the public in-process pipeline. CLI integration tests cover argument parsing and file handling. Install `cargo-fuzz` and the Rust nightly toolchain once: ```bash cargo install cargo-fuzz rustup toolchain install nightly ``` Run it for 60 seconds, or pass a longer duration: ```bash just fuzz-model-yaml just fuzz-model-yaml 600 ``` The run limits each input to 64 KiB, each table to eight rows, each input to five seconds, and each process to explicit allocation and resident-memory bounds. Minimize the accumulated corpus after a long or parallel run: ```bash just fuzz-model-yaml-cmin ``` Crashing inputs are written to `fuzz/artifacts/model_yaml/`. Reproduce and minimize one with: ```bash cargo +nightly fuzz run model_yaml fuzz/artifacts/model_yaml/ cargo +nightly fuzz tmin model_yaml fuzz/artifacts/model_yaml/ ``` ## Test Fixtures [Section titled “Test Fixtures”](#test-fixtures) Located in `tests/fixtures/`: ```plaintext tests/fixtures/ ├── generated/ # Test data generated at test time (gitignored) └── static/ ├── mysql/ │ ├── duplicate_pk.sql │ ├── escaped_strings.sql │ ├── fk_missing_parent.sql │ └── missing_table.sql ├── postgres/ │ └── copy_with_nulls.sql ├── sqlite/ │ └── minimal_tenant.sql └── mssql/ ├── edge_cases.sql ├── multi_tenant.sql └── simple.sql ``` ## Writing Tests [Section titled “Writing Tests”](#writing-tests) ### Test a New Feature [Section titled “Test a New Feature”](#test-a-new-feature) 1. Create fixture files in `tests/fixtures/static/` 2. Write unit tests in the module 3. Write integration test in `tests/` 4. Run `cargo nextest run` ### Test Fixture Example [Section titled “Test Fixture Example”](#test-fixture-example) ```sql -- tests/fixtures/static/mysql/my_feature.sql CREATE TABLE test ( id INT AUTO_INCREMENT PRIMARY KEY, name VARCHAR(100) ); INSERT INTO test VALUES (1, 'Alice'); INSERT INTO test VALUES (2, 'Bob'); ``` ### Test Code Example [Section titled “Test Code Example”](#test-code-example) ```rust #[test] fn test_my_feature() { let result = my_function("tests/fixtures/static/mysql/my_feature.sql"); assert!(result.is_ok()); assert_eq!(result.unwrap().table_count, 1); } ``` ## Code Coverage [Section titled “Code Coverage”](#code-coverage) ```bash cargo install cargo-tarpaulin cargo tarpaulin --out Html ``` Open `tarpaulin-report.html` to view coverage. # Website OG Images > How OG (OpenGraph) images are generated for the documentation website OG images are automatically generated for all documentation pages at build time. ## Overview [Section titled “Overview”](#overview) The website generates an OpenGraph image (1200x630px) for every documentation page, for sharing on social media. Each image includes: * Category-coded accent strip (color varies by section) * Project logo (`;` + “sql-splitter”) * Category label (Command, Guide, Reference, etc.) * Page title * Description from frontmatter ## Category Colors [Section titled “Category Colors”](#category-colors) | Section | Color | Hex | | --------------- | ------ | --------- | | Commands | Blue | `#58a6ff` | | Getting Started | Green | `#3fb950` | | Guides | Green | `#3fb950` | | Reference | Purple | `#d2a8ff` | | Advanced | Orange | `#ffa657` | | Contributing | Pink | `#f778ba` | | Tools | Blue | `#79c0ff` | | Roadmap | Purple | `#a371f7` | ## Implementation [Section titled “Implementation”](#implementation) OG images are generated using **Satori** and converted from SVG to PNG with **Sharp**. ### Key Files [Section titled “Key Files”](#key-files) | File | Purpose | | ------------------------------- | --------------------------------------------------- | | `src/pages/og/[...slug].png.ts` | Dynamic route that generates OG images for each doc | | `src/pages/og/satori-lib.mjs` | Satori template with category detection and styling | ### Generation Process [Section titled “Generation Process”](#generation-process) 1. At build time, Astro fetches all docs from the `docs` content collection 2. For each doc, `getStaticPaths()` creates a route at `/og/{slug}.png` 3. The `GET` handler renders the Satori template to SVG 4. Sharp converts SVG to PNG 5. Images are saved to `dist/og/` with `.png` extension ### Example Output [Section titled “Example Output”](#example-output) ```bash dist/og/index.png # Homepage dist/og/commands/split.png # split command page dist/og/getting-started/installation.png dist/og/guides.png # Section index pages dist/og/reference/dialects.png ... ``` ## Modifying the Template [Section titled “Modifying the Template”](#modifying-the-template) Edit `src/pages/og/satori-lib.mjs` to change the OG image design: ```javascript export async function generateOgSvg({ title, description, slug }) { // ... category detection and color assignment const element = { type: 'div', props: { style: { /* Satori JSX element */ }, children: [/* ... */], }, }; return await satori(element, { width: 1200, height: 630, fonts: [...] }); } ``` ## Regenerating Images [Section titled “Regenerating Images”](#regenerating-images) ```bash cd website bun run build ``` OG images are rebuilt with every static build (not during dev mode). ## Homepage Image [Section titled “Homepage Image”](#homepage-image) The homepage (`src/pages/index.astro`) uses a separate BaseLayout that references a static OG image at `/og-image.png`. This image should be placed in the `public/` directory. To set up the homepage OG image: 1. Create `website/public/og-image.png` (1200x630px) 2. The file will be served at `https://sql-splitter.dev/og-image.png` Alternatively, you can update `src/layouts/BaseLayout.astro` to use the dynamically generated `/og/index.png` instead. # Analytics Queries > Query SQL dumps with DuckDB without restoring Run SQL analytics directly on dump files using the embedded DuckDB engine. ## Quick Start [Section titled “Quick Start”](#quick-start) ```bash sql-splitter query dump.sql "SELECT COUNT(*) FROM users" ``` No database server needed. The dump is parsed and loaded into DuckDB for querying. ## Interactive Mode [Section titled “Interactive Mode”](#interactive-mode) ```bash sql-splitter query dump.sql --interactive ``` REPL commands: ```plaintext sql> .tables users orders products sql> .schema users CREATE TABLE users ( id INT PRIMARY KEY, name VARCHAR(100), email VARCHAR(255) ); sql> SELECT COUNT(*) FROM users; ┌──────────────┐ │ count_star() │ ├──────────────┤ │ 1500 │ └──────────────┘ sql> .sample orders 5 ┌────┬─────────┬────────┐ │ id │ user_id │ total │ ├────┼─────────┼────────┤ │ 1 │ 42 │ 99.99 │ │ 2 │ 17 │ 149.50 │ ... sql> .exit ``` ## Output Formats [Section titled “Output Formats”](#output-formats) ### Table (default) [Section titled “Table (default)”](#table-default) ```bash sql-splitter query dump.sql "SELECT * FROM users LIMIT 5" ``` ### JSON [Section titled “JSON”](#json) ```bash sql-splitter query dump.sql "SELECT * FROM users" -f json ``` ```json [ { "id": 1, "name": "Alice", "email": "alice@example.com" }, { "id": 2, "name": "Bob", "email": "bob@example.com" } ] ``` ### CSV [Section titled “CSV”](#csv) ```bash sql-splitter query dump.sql "SELECT * FROM users" -f csv -o users.csv ``` ### JSON Lines [Section titled “JSON Lines”](#json-lines) ```bash sql-splitter query dump.sql "SELECT * FROM users" -f jsonl ``` ## Common Analytics [Section titled “Common Analytics”](#common-analytics) ### Table sizes [Section titled “Table sizes”](#table-sizes) ```bash sql-splitter query dump.sql " SELECT 'users' as tbl, COUNT(*) as rows FROM users UNION ALL SELECT 'orders', COUNT(*) FROM orders UNION ALL SELECT 'products', COUNT(*) FROM products " ``` ### Data quality checks [Section titled “Data quality checks”](#data-quality-checks) ```bash # Find duplicates sql-splitter query dump.sql " SELECT email, COUNT(*) as cnt FROM users GROUP BY email HAVING cnt > 1 " # Find orphan records sql-splitter query dump.sql " SELECT o.* FROM orders o LEFT JOIN users u ON o.user_id = u.id WHERE u.id IS NULL " ``` ### Export subsets [Section titled “Export subsets”](#export-subsets) ```bash # Export active users to CSV sql-splitter query dump.sql \ "SELECT * FROM users WHERE active = 1" \ -f csv -o active_users.csv # Export high-value orders to JSON sql-splitter query dump.sql \ "SELECT * FROM orders WHERE total > 1000" \ -f json -o high_value.json ``` ## Caching for Speed [Section titled “Caching for Speed”](#caching-for-speed) First query imports the dump (slow): ```bash sql-splitter query dump.sql "SELECT COUNT(*) FROM users" --cache # Importing dump... done (5.2s) # 1500 ``` Subsequent queries use cache (fast): ```bash sql-splitter query dump.sql "SELECT * FROM users WHERE active = 1" --cache # Using cached database # ... ``` Manage cache: ```bash # List cached databases sql-splitter query --list-cache # Clear cache sql-splitter query --clear-cache ``` ## Large Files [Section titled “Large Files”](#large-files) For dumps larger than available RAM: ```bash sql-splitter query huge.sql "SELECT ..." --disk ``` This stores data on disk instead of in memory. ## DuckDB SQL Features [Section titled “DuckDB SQL Features”](#duckdb-sql-features) DuckDB supports modern SQL features: ```sql -- Window functions SELECT id, name, SUM(total) OVER (PARTITION BY user_id ORDER BY created_at) as running_total FROM orders; -- CTEs WITH active_users AS ( SELECT * FROM users WHERE active = 1 ) SELECT * FROM orders WHERE user_id IN (SELECT id FROM active_users); -- QUALIFY SELECT * FROM orders QUALIFY ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY created_at DESC) = 1; ``` See [DuckDB SQL Reference](https://duckdb.org/docs/sql/introduction) for full documentation. # CI Validation > Validate SQL dumps in CI/CD pipelines Integrate sql-splitter validation into your CI/CD pipeline. ## Basic CI Gate [Section titled “Basic CI Gate”](#basic-ci-gate) ```bash #!/bin/bash set -euo pipefail sql-splitter validate "dumps/*.sql" --strict --fail-fast --progress ``` Exit codes: * `0`: All dumps valid * `1`: Validation errors or warnings (with `--strict`) ## GitHub Actions [Section titled “GitHub Actions”](#github-actions) .github/workflows/validate-dumps.yml ```yaml name: Validate SQL Dumps on: push: paths: - "dumps/**/*.sql" pull_request: paths: - "dumps/**/*.sql" jobs: validate: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Install sql-splitter run: cargo install sql-splitter - name: Validate dumps run: | sql-splitter validate "dumps/**/*.sql" \ --strict \ --fail-fast \ --json > validation-results.json - name: Upload results if: always() uses: actions/upload-artifact@v4 with: name: validation-results path: validation-results.json ``` ## GitLab CI [Section titled “GitLab CI”](#gitlab-ci) .gitlab-ci.yml ```yaml validate-dumps: stage: test image: rust:slim before_script: - cargo install sql-splitter script: - sql-splitter validate "dumps/**/*.sql" --strict --fail-fast rules: - changes: - dumps/**/*.sql ``` ## JSON Output for Reporting [Section titled “JSON Output for Reporting”](#json-output-for-reporting) ```bash sql-splitter validate "dumps/*.sql" --json | jq '.results[] | select(.passed == false)' ``` Example output: ```json { "file": "dumps/broken.sql", "passed": false, "size_mb": 12.4, "summary": { "dialect": "mysql", "issues": [ { "code": "DDL_MISSING_TABLE", "severity": "error", "message": "INSERT/COPY references table 'users' with no CREATE TABLE", "location": { "table": "users", "statement_index": 1 } } ], "summary": { "errors": 1, "warnings": 0, "info": 0 }, "checks": { "syntax": "ok", "ddl_dml_consistency": { "failed": 1 } } } } ``` ## Parallel Validation [Section titled “Parallel Validation”](#parallel-validation) For many dump files: ```bash find dumps -name '*.sql.gz' -print0 | \ xargs -0 -n1 -P4 sql-splitter validate --strict ``` ## Pre-commit Hook [Section titled “Pre-commit Hook”](#pre-commit-hook) .git/hooks/pre-commit ```bash #!/bin/bash # Find staged SQL files staged_sql=$(git diff --cached --name-only --diff-filter=ACM | grep '\.sql$' || true) if [ -n "$staged_sql" ]; then echo "Validating SQL files..." echo "$staged_sql" | xargs -n1 sql-splitter validate --strict fi ``` ## Validation in Docker [Section titled “Validation in Docker”](#validation-in-docker) ```dockerfile FROM rust:slim AS builder RUN cargo install sql-splitter FROM debian:bookworm-slim COPY --from=builder /usr/local/cargo/bin/sql-splitter /usr/local/bin/ ENTRYPOINT ["sql-splitter"] ``` ```bash docker build -t sql-splitter . docker run -v $(pwd):/data sql-splitter validate /data/dump.sql --strict ``` ## Best Practices [Section titled “Best Practices”](#best-practices) 1. **Use `--strict`** - Treat warnings as errors 2. **Use `--fail-fast`** - Stop on first error for faster feedback 3. **Use `--json`** - Machine-readable output for reporting 4. **Run on PRs** - Catch issues before merge 5. **Cache the binary** - Speed up CI runs # Database Migration > Migrate between MySQL, PostgreSQL, SQLite, and MSSQL This guide walks through migrating a database from one dialect to another using sql-splitter. ## MySQL to PostgreSQL [Section titled “MySQL to PostgreSQL”](#mysql-to-postgresql) The most common migration path. ### Step 1: Validate Source [Section titled “Step 1: Validate Source”](#step-1-validate-source) ```bash sql-splitter validate prod_mysql.sql.gz --strict --progress ``` ### Step 2: Analyze the Dump [Section titled “Step 2: Analyze the Dump”](#step-2-analyze-the-dump) ```bash sql-splitter analyze prod_mysql.sql.gz --progress ``` Review table sizes to estimate migration time. ### Step 3: Convert [Section titled “Step 3: Convert”](#step-3-convert) ```bash sql-splitter convert prod_mysql.sql.gz \ --to postgres \ -o prod_postgres.sql \ --progress ``` ### Step 4: Validate Output [Section titled “Step 4: Validate Output”](#step-4-validate-output) ```bash sql-splitter validate prod_postgres.sql \ --dialect=postgres \ --strict \ --progress ``` ### Step 5: Import [Section titled “Step 5: Import”](#step-5-import) ```bash psql "$PG_CONN" < prod_postgres.sql ``` Or stream directly (output goes to stdout when `-o` is omitted): ```bash sql-splitter convert prod_mysql.sql.gz --to postgres | psql "$PG_CONN" ``` ## Handling Conversion Warnings [Section titled “Handling Conversion Warnings”](#handling-conversion-warnings) Not all features convert perfectly. Common warnings: ### ENUM Types [Section titled “ENUM Types”](#enum-types) MySQL ENUMs don’t exist in PostgreSQL. Options: 1. Convert to `VARCHAR(255)` (automatic) 2. Create custom TYPE (manual) ### Auto-Increment [Section titled “Auto-Increment”](#auto-increment) Conversion is automatic: * MySQL `AUTO_INCREMENT` → PostgreSQL `SERIAL` * PostgreSQL `SERIAL` → MySQL `AUTO_INCREMENT` ### Session Commands [Section titled “Session Commands”](#session-commands) MySQL-specific `SET` commands are stripped when converting to PostgreSQL. ## MSSQL to MySQL [Section titled “MSSQL to MySQL”](#mssql-to-mysql) ```bash # Convert sql-splitter convert mssql_dump.sql \ --from mssql \ --to mysql \ -o mysql_dump.sql \ --progress # Validate sql-splitter validate mysql_dump.sql --dialect=mysql --strict ``` ### MSSQL-Specific Handling [Section titled “MSSQL-Specific Handling”](#mssql-specific-handling) * `GO` batch separators are removed * `IDENTITY(1,1)` → `AUTO_INCREMENT` * `NVARCHAR` → `VARCHAR` (the output header sets `SET NAMES utf8mb4`) * `DATETIME2` → `DATETIME(6)` ## Best Practices [Section titled “Best Practices”](#best-practices) 1. **Always validate before and after** - Catch issues early 2. **Use `--strict` for CI** - Don’t ignore warnings 3. **Test with a sample first** - Convert a small sample before full dump 4. **Review warnings** - Some may require manual fixes ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) ### ”Unsupported feature” warnings [Section titled “”Unsupported feature” warnings”](#unsupported-feature-warnings) Warnings are printed to stderr by default; use `--json` to inspect them, then address manually: ```bash sql-splitter convert dump.sql --to postgres -o pg.sql --json | jq '.warnings' ``` Add `--strict` to fail the conversion when any warnings are generated. ### Encoding issues [Section titled “Encoding issues”](#encoding-issues) Ensure source dump is UTF-8: ```bash file dump.sql # Should show: UTF-8 Unicode text ``` ### Large file performance [Section titled “Large file performance”](#large-file-performance) For multi-GB dumps: ```bash sql-splitter convert big.sql.gz --to postgres | psql "$PG_CONN" ``` This streams without writing an intermediate file. # Dev Data Seeding > Create anonymized development datasets Create realistic but safe development datasets using sample and redact. ## The Problem [Section titled “The Problem”](#the-problem) Production data is realistic but contains sensitive PII. Random test data lacks realistic relationships and edge cases. ## The Solution [Section titled “The Solution”](#the-solution) 1. **Sample** production data to reduce size 2. **Redact** PII to anonymize 3. **Validate** the result ## Step-by-Step [Section titled “Step-by-Step”](#step-by-step) 1. **Analyze Production Dump** ```bash sql-splitter analyze prod.sql.gz --progress ``` Review table sizes to plan sampling. 2. **Sample with FK Preservation** ```bash sql-splitter sample prod.sql.gz \ -o sampled.sql \ --percent 10 \ --preserve-relations \ --seed 42 \ --progress ``` Key options: * `--preserve-relations`: Ensures FK integrity * `--seed 42`: Reproducible results 3. **Generate Redaction Config** ```bash sql-splitter redact sampled.sql \ --generate-config \ -o redact.yaml ``` Review and customize `redact.yaml`: ```yaml seed: 12345 locale: en rules: - column: "*.email" strategy: hash - column: "*.name" strategy: fake generator: name - column: "*.ssn" strategy: "null" - column: "*.phone" strategy: fake generator: phone - column: "*.credit_card" strategy: mask pattern: "****-****-****-XXXX" skip_tables: - schema_migrations - ar_internal_metadata ``` 4. **Apply Redaction** ```bash sql-splitter redact sampled.sql \ -o dev.sql \ --config redact.yaml \ --progress ``` 5. **Validate Final Output** ```bash sql-splitter validate dev.sql --strict ``` ## One-Liner Approach [Section titled “One-Liner Approach”](#one-liner-approach) For quick anonymization without a config file: ```bash sql-splitter sample prod.sql.gz --percent 10 --preserve-relations -o sampled.sql && \ sql-splitter redact sampled.sql \ --hash "*.email" \ --fake "*.name,*.phone" \ --null "*.ssn,*.password" \ -o dev.sql ``` ## Choosing a Redaction Strategy [Section titled “Choosing a Redaction Strategy”](#choosing-a-redaction-strategy) | Data Type | Strategy | Why | | ------------- | -------- | ----------------------------------------------------- | | Emails | `hash` | Preserves FK relationships (same input → same output) | | Names | `fake` | Realistic looking data | | SSN/Passwords | `null` | Remove entirely | | Credit Cards | `mask` | Keep format for testing | | Timestamps | `skip` | Usually not PII | ## Reproducible Builds [Section titled “Reproducible Builds”](#reproducible-builds) Use `--seed` for deterministic results: ```bash # Same seed = same fake data every time sql-splitter sample prod.sql --percent 10 --seed 42 -o sampled.sql && \ sql-splitter redact sampled.sql --config redact.yaml --seed 42 -o dev.sql ``` ## CI/CD Integration [Section titled “CI/CD Integration”](#cicd-integration) .github/workflows/seed-dev.yml ```yaml jobs: seed-dev: runs-on: ubuntu-latest steps: - name: Download production backup run: aws s3 cp s3://backups/prod.sql.gz . - name: Create dev dataset run: | sql-splitter sample prod.sql.gz \ --percent 10 \ --preserve-relations \ --seed ${{ github.run_id }} \ -o sampled.sql sql-splitter redact sampled.sql \ --config redact.yaml \ -o dev.sql - name: Validate run: sql-splitter validate dev.sql --strict - name: Upload run: aws s3 cp dev.sql s3://dev-data/ ``` # Docker Usage > Running sql-splitter in Docker containers sql-splitter works seamlessly in Docker containers for CI/CD pipelines and reproducible environments. ## Quick Start [Section titled “Quick Start”](#quick-start) ### Pre-built Images [Section titled “Pre-built Images”](#pre-built-images) Multi-arch images (amd64, arm64) are published to Docker Hub and GHCR on every release: ```bash # Docker Hub docker run --rm -v $(pwd):/data helgesverre/sql-splitter analyze /data/dump.sql # GitHub Container Registry docker run --rm -v $(pwd):/data ghcr.io/helgesverre/sql-splitter analyze /data/dump.sql ``` Tags: `latest`, exact versions (`1.13.8`), and partial semver (`1.13`, `1`). ### Building the Image [Section titled “Building the Image”](#building-the-image) ```bash # Clone and build git clone https://github.com/helgesverre/sql-splitter cd sql-splitter # Build optimized image docker build -t sql-splitter -f - . <<'EOF' FROM rust:slim AS builder WORKDIR /app COPY . . RUN cargo build --release FROM debian:bookworm-slim COPY --from=builder /app/target/release/sql-splitter /usr/local/bin/ ENTRYPOINT ["sql-splitter"] EOF # Run docker run --rm -v $(pwd):/data sql-splitter analyze /data/dump.sql ``` ## CI/CD Examples [Section titled “CI/CD Examples”](#cicd-examples) * GitHub Actions ```yaml name: Validate SQL Dumps on: push: paths: - 'migrations/*.sql' jobs: validate: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Install sql-splitter run: cargo install sql-splitter - name: Validate migrations run: sql-splitter validate "migrations/*.sql" --strict - name: Check for breaking changes run: | sql-splitter diff migrations/prev.sql migrations/current.sql \ --format json > diff.json if jq -e '.schema.tables_removed | length > 0' diff.json; then echo "Breaking change: tables removed" exit 1 fi ``` * GitLab CI ```yaml validate-sql: image: rust:slim before_script: - cargo install sql-splitter script: - sql-splitter validate "dumps/*.sql" --strict --json > validation.json artifacts: paths: - validation.json when: always ``` * CircleCI ```yaml version: 2.1 jobs: validate: docker: - image: rust:slim steps: - checkout - run: name: Install sql-splitter command: cargo install sql-splitter - run: name: Validate SQL dumps command: sql-splitter validate "*.sql" --strict ``` ## Docker Compose [Section titled “Docker Compose”](#docker-compose) ### Development Database Seeding [Section titled “Development Database Seeding”](#development-database-seeding) docker-compose.yml ```yaml version: "3.8" services: db: image: mysql:8.0 environment: MYSQL_ROOT_PASSWORD: root MYSQL_DATABASE: app volumes: - ./init:/docker-entrypoint-initdb.d healthcheck: test: mysqladmin ping -h localhost interval: 5s timeout: 5s retries: 10 seed: image: rust:slim depends_on: db: condition: service_healthy volumes: - ./dumps:/data command: > sh -c " cargo install sql-splitter && sql-splitter sample /data/prod.sql.gz \ --percent 10 \ --preserve-relations \ -o /data/dev.sql && sql-splitter redact /data/dev.sql \ --hash '*.email' \ --fake '*.name' \ -o /data/safe.sql " ``` ### Multi-Tenant Extraction [Section titled “Multi-Tenant Extraction”](#multi-tenant-extraction) ```yaml # Extract tenant data for isolated testing services: extract-tenant: build: context: . dockerfile: Dockerfile.sql-splitter volumes: - ./dumps:/data command: > shard /data/prod.sql --tenant-column company_id --tenant-values "1,2,3" -o /data/tenants/ ``` ## Dockerfile Best Practices [Section titled “Dockerfile Best Practices”](#dockerfile-best-practices) ### Multi-stage Build (Smallest Image) [Section titled “Multi-stage Build (Smallest Image)”](#multi-stage-build-smallest-image) ```dockerfile # Build stage FROM rust:slim AS builder WORKDIR /app RUN apt-get update && apt-get install -y pkg-config libssl-dev COPY . . RUN cargo build --release # Runtime stage FROM debian:bookworm-slim RUN apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/* COPY --from=builder /app/target/release/sql-splitter /usr/local/bin/ ENTRYPOINT ["sql-splitter"] ``` ### With Shell for Piping [Section titled “With Shell for Piping”](#with-shell-for-piping) ```dockerfile FROM debian:bookworm-slim RUN apt-get update && \ apt-get install -y ca-certificates curl && \ rm -rf /var/lib/apt/lists/* # Install from GitHub releases (replace VERSION with actual version) ARG VERSION=1.13.7 RUN apt-get update && apt-get install -y xz-utils && rm -rf /var/lib/apt/lists/* RUN curl -L https://github.com/helgesverre/sql-splitter/releases/download/v${VERSION}/sql-splitter-x86_64-unknown-linux-gnu.tar.xz | \ tar -xJ --strip-components=1 -C /usr/local/bin/ \ sql-splitter-x86_64-unknown-linux-gnu/sql-splitter # Use shell entrypoint for piping support ENTRYPOINT ["/bin/sh", "-c"] ``` ## Volume Mounting [Section titled “Volume Mounting”](#volume-mounting) ### Read-Only Input [Section titled “Read-Only Input”](#read-only-input) ```bash # Mount input as read-only, output as read-write docker run --rm \ -v $(pwd)/input:/input:ro \ -v $(pwd)/output:/output \ sql-splitter split /input/dump.sql -o /output/tables/ ``` ### Working with Compressed Files [Section titled “Working with Compressed Files”](#working-with-compressed-files) ```bash # Process compressed files (sql-splitter handles .gz, .bz2, .xz, .zst) docker run --rm \ -v $(pwd):/data \ sql-splitter analyze /data/backup.sql.gz --progress ``` ## Performance Tips [Section titled “Performance Tips”](#performance-tips) ### Use Native CPU Features [Section titled “Use Native CPU Features”](#use-native-cpu-features) When building for your specific architecture: ```dockerfile FROM rust:slim AS builder WORKDIR /app COPY . . ENV RUSTFLAGS="-C target-cpu=native" RUN cargo build --release ``` ### Memory Limits [Section titled “Memory Limits”](#memory-limits) sql-splitter uses \~20–150 MB memory regardless of file size, so you can set conservative limits: ```bash docker run --rm \ --memory=256m \ -v $(pwd):/data \ sql-splitter split /data/huge.sql.gz -o /data/tables/ ``` ### Parallel Processing [Section titled “Parallel Processing”](#parallel-processing) For many files, run multiple containers: ```bash # Process files in parallel using GNU parallel find dumps -name '*.sql.gz' | \ parallel -j4 docker run --rm -v $(pwd):/data sql-splitter validate /data/{} ``` ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) ### Permission Denied [Section titled “Permission Denied”](#permission-denied) ```bash # Run as current user docker run --rm \ --user $(id -u):$(id -g) \ -v $(pwd):/data \ sql-splitter split /data/dump.sql -o /data/output/ ``` ### File Not Found [Section titled “File Not Found”](#file-not-found) ```bash # Ensure absolute paths inside container docker run --rm \ -v /absolute/path/to/dumps:/data \ sql-splitter analyze /data/dump.sql ``` ## See Also [Section titled “See Also”](#see-also) * [CI Validation](/cookbook/ci-validation/) - Automated validation workflows * [Compression](/reference/compression) - Supported compression formats * [Performance](/advanced/performance) - Optimization tips # Multi-Tenant Sharding > Extract tenant-specific data from multi-tenant dumps Extract data for specific tenants from multi-tenant SQL dumps. ## Use Cases [Section titled “Use Cases”](#use-cases) * **Tenant isolation**: Give customers their own data export * **Debugging**: Reproduce issues with specific tenant data * **Testing**: Create tenant-specific test fixtures * **Compliance**: Data portability requests ## Basic Extraction [Section titled “Basic Extraction”](#basic-extraction) ```bash sql-splitter shard multi_tenant.sql \ --tenant-value 123 \ -o tenant_123.sql \ --progress ``` ## How It Works [Section titled “How It Works”](#how-it-works) 1. Identifies tables with the tenant column 2. Finds all rows matching the tenant value 3. Follows foreign keys to include related data 4. Outputs a complete, consistent dump for that tenant ## Tenant Column Detection [Section titled “Tenant Column Detection”](#tenant-column-detection) sql-splitter auto-detects common column names: * `company_id` * `tenant_id` * `organization_id` * `org_id` * `account_id` * `team_id` * `workspace_id` Override if needed: ```bash sql-splitter shard dump.sql \ --tenant-column customer_id \ --tenant-value 42 \ -o tenant.sql ``` ## Multiple Tenants [Section titled “Multiple Tenants”](#multiple-tenants) Extract multiple tenants to separate files: ```bash sql-splitter shard multi_tenant.sql \ --tenant-values "1,2,3" \ -o shards/ \ --progress ``` Creates: ```plaintext shards/ ├── tenant_1.sql ├── tenant_2.sql └── tenant_3.sql ``` ## Handling Global Tables [Section titled “Handling Global Tables”](#handling-global-tables) Multi-tenant apps often have shared/lookup tables. Control how they’re handled: ```bash # Include small lookup tables (default) sql-splitter shard dump.sql --tenant-value 123 --include-global lookups # Include all non-tenant tables sql-splitter shard dump.sql --tenant-value 123 --include-global all # Skip global tables entirely sql-splitter shard dump.sql --tenant-value 123 --include-global none ``` ## Example Schema [Section titled “Example Schema”](#example-schema) ```sql -- Global tables (no tenant_id) CREATE TABLE countries (id INT PRIMARY KEY, name VARCHAR(100)); CREATE TABLE currencies (id INT PRIMARY KEY, code VARCHAR(3)); -- Tenant tables CREATE TABLE tenants (id INT PRIMARY KEY, name VARCHAR(100)); CREATE TABLE users ( id INT PRIMARY KEY, tenant_id INT REFERENCES tenants(id), name VARCHAR(100) ); CREATE TABLE orders ( id INT PRIMARY KEY, tenant_id INT REFERENCES tenants(id), user_id INT REFERENCES users(id) ); ``` Sharding for tenant 42: * Includes all `users` and `orders` where `tenant_id = 42` * Follows FK to include the `tenants` row * Includes `countries` and `currencies` (with `--include-global lookups`) ## Combining with Sample [Section titled “Combining with Sample”](#combining-with-sample) For large tenants, combine with sampling: ```bash # First shard sql-splitter shard dump.sql --tenant-value 123 -o tenant_123.sql # Then sample sql-splitter sample tenant_123.sql \ -o tenant_123_sample.sql \ --percent 10 \ --preserve-relations ``` ## Combining with Redact [Section titled “Combining with Redact”](#combining-with-redact) Anonymize after sharding: ```bash sql-splitter shard dump.sql --tenant-value 123 -o tenant_123.sql && \ sql-splitter redact tenant_123.sql \ --hash "*.email" \ --fake "*.name" \ -o tenant_123_safe.sql ``` # Schema Visualization > Generate ERD diagrams from SQL dumps Visualize database schemas as Entity-Relationship Diagrams. ## Quick Start [Section titled “Quick Start”](#quick-start) ```bash # Interactive HTML diagram sql-splitter graph dump.sql -o schema.html ``` Open `schema.html` in a browser for an interactive diagram with: * Dark/light theme toggle * Pan and zoom * Click to highlight relationships ## Output Formats [Section titled “Output Formats”](#output-formats) ### HTML (Interactive) [Section titled “HTML (Interactive)”](#html-interactive) ```bash sql-splitter graph dump.sql -o schema.html ``` Best for exploration and presentations. ### Mermaid (Markdown) [Section titled “Mermaid (Markdown)”](#mermaid-markdown) ```bash sql-splitter graph dump.sql -o schema.mmd --format mermaid ``` Embed in Markdown documentation: ````markdown ```mermaid erDiagram users { INT id PK VARCHAR name VARCHAR email } orders { INT id PK INT user_id FK } orders }|--|| users : user_id ``` ```` ### Graphviz DOT [Section titled “Graphviz DOT”](#graphviz-dot) ```bash sql-splitter graph dump.sql -o schema.dot ``` Render with Graphviz: ```bash dot -Tpng schema.dot -o schema.png dot -Tsvg schema.dot -o schema.svg dot -Tpdf schema.dot -o schema.pdf ``` Or use `--render`: ```bash sql-splitter graph dump.sql -o schema.png --render ``` ### JSON (Data) [Section titled “JSON (Data)”](#json-data) ```bash sql-splitter graph dump.sql --json -o schema.json ``` For programmatic processing. ## Filtering Tables [Section titled “Filtering Tables”](#filtering-tables) ### Include specific tables [Section titled “Include specific tables”](#include-specific-tables) ```bash sql-splitter graph dump.sql --tables "user*,order*" -o filtered.html ``` ### Exclude tables [Section titled “Exclude tables”](#exclude-tables) ```bash sql-splitter graph dump.sql --exclude "log*,audit*,temp*" -o clean.html ``` ## Focus Mode [Section titled “Focus Mode”](#focus-mode) ### Show dependencies [Section titled “Show dependencies”](#show-dependencies) Show a table and everything it depends on: ```bash sql-splitter graph dump.sql --table orders --transitive -o orders-deps.html ``` ### Show dependents [Section titled “Show dependents”](#show-dependents) Show a table and everything that depends on it: ```bash sql-splitter graph dump.sql --table users --reverse -o users-deps.html ``` ### Limit depth [Section titled “Limit depth”](#limit-depth) ```bash sql-splitter graph dump.sql --table orders --transitive --max-depth 2 ``` ## Detecting Cycles [Section titled “Detecting Cycles”](#detecting-cycles) Find circular foreign key dependencies: ```bash sql-splitter graph dump.sql --cycles-only -o cycles.html ``` The cycle list is printed to stderr, and the diagram itself is filtered to only the tables involved: ```plaintext Cycles detected (1): 1. orders -> payments -> invoices -> orders ``` ## Layout Options [Section titled “Layout Options”](#layout-options) ```bash # Horizontal (left-to-right) sql-splitter graph dump.sql --layout lr -o horizontal.html # Vertical (top-to-bottom) sql-splitter graph dump.sql --layout tb -o vertical.html ``` ## Documentation Integration [Section titled “Documentation Integration”](#documentation-integration) ### GitHub README [Section titled “GitHub README”](#github-readme) ```bash sql-splitter graph dump.sql -o schema.mmd --format mermaid ``` Include in README.md: ````markdown ## Database Schema ```mermaid %%{init: {'theme': 'dark'}}%% erDiagram ... ``` ```` ### Notion/Confluence [Section titled “Notion/Confluence”](#notionconfluence) 1. Generate Mermaid output 2. Paste into Mermaid code block ### Static Site [Section titled “Static Site”](#static-site) 1. Generate HTML 2. Embed in an iframe or link directly # Overview > Technical reference documentation for sql-splitter Detailed technical documentation. | Reference | Description | | ------------------------------------------- | ---------------------------------------- | | [Dialects](/reference/dialects) | MySQL, PostgreSQL, SQLite, MSSQL details | | [Type Mappings](/reference/type-mappings) | Data type conversion tables | | [JSON Output](/reference/json-output) | JSON schemas for `--json` output | | [Compression](/reference/compression) | Supported compression formats | | [Redact Config](/reference/redact-config) | YAML format for redaction rules | | [Exit Codes](/reference/exit-codes) | CLI exit codes and meanings | | [Known Limitations](/reference/limitations) | What sql-splitter does not parse | | [Library Usage](/reference/library-usage) | Using sql-splitter as a Rust crate | | [Glossary](/reference/glossary) | Database and SQL terminology | # Compression Support > Supported compression formats for input files sql-splitter automatically detects and decompresses compressed input files based on file extension. ## Supported Formats [Section titled “Supported Formats”](#supported-formats) | Format | Extension | Library | | --------- | --------- | ------- | | Gzip | `.gz` | flate2 | | Bzip2 | `.bz2` | bzip2 | | XZ/LZMA | `.xz` | xz2 | | Zstandard | `.zst` | zstd | ## Usage [Section titled “Usage”](#usage) Simply pass a compressed file—no flags needed: ```bash # Gzip sql-splitter split backup.sql.gz -o tables/ # Bzip2 sql-splitter analyze database.sql.bz2 # XZ sql-splitter validate dump.sql.xz # Zstandard sql-splitter convert mysql.sql.zst --to postgres -o pg.sql ``` ## Commands Supporting Compression [Section titled “Commands Supporting Compression”](#commands-supporting-compression) Nearly every command that accepts an input file supports compressed input: ```bash sql-splitter split backup.sql.gz -o tables/ sql-splitter analyze backup.sql.gz sql-splitter merge tables/ -o merged.sql # Note: merge reads directory, not compressed file sql-splitter sample backup.sql.gz --percent 10 -o sample.sql sql-splitter shard backup.sql.gz --tenant-value 123 -o tenant.sql sql-splitter convert backup.sql.gz --to postgres -o pg.sql sql-splitter validate backup.sql.gz --strict sql-splitter diff old.sql.gz new.sql.gz sql-splitter graph backup.sql.gz -o schema.html sql-splitter order backup.sql.gz -o ordered.sql sql-splitter query backup.sql.gz "SELECT COUNT(*) FROM users" ``` **Exception:** `redact` does not decompress input. Decompress first, then redact: ```bash zcat backup.sql.gz > backup.sql sql-splitter redact backup.sql --hash "*.email" -o safe.sql ``` ## Compression on Output [Section titled “Compression on Output”](#compression-on-output) sql-splitter does not compress output directly. Use pipes for compressed output: ```bash # Zstandard output (faster, better compression) sql-splitter sample dump.sql --percent 10 | zstd > sample.sql.zst # Bzip2 output sql-splitter convert mysql.sql --to postgres | bzip2 > pg.sql.bz2 # Merge and compress sql-splitter merge tables/ | gzip > merged.sql.gz ``` When output goes to stdout, status lines are written to stderr, so piped SQL stays clean. ## Performance Notes [Section titled “Performance Notes”](#performance-notes) * **Gzip**: Good balance of speed and compression, widely supported * **Zstandard**: Fastest decompression, excellent compression ratio, recommended for large files * **XZ**: Best compression ratio but slower, good for archival * **Bzip2**: Moderate speed and compression, legacy format For best performance with very large dumps, Zstandard (`.zst`) is recommended: ```bash # Compress with zstd for optimal speed zstd -T0 huge-dump.sql -o huge-dump.sql.zst # Process compressed file sql-splitter analyze huge-dump.sql.zst --progress ``` ## No Stdin Input [Section titled “No Stdin Input”](#no-stdin-input) sql-splitter reads from file paths only—`-` for stdin is not supported, and process substitution (`<(zcat ...)`) fails because the input is read more than once (dialect detection, then parsing). Pass the compressed file directly instead: ```bash sql-splitter analyze backup.sql.gz ``` ## See Also [Section titled “See Also”](#see-also) * [Unix Piping](/advanced/piping) - Composing commands with pipes * [Performance](/advanced/performance) - Optimization tips # SQL Dialects > Supported SQL dialects and their features sql-splitter supports four SQL dialects. * MySQL **Source**: `mysqldump` **Detection signals**: * `MySQL dump` / `MariaDB dump` header comment * Conditional comments (`/*!40...`, `/*!50...`) * `LOCK TABLES` * Backtick quoting (`` `table` ``) **Features supported**: * `CREATE TABLE` with inline keys * `INSERT INTO` (single and multi-row) * `CREATE INDEX` * `ALTER TABLE` * Session commands (`SET`, `USE`) are recognized but carry no table, so `split` does not write them to any output file **Example**: ```sql CREATE TABLE `users` ( `id` INT AUTO_INCREMENT PRIMARY KEY, `name` VARCHAR(100) NOT NULL, `email` VARCHAR(255), INDEX `idx_email` (`email`) ) ENGINE=InnoDB; INSERT INTO `users` VALUES (1, 'Alice', 'alice@example.com'); ``` * PostgreSQL **Source**: `pg_dump` (plain text format) **Detection signals**: * `pg_dump` / `PostgreSQL database dump` header comment * `COPY ... FROM stdin` * `search_path` * Dollar quoting (`$$`) and `CREATE EXTENSION` **Features supported**: * `CREATE TABLE` * `COPY ... FROM stdin` (tab-separated data) * `INSERT INTO` * `CREATE INDEX` * `ALTER TABLE` * Sequences (`CREATE SEQUENCE`) are recognized but carry no table, so `split` does not write them to any output file **Example**: ```sql CREATE TABLE users ( id SERIAL PRIMARY KEY, name VARCHAR(100) NOT NULL, email VARCHAR(255) ); COPY users (id, name, email) FROM stdin; 1 Alice alice@example.com 2 Bob bob@example.com \. ``` **Note**: Binary/custom `pg_dump` formats are not supported. Use `pg_dump -Fp` (plain). * SQLite **Source**: `.dump` command **Detection signals**: * `SQLite` header comment * `PRAGMA` statements * `BEGIN TRANSACTION` **Features supported**: * `CREATE TABLE` * `INSERT INTO` * `CREATE INDEX` * `PRAGMA` statements are recognized but carry no table, so `split` does not write them to any output file **Example**: ```sql CREATE TABLE users ( id INTEGER PRIMARY KEY, name TEXT NOT NULL, email TEXT ); INSERT INTO users VALUES (1, 'Alice', 'alice@example.com'); ``` * MSSQL **Source**: `sqlcmd`, SSMS “Generate Scripts” **Detection signals**: * `SET ANSI_NULLS`, `SET QUOTED_IDENTIFIER` * `GO` batch separator * `IDENTITY(` * Square bracket quoting (`[table]`) * `NVARCHAR`, `N'...'` literals **Features supported**: * `CREATE TABLE` * `INSERT INTO` * `CREATE INDEX` * `GO` batch handling **Example**: ```sql CREATE TABLE [users] ( [id] INT IDENTITY(1,1) PRIMARY KEY, [name] NVARCHAR(100) NOT NULL, [email] NVARCHAR(255) ); GO INSERT INTO [users] ([name], [email]) VALUES (N'Alice', N'alice@example.com'); GO ``` ## Auto-Detection [Section titled “Auto-Detection”](#auto-detection) sql-splitter scores dialect markers in the first 8 KB of the file. Ambiguous files default to MySQL. Override with `--dialect`: ```bash sql-splitter split dump.sql --dialect=postgres sql-splitter convert dump.sql --from mysql --to postgres ``` Detection can guess wrong when marker-like text appears in data or comments early in the file — see [Known Limitations](/reference/limitations/) for the failure modes, plus what each dialect parser does not handle (stored procedures, binary formats, and more). # Exit Codes > CLI exit codes and their meanings sql-splitter uses standard Unix exit codes. ## Exit Codes [Section titled “Exit Codes”](#exit-codes) | Code | Meaning | When | | ---- | ----------------- | --------------------------------------- | | `0` | Success | Command completed without errors | | `1` | Error | Runtime error (I/O, syntax, validation) | | `2` | Invalid arguments | Bad CLI arguments | ## Usage in Scripts [Section titled “Usage in Scripts”](#usage-in-scripts) ### Bash [Section titled “Bash”](#bash) ```bash #!/bin/bash set -e # Exit on error sql-splitter validate dump.sql --strict echo "Validation passed" ``` Or with explicit handling: ```bash if sql-splitter validate dump.sql --strict; then echo "Valid" else echo "Invalid" exit 1 fi ``` ### CI/CD [Section titled “CI/CD”](#cicd) ```yaml # GitHub Actions - name: Validate run: sql-splitter validate dump.sql --strict # Fails step if exit code != 0 ``` ## Validation Exit Codes [Section titled “Validation Exit Codes”](#validation-exit-codes) With `--strict`: * Exit `0`: No errors AND no warnings * Exit `1`: Errors or warnings present Without `--strict`: * Exit `0`: No errors (warnings are ignored) * Exit `1`: Errors present ## Checking Exit Codes [Section titled “Checking Exit Codes”](#checking-exit-codes) ```bash sql-splitter validate dump.sql --strict EXIT_CODE=$? case $EXIT_CODE in 0) echo "Success" ;; 1) echo "Validation failed" ;; 2) echo "Invalid arguments" ;; *) echo "Unknown error: $EXIT_CODE" ;; esac ``` ## JSON Output with Exit Codes [Section titled “JSON Output with Exit Codes”](#json-output-with-exit-codes) Single-file JSON output includes an error count in `summary`: ```bash result=$(sql-splitter validate dump.sql --json) errors=$(echo "$result" | jq -r '.summary.errors') if [ "$errors" = "0" ]; then echo "Valid" else echo "Invalid" fi ``` In multi-file mode (glob input), each entry in `results` has a boolean `passed` field: ```bash sql-splitter validate "backups/*.sql" --json | jq '.results[] | select(.passed | not) | .file' ``` # Glossary > Technical terms used throughout sql-splitter documentation This glossary explains database and SQL terminology used in sql-splitter documentation. Terms are grouped by category for easier reference. ## Keys, Constraints, and Integrity [Section titled “Keys, Constraints, and Integrity”](#keys-constraints-and-integrity) ### Primary Key (PK) [Section titled “Primary Key (PK)”](#primary-key-pk) A column (or set of columns) whose values uniquely identify each row in a table. Primary keys cannot be NULL and are often referenced by foreign keys in other tables. ### Foreign Key (FK) [Section titled “Foreign Key (FK)”](#foreign-key-fk) A constraint that requires values in one table’s column(s) to match existing values in another table’s primary key. Foreign keys are the main mechanism for representing relationships between tables. ### Constraint [Section titled “Constraint”](#constraint) A rule the database enforces on table data or structure. Common constraints include: * **PRIMARY KEY** - Unique row identifier * **FOREIGN KEY** - Reference to another table * **UNIQUE** - No duplicate values allowed * **NOT NULL** - Value required * **CHECK** - Custom validation (e.g., `age >= 0`) ### Referential Integrity [Section titled “Referential Integrity”](#referential-integrity) The guarantee that relationships between tables remain consistent. For example, you can’t have an `orders.user_id` that references a `users.id` that doesn’t exist. Foreign keys enforce referential integrity. ### CASCADE [Section titled “CASCADE”](#cascade) A referential action on a foreign key that automatically propagates changes. For example, `ON DELETE CASCADE` deletes child rows when the parent row is deleted. Similarly, `ON UPDATE CASCADE` updates child foreign key values when the parent key changes. ### Deferred Constraints [Section titled “Deferred Constraints”](#deferred-constraints) Constraints that are checked at the end of a transaction instead of immediately after each statement. This allows temporarily “inconsistent” states while loading data, as long as everything is valid by commit time. Useful for loading tables with circular dependencies. ```sql -- PostgreSQL syntax SET CONSTRAINTS ALL DEFERRED; ``` ## SQL Structure [Section titled “SQL Structure”](#sql-structure) ### Schema [Section titled “Schema”](#schema) The organized definition of database objects—tables, columns, types, constraints, indexes, etc. In PostgreSQL, “schema” also refers to a namespace (like `public`), but in sql-splitter docs, “schema” typically means the overall database structure. ### DDL (Data Definition Language) [Section titled “DDL (Data Definition Language)”](#ddl-data-definition-language) SQL statements that define or change database structure: * `CREATE TABLE`, `ALTER TABLE`, `DROP TABLE` * `CREATE INDEX`, `DROP INDEX` * `CREATE VIEW` DDL describes *what the database looks like*. ### DML (Data Manipulation Language) [Section titled “DML (Data Manipulation Language)”](#dml-data-manipulation-language) SQL statements that work with data: * `INSERT`, `UPDATE`, `DELETE` * `COPY ... FROM stdin` (PostgreSQL bulk load) DML describes *what data is in the database*. ## Dialect-Specific Syntax [Section titled “Dialect-Specific Syntax”](#dialect-specific-syntax) ### SQL Dialect [Section titled “SQL Dialect”](#sql-dialect) The specific flavor of SQL supported by a database engine. sql-splitter supports four dialects: | Dialect | Key Features | | ---------- | ----------------------------------------------------- | | MySQL | Backtick quoting, `AUTO_INCREMENT`, `ENGINE=InnoDB` | | PostgreSQL | Double-quote identifiers, `SERIAL`, `COPY FROM stdin` | | SQLite | `INTEGER PRIMARY KEY` for auto-increment | | MSSQL | `IDENTITY`, `GO` batch separators, `NVARCHAR` | ### AUTO\_INCREMENT (MySQL) [Section titled “AUTO\_INCREMENT (MySQL)”](#auto_increment-mysql) A column attribute that automatically generates increasing numeric values. Commonly used for surrogate primary keys. ```sql id INT AUTO_INCREMENT PRIMARY KEY ``` ### SERIAL (PostgreSQL) [Section titled “SERIAL (PostgreSQL)”](#serial-postgresql) A convenience type that creates an integer column backed by an auto-incrementing sequence. Modern PostgreSQL prefers `IDENTITY`. ```sql id SERIAL PRIMARY KEY -- Equivalent to: id INTEGER DEFAULT nextval('tablename_id_seq') ``` ### IDENTITY (PostgreSQL/MSSQL) [Section titled “IDENTITY (PostgreSQL/MSSQL)”](#identity-postgresqlmssql) A standards-aligned way to define auto-generated numeric columns. ```sql -- PostgreSQL id INT GENERATED BY DEFAULT AS IDENTITY -- MSSQL id INT IDENTITY(1,1) ``` ### COPY FROM stdin (PostgreSQL) [Section titled “COPY FROM stdin (PostgreSQL)”](#copy-from-stdin-postgresql) A PostgreSQL bulk-loading command that reads table data from the client connection. In `pg_dump` output, it appears as: ```sql COPY users (id, name, email) FROM stdin; 1 Alice alice@example.com 2 Bob bob@example.com \. ``` The `\.` marks the end of data. sql-splitter’s [`convert`](/commands/convert) command transforms this to INSERT statements for other dialects. ## Dependency Graphs and Ordering [Section titled “Dependency Graphs and Ordering”](#dependency-graphs-and-ordering) ### Dependency Graph [Section titled “Dependency Graph”](#dependency-graph) A directed graph where nodes represent tables and edges represent foreign key relationships. Used to determine safe ordering for creating tables and loading data. ### Topological Order / Topological Sort [Section titled “Topological Order / Topological Sort”](#topological-order--topological-sort) An ordering of tables such that every dependency (parent table) appears before the table that depends on it (child table). In SQL dumps, this means parent tables are created and populated before child tables, satisfying foreign key constraints during import. See [`order`](/commands/order) for reordering dumps topologically. ### Circular Dependency / Cycle [Section titled “Circular Dependency / Cycle”](#circular-dependency--cycle) A situation where following foreign key relationships leads back to the starting table. For example: `orders → payments → invoices → orders`. Cycles prevent simple topological ordering and require special handling: * Defer constraints during import * Disable FK checks temporarily * Split data loading into phases See [`graph --cycles-only`](/commands/graph) to detect and visualize cycles. ### Root Table [Section titled “Root Table”](#root-table) A table with no outbound foreign keys (it doesn’t reference other tables). Root tables can be loaded first since they have no dependencies. ### Parent Table [Section titled “Parent Table”](#parent-table) The referenced table in a foreign key relationship—the table that “owns” the primary key being referenced. Parent tables must generally exist before child tables can reference them. ### Child Table [Section titled “Child Table”](#child-table) The referencing table in a foreign key relationship—it contains the foreign key column(s). Child rows depend on parent rows existing. ## Data Processing [Section titled “Data Processing”](#data-processing) ### Streaming [Section titled “Streaming”](#streaming) Processing a dump file incrementally as it’s read, rather than loading the entire file into memory. **Commands that stream (bounded memory, independent of file size):** * `split`, `merge`, `convert`, `analyze`, `redact` **Commands that require more memory:** * `query` - Loads data into DuckDB (use `--disk` for large files) * `diff` - Tracks PK hashes for comparison (use `--schema-only` to reduce) * `validate` - Tracks PKs/FKs when checking integrity (use `--no-fk-checks` to skip) * `order`, `graph` - Build dependency graph (memory scales with number of tables/FKs) * `sample`, `shard` - Track selected rows when preserving FK relationships See [Memory & Streaming](/advanced/memory-streaming) for detailed memory profiles per command. ### Deterministic [Section titled “Deterministic”](#deterministic) Given the same input and configuration, the tool produces identical output every run. Determinism is important for CI pipelines, debugging, and reproducible data workflows. Use `--seed` with sampling/redaction operations for deterministic “random” results. ### Seed [Section titled “Seed”](#seed) A fixed value used to initialize a pseudo-random number generator. Using the same seed produces the same “random” results every time, making operations reproducible. ```bash sql-splitter sample dump.sql --percent 10 --seed 42 ``` ## Data Privacy and Anonymization [Section titled “Data Privacy and Anonymization”](#data-privacy-and-anonymization) ### PII (Personally Identifiable Information) [Section titled “PII (Personally Identifiable Information)”](#pii-personally-identifiable-information) Data that can identify a person directly or indirectly: names, email addresses, phone numbers, IP addresses, government IDs, etc. Many regulations require protecting PII, including [GDPR](https://gdpr.eu/) (EU) and [CCPA](https://oag.ca.gov/privacy/ccpa) (California). ### Redaction [Section titled “Redaction”](#redaction) Removing or masking sensitive values entirely, typically replacing them with NULL, asterisks, or a placeholder like `[REDACTED]`. Redaction prioritizes safety over data usefulness. ### Anonymization [Section titled “Anonymization”](#anonymization) Transforming data to reduce identification risk while keeping it useful for testing or development. Techniques include replacing real names with fake names, hashing emails, or shuffling values. ### Hashing [Section titled “Hashing”](#hashing) Converting a value into a fixed-length digest (e.g., SHA-256) that cannot be reversed. Hashing preserves equality (same input → same hash), which is useful for maintaining foreign key relationships during anonymization. ```bash sql-splitter redact dump.sql --hash "*.email" -o safe.sql ``` ## Multi-Tenant Concepts [Section titled “Multi-Tenant Concepts”](#multi-tenant-concepts) ### Tenant [Section titled “Tenant”](#tenant) A customer, account, or organization whose data is logically separated from others within the same database. Multi-tenant systems store multiple tenants’ data in shared infrastructure. ### Multi-Tenant [Section titled “Multi-Tenant”](#multi-tenant) An architecture where a single database serves multiple tenants, typically with row-level isolation via a tenant identifier column. This is common in SaaS applications. ### Sharding [Section titled “Sharding”](#sharding) In sql-splitter context, extracting a subset of data belonging to a specific tenant from a multi-tenant database dump. See [`shard`](/commands/shard). ### Tenant Column [Section titled “Tenant Column”](#tenant-column) A column (typically `tenant_id`, `org_id`, `company_id`, `account_id`) present in tables that indicates which tenant owns each row. sql-splitter auto-detects common tenant column names. ## Visualization and Documentation [Section titled “Visualization and Documentation”](#visualization-and-documentation) ### ERD (Entity Relationship Diagram) [Section titled “ERD (Entity Relationship Diagram)”](#erd-entity-relationship-diagram) A diagram showing tables, their columns, and the relationships (foreign keys) between them. ERDs help visualize database structure and identify dependency cycles. See [`graph`](/commands/graph) for generating ERDs from SQL dumps. ### Schema Drift [Section titled “Schema Drift”](#schema-drift) When database structure differs across environments (development, staging, production) or differs from expected migrations. Drift can cause import failures or data inconsistencies. See [`diff`](/commands/diff) for detecting schema differences between dumps. ## File Patterns [Section titled “File Patterns”](#file-patterns) ### Glob Pattern [Section titled “Glob Pattern”](#glob-pattern) A simple pattern syntax for matching file paths: | Pattern | Matches | | ----------------------- | ----------------------------------------------- | | `*.sql` | All SQL files in current directory | | `**/*.sql` | All SQL files recursively | | `dump_*.sql.gz` | Files starting with `dump_` ending in `.sql.gz` | | `backups/202[3-4]*.sql` | Backups from 2023-2024 | Most sql-splitter commands accept glob patterns: ```bash sql-splitter validate "backups/**/*.sql" --fail-fast ``` # JSON Output Reference > Machine-readable JSON output formats and schemas for each command Most commands support `--json` for machine-readable output. This page documents the JSON structure for each command. ## General Notes [Section titled “General Notes”](#general-notes) * All commands output valid JSON to stdout * Use `jq` for parsing and filtering * The `query` command uses `--format json` instead of `--json` * Multi-file operations return aggregate results with a `results` array ## JSON Schema Files [Section titled “JSON Schema Files”](#json-schema-files) Formal JSON Schema files (draft 2020-12) are available in the [schemas/](/schemas/) directory for validation and code generation: | Command | Schema File | | ---------- | ----------------------------------------------------- | | `analyze` | [analyze.schema.json](/schemas/analyze.schema.json) | | `validate` | [validate.schema.json](/schemas/validate.schema.json) | | `split` | [split.schema.json](/schemas/split.schema.json) | | `merge` | [merge.schema.json](/schemas/merge.schema.json) | | `sample` | [sample.schema.json](/schemas/sample.schema.json) | | `convert` | [convert.schema.json](/schemas/convert.schema.json) | | `redact` | [redact.schema.json](/schemas/redact.schema.json) | | `graph` | [graph.schema.json](/schemas/graph.schema.json) | | `shard` | [shard.schema.json](/schemas/shard.schema.json) | ## Common Fields [Section titled “Common Fields”](#common-fields) ### Dialect [Section titled “Dialect”](#dialect) Most commands include a `dialect` field (`convert` reports `conversion.from`/`conversion.to` instead, and `redact` and `graph` omit it): ```json "dialect": "mysql" | "postgres" | "sqlite" | "mssql" ``` ### Statistics [Section titled “Statistics”](#statistics) Most commands include timing and throughput: ```json "statistics": { "elapsed_secs": 0.123, "throughput_mb_per_sec": 456.78 } ``` *** ## analyze [Section titled “analyze”](#analyze) ```bash sql-splitter analyze dump.sql --json ``` ```json { "input_file": "dump.sql", "dialect": "mysql", "size_mb": 125.5, "elapsed_secs": 0.28, "summary": { "total_tables": 15, "total_inserts": 45000, "total_bytes": 131596288 }, "tables": [ { "name": "users", "inserts": 1000, "creates": 1, "statements": 2, "bytes": 125000, "size_mb": 0.12 } ] } ``` ### Multi-file mode [Section titled “Multi-file mode”](#multi-file-mode) ```bash sql-splitter analyze "*.sql" --json ``` ```json { "total_files": 3, "succeeded": 3, "failed": 0, "elapsed_secs": 1.25, "results": [ { "file": "a.sql", "size_mb": 125.5, "dialect": "mysql", "tables_count": 15, "total_inserts": 50000, "total_size_mb": 124.8, "tables": [ { "name": "users", "inserts": 1000, "creates": 1, "statements": 2, "bytes": 125000, "size_mb": 0.12 } ], "status": "success" } ] } ``` *** ## split [Section titled “split”](#split) ```bash sql-splitter split dump.sql -o tables/ --json ``` ```json { "input_file": "dump.sql", "output_dir": "tables/", "dialect": "mysql", "dialect_confidence": "high", "dry_run": false, "statistics": { "tables_found": 15, "statements_processed": 45, "bytes_processed": 131596288, "elapsed_secs": 0.21, "throughput_mb_per_sec": 625.5 }, "tables": ["users", "orders", "products"] } ``` A `compression` field (`"gzip"`, `"bzip2"`, `"xz"`, or `"zstd"`) is included only when the input file is compressed. *** ## merge [Section titled “merge”](#merge) ```bash sql-splitter merge tables/ -o merged.sql --json ``` ```json { "input_dir": "tables/", "output_file": "merged.sql", "dialect": "mysql", "dry_run": false, "statistics": { "tables_merged": 15, "bytes_written": 131596288, "elapsed_secs": 0.15, "throughput_kb_per_sec": 856.4 }, "tables": ["users", "orders", "products"], "options": { "transaction": false, "header": true } } ``` > **Note:** When output goes to stdout (no `-o`), JSON is written to stderr to avoid mixing with SQL output. *** ## validate [Section titled “validate”](#validate) ```bash sql-splitter validate dump.sql --json ``` ```json { "dialect": "mysql", "issues": [ { "code": "FK_MISSING_PARENT", "severity": "error", "message": "FK violation in 'orders': references missing row in 'users'", "location": { "table": "orders", "statement_index": 4 } } ], "summary": { "errors": 1, "warnings": 0, "info": 0, "tables_scanned": 15, "statements_scanned": 450 }, "checks": { "syntax": "ok", "encoding": "ok", "ddl_dml_consistency": "ok", "pk_duplicates": "ok", "fk_integrity": { "failed": 1 } } } ``` ### Check status values [Section titled “Check status values”](#check-status-values) * `"ok"` - Check passed * `{"failed": 3}` - Check failed with count * `{"skipped": "reason"}` - Check was skipped ### Multi-file mode [Section titled “Multi-file mode”](#multi-file-mode-1) ```bash sql-splitter validate "*.sql" --json ``` ```json { "total_files": 5, "passed": 4, "failed": 1, "elapsed_secs": 2.5, "results": [ { "file": "a.sql", "passed": true, "size_mb": 125.5, "summary": { "dialect": "mysql", "issues": [], "summary": { "errors": 0, "warnings": 0, "info": 0, "tables_scanned": 15, "statements_scanned": 450 }, "checks": { "syntax": "ok", "encoding": "ok", "ddl_dml_consistency": "ok", "pk_duplicates": "ok", "fk_integrity": "ok" } } } ] } ``` Each entry’s `summary` is the full single-file report (with its own nested `summary`). *** ## sample [Section titled “sample”](#sample) ```bash sql-splitter sample dump.sql --percent 10 -o dev.sql --json ``` ```json { "input_file": "dump.sql", "output_file": "dev.sql", "dialect": "mysql", "dry_run": false, "mode": { "type": "percent", "value": 10 }, "statistics": { "tables_sampled": 15, "tables_skipped": 0, "rows_selected": 10500, "rows_total": 100000, "sample_rate_percent": 10.5, "fk_orphans_rejected": 0 }, "tables": [ { "name": "users", "classification": "root", "rows_selected": 100, "rows_total": 1000, "sample_rate_percent": 10.0 } ] } ``` ### Classification values [Section titled “Classification values”](#classification-values) * `root` - No foreign keys referencing other tables * `lookup` - Referenced by many tables, small row count * `system` - System/metadata tables * `junction` - Many-to-many junction tables * `normal` - Standard table with foreign keys *** ## shard [Section titled “shard”](#shard) ```bash sql-splitter shard dump.sql --tenant-value 123 -o tenant.sql --json ``` ```json { "input_file": "dump.sql", "output_file": "tenant.sql", "dialect": "mysql", "dry_run": false, "tenant": { "column": "company_id", "value": "123", "auto_detected": false }, "statistics": { "tables_processed": 12, "tables_with_data": 10, "tables_skipped": 2, "rows_selected": 5000, "rows_total": 100000, "reduction_percent": 95.0, "fk_orphans_skipped": 0 }, "tables": [ { "name": "users", "classification": "tenantroot", "rows_selected": 50, "rows_total": 1000 } ] } ``` ### Classification values [Section titled “Classification values”](#classification-values-1) * `tenantroot` - Contains the tenant column directly * `tenantdependent` - Linked to tenant via foreign keys * `junction` - Many-to-many junction tables * `lookup` - Shared lookup tables (all rows included) * `system` - System/metadata tables * `unknown` - Could not determine relationship *** ## convert [Section titled “convert”](#convert) ```bash sql-splitter convert mysql.sql --to postgres -o pg.sql --json ``` ```json { "input_file": "mysql.sql", "output_file": "pg.sql", "conversion": { "from": "auto", "to": "postgres" }, "dry_run": false, "statistics": { "statements_processed": 450, "statements_converted": 45, "statements_unchanged": 400, "statements_skipped": 5 }, "warnings": [ { "type": "unsupported_feature", "feature": "ENUM type 'status'", "suggestion": "Use VARCHAR with CHECK constraint" }, { "type": "lossy_conversion", "from_type": "MEDIUMINT", "to_type": "INTEGER", "table": "users", "column": "age" } ] } ``` ### Warning types [Section titled “Warning types”](#warning-types) * `unsupported_feature` - Feature not available in target dialect * `lossy_conversion` - Data type conversion may lose precision * `skipped_statement` - Statement was skipped during conversion * `copy_not_converted` - PostgreSQL COPY statement needs manual conversion *** ## redact [Section titled “redact”](#redact) ```bash sql-splitter redact dump.sql --hash "*.email" -o safe.sql --json ``` ```json { "tables_processed": 15, "rows_redacted": 50000, "columns_redacted": 8, "table_stats": [ { "name": "users", "rows_processed": 1000, "columns_redacted": 3 } ], "warnings": [] } ``` *** ## graph [Section titled “graph”](#graph) ```bash sql-splitter graph dump.sql --json ``` ```json { "tables": [ { "name": "users", "columns": [ { "name": "id", "type": "INT", "is_primary_key": true, "is_foreign_key": false, "is_nullable": false }, { "name": "email", "type": "VARCHAR", "is_primary_key": false, "is_foreign_key": false, "is_nullable": true } ] } ], "relationships": [ { "from_table": "orders", "from_column": "user_id", "to_table": "users", "to_column": "id", "cardinality": "ManyToOne" } ], "stats": { "table_count": 2, "column_count": 5, "relationship_count": 1 } } ``` *** ## query [Section titled “query”](#query) The `query` command uses `--format` instead of `--json`: ```bash sql-splitter query dump.sql "SELECT * FROM users LIMIT 2" --format json ``` ```json [ { "id": 1, "name": "Alice", "email": "alice@example.com" }, { "id": 2, "name": "Bob", "email": "bob@example.com" } ] ``` ### JSON Lines format [Section titled “JSON Lines format”](#json-lines-format) ```bash sql-splitter query dump.sql "SELECT * FROM users" --format jsonl ``` ```plaintext {"id": 1, "name": "Alice", "email": "alice@example.com"} {"id": 2, "name": "Bob", "email": "bob@example.com"} ``` *** ## Parsing with jq [Section titled “Parsing with jq”](#parsing-with-jq) ```bash # Get table names from analyze sql-splitter analyze dump.sql --json | jq -r '.tables[].name' # Check if validation passed sql-splitter validate dump.sql --json | jq '.summary.errors == 0' # Get tables with most rows sql-splitter analyze dump.sql --json | jq '.tables | sort_by(-.inserts) | .[0:5]' # Filter failed validations in multi-file mode sql-splitter validate "*.sql" --json | jq '.results[] | select(.passed | not)' ``` ## Parsing in Code [Section titled “Parsing in Code”](#parsing-in-code) ### Python [Section titled “Python”](#python) ```python import subprocess import json result = subprocess.run( ["sql-splitter", "analyze", "dump.sql", "--json"], capture_output=True, text=True ) data = json.loads(result.stdout) print(f"Found {data['summary']['total_tables']} tables") ``` ### Node.js [Section titled “Node.js”](#nodejs) ```javascript import { execSync } from "child_process"; const output = execSync("sql-splitter analyze dump.sql --json"); const data = JSON.parse(output); console.log(`Found ${data.summary.total_tables} tables`); ``` ## Schema Validation [Section titled “Schema Validation”](#schema-validation) You can validate command output against schemas using any JSON Schema validator: ```bash # Using ajv (Node.js) bun add -g ajv-cli sql-splitter analyze dump.sql --json > output.json ajv validate -s schemas/analyze.schema.json -d output.json ``` ## See Also [Section titled “See Also”](#see-also) * [Exit Codes](/reference/exit-codes) - Exit code meanings * [Glob Patterns](/advanced/glob-patterns) - Multi-file processing * [AI Integration](/advanced/ai-integration/) - Machine-readable outputs for AI agents # Library Usage > Use sql-splitter as a Rust crate — split, parse, convert, redact, sample, and analyze SQL dumps programmatically sql-splitter is published on [crates.io](https://crates.io/crates/sql-splitter) as both a CLI and a library. Every subcommand is backed by a public module you can call from Rust. ```toml [dependencies] sql-splitter = { version = "1", default-features = false, features = ["compression"] } ``` | Feature | Default | Effect | | -------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `compression` | on | Read `.gz`, `.bz2`, `.xz`, `.zst` input. Without it, compressed files fail with an `Unsupported` I/O error; plain `.sql` still works. | | `duckdb-query` | on | The `duckdb` module (backing the `query` command). Pulls in bundled DuckDB — a large build dependency. Implies `compression`. | For library use, `default-features = false` with `compression` is the practical baseline; add `duckdb-query` only if you need it. ## Conventions [Section titled “Conventions”](#conventions) All high-level APIs follow the same shape: build a config (struct literal or builder), call a runner, get back a `Serialize`-able stats struct. Errors are `anyhow::Result` throughout; the low-level parser returns `std::io::Result`. Two things to know before starting: * **High-level runners work on file paths**, not readers. `Splitter`, `Analyzer`, `Redactor`, and `Validator` open files themselves. For in-memory or streaming input, drop down to [`Parser`](#stream-statements-with-parser) and [`Converter`](#convert-between-dialects), which work over any `Read` / byte slice. * **`output: None` means process stdout** for `convert::run`, `sample::run`, `Redactor::run`, and `Merger::merge` — same behavior as the CLI’s `-o -`. ## Detect the dialect [Section titled “Detect the dialect”](#detect-the-dialect) `SqlDialect` is the enum passed to every API. Detect it from a file header, or parse it from a string (`"mysql"`, `"postgres"`, `"sqlite"`, `"mssql"` and common aliases): ```rust use sql_splitter::parser::{detect_dialect_from_file, DialectConfidence, SqlDialect}; fn main() -> anyhow::Result<()> { let detection = detect_dialect_from_file("dump.sql".as_ref())?; if detection.confidence == DialectConfidence::Low { eprintln!("warning: low-confidence dialect guess: {}", detection.dialect); } let dialect: SqlDialect = "postgres".parse().unwrap(); println!("detected={} explicit={}", detection.dialect, dialect); Ok(()) } ``` Detection reads the first 8 KB and can guess wrong — see [Known Limitations](/reference/limitations/#dialect-auto-detection). Pass an explicit dialect when you know the source. ## Split a dump [Section titled “Split a dump”](#split-a-dump) `Splitter` writes one `.sql` file per table into an output directory and returns `Stats`: ```rust use sql_splitter::parser::SqlDialect; use sql_splitter::splitter::Splitter; fn main() -> anyhow::Result<()> { let stats = Splitter::new("dump.sql".into(), "tables/".into()) .with_dialect(SqlDialect::MySql) .with_table_filter(vec!["users".into(), "orders".into()]) // optional .with_progress(|bytes| eprintln!("{bytes} bytes read")) // optional .split()?; println!( "{} tables, {} statements, {} bytes", stats.tables_found, stats.statements_processed, stats.bytes_processed ); Ok(()) } ``` Compressed input is detected from the file extension. Use `.with_dry_run(true)` to collect stats without writing files, and `.with_content_filter(ContentFilter::SchemaOnly)` (or `DataOnly`) to mirror the CLI’s `--schema-only`/`--data-only`. ## Stream statements with Parser [Section titled “Stream statements with Parser”](#stream-statements-with-parser) `Parser` is the streaming engine everything else builds on. It works over any `Read`, yields one statement at a time as raw bytes, and never loads the whole dump into memory: ```rust use std::fs::File; use sql_splitter::parser::{determine_buffer_size, Parser, SqlDialect, StatementType}; fn main() -> anyhow::Result<()> { let dialect = SqlDialect::Postgres; let file = File::open("dump.sql")?; let buffer_size = determine_buffer_size(file.metadata()?.len()); let mut parser = Parser::with_dialect(file, buffer_size, dialect); let mut inserts = 0u64; while let Some(stmt) = parser.read_statement()? { let (stmt_type, table) = Parser::<&[u8]>::parse_statement_with_dialect(&stmt, dialect); if stmt_type == StatementType::Insert { inserts += 1; let _ = table; // table name the statement was attributed to } } println!("{inserts} INSERT statements"); Ok(()) } ``` `read_statement` handles quoting, escapes, PostgreSQL dollar-quoted bodies and `COPY` data blocks, and MSSQL `GO` batches. Statement types are `CreateTable`, `Insert`, `CreateIndex`, `AlterTable`, `DropTable`, `Copy`, or `Unknown` — see [Known Limitations](/reference/limitations/) for what falls into `Unknown`. ## Convert between dialects [Section titled “Convert between dialects”](#convert-between-dialects) Whole-file conversion mirrors the `convert` command: ```rust use sql_splitter::convert::{self, ConvertConfig}; use sql_splitter::parser::SqlDialect; fn main() -> anyhow::Result<()> { let stats = convert::run(ConvertConfig { input: "mysql.sql".into(), output: Some("postgres.sql".into()), // None writes to stdout from_dialect: None, // None auto-detects to_dialect: SqlDialect::Postgres, ..Default::default() })?; println!("{} converted, {} skipped", stats.statements_converted, stats.statements_skipped); for warning in &stats.warnings { eprintln!("warning: {warning:?}"); } Ok(()) } ``` For statement-level control (custom pipelines, non-file sources), use `Converter` directly. An `Ok` with an empty `Vec` means the statement was skipped for the target dialect: ```rust use sql_splitter::convert::Converter; use sql_splitter::parser::SqlDialect; fn main() { let mut converter = Converter::new(SqlDialect::MySql, SqlDialect::Postgres); match converter.convert_statement( b"CREATE TABLE `users` (`id` INT AUTO_INCREMENT PRIMARY KEY);", ) { Ok(converted) => println!("{}", String::from_utf8_lossy(&converted)), Err(warning) => eprintln!("unsupported: {warning:?}"), } } ``` Conversion is lossy for some constructs (`ENUM`, `UNSIGNED`, PostgreSQL-only DDL) — the details are in [Known Limitations](/reference/limitations/#conversion-convert) and [Type Mappings](/reference/type-mappings/). ## Redact sensitive data [Section titled “Redact sensitive data”](#redact-sensitive-data) `RedactConfig` uses a builder; patterns are `table.column` globs, identical to the CLI flags: ```rust use sql_splitter::redactor::{RedactConfig, Redactor}; fn main() -> anyhow::Result<()> { let config = RedactConfig::builder() .input("dump.sql".into()) .output(Some("safe.sql".into())) .hash_patterns(vec!["*.email".into()]) .fake_patterns(vec!["users.name".into()]) .null_patterns(vec!["*.ssn".into()]) .seed(Some(42)) // deterministic output .build()?; let stats = Redactor::new(config)?.run()?; println!("{} rows redacted across {} tables", stats.rows_redacted, stats.tables_processed); Ok(()) } ``` `Redactor::new` makes a first pass over the input to build the schema (so patterns can be validated against real columns); `run` makes the second pass and writes the output. See the [redact command](/commands/redact/) for the available strategies and [redact config](/reference/redact-config/) for the YAML format (`RedactYamlConfig::load` reads the same files). ## Sample a subset [Section titled “Sample a subset”](#sample-a-subset) ```rust use sql_splitter::parser::SqlDialect; use sql_splitter::sample::{self, SampleConfig, SampleMode}; fn main() -> anyhow::Result<()> { let stats = sample::run(SampleConfig { input: "prod.sql".into(), output: Some("dev.sql".into()), dialect: SqlDialect::MySql, mode: SampleMode::Percent(10), preserve_relations: true, // keep FK integrity seed: 42, ..Default::default() })?; println!( "{} of {} rows kept, {} orphans rejected", stats.total_rows_selected, stats.total_rows_seen, stats.fk_orphans_rejected ); Ok(()) } ``` `SampleConfig::default()` uses a random seed — set `seed` explicitly for reproducible samples. ## Analyze, validate, merge [Section titled “Analyze, validate, merge”](#analyze-validate-merge) `Analyzer` is the cleanest read-only entry point — it returns data and writes nothing: ```rust use sql_splitter::analyzer::Analyzer; use sql_splitter::parser::SqlDialect; fn main() -> anyhow::Result<()> { let tables = Analyzer::new("dump.sql".into()) .with_dialect(SqlDialect::MySql) .analyze()?; // sorted by insert count, descending for t in &tables { println!("{}: {} inserts, {} bytes", t.table_name, t.insert_count, t.total_bytes); } Ok(()) } ``` `Validator` runs the same checks as the `validate` command and returns a `ValidationSummary`. `ValidateOptions` has no `Default` — construct it fully (the CLI’s default row limit is 1,000,000): ```rust use sql_splitter::validate::{ValidateOptions, Validator}; fn main() -> anyhow::Result<()> { let summary = Validator::new(ValidateOptions { path: "dump.sql".into(), dialect: None, // auto-detect progress: false, strict: true, json: false, max_rows_per_table: 1_000_000, fk_checks_enabled: true, max_pk_fk_keys: None, }) .validate()?; if summary.has_errors() { for issue in &summary.issues { eprintln!("{}: {}", issue.code, issue.message); } anyhow::bail!("validation failed"); } Ok(()) } ``` `Merger` recombines a directory of per-table files (the inverse of `Splitter`): ```rust use sql_splitter::merger::Merger; fn main() -> anyhow::Result<()> { let stats = Merger::new("tables/".into(), Some("restored.sql".into())) .with_transaction(true) // wrap in BEGIN/COMMIT .merge()?; println!("{} tables merged, {} bytes", stats.tables_merged, stats.bytes_written); Ok(()) } ``` ## Caveats [Section titled “Caveats”](#caveats) * The `cmd` module contains the CLI layer — it takes parsed clap arguments and prints to the terminal. Don’t call it from library code; use the module APIs above. * All stats structs implement `serde::Serialize` (handy for JSON pipelines). * Progress callbacks take cumulative bytes read: `Fn(u64) + 'static` — except `Validator`, which requires `Fn(u64) + Send + Sync + 'static`. * `sample` and `redact` read uncompressed input only; `split`, `analyze`, and `convert` auto-detect compression by extension (with the `compression` feature). ## See also [Section titled “See also”](#see-also) * [Known Limitations](/reference/limitations/) — what the parser does not handle * [Type Mappings](/reference/type-mappings/) — conversion matrix between dialects * [JSON Output](/reference/json-output/) — the same stats structures, as emitted by the CLI # Known Limitations > What sql-splitter does not parse, per dialect — stored procedures, binary formats, detection edge cases, and lossy conversions sql-splitter is a streaming statement splitter, not a full SQL parser. It recognizes six statement types — `CREATE TABLE`, `INSERT`, `CREATE INDEX`, `ALTER TABLE`, `DROP TABLE`, and PostgreSQL `COPY` — and attributes each to a table. Everything else (views, triggers, procedures, grants, session commands) passes through as *unknown* and is **skipped by `split`**, since it cannot be attributed to a table. This page lists what does not work, so you can check your dump against it before relying on the output. ## All dialects [Section titled “All dialects”](#all-dialects) * **Statements without a table are dropped by `split`.** `CREATE VIEW`, `CREATE TRIGGER`, `CREATE PROCEDURE`/`FUNCTION`, `GRANT`/`REVOKE`, `TRUNCATE`, `SET`, and `USE` are not written to any output file. Run `validate` or compare statement counts if your dump relies on them. * **Schema qualifiers are discarded.** Table attribution keys on the bare table name: `sales.orders` and `hr.orders` both land in `orders.sql`. Multi-schema dumps merge colliding names into one file. * **A `;` inside a backtick-quoted identifier or inside a `/* */` block comment splits the statement early.** Quoted *strings* are handled correctly; backtick identifiers and mid-statement block comments are not. * **A leading UTF-8 BOM** can make the first statement unrecognized. Strip it first (`sed -i '1s/^\xEF\xBB\xBF//' dump.sql`). * **No character-set transcoding.** Input bytes pass through verbatim; latin1 dumps are not converted to UTF-8. `validate` emits an `ENCODING` warning for statements containing invalid UTF-8. * **Compressed input** (`.gz`, `.bz2`, `.xz`, `.zst`) requires the `compression` Cargo feature (on by default in released binaries). ## MySQL / MariaDB [Section titled “MySQL / MariaDB”](#mysql--mariadb) * **`DELIMITER` is not supported.** Dumps containing stored procedures, functions, or triggers wrapped in `DELIMITER $$ ... $$` are split at every internal `;`, producing broken fragments that `split` then drops. Dump with `--skip-triggers` and without `--routines`, or handle routine definitions separately. * Conditional comments (`/*!40101 SET ... */;`) are tolerated as leading comments; the wrapped statement is still detected. * Multi-database dumps using `USE db;` are not namespaced — same-named tables from different databases collide into one file. ## PostgreSQL [Section titled “PostgreSQL”](#postgresql) * **Plain-text format only.** `pg_dump -Fc` (custom), `-Fd` (directory), and `-Ft` (tar) are binary formats and will not parse. Use `pg_dump -Fp`, or convert an existing archive with `pg_restore -f dump.sql archive.dump`. * **`COPY` data must be text format**: tab-separated with `\N` for NULL (pg\_dump’s default). `COPY ... WITH (FORMAT csv)` and `WITH BINARY` are not supported. * Dollar-quoted bodies (`$$ ... $$`, `$tag$ ... $tag$`) are handled — function bodies with internal `;` split correctly. * `CREATE FUNCTION`/`TYPE`/`DOMAIN`/`SEQUENCE`/`EXTENSION`, `COMMENT ON`, `GRANT`, and `ALTER ... OWNER TO` carry no table and are dropped by `split` (and skipped with a warning by `convert`). ## SQLite [Section titled “SQLite”](#sqlite) * `PRAGMA`, `BEGIN TRANSACTION`, and `COMMIT` are dropped by `split` — re-add them when re-importing (`sql-splitter merge --transaction` restores a transaction wrapper). * `CREATE VIEW` and `CREATE TRIGGER` are dropped by `split`. ## MSSQL [Section titled “MSSQL”](#mssql) * **`GO` must be alone on its own line** (optionally `GO `). An inline `GO` or one followed by a comment does not separate batches. * Both `GO` *and* `;` terminate statements, so multi-statement batches (procedure bodies, `BEGIN ... END` blocks with internal semicolons) are split apart. * `CREATE PROCEDURE`/`FUNCTION`/`TRIGGER`/`VIEW`, `MERGE`, and `EXEC` are dropped by `split`. ## Dialect auto-detection [Section titled “Dialect auto-detection”](#dialect-auto-detection) Detection reads the **first 8 KB** of the file and scores dialect markers (`pg_dump` headers, backticks, `GO` lines, `IDENTITY(`, square brackets, …). The highest score wins; ties default to MySQL. It can guess wrong when: * Marker text appears inside **data or comments** in the first 8 KB — matching is substring-based, so a literal `pg_dump` or `SQLite` in a value counts. * A MySQL or PostgreSQL dump contains `[...]` or `N'...'` early in the file — MSSQL markers are weighted heavily. * A long license or comment banner pushes real markers past the first 8 KB. Detection confidence is advisory: commands proceed even on a low-confidence guess. **Pass `--dialect` explicitly whenever you know the source** — a wrong guess silently produces wrong splits (for example, unrecognized `COPY` blocks). ## Conversion (`convert`) [Section titled “Conversion (convert)”](#conversion-convert) Conversion is string-based rewriting, not AST transformation. It is deliberately lossy for constructs the target dialect lacks; every drop or rewrite is reported as a warning (deduplicated per feature, capped at 100). * **Skipped**: session commands (`SET NAMES`, `SET ANSI_NULLS`, `LOCK TABLES`, `PRAGMA`, …), and PostgreSQL-only DDL (`CREATE FUNCTION`/`TYPE`/`DOMAIN`/`SEQUENCE`/`EXTENSION`/`TRIGGER`, `COMMENT ON`, `GRANT`) — each with a warning. * **Rewritten lossily**: MySQL `ENUM(...)` and `SET(...)` become `VARCHAR(255)`/`TEXT` (constraint semantics lost); `UNSIGNED` and `ZEROFILL` are removed; inline `COMMENT` clauses are stripped for PostgreSQL targets. * **Warned but left as-is**: PostgreSQL array types (`integer[]`) and `INHERITS` — no equivalent is generated. * **Not detectable**: SQLite auto-increment (`INTEGER PRIMARY KEY`) passes through unchanged when converting *from* SQLite; MSSQL `TIMESTAMP` (rowversion) is only converted when written as `[TIMESTAMP]`. * `COPY` blocks are converted to multi-row `INSERT`s (1000 rows per statement) for non-PostgreSQL targets. Use `--strict` to fail on the first unsupported construct instead of collecting warnings. See [Type Mappings](/reference/type-mappings/) for the full type conversion matrix. ## Practical guidance [Section titled “Practical guidance”](#practical-guidance) * Dumps that are **schema + data only** (the common `mysqldump`/`pg_dump -Fp` case, no routines) split and convert reliably — this is the designed-for path. * Always spot-check with `sql-splitter validate "output/*.sql" --strict` after splitting or converting a dump that contains anything beyond tables, indexes, and rows. * If in doubt, run with `--dry-run` first and compare the reported table list against your expectations. # Redact Config > YAML configuration format for data redaction The redact command supports YAML configuration for complex redaction rules. ## Basic Structure [Section titled “Basic Structure”](#basic-structure) ```yaml # Global settings seed: 12345 locale: en # Default strategy for unmatched columns # (note the nested `strategy` key) defaults: strategy: strategy: skip # Column-specific rules rules: - column: "*.email" strategy: hash - column: "*.name" strategy: fake generator: name # Tables to skip entirely skip_tables: - schema_migrations - ar_internal_metadata ``` ## Global Settings [Section titled “Global Settings”](#global-settings) ### seed [Section titled “seed”](#seed) Random seed for reproducible fake data generation: ```yaml seed: 12345 ``` Same seed = same fake data on every run. ### locale [Section titled “locale”](#locale) Locale for fake data generation: ```yaml locale: en # English (default) locale: de_de # German locale: fr_fr # French locale: ja_jp # Japanese locale: zh_cn # Chinese ``` ## Rules [Section titled “Rules”](#rules) Each rule matches columns and applies a strategy. ### Column Patterns [Section titled “Column Patterns”](#column-patterns) ```yaml rules: # All columns named 'email' in any table - column: "*.email" # Specific table.column - column: "users.password" # Wildcard in column name - column: "*.ssn*" ``` For a schema-qualified table, put the complete table identity before the final column separator. For example, `tenant_a.users.email` targets the `email` column in `tenant_a.users`. Each rule takes a single glob pattern. To match multiple patterns, use one rule per pattern: ```yaml rules: - column: "*.phone" strategy: fake generator: phone - column: "*.mobile" strategy: fake generator: phone ``` ### Strategies [Section titled “Strategies”](#strategies) #### null [Section titled “null”](#null) Replace with NULL: ```yaml - column: "*.ssn" strategy: "null" ``` The quotes are required—unquoted `null` is a YAML null value and fails to parse. #### constant [Section titled “constant”](#constant) Replace with a fixed value: ```yaml - column: "*.status" strategy: constant value: "REDACTED" ``` #### hash [Section titled “hash”](#hash) SHA256 hash (deterministic): ```yaml - column: "*.email" strategy: hash ``` Same input always produces same hash, preserving FK relationships. Optionally keep the email domain (`user@domain.com` → `hash@domain.com`): ```yaml - column: "*.email" strategy: hash preserve_domain: true ``` #### fake [Section titled “fake”](#fake) Generate realistic fake data: ```yaml - column: "*.name" strategy: fake generator: name ``` Available generators: | Generator | Example Output | | ------------- | --------------------------------- | | `email` | `jessica.smith@example.com` | | `name` | `Robert Johnson` | | `first_name` | `Sarah` | | `last_name` | `Williams` | | `phone` | `+1 (555) 234-5678` | | `address` | `123 Oak Street, Springfield, IL` | | `city` | `Portland` | | `state` | `California` | | `zip` | `90210` | | `country` | `United States` | | `company` | `Acme Corporation` | | `job_title` | `Software Engineer` | | `username` | `jsmith42` | | `url` | `https://example.com/page` | | `ip` | `192.168.1.100` | | `ipv6` | `2001:db8::1` | | `uuid` | `550e8400-e29b-41d4-...` | | `date` | `1985-07-23` | | `datetime` | `2023-01-15 14:30:00` | | `credit_card` | `4532015112830366` | | `iban` | `DE89370400440532013000` | | `ssn` | `123-45-6789` | | `lorem` | `Lorem ipsum dolor sit...` | | `paragraph` | `Full paragraph of text...` | | `sentence` | `A complete sentence.` | #### mask [Section titled “mask”](#mask) Partial masking with pattern: ```yaml - column: "*.credit_card" strategy: mask pattern: "****-****-****-XXXX" ``` Pattern symbols: * `*` - Replace with asterisk * `X` - Keep original character * `#` - Random digit Examples: * `****-****-****-XXXX` → `****-****-****-1234` * `XXX-XX-####` → `123-45-6789` (SSN with random last 4) * `***@XXXXX` → `***@gmail.com` (mask email prefix) #### shuffle [Section titled “shuffle”](#shuffle) `shuffle` configuration is rejected. Correct shuffling requires a two-pass rewrite so the output never retains the original value at its source row. #### skip [Section titled “skip”](#skip) No redaction (passthrough): ```yaml - column: "admins.email" strategy: skip ``` ## skip\_tables [Section titled “skip\_tables”](#skip_tables) Skip entire tables from redaction: ```yaml skip_tables: - schema_migrations - ar_internal_metadata - _global ``` ## Complete Example [Section titled “Complete Example”](#complete-example) ```yaml seed: 42 locale: en defaults: strategy: strategy: skip rules: # PII - column: "*.email" strategy: hash - column: "*.name" strategy: fake generator: name - column: "*.first_name" strategy: fake generator: first_name - column: "*.last_name" strategy: fake generator: last_name - column: "*.phone" strategy: fake generator: phone - column: "*.mobile" strategy: fake generator: phone - column: "*.ssn" strategy: "null" - column: "*.password" strategy: constant value: "$2a$10$REDACTED" # Financial - column: "*.credit_card" strategy: mask pattern: "****-****-****-XXXX" # Addresses - column: "*.address" strategy: fake generator: address - column: "*.city" strategy: fake generator: city - column: "*.zip" strategy: fake generator: zip - column: "*.postal_code" strategy: fake generator: zip # Exceptions - column: "admins.email" strategy: skip skip_tables: - schema_migrations - ar_internal_metadata ``` # Type Mappings > Data type conversions between SQL dialects sql-splitter handles 50+ data type mappings during conversion. ## Numeric Types [Section titled “Numeric Types”](#numeric-types) | MySQL | PostgreSQL | SQLite | MSSQL | | -------------- | ------------------ | --------- | -------------- | | `TINYINT` | `SMALLINT` | `INTEGER` | `TINYINT` | | `SMALLINT` | `SMALLINT` | `INTEGER` | `SMALLINT` | | `MEDIUMINT` | `INTEGER` | `INTEGER` | `INT` | | `INT` | `INTEGER` | `INTEGER` | `INT` | | `BIGINT` | `BIGINT` | `INTEGER` | `BIGINT` | | `FLOAT` | `REAL` | `REAL` | `REAL` | | `DOUBLE` | `DOUBLE PRECISION` | `REAL` | `FLOAT` | | `DECIMAL(p,s)` | `DECIMAL(p,s)` | `REAL` | `DECIMAL(p,s)` | ## String Types [Section titled “String Types”](#string-types) | MySQL | PostgreSQL | SQLite | MSSQL | | ------------ | ------------ | ------ | --------------- | | `CHAR(n)` | `CHAR(n)` | `TEXT` | `CHAR(n)` | | `VARCHAR(n)` | `VARCHAR(n)` | `TEXT` | `VARCHAR(n)` | | `TEXT` | `TEXT` | `TEXT` | `TEXT` | | `MEDIUMTEXT` | `TEXT` | `TEXT` | `NVARCHAR(MAX)` | | `LONGTEXT` | `TEXT` | `TEXT` | `NVARCHAR(MAX)` | `CHAR`, `VARCHAR`, and `TEXT` are valid MSSQL types and pass through unchanged. ## Binary Types [Section titled “Binary Types”](#binary-types) | MySQL | PostgreSQL | SQLite | MSSQL | | -------------- | ---------- | ------ | ---------------- | | `BINARY(n)` | `BYTEA` | `BLOB` | `BINARY(n)` | | `VARBINARY(n)` | `BYTEA` | `BLOB` | `VARBINARY(n)` | | `BLOB` | `BYTEA` | `BLOB` | `VARBINARY(MAX)` | | `MEDIUMBLOB` | `BYTEA` | `BLOB` | `VARBINARY(MAX)` | | `LONGBLOB` | `BYTEA` | `BLOB` | `VARBINARY(MAX)` | ## Date/Time Types [Section titled “Date/Time Types”](#datetime-types) | MySQL | PostgreSQL | SQLite | MSSQL | | ----------- | ----------- | ------ | ----------- | | `DATE` | `DATE` | `TEXT` | `DATE` | | `TIME` | `TIME` | `TEXT` | `TIME` | | `DATETIME` | `TIMESTAMP` | `TEXT` | `DATETIME2` | | `TIMESTAMP` | `TIMESTAMP` | `TEXT` | `TIMESTAMP` | MySQL `YEAR` and `TIMESTAMP` (for MSSQL targets) are not rewritten and pass through unchanged. ## Boolean Types [Section titled “Boolean Types”](#boolean-types) | MySQL | PostgreSQL | SQLite | MSSQL | | ------------ | ---------- | --------- | ----- | | `TINYINT(1)` | `BOOLEAN` | `INTEGER` | `BIT` | The `BOOL`/`BOOLEAN` aliases are not rewritten—only the literal `TINYINT(1)` form is detected. ## Auto-Increment [Section titled “Auto-Increment”](#auto-increment) | MySQL | PostgreSQL | SQLite | MSSQL | | ----------------------- | ----------- | --------------------- | ---------------------- | | `INT AUTO_INCREMENT` | `SERIAL` | `INTEGER PRIMARY KEY` | `INT IDENTITY(1,1)` | | `BIGINT AUTO_INCREMENT` | `BIGSERIAL` | `INTEGER PRIMARY KEY` | `BIGINT IDENTITY(1,1)` | ## JSON Types [Section titled “JSON Types”](#json-types) | MySQL | PostgreSQL | SQLite | MSSQL | | ------ | ---------- | ------ | --------------- | | `JSON` | `JSONB` | `TEXT` | `NVARCHAR(MAX)` | ## Special Types [Section titled “Special Types”](#special-types) ### ENUM (MySQL) [Section titled “ENUM (MySQL)”](#enum-mysql) MySQL ENUMs are converted to: * **PostgreSQL**: `VARCHAR(255)` (a warning suggests adding a CHECK constraint manually) * **SQLite**: `TEXT` * **MSSQL**: `NVARCHAR(255)` ### UUID (PostgreSQL) [Section titled “UUID (PostgreSQL)”](#uuid-postgresql) PostgreSQL `UUID` is converted to: * **MySQL**: `VARCHAR(36)` * **SQLite**: `TEXT` * **MSSQL**: `UNIQUEIDENTIFIER` ### Arrays (PostgreSQL) [Section titled “Arrays (PostgreSQL)”](#arrays-postgresql) PostgreSQL array types (e.g. `integer[]`) generate a warning but are **not converted**—they pass through unchanged. The warning suggests using JSON in the target dialect. ## Unsupported Features [Section titled “Unsupported Features”](#unsupported-features) These features generate warnings during conversion: * Custom types (`CREATE TYPE`) - skipped when converting from PostgreSQL * Triggers (`CREATE TRIGGER`) - skipped when converting from PostgreSQL * Stored procedures and functions (`CREATE PROCEDURE`/`CREATE FUNCTION`) - skipped when converting from PostgreSQL * Table inheritance (`INHERITS`) * Partitioning (`PARTITION BY`, when converting PostgreSQL to SQLite) * MySQL `UNSIGNED`/`ZEROFILL` modifiers (removed) See [Known Limitations](/reference/limitations/) for the full list of skipped and lossily rewritten constructs. # Roadmap > sql-splitter development roadmap and planned features **Current Version**: 1.21.0 ## Completed Features [Section titled “Completed Features”](#completed-features) All core features have been released: | Version | Feature | Status | | ------- | -------------------------------------------------------------------------------------- | -------- | | v1.4.0 | Test Data Generator + Merge (fixture generator superseded by `generate`, see below) | Released | | v1.5.0 | Sample (FK-aware sampling) | Released | | v1.6.0 | Shard (tenant extraction) | Released | | v1.7.0 | Convert (dialect conversion) | Released | | v1.8.x | Validate + Glob Patterns | Released | | v1.9.x | Diff (schema + data comparison) | Released | | v1.10.0 | Redact (data anonymization) | Released | | v1.11.0 | Graph + Order (ERD + FK ordering) | Released | | v1.12.0 | Query (DuckDB analytics) | Released | | v1.12.x | MSSQL Support (fourth dialect) | Released | | v1.13.0 | JSON Schema Generation | Released | | v1.13.x | Library feature flags, Docker images, convert fixes | Released | | v1.14.0 | `-o -` stdout support, redact compression fix | Released | | v1.15.0 | Output compression (`--compress`), archive output (tar.\*/zip), 2.4–4.4× split speedup | Released | | v1.16.0 | Zip input (all commands), adaptive `--io-strategy` for slow devices | Released | | v1.17.0 | Synthetic data generation (`generate`) | Released | ## Released in v1.17.0 [Section titled “Released in v1.17.0”](#released-in-v1170) ### `generate` — Synthetic Data Generation [Section titled “generate — Synthetic Data Generation”](#generate--synthetic-data-generation) Generate production-shaped synthetic SQL from a dump, schema, or reusable YAML model. This supersedes the old `test_data_gen` fixture crate from v1.4.0 (removed) as the product feature for synthetic data: ```bash sql-splitter generate production.sql -o synthetic.sql sql-splitter generate production.sql --emit-config synthetic.yaml sql-splitter generate --config synthetic.yaml --verify -o synthetic.sql ``` **Features:** * Bounded basic/full profiling and exact emitted row counts * Editable per-table and per-column YAML rules, with generators, modifiers, and planners for relational and multi-column invariants (see [`generate`](/commands/generate/)) * Stable global, table, column, and operator seed streams * Public library API using the same compiler and engine as the CLI * Neutral profile evidence reusable by the future `infer` command (see below) — profiler internals are not frozen around `infer`’s unbuilt needs Full documentation: [`generate` command reference](/commands/generate/), [model reference](/commands/generate/model-reference/), [generator catalog](/commands/generate/generators/), [planner catalog](/commands/generate/planners/), and [library API](/commands/generate/library-api/). ### v1.16.0 — Zip Input + Adaptive I/O Profiles (shipped) [Section titled “v1.16.0 — Zip Input + Adaptive I/O Profiles (shipped)”](#v1160--zip-input--adaptive-io-profiles-shipped) `.zip` dumps are now accepted as input across all commands — real-world exports often arrive as `dump.sql.zip`: ```bash sql-splitter split reflow_latest.sql.zip -o tables/ ``` **Zip input:** * Streams the zip’s `.sql` member through the existing decompression pipeline (no full extraction) * Exactly-one-`.sql`-member policy, with clear errors for multi-member or encrypted archives * No new dependencies — reuses the `zip` crate already powering archive output **Adaptive I/O strategys** — measured on a USB spinning disk, the default (SSD-tuned) write path runs at \~22–33 MB/s while a single writer with large buffers reaches \~55 MB/s (2.5×, near the drive’s physical ceiling): ```bash sql-splitter split dump.sql -o tables/ --io-strategy auto # default sql-splitter split dump.sql -o tables/ --io-strategy hdd # pin it ``` * An fsync probe picks the opening profile; after that, a feedback loop driven by the pipeline’s own backpressure steps between fast/HDD/slow-flash profiles — no device fingerprinting, works for network mounts and cheap USB sticks too * Output stays byte-identical across all profiles (verified by cross-profile hash tests) ## Upcoming Features [Section titled “Upcoming Features”](#upcoming-features) ### v1.18.0 — Enum Type Conversion [Section titled “v1.18.0 — Enum Type Conversion”](#v1180--enum-type-conversion) Proper bidirectional enum conversion between PostgreSQL and MySQL. Today, `ENUM('a','b')` converts lossily to `VARCHAR(255)` — this replaces that with real type conversion: **Features:** * PostgreSQL `CREATE TYPE ... AS ENUM` → MySQL inline `ENUM()` * MySQL inline `ENUM()` → PostgreSQL `CREATE TYPE` (deterministic naming) * Registry-based type tracking across statements (streaming-safe) * Strip `::type` casts in DML statements * `VARCHAR` fallback with a warning for unknown types ### v1.19.0 — Migration Generation [Section titled “v1.19.0 — Migration Generation”](#v1190--migration-generation) Generate ALTER statements from schema differences: ```bash sql-splitter migrate old.sql new.sql -o migration.sql sql-splitter migrate old.sql new.sql --rollback -o rollback.sql sql-splitter migrate old.sql new.sql --breaking-changes ``` **Features:** * Generate migration scripts (ALTER TABLE, CREATE INDEX) * Multi-dialect migration output * Rollback script generation * Breaking change detection ### v1.20.0 — DBML Import/Export [Section titled “v1.20.0 — DBML Import/Export”](#v1200--dbml-importexport) Schema documentation and interoperability with [DBML](https://dbml.dbdiagram.io/): **Features:** * Export SQL dump schemas to DBML for dbdiagram.io visualization (extends `graph`) * Generate SQL DDL from DBML schema-as-code definitions (extends `convert`) ### v2.0.0 — Parallel Processing [Section titled “v2.0.0 — Parallel Processing”](#v200--parallel-processing) Multi-threaded performance: ```bash sql-splitter split dump.sql -o tables/ --parallel 8 sql-splitter convert dump.sql --parallel auto ``` **Targets:** * 4x speedup on 8-core systems * Linear scaling up to available cores * Memory-bounded parallel processing ### v2.1.0 — Schema Inference [Section titled “v2.1.0 — Schema Inference”](#v210--schema-inference) Generate DDL from data-only dumps: ```bash sql-splitter infer data-only.sql -o schema.sql sql-splitter infer data.csv --table users --dialect mysql ``` **Features:** * Type inference from INSERT values * Primary key detection * Index suggestion * FK inference (heuristic) * Reuses or extends `generate`’s bounded neutral profile evidence ## Non-Goals [Section titled “Non-Goals”](#non-goals) These features are explicitly out of scope: * **GUI interface** — CLI only * **Database connection** — File-based only * **Binary backup formats** — No .bak (MSSQL) * **Stored procedure conversion** — Too complex * **Real-time streaming** — Batch processing only * **Cloud storage integration** — Use pipes ## Contributing [Section titled “Contributing”](#contributing) Want to contribute to a roadmap feature? See [Contributing](/contributing/). For feature requests, open an issue on [GitHub](https://github.com/helgesverre/sql-splitter/issues).