Laravel Data Sync

An external system drops a file on your server: a CRM export, a supplier price list, a bank statement. It has to end up in your tables. Laravel Data Sync turns that into a class that names the source, the format, the model, the field mapping, and the columns to match on.

The package handles discovery, reading, validation, queued writes, and the audit trail. A process-once ledger means the same file never lands twice. Your application keeps ownership of its models, its business rules, and whatever it does with the data afterwards.

Read the full documentation for definitions, readers, write policies, queued execution, transfers, retries, retention, and every Artisan command.

Installation

Install the package, publish its configuration and migration, and migrate:

composer require byrcsc/laravel-data-sync
php artisan vendor:publish --tag="data-sync-config"
php artisan vendor:publish --tag="data-sync-migrations"
php artisan migrate

Publish the configuration before the migration when you want custom table names; the migration reads them from data-sync.tables.

Queued syncs fan chunks out through Bus::batch(), so the framework's job_batches table must exist. FTP and SFTP sources need a Flysystem adapter (league/flysystem-ftp or league/flysystem-sftp-v3), neither of which is installed by default.

Run php artisan sync:doctor after installing. It checks tables, disks, source reachability, cache locks, and matchOn indexes before your first real feed arrives. See the installation guide for disks, connections, and queue routing.

Quick start

Generate a definition:

php artisan make:sync ProductsSync

Fill in the source, the format, the model, the fields, and the match columns:

namespace App\Syncs;

use App\Models\Product;
use ByRcsc\LaravelDataSync\Contracts\Reader as ReaderContract;
use ByRcsc\LaravelDataSync\Definitions\Field;
use ByRcsc\LaravelDataSync\Definitions\Reader;
use ByRcsc\LaravelDataSync\Definitions\Source;
use ByRcsc\LaravelDataSync\Definitions\SyncDefinition;

final class ProductsSync extends SyncDefinition
{
    public function source(): Source
    {
        return Source::disk('incoming')->path('products');
    }

    public function format(): ReaderContract
    {
        return Reader::csv();
    }

    public function model(): string
    {
        return Product::class;
    }

    public function fields(): array
    {
        return [
            Field::make('sku', from: 'SKU'),
            Field::make('name', from: 'Description'),
            Field::make('price_cents', from: 'Price')
                ->transform(fn (string $value): int => (int) round((float) $value * 100)),
        ];
    }

    public function matchOn(): array
    {
        return ['sku'];
    }

    public function rules(): array
    {
        return [
            'sku' => ['required', 'string', 'max:32'],
            'price_cents' => ['required', 'integer', 'min:0'],
        ];
    }
}

Register it in config/data-sync.php and run it:

'syncs' => [
    App\Syncs\ProductsSync::class,
],

php artisan sync:run products          # queued
php artisan sync:run products --now    # in this process
php artisan sync:status

The name is derived from the class: ProductsSync becomes products.

What is included

• Class-based sync definitions with a declarative field mapping, per-row validation rules, and configurable write, duplicate, and unknown-column policies.
• Streaming readers for CSV and other delimited text, XLSX, JSON, NDJSON, and fixed-width files.
• Queued processing that fans a file's chunks out through Bus::batch(), plus an in-process mode for small feeds and an atomic() whole-file mode.
• A SHA-256 ledger that makes reprocessing a no-op, checksum-verified archives for completed files, and preserved copies for failed ones.
• Run history with counters, batch progress, failed-row records, and an error summary, surfaced through sync:status.
• sync:retry for a whole finished run, or --failed-rows to replay only the captured failed-row payloads.
• File transfers with filename-pattern metadata, templated destination paths, checksum verification, and optional model linking.
• Schedule registration, retention pruning, and a sync:doctor health check for the whole installation.

Important behavior

• File identity is the SHA-256 checksum of the contents. Renaming a file does not make it new; changing one byte does.
• Only a successful run blocks reprocessing. A failed checksum is retried on the next run without --force.
• Chunks commit independently. Only atomic() gives a file all-or-nothing semantics, at the cost of parallelism.
• Model events fire only on the Eloquent write path, which a definition opts into by declaring beforeSave() or afterSave().
• Package events are dispatched synchronously and never implement ShouldQueue. Queue your listeners.
• Multi-server deployments need a shared staging disk. A local staging disk is a single-machine configuration.
• --dry-run skips prepare(), so its counts are a preview and not a guarantee.
• Reader::json() reads a whole document into memory. Use NDJSON for feeds that need to stream.
• The sync_files ledger is never pruned; runs, failed rows, and archived files are. sync:prune does not sweep the staging disk, so successful runs leave their staged copy behind.
• Rows that fail are recorded and the run continues. maxErrors() is the only thing that stops a bad feed part way.

Out of scope

The package draws its edges deliberately. What follows describes what it sets out to do rather than what it might do later: treat none of it as planned work, and none of it as ruled out forever.

• Model pruning for absent rows. The package never deletes application records that stop appearing in a feed. It is written for incremental and delta feeds rather than full snapshots.
• PDF and OCR extraction. Transfers move PDFs; nothing reads inside them.
• AI-assisted extraction. Mapping is declarative and deterministic.
• Two-way sync. Data flows from files into models, never back out.
• API connectors. Sources are filesystems. An API feed has to be written to a disk first.
• Merge semantics for captured JSON. UnknownColumnPolicy::Capture overwrites the target attribute; it does not merge with what is already there.
• Retry from a date. sync:retry takes a run, not a time range.
• Glob and pattern discovery. A source names either a directory, including files in its subdirectories, or a single file. There is no filtering by name or extension.

Documentation

• Installation and setup
• Quick start
• Matching and writing
• Running a sync
• File transfers
• Failures and retries
• Configuration
• Troubleshooting

Development

The local checks mirror CI:

composer install
composer test
composer analyse
vendor/bin/pint --test

PHPStan runs at level max with no baseline. Tests use SQLite locally and run against MySQL and PostgreSQL in CI.

workbench/ is a bootable demo application that installs the package the way a real application would. composer build sets it up; see workbench/README.md for the demo loop.

Versioning

The package follows semantic versioning.

• Upgrading within 1.x is safe. Nothing you use will break.
• Only a new major version, like 2.0.0, can break your code.
• If the README or the documentation describes it, it is safe to build on. If they don't, treat it as internal and expect it to change.

Bug fixes go into the newest version only. To get a fix, upgrade to it.

Questions and issues

• Stuck, or have an idea? Start a discussion. Usage questions and feature ideas both live there.
• Found a bug you can reproduce? Open an issue. A failing test is the fastest way to a fix, and a short reproduction is the next best thing.
• Found a security problem? Please don't open a public issue. See SECURITY.md for how to report it privately.
• Planning a pull request? CONTRIBUTING.md covers the setup and the three checks it needs to pass.

This package is maintained by one person, so replies can take a while. Everything gets read.

Credits

• Ryan Catapang
• All contributors

License

MIT. See LICENSE.md. Changelog in CHANGELOG.md.