CDC (Change Data Capture)

Change Data Capture (CDC) is the technique of detecting the inserts, updates and deletes made to a database (or any data source) and turning them into an ordered stream of change events that other systems can consume - in real time or near real time. Instead of copying a whole table again, CDC copies only what changed.

That makes CDC the backbone of data integration: keeping a warehouse, cache, search index or microservice in sync with the system of record, without a nightly full reload.

Why CDC instead of a full reload

A traditional batch ETL job reads the entire table. On a large table that means re-reading millions of unchanged rows, loading the source database, and moving the same bytes every night. CDC reads only the changed rows, so it:

• puts far less load (IO) on the source database;
• moves much less data over the network, and stores less downstream;
• delivers changes in seconds rather than hours;
• keeps a history of changes for auditing or replay.

The three kinds of change

• Insert - a new row appears.
• Update - an existing row is modified (ideally with before/after values).
• Delete - a row is removed.

A correct CDC stream captures all three and preserves their order, so a later delete cannot be undone by an earlier update arriving late.

How CDC is implemented

There are four common approaches. Most production systems use log-based capture, often combined with an initial snapshot.

Mode	How it works	Pros	Cons / when to avoid
Log-based	Parse the database transaction log: MySQL binlog, PostgreSQL WAL, Oracle redo, SQL Server CDC.	Millisecond latency, low impact on the source, captures inserts/updates/deletes and ordering.	Needs log access or extra permissions; format is database-specific.
Trigger-based	Database triggers write each change into a change/audit table.	Works on databases without usable logs; change table is easy to read.	Adds overhead to every write; triggers must be maintained per table.
Timestamp / query-based	Poll for rows whose updated_at is newer than the last run.	Simple, no database internals required.	Misses hard deletes; polling adds repeated load; needs a reliable timestamp column.
Snapshot diff	Compare two full snapshots (e.g. last night vs tonight) and keep the differences.	No source changes needed; detects deletes.	Expensive on large tables; latency equals the snapshot interval.

Common use cases

• Incremental loading into a data warehouse or data mart, keeping cross-service data consistent.
• Real-time refresh of caches and search indexes such as Redis and Elasticsearch.
• Database replication, disaster recovery and migration with near-zero downtime.
• Event-driven features - fraud/risk checks, recommendations, notifications - that must react to a change as it happens.

From changes to a pipeline

Capture is only half the job: the changes still have to be ordered, delivered and applied. That architecture - source, broker, sink, ordering, exactly-once and idempotency - is described in CDC pipeline.

Working with the changed data

Once changes are written to files or tables, Withdata's tools move them between formats and systems: DB to File (export tables to CSV, JSON, Excel, XML), File to DB (load files back into a database) and DataFileConverter. The free online converter handles quick CSV, JSON and SQL conversions in the browser.

FAQ

What is Change Data Capture (CDC)?
It detects inserts, updates and deletes in a database and turns them into an ordered stream of change events, so other systems stay in sync in real time without reloading the whole table.

Which CDC method should I use?
Log-based capture when the database offers it (best latency and completeness); trigger-based when it does not; timestamp polling only for simple, delete-free cases.

Does CDC handle deletes?
Log-based CDC does. Timestamp/query-based capture needs soft deletes or a separate delete log, because the deleted row no longer exists to poll.