Understanding Change Data Capture: A Comprehensive Overview
Written on
Chapter 1: The Fundamentals of Change Data Capture
In my professional journey within Big Data analysis and Data Engineering, I have encountered a variety of projects. Despite their differences, they consistently adhere to a unified framework: the aim is to develop a data platform that aggregates information from diverse sources, processes it, and presents the refined data for end-users.
This framework is often encapsulated in the concepts of Data Lake/Data Lakehouse and ETL (Extract-Transform-Load) workflows. The methods for extracting data from source systems generally fall into two categories: batch, where the entire dataset is extracted in one go, and streaming, which involves continuous monitoring of the source for changes and extracting data as modifications occur.
While new technologies and architectures emerge annually, one methodology that remains prevalent is Change Data Capture (CDC).
Section 1.1: What is Change Data Capture (CDC)? 🤓
Change Data Capture is a design strategy that allows the tracking of changes in a data source. It provides a continuous stream of data modifications, which can serve various purposes, including:
- Data Lake/Data Lakehouse: Feeding a datalake with incremental updates.
- Real-time Analytics: Facilitating immediate analysis of data alterations.
- Event-driven Applications: Activating actions based on data modifications.
- Data Replication: Synchronizing multiple data copies.
Section 1.2: How Does CDC Function? 🧐
There are multiple ways to implement this pattern, but contemporary methods typically combine two key concepts:
- Transaction Log: Databases maintain a log of all operations performed on the data.
- Pub/Sub Queues: The CDC system periodically checks the data source for changes (new entries in the transaction log) and publishes these updates in a queue.
This approach employs various components and is ideal for scenarios demanding real-time processing and decoupled architecture. An example of this implementation can be found in my earlier posts, where I explained the procedure using Databricks.
The architecture described includes the following elements:
- Transactional DB: A SQL database capable of publishing real-time changes.
- Kafka: An open-source distributed streaming platform.
- Kafka Storage Sink Connector: This connector regularly polls data from Kafka and uploads it to cloud storage as Parquet files.
- Cloud Storage: The repository for all Parquet files.
- Data Lakehouse: A data lakehouse designed with a multi-hop structure.
The process is straightforward: the SQL database conveys each change as a message on Kafka, transmitting the entire updated row and indicating whether the operation is an insert, update, or delete. The Kafka connector retrieves the data under specific conditions, such as reaching a set number of queued messages or a predefined time limit.
Consequently, in the Data Lakehouse, messages are stored in an append-only log format, ensuring data integrity and order, ultimately reconstructing the source table by sequentially applying the messages.
Another method, instead of using Pub/Sub queues, involves replication tools (e.g., Oracle Golden Gate) that read from the transaction log and immediately execute operations on a target database. While this design is more straightforward, it may lack the scalability of the Pub/Sub queue approach.
Chapter 2: Advantages of Change Data Capture 💪
Despite its architectural intricacies, the CDC model offers numerous advantages:
- Reduced Latency: CDC enables near-real-time updates, minimizing pipeline latency.
- Enhanced Scalability: CDC can be scaled to accommodate large volumes of data changes.
- Decreased Workload on Source Systems: Unlike full data extractions, CDC captures only changes, reducing the performance impact on the source system.
Conclusion
Change Data Capture is a robust methodology for tracking changes within a data source. It serves various functions, including data replication, feeding datalakes, real-time analytics, and supporting event-driven applications.
In today's big data and AI landscape, CDC equips you to navigate the evolving data environment effectively.
Thank you for reading! If you found this information valuable, please consider clapping and following me! 😉
Here are some additional resources that may be beneficial:
This video titled "Change Data Capture (CDC) Explained (with examples)" provides a thorough explanation of CDC concepts, illustrating its practical applications.
In this video, "What Is Change Data Capture - Understanding Data Engineering 101," you will gain insights into the fundamentals of CDC and its relevance in data engineering.