Now, did you know that you can replicate tables in Iceberg format using GG for DAA from version 23.7?
In a nutshell, Apache Iceberg is an open source table format designed for large-scale analysis in data lakes. In other words, the Iceberg format is a high-performance table format for extremely large analytical tables, designed to provide scalable and efficient data management.
Iceberg brings the reliability and simplicity of SQL tables to GG for DAA, while enabling engines such as Spark, Trino, Flink, Presto, Hive and Impala to work securely with the same tables at the same time.
And how can I do that? By using GG for DAA Handlers.
GG for DAA Handlers are native source and destination connectors for message streaming data/delta lake, cloud warehouse and NoSQL database technologies. They provide low-impact capture and real-time data ingestion capabilities with high accuracy and data throughput.
The OGG for DAA can be configured to work with any of the formats supported by Iceberg:
- Parquet
- Avro
- ORC
- Hadoop catalog
- Nessie Catalog
- AWS Glue Catalog
- Polaris Catalog
- REST Catalog
- JDBC Catalog
- INSERT: Generates files for insert operations.
- UPDATE: Generates data files and delete files for update operations.
- DELETE: Generates delete files for delete operations.
- TRUNCATE: Generates a delete file with a condition of always true to truncate the target table.
Oracle GoldenGate Iceberg Replicat can also replicate GoldenGate trail records to Iceberg tables. The files can be written to local files, AWS S3, Google Cloud Storage (GCS) or Azure DataLake Storage (ADLS).
Iceberg supports two types of delete files:
- Exclusions by equality: The excluded records are identified by the equality of the values in the columns specified in the exclusion file.
- Exclusions by position: The excluded records are identified by the position of the records in the Iceberg data file.
- Nessie Catalog
- AWS Glue Catalog
- Polaris Catalog
- REST Catalog
- JDBC Catalog
- Hadoop Catalog
And if you want to know more details, you can check it out here and here.
See you.