Airbyte, creators of the fastest-growing open-source data integration platform, today launched three new features – checkpointing, column selection, and schema propagation now available on both Airbyte Cloud and Airbyte Open Source – improving reliability, optimizing efficiency and resource usage while giving users more flexibility and enhancing privacy compliance.
“We’re getting closer to making Airbyte best-in-class with this significant release while furthering our mission to commoditize data integration,” said Michel Tricot, co-founder and CEO, Airbyte. “At the same time, we’re reducing resource requirements and giving users of both Airbyte Open Source and Airbyte Cloud more flexibility in how they comply with their internal or regulatory requirements. And, there are even more updates coming in the weeks ahead.”
Airbyte's adoption of checkpointing – a data processing technique that enables recovery from interruptions – now gives users improved levels of data reliability. By creating a saved state of a data stream at regular intervals, checkpointing allows the system to resume from the most recent checkpoint in the event of a failure, avoiding data loss or redundant processing – essential for large-scale data integration tasks.
The column selection feature enables users to select specific columns from a source for replication, rather than being required to integrate the entire object or table. With this added flexibility, users can now focus on the data most relevant to their needs, optimizing resource usage and efficiency. Users can also protect their data by excluding sensitive data, and more easily comply with data privacy regulations. Here is a video that demonstrates how column selection works.
The schema propagation feature automatically adopts changes from the source schema to the destination schema, saving users from manually making these changes. As data schemas change frequently in real-world scenarios, this capability enhances usability and reduces administrative overhead. Users are able to control how schema propagation is done for each connection. This enhances usability and reliability of the Airbyte platform.
Airbyte makes moving data easy and affordable across almost any source and destination, helping enterprises provide their users with access to the right data for analysis and decision-making. Airbyte has the largest data engineering contributor community – with more than 800 contributors – and the best tooling to build and maintain connectors.
About Airbyte
Airbyte is the open-source data integration leader running in the safety of your cloud and syncing data from applications, APIs, and databases to data warehouses, lakes, and other destinations. Airbyte was co-founded by Michel Tricot (former director of engineering and head of integrations at Liveramp and RideOS) and John Lafleur (serial entrepreneur of dev tools and B2B). The company is headquartered in San Francisco with a distributed team around the world. To learn more, visit airbyte.com.