Decodable, a real-time data engineering company announced today a faster, simpler, and less costly method of ingesting streaming data into Databricks. The announcement was made today at the Data + AI Summit in San Francisco and was made possible by the inclusion of a new connection for Decodable SaaS users.
“Ingesting streaming data into Databricks unlocks a host of powerful analytics capabilities. Unfortunately, for many developers of real-time applications, the present process to make this happen is overly complicated and cost prohibitive. We built the Delta Lake connector to solve those problems and bring the power of Databricks to a whole new set of use cases that presently are blocked due to the cost and complexity of ingestion.”
Eric Sammer, CEO and founder of Decodable
Delta Lake is an open source storage system for working with data from Amazon S3, Google Cloud, Azure Data Lake, Hadoop, and a variety of other data lakes.
Any Decodable user who wishes to utilize Databricks with data from other systems can use the new Delta Lake connection. It allows for data intake into Databricks at the Bronze and Silver levels of the Databricks medallion data layer architecture. Using the Decodable service in conjunction with the new connector is the easiest way to get data into Databricks so that application developers and data engineers can leverage Databricks' AI-driven power while doing so through a data ingestion pipeline that is less expensive, less complex, and simpler than current batch ingestion.
*How to Begin with Decodable and the Delta Lake Connector*
Interested application developers may get started with a free Decodable Developer account, which includes the Databricks Delta Lake connection, to create their first Decodable pipeline.
Decodable is a real-time data engineering service that helps developers transport data between data stores, analytic databases, and messaging systems. Decodable is a completely managed service with no infrastructure to maintain and no batching to configure or monitor. This lets consumers concentrate on the data and the application rather than the infrastructure.
Decodable gives Databricks users get straightforward, dependable data input from any source on any cloud. Decodable's on-the-wire processing employs SQL to alter and optimize data records before they arrive at Databricks, lowering storage and processing costs while boosting data quality in the Medallion Bronze and Silver tiers.
Decodable users can quickly design SQL pipelines to conduct basic or compound transformations such as filtering, masking, routing, aggregating, triggering, and more complicated windowing tasks using simple abstractions. Utilizing Decodable to prepare data for Databricks reduces data quantities dramatically, cutting computing and storage costs at the destination.