Analyze Data Faster with an Open Source Columnar Database

Fanatics, a popular sports apparel website and fan gear merchandiser, needed to ingest terabytes of data from multiple historical and streaming sources transactional, e-commerce, and back-office systems to a data lake on Amazon S3. Once ingested, the data would be analyzed to better identify, predict, and fulfill customer needs related to the products Fanatics offers in over 300 online and offline stores.

Few things bring down the value of data faster than confusion and uncertainty about what it is, where it came from, and whether it is good quality data. Yet as more users seek to access and interact with data and reports for business intelligence and analytics and as data sources become larger and more varied, confusion and uncertainty spread fast. Executives, managers, regulatory administrators, and other key personnel cannot rely on their reports, KPIs, and dashboards. Users cannot even find reports that the organization is producing. Instead, users spend more of their time trying to locate data and reports and correcting mistakes than they do applying data insights to solve business problems.

Building performant ETL pipelines to address analytics requirements is hard as data volumes and variety grow at an explosive pace. With existing technologies, data engineers are challenged to deliver data pipelines to support the real-time insight business owners demand from their analytics. Databricks Delta is the next generation of evolution in big data processing from Databricks, the company founded by the original creators of Apache Spark.

Cloudera DataFlow (CDF) is a scalable, real-time streaming data platform that collects, curates, and analyzes data so customers gain key insights for immediate actionable intelligence. It meets the challenges faced with data-in-motion, such as real-time stream processing, data provenance, and data ingestion from IoT devices and