Big Data Innovation – From Experiment to Production

Do you have your Enterprise’s data privacy under control? Do you know what datasets are sensitive and who has access? Many companies are behind on privacy and regulatory compliance (GDPR, CCPA, etc.). Legacy tools and manual processes are inaccurate and error prone and can force you to choose between delaying data access by months, or increased compliance risk.

In this webinar we will discuss a more modern view of the data lake and consider best practices for planning and implementing a scalable enterprise data lake. The flaws in early data lakes were often rooted in the expectations of data consumers who put a premium on self-service data analytics. However, with no data governance mechanisms, data lakes quickly became more of a glorified “dumping ground,” “data swamp,” or “beta lake” for organizational data.In recent years, though, some innovations have allowed the data lake to evolve into an agile yet managed environment for accumulating shared data resources that can be optimally used for competitive advantage. Data lakes have evolved beyond the original on-premises concept based solely on Hadoop and now include pretty much any distributed computing platform (Hadoop, Spark, EMR, serverless, etc.) and any storage mechanism (HDFS, S3, ADLS), either on-premises or in the cloud.

As part of its platform, GoodData provides a fault-tolerant, high performance and scalable system for data integration. While built for large-scale analytic applications, it is a metadata-driven, modular system that can start small and grow with your business. In this session, Cameron demonstrates how to set up and schedule regular data extraction from SQL databases and other sources. He also covers some of the issues requiring attention in data extraction such as data merging and incremental loads. A future session will cover transformations and data enrichment along with data distribution.

TDWI research has found that organizations are increasingly modernizing their data warehouse environments. Often the current environment is not sufficient to support new analytics initiatives or they need to support new data types for analytics. Many enterprises are moving to the cloud as part of this journey. In fact, cloud data warehouses and cloud data lakes are already mainstream. The popularity of automated tools is growing as environments become more complex.