Building the Enterprise Data Lake – Important Considerations Before You Jump In

The world of data and analytics is evolving faster than ever before. Those that stay ahead of this change will be able to navigate the challenging economy and dominate the decade of data. Data is driving tectonic shifts across our personal and professional lives, creating new opportunities for organizations - and new challenges. But which trends are truly transformative and which will fail to live up to the hype?

Building performant ETL pipelines to address analytics requirements is hard as data volumes and variety grow at an explosive pace. With existing technologies, data engineers are challenged to deliver data pipelines to support the real-time insight business owners demand from their analytics. Databricks Delta is the next generation of evolution in big data processing from Databricks, the company founded by the original creators of Apache Spark.

At its core, Data Governance (DG) is all about managing data with guidance. This immediately provokes the question: Would you tolerate any of your assets to be managed without guidance? (In all likelihood, your organization has been managing data without adequate guidance and this accounts for its current, less-than-optimal state.) This program provides a practical guide to implementing DG or recharging your existing program. It provides an understanding of what Data Governance functions are required and how they fit with other Data Management disciplines. Understanding these aspects is a prerequisite to eliminate the ambiguity that often surrounds initial discussions and implement effective Data Governance/Stewardship programs that manage data in support of organizational strategy. Delegates will understand why Data Governance can be tricky for organizations due to data’s confounding characteristics.

Modern-day systems are transforming into complex, open-source, cloud-native services running on various environments and being developed/deployed at lightning speed by distributed teams. When working on these systems, identifying a broken link in the chain can be near impossible. Everything fails at one point or another, whether due to code bugs, infrastructure overload, or changes in end-user behavior or market driven factors or errors in data collection. This has led to the rise of DataOps with a focus on changing the organizational speed and trust in delivering data pipelines and the related artifacts by co-creating “decision quality” data with the consumers. This development has led to the idea of observability that includes monitoring, tracking, and triaging incidents to prevent downtime of the systems and around several factors such as freshness, distribution, volume, schema, lineage.