Why BI Programs Fail to Scale: The Data Decision Gap

Much like project management and home improvements, Data Governance sounds a lot simpler than it actually is. In a nutshell, Data Governance can be explained as “managing data with guidance.” In general, the perceived utility of these programs increases with the specificity of desired data and processing improvements. Whether restarting or starting your Data Governance programs, it is critical to be guided by a periodically revised Data Strategy that links support for organizational strategy to specific operational data improvements. Understanding these and other aspects of governance is necessary to eliminate the ambiguity that often surrounds the implementation of effective Data Management and stewardship programs.

Today’s data-driven companies have a choice to make – where do we store our data? As the move to the cloud continues to be a driving factor, the choice becomes either the data warehouse (Snowflake et al) or the data lake (AWS S3 et al). There are pro’s and con’s for each approach. While the data warehouse will give you strong data management with analytics, they don’t do well with semi-structured and unstructured data with tightly coupled storage and compute, not to mention expensive vendor lock-in. On the other hand, data lakes allow you to store all kinds of data and are extremely affordable, but they’re only meant for storage and by themselves provide no direct value to an organization.

One of the biggest misunderstandings about data governance? An assumption that you need data governance only because you’re working with inherently low-quality or low-value data. The data governance team at Genworth Financial focused on changing the way the organization thinks about governance in order to improve how it captures, stores, transforms, analyzes, and presents data—which in turn unlocked their data’s full value.

Whether to take data ingestion cycles off the ETL tool and the Data Warehouse or to facilitate competitive Data Science and building algorithms in the organization, the Data Lake a place for unmodeled and vast data will be provisioned widely in 2019. Though it doesn’t have to be complicated, the Data Lake has a few key design points that are critical, and it does need to follow some principles for success. Avoid building the Data Swamp, but not the Data Lake! The tool ecosystem is building up around the Data Lake and soon many will have a robust Lake and Data Warehouse. We will discuss policy to keep them straight, send “horses to courses,” and keep up users’ confidence in the Data Platforms.