Building Next-Gen Data Pipelines with Databricks Delta

Whether to take data ingestion cycles off the ETL tool and the Data Warehouse or to facilitate competitive Data Science and building algorithms in the organization, the Data Lake a place for unmodeled and vast data will be provisioned widely in 2019. Though it doesn’t have to be complicated, the Data Lake has a few key design points that are critical, and it does need to follow some principles for success. Avoid building the Data Swamp, but not the Data Lake! The tool ecosystem is building up around the Data Lake and soon many will have a robust Lake and Data Warehouse. We will discuss policy to keep them straight, send “horses to courses,” and keep up users’ confidence in the Data Platforms.

As data and analytics environments become increasingly complex, organizations can no longer afford to perform many operations manually. According to TDWI research, automation (in general) is one of the top three priorities for analytics. We see automation occurring throughout the data and analytics life cycle. Automation increasingly leverages embedded AI/ML algorithms (i.e., infused in the software) to help perform tasks such as profiling and cleansing data, identifying sensitive data, data mapping, surfacing insights, or building machine learning models.

If your organization is in a highly-regulated industry or relies on data for competitive advantage data governance is undoubtedly a top priority. Whether you’re focused on “defensive” data governance (supporting regulatory compliance and risk management) or “offensive” data governance (extracting the maximum value from your data assets, and minimizing the cost of bad data), data quality plays a critical role in ensuring success.

We all strive for great data and insights to feed into decisions that propel business’ forward. In today’s fast-spinning world we need to do that ever more quickly and cost-effectively. But when moving quickly, the research participant and their needs can be forgotten, and this can have negative consequences for data quality.