Databricks Conquers AI Dilemma with Unified Analytics

Despite the allure of artificial intelligence (AI), most enterprises are struggling to succeed with AI. Preliminary findings of a Databricks commissioned research study reveals that 96 percent of organizations say data-related challenges are the most common obstacle when moving AI projects to production. Data is the key to AI, but data and AI sit in technology and organizational silos. Databricks, a leader in unified analytics and founded by the original creators of Apache Spark™, addresses this AI dilemma with the Unified Analytics Platform. The company launched new capabilities to lower the barrier for enterprises to innovate with AI. These new capabilities unify data and AI teams and technologies: MLflow for developing an end-to-end machine learning workflow, Databricks Runtime for ML to simplify distributed machine learning; and Databricks Delta for data reliability and performance at scale. Data is critical to both training and productionizing machine learning. But using machine learning in production is difficult because the development process is ad hoc, lacking tools to reproduce results, track experiments and manage models. To address this problem, Databricks introduces MLflow, an open source, cross-cloud framework that can dramatically simplify the machine learning workflow. With MLflow, organizations can package their code for reproducible runs, execute and compare hundreds of parallel experiments, leverage any hardware or software platform, and deploy models to production on a variety of serving platforms. MLflow integrates with Apache Spark, SciKit-Learn, TensorFlow and other open source machine learning frameworks.