Business Intelligence, Big Data Management

Fivetran Introduces Metadata API – Enables End-to-End Data Analysis and Visibility

Fivetran | September 20, 2022 | Read time : 03:00 min

Fivetran
Fivetran, the global leader in modern data integration, today announced the availability of Fivetran’s Metadata API to enable the tracking of data “in-flight” from source to destination as it moves through Fivetran-managed pipelines. With this additional visibility, customers can integrate with governance and observability tools to give data teams more control over who has access to what data. Enabling automated data governance, the Fivetran Metadata API also provides data stewards, security teams and data engineering teams the needed visibility to answer where the data came from, who accessed it, and what changes have occurred in the pipeline.

“Every enterprise knows it must be data-driven, but traditional data governance has been a barrier with manual processes and reactive enforcement of policies. That's not a scalable approach, especially as data infrastructure grows to thousands of pipelines. “With Metadata API, our customers get out-of-the-box data governance automations and data quality workflows so they can proactively identify and take action on governance issues before they become a problem. Our automated in-flight approach enables data access at scale without increasing risk to the business.”

Fraser Harris, VP of Product at Fivetran

Complex legislation such as Europe’s GDPR, the California Consumer Privacy Act (CCPA), and other state laws impose fines on companies that do not comply with complex rules to safeguard data privacy. The Fivetran Metadata API helps enterprises address compliance requirements, easily integrating into their existing privacy and security strategies while bolstering the value of investments they’ve made in data catalogs and data quality solutions.

With the Fivetran Metadata API:

  • Data analysts are provided with a deep understanding of where data is coming from and are able to run impact analyses on it.
  • Data stewards know end users have access to data that has been handled securely and is compliant with governance requirements.
  • Security and legal teams can complete security audits and ensure the data moved is in compliance with organizational policies.
  • Data architects and engineers will soon be able to understand upstream schema changes and ensure downstream processes are updated.

Creating an automated data governance experience – Fivetran partners with leading data catalogs

Fivetran is excited to launch better metadata management with four leading data catalog vendors: Atlan, data.world, Alation and Collibra. Here are the combined benefits:

  • Information about all data can be consolidated into a single data catalog, for a complete view and seamless user experience.
  • End-to-end data lineage graphs are available for data, despite data passing through multiple systems and tools.
  • By centralizing governance in a single tool, data stewards can better ensure policies and processes are being applied to data as necessary.
  • The ability to source trace data at a column level back to its origin helps confirm data quality and builds trust that the data is accurate and safe to use.

“Fivetran’s Metadata API solves a major gap in extracting data from operational systems into modern analytical systems by delivering the much-needed context. The availability of the Metadata API will accelerate development of reliable and secure data-intensive applications by exposing lineage, impact analysis, and security and privacy aspects,�� said Sanjeev Mohan, Principal at SanjMo, a data and analytics expert and former Research Vice President at Gartner.

“Our customers are excited to move faster with more visibility, powered by metadata from Fivetran,” said Prukalpa Sankar, Co-Founder at Atlan. “No need to manually stitch together sources anymore. With the new Metadata API, customers can now bring metadata from Fivetran into Atlan for a truly automated data governance experience. This end-to-end lineage covers everything from upstream sources to the warehouse and BI layer, helping customers stay informed across complex data stacks with powerful root cause and impact analysis.”

“data.world's interface with Fivetran enables data producers and data consumers to understand and trust the data synchronization and mapping along all databases, data sources and applications that matter to enterprises,” said Jon Loyens, CPO & Co-Founder at data.world. “Together with Fivetran, we provide granular data visibility, including metadata of source-destination column pairs, simplifying data discovery, data governance and actionable insights.”

A new report issued by Vanson Bourne for Fivetran highlights that companies continue to struggle with proper data governance – a necessity for compliance. All (100%) U.S. respondents said their companies could improve data governance roles, policies and standards, with 90% of those polled in France, 82% of those surveyed in the U.K. and Ireland, and 82% of respondents in Germany saying the same.

About Fivetran
Fivetran is the global leader in modern data integration. Our mission is to make access to data as simple and reliable as electricity. Built for the cloud, Fivetran enables data teams to effortlessly centralize and transform data from hundreds of SaaS and on-prem data sources into high-performance cloud destinations. Fast-moving startups to the world’s largest companies use Fivetran to accelerate modern analytics and operational efficiency, fueling data-driven business growth. Fivetran is headquartered in Oakland, California, with offices around the world.

Spotlight

User Entity and Behavior Analytics (UEBA) is a cybersecurity technology and approach that focuses on analyzing the behavior of users and entities (such as devices, applications, and systems) within an organization's IT environment. By using advanced data analytics, machine learning algorithms, and artificial intelligence, UEBA aims to detect and prevent cyber threats by identifying anomalies, deviations, or patterns in user and entity activities that might indicate potential security risks.


Other News
Big Data Management

Kinetica Redefines Real-Time Analytics with Native LLM Integration

Kinetica | September 22, 2023

Kinetica, a renowned speed layer for generative AI and real-time analytics, has recently unveiled a native Large Language Model (LLM) integrated with Kinetica's innovative architecture. This empowers users to perform ad-hoc data analysis on real-time, structured data with the ease of natural language, all without the need for external API calls and without data ever leaving the secure confines of the customer's environment. This significant milestone follows Kinetica's prior innovation as the first analytic database to integrate with OpenAI. Amid the LLM fervor, enterprises and government agencies are actively seeking inventive ways to automate various business functions while safeguarding sensitive information that could be exposed through fine-tuning or prompt augmentation. Public LLMs, exemplified by OpenAI's GPT 3.5, raise valid concerns regarding privacy and security. These concerns are effectively mitigated through native offerings, seamlessly integrated into the Kinetica deployment, and securely nestled within the customer's network perimeter. Beyond its superior security features, Kinetica's native LLM is finely tuned to the syntax and industry-specific data definitions, spanning domains such as telecommunications, automotive, financial services, logistics, and more. This tailored approach ensures the generation of more reliable and precise SQL queries. Notably, this capability extends beyond conventional SQL, enabling efficient handling of intricate tasks essential for enhanced decision-making capabilities, particularly for time-series, graph, and spatial inquiries. Kinetica's approach to fine-tuning places emphasis on optimizing SQL generation to deliver consistent and accurate results, in stark contrast to more conventional methods that prioritize creativity but yield diverse and unpredictable responses. This steadfast commitment to reliable SQL query outcomes offers businesses and users the peace of mind they deserve. Illustrating the practical impact of this innovation, the US Air Force has been collaborating closely with Kinetica to leverage advanced analytics on sensor data, enabling swift identification and response to potential threats. This partnership contributes significantly to the safety and security of the national airspace system. The US Air Force now employs Kinetica's embedded LLM to detect airspace threats and anomalies using natural language. Kinetica's database excels in converting natural language queries into SQL, delivering responses in mere seconds, even when faced with complex or unfamiliar questions. Furthermore, Kinetica seamlessly combines various analytics modes, including time series, spatial, graph, and machine learning, thereby expanding the range of queries it can effectively address. What truly enables Kinetica to excel in conversational query processing is its ingenious use of native vectorization. In a vectorized query engine, data is organized into fixed-size blocks called vectors, enabling parallel query operations on these vectors. This stands in contrast to traditional approaches that process individual data elements sequentially. The result is significantly accelerated query execution, all within a smaller compute footprint. This remarkable speed is made possible by the utilization of GPUs and the latest CPU advancements, which enable simultaneous calculations on multiple data elements, thereby greatly enhancing the processing speed of computation-intensive tasks across multiple cores or threads. About Kinetica Kinetica is a pioneering company at the forefront of real-time analytics and is the creator of the groundbreaking real-time analytical database specially designed for sensor and machine data. The company offers native vectorized analytics capabilities in the fields of generative AI, spatial analysis, time-series modeling, and graph processing. A distinguished array of the world's largest enterprises spanning diverse sectors, including the public sector, financial services, telecommunications, energy, healthcare, retail, and automotive industries, entrusts Kinetica to forge novel solutions in the realms of time-series data and spatial analysis. The company's clientele includes various illustrious organizations such as the US Air Force, Citibank, Ford, T-Mobile, and numerous others.

Read More

Big Data Management

Congruity360 Delivers Intelligent Data Migrations and Storage Tiering

PR Newswire | September 27, 2023

Congruity360, a leading unstructured data management and risk mitigation provider, announces the addition of data mobility in Enterprise Insights. As unstructured data grows at the annual rate of 55% to 65% and accounts for more than 80% of all enterprise data, businesses must find a way to identify, classify and move data intelligently and automatically during its lifecycle. As enterprises grow, their valuable data must mature with their business. This may require a journey to the cloud, SLA changes which optimize storage costs, classification to mitigate risk, and moving the right data to additional key AI platform initiatives. A simple, scalable, high-performance data classification engine, Enterprise Insights delivers next-generation data lifecycle management for storage optimization, security and risk optimization, and IT business optimization. Enterprise Insights Approach to Successful Data Optimization: Identify – Securely analyze PBs of unstructured data across on premises (NAS & object) and cloud (files/objects & SaaS) sources by harnessing the power of the platform's rapid insights and auto-discover technologies, which can reduce data identification times by 1,000%. Classify – Quickly identify key client data attributes for cost savings, risk mitigation, and business impact with simple to consume dashboards and drill down capabilities. Review – Confidently create and take actions by leveraging the comprehensive search engine to quickly find and preview data for movement without ever leaving the platform. Remediate – Seamlessly take action (migrate and tier) on classified data to ensure it's properly protected, optimally stored, and most effectively serving the business. Enterprise Insights offers three use case-driven insight analysis modules: Storage and Migration Optimization – Insights into over 35 file data attributes including systems' aged, stale, obsolete, redundant, trivial, and types of systems files. Business Optimization – Insight into and classification by business units' or cost centers' aged, stale, obsolete, redundant, trivial, and types of files. Data Security and Risk Optimization – Insights into files containing PII and SPII, financial, legal, security, and risk data, as well as open shares and other network & storage security vulnerabilities. By leveraging Enterprise Insights, clients can classify data for simple and secure migration both on premise and in the cloud. Equally important is Insights data tiering capabilities, enabling users to match data storage costs to data usage. Powered by the Classify360 Platform, Enterprise Insights' secure hybrid approach to data analysis scales capabilities to exabyte levels at unmatched speed. Enterprise Insights is the industry's most powerful weapon to tackle the costs, time, and complexity of cloud migration projects, backup modernization, storage tiering, hardware refresh, and security posture management. By providing users with dashboards highlighting their existing storage costs and risks, Enterprise Insights frees clients from hidden, legacy, CapEx and OpEx expenditures, performance, and scalability bottlenecks while discovering and acting on sensitive and risk data. Unstructured data insanity is treating all data equally with zero insights into its business impact, said Brian Davidson, Chief Executive Officer and Managing Partner of Congruity360. Enterprise Insights is the first step in implementing optimized data lifecycle management. With historically high data growth and new business uses for unstructured data, it is essential to attack the costs and risks inherent in unmanaged data. Our customers have realized 7-10x returns on their data lifecycle management implementations while reducing risk in an auditable compliance framework. As AI continues to gain steam, don't overpay by moving useless data to your expensive AI platforms. The Classify360 Platform is comprehensive, simple to implement, scale, and operate. Businesses leverage the Classify360 Platform for unstructured data discovery, classification, business workflows, remediation actions, and insightful reporting. Congruity360 continues to tackle additional data governance challenges through innovations to the Classify360 Platform to continue delivering revolutionary data governance and classification, at scale, to the enterprise world. ABOUT CONGRUITY360 Congruity360 delivers the only data life cycle management solution built on a foundation of classification, by expert data storage engineers alongside expert data privacy consultants. The Classify360 Platform is easy to implement, requires no outside consultants, and quickly analyzes your data at the petabyte scale in days, not weeks or months.

Read More

Data Visualization

Airbyte Integrates with Datadog for Superior Data Pipeline Management

Airbyte | September 11, 2023

Airbyte, the leading open-source data movement platform, has announced a strategic integration with Datadog, Inc., a prominent cloud application monitoring and security platform. This integration offers customers a comprehensive solution to monitor and analyze data pipelines with access to nearly 50 metrics, all at no additional cost. The integration between Airbyte Self-Managed and Datadog's data observability and security monitoring capabilities allows organizations to maintain a close watch on the health of their critical data pipelines. Key features of this integration include: A centralized overview of Airbyte data pipeline performance Real-time detection and immediate alerts for failing syncs or connections Notifications regarding long-running jobs, which could indicate potential latency issues Michel Tricot, CEO of Airbyte, emphasized the significance of this integration, stating, The new Datadog integration provides transparency and actionable insights, empowering users to optimize performance and ensure reliable data pipelines by proactively addressing potential data issues. [Source: Business Wire] Yrieix Garnier, Vice President of Product at Datadog, further elaborated on the benefits, explaining, Airbyte's data extraction and loading process involves numerous complex components. The integration with Datadog offers users peace of mind, enabling them to monitor data pipelines across their organization and troubleshoot any potential data integration workflow issues, ultimately ensuring data quality. [Source: Business Wire] This integration will be immediately available to users. Existing Datadog customers can configure their Airbyte deployments to send metrics to Datadog. For those not already using Datadog, a free trial is available. Similarly, users new to Airbyte can sign up for free. Airbyte continues its commitment to delivering robust data integration and analysis solutions to organizations. The Datadog integration represents a significant milestone in Airbyte's mission to empower businesses with efficient data integration capabilities. Airbyte simplifies data movement across various sources and destinations, making it accessible and cost-effective for enterprises. With the largest data engineering contributor community, boasting over 800 contributors as well as top-tier tools for connector development and maintenance, Airbyte remains at the forefront of the data integration landscape. About Airbyte Founded in 2020, Airbyte is an open-source platform for EL (T) that enables data teams to replicate data from various sources to different destinations. The company, which has raised $181 million in funding, believes in the power of open source to address data integration challenges and offers over 200 connectors for data syncing. Currently, it serves over 25,000 companies.

Read More

Data Science

J.D. Power Acquires Autovista Group to Expand Automotive Data Portfolio

J.D. Power | September 18, 2023

J.D. Power, a prominent global leader in data analytics, has recently announced a definitive agreement to acquire Autovista Group, a renowned pan-European and Australian automotive data, analytics, and industry insights provider. This strategic acquisition complements J.D. Power's existing strengths in vehicle valuation and intricate vehicle specification data and analytics while significantly expanding its presence within the European and Australian automotive markets. This acquisition represents a crucial moment, as it delivers substantial value to the customers of both companies. It brings together Autovista Group's extensive European and Australian market intelligence with J.D. Power's market-leading predictive analytics, valuation data, and customer experience datasets. These complementary offerings will empower original equipment manufacturers (OEMs), insurers, dealers, and financing companies with a truly global perspective on critical industry trends. They will also provide the tools to accurately predict risk, capitalize on emerging trends, and align sales strategies with real-time market dynamics. Pete Cimmet, Chief Strategy Officer at J.D. Power, stated: The addition of Autovista Group broadens our global presence allowing us to serve our customers across key global markets including North America, Europe and Asia/Australia. We look forward to partnering with the Autovista team to launch innovative new products and pursue strategic add-on acquisitions in Europe and Australia. [Source: Business Wire] Autovista Group, through its five prominent brands—Autovista, Glass's, Eurotax, Schwacke, and Rødboka—standardizes and categorizes a multitude of technical attributes for nearly every vehicle manufactured in European and Australian markets. This comprehensive approach offers clients a 360-degree view of detailed vehicle data, which is invaluable for valuations, forecasts, and repair estimates. Furthermore, Autovista Group's robust analytical solutions and its team of seasoned experts are trusted by stakeholders across the automobile industry for their in-depth insights and benchmarks related to vehicle values, ownership, replacements, and repair costs. Under this agreement, Autovista Group's senior leadership, along with its 700 employees, will remain part of the organization, serving as J.D. Power's automotive data and analytics platform for Australia and Europe. Lindsey Roberts will continue to lead the team in her role as President of J.D. Power Europe, reporting to CEO Dave Habiger. Currently, Autovista Group is owned by Hayfin Capital Management, a prominent European alternative asset management firm. The anticipated closure of the Autovista Group acquisition is set for conclusion by the end of 2023, pending customary closing conditions and regulatory review and approval. For this transaction, RBC Capital Markets acted as the exclusive financial advisor, and Kirkland & Ellis provided legal counsel to J.D. Power. TD Cowen served as the exclusive financial advisor, with Macfarlanes, Cravath, Swaine & Moore, and Mishcon de Reya acting as legal advisors to Autovista Group and Hayfin. About J.D. Power J.D. Power, a renowned consumer insights, advisory services, and data and analytics firm, has consistently spearheaded the use of big data, artificial intelligence (AI), and algorithmic modeling to illuminate the intricacies of consumer behavior for more than half a century. With a storied legacy of providing in-depth industry intelligence on customer interactions with brands and products, J.D. Power serves as the trusted leader for the world's preeminent enterprises, spanning diverse major sectors, profoundly influencing and refining their customer-centric strategies.

Read More

Spotlight

User Entity and Behavior Analytics (UEBA) is a cybersecurity technology and approach that focuses on analyzing the behavior of users and entities (such as devices, applications, and systems) within an organization's IT environment. By using advanced data analytics, machine learning algorithms, and artificial intelligence, UEBA aims to detect and prevent cyber threats by identifying anomalies, deviations, or patterns in user and entity activities that might indicate potential security risks.

Resources