Twitter removes storage bottlenecks, speeds up Hadoop analytics by 50%

venturebeat.com | October 03, 2019

Twitter removes storage bottlenecks, speeds up Hadoop analytics by 50%
Think it’s hard keeping up with your Twitter feed? Imagine keeping track of all of Twitter. “Every tweet is comprised of over 100 data points,” says Matt Singer, a senior staff hardware engineer responsible for server architecture at Twitter. Data from every retweet, “unfollow”, link-click and other actions feeds analytic and deep learning systems serving operational, advertising. and other vital functions. It’s a nonstop stream exceeding 1.5 trillion events per day.How does an organization handle such hyper-scale demands? Twitter relies upon one of the world’s biggest deployments of Hadoop clusters. The open source Big Data analytics software helps the company generate business insights that allow it to manage and grow its vast global network. To enhance its reputation as the premier streaming information service, Twitter turned to Intel to help amplify its growth by maximizing performance and slashing rising storage costs.

Spotlight

Here's a great visual overview of what you need to get started with a data-driven customer loyalty programme: the questions to ask before getting started and an overview of all the possible data sources to consider.

Spotlight

Here's a great visual overview of what you need to get started with a data-driven customer loyalty programme: the questions to ask before getting started and an overview of all the possible data sources to consider.

Related News
DATA SCIENCE

Equity Data Science Appoints Theresa Elamparo as Head of Marketing

Equity Data Science | December 06, 2021

Equity Data Science (“EDS”), a cloud-based analytics platform provider that delivers decision support tools for the investment process to hedge funds and asset managers, has named Theresa Elamparo as Head of Marketing to accelerate the company’s business strategy and growth plan. Elamparo will lead all brand, marketing and communications strategies to continue to build the brand and to focus on the expansion of the business. She will be based in New York. Elamparo brings 23 years of marketing communications experience, serving the fintech community for 16 years, most recently as Chief Marketing Officer at Tier1 Financial Solutions where she was the recipient of the 2019 Markets Media Women in Finance award for Excellence in Marketing and Communications for her work leading Tier1’s rebrand and building the firm’s marketing organization. Prior to that, she held marketing leadership roles at fintech firms including Ipreo, Investment Technology Group and Tradeweb Markets. We are thrilled to have Theresa join our leadership team. Her wealth of experience within the financial services industry and strategic marketing expertise will be integral to expanding our global footprint as we deliver on growing demand for data aggregation, analytics, workflow and scalable decision support for the fundamental investment process.” Greg McCall, President and Co-founder at EDS “I’m delighted to join EDS at such an exciting time,” Elamparo said. “The fundamental investment community is faced with fragmentation, underutilized data and technical inefficiencies. EDS provides a modular, decision-support workflow platform for the full investment lifecycle, helping clients better manage their process to maximize returns.” Throughout 2021, EDS invested heavily in building out solid leadership across key functions, including the appointment of Jen Vermeulen, CFA as Head of Sales, and Erin Greenfield as Head of Customer Success. These recent appointments strengthen EDS’s ability to expand its position in fundamental investing. At the start of the year, Northern Trust announced a strategic investment in EDS, allowing for integration of EDS’s decision-support tools with Northern Trust’s core technology platforms to provide highly specialized and innovative solutions to the most sophisticated institutional investors across the globe. ABOUT EQUITY DATA SCIENCE Equity Data Science (EDS) empowers fundamental investors to build, operate and sustain a modernized, repeatable investment process by aggregating data sources and refining workflows to govern investment decisions. EDS provides a fully configurable, measurable, and scalable platform with purpose-built analytics to support idea generation, research management, portfolio construction and risk management

Read More

BIG DATA MANAGEMENT

BigID Introduces Tableau Metadata Exchange App to Deliver New Insights for High-Value Data

BigID | December 03, 2021

BigID, the leader in data discovery and intelligence for privacy, protection, and perspective, today introduced the first-of-its-kind data catalog integration app for Tableau, the world's leading analytics platform. Joint customers can get more value from their data through powerful analytics fueled with the highest quality and security and privacy vetted data. BigID provides Tableau users ML-based capabilities for finding high-value, correct, and meaningful data that simplify data protection and governance strategies. Using BigID and Tableau customers can: Automatically discover, classify, catalog, and correlate high-value data across all types of data, structured and unstructured Provide visual cues on data sources, databases, workbooks, tables, and columns to indicate high-value data, data quality results, and risk scorings Improve analytics governance with deep and contextual data knowledge at scale "Tableau has always been committed to helping everyone see and understand data," said Brian Matsubara, RVP, Global Technology Alliances, Tableau. "By integrating with BigID, we're making it easier to trust and discover the right data to drive insights." "BigID is the most comprehensive data intelligence platform in the market to enrich your Tableau Catalog. By providing deep data insights across all data sources, we enable our customers to curate their catalog of analytics content, making it easier for analytic consumers to know not only what content is available, but which is the most accurate and relevant." Dimitri Sirota, CEO and founder of BigID BigID joins Tableau's effort in helping organizations build a Data Culture by enabling them to use the highest quality data to solve problems and protect the high-value data. About BigID BigID's data intelligence platform enables organizations to know their enterprise data and take action for privacy, protection, and perspective. Customers deploy BigID to proactively discover, manage, protect, and get more value from their regulated, sensitive, and personal data across their data landscape. BigID has been recognized for its data intelligence innovation as a 2019 World Economic Forum Technology Pioneer, named to the 2021 Forbes Cloud 100, the 2021 Inc 5000 as the #19th fastest growing company and #1 in Security, a Business Insider 2020 AI Startup to Watch, and an RSA Innovation Sandbox winner.

Read More

DATA ARCHITECTURE

Modak Joins Starburst Partner Program to Create Domain-Driven Data Products Based on a Data Mesh Architecture

Modak | December 02, 2021

Modak, a leading provider of data engineering solutions, today announced they have joined the Starburst Orbit partner program. This partnership will empower organizations to manage their siloed data landscape, optimize cloud operating costs and accelerate the creation of domain-specific data products accessed through a data mesh architecture. "The partnership will empower organizations to transition from centralized to decentralized data processing and de-couple compute from storage. The unique value proposition will fast-track the creation of contextual business data domain products at reduced time and operational costs." Milind Chitgupakar, Chief Analytics Officer, and co-founder, at Modak. "We're delighted to have Modak join our partner program to deliver their unique expertise to the pharmaceutical and healthcare market. Starburst is allowing companies to unlock the value of their data by making it fast and easy to access, no matter where it lives, without data movement. We are excited to help our customers bring a more personalized experience to their customers, patients, partners and to build data into their core business," says Tony Li, Global Head of SIs at Starburst Data. Enterprise Data Management has evolved, from data warehouses to data lakes, and now to multimodal cloud architecture. All these data architecture models have undergone evolutionary enhancements. But, despite all advancements, some things remain unchanged. Data management architecture and technology are still monolithic, and data remains centralized. There is a need for a decentralized data management architecture that allows for easier access to data across different data domains, i.e., through a Data Mesh. The fundamental concept behind data mesh architecture is to transfer data ownership to the business owners and users who create the data. Operationally, the Data mesh architecture allows organizations to treat "data as products", just like software applications, rather than fragmented data sets centrally managed by IT. With Modak's digital accelerators, such as Modak Nabu™ and Starburst's enterprise-ready, powerful query engine, organizations will be able to create contextual business data domain products accessible through APIs and will move to federated data governance and self-service analytics capabilities. Modak recently enabled a Top 5 pharma company to productionize six domain data products within months, using cloud and data mesh architecture. The six domain data products are now accessible to thousands of users across the enterprise. About Modak Modak is a solutions company that enables enterprises to manage and utilize their data landscape effectively. Modak provides technology, cloud, and vendor-agnostic software and services to accelerate data migration initiatives. Modak uses machine learning (ML) techniques to transform how structured and unstructured data is prepared, consumed, and shared. Modak's portfolio of Data Engineering and DataOps studios provides best-in-class delivery services, managed data operations, enterprise data lake, data mesh, augmented data preparation, data quality, and governed data lake solutions. About Starburst Starburst is the analytics engine for the Data Mesh. We unlock the value of distributed data by making it fast and easy to access, no matter where it lives. Starburst queries data across any database, making it instantly actionable for data-driven organizations. With Starburst, teams can lower the total cost of their infrastructure and analytics investments, prevent vendor lock-in, and use the existing tools that work for their business. Trusted by companies like Comcast, FINRA, and Condé Nast, Starburst helps companies make better decisions faster on all data.

Read More