3 steps to build a data fabric to integrate all your data tools

MARY SHACKLETT | May 17, 2021

article image
One approach for better data utilization is the data fabric, a data management approach that arranges data in a single "fabric" that spans multiple systems and endpoints. The goal of the fabric is to link all data so it can easily be accessed.

"DataOps and data fabric are two different but related things," said Ed Thompson, CTO at Matillion, which provides a cloud data integration platform. "DataOps is about taking practices which are common in modern software development and applying them to data projects. Data fabric is about the type of data landscape that you create and how the tools that you use work together."

Spotlight

Redwood Algorithms

Redwood Algorithms (previously known as Redwood Associates) is an Analytics company that helps transform decision-making, by individuals and organizations every day, using Data & Analytics knowledge, expertise and tools.

OTHER ARTICLES

Can you really trust Amazon Product Recommendation?

Article | January 28, 2021

Since the internet became popular, the way we purchase things has evolved from a simple process to a more complicated process. Unlike traditional shopping, it is not possible to experience the products first-hand when purchasing online. Not only this, but there are more options or variants in a single product than ever before, which makes it more challenging to decide. To not make a bad investment, the consumer has to rely heavily on the customer reviews posted by people who are using the product. However, sorting through relevant reviews at multiple eCommerce platforms of different products and then comparing them to choose can work too much. To provide a solution to this problem, Amazon has come up with sentiment analysis using product review data. Amazon performs sentiment analysis on product review data with Artificial Intelligence technology to develop the best suitable products for the customer. This technology enables Amazon to create products that are most likely to be ideal for the customer. A consumer wants to search for only relevant and useful reviews when deciding on a product. A rating system is an excellent way to determine the quality and efficiency of a product. However, it still cannot provide complete information about the product as ratings can be biased. Textual detailed reviews are necessary to improve the consumer experience and in helping them make informed choices. Consumer experience is a vital tool to understand the customer's behavior and increase sales. Amazon has come up with a unique way to make things easier for their customers. They do not promote products that look similar to the other customer's search history. Instead, they recommend products that are similar to the product a user is searching for. This way, they guide the customer using the correlation between the products. To understand this concept better, we must understand how Amazon's recommendation algorithm has upgraded with time. The history of Amazon's recommendation algorithm Before Amazon started a sentiment analysis of customer product reviews using machine learning, they used the same collaborative filtering to make recommendations. Collaborative filtering is the most used way to recommend products online. Earlier, people used user-based collaborative filtering, which was not suitable as there were many uncounted factors. Researchers at Amazon came up with a better way to recommend products that depend on the correlation between products instead of similarities between customers. In user-based collaborative filtering, a customer would be shown recommendations based on people's purchase history with similar search history. In item-to-item collaborative filtering, people are shown recommendations of similar products to their recent purchase history. For example, if a person bought a mobile phone, he will be shown hints of that phone's accessories. Amazon's Personalization team found that using purchase history at a product level can provide better recommendations. This way of filtering also offered a better computational advantage. User-based collaborative filtering requires analyzing several users that have similar shopping history. This process is time-consuming as there are several demographic factors to consider, such as location, gender, age, etc. Also, a customer's shopping history can change in a day. To keep the data relevant, you would have to update the index storing the shopping history daily. However, item-to-item collaborative filtering is easy to maintain as only a tiny subset of the website's customers purchase a specific product. Computing a list of individuals who bought a particular item is much easier than analyzing all the site's customers for similar shopping history. However, there is a proper science between calculating the relatedness of a product. You cannot merely count the number of times a person bought two items together, as that would not make accurate recommendations. Amazon research uses a relatedness metric to come up with recommendations. If a person purchased an item X, then the item Y will only be related to the person if purchasers of item X are more likely to buy item Y. If users who purchased the item X are more likely to purchase the item Y, then only it is considered to be an accurate recommendation. Conclusion In order to provide a good recommendation to a customer, you must show products that have a higher chance of being relevant. There are countless products on Amazon's marketplace, and the customer will not go through several of them to figure out the best one. Eventually, the customer will become frustrated with thousands of options and choose to try a different platform. So Amazon has to develop a unique and efficient way to recommend the products that work better than its competition. User-based collaborative filtering was working fine until the competition increased. As the product listing has increased in the marketplace, you cannot merely rely on previous working algorithms. There are more filters and factors to consider than there were before. Item-to-item collaborative filtering is much more efficient as it automatically filters out products that are likely to be purchased. This limits the factors that require analysis to provide useful recommendations. Amazon has grown into the biggest marketplace in the industry as customers trust and rely on its service. They frequently make changes to fit the recent trends and provide the best customer experience possible.

Read More

Value Vs Cost: 3 Core Components to Evaluate a Data and Analytics Solution

Article | July 13, 2021

All business functions whether it is finance, marketing, procurement, or others find using data and analytics to drive success an imperative for today. They want to make informed decisions and be able to predict trends that are based on trusted data and insights from the business, operations, and customers. The criticality of delivering these capabilities was emphasised in a recent report, “The Importance of Unified Data and Analytics, Why and How Preintegrated Data and Analytics Solutions Drive Busines Success,” from Forrester Consulting. For approximately two-thirds of the global data warehouse and analytics strategy decision-makers surveyed in the research, their key data and analytics priorities are:

Read More

Time Machine Big Data of the Past for the Future of Europe

Article | February 24, 2020

Emerging technology has the power to transform history and cultural heritage into a living resource. The Time Machine project will digitise archives from museums and libraries, using Artificial Intelligence and Big Data mining, to offer richer interpretations of our past. An inclusive European identity benefits from a deep engagement with the region’s past. The Time Machine project set out to offer this by exploiting already freely accessible Big Data sources. EU support for a preparatory action enabled the development of a decade-long roadmap for the large-scale digitisation of kilometres of archives, from large museum and library collections, into a distributed information system. Artificial Intelligence (AI) will play a key role at each step, from digitisation planning to document interpretation and fact-checking. Once embedded, this infrastructure could create new business and employment opportunities across a range of sectors including ICT, the creative industries and tourism.

Read More

A learning guide to accelerate data analysis with SPSS Statistics

Article | May 20, 2021

IBM SPSS Statistics provides a powerful suite of data analytics tools which allows you to quickly analyze your data with a simple point-and-click interface and enables you to extract critical insights with ease. During these times of rapid change that demand agility, it is imperative to embrace data driven decision-making to improve business outcomes. Organizations of all kinds have relied on IBM SPSS Statistics for decades to help solve a wide range of business and research problems.

Read More

Spotlight

Redwood Algorithms

Redwood Algorithms (previously known as Redwood Associates) is an Analytics company that helps transform decision-making, by individuals and organizations every day, using Data & Analytics knowledge, expertise and tools.

Events