IT wants (but struggles) to operationalize big data

THOR OLAVSRUD |

article image
Big data leaders at large companies are confident their big data strategies are headed in the right direction, but most also feel that they're struggling to operationalize them, according to a new survey…

Spotlight

Decisive Data

Decisive Data is an end-to-end business analytics consultancy and thought leader. We deliver repeatable, actionable insight for making critical business decisions informed by data and science. We’re the best in the world at data visualization – our visualization methodology is widely recognized by industry leaders. We offer a full data analytics stack; data science, predictive analytics, data management, and application development.

OTHER ARTICLES

IS YOUR ORGANISATION DATA SCIENCE READY

Article | February 19, 2020

With the increasing amount of data in modern businesses, data science has been receiving a lot of attention. A growing number of companies are, nowadays investing in data science researchers and experts to implement technologies like artificial intelligence and machine learning in their organisation in order to derive actionable insights. But, to place such a massive transformation in an organisation, one has to ensure complete business readiness for data science. Although it is interesting to imagine the potential benefits data science can provide for your organisation, it is worth evaluating how much your organisation is prepared to accommodate a team of data scientists.

Read More
THEORY AND STRATEGIES

Can you really trust Amazon Product Recommendation?

Article | February 19, 2020

Since the internet became popular, the way we purchase things has evolved from a simple process to a more complicated process. Unlike traditional shopping, it is not possible to experience the products first-hand when purchasing online. Not only this, but there are more options or variants in a single product than ever before, which makes it more challenging to decide. To not make a bad investment, the consumer has to rely heavily on the customer reviews posted by people who are using the product. However, sorting through relevant reviews at multiple eCommerce platforms of different products and then comparing them to choose can work too much. To provide a solution to this problem, Amazon has come up with sentiment analysis using product review data. Amazon performs sentiment analysis on product review data with Artificial Intelligence technology to develop the best suitable products for the customer. This technology enables Amazon to create products that are most likely to be ideal for the customer. A consumer wants to search for only relevant and useful reviews when deciding on a product. A rating system is an excellent way to determine the quality and efficiency of a product. However, it still cannot provide complete information about the product as ratings can be biased. Textual detailed reviews are necessary to improve the consumer experience and in helping them make informed choices. Consumer experience is a vital tool to understand the customer's behavior and increase sales. Amazon has come up with a unique way to make things easier for their customers. They do not promote products that look similar to the other customer's search history. Instead, they recommend products that are similar to the product a user is searching for. This way, they guide the customer using the correlation between the products. To understand this concept better, we must understand how Amazon's recommendation algorithm has upgraded with time. The history of Amazon's recommendation algorithm Before Amazon started a sentiment analysis of customer product reviews using machine learning, they used the same collaborative filtering to make recommendations. Collaborative filtering is the most used way to recommend products online. Earlier, people used user-based collaborative filtering, which was not suitable as there were many uncounted factors. Researchers at Amazon came up with a better way to recommend products that depend on the correlation between products instead of similarities between customers. In user-based collaborative filtering, a customer would be shown recommendations based on people's purchase history with similar search history. In item-to-item collaborative filtering, people are shown recommendations of similar products to their recent purchase history. For example, if a person bought a mobile phone, he will be shown hints of that phone's accessories. Amazon's Personalization team found that using purchase history at a product level can provide better recommendations. This way of filtering also offered a better computational advantage. User-based collaborative filtering requires analyzing several users that have similar shopping history. This process is time-consuming as there are several demographic factors to consider, such as location, gender, age, etc. Also, a customer's shopping history can change in a day. To keep the data relevant, you would have to update the index storing the shopping history daily. However, item-to-item collaborative filtering is easy to maintain as only a tiny subset of the website's customers purchase a specific product. Computing a list of individuals who bought a particular item is much easier than analyzing all the site's customers for similar shopping history. However, there is a proper science between calculating the relatedness of a product. You cannot merely count the number of times a person bought two items together, as that would not make accurate recommendations. Amazon research uses a relatedness metric to come up with recommendations. If a person purchased an item X, then the item Y will only be related to the person if purchasers of item X are more likely to buy item Y. If users who purchased the item X are more likely to purchase the item Y, then only it is considered to be an accurate recommendation. Conclusion In order to provide a good recommendation to a customer, you must show products that have a higher chance of being relevant. There are countless products on Amazon's marketplace, and the customer will not go through several of them to figure out the best one. Eventually, the customer will become frustrated with thousands of options and choose to try a different platform. So Amazon has to develop a unique and efficient way to recommend the products that work better than its competition. User-based collaborative filtering was working fine until the competition increased. As the product listing has increased in the marketplace, you cannot merely rely on previous working algorithms. There are more filters and factors to consider than there were before. Item-to-item collaborative filtering is much more efficient as it automatically filters out products that are likely to be purchased. This limits the factors that require analysis to provide useful recommendations. Amazon has grown into the biggest marketplace in the industry as customers trust and rely on its service. They frequently make changes to fit the recent trends and provide the best customer experience possible.

Read More
PERFORMANCE MANAGEMENT

Living upto Learn, Re Learn and Unlearn

Article | February 19, 2020

Learn, re Learn and Unlearn The times we are living in, we have to upgrade ourselves constantly in order to stay afloat with the industry be it Logistics, Traditional business, Agriculture, etc.. Technology is constantly changing our lives the way we used to live, living and will live. Anyone who thinks technology is not their cup of tea then I would say he /she will have no place in the world to live. It’s a blessing or curse on human race, only time will tell but effects are already surfacing in the market in the form of Job cut, poverty, some roles are no longer needed or replaced with. Poor is getting poorer and rich is getting richer. Covid19 has not only brought the curse on human race but it has been a blessing in disguise for Tech giants and E-commerce. Technology not only changing the business but every human’s outlook towards life, family structure, the globalization of talents etc. It is nerve wrenching to imagine just what the world will look like in coming 20 years from now. Can all of us adapt to learn, re learn and unlearn quote? Or we have to depend upon countries/Governments to announce Minimum Wage to sustain our basic needs? Uncertainties are looming as the world is coming closer due to technology but emotionally going far. It’s sad to see children, colleagues communicating via emails and messages in the same home and office. Human is losing its touch and feel. Repercussion to resists of learning, unlearning and relearning can bring down choices to none in the long run. Delay in adapting to change can be increasingly expensive as one can lose their place in a world earlier than one think. From 1992, where fewer people used to have facility of internet around , People used to stay in jobs for life but same people are now not wanted in the jobs when they go for interview as they lack in experience just because they have been doing what they were doing in one job without exposing themselves to the world’s new requirement of learn , re learn and unlearn. Chances of this group, getting a job will be negative. World has thrown different types of challenges to people, community, jobs, businesses , those people used to be applauded for remaining On one job for life ,same group of people are looked differently by corporate firms as redundant due to technology. So should people keep changing jobs after few years to just get on to learn, re learn and unlearn or continue waiting for their existing companies to face challenges and go off from the market? Only time and technology will determine what is store for human race next. According to some of the studies, its shown the longer the delay in adopting technology for any given nation, the lower the per capita income of that nation. It shows extreme reliance on Technology but can all of us adopt to the technology at the same rate as its been introduced to us? Can our children or upcoming next generations adopt technology at same scale? Or future is Either Technology or nothing, in Short Job or Jobless there is no in between option? Stephen Goldsmith, director of the Innovations in Government Program and Data-Smart City Solutions at the John F. Kennedy School of Government at Harvard University, said that in some areas, technological advancements have exceeded expectations made in 2000. The Internet also has exploded beyond expectations. From 2000 to 2010, the number of Internet users increased 500 percent, from 361 million worldwide to almost 2 billion. Now, close to 4 billion people throughout the world use the Internet. People go online for everything from buying groceries and clothes to finding a date. They can register their cars online, earn a college degree, shop for houses and apply for a mortgage but again same question is arising , Can each one of us at the same scale use or advance their skill to use technology or we are leaving our senior generations behind and making them cripple in today’s society? Or How about Mid age people who are in their 50s and soon going to take over senior society , Can they get the job and advance their skill to meet technology demands or learn, unlearn and re learn or Not only pandemic but even Technology is going to make human redundant before their actual retirement and their knowledge, skill obsolete. There should be a way forward to achieve balance, absolute reliance on Technology is not only cyber threat to governments but in long term, Unemployment, Creating Jobs or paying minimum wage to unemployed mass will be a huge worry. At the end of the day, humans need basic and then luxury. Technology can bring ease of doing business, connecting businesses and out flows, connecting Wholesalers to end users but in between many jobs, heads will be slashed down and impact will be dire. Therefore Humans have to get themselves prepared to learn, unlearn and re learn to meet today’s technology requirement or prepare themselves for early retirement.

Read More

A Tale of Two Data-Centric Services

Article | February 19, 2020

The acronym DMaaS can refer to two related but separate things: data center management-as-a-service referred to here by its other acronym, DCMaaS and data management-as-a-service. The former looks at infrastructure-level questions such as optimization of data flows in a cloud service, the latter refers to master data management and data preparation as applied to federated cloud services.DCMaaS has been under development for some years; DMaaS is slightly younger and is a product of the growing interest in machine learning and big data analytics, along with increasing concern over privacy, security, and compliance in a cloud environment.DMaaS responds to a developing concern over data quality in machine learning due to the large amount of data that must be used for training and the inherent dangers posed by divergence in data structure from multiple sources. To use the rapidly growing array of cloud data, including public cloud information and corporate internal information from hybrid clouds, you must aggregate data in a normalized way so it can be available for model training and processing with ML algorithms. As data volumes and data diversity increase, this becomes increasingly difficult.

Read More

Spotlight

Decisive Data

Decisive Data is an end-to-end business analytics consultancy and thought leader. We deliver repeatable, actionable insight for making critical business decisions informed by data and science. We’re the best in the world at data visualization – our visualization methodology is widely recognized by industry leaders. We offer a full data analytics stack; data science, predictive analytics, data management, and application development.

Events