PLAN FOR SUCCESS WITH HIGH-STAKES DATA PROJECTS

|

article image
Throw away your long-held traditions. The pursuit of your enterprise goals doesn’t have to end as analysts predict—as one of many free falls into a growing pool of IT project failures. You can leverage all of your data to reach your objectives with less time and expense than you might imagine. The key is to start with the end in mind by undertaking your next high-stakes data project armed with the right technology and mindset to succeed. Change is good

Spotlight

WGSigma Systems, Inc.

WGSigma Systems provides Sigma, a Big Data Application Server. Sigma supports the creation of cognitive applications which operates using a NoSQL topology foundation to increase information value for business real time use cases.With business solutions like Sigma iContractRisk (Construction Contract Risk) and Sigma iWorkplace (IT Service Management), Sigma utilizes a unique method of data context injection to enrich data during the information life cycle. Sigma also provides the underlying infrastructure to support applications to be executed against the enriched information for semantic insight and support of business real time interactions.

OTHER ARTICLES

Is Augmented Analytics the Future of Big Data Analytics?

Article | August 3, 2021

We currently live in the age of data. It’s not just any kind of data, but big data. The current data sets have become huge, complicated, and quick, making it difficult for traditional business intelligence (BI) solutions to handle. These dated BI solutions are either unable to get the data, deal with the data, or understand the data. It is vital to handle the data aptly since data is everywhere and is being produced constantly. Your organization needs to discover any hidden insights in your datasets. Going through all the data will be doable with the right tools like machine learning (ML) and augmented analytics. According to Gartner, augmented analytics is the future of data analytics and defines it as: “Augmented analytics uses machine learning/artificial intelligence (ML/AI) techniques to automate data preparation, insight discovery, and sharing. It also automates data science and ML model development, management, and deployment.” Augmented analytics is different from BI tools because ML technologies work behind the scenes continuously to learn and enhance results. Augmented analytics facilitates this process faster to derive insights from large amounts of structured and unstructured data to gain ML-based recommendations. In addition, it helps to find patterns in the data that usually go unnoticed, removes human bias, and allows predictive capabilities to inform an organization of what to do next. Artificial intelligence has brought about an augmented analytics trend, and there has been a significant increase in the demand for augmented analytics. Benefits of Augmented Analytics Organizations now understand the benefits of augmented analytics which has led them to adopt it to deal with the increasing volume of structured and unstructured data. Oracle identified top four reasons organizations are opting for augmented analytics: Data Democratization Augmented data science availability to everyone has become a possibility thanks to augmented analytics. Augmented analytics solutions come prebuilt with models and algorithms, so data scientists are not needed to do this work. In addition, these augmented analytics models have user-friendly interfaces, making it easier for business users and executives to use them. Quicker Decision-making You will receive suggestions and recommendations through augmented analytics about which datasets to incorporate in analyses, alert users with dataset upgrades, and recommend new datasets when the results are not what the users expect. With just one click, augmented analytics provides precise forecasts and predictions on historical data. Programmed Recommendations Natural language processing (NLP) is featured on the augmented analytics platforms enabling non-technical users to question the source data easily. Interpreting the complex data into text with intelligent recommendations is automated by natural language generation (NLG), thus speeding up the analytic insights. Anyone using the tools can find out hidden patterns and predict trends to optimize the time it takes to go from data to insights to decisions using automated recommendations for data improvement and visualization. Non-expert users can use NLP technology to make sense of large amounts of data. Users can ask doubts about data using typical business terms. The software will find and question the correct data, making the results easy to digest using visualization tools or natural language output. Grow into a Data-driven Company It is more significant to understand data and business while organizations are rapidly adjusting to changes. Analytics has become more critical to doing everything from understanding sales trends, to segment customers, based on their online behaviors, and predicting how much inventory to hold to strategizing marketing campaigns. Analytics is what makes data a valuable asset. Essential Capabilities of Augmented Analytics Augmented analytics reduces the repetitive processes data analysts need to do every time they work with new datasets. It helps to decrease the time it takes to clean data through the ETL process. Augmented analytics allows more time to think about the data implications, discover patterns, auto-generated code, create visualizations, and propose recommendations from the insights it derives. Augmented analytics considers intents and behaviors and turns them into contextual insights. It presents new directions to look at data and identify patterns and insights companies would have otherwise missed out on completely- thus altering the way analytics is used. The ability to highlight the most relevant hidden insights is a powerful capability. Augmented analytics, for example, can help users manage the context at the explanatory process stage. It understands the values of data that are associated with or unrelated to that context, which results in powerful and relevant suggestions that are context-aware. Modern self-service BI tools have a friendly user interface that enables business users with low to no technical skills to derive insights from data in real-time. In addition, these tools can easily handle large datasets from various sources in a quickly and competently. The insights from augmented analytics tools can tell you what, why, and how something happened. In addition, it can reveal important insights, recommendations, and relationships between data points in real-time and present it to the user in the form of reports in conversational language. Users can have data queries to get insights through the augmented analytics tools. For example, business users can ask, “How was the company’s performance last year?” or “What was the most profitable quarter of the year?” The systems provide in-depth explanations and recommendations around data insights, clearly understanding the “what” and the “why” of the data. It enhances efficiency, decision-making, and collaboration between users and encourages data literacy and data democracy throughout an organization. Augmented Analytics: What’s Next? Augmented analytics is going to change the way people understand and examine data. It has become a necessity for businesses to survive. It will simplify and speed up the augmented data preparation, cleansing, and standardization of data, thus assist businesses to focus all their efforts on data analysis. BI and analytics will become an immersive environment with integrations allowing users to interact with their data. New insights and data will be easier to access through various devices and interfaces like mobile phones, virtual assistants, or chatbots. In addition, it will help decision-making by notifying the users of alerts that need immediate attention. This will help businesses to stay updated about any changes happening in real-time. Frequently Asked Questions What are the benefits of augmented analytics? Augmented analytics helps companies become more agile, gain access to analytics, helps users make better, faster, and data-driven decisions, and reduces costs. How important is augmented analytics? Augmented analytics build efficiency into the data analysis process, equips businesses and people with tools that can answer data-based questions within seconds, and assist companies in getting ahead of their competitors. What are the examples of augmented analytics? Augmented analytics can help retain existing customers, capitalize on customer needs, drive revenue through optimized pricing, and optimize operations in the healthcare sector for better patient outcomes. These are some of the examples of the use of augmented analytics. { "@context": "https://schema.org", "@type": "FAQPage", "mainEntity": [{ "@type": "Question", "name": "What are the benefits of augmented analytics?", "acceptedAnswer": { "@type": "Answer", "text": "Augmented analytics helps companies become more agile, gain access to analytics, helps users make better, faster, and data-driven decisions, and reduces costs." } },{ "@type": "Question", "name": "How important is augmented analytics?", "acceptedAnswer": { "@type": "Answer", "text": "Augmented analytics build efficiency into the data analysis process, equips businesses and people with tools that can answer data-based questions within seconds, and assist companies in getting ahead of their competitors." } },{ "@type": "Question", "name": "What are the examples of augmented analytics?", "acceptedAnswer": { "@type": "Answer", "text": "Augmented analytics can help retain existing customers, capitalize on customer needs, drive revenue through optimized pricing, and optimize operations in the healthcare sector for better patient outcomes. These are some of the examples of the use of augmented analytics." } }] }

Read More

Natural Language Desiderata: Understanding, explaining and interpreting a model.

Article | August 3, 2021

Clear conceptualization, taxonomies, categories, criteria, properties when solving complex real-life contextualized problems is non-negotiable, a “must” to unveil the hidden potential of NPL impacting on the transparency of a model. It is common knowledge that many authors and researchers in the field of natural language processing (NLP) and machine learning (ML) are prone to use explainability and interpretability interchangeably, which from the start constitutes a fallacy. They do not mean the same, even when looking for a definition from different perspectives. A formal definition of what explanation, explainable, explainability mean can be traced to social science, psychology, hermeneutics, philosophy, physics and biology. In The Nature of Explanation, Craik (1967:7) states that “explanations are not purely subjective things; they win general approval or have to be withdrawn in the face of evidence or criticism.” Moreover, the power of explanation means the power of insight and anticipation and why one explanation is satisfactory involves a prior question why any explanation at all should be satisfactory or in machine learning terminology how a model is performant in different contextual situations. Besides its utilitarian value, that impulse to resolve a problem whether or not (in the end) there is a practical application and which will be verified or disapproved in the course of time, explanations should be “meaningful”. We come across explanations every day. Perhaps the most common are reason-giving ones. Before advancing in the realm of ExNLP, it is crucial to conceptualize what constitutes an explanation. Miller (2017) considered explanations as “social interactions between the explainer and explainee”, therefore the social context has a significant impact in the actual content of an explanation. Explanations in general terms, seek to answer the why type of question. There is a need for justification. According to Bengtsson (2003) “we will accept an explanation when we feel satisfied that the explanans reaches what we already hold to be true of the explanandum”, (being the explanandum a statement that describes the phenomenon to be explained (it is a description, not the phenomenon itself) and the explanan at least two sets of statements, used for the purpose of elucidating the phenomenon). In discourse theory (my approach), it is important to highlight that there is a correlation between understanding and explanation, first and foremost. Both are articulated although they belong to different paradigmatic fields. This dichotomous pair is perceived as a duality, which represents an irreducible form of intelligibility. When there are observable external facts subject to empirical validation, systematicity, subordination to hypothetic procedures then we can say that we explain. An explanation is inscribed in the analytical domain, the realm of rules, laws and structures. When we explain we display propositions and meaning. But we do not explain in a vacuum. The contextual situation permeates the content of an explanation, in other words, explanation is an epistemic activity: it can only relate things described or conceptualized in a certain way. Explanations are answers to questions in the form: why fact, which most authors agree upon. Understanding can mean a number of things in different contexts. According to Ricoeur “understanding precedes, accompanies and swathes an explanation, and an explanation analytically develops understanding.” Following this line of thought, when we understand we grasp or perceive the chain of partial senses as a whole in a single act of synthesis. Originally, belonging to the field of the so-called human science, then, understanding refers to a circular process and it is directed to the intentional unit of discourse whereas an explanation is oriented to the analytical structure of a discourse. Now, to ground any discussion on what interpretation is, it is crucial to highlight that the concept of interpretation opposes the concept of explanation. They cannot be used interchangeably. If considered as a unit, they composed what is called une combinaison éprouvé (a contrasted dichotomy). Besides, in dissecting both definitions we will see that the agent that performs the explanation differs from the one that produce the interpretation. At present there is a challenge of defining—and evaluating—what constitutes a quality interpretation. Linguistically speaking, “interpretation” is the complete process that encompasses understanding and explanation. It is true that there is more than one way to interprete an explanation (and then, an explanation of a prediction) but it is also true that there is a limited number of possible explanations if not a unique one since they are contextualized. And it is also true that an interpretation must not only be plausible, but more plausible than another interpretation. Of course there are certain criteria to solve this conflict. And to prove that an interpretation is more plausible based on an explanation or the knowledge could be related to the logic of validation rather than to the logic of subjective probability. Narrowing it down How are these concepts transferred from theory to praxis? What is the importance of the "interpretability" of an explainable model? What do we call a "good" explainable model? What constitutes a "good explanation"? These are some of the many questions that researchers from both academia and industry are still trying to answer. In the realm on machine learning current approaches conceptualize interpretation in a rather ad-hoc manner, motivated by practical use cases and applications. Some suggest model interpretability as a remedy, but only a few are able to articulate precisely what interpretability means or why it is important. Hence more, most in the research community and industry use this term as synonym of explainability, which is certainly not. They are not overlapping terms. Needless to say, in most cases technical descriptions of interpretable models are diverse and occasionally discordant. A model is better interpretable than another model if its decisions are easier for a human to comprehend than decisions from the other model (Molnar, 2021). For a model to be interpretable (being interpretable the quality of the model), the information conferred by an interpretation may be useful. Thus, one purpose of interpretations may be to convey useful information of any kind. In Molnar’s words the higher the interpretability of a machine learning model, the easier it is for someone to comprehend why certain decisions or predictions have been made.” I will make an observation here and add “the higher the interpretability of an explainable machine learning model”. Luo et. al. (2021) defines “interpretability as ‘the ability [of a model] to explain or to present [its predictions] in understandable terms to a human.” Notice that in this definition the author includes “understanding” as part of the definition, giving the idea of completeness. Thus, the triadic closure explanation-understanding-interpretation is fulfilled, in which the explainer and interpretant (the agents) belong to different instances and where interpretation allows the extraction and formation of additional knowledge captured by the explainable model. Now are the models inherently interpretable? Well, it is more a matter of selecting the methods of achieving interpretability: by (a) interpreting existing models via post-hoc techniques, or (b) designing inherently interpretable models, which claim to provide more faithful interpretations than post-hoc interpretation of blackbox models. The difference also lies in the agency –like I said before– , and how in one case interpretation may affect the explanation process, that is model’s inner working or just include natural language explanations of learned representations or models.

Read More

How big data can help the homeless

Article | August 3, 2021

Homeless policy needs to join the big data revolution. A data tsunami is transforming our world. Ninety percent of existing data was created in the last two years, and Silicon Valley is leveraging it with powerful analytics to create self-driving cars and to revolutionize business decision-making in ways that drive innovation and efficiency.Unfortunately, this revolution has yet to help the homeless. It is not due to a lack of data. Sacramento alone maintains data on half a million service interactions with more than 65,000 homeless individuals. California is considering integrating the data from its 44 continuums of care to create a richer pool of data. Additionally, researchers are uncovering troves of relevant information in educational and social service databases.These data, however, are only useful if they are aggressively mined for insights, looking for problems to solve and successful practices to replicate. At that juncture California falls short.

Read More
THEORY AND STRATEGIES

Can you really trust Amazon Product Recommendation?

Article | August 3, 2021

Since the internet became popular, the way we purchase things has evolved from a simple process to a more complicated process. Unlike traditional shopping, it is not possible to experience the products first-hand when purchasing online. Not only this, but there are more options or variants in a single product than ever before, which makes it more challenging to decide. To not make a bad investment, the consumer has to rely heavily on the customer reviews posted by people who are using the product. However, sorting through relevant reviews at multiple eCommerce platforms of different products and then comparing them to choose can work too much. To provide a solution to this problem, Amazon has come up with sentiment analysis using product review data. Amazon performs sentiment analysis on product review data with Artificial Intelligence technology to develop the best suitable products for the customer. This technology enables Amazon to create products that are most likely to be ideal for the customer. A consumer wants to search for only relevant and useful reviews when deciding on a product. A rating system is an excellent way to determine the quality and efficiency of a product. However, it still cannot provide complete information about the product as ratings can be biased. Textual detailed reviews are necessary to improve the consumer experience and in helping them make informed choices. Consumer experience is a vital tool to understand the customer's behavior and increase sales. Amazon has come up with a unique way to make things easier for their customers. They do not promote products that look similar to the other customer's search history. Instead, they recommend products that are similar to the product a user is searching for. This way, they guide the customer using the correlation between the products. To understand this concept better, we must understand how Amazon's recommendation algorithm has upgraded with time. The history of Amazon's recommendation algorithm Before Amazon started a sentiment analysis of customer product reviews using machine learning, they used the same collaborative filtering to make recommendations. Collaborative filtering is the most used way to recommend products online. Earlier, people used user-based collaborative filtering, which was not suitable as there were many uncounted factors. Researchers at Amazon came up with a better way to recommend products that depend on the correlation between products instead of similarities between customers. In user-based collaborative filtering, a customer would be shown recommendations based on people's purchase history with similar search history. In item-to-item collaborative filtering, people are shown recommendations of similar products to their recent purchase history. For example, if a person bought a mobile phone, he will be shown hints of that phone's accessories. Amazon's Personalization team found that using purchase history at a product level can provide better recommendations. This way of filtering also offered a better computational advantage. User-based collaborative filtering requires analyzing several users that have similar shopping history. This process is time-consuming as there are several demographic factors to consider, such as location, gender, age, etc. Also, a customer's shopping history can change in a day. To keep the data relevant, you would have to update the index storing the shopping history daily. However, item-to-item collaborative filtering is easy to maintain as only a tiny subset of the website's customers purchase a specific product. Computing a list of individuals who bought a particular item is much easier than analyzing all the site's customers for similar shopping history. However, there is a proper science between calculating the relatedness of a product. You cannot merely count the number of times a person bought two items together, as that would not make accurate recommendations. Amazon research uses a relatedness metric to come up with recommendations. If a person purchased an item X, then the item Y will only be related to the person if purchasers of item X are more likely to buy item Y. If users who purchased the item X are more likely to purchase the item Y, then only it is considered to be an accurate recommendation. Conclusion In order to provide a good recommendation to a customer, you must show products that have a higher chance of being relevant. There are countless products on Amazon's marketplace, and the customer will not go through several of them to figure out the best one. Eventually, the customer will become frustrated with thousands of options and choose to try a different platform. So Amazon has to develop a unique and efficient way to recommend the products that work better than its competition. User-based collaborative filtering was working fine until the competition increased. As the product listing has increased in the marketplace, you cannot merely rely on previous working algorithms. There are more filters and factors to consider than there were before. Item-to-item collaborative filtering is much more efficient as it automatically filters out products that are likely to be purchased. This limits the factors that require analysis to provide useful recommendations. Amazon has grown into the biggest marketplace in the industry as customers trust and rely on its service. They frequently make changes to fit the recent trends and provide the best customer experience possible.

Read More

Spotlight

WGSigma Systems, Inc.

WGSigma Systems provides Sigma, a Big Data Application Server. Sigma supports the creation of cognitive applications which operates using a NoSQL topology foundation to increase information value for business real time use cases.With business solutions like Sigma iContractRisk (Construction Contract Risk) and Sigma iWorkplace (IT Service Management), Sigma utilizes a unique method of data context injection to enrich data during the information life cycle. Sigma also provides the underlying infrastructure to support applications to be executed against the enriched information for semantic insight and support of business real time interactions.

Events