Leonid Bershidsky: Big data is still only a little helpful

| June 3, 2016

article image
Big data” is one of the tech world’s ubiquitous buzzwords. In the old days, people just called it data, but in Silicon Valley it’s not a thing unless it’s big. It’s not yet obvious, however, that data collected by various internet services is any more useful than those mined in more traditional ways —through surveys, for example. As consumers, we know the data collected by big internet companies can be used to push goods and services — mostly things very much like those we have already purchased…

Spotlight

PushCV

PushCV is the largest pool of pre-screened candidates in Africa. We provide a platform where only the best talents are connected to top employers and recruiters.

OTHER ARTICLES
DATA SCIENCE

Soft Skills in Data Science

Article | April 29, 2021

We live in a world convulsed by new technologies and we are witnessing how more and more processes are automated in order to be executed with the same skill or even with better results than if they were carried out by a human, all this in order to be more efficient and effective. In this context the world of work is becoming increasingly competitive, because to remain employable we need to learn to manage or find a way to adapt our knowledge and skills to new technologies. With the spread of e-learning platforms and the tutorials that we can find available on the internet, acquiring new knowledge is within everyone's reach. For this reason, it is necessary to differentiate ourselves in order to stand out from other professionals, who have the hard skills similar to ours and this is precisely where Soft Skills play a very important role. What are Soft Skills? Soft skills are actually a combination of individual social skills, communication skills, personality traits, attitudes, social intelligence and emotional intelligence. Which facilitate relationships with others, making us more effective when interacting with other people. We could say that Soft Skills are the human interface that allow us to adapt to different working environments and industries. They are powerful tools for personal and professional growth. Why are Soft Skills key in our professional growth? Nowadays, standing out in the world of work is getting increasingly difficult, regardless of whether you are part of a corporation or work independently, due to the great competition within the labor market. That is why we must develop certain skills and attitudes that help us to function properly and successfully meet professional demands. Soft Skills are the point of differentiation that allows us to be selected for a position. The reason is very simple, we could be applying for a position and competing with people that are equal or even more qualified than us at a technical level, but to achieve the collaborative objectives of the company, more is required than just the technical and rational part. Also the way of communicating, values, ethics, as well as personality traits are highly valued factors since they help to drive organizations through high-performance teams, guaranteeing the achievement of their objectives. The background of the Soft Skills that we have trained throughout our lives make us unique, because it is unlikely that two people have the same combination of Soft Skills and been trained in a similar way, and that makes us more competitive against certain job opportunities where perhaps many will have the same Hard Skills, but where our Soft Skills will be the ones that will make us stand out to continue advancing in our professional career. How to sharpen our Soft Skills? To perform in any job we necessarily need to interact with other people, even if we work independently or remotely, so we must have the necessary skills that allow us to connect successfully with our teammates and stakeholders. Starting from the fact that Soft Skills are human skills, we can say that we have them pre-installed and the way to start using them (installing them) is through the experiences we undergo every day. Imagine being able to communicate assertively in your work environment and in your personal life. Master the use of tools installed in you to improve your interpersonal relationships within your work teams and reduce conflict. This would allow you to foster a healthy working environment and be able to lead any team in any environment in a strategic and effective way. Think of Soft Skills as a set of Apps that are ready to be used (like a toolbox) and that according to the experiences that are presented in our personal and / or professional lives, we are going to choose to use these applications to achieve our goals. Every time we access one of these applications, we are giving it the opportunity to collect data that will allow it to personalize its insights according to our needs and to fine-tune its effectiveness each time we use it. One of the best ways to train our Soft Skills is by leaving our comfort zone, because that will allow us to 'install' more and more Soft Skills. Another way to refine our Soft Skills is by participating in activities that involve people we do not know and even better if we involve people from other cultures, because we will achieve a beneficial exchange of experiences and knowledge for both parties that will enrich and make the training of our Soft Skills even more valuable. Some examples of activities that will enhance your Soft Skills: • Participate in competitions (e.g. Hackathons) • Found or be a lead of a community that shares your interests, and organizes small or large projects. • Organize a study group aimed at carrying out a technical or business project in order to confront professionals from various fields or industries. • Find resources and experts to help you. There are Soft Skills trainers who know useful techniques and tips to develop/sharpen your skills. • Participate in volunteer activities. You will meet new people with whom to put your Soft Skills in action. These activities will train/sharpen your leadership skills, teamwork, delegation, interpersonal communication, persuasion, etc. These are skills that we do not have as much facility to train while we are students or when we have just started working after finishing our studies, and that are required in the labor market to continue climbing in our professional career. Why do Soft Skills matter in the Data Science universe? A consequence of the use of Artificial Intelligence and Data Science is that many of the jobs that we know today will be automated and this is a matter of concern for many professionals who see their careers are in danger, but the good news is that in the future many new jobs the Soft Skills will be the main protagonists, this is what John Thompson explains us in his book "Building Analytics Teams" In other words, it is precisely our human skills that will allow us to be more employable in the future, and they will be highly requested skills because according to what the experts envision which is, that the machines will not be able to match us in this field, and that is why training our Soft Skills becomes a priority because they will allow us to be the key players of the future. On the other hand, Data Science is an interdisciplinary field where Soft Skills such as cooperation and communication are essential to achieve the goals set. Denis Rothman, author of the book "Transformers for Natural Language Processing" in an interview that I conducted, mentioned that The Human Quality is the most important thing for him when choosing the members of his work team. These are some considerations to take into account to generate a culture of cooperation: • People work harder and need less supervision, when they themselves control their work and have more freedom to choose how to do it. When they work as a team, they show greater motivation, their sense of pride increases and productivity reaches higher levels. • Solid teams that seek quality and excellence correct themselves; that is, they identify problems and correct them very quickly. Thus, they gain work experience and increase their performance. • Forming a solid and efficient work team requires patience. You need to give them time to see your results. They will have to establish procedures to complete tasks, handle administrative functions and work together efficiently, they will even have to adapt to their own decisions and accept their consequences. • A manager or team leader must recognize the team building process without expecting immediate results. The group will have to go through a learning process and this will take longer in some groups than in others. Another key component to achieving high levels of cooperation is fluid communication among team members and stakeholders. For instance defining the communication channels and the contact points in the different teams involved, guarantees the constant flow of communication during the life cycle of a Data Science project. One of the most critical moments is the presentation of the results to the stakeholders. In some cases the results of a project are not taken into consideration not so much because the expected results are not achieved, but because the way in which these results are presented are not meaningful for the stakeholders, and this, in most cases, it is due to the existence of communication barriers that is a consequence of the use of a language (terminologies) used in the technical world but not in the business world. After taking a tour of the world of Soft Skills, we can conclude by saying that Soft Skills are like superpowers that are waiting for the opportunity to be put into action, to make you a superhero or superheroine. Keep climbing positions in your professional career depends on you, on how much you use these superpowers but above all on your skills to refine them and make them available to the work team of which you are part. Don't wait any longer and start discovering your potential, start training your Soft Skills! If you want to know more about Soft Skills, I invite you to visit The Soft Skills Show

Read More
BIG DATA MANAGEMENT

How can machine learning detect money laundering?

Article | December 16, 2020

In this article, we will explore different techniques to detect money laundering activities. Notwithstanding, regardless of various expected applications inside the financial services sector, explicitly inside the Anti-Money Laundering (AML) appropriation of Artificial Intelligence and Machine Learning (ML) has been generally moderate. What is Money Laundering, Anti Money Laundering? Money Laundering is where someone unlawfully obtains money and moves it to cover up their crimes. Anti-Money Laundering can be characterized as an activity that forestalls or aims to forestall money laundering from occurring. It is assessed by UNO that, money-laundering exchanges account in one year is 2–5% of worldwide GDP or $800 billion — $3 trillion in USD. In 2019, regulators and governmental offices exacted fines of more than $8.14 billion. Indeed, even with these stunning numbers, gauges are that just about 1 % of unlawful worldwide money related streams are ever seized by the specialists. AML activities in banks expend an over the top measure of manpower, assets, and cash flow to deal with the process and comply with the guidelines. What are the punishments for money laundering? In 2019, Celent evaluated that spending came to $8.3 billion and $23.4 billion for technology and operations, individually. This speculation is designated toward guaranteeing anti-money laundering. As we have seen much of the time, reputational costs can likewise convey a hefty price. In 2012, HSBC laundering of an expected £5.57 billion over at least seven years.   What is the current situation of the banks applying ML to stop money laundering? Given the plenty of new instruments the banks have accessible, the potential feature risk, the measure of capital involved, and the gigantic expenses as a form of fines and punishments, this should not be the situation. A solid impact by nations to curb illicit cash movement has brought about a huge yet amazingly little part of money laundering being recognized — a triumph rate of about 2% average. Dutch banks — ABN Amro, Rabobank, ING, Triodos Bank, and Volksbank announced in September 2019 to work toward a joint transaction monitoring to stand-up fight against Money Laundering. A typical challenge in transaction monitoring, for instance, is the generation of a countless number of alerts, which thusly requires operation teams to triage and process the alarms. ML models can identify and perceive dubious conduct and besides they can classify alerts into different classes such as critical, high, medium, or low risk. Critical or High alerts may be directed to senior experts on a high need to quickly explore the issue. Today is the immense number of false positives, gauges show that the normal, of false positives being produced, is the range of 95 and 99%, and this puts extraordinary weight on banks. The examination of false positives is tedious and costs money. An ongoing report found that banks were spending near 3.01€ billion every year exploring false positives. Establishments are looking for increasing productive ways to deal with crime and, in this specific situation, Machine Learning can end up being a significant tool. Financial activities become productive, the gigantic sum and speed of money related exchanges require a viable monitoring framework that can process exchanges rapidly, ideally in real-time.   What are the types of machine learning algorithms which can identify money laundering transactions? Supervised Machine Learning, it is essential to have historical information with events precisely assigned and input variables appropriately captured. If biases or errors are left in the data without being dealt with, they will get passed on to the model, bringing about erroneous models. It is smarter to utilize Unsupervised Machine Learning to have historical data with events accurately assigned. It sees an obscure pattern and results. It recognizes suspicious activity without earlier information of exactly what a money-laundering scheme resembles. What are the different techniques to detect money laundering? K-means Sequence Miner algorithm: Entering banking transactions, at that point running frequent pattern mining algorithms and mining transactions to distinguish money laundering. Clustering transactions and dubious activities to money laundering lastly show them on a chart. Time Series Euclidean distance: Presenting a sequence matching algorithm to distinguish money laundering detection, utilizing sequential detection of suspicious transactions. This method exploits the two references to recognize dubious transactions: a history of every individual’s account and exchange data with different accounts. Bayesian networks: It makes a model of the user’s previous activities, and this model will be a measure of future customer activities. In the event that the exchange or user financial transactions have. Cluster-based local outlier factor algorithm: The money laundering detection utilizing clustering techniques combination and Outliers.   Conclusion For banks, now is the ideal opportunity to deploy ML models into their ecosystem. Despite this opportunity, increased knowledge and the number of ML implementations prompted a discussion about the feasibility of these solutions and the degree to which ML should be trusted and potentially replace human analysis and decision-making. In order to further exploit and achieve ML promise, banks need to continue to expand on its awareness of ML strengths, risks, and limitations and, most critically, to create an ethical system by which the production and use of ML can be controlled and the feasibility and effect of these emerging models proven and eventually trusted.

Read More

Big Data Is Helping Us Fight The Coronavirus But At What Cost To Our Privacy

Article | April 9, 2020

In some ways at least, technology has been able to tell us more about how and where the virus is spreading. Mostly, this has involved creatively harnessing the power of big data using temperature readings from smart thermometers to detect COVID-19 hot spots, or aggregating cellphone location data to point to the areas of the country where people are staying home. But against a backdrop of debate between civil liberties and public health, we also need to be asking where the line is digitally: How much surveillance is acceptable in the service of the greater good.

Read More

Understanding Big Data and Artificial Intelligence

Article | June 18, 2021

Data is an important asset. Data leads to innovation and organizations tend to compete for leading these innovations on a global scale. Today, every business requires data and insights to stay relevant in the market. Big Data has a huge impact on the way organizations conduct their businesses. Big Data is used in different enterprises like travel, healthcare, manufacturing, governments, and more. If they need to determine their audience, understand what clients want, forecast the needs of the customers and the clients, AI and big data analysis is vital to every decision-making scenario. When companies process the collected data accurately, they get the desired results, which leads them to their desired goals. The term Big Data has been around since the 1990s. By the time we could fully comprehend it, Big Data had already amassed a huge amount of stored data. If this data is analyzed properly, it would reveal valuable industry insights into the industry to which the data belonged. IT professionals and computer scientists realized that going through all of the data and analyzing it for the purpose was too big of a task for humans to undertake. When artificial intelligence (AI) algorithm came into the picture, it accomplished analyzing the accumulated data and deriving insights. The use of AI in Big Data is fundamental to get desired results for organizations. According to Northeastern University, the amount of data in the world was 4.4 zettabytes in 2013. By of 2020, the data rose to 44 zettabytes. When there is this amount of data produced globally, this information is invaluable to the enterprises and now can leverage AI algorithms to process it. Because of this, the companies can understand and influence customer behavior. By 2018, over 50% of countries had adopted Big Data. Let us understand what Big Data, convergence of big data and AI, and impact of AI on big data analytics. Understanding Big Data In simple words, Big Data is a term that comprises every tool and process that helps people use and manage vast sets of data. According to Gartner, Big Data is a “high-volume and/or high-variety information assets that demand cost-effective, innovative forms of information processing to enable enhanced insight, decision-making, and process automation.” The concept of Big Data was created to capture trends, preferences, and user behavior in one place called the data lake. Big Data in enterprises can help them analyze and configure their customers’ motivations and come up with new ideas for the creation of new offerings. Big Data studies different methods of extracting, analyzing, or dealing with data sets that are too complicated for traditional data processing systems. To analyze a large amount of data requires a system designed to stretch its extraction and analysis capability. Data is everywhere. This stockpile of data can give us insights and business analytics to the industry belonging to the data set. Therefore, the AI algorithms are written to benefit from large and complex data. Importance of Big Data Data is an integral part of understanding customer demographics and their motivations. When customers interact with technology in active or passive manner, these actions create a new set of data. What contributes to this data creation is what they carry with them every day - their smartphones. Their cameras, credit cards, purchased products all contribute to their growing data profile. A correctly done analysis can tell a lot about their behavior patterns, personality, and events in the customer’s life. Companies can use this information to rethink their strategies, improve on their product, and create targeted marketing campaigns, which would ultimately lead them to their target customer. Industry experts, for years and years, have discussed Big Data and its impact on businesses. Only in recent years, however, has it become possible to calculate that impact. Algorithms and software can now analyze large datasets quickly and efficiently.The forty-four zettabyte of data will only quadruple in the coming years. This collection and analysis of the data will help companies get the AI insights that will aid them in generating profits and be future-ready. Organizations have been using Big Data for a long time. Here’s how those organizations are using Big Data to drive success: Answering customer questions Using big data and analytics, companies can learn the following things: • What do customers want? • Where are they missing out on? • Who are their best and loyal customers? • Why people choose different products? Every day, as organizations gather more information, they can get more insights into sales and marketing. Once they get this data, they can optimize their campaigns to suit the customer’s needs. Learning from their online habits and with correct analysis, companies can send personalized promotional emails. These emails may prompt this target audience to convert into full-time customers. Making confident decisions As companies grow, they all need to make complex decisions. With in-depth analysis of marketplace knowledge, industry, and customers, Big Data can help you make confident choices. Big Data gives you a complete overview of everything you need to know. With the help of this, you can launch your marketing campaign or launch a new product in the market, or make a focused decision to generate the highest ROI. Once you add machine learning and AI to the mix, your Big Data collections can form a neural network to help your AI suggest useful company changes. Optimizing and Understanding Business Processes Cloud computing and machine learning help you to stay ahead by identifying opportunities in your company’s practices. Big Data analytics can tell you if your email strategy is working even when your social media marketing isn’t gaining you any following. You can also check which parts of your company culture have the right impact and result in the desired turnover. The existing evidence can help you make quick decisions and ensure you spend more of your budget on things that help your business grow. Convergence of Big Data and AI Big Data and Artificial Intelligence have a synergistic relationship. Data powers AI. The constantly evolving data sets or Big Data makes it possible for machine learning applications to learn and acquire new skills. This is what they were built to do. Big Data’s role in AI is supplying algorithms with all the essential information for developing and improving features, pattern recognition capabilities. AI and machine learning use data that has been cleansed of duplicate and unnecessary data. This clean and high-quality big data is then utilized to create and train intelligent AI algorithms, neural networks, and predictive models. AI applications rarely stop working and learning. Once the “initial training” is done (initial training is preparing already collected data), they adjust their work as and when the data changes. This makes it necessary for data to be constantly collected. When it comes to businesses using this technology, AI helps them use Big Data for analytics by making advanced tools accessible and obtainable to help users gain insights that would otherwise have been hidden in the huge amount of data. Once firms and businesses gain a hold on using AI and Big Data, they can provide decision-makers with a clear understanding of factors that affect their businesses. Impact of AI on Big Data Analytics AI supports users in the Big Data cycle, including aggregation, storage, and retrieval of diverse data types from different data sources. This includes data management, context management, decision management, action management, and risk management. Big Data can help alert problems and help find new solutions and get ideas about any new prospects. With the amount of information stream that comes in, it can be difficult to determine what is important and what isn’t. This is where AI and machine learning come in. It can help identify unusual patterns in the processes, help in the analysis, and suggest further steps to be taken. It can also learn how users interact with analytics and learn subtle differences in meanings or context-specific nuances to understand numeric data sources. AI can also caution users about anomalies, unforeseen data patterns, monitoring events, and threats from system logs or social networking data. Application of Big Data and Artificial Intelligence After establishing how AI and Big Data work together, let us look at how some applications are benefitting from their synergy: Banking and financial sectors The banking and financial sectors apply these to monitor financial marketing activities. These institutions also use AI to keep an eye on any illegal trading activities. Trading data analytics are obtained for high-frequency trading, and decision making based on trading, risk analysis, and predictive analysis. It is also used for fraud warning and detection, archival and analysis of audit trails, reporting enterprise credit, customer data transformation, etc. Healthcare AI has simplified health data prescriptions and health analysis, thus benefitting healthcare providers from the large data pool. Hospitals are using millions of collected data that allow doctors to use evidence-based medicine. Chronic diseases can be tracked faster by AI. Manufacturing and supply chain AI and Big Data in manufacturing, production management, supply chain management and analysis, and customer satisfaction techniques are flawless. The quality of products is thus much better with higher energy efficiency, reliable increase in levels, and profit increase. Governments Governments worldwide use AI applications like facial recognition, vehicle recognition for traffic management, population demographics, financial classifications, energy explorations, environmental conservation, criminal investigations, and more. Other sectors that use AI are mainly retail, entertainment, education, and more. Conclusion According to Gartner’s predictions, artificial intelligence will replace one in five workers by 2022. Firms and businesses can no longer afford to avoid using artificial intelligence and Big Data in their day-to-day. Investments in AI and Big Data analysis will be beneficial for everyone. Data sets will increase in the future, and with it, its application and investment will grow over time. Human relevance will continue to decrease as time goes by. AI enables machine learning to be the future of the development of business technologies. It will automate data analysis and find new insights that were previously impossible to imagine by processing data manually. With machine learning, AI, and Big Data, we can redraw the way we approach everything else. Frequently Asked Questions Why does big data affect artificial intelligence? Big Data and AI customize business processes and make better-suited decisions for individual needs and expectations. This improves its efficiency of processes and decisions. Data has the potential to give insights into a variety of predicted behaviors and incidents. Is AI or big data better? AI becomes better as it is fed more and more information. This information is gathered from Big Data which helps companies understand their customers better. On the other hand, Big Data is useless if there is no AI to analyze it. Humans are not capable of analyzing the data on a large scale. Is AI used in big data? When the gathered Big Data is to be analyzed, AI steps in to do the job. Big Data makes use of AI. What is the future of AI in big data? AI’s ability to work so well with data analytics is the primary reason why AI and Big Data now seem inseparable. AI machine learning and deep learning are learning from every data input and using those inputs to generate new rules for future business analytics. { "@context": "https://schema.org", "@type": "FAQPage", "mainEntity": [{ "@type": "Question", "name": "Why does big data affect artificial intelligence?", "acceptedAnswer": { "@type": "Answer", "text": "Big Data and AI customize business processes and make better-suited decisions for individual needs and expectations. This improves its efficiency of processes and decisions. Data has the potential to give insights into a variety of predicted behaviors and incidents." } },{ "@type": "Question", "name": "Is AI or big data better?", "acceptedAnswer": { "@type": "Answer", "text": "AI becomes better as it is fed more and more information. This information is gathered from Big Data which helps companies understand their customers better. On the other hand, Big Data is useless if there is no AI to analyze it. Humans are not capable of analyzing the data on a large scale." } },{ "@type": "Question", "name": "Is AI used in big data?", "acceptedAnswer": { "@type": "Answer", "text": "When the gathered Big Data is to be analyzed, AI steps in to do the job. Big Data makes use of AI." } },{ "@type": "Question", "name": "What is the future of AI in big data?", "acceptedAnswer": { "@type": "Answer", "text": "AI’s ability to work so well with data analytics is the primary reason why AI and Big Data now seem inseparable. AI machine learning and deep learning are learning from every data input and using those inputs to generate new rules for future business analytics." } }] }

Read More

Spotlight

PushCV

PushCV is the largest pool of pre-screened candidates in Africa. We provide a platform where only the best talents are connected to top employers and recruiters.

Events