Ryne Sherman on Big Data

| January 8, 2019

article image
Ryne Sherman, Hogan's Chief Science Officer, discusses Big Data and how it has been part of Hogan's core business long before Big Data got its name. Simply put, you can have nearly infinite data points, but it's useless if you don't know what to do with it.

Spotlight

C3 IoT

C3 IoT provides a next-generation enterprise platform (PaaS) for the rapid design, development, and deployment of large-scale big data, AI, and IoT applications. By leveraging telemetry, elastic cloud computing, analytics, and machine learning, C3 IoT brings the power of predictive insights to any business value chain. C3 IoT also provides a family of turn-key SaaS IoT applications including predictive maintenance, fraud detection, sensor network health, supply chain optimization, and customer engagement. Large enterprise customers use C3 IoT’s prebuilt SaaS applications or build custom applications using PaaS, with a combined total of 100 million sensors under management of the C3 IoT Platform today.

OTHER ARTICLES

Will Quantum Computers Make Supercomputers Obsolete in the Field of High Performance Computing?

Article | May 12, 2021

If you want an explicit answer without having to know the extra details, then here it is: Yes, there is a possibility that quantum computers can replace supercomputers in the field of high performance computing, under certain conditions. Now, if you want to know how and why this scenario is a possibility and what those conditions are, I’d encourage you to peruse the rest of this article. To start, we will run through some very simple definitions. Definitions If you work in the IT sector, you probably would have heard of the terms ‘high performance computing’, ‘supercomputers’ and ‘quantum computers’ many times. These words are thrown around quite often nowadays, especially in the area of data science and artificial intelligence. Perhaps you would have deduced their meanings from their context of use, but you may not have gotten the opportunity to explicitly sit down and do the required research on what they are and why they are used. Therefore, it is a good idea to go through their definitions, so that you have a better understanding of each concept. High Performance Computing: It is the process of carrying out complex calculations and computations on data at a very high speed. It is much faster than regular computing. Supercomputer: It is a type of computer that is used to efficiently perform powerful and quick computations. Quantum Computing: It is a type of computer that makes use of quantum mechanics’ concepts like entanglement and superposition, in order to carry out powerful computations. Now that you’ve gotten the gist of these concepts, let’s dive in a little more to get a wider scope of how they are implemented throughout the world. Background High performance computing is a thriving area in the sector of information technology, and rightly so, due to the rapid surge in the amount of data that is produced, stored, and processed every second. Over the last few decades, data has become increasingly significant to large corporations, small businesses, and individuals, as a result of its tremendous potential in their growth and profit. By properly analysing data, it is possible to make beneficial predictions and determine optimal strategies. The challenge is that there are huge amounts of data being generated every day. If traditional computers are used to manage and compute all of this data, the outcome would take an irrationally long time to be produced. Massive amounts of resources like time, computational power, and expenses would also be required in order to effectuate such computations. Supercomputers were therefore introduced into the field of technology to tackle this issue. These computers facilitate the computation of huge quantities of data at much higher speeds than a regular computer. They are a great investment for businesses that require data to be processed often and in large amounts at a time. The main advantage of supercomputers is that they can do what regular computers need to do, but much more quickly and efficiently. They have an overall high level of performance. Till date, they have been applied in the following domains: • Nuclear Weapon Design • Cryptography • Medical Diagnosis • Weather Forecasting • Online Gaming • Study of Subatomic Particles • Tackling the COVID-19 Pandemic Quantum computers, on the other hand, use a completely different principle when functioning. Unlike regular computers that use bits as the smallest units of data, quantum computers generate and manipulate ‘qubits’ or ‘quantum bits’, which are subatomic particles like electrons or photons. These qubits have two interesting quantum properties which allow them to powerfully compute data – • Superposition: Qubits, like regular computer bits, can be in a state of 1 or 0. However, they also have the ability to be in both states of 1 and 0 simultaneously. This combined state allows quantum computers to calculate a large number of possible outcomes, all at once. When the final outcome is determined, the qubits fall back into a state of either 1 or 0. This property iscalled superposition. • Entanglement: Pairs of qubits can exist in such a way that two members of a pair of qubits exist in a single quantum state. In such a situation, changing the state of one of the qubits can instantly change the state of the other qubit. This property is called entanglement. Their most promising applications so far include: • Cybersecurity • Cryptography • Drug Designing • Financial Modelling • Weather Forecasting • Artificial Intelligence • Workforce Management Despite their distinct features, both supercomputers and quantum computers are immensely capable of providing users with strong computing facilities. The question is, how do we know which type of system would be the best for high performance computing? A Comparison High performance computing requires robust machines that can deal with large amounts of data - This involves the collection, storage, manipulation, computation, and exchange of data in order to derive insights that are beneficial to the user. Supercomputers have successfully been used so far for such operations. When the concept of a quantum computer first came about, it caused quite a revolution within the scientific community. People recognised its innumerable and widespread abilities, and began working on ways to convert this theoretical innovation into a realistic breakthrough. What makes a quantum computer so different from a supercomputer? Let’s have a look at Table 1.1 below. From the table, we can draw the following conclusions about supercomputers and quantum computers - 1. Supercomputers have been around for a longer duration of time, and are therefore more advanced. Quantum computers are relatively new and still require a great depth of research to sufficiently comprehend their working and develop a sustainable system. 2. Supercomputers are easier to provide inputs to, while quantum computers need a different input mechanism. 3. Supercomputers are fast, but quantum computers are much faster. 4. Supercomputers and quantum computers have some similar applications. 5. Quantum computers can be perceived as extremely powerful and highly advanced supercomputers. Thus, we find that while supercomputers surpass quantum computers in terms of development and span of existence, quantum computers are comparatively much better in terms of capability and performance. The Verdict We have seen what supercomputers and quantum computers are, and how they can be applied in real-world scenarios, particularly in the field of high performance computing. We have also gone through their differences and made significant observations in this regard. We find that although supercomputers have been working great so far, and they continue to provide substantial provisions to researchers, organisations, and individuals who require intense computational power for the quick processing of enormous amounts of data, quantum computers have the potential to perform much better and provide faster and much more adequate results. Thus, quantum computers can potentially make supercomputers obsolete, especially in the field of high performance computing, if and only if researchers are able to come up with a way to make the development, deployment, and maintenance of these computers scalable, feasible, and optimal for consumers.

Read More

Self-supervised learning The plan to make deep learning data-efficient

Article | March 23, 2020

Despite the huge contributions of deep learning to the field of artificial intelligence, there’s something very wrong with it: It requires huge amounts of data. This is one thing that both the pioneers and critics of deep learning agree on. In fact, deep learning didn’t emerge as the leading AI technique until a few years ago because of the limited availability of useful data and the shortage of computing power to process that data.Reducing the data-dependency of deep learning is currently among the top priorities of AI researchers.

Read More

Taking a qualitative approach to a data-driven market

Article | February 18, 2021

While digital transformation is proving to have many benefits for businesses, what is perhaps the most significant, is the vast amount of data there is available. And now, with an increasing number of businesses turning their focus to online, there is even more to be collected on competitors and markets than ever before. Having all this information to hand may seem like any business owner’s dream, as they can now make insightful and informed commercial decisions based on what others are doing, what customers want and where markets are heading. But according to Nate Burke, CEO of Diginius, a propriety software and solutions provider for ecommerce businesses, data should not be all a company relies upon when making important decisions. Instead, there is a line to be drawn on where data is required and where human expertise and judgement can provide greater value. Undeniably, the power of data is unmatched. With an abundance of data collection opportunities available online, and with an increasing number of businesses taking them, the potential and value of such information is richer than ever before. And businesses are benefiting. Particularly where data concerns customer behaviour and market patterns. For instance, over the recent Christmas period, data was clearly suggesting a preference for ecommerce, with marketplaces such as Amazon leading the way due to greater convenience and price advantages. Businesses that recognised and understood the trend could better prepare for the digital shopping season, placing greater emphasis on their online marketing tactics to encourage purchases and allocating resources to ensure product availability and on-time delivery. While on the other hand, businesses who ignored, or simply did not utilise the information available to them, would have been left with overstocked shops and now, out of season items that would have to be heavily discounted or worse, disposed of. Similarly, search and sales data can be used to understand changing consumer needs, and consequently, what items businesses should be ordering, manufacturing, marketing and selling for the best returns. For instance, understandably, in 2020, DIY was at its peak, with increases in searches for “DIY facemasks”, “DIY decking” and “DIY garden ideas”. For those who had recognised the trend early on, they had the chance to shift their offerings and marketing in accordance, in turn really reaping the rewards. So, paying attention to data certainly does pay off. And thanks to smarter and more sophisticated ways of collecting data online, such as cookies, and through AI and machine learning technologies, the value and use of such information is only likely to increase. The future, therefore, looks bright. But even with all this potential at our fingertips, there are a number of issues businesses may face if their approach relies entirely on a data and insight-driven approach. Just like disregarding its power and potential can be damaging, so can using it as the sole basis upon which important decisions are based. Human error While the value of data for understanding the market and consumer patterns is undeniable, its value is only as rich as the quality of data being inputted. So, if businesses are collecting and analysing their data on their own activity, and then using this to draw meaningful insight, there should be strong focus on the data gathering phase, with attention given to what needs to be collected, why it should be collected, how it will be collected, and whether in fact this is an accurate representation of what it is you are trying to monitor or measure. Human error can become an issue when this is done by individuals or teams who do not completely understand the numbers and patterns they are seeing. There is also an obstacle presented when there are various channels and platforms which are generating leads or sales for the business. In this case, any omission can skew results and provide an inaccurate picture. So, when used in decision making, there is the possibility of ineffective and unsuccessful changes. But while data gathering becomes more and more autonomous, the possibility of human error is lessened. Although, this may add fuel to the next issue. Drawing a line The benefits of data and insights are clear, particularly as the tasks of collection and analysis become less of a burden for businesses and their people thanks to automation and AI advancements. But due to how effortless data collection and analysis is becoming, we can only expect more businesses to be doing it, meaning its ability to offer each individual company something unique is also being lessened. So, businesses need to look elsewhere for their edge. And interestingly, this is where a line should be drawn and human judgement should be used in order to set them apart from the competition and differentiate from what everyone else is doing. It makes perfect sense when you think about it. Your business is unique for a number of reasons, but mainly because of the brand, its values, reputation and perceptions of the services you are upheld by. And it’s usually these aspects that encourage consumers to choose your business rather than a competitor. But often, these intangible aspects are much more difficult to measure and monitor through data collection and analysis, especially in the autonomous, number-driven format that many platforms utilise. Here then, there is a great case for businesses to use their own judgements, expertise and experiences to determine what works well and what does not. For instance, you can begin to determine consumer perceptions towards a change in your product or services, which quantitative data may not be able to pick up until much later when sales figures begin to rise or fall. And while the data will eventually pick it up, it might not necessarily be able to help you decide on what an appropriate alternative solution may be, should the latter occur. Human judgement, however, can listen to and understand qualitative feedback and consumer sentiments which can often provide much more meaningful insights for businesses to base their decisions on. So, when it comes to competitor analysis, using insights generated from figure-based data sets and performance metrics is key to ensuring you are doing the same as the competition. But if you are looking to get ahead, you may want to consider taking a human approach too.

Read More

Big Data Could Undermine the Covid-19 Response

Article | April 13, 2020

THE CORONAVIRUS PANDEMIC has spurred interest in big data to track the spread of the fast-moving pathogen and to plan disease prevention efforts. But the urgent need to contain the outbreak shouldn’t cloud thinking about big data’s potential to do more harm than good.Companies and governments worldwide are tapping the location data of millions of internet and mobile phone users for clues about how the virus spreads and whether social distancing measures are working. Unlike surveillance measures that track the movements of particular individuals, these efforts analyze large data sets to uncover patterns in people’s movements and behavior over the course of the pandemic.

Read More

Spotlight

C3 IoT

C3 IoT provides a next-generation enterprise platform (PaaS) for the rapid design, development, and deployment of large-scale big data, AI, and IoT applications. By leveraging telemetry, elastic cloud computing, analytics, and machine learning, C3 IoT brings the power of predictive insights to any business value chain. C3 IoT also provides a family of turn-key SaaS IoT applications including predictive maintenance, fraud detection, sensor network health, supply chain optimization, and customer engagement. Large enterprise customers use C3 IoT’s prebuilt SaaS applications or build custom applications using PaaS, with a combined total of 100 million sensors under management of the C3 IoT Platform today.

Events