10 tips for building long-running clusters using Cloud Dataproc

| February 1, 2019

article image
Google’s Cloud Dataproc is a fast, easy-to-use, fully managed cloud service for running Apache Spark and Apache Hadoop clusters in a simple, cost-efficient way. Google Cloud Platform (GCP) customers like Pandora and Outbrain depend on Cloud Dataproc to run their Hadoop and Spark jobs. A key differentiator for Cloud Dataproc is that it is optimized to create ephemeral job-scoped clusters in around 90 seconds. This speed of deployment means that a single job can have a dedicated cluster, containing just the resources needed to run the job, that is shut down upon job completion. On the Cloud Dataproc team, we’ve worked with countless customers who are creating clusters for their particular use cases. However, not all Hadoop and Spark workloads are appropriately served by an ephemeral job-scoped cluster model. Our goal on the Cloud Dataproc team is to make sure every customer’s use case can be addressed. To that end, we’re excited to share these tips and recommendations for using Cloud Dataproc in a non-ephemeral model.

Spotlight

Guavus

Founded in February 2006, Guavus has been built from the ground up to solve the world’s most complex data problems. Proven across Fortune 500 enterprises, Guavus provides the only operational intelligence platform that delivers an end-to-end, real-time view across business operations and currently processes over half a trillion records every day. The Guavus Reflex® operational intelligence platform correlates and analyzes massive petabytes of streaming and stored business, operational and sensor data from multiple, disparate source systems in real-time. The resulting “live” operational analytics are fed into a new generation of analytically-powered applications to address specific business problems for network, service assurance & operations, marketing and the Internet of Things. These solutions, which can be embedded into automated workflows and business processes, make our customers more agile, competitive and profitable by transforming them into real-time, data-driven enterprises. B

OTHER ARTICLES

Data privacy concerns over Internet of Things

Article | February 25, 2020

Internet of Things, according to congressional research service (CRS) report 2020, is a system of interrelated devices connected to a network and/or to one another, exchanging data without necessarily requiring human to machine interaction.The report cites smart factories, smart home devices, medical monitoring devices, wearable fitness trackers, smart city infrastructures, and vehicular telematics as examples of IoT.

Read More

How Incorta Customers are Leveraging Real-Time Operational Intelligence to Quickly & Effectively Respond to 3 Likely Scenarios Caused by COVID-19

Article | March 30, 2020

Most businesses do not have contingency or business continuity plans that correlate to the world we see unfold before us—one in which we seem to wake up to an entirely new reality each day. Broad mandates to work at home are now a given. But how do we move beyond this and strategically prepare for—and respond to—business implications resulting from the coronavirus pandemic? Some of our customers are showing us how. These organizations have developed comprehensive, real-time operational intelligence views of their global teams—some in only 24-48 hours—that help them better protect their remote workforces, customers, and business at hand.

Read More

Top 6 Marketing Analytics Trends in 2021

Article | June 21, 2021

The marketing industry keeps changing every year. Businesses and enterprises have the task of keeping up with the changes in marketing trends as they evolve. As consumer demands and behavior changed, brands had to move from traditional marketing channels like print and electronic to digital channels like social media, Google Ads, YouTube, and more. Businesses have begun to consider marketing analytics a crucial component of marketing as they are the primary reason for success. In uncertain times, marketing analytics tools calculate and evaluate the market status and enhances better planning for enterprises. As Covid-19 hit the world, organizations that used traditional marketing analytics tools and relied on historical data realized that many of these models became irrelevant. The pandemic rendered a lot of data useless. With machine learning (ML) and artificial intelligence (AI) in marketers’ arsenal, marketing analytics is turning virtual with a shift in the marketing landscape in 2021. They are also pivoting from relying on just AI technologies but rather combining big data with it. AI and machine learning help advertisers and marketers to improve their target audience and re-strategize their campaigns through advanced marketing attributes, which in turn increases customer retention and customer loyalty. While technology is making targeting and measuring possible, marketers have had to reassure their commitment to consumer privacy and data regulations and governance in their initiatives. They are also relying on third-party data. These data and analytics trends will help organizations deal with radical changes and uncertainties, with opportunities they bring with them over the next few years. To know why businesses are gravitating towards these trends in marketing analytics, let us look at why it is so important. Importance of Marketing Analytics As businesses extended into new marketing categories, new technologies were implemented to support them. This new technology was usually deployed in isolation, which resulted in assorted and disconnected data sets. Usually, marketers based their decisions on data from individual channels like website metrics, not considering other marketers channels. Website and social media metrics alone are not enough. In contrast, marketing analytics tools look at all marketing done across channels over a period of time that is vital for sound decision-making and effective program execution. Marketing analytics helps understand how well a campaign is working to achieve business goals or key performance indicators. Marketing analytics allows you to answer questions like: • How are your marketing initiatives/ campaigns working? What can be done to improve them? • How do your marketing campaigns compare with others? What are they spending their time and money on? What marketing analytics software are they using that helps them? • What should be your next step? How should you allocate the marketing budget according to your current spending? Now that the advantages of marketing analytics are clear, let us get into the details of the trends in marketing analytics of 2021: Rise of real-time marketing data analytics Reciprocation to any action is the biggest trend right now in digital marketing, especially post Covid. Brands and businesses strive to respond to customer queries and provide them with solutions. Running queries in a low-latency customer data platform have allowed marketers to filter the view by the audience and identify underachieving sectors. Once this data is collected, businesses and brands can then readjust their customer targeting and messaging to optimize their performance. To achieve this on a larger scale, organizations need to invest in marketing analytics software and platforms to balance data loads with processing for business intelligence and analytics. The platform needs to allow different types of jobs to run parallel by adding resources to groups as required. This gives data scientists more flexibility and access to response data at any given time. Real-time analytics will also aid marketers in identifying underlying threats and problems in their strategies. Marketers will have to conduct a SWOT analysis and continuously optimize their campaigns to suit them better. . Data security, regulatory compliance, and protecting consumer privacy Protecting market data from a rise in cybercrimes and breaches are crucial problems to be addressed in 2021. This year has seen a surge in data breaches that have damaged businesses and their infrastructures to different levels. As a result, marketers have increased their investments in encryption, access control, network monitoring, and other security measures. To help comply with the General Data Protection Regulation (GDPR) of the European Union, the California Consumer Privacy Act (CCPA), and other regulatory bodies, organizations have made the shift to platforms where all consumer data is in one place. Advanced encryptions and stateless computing have made it possible to securely store and share governed data that can be kept in a single location. Interacting with a single copy of the same data will help compliance officers tasked with identifying and deleting every piece of information related to a particular customer much easier and the possibility of overseeing something gets canceled. Protecting consumer privacy is imperative for marketers. They offer consumers the control to opt out, eradicate their data once they have left the platform, and remove information like location, access control to personally identifiable information like email addresses and billing details separated from other marketing data. Predictive analytics Predictive analytics’ analyzes collected data and predicts future outcomes through ML and AI. It maps out a lookalike audience and identifies which strata are most likely to become a high-value customer and which customer strata has the highest likelihood of churn. It also gauges people’s interests based on their browsing history. With better ML models, predictions have become better overtime, leading to increased customer retention and a drop in churn. According to the research by Zion Market Research, by 2022, the global market for predictive analytics is set to hit $11 billion. Investment in first-party data Cookies-enabled website tracking led marketers to know who was visiting their website and re-calibrate their ads to these people throughout the web. However, in 2020, Google announced cookies would be phased out of Chrome within two years while they had already removed them from Safari and Firefox. Now that adding low-friction tracking to web pages will be tough, marketers will have to gather more limited data. This will then be then integrated with first-party data sets to get a rounded view of the customer. Although a big win for consumer privacy activists, it is difficult for advertisers and agencies to find it more difficult to retarget ads and build audiences in their data management platforms. In a digital world without cookies, marketers now understand how customer data is collected, introspect on their marketing models, and evaluate their marketing strategy. Emergence of contextual customer experience These trends in marketing analytics have become more contextually conscious since the denunciation of cookies. Since marketers are losing their data sets and behavioral data, they have an added motivation to invest in insights. This means that marketers have to target messaging based on known and inferred customer characteristics like their age, location, income, brand affinity, and where these customers are in their buying journey. For example, marketers should tailor messaging in ads to make up consumers based on the frequency of their visits to the store. Effective contextual targeting hinges upon marketers using a single platform for their data and creates a holistic customer profile. Reliance on third-party data Even though there has been a drop in third-party data collection, marketers will continue to invest in third-party data which have a complete understanding of their customers that augments the first-party data they have. Historically, third-party data has been difficult to source and maintain for marketers. There are new platforms that counter improvement of data like long time to value, cost of maintaining third-party data pipelines, and data governance problems. U.S. marketers have spent upwards of $11.9 billion on third-party audience data in 2019, up 6.1% from 2018, and this reported growth curve is going to be even steeper in 2021, according to a study by Interactive Advertising Bureau and Winterberry Group. Conclusion Marketing analytics enables more successful marketing as it shows off direct results of the marketing efforts and investments. These new marketing data analytics trends have made their definite mark and are set to make this year interesting with data and AI-based applications mixed with the changing landscape of marketing channels. Digital marketing will be in demand more than ever as people are purchasing more online. Frequently Asked Questions Why is marketing analytics so important? Marketing analytics has two main purposes; to gauge how well your marketing efforts perform and measure the effectiveness of marketing activity. What is the use of marketing analytics? Marketing analytics help us understand how everything plays off of each other and decide how to invest, whether to re-prioritize or keep going with the current methods. Which industries use marketing analytics? Commercial organizations use it to analyze data from different sources, use analytics to determine the success of a marketing campaign, and target customers specifically. What are the types of marketing analytics tools? Some marketing analytics’ tools are Google Analytics, HubSpot Marketing Hub, Semrush, Looker, Optimizely, etc. { "@context": "https://schema.org", "@type": "FAQPage", "mainEntity": [{ "@type": "Question", "name": "Why is marketing analytics so important?", "acceptedAnswer": { "@type": "Answer", "text": "Marketing analytics has two main purposes; to gauge how well your marketing efforts perform and measure the effectiveness of marketing activity." } },{ "@type": "Question", "name": "What is the use of marketing analytics?", "acceptedAnswer": { "@type": "Answer", "text": "Marketing analytics help us understand how everything plays off of each other and decide how to invest, whether to re-prioritize or keep going with the current methods." } },{ "@type": "Question", "name": "Which industries use marketing analytics?", "acceptedAnswer": { "@type": "Answer", "text": "Commercial organizations use it to analyze data from different sources, use analytics to determine the success of a marketing campaign, and target customers specifically." } },{ "@type": "Question", "name": "What are the types of marketing analytics tools?", "acceptedAnswer": { "@type": "Answer", "text": "Some marketing analytics’ tools are Google Analytics, HubSpot Marketing Hub, Semrush, Looker, Optimizely, etc." } }] }

Read More

Man Vs. Machine: Peaking into the Future of Artificial Intelligence

Article | March 15, 2021

Stephen Hawking, one of the finest minds to have ever lived, once famously said, “AI is likely to be either the best or the worst thing to happen to humanity.” This is of course true, with valid arguments both for and against the proliferation of AI. As a practitioner, I have witnessed the AI revolution at close quarters as it unfolded at breathtaking pace over the last two decades. My personal view is that there is no clear black and white in this debate. The pros and cons are very contextual – who is developing it, for what application, in what timeframe, towards what end? It always helps to understand both sides of the debate. So let’s try to take a closer look at what the naysayers say. The most common apprehensions can be clubbed into three main categories: A. Large-scale Unemployment: This is the most widely acknowledged of all the risks of AI. Technology and machines replacing humans for doing certain types of work isn’t new. We all know about entire professions dwindling, and even disappearing, due to technology. Industrial Revolution too had led to large scale job losses, although many believe that these were eventually compensated for by means of creating new avenues, lowering prices, increasing wages etc. However, a growing number of economists no longer subscribe to the belief that over a longer term, technology has positive ramifications on overall employment. In fact, multiple studies have predicted large scale job losses due to technological advancements. A 2016 UN report concluded that 75% of jobs in the developing world are expected to be replaced by machines! Unemployment, particularly at a large scale, is a very perilous thing, often resulting in widespread civil unrest. AI’s potential impact in this area therefore calls for very careful political, sociological and economic thinking, to counter it effectively. B. Singularity: The concept of Singularity is one of those things that one would have imagined seeing only in the pages of a futuristic Sci-Fi novel. However, in theory, today it is a real possibility. In a nutshell, Singularity refers to that point in human civilization when Artificial Intelligence reaches a tipping point beyond which it evolves into a superintelligence that surpasses human cognitive powers, thereby potentially posing a threat to human existence as we know it today. While the idea around this explosion of machine intelligence is a very pertinent and widely discussed topic, unlike the case of technology driven unemployment, the concept remains primarily theoretical. There is as yet no consensus amongst experts on whether this tipping point can ever really be reached in reality. C. Machine Consciousness: Unlike the previous two points, which can be regarded as risks associated with the evolution of AI, the aspect of machine consciousness perhaps is best described as an ethical conundrum. The idea deals with the possibility of implanting human-like consciousness into machines, taking them beyond the realm of ‘thinking’ to that of ‘feeling, emotions and beliefs’. It’s a complex topic and requires delving into an amalgamation of philosophy, cognitive science and neuroscience. ‘Consciousness’ itself can be interpreted in multiple ways, bringing together a plethora of attributes like self-awareness, cause-effect in mental states, memory, experiences etc. To bring machines to a state of human-like consciousness would entail replicating all the activities that happen at a neural level in a human brain – by no means a meagre task. If and when this were to be achieved, it would require a paradigm shift in the functioning of the world. Human society, as we know it, will need a major redefinition to incorporate machines with consciousness co-existing with humans. It sounds far-fetched today, but questions such as this need pondering right now, so as to be able to influence the direction in which we move when it comes to AI and machine consciousness, while things are still in the ‘design’ phase so to speak. While all of the above are pertinent questions, I believe they don’t necessarily outweigh the advantages of AI. Of course, there is a need to address them systematically, control the path of AI development and minimize adverse impact. In my opinion, the greatest and most imminent risk is actually a fourth item, not often taken into consideration, when discussing the pitfalls of AI. D. Oligarchy: Or to put it differently, the question of control. Due to the very nature of AI – it requires immense investments in technology and science – there are realistically only a handful of organizations (private or government) that can make the leap into taking AI into the mainstream, in a scalable manner, and across a vast array of applications. There is going to be very little room for small upstarts, however smart they might be, to compete at scale against these. Given the massive aspects of our lives that will likely be steered by AI enabled machines, those who control that ‘intelligence’ will hold immense power over the rest of us. That all familiar phrase ‘with great power, comes great responsibility’ will take a whole new meaning – the organizations and/or individuals that are at the forefront of the generally available AI applications would likely have more power than the most despotic autocrats in history. This is a true and real hazard, aspects of which are already becoming areas of concern in the form of discussions around things like privacy. In conclusion, AI, like all major transformative events in human history, is certain to have wide reaching ramifications. But with careful forethought these can be addressed. In the short to medium term, the advantages of AI in enhancing our lives, will likely outweigh these risks. Any major conception that touches human lives in a broad manner, if not handled properly, can pose immense danger. The best analogy I can think of is religion – when not channelled appropriately, it probably poses a greater threat than any technological advancement ever could.

Read More

Spotlight

Guavus

Founded in February 2006, Guavus has been built from the ground up to solve the world’s most complex data problems. Proven across Fortune 500 enterprises, Guavus provides the only operational intelligence platform that delivers an end-to-end, real-time view across business operations and currently processes over half a trillion records every day. The Guavus Reflex® operational intelligence platform correlates and analyzes massive petabytes of streaming and stored business, operational and sensor data from multiple, disparate source systems in real-time. The resulting “live” operational analytics are fed into a new generation of analytically-powered applications to address specific business problems for network, service assurance & operations, marketing and the Internet of Things. These solutions, which can be embedded into automated workflows and business processes, make our customers more agile, competitive and profitable by transforming them into real-time, data-driven enterprises. B

Events