What you missed in Big Data: Making order out of spreadsheet chaos

MARIA DEUTSCHER | May 12, 2016

article image
For a solution category dedicated to keeping data organized and accessible, spreadsheeting software can be immensely unwieldy at times. The issue returned to the fore last week after Alphabet Inc. integrated its BigQuery business intelligence service with Google Sheets to try and make it easier for users to tap their structured records. The addition not only makes it possible to import spreadsheets through the native interface but also export the results of an analysis in the other direction with similar ease. And perhaps most significantly, BigQuery can now automatically pick up changes in a document that it’s being used to process and account for the new data during the next access request. The feature has the potential to save a lot of manual work for a business intelligence team with a large number of spreadsheets to manage.

Spotlight

CI&T

CI&T, the lean digital transformation partner for the world's biggest companies, is a pioneer at delivering speed-at-scale through the application of design thinking, lean methodologies and advanced tech, including Machine Learning/AI, Advanced Analytics, Cloud and Mobility. For over 20 years, CI&T has been a trusted partner for the most complex global engagements inside companies including Coca-Cola, Johnson & Johnson, AB-InBev, Itau Bank and Motorola. Together with Comrade, a strategy and customer experience design agency in the San Francisco Bay Area, we quickly and efficiently deliver high quality products and experiences people love.

OTHER ARTICLES

Advanced Data and Analytics Can Add Value in Private Equity Industry!

Article | January 6, 2021

As the organizations go digital the amount of data generated whether in-house or from outside is humongous. In fact, this data keeps increasing with every tick of the clock. There is no doubt about the fact that most of this data can be junk, however, at the same time this is also the data set from where an organization can get a whole lot of insight about itself. It is a given that organizations that don’t use this generated data to build value to their organization are prone to speed up their obsolescence or might be at the edge of losing the competitive edge in the market. Interestingly it is not just the larger firms that can harness this data and analytics to improve their overall performance while achieving operational excellence. Even the small size private equity firms can also leverage this data to create value and develop competitive edge. Thus private equity firms can achieve a high return on an initial investment that is low. Private Equity industry is skeptical about using data and analytics citing the reason that it is meant for larger firms or the firms that have deep pockets, which can afford the revamping cost or can replace their technology infrastructure. While there are few private equity investment professionals who may want to use this advanced data and analytics but are not able to do so for the lack of required knowledge. US Private Equity Firms are trying to understand the importance of advanced data and analytics and are thus seeking professionals with the expertise in dealing with data and advanced analytics. For private equity firms it is imperative to comprehend that data and analytics’ ability is to select the various use cases, which will offer the huge promise for creating value. Top Private Equity firms all over the world can utilize those use cases and create quick wins, which will in turn build momentum for wider transformation of businesses. Pinpointing the right use cases needs strategic thinking by private equity investment professionals, as they work on filling the relevant gaps or even address vulnerabilities. Private Equity professionals most of the time are also found thinking operationally to recognize where can they find the available data. Top private equity firms in the US have to realize that the insights which Big data and advanced analytics offer can result in an incredible opportunity for the growth of private equity industry. As Private Equity firms realize the potential and the power of big data and analytics they will understand the invaluableness of the insights offered by big data and analytics. Private Equity firms can use the analytics insights to study any target organization including its competitive position in the market and plan their next move that may include aggressive bidding for organizations that have shown promise for growth or leaving the organization that is stuffed with loads of underlying issues. But for all these and also to build careers in private equity it is important to have reputed qualification as well. A qualified private equity investment professional will be able to devise information-backed strategies in no time at all. In addition, with Big Data and analytics in place, private equity firms can let go of numerous tasks that are done manually and let the technology do the dirty work. There have been various studies that show how big data and analytics can help a private Equity firm.

Read More

Why Data Science Needs DataOps

Article | March 31, 2020

DataOps helps reduce the time data scientists spend preparing data for use in applications. Such tasks consume roughly 80% of their time now.We’re still hopeful that the digital transformation will provide the insights businesses need from big data. As a data scientist, you’re probably aware of the growing pressure from companies to extract meaningful insights from data and find the stories needed for impact.No matter how in-demand data science is in the employment numbers, equal pressure is rising for data scientists to deliver business value and no wonder. We’re approaching the age where data science and AI draw a line in the sand for which companies remain competitive and which ones collapse.One answer to this pressure is the rise of DataOps. Let’s take a look at what it is and how it could provide a path for data scientists to give businesses what they’ve been after.

Read More

Evolution of capabilities of Data Platforms & data ecosystem

Article | October 27, 2020

Data Platforms and frameworks have been constantly evolving. At some point of time; we are excited by Hadoop (well for almost 10 years); followed by Snowflake or as I say Snowflake Blizzard (who managed to launch biggest IPO win historically) and the Google (Google solves problems and serves use cases in a way that few companies can match). The end of the data warehouse Once upon a time, life was simple; or at least, the basic approach to Business Intelligence was fairly easy to describe… A process of collecting information from systems, building a repository of consistent data, and bolting on one or more reporting and visualisation tools which presented information to users. Data used to be managed in expensive, slow, inaccessible SQL data warehouses. SQL systems were notorious for their lack of scalability. Their demise is coming from a few technological advances. One of these is the ubiquitous, and growing, Hadoop. On April 1, 2006, Apache Hadoop was unleashed upon Silicon Valley. Inspired by Google, Hadoop’s primary purpose was to improve the flexibility and scalability of data processing by splitting the process into smaller functions that run on commodity hardware. Hadoop’s intent was to replace enterprise data warehouses based on SQL. Unfortunately, a technology used by Google may not be the best solution for everyone else. It’s not that others are incompetent: Google solves problems and serves use cases in a way that few companies can match. Google has been running massive-scale applications such as its eponymous search engine, YouTube and the Ads platform. The technologies and infrastructure that make the geographically distributed offerings perform at scale are what make various components of Google Cloud Platform enterprise ready and well-featured. Google has shown leadership in developing innovations that have been made available to the open-source community and are being used extensively by other public cloud vendors and Gartner clients. Examples of these include the Kubernetes container management framework, TensorFlow machine learning platform and the Apache Beam data processing programming model. GCP also uses open-source offerings in its cloud while treating third-party data and analytics providers as first-class citizens on its cloud and providing unified billing for its customers. The examples of the latter include DataStax, Redis Labs, InfluxData, MongoDB, Elastic, Neo4j and Confluent. Silicon Valley tried to make Hadoop work. The technology was extremely complicated and nearly impossible to use efficiently. Hadoop’s lack of speed was compounded by its focus on unstructured data — you had to be a “flip-flop wearing” data scientist to truly make use of it. Unstructured datasets are very difficult to query and analyze without deep knowledge of computer science. At one point, Gartner estimated that 70% of Hadoop deployments would not achieve the goal of cost savings and revenue growth, mainly due to insufficient skills and technical integration difficulties. And seventy percent seems like an understatement. Data storage through the years: from GFS to Snowflake or Snowflake blizzard Developing in parallel with Hadoop’s journey was that of Marcin Zukowski — co-founder and CEO of Vectorwise. Marcin took the data warehouse in another direction, to the world of advanced vector processing. Despite being almost unheard of among the general public, Snowflake was actually founded back in 2012. Firstly, Snowflake is not a consumer tech firm like Netflix or Uber. It's business-to-business only, which may explain its high valuation – enterprise companies are often seen as a more "stable" investment. In short, Snowflake helps businesses manage data that's stored on the cloud. The firm's motto is "mobilising the world's data", because it allows big companies to make better use of their vast data stores. Marcin and his teammates rethought the data warehouse by leveraging the elasticity of the public cloud in an unexpected way: separating storage and compute. Their message was this: don’t pay for a data warehouse you don’t need. Only pay for the storage you need, and add capacity as you go. This is considered one of Snowflake’s key innovations: separating storage (where the data is held) from computing (the act of querying). By offering this service before Google, Amazon, and Microsoft had equivalent products of their own, Snowflake was able to attract customers, and build market share in the data warehousing space. Naming the company after a discredited database concept was very brave. For those of us not in the details of the Snowflake schema, it is a logical arrangement of tables in a multidimensional database such that the entity-relationship diagram resembles a snowflake shape. … When it is completely normalized along all the dimension tables, the resultant structure resembles a snowflake with the fact table in the middle. Needless to say, the “snowflake” schema is as far from Hadoop’s design philosophy as technically possible. While Silicon Valley was headed toward a dead end, Snowflake captured an entire cloud data market.

Read More

Taking a qualitative approach to a data-driven market

Article | February 18, 2021

While digital transformation is proving to have many benefits for businesses, what is perhaps the most significant, is the vast amount of data there is available. And now, with an increasing number of businesses turning their focus to online, there is even more to be collected on competitors and markets than ever before. Having all this information to hand may seem like any business owner’s dream, as they can now make insightful and informed commercial decisions based on what others are doing, what customers want and where markets are heading. But according to Nate Burke, CEO of Diginius, a propriety software and solutions provider for ecommerce businesses, data should not be all a company relies upon when making important decisions. Instead, there is a line to be drawn on where data is required and where human expertise and judgement can provide greater value. Undeniably, the power of data is unmatched. With an abundance of data collection opportunities available online, and with an increasing number of businesses taking them, the potential and value of such information is richer than ever before. And businesses are benefiting. Particularly where data concerns customer behaviour and market patterns. For instance, over the recent Christmas period, data was clearly suggesting a preference for ecommerce, with marketplaces such as Amazon leading the way due to greater convenience and price advantages. Businesses that recognised and understood the trend could better prepare for the digital shopping season, placing greater emphasis on their online marketing tactics to encourage purchases and allocating resources to ensure product availability and on-time delivery. While on the other hand, businesses who ignored, or simply did not utilise the information available to them, would have been left with overstocked shops and now, out of season items that would have to be heavily discounted or worse, disposed of. Similarly, search and sales data can be used to understand changing consumer needs, and consequently, what items businesses should be ordering, manufacturing, marketing and selling for the best returns. For instance, understandably, in 2020, DIY was at its peak, with increases in searches for “DIY facemasks”, “DIY decking” and “DIY garden ideas”. For those who had recognised the trend early on, they had the chance to shift their offerings and marketing in accordance, in turn really reaping the rewards. So, paying attention to data certainly does pay off. And thanks to smarter and more sophisticated ways of collecting data online, such as cookies, and through AI and machine learning technologies, the value and use of such information is only likely to increase. The future, therefore, looks bright. But even with all this potential at our fingertips, there are a number of issues businesses may face if their approach relies entirely on a data and insight-driven approach. Just like disregarding its power and potential can be damaging, so can using it as the sole basis upon which important decisions are based. Human error While the value of data for understanding the market and consumer patterns is undeniable, its value is only as rich as the quality of data being inputted. So, if businesses are collecting and analysing their data on their own activity, and then using this to draw meaningful insight, there should be strong focus on the data gathering phase, with attention given to what needs to be collected, why it should be collected, how it will be collected, and whether in fact this is an accurate representation of what it is you are trying to monitor or measure. Human error can become an issue when this is done by individuals or teams who do not completely understand the numbers and patterns they are seeing. There is also an obstacle presented when there are various channels and platforms which are generating leads or sales for the business. In this case, any omission can skew results and provide an inaccurate picture. So, when used in decision making, there is the possibility of ineffective and unsuccessful changes. But while data gathering becomes more and more autonomous, the possibility of human error is lessened. Although, this may add fuel to the next issue. Drawing a line The benefits of data and insights are clear, particularly as the tasks of collection and analysis become less of a burden for businesses and their people thanks to automation and AI advancements. But due to how effortless data collection and analysis is becoming, we can only expect more businesses to be doing it, meaning its ability to offer each individual company something unique is also being lessened. So, businesses need to look elsewhere for their edge. And interestingly, this is where a line should be drawn and human judgement should be used in order to set them apart from the competition and differentiate from what everyone else is doing. It makes perfect sense when you think about it. Your business is unique for a number of reasons, but mainly because of the brand, its values, reputation and perceptions of the services you are upheld by. And it’s usually these aspects that encourage consumers to choose your business rather than a competitor. But often, these intangible aspects are much more difficult to measure and monitor through data collection and analysis, especially in the autonomous, number-driven format that many platforms utilise. Here then, there is a great case for businesses to use their own judgements, expertise and experiences to determine what works well and what does not. For instance, you can begin to determine consumer perceptions towards a change in your product or services, which quantitative data may not be able to pick up until much later when sales figures begin to rise or fall. And while the data will eventually pick it up, it might not necessarily be able to help you decide on what an appropriate alternative solution may be, should the latter occur. Human judgement, however, can listen to and understand qualitative feedback and consumer sentiments which can often provide much more meaningful insights for businesses to base their decisions on. So, when it comes to competitor analysis, using insights generated from figure-based data sets and performance metrics is key to ensuring you are doing the same as the competition. But if you are looking to get ahead, you may want to consider taking a human approach too.

Read More

Spotlight

CI&T

CI&T, the lean digital transformation partner for the world's biggest companies, is a pioneer at delivering speed-at-scale through the application of design thinking, lean methodologies and advanced tech, including Machine Learning/AI, Advanced Analytics, Cloud and Mobility. For over 20 years, CI&T has been a trusted partner for the most complex global engagements inside companies including Coca-Cola, Johnson & Johnson, AB-InBev, Itau Bank and Motorola. Together with Comrade, a strategy and customer experience design agency in the San Francisco Bay Area, we quickly and efficiently deliver high quality products and experiences people love.

Events