Data management professionals getting the central stage in business, focus on ethics increasing, pressure on Big Tech affecting the landscape of the public web data industry. Thesewill be among the most prominent trends in the big data industry in 2022, according to a public data gathering solutions provider, Oxylabs. Their experts predict what to expect in the year ahead.
Growing Markets for External Data
Tomas Montvilas, Chief Commercial Officer at Oxylabs, says that more industries will discover the benefits of using external data in the upcoming year. He lists a few:
“The market of SaaS products that use external data to provide insights for their clients will grow further in 2022. The successful IPOs of companies like Semrush, Similarweb, Zoominfo and others are driving further investments in the field, and we are likely to see more stars emerging,” - Tomas says.
Another important area he sees for the web scraping industry’s growth is cybersecurity. Cyber threats are becoming more advanced and require new measures of defense. This is where web monitoring and scraping technologies come in. “Constant monitoring of both the public and dark web can help identify malicious sites and programs early. It can also help catch data leaks sooner by finding data sets when they go for sale on the dark web and recognize the actions of hacker groups. Meanwhile, proxies can help with email security by allowing you to scan emails from different IP addresses," he explains.
Data Management Role in Business Further Increasing
With the recent explosion in digitizing everything, data management and analytics have become pivotal in business. Data departments have been experiencing exponential growth during the past few years and the growth will continue well into 2022.
Gediminas Rickevicius, Vice President of Global Partnerships at Oxylabs, notes that the increasing importance of data departments can be easily illustrated by budgeting trends. According to several recent surveys Oxylabs conducted in the UK's finance and ecommerce industries, most data departments are expecting to increase their budgets (51% ecommerce, 43% financial services). Another trend Gediminas predicts for data departments will be the increasing outsourcing of automated public web data gathering tools. There will be several reasons for this. First of all, it is obvious that as companies become dependent on external data, manual data gathering processes are simply not sufficient. Another important factor is the current job market landscape.
Pressure for Big Tech Could Affect Web Data Industry
“With “the great resignation” and lack of human resources being the dominant topics of 2021, it became even harder to find in-house professionals that could dedicate all their time to maintaining and adjusting web scraping infrastructure. Outsourcing this task allows optimizing resources and focusing on data analysis rather than acquisition.” - says Gediminas.
Recent years have been marked by the growing pressure on Big Tech from governments around the world. 2022 will be no different; there will likely be a push for new regulations, especially around personal data and its acquisition and aggregation. According to Denas Grybauskas, Head of Legal at Oxylabs, the data gathering industry should not turn a blind eye to these processes. In light of government pressure, some big tech companies might already be in the process of restricting access to public web data, which could affect many businesses.
Moving Towards Industry Self-regulation
“Some companies are preparing for the old death tactic of pointing fingers. At least in accordance with the leaked emails, Meta (Facebook) is planning to do in terms of personal data leaks and data scraping companies - to shift the attention from leaks by stating that personal data got out in the wild not due to Facebook’s mistakes, but those of scrapers”, - Denas says.
When it comes to the strategic development of the data gathering industry, ethics and legal implications will remain the hot topics in 2022, pushing the industry to continue raising the standards. Ethical proxy acquisition and strong KYC practices will dominate the conversation, predicts Julius Cerniauskas, CEO of Oxylabs. He explains that, as with most new technologies, web scraping is developing faster than the regulations that could safeguard it from potential misuse cases. Therefore, the industry itself has to take the lead in developing self-regulation guidelines and standards for the proper use of technology.
“For several reasons, the issue is set to become more mainstream in 2022. First of all, as the largest industry players are setting the tone, smaller players are likely to follow. Secondly, brands that use proxy services are putting more emphasis on the nature of proxies too, as potential misuse could damage their reputation as well,” - says Julius.
Authors: Julius Cerniauskas, CEO, Oxylabs, Tomas Montvilas, Chief Commercial Officer, Oxylabs, Gediminas Rickevičius, VP of Global Partnerships, Oxylabs, Denas Grybauskas, Head of Legal, Oxylabs