Web Crawling vs API: Who’s Winner? (Never-Ending Issues)
With 4th industrial revolution, as a value of data is more important, a technology to collect data has been improved. A technology to gather various data and information on the online is called ‘Web Crawling’. To make valuable something, it is necessary to gather and integrate data or information by web crawling. A Process for Integrating data or information is called ‘Big Data’
Through these technologies, various start-up companies are emerging, and the existing companies try hard to make new businesses. Recently, start-up companies is utilizing web crawling to gather data or service which the existing companies have opened, and making new services by integrating those data, services and information.
In the past, it was hard to gather data or information because technology of web crawling was not good. Therefore, start-up companies communicated with the existing companies, and developed API to get data or information in each task. However, now, they are free to make new service or business without limitations as new technology, web crawling, has been emerging.
However, due to this trend, some issues have appeared. A conflict between start-up and the existing companies is begun because data or services of the existing companies are utilized without agreements or approvals. Of course, those data or information were publicly made for their customers. However, a critical thing is that web crawling can have an effect on a traffic on the server of the existing companies.
For example of the recent start-up company offering aggregation service for insurance, they make an interface to gather a customer’s insurance information to each insurance company, and show them in the integrated page. Before that, they get agreement of personal information from the customer. For this service, instead of the customer, the company get and show all insurance information by doing login in each insurance company web site. This process has an effect on a server of the existing insurance companies. Actually, because the companies utilizing web crawling have an effect on the server without paying, the existing companies get a burden.
Even though these issues are not real issue in the initial stage of these technologies, those became real issues, as technologies have been improved and customers need new creative services. Also, one of these issues is caused by increase of service usage rate. As a result of these, some existing companies reinforced security’s policies, and few companies detected and blocked specific IPs by start-up companies.
To minimize these issues, few start-up companies already changed their strategy, which is to have official affiliation to develop each API to gather data or information from the existing companies. So then, what is the end of these issues?
These issues are far from over.
There are some reasons.
- A shortage of making related laws and guidelines
Basically, optimized laws related to these technologies are absent. Even though there are some guidelines made by the government, it lacks to solve these issues. Also, the guidelines does not have a legal force. - A lack of each stakeholder in this area
It is not easy to narrow each interest of start-up companies and the existing companies. Actually, because services made by start-up companies are new and creative, more customers have been moving to those services. This being so, the existing companies started to hold them in check because start-up companies are not rising star, but real competitor.
Except for the above, a lot of issues are remaining.
Web crawling is not bad technology. But, it is needed to do social discussion and compromise for whole use of this technology, even though there are some issues. Obviously, I expect that this technology is more popularized.
Until now, API technology do not have any issues and is more practical than web crawling technology. However, I consider that web crawling technology should be more popularized then now because improvement of technology is unstoppable. If not solving some issues by this technology, it will be hard to expand and improve this technology.
Even though there is no answer, I fundamentally considered about ‘Shared Society’, through this case.
Web Crawling vs API: Who’s Winner? (Never-Ending Issues)
Recent Related Articles:
- Chatbot Trend 2020: Executive Summary (Prologue / Epilogue)
- Shake Shack Open Review: Jong-Gak / Gwang-Hwa Moon, Korea
- Will Blue Bottle be Think Coffee?
- KAKAO Friends Popcorn and Contents Marketing (Feat. LOTTE Cinema)
- E-Receipt by Messenger Platform: KAKAO PAY Receipt Service
Related External References to this article: