DIGITAL ENG TREND

Web Crawling vs API: Who’s Winner? (Never-Ending Issues)

Web Crawling vs API: Who’s Winner? (Never-Ending Issues)

With 4th industrial revolution, as a value of data is more important, a technology to collect data has been improved. A technology to gather various data and information on the online is called ‘Web Crawling’. To make valuable something, it is necessary to gather and integrate data or information by web crawling. A Process for Integrating data or information is called ‘Big Data’

웹 크롤링과 빅데이터 기술은 4차 산업의 대표 기술이다.
Web Crawling and Big Data are the typical technology in the 4th industrial revolution
Web Crawling and Big Data are the typical technology in the 4th industrial revolution

Through these technologies, various start-up companies are emerging, and the existing companies try hard to make new businesses. Recently, start-up companies is utilizing web crawling to gather data or service which the existing companies have opened, and making new services by integrating those data, services and information.

In the past, it was hard to gather data or information because technology of web crawling was not good. Therefore, start-up companies communicated with the existing companies, and developed API to get data or information in each task. However, now, they are free to make new service or business without limitations as new technology, web crawling, has been emerging.

웹 크롤링 기술로 수 많은 스타트업들이 등장하고 있다.
A lot of Start-Up Companies have been introduced by Web Crawling Technology
A lot of Start-Up Companies have been introduced by Web Crawling Technology

However, due to this trend, some issues have appeared. A conflict between start-up and the existing companies is begun because data or services of the existing companies are utilized without agreements or approvals. Of course, those data or information were publicly made for their customers. However, a critical thing is that web crawling can have an effect on a traffic on the server of the existing companies.

For example of the recent start-up company offering aggregation service for insurance, they make an interface to gather a customer’s insurance information to each insurance company, and show them in the integrated page. Before that, they get agreement of personal information from the customer. For this service, instead of the customer, the company get and show all insurance information by doing login in each insurance company web site. This process has an effect on a server of the existing insurance companies. Actually, because the companies utilizing web crawling have an effect on the server without paying, the existing companies get a burden.

웹 크롤링 기술은 기존 기업들의 서버에 부하를 줄 수 있다.
Web Crawling Technology can have an impact on a server of the existing companies.
Web Crawling Technology can have an impact on a server of the existing companies.

Even though these issues are not real issue in the initial stage of these technologies, those became real issues, as technologies have been improved and customers need new creative services. Also, one of these issues is caused by increase of service usage rate. As a result of these, some existing companies reinforced security’s policies, and few companies detected and blocked specific IPs by start-up companies.

To minimize these issues, few start-up companies already changed their strategy, which is to have official affiliation to develop each API to gather data or information from the existing companies. So then, what is the end of these issues?

규제와 이해 관계자들의 충돌로 인하여 스타트업들의 전략이 일부 바뀌고 있다.
Strategy of Start-Up companies is being changed by regulation issue and conflict to the existing companies.
Strategy of Start-Up companies is being changed by regulation issue and conflict to the existing companies.

These issues are far from over.
There are some reasons.

  1. A shortage of making related laws and guidelines
    Basically, optimized laws related to these technologies are absent. Even though there are some guidelines made by the government, it lacks to solve these issues. Also, the guidelines does not have a legal force.
  2. A lack of each stakeholder in this area
    It is not easy to narrow each interest of start-up companies and the existing companies. Actually, because services made by start-up companies are new and creative, more customers have been moving to those services. This being so, the existing companies started to hold them in check because start-up companies are not rising star, but real competitor.

Except for the above, a lot of issues are remaining.

Web crawling is not bad technology. But, it is needed to do social discussion and compromise for whole use of this technology, even though there are some issues. Obviously, I expect that this technology is more popularized.

웹 크롤링은 분명 언젠가는 자리 잡을 것이다. 하지만 해결 해야할 과제가 있다.
Web Crawling Technology will settle down. But, there are some issues to solve
Web Crawling Technology will settle down. But, there are some issues to solve

Until now, API technology do not have any issues and is more practical than web crawling technology. However, I consider that web crawling technology should be more popularized then now because improvement of technology is unstoppable. If not solving some issues by this technology, it will be hard to expand and improve this technology.

Even though there is no answer, I fundamentally considered about ‘Shared Society’, through this case.

Web Crawling vs API: Who’s Winner? (Never-Ending Issues)

Recent Related Articles:

Related External References to this article:

댓글 남기기

이 사이트는 스팸을 줄이는 아키스밋을 사용합니다. 댓글이 어떻게 처리되는지 알아보십시오.

%d 블로거가 이것을 좋아합니다: