Spiders, bots and other creepy crawlers: protecting your company website

Download document:

Spiders, bots and other creepy crawlers: protecting your company website - .PDF file.

By Gina Durham

Websites contain a wealth of information. Unfortunately, the human internet users you hope are accessing your site are not the only ones attracted to it. Automated tools, frequently referred to as spiders, bots and screen scrapers, may be crawling your company website too.

Given the potential of the internet to consolidate and manipulate information, automated data aggregation has become a business model for many companies. Using specialised software applications, a scraper can almost instantaneously access numerous websites, submit queries to the sites, extract data from them and gather copies of the actual web pages viewed. Websites containing unique collections of proprietary, pricing and consumer information are frequent targets of scrapers.

Some aggregators, such as reputable online travel sites that provide a collection of airline and hotel options, may ‘scrape’ source websites with the authorisation of the website owners. Others, however, attack competitors’ sites, engaging in unauthorised scraping of large volumes of data. Yet others may crawl sites searching for email addresses, credit card numbers or other sensitive information…

If you are registered and logged in to the site, click on the link below to read the rest of the DLA Piper briefing. If not, please register or sign in with your details below.