Spiders, bots and other creepy crawlers: protecting your company website

By Gina Durham

Websites contain a wealth of information. Unfortunately, the human internet users you hope are accessing your site are not the only ones attracted to it. Automated tools, frequently referred to as spiders, bots and screen scrapers, may be crawling your company website too.

Given the potential of the internet to consolidate and manipulate information, automated data aggregation has become a business model for many companies. Using specialised software applications, a scraper can almost instantaneously access numerous websites, submit queries to the sites, extract data from them and gather copies of the actual web pages viewed. Websites containing unique collections of proprietary, pricing and consumer information are frequent targets of scrapers.

Some aggregators, such as reputable online travel sites that provide a collection of airline and hotel options, may ‘scrape’ source websites with the authorisation of the website owners. Others, however, attack competitors’ sites, engaging in unauthorised scraping of large volumes of data. Yet others may crawl sites searching for email addresses, credit card numbers or other sensitive information…

If you are registered and logged in to the site, click on the link below to read the rest of the DLA Piper briefing. If not, please register or sign in with your details below.

Sign in or Register to continue reading this article

Sign in


It's quick, easy and free!

It takes just 5 minutes to register. Answer a few simple questions and once completed you’ll have instant access.

Register now

Why register to The Lawyer


Industry insight

In-depth, expert analysis into the stories behind the headlines from our leading team of journalists.


Market intelligence

Identify the major players and business opportunities within a particular region through our series of free, special reports.


Email newsletters

Receive your pick of The Lawyer's daily and weekly email newsletters, tailored by practice area, region and job function.

More relevant to you

To continue providing the best analysis, insight and news across the legal market we are collecting some information about who you are, what you do and where you work to improve The Lawyer and make it more relevant to you.

Briefings from DLA Piper

View more briefings from DLA Piper

Analysis from The Lawyer

View more analysis from The Lawyer


3 Noble Street

Turnover (£m): 1,566.29
No. of lawyers: 3,961 (UK 200)
Jurisdiction: global
No. of offices: more than 75
No. of qualified lawyers: 542 (International 50)
No. of partners: 142.6