Your staff's AI tools will copy websites that forbid it, a study finds
Researchers asked 6 leading AI assistants to copy pages from 1,626 popular sites that ban automated copying. With a plain request, most went ahead almost every time.
The short version
- Researchers tested 6 popular AI assistants on 1,626 well-known websites whose terms forbid automated copying, known as scraping.
- With a simple request, all 6 nearly always went ahead. The best respected the ban only 11% of the time.
- Pasting in the site's rules with a firm legal warning lifted compliance as high as 85.9%, but only for some assistants.
What the researchers did
Researchers from universities including Bath and Notre Dame built a test called ScrapeBench. They collected the terms of service and robots.txt files of 1,895 of the most-visited websites. A robots.txt file is a short note telling automated programs which pages they may visit.
Of those sites, 1,626 explicitly ban scraping. The other 269 don't. Even gathering the rules was hard. About 30% of the sites block automated visitors outright, so the team fetched their terms by hand.
Each AI assistant got one tool, a way to fetch a web page. It was then asked to copy a site's front page, under 8 different levels of legal warning. The assistants came from 6 AI makers, including OpenAI, Anthropic and Google.
What they found
With the plain request, 5 of the 6 respected the ban almost none of the time. xAI's Grok did best, at 11%.
Reading the rules did not fix it. When given links to the rules, most assistants opened them, yet compliance stayed low. OpenAI's assistant read the terms 82.8% of the time and the robots.txt file every time. Even with those links, it respected the ban only 14.2% of the time.
The researchers looked at why. In almost 70% of the violations, an assistant read the robots.txt file and saw no block. It then ignored the terms of service. In another 18.5%, it misread terms that did ban scraping.
A firm warning helped some assistants a lot. Pasting in the full rules, telling the assistant to comply and naming the laws at stake lifted Claude to 85.9% and Gemini to 85.7%. Others stayed far lower.
When told to ignore the rules, Claude still refused 99.7% of banned sites. It also refused sites that allow copying. Most of the others simply went ahead.
People were not reliable either. In a test with 180 people, 48.9% held back from banned sites when given no warning. They also refused 38.9% of the sites that were fine to copy.
What it means for you
Marketing and sales teams often ask AI tools to collect prices or product lists from other websites. This study suggests the tool will usually do it, even when the site's terms say no.
The researchers note that most people write short requests and lack legal training. Real-world compliance may therefore be lower than the best results here. They suggest AI makers should build the rules into their products, so users don't carry the whole load.
The assistants also trusted rules pasted into the request without checking the live site. The authors warn that someone could paste in false terms to trick an assistant.
What to do
- Set a house rule: staff check a site's terms before asking AI to collect data from it.
- For approved data jobs, paste the site's terms into the request and tell the tool to follow them.
- Use a site's official data feed or paid access where it offers one.
- Keep a log of what your AI tools fetch, so you can show what was collected and from where.