The Internet Advertising Bureau said it had reached an agreement with the on-line auditing service ABCi to track all internet traffic originating from automated robots and spiders, programs that visit web sites and reads their pages and other information in order to create entries for a search engine index. This traffic should not be included in counts claiming to detail the amount of human visitation to sites, but is often responsible for confusing data.
“Spider and robot activity is believed to be the largest single contributor to inaccurate measurement," ABCi President Bruce Smith explained.
The two groups have agreed to update the list of spiders and robots every month and make this information available to their members every month free of charge. It is hoped that this will increase the transparency of web traffic measurements, a key indicator used by advertisers when deciding where to place ad campaigns.