September 26, 2005

Abusive Spidering: don't do it

Without naming names, I discovered that a particular site was requesting our RSS feed to the tune of 15,000+ times last month. I'd tried various ways to throttle it, and then to ban it, all of which were unsuccessful. Given that I don't generally post multiple times a day, there's simply no reason to hit the feed 500 times a day (at nearly a dozen hits per second each time), now is there?

Finally I called the owner of the domain from which the requests were emanating. Turns out he was the host. After a little discussion — during which he noted that none of his clients were running such a spider — I figured the real domain was probably cloaked, and let it go.

However — surprise! — this was no lame host. He called back a few times to keep me posted, and actually went through each and every one of his clients' sites, found the culprit and called him. After which said culprit called me claiming that he was featuring my feeds on his site and demanding to know (threatening tone and all) whether I "really" wanted to be removed. Given that I've seen exactly zero referrals from his site, that's a threat that can't hold any water. At any rate, the abusive spidering has stopped.

Listen: anyone spidering overmuch just creates a problem for the site owner. It's good to spider politely.

Have your say ...

First-time comments will be held for moderation (but comments are appreciated). Otherwise, just be courteous. If your name is a bunch of keywords, your comment will be deleted. Don't post links unless highly pertinent. Posters must be 16 or older.

Manage your subscriptions

Archives
© 2004-2017 DianeV Web Design Studio. All Rights Reserved.
33 queries. 0.225 seconds.