There are currently 12 Baidu crawlers infesting our beloved forums, in contrast there are only 2 google crawlers. It's an invasion I tell thee.

I don't really know what Baidu does, but I'm struggling to work out why they need 10 spiders all trawling the site.

Only 10 now, the other two must have gone back for reinforcements.   

They run a web search site, like google, amongst other options. Quite why they require so many spiders I don't know, spying for the Chinese government? 

Oops, did I let that Chinese secret out the bag? 

Baidu has much to do, it is looking at threads way back in time. Like; http://www.tekforums.net/sports-hobbies-cycling/pompey-lift-asia-cup/

Baidu seems to like a lot of your threads at the moment bear

has a lot of catching up to do it seems

How does one ban it from crawling a site/forum?

Does one and should one ban it more my query, I would think the more engines indexing the site the better?

QuoteDuring Q4 of 2010, it is estimated that there were 4.02 billion search queries in China of which Baidu had a market share of 56.6%. China's internet-search revenue share in second quarter 2011 by Baidu is 76%[6] In December 2007, Baidu became the first Chinese company to be included in the NASDAQ-100 index.

Seems it is China's answer to Google.


Oh and to answer your question directly, you can add exclusions to your robots.txt file for search engine spiders, or if your forum/CMS software supports it there might be a way of blocking them directly.

Robots.txt only works if the spider is 'polite', but there are supposedly ways of 'trapping' impolite spiders who ignore it.

I think we have Baidu to thank for the recent spam posts, it seems they all have Chinese IPs

14 of the tasty spam spiders at the moment