Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

poacher69, we crawl the public web. Anyone that blocks us out with robots.txt, we won't crawl. If you check bing.com/robots.txt, it has "Disallow: /search" . So no, we won't crawl Bing's search results pages. If anything, users tend to complain when search results from Lycos or wherever show up in Google.


http://www.bing.com/robots.txt User-agent: * Disallow: /search

Funny thing: http://www.google.com/search?q=site%3Abing.com%2Fsearch%2F

I was gonna call out Matt for crawling bing's search results but I'm guessing Microsoft hasn't realized they return results from the /Search/ folder. ;)


Once again Microsoft is bitten by expecting case insensitivity.


matt, how does google do competitive relevance evaluations without scraping Bing?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: