Did you respect robots.txt? | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

alecco on July 23, 2017 | parent | context | favorite | on: Ask HN: What do people use to prevent crawlers?

Did you respect robots.txt?

333c on July 24, 2017 | [–]

It sounds to me like an obvious no, if they have a large team to get around countermeasures.

rk06 on July 24, 2017 | [–]

Considering the effort that went into it.

I am pretty sure crawling robot.txt links was their P1 requirement.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact