That strongly depends on the data you have. I have written boots to scrape sites with government data so that I could do searches that wasn't possible using their online forms. I did not look at, nor attempt to obey, their robots text, nor would I have given a rats ass about breaking whatever they would put up to stop me.

There is also the argument that when you make something available on the web you make it available to everybody.