i see the bigger site have the bots not crawling them

[indodpendo]

I see the big site disallows all the big engines for crawling there site why would some not want them to crawl there site?

[ThomasB]

indodpendo, there are a lot of reasons, I just want to mention a few of them:
- Bandwidth (crawling needs bandwith, that costs money)
- Ressources (CPU, memory, ...)
- Content Theft
- Archiving
- Mistake