Welcome to WebmasterWorld Guest from

Forum Moderators: bakedjake

Message Too Old, No Replies

Mobile site design question

Managing scrapers and bots?

7:16 pm on Jul 15, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 20, 2004
votes: 0

Hello everyone,

Been working on my mobile site for a few days now. Getting close to rolling it out. I have some concerns about scrapers and such abusing my mobile site for quick access to my data/content.

Anyone have any experiences with this?

For now I am thinking of limiting my mobile scripts that page through my content to XX number of pages. Picking a number that most mobile users would never reach, but a bot could hit rather quickly. Then telling them to go to our non-mobile site for our full index of content.

I figured that might thwart the bot scrapers that stumble into my juicy mobile version of the site.

Obviously this would slow down any good bots from crawling the mobile site completely as well. I was not sure if I should even care about that given that I can submit a mobile sitemap to the major search engines anyway?

What do you all think?