Welcome to WebmasterWorld Guest from 54.159.19.75

Message Too Old, No Replies

Blocking PDFs For A Step Towards Better Quality

     
3:52 pm on Nov 26, 2012 (gmt 0)

Preferred Member

joined:Mar 22, 2011
posts:395
votes: 0


I have a number of data only PDFs that users find highly valuable on a few of my sites. All of those sites were hit by Panda 1. The only real low-quality work on those sites are these data-only (numbers) PDFs, thousands of them.

Would a robot.txt disallow of the directory of PDFs re-establish quality?

With HTML I usually just “noindex” and robot.txt disallow, but PDFs don’t allow for that option.

Any ideas guys?
5:11 pm on Nov 26, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member bwnbwn is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Oct 25, 2005
posts:3492
votes: 3


I have a number of data only PDFs that users find highly valuable

Why are they considered by u low quality when the user finds them of high quality, or are a few of them high but the vast majority are considered low?
To add them to a robots.txt file you will need to add the PDF's into a new folder called PDF then the whole folder can be blocked if this is the way you want to go.
5:35 pm on Nov 26, 2012 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month

joined:Apr 9, 2011
posts:12720
votes: 244


Do you want no-crawl or no-index? If no-index, add this simple package to your htaccess or config file, changing the endings as appropriate:

<FilesMatch "\.(js|txt|xml)$">
Header set X-Robots-Tag "noindex"
</FilesMatch>

Also good for preventing them from indexing your robots.txt and sitemap, since you obviously can't block them from crawling those ;)
4:09 pm on Nov 27, 2012 (gmt 0)

Preferred Member

joined:Mar 22, 2011
posts:395
votes: 0


Great ideas. Thanks.

If you block the bot from crawling files via "robot.txt" will G ever count those towards the quality score of your site? I'm just worried about sometime down the line getting hit by Panda if 1/10 of my site is data driven PDFs.
3:46 am on Nov 28, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
posts:37301
votes: 0


Google also obeys wild card pattern patching in a robots.txt file, even though that syntax was not part of the original robots.txt standard. This means you can use:

User-agent: Googlebot
Disallow: /*.pdf$

1) The asterisk character [*] stands for any number of characters including directory names.
2) The dollar sign character [$] stands for "end of the character string".

This approach disallows any pdf file no matter which directory holds it, and also guards against any accidental use of the character string ".pdf" in a file path other than an actual PDF file.

If you block the bot from crawling files via "robot.txt" will G ever count those towards the quality score of your site?

If the files are not even crawled, then they cannot be directly evaluated for quality.

[edited by: tedster at 3:38 pm (utc) on Nov 29, 2012]

3:13 pm on Nov 28, 2012 (gmt 0)

Preferred Member

joined:Mar 22, 2011
posts:395
votes: 0


@tedster

Thank you! As usual you save the day!

We need to have an appreciation day for you.