Forum Moderators: open
Since i have to remove a handfull of redirect script urls i wonder if a robots.txt entry is enough.
History: i recently changed all links that people post within my forum from direct urls to redirect urls /jump.php?go=anydomain.ws. I blocked the jump.php through a robots.txt entry. However, since google allready has a older version of my robots.txt file, the new jump.php links were crawled and are now listed at google.
I made this script to avoid unwanted links to bad neighbourhoods - but that doesn's work now until google recrawls the robots.txt. Therefor i would like to fill removal requests for the allready crawled urls.
So, does google recrawl the robots.txt file in case of a removal request? Or does the removal not work for such redirect urls where there's no meta tag?
They require you to change each page, AFAIK. This is their "proof" that you own and control the page, and are not submitting someone else's pages for removal.
Also, note that an entry in robots.txt prevents Google from crawling a page, but it may list that page's URL (with no title or description) if it finds a link to it. I have obtained best results controlling Google (and AJ/Teoma) by using the <meta robots> tag for all pages which have incoming links from other indexed pages.
Jim
jdMorgan,
yah, i know this. The problem is that i have to remove this kind of links:
- [widgets.ws...]
no Description, no title, just the url.
These are redirect links. I can't put a meta tag at them because they don't exist as pages. The question is: if google visits them because of a removal request - will the robots.txt file get recrawled first or does google simply look if the url returns a valid exclusion meta tag and if not moves on?
Never tried it with a rewritten URL. I'd suggest you contact them and ask, explaining that your site is dynamically generated by a script.
I don't know if these removal requests are processed completely automatically, or if there is a human review.
Jim
