The Robots meta tag tells searchengines what to add to their index and where to follow links. It`s different to the robots.txt file but sets similar rules. You can tell searchengines not to look at a page and deindex it from their database so this meta tag has significant influence and should be handled with care.
The meta name ‘robots’ is considered by google, yahoo and bing.
<meta name="robots" content="noindex, follow"> <meta name="robots" content="index, nofollow"> <meta name="robots" content="noindex, nofollow">
- noindex, follow – The spider will drop that page from their database but follows links on this page. This may be an option for a TOS page or a page where you don’t want visitors to enter from search engine result pages.
- index, nofollow – The page is listed but links on that page are not followed.
- noindex, nofollow – Don’t do anything with this page. Do not list it and don’t follow any links.
- noimageindex – Disallow search engines to spider images from that page. This does not guarantee that google won’t find a link to the image somewhere else.
- noarchive – searchengines won’t show a cached version of this url
- NONE – equivalent to “noindex, nofollow”.
There are a few other attributes but index and follow are the most useful. If the meta tag is not set google will use “index, follow” by default.