Articles in The 'robots-meta-tag' Tag

3 Things Needed to Keep Your Pages from Being Indexed by Search

by Lee Zoumas

Sometimes you may have certain pages on your website that you do not want indexed by search engines. Just recently, we developed an online order form for a client that should not show up in search engines. To ensure that a page does not get indexed or crawled by search engines, it is important to do 3 things:

1. Add a rule in your website’s robots.txt file. Assuming the page you don’t want indexed is order.htm:

 User-agent: *
 Disallow: /order.htm

2. Add a “noindex, nofollow” robots meta tag in the head section of the page that you don’t want indexed or crawled:

3. For each link leading to the page that you don’t want indexed or crawled add the following “rel” attribute to the anchor tag with a value of “nofollow”:

That’s pretty much it. It should be noted, that until recently, it was thought that you only had to add the rule to your robots.txt file to prevent pages from being indexed and crawled. However, Matt Cutts from Google gave some insight stating otherwise in a video posted on his blog (http://www.mattcutts.com/blog/robots-txt-remove-url/).

Using the “nofollow” link attribute for fun and profit

by Marjory Meechan

On Wikipedia and any number of message boards and forums across the internet, it is common to find external links marked with a “nofollow” attribute like this one here:Marjory Meechan

The “nofollow” attribute for links was adopted a few years ago by major search engines to combat the use of spam links that were showing up in beleaguered forum pages all over the internet. In an attempt to build ranking for pages on their sites, spammers would insert link references to those pages using their chosen keywords as anchor text in the comments section of message boards, forums and blog posts. In many cases, the comments were completely irrelevant to the content of the discussion and were a big nuisance for these sites and the search engines.

To help discourage this practice, forum owners were encouraged to place the “no follow” attribute on their links and all the major search engines got together and announced that they would not credit these inbound links to sites for the purposes of calculating search engine results ranks. This turned out to be an excellent solution to the problem and is now the standard for blog comments and message board comments.

Read More