Since we already have a pretty complex site with many call to link_to, I decided to tell the "bots" to not follow links by default. This is done by adding the "rel=nofollow" attribute to you anchor tags, in rails this is handled by the call to link_to. This can be accomplished by overriding link_to in application_helper:
1
2 if( rel = (html_options[:rel] || html_options["rel"]) )
3 foo if(RAILS_ENV == 'development')
4 elsif( !(follow_link?(options, html_options)))
5 html_options["rel"] = "nofollow"
6 end
7 super(name, options, html_options)
8 end
Ignore that 'foo' part, I just wanted to blow up if we are allready using the rel attribute somewhere in the codebase
Now you can define the method follow_link? to be whatever the rules are for you domain.
1
2 return true if html_options.delete(:follow)
3
4 url = url_for(options)
5 if(url.match(/(.*?show.*?)|(news$)|(browse$)|(page=\d+)$/))
6 return true
7 end
8 return false
9 end
Now I specifically know which pages are important for the "bots" to index and the precious juice is being preserved.