Since we already have a pretty complex site with many call to link_to, I decided to tell the "bots" to not follow links by default. This is done by adding the "rel=nofollow" attribute to you anchor tags, in rails this is handled by the call to link_to. This can be accomplished by overriding link_to in application_helper:
2 if( rel = (html_options[:rel] || html_options["rel"]) )
3 foo if(RAILS_ENV == 'development')
4 elsif( !(follow_link?(options, html_options)))
5 html_options["rel"] = "nofollow"
7 super(name, options, html_options)
Ignore that 'foo' part, I just wanted to blow up if we are allready using the rel attribute somewhere in the codebase
Now you can define the method follow_link? to be whatever the rules are for you domain.
2 return true if html_options.delete(:follow)
4 url = url_for(options)
6 return true
8 return false
Now I specifically know which pages are important for the "bots" to index and the precious juice is being preserved.