Make the default robot policy a configurable option.

Make the default robot policy a configurable option.

Also replace some odd hardcoded robot policies (for Special:Specialpages and for nonexistent images) with 'noindex,nofollow', which seems more
appropriate at least for the latter.  I'm not quite sure _why_ Special:Specialpages was marked as 'index,nofollow' before; it's been that way since
r1284.  It might be safe to just leave it at the default setting, but I don't really think it makes much difference either way.