I’ve been doing a lot of development work on a fun new project and suddenly feel like using my blog to document some of the tools I am enjoying.
First up, I’m cleaning up the nagging error.log messages for things like favicon.ico and robots.txt
Here is a handy favicon generator at favicon.cc
This one was pretty easy to use. It has a full color picker, with transparency, easy right click delete, and you can drag your work around to fix the alignment and centering with a nice preview of what it might look like in an address bar. I quick click on Download Favicon and you are done. Or, if you want to save your work and return to it later you can register. (You can also make animated icons. But, please, don’t…)
The other item is the robots.txt file. You can get some handy information from http://www.robotstxt.org/, where you will learn (or be reminded) that the robots.txt is optional and in no way secures your site or guarantees a search engine won’t index content you ‘disallow’.
So why would you use it? It might be helpful to request search engines not download and index all of your images (to save bandwidth), especially if having searchable images doesn’t help customers find you. Another handy use of the robots.txt file is on your test servers so that search engines don’t end up making error pages searchable, or exposing information that isn’t ready to be released.
In all cases the robots.txt file is a polite suggestion, and if you really want to keep prying eyes out, you better use firewalls or the .htaccess (or whatever your server uses) to properly restrict access.