spiders, crawlers and bots

You may not notice them but the world of the internet is full of robots! They’re not physical robots, they’re software ‘bots’. Most of the major search engines use ‘spiders’ and ‘webcrawlers’ to travel from webpage to webpage to index the world wide web. ‘Bots’ are slightly different in that they take on a specific task, find a suitable answer and come back and tell you the results.