Crawler

This is a search engine component, which browsers through all he pages n the internet and collects relevant information from them and indexes them. It is also referred to as spider or robot.

Related posts:

1. Bot
2. AllTheWeb
3. Doorway page
4. Crawl
5. Bait and Switch
6. Cgi-bin
7. Filter Word
8. Agent Name
9. Entry Page
10. Cache