Crawler

This is a search engine component, which browsers through all he pages n the internet and collects relevant information from them and indexes them. It is also referred to as spider or robot.

Related posts:

1. Bot
2. AllTheWeb
3. Doorway page
4. Crawl
5. Bait and Switch
6. Agent Name
7. Cgi-bin
8. Filter Word
9. Entry Page
10. Banned