Introduction

A search engine bot or web crawler is a computerized system that crawls and indexes sites so that an online search engine can retrieve associated results for users.