Crawlers (or bots) are used to gather data obtainable on the web. By using website navigation menus, and reading inner and external hyperlinks, the bots start to understand the context of a page. Of course, the words, images, and other knowledge on pages also assist search engines like google and ya