The robots
https://numberfields.asu.edu/NumberFields/show_user.php?userid=6139696
The robots.txt file courses seek engine crawlers on which pages to index or no longer, guaranteeing that simplest primary content will get prioritized in search outcomes.