showing posts tagged #Crawl
@kas I updated the registry to interpret the robots.txt (your entries won't be indexed anymore) + the Crawl-Delay directive of robots.txt https://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive