Amazon Web Services has started an investigation to determine whether Perplexity AI is breaking its rules, according to Wired. To, be precise, the company’s cloud division is looking into allegations that the service is using a crawler, which is hosted on its servers, that ignores the Robots Exclusion Protocol. This protocol is a web standard, wherein developers put a robots.txt file on a domain containing instructions on whether bots can or can’t access a particular page. Complying with those instructions is voluntary, but crawlers from reputable companies have generally been respecting them since web
→ Continue reading at Engadget
