@robotmachine All of the companies have made their ip addresses public.
This means if you are a website admin or host your own site, you can edit configuration files that either block, redirect or serve 404 pages to specific incoming requests. Essentially website owners can prevent AI bots scraping their servers with this configuration (probably the block or redirect as traffic can have an effect on bills for hosting).
However Perplexity lied about their IP addresses. So blocking what they published as their ip address range did not work.
"The magic trick that’s made Perplexity worth 10 figures, in other words, appears to be that it’s both doing what it says it isn’t and not doing what it says it is."
I also think that seeing as these companies are so rampant to ingest everything and maybe after the PerplexityAI horseshit, they might also have found ways to try and bypass protocols, firewalls, server rules and so on. A race to the bottom.
This means if you are a website admin or host your own site, you can edit configuration files that either block, redirect or serve 404 pages to specific incoming requests. Essentially website owners can prevent AI bots scraping their servers with this configuration (probably the block or redirect as traffic can have an effect on bills for hosting).
However Perplexity lied about their IP addresses. So blocking what they published as their ip address range did not work.
See
https://www.wired.com/story...
https://rknight.me/blog...
https://blog.cloudflare.com/perplexi...
My favourite quote from the first article
"The magic trick that’s made Perplexity worth 10 figures, in other words, appears to be that it’s both doing what it says it isn’t and not doing what it says it is."
I also think that seeing as these companies are so rampant to ingest everything and maybe after the PerplexityAI horseshit, they might also have found ways to try and bypass protocols, firewalls, server rules and so on. A race to the bottom.
Sorry for such a long rant.