On one hand, this seems like a great opportunity to gain visibility for our content if searchers click through. On the other hand, they may just utilize the AI-generated summaries without ever clicking through. I’m encountering mixed opinions on the pros and cons everywhere. Has anyone indexed their blog on AI platforms yet? What are your thoughts?
I’ve blocked AI bots from scraping my blog because I don’t want my content indexed by them. I see them as a way for big tech to exploit creators by profiting off our work without compensation.
Arthur said:
I’ve blocked AI bots from scraping my blog because I don’t want my content indexed by them. I see them as a way for big tech to exploit creators by profiting off our work without compensation.
Could you share how to block those bots?
@Lucypiper
It’s simple if you’re using a CDN like Cloudflare or QUIC Cloud. In Cloudflare, navigate to Security → Bots and enable both Bot Fight Mode and Block AI Bots. On QUIC Cloud, you can add them to the User Agent block list by typing in CCBot, GPTBot, etc. I’ll write a detailed article on how to do this for others interested.
@Arthur
Thank you! I’ll look out for your post on this.
Arthur said:
I’ve blocked AI bots from scraping my blog because I don’t want my content indexed by them. I see them as a way for big tech to exploit creators by profiting off our work without compensation.
I can relate to that. I’ve been writing for years and have survived the decline of print media without wanting to engage with AI. It’s sad to think our creativity might be overshadowed. Talking to my tech-savvy husband, I understand that AI search engines and shady databases scraping our content are separate. ChatGPT functions somewhat like Google. But even so, I’m concerned about the potential negative effects of indexing my content in the future.
@SHAWN
You’re correct. AI search engines utilize data scraped by bots. ChatGPT uses its scraper (like CCBot, which is also used by Google Bard). Some top scrapers run by companies like DeepAI can be blocked, but what about lesser-known scrapers? These operate in a grey area and are more problematic. While the recognizable operators might purchase data scraped by shady bots, you end up in a tricky situation with Google indexing. If you avoid indexing, you limit potential growth, but if you do, who knows what happens to your data?
@Arthur
Thanks for the in-depth explanation. The complexities really show how tricky this all is. It makes me wish I could rewind the internet a decade!
@SHAWN
ChatGPT search actually uses Bing as its data source. It pulls from what Bing’s web crawlers index.
Danielle said:
@SHAWN
ChatGPT search actually uses Bing as its data source. It pulls from what Bing’s web crawlers index.
That’s fascinating! So, if my blog is indexed on Bing, does that mean it is also on ChatGPT Search? Would it be suggested to users searching for blogs in my niche, or does it just scrape certain content to summarize?
@SHAWN
Any content indexed by Bing shows up in ChatGPT search results if someone is looking for related content.
Danielle said:
@SHAWN
Any content indexed by Bing shows up in ChatGPT search results if someone is looking for related content.
Thanks for clarifying!
I’ve recently seen some traffic from ChatGPT and Perplexity according to my analytics. Not a huge amount—about 15-20 visits—but it’s a new occurrence so I’ll have to track it.
Ameliascarlet said:
I’ve recently seen some traffic from ChatGPT and Perplexity according to my analytics. Not a huge amount—about 15-20 visits—but it’s a new occurrence so I’ll have to track it.
That’s good to hear! 15-20 is better than nothing. How long have you been indexed? Do you worry about any potential drawbacks?
@SHAWN
I noticed the traffic about a month ago, with sporadic visits at first. With the increase, I wonder if some strong keywords are being siphoned off by AI search engines. But right now, I don’t mind how they find my site as long as they do.
@Ameliascarlet
I appreciate your perspective! It sounds like you joined at a good time without overwhelming traffic. You can differentiate organic traffic from ChatGPT’s influence. I’m also new to blogging and want to assess whether ChatGPT Search is worth it for me. Will you update us on your traffic patterns in the coming weeks?
@SHAWN
Definitely! I’ll keep tracking it and share my insights in a few weeks.