Cloudflare Crawler Hints Caused Site to De-Index in Bing

How turning-off crawler hints helped to get site indexed in Bing

After year of enabling Cloudflare Crawler Hints which uses IndexNow to send every URL to Bing to be crawled and indexed actually has one disadvantage which caused my website to deindex in Bing. This happened because Cloudflare crawler hints was sending even non-existant URLs to Bing search engine. Bots visiting my site via cloudflare were requesting non-existant URLs and so each hit was been sent to Bing. Crawler Hints was sending 2K to 3K URLs to Bing on daily basis. Over 500 URLs were returning 404 error page and Bing then started to de-index the site. First it stopped crawling new pages (actual posts published) and later it took the entire site out from its index.

Turning-off crawler hints in Cloudflare helped

After reading many posts on reddit, quora, and even cloudflare support forum, I found that disabling crawler hints inside Cloudflare is going to help as it will stop sending un-necessary URLs to Bing. Also, my crawl rate and crawl quota will be saved. After disabling crawler hints I simply added Indexnow plugin and it only sent signal to Bing when only new post were published or old posts was updated. After a month, I could see my website appearing back in Bing. Not only that, Bing started crawling all new posts too!

If you are too facing such issue on Bing search, and you are using Cloudflare crawler-hints then better disable it for a while to solve the indexing and crawling issue in Bing.

Social Share or Summarize with AI

Similar Topics

  • The Savala Vada Instagram Profile BANNED

    Satire, parody profile The Savala Vada banned in India An Instagram page named The Savala Vada (@thesavalavada) which posts about satire and parody is now…

  • RackNerd Black Friday Deals 2024

    RackNerd VPS, Cpanel Hosting, Reseller Hosting Deals This year 2024 Racknerd.com is offering the best deals on VPS, shared hosting, reseller hosting, dedicated servers and…

  • Secureserver.net Domain Registration 50% OFF

    Register domain name at secureserver.net at 50% discount today! Secureserver.net is a leading domain name registration provider since last 20 years! Now you can register…

  • Offerwall in Google Ad Manager

    Google Ad manager Offerwall is now generally available Starting today Google Ad manager (GAM) is making Offerwall generally available for all users. Offerwall helps publishers…