Send Inputs and Feedbacks on Crawl-Delay in Bing

Bing is looking for input on Crawl-Delay:

In keeping with Bing themes of rapid change and responding to input and feedback from the webmaster community, Bing webmaster teams are working on future feature planning for the Bing webmaster tools and they would like to hear user and webmasters thoughts in a few areas. They starts from Crawl-Delay.

Today, Bing fully support crawl delay which is specified in robots.txt.

Now Bing are considering creating a tool which allows to directly control the frequency, and perhaps even the timing, of msnbot/bingbot.

A few questions from Bing Team:

  1. Do you prefer for us to continue to support crawl-delay in robots.txt?
  2. Would you like to be able to control the crawl rate based on time of day?
  3. What are the ranges of crawl rate you would like to see if you had control over the time of day?
  4. Would you want to turn crawling on/off by time, or just specify different crawl rates based on the time of day?
  5. If you had this ability to directly control the crawl rate through a webmaster tool, how often would you adjust the settings?
  6. If we built this tool, would you like it to control our fetching directly, or would you rather have the tool output the proper robots.txt entries which you then insert into your robots.txt file?

Feel free to send reply to this post on email to bwmc@microsoft.com

( Source : Bing Webmaster Central Blog )

Scroll to Top