Bot traffic is non-user traffic that can hurt your web performance significantly. It is essential to safeguard your website from abusive bot traffic and, there are a few ways to do it. In this article, you will learn to block bots using the robot.txt file with your cPanel account.
Located in the root directory of your website, robot.txt is a text file that enables the search engine bots to crawl and index your website. Apparently, it also decides what files should be accessible to the bots making its configuration essential. It is also why you can restrict these bots from indexing your website in the search engines using the robot.txt file.
To Block Bots Using Robot.txt File
1. Log in to your cPanel account.
2. Go to the FILES section and click on File Manager.
The File Manager interface will open. You will have to create the robot.txt file.
3. Click the +File button on the top left corner of the interface.
4. Enter the file name as robot.txt and click the Create new file button.
5. Locate and right-click on the robot.txt file. Select the Edit option.
The file will open for you to enter/edit its information.
6. Copy this code in the file.
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Baiduspider [NC]
RewriteRule .* – [F,L]
This code will block the bot from the Baidu search engine.
7. Click the Save Changes button and Close the file.
With this robot.txt file, the Baidu search engine cannot use bots on your website.
You can also add the bot crawler of any other search engine to the code preventing it from indexing your website.
In short, you can even block bots from other search engines with the robot.txt file by editing it via cPanel File Manager.
