Rogue Robots and robots.txt
Most people who have had business websites for even a short time know that the robots.txt file contains statements to tell search engine robots which directories and/or pages not to access.
So-called legitimate search engine robots follow the directives in the robots.txt file.
Rogue robots might use the robots.txt file to determine where your secret pages are. Which must have tasty information, because you're trying to hide it (may be the thinking of rogue robot handlers).
So, what does a person do?
One solution is to list a directory in the robots.txt file. Legitimate robots will stay away. Expect rogue robots to enter.
In that directory, put an index.html file with any non-confidential content, or no content, so long as the file is there. That will prevent rogue robots from getting a directory listing.
Now, in that directory, you can create secret directories with names unlikely to be guessed. The secret directory names are seemingly random combinations of characters. The Password Generator could be used to come up with good/strong secret directory names.
This protection by obfuscation can be very effective protection from rogue robots and even from snooping humans.
Content can be put in those secret directories without fear they will be discovered by rogue robots or their handlers.
Will Bontrager
Was this blog post helpful to you?
(anonymous form)
All information in WillMaster Blog articles is presented AS-IS.
We only suggest and recommend what we believe is of value.
As remuneration for the time and research involved to provide quality links,
we generally use affiliate links when we can.
Whenever we link to something not our own, you should assume
they are affiliate links or that we benefit in some way.
Recent Articles in the Library
Characters for Hyphenation
The CSS hyphenate-character property can be used to tell the browser an alternate character or set of characters should be used where a hyphen would be inserted at end of lines.
Automatic Wrap Balancing for Headlines
Use the CSS text-wrap:balance; property to better balance the line lengths of multi-line headlines.
Redirect With Method POST
When you can't use method POST, but you must, read this.
Gradient-colored Text
The information you need to make color-gradient text.
The HTML Q Tag
Create a style then use the HTML q tag to apply the style. This is about as easy as it gets for attracting attention to quotes.
Plain Text To HTML Converter
Here is a function to convert plain text from a form textarea field into HTML for publishing on a web page or in an email.
HTML for Meme Making
No fancy image creation software is needed to make memes. An easy way is to create a web page with text over an image or color background and screenshot it.
