Master 404-Terminator™

M A N U A L

Requirements

To install and use Master 404-Terminator™, your domain server must be Unix or Linux and have Perl version 5+ installed. Further, CGI programs must be able to create subdirectories and to create and update files in those subdirectories. You may use Master Pre-Installation Tester to test for the ability to create/update directories and files.

Use an FTP program to install Master 404-Terminator™. CuteFTP and WS_FTP for Windows, and Transmit for Macintosh, are among the many FTP programs that allow you to CHMOD (change directory and file permissions). FTP programs are all listed at http://tucows.com/

List of Files

{top}

Here is a list of the files you should have received when you generated Master 404-Terminator™.

If you did not receive all of the above files, please contact us at the technical support URL in the README.txt file — include a list of the files you did receive and the domain name Master 404-Terminator™ was generated for.

Getting Started

To install Master 404-Terminator™, upload these three program files

Master404T.cgi
Master404Tcp.cgi
Master404T.pm

You may wish to make a subdirectory in your cgi-bin within which to install Master 404-Terminator™, a subdirectory named 404, for example. Installing it in its own directory can provide a better sense of organization when you have or anticipate having other programs in your cgi-bin.

The files must be uploaded with an FTP program and uploaded as plain text files — not as binary files. Once uploaded, give each file 755 permissions (see Setting Permissions).

Once the program files are installed, use the control panel's Configuration page to obtain the information for your .htaccess file. If you wish, you can have Master 404-Terminator™ update the .htaccess file for you.

If you decide to do it yourself —

Check whether or not the document root directory on your server contains a file named ".htaccess" (without the quotes). The "document root directory" is where your main index page is located, the page that loads when you type only your domain name URL into your browser's address bar.

If the server contains the .htaccess file, download it as a plain text file.

Note that some servers are configured not to display the .htaccess file name in FTP programs. If your server is configured that way, you can download a copy of the .htaccess file by manually typing in the file name for your FTP program to download. (If your FTP program doesn't allow you to do that, pick one up that does. See Requirements.)

If your document root directory does not contain a file named .htaccess then you'll need to create one.

On the control panel's Configuration page is an example of the line of text you'll need to add to your .htaccess file.

If your .htaccess file already contains a line that begins with the words
ErrorDocument 404
replace that line with the line from the control panel. Otherwise, paste the line from the control panel at the top of the .htaccess file.

The URL in the line you pasted in might need to be edited, but probably not. Master 404-Terminator™ does a good job of guessing, getting it right for most servers. (The correct URL would be the URI portion, which is the URL to Master404T.cgi except with "http://" and domain name parts removed. What's left over is what the .htaccess line requires.)

Upload the new or edited .htaccess file to the document root directory on your server. Upload it as a plain text file.

Here are a few installation notes:

cgi-bin |
        | 404 |
              | m404Tdata

Setting Permissions

{top}

Permissions can be set via telnet or with some FTP programs.

Some servers require permission 766 or 777 for directories that will write files or for the files themselves. If a program creates its own directories and files, don't change the permissions unless you know what you're doing. (In other words, if it works like it should then don't fix it.)

(On some servers, the directory where the scripts are installed needs to have 777 permissions before the scripts can create data directories — if you need to do that, install the scripts in a separate subdirectory and then give that subdirectory 777 permissions.)

Here are the permission numbers translated into defining attributes:

Some FTP programs use the term "Other" or "Global" instead of the term "World" as used above.

The Control Panel

{top}

To access the Webmaster's control panel, type the URL of Master404Tcp.cgi into your browser.

If you installed Master404Tcp.cgi into a directory named 404, then the URL might be something like this:

http://example.com/cgi-bin/404/Master404Tcp.cgi

The first time you access your control panel, you will be asked to specify a password for future access.

Once access is granted, the fact is stored in a cookie on your browser until either one of these happens:

• You click the "Log Off" link on the control panel.
• You close your browser.

When either of the above happens, the cookie is removed from your browser. (Cookies must be turned on to use the control panel.)

After you specify a password for future access, use the Configuration link first. See below.

Configuration

{top}

This section of the control panel is for determining the line for the .htaccess file and for specifying vital information Master 404-Terminator™ needs to operate.

.htaccess File

You'll need to know the location of your .htaccess file (the location where you want it, it doesn't actually have to exist prior to now), and you'll need to know the URL of the 404 handler script.

Location of .htaccess file

Probably most installations will want to have the .htaccess file in the document root directory, which is the directory where your domain's index/main web page resides.

If that's true for you, you can leave the control panel's relevant form field blank. Otherwise, type the subdirectory name (or subdirectory path if the .htaccess file will be several subdirectories below the document root).

URL of the 404 handler script

The URL of the 404 handler script would be the URL of Master404T.cgi without the domain name part. If the browser URL of Master404T.cgi were
http://example.com/cgi-bin/404/Master404T.cgi
then the URL for this control panel form field would be
/cgi-bin/404/Master404T.cgi

If you wish to have Master 404-Terminator™ update your .htaccess file (or create it, if necessary) check the relevant checkbox prior to clicking the "Update..." button at the bottom of the control panel page.

To manually update your .htaccess file, see the Getting Started section of this manual.

Domain URL and Redirect Defaults

If the form fields in this section are not filled in, 404-Terminator™ will make some assumptions.

The first field is for the URL of the domain where 404-Terminator™ is installed. If left blank, http://example.com is assumed.

The second form field is for the default redirection URL, where 404's are redirected to when an entry for the 404 URL has not yet been put into the redirection database. If left blank, http://example.com/ is assumed.

The third is a dropdown list to select the default redirection status code to send to browsers when an entry for the 404 URL has not yet been put into the redirection database. If nothing else is selected, "301 Moved Permanently" is sent.

404 Management Maximum Listing

The Manage Current 404s control panel page lists the 404 URLs that had no entry in the redirection database (and were redirected to the default redirection URL with the default status code).

If you have many unmanaged 404 URLs, the large list could be daunting. A very large list could actually be more than the browser or computer resources can easily handle.

If you want to limit the number of 404 URLs on the Manage Current 404s control panel page, specify the maximum number that may be listed.

Manage Current 404s

{top}

This control panel page lists the 404 URLs that had no entry in the redirection database (and were redirected to the default redirection URL with the default status code as specified at the Configuration control panel page).

For every unique 404 URL, you'll see:

1. A dropdown list of IP addresses and a colored bar.

   The first item in the dropdown list of IP addresses contains a number representing the number of times that particular 404 URL was used. Below the first item is a list of each unique IP address involved with the 404 URL, along with the number of times the IP address was involved if more than once.

   To the right of the IP address dropdown list is a colored bar for visualizing the number of times that particular 404 URL was used.

2. The 404 URL (minus the domain name part).

   The 404 URL is in an editable form field. (See later in this section for editing suggestions.)

   To the left of the 404 URL form field is an "X" between square brackets. If you click on the X, the 404 form field will become blank. If you click on it again, the 404 form field will be restored to the way it was when the control panel page was loaded into your browser.

3. Two radio buttons, one with a dropdown list.

   The radio button options are "replace" and "redirect".

   replace allows you to replace a file, sending the file directly to the browser instead of redirecting to another URL. Examples of when "replace" might be used are 404 URLs to image, CSS, or JavaScript files. (Sending the file directly to the browser allows the browser to use the image, etc. to render the web page being loaded.)

   redirect allows you to redirect the browser to another URL. When the redirect happens, the dropdown list is used to select the type of status code that shall be sent to the browser at the time of the redirect.

   The status code dropdown list contains:

   1. 301 Moved Permanently
   2. 302 Found
   3. 307 Temporary Redirect
   4. 403 Forbidden
   5. 410 Gone

   (See later in this section for more comprehensive instructions.)

4. A "Location" form field.

   This form field is used to specify the location of the replaced file or the redirect URL.

Editing suggestions and more comprehensive instructions.

First, let me mention that at the top of the list of 404 URLs on this control panel page is a "default" section. Here, you can specify a default radio button, status dropdown list, and/or location. This may save some time if you have many 404 URLs to deal with and most to be handled the same way.

Once a 404 URL has been managed, it is removed from the unmanaged database. The 404 URL will in most cases be copied into the redirection database, the database Master 404-Terminator™ uses when determining the disposition of 404 URLs it is asked to handle.

Here are things you can do with each unique 404 URL:

1. The 404 URL.

   This may be edited.

   If the 404 URL is to a directory that no longer exists, or if you want to specify all of the same class of files (all .html or all .gif, for examples), then one or more asterisk wild card characters may be used. When the 404 URL is copied into the redirection database, it can have an effect on many 404 URLs.

   See the Wild Cards section of this manual for instructions on how to use asterisk wild cards.

   If you make this field blank, the 404 URL record will be removed from the unhandled database and will not be copied into the redirection database. The 404 URL will just disappear (unless it's used again). This is a way to deal with 404 URLs that are unlikely to reoccur.

   To the left of the 404 URL form field is an "X" between square brackets. If you click on the X, the 404 form field will become blank. If you click on it again, the 404 form field will be restored to the way it was when the control panel page was loaded into your browser.

   Always omit the http://example.com part of any URL you specify here.

2. Two radio buttons, one with a dropdown list.

   Here, choose between

   • replacing the 404 URL with a file sent directly to the browser or
   • redirecting the browser to the correct URL.

   Click the radio button of your choice, "replace" or "redirect".

   replace replaces a file by sending it directly to the browser instead of redirecting the browser to another URL. Use this for files that are inserted into web pages, such as image, CSS, and JavaScript files.

   Commonly, it is more appropriate to replace missing non-web page files by sending the replacement directly to the browser than it is to redirect browsers to a new URL. Exceptions would be files meant to be viewed directly, such as text or image files not meant for display from within an HTML coded web page.

   redirect redirects the browser to another URL. Use this to point the browser to a different web page.

   When redirecting, use the dropdown list to select the type of status code that shall be sent to the browser at the time of the redirect.

   The options are:

   1. 301 Moved Permanently
   2. 302 Found
   3. 307 Temporary Redirect
   4. 403 Forbidden
   5. 410 Gone

   The status code you select will be sent to the browser.

   The last two on the list do not require a redirect URL in the "Location" form field. When "403" or "410" is selected, a web page with an appropriate message is sent to the browser instead of redirecting it.

3. The "Location" form field.

   This form field is used to specify the location of the replaced file or the redirect URL.

   For the replace file —

   Specify either the server location of the file or its URL.

   The server location will in most cases be faster.

   The server location can be the absolute server location or it can be the location relative to the document root. If relative to the document root, it would be the file's URL without the domain name part. For example, "http://example.com/" would be specified as "/" and "http://example.com/subdomain/" would be specified as "/subdomain/"

   However, sometimes an http://... URL must be used, such as when

   • the replacement file is not on domain example.com or
   • the file to be replaced is generated on demand (like a CGI program for example).

   If an http://... URL is used, Master 404-Terminator™ will retrieve the file using the HTTP protocol that browsers use. This means the file must be available to public browsers, or Master 404-Terminator™ will retrieve an error message page.

   The Master 404-Terminator™ subroutine that retrieves these pages with http://... URLs is not optimized for connections with secure servers or getting into password protected directories, and it pays no heed to JavaScript or frames. If you're uncertain about whether or not a URL can be retrieved, you can use Master Snooper to try and retrieve it. If Master Snooper can get the page, the URL is good.

   For the redirect URL —

   If the "Location" is on the example.com domain, specify the URL without the domain name part. For example, "http://example.com/" would be "/" and "http://example.com/subdomain/" would be "/subdomain/"

   However, if the "Location" is not on the example.com domain, specify the complete http://... URL. For example, if someone tries to load a nonexistent "goodscripts.html" page, you might redirect them to URL "http://willmaster.com/master/?MSAC" (without the quotes, of course), where "MSAC" is your Master Series Affiliate Code.

   If the dropdown list specifies status code "403 Forbidden" or "410 Gone" then no redirect URL is required, and will not be used if provided.

Edit Redirect URLs

{top}

Here is where you'll find the 404 URL/redirect URL pairs existing in the redirection database. These may be edited.

Before viewing the contents of the redirection database, you'll have the opportunity to specify a range of URLs to view. The range can be a range of 404 URLs or a range of redirect URLs.

If you leave these blank, the entire redirection database will be displayed.

When the redirection database is large or when you want to have only a few on the edit page, specify a "from" range and a "to" range.

URLs beginning with what's specified in the "from" field or beginning with what's specified in the "to" field, or falling between those two ranges, will be displayed on the edit page,

To edit any of the URLs, simply re-type the relevant form fields and/or re-select from the radio buttons/dropdown list. See Wild Cards for example URLs using asterisk wild cards.

To delete a 404 URL/redirect URL pair, delete the 404 URL. This can be done easily by clicking the "[X]" on the left of the 404 URL form field. (A subsequent click on the "[X]" will restore the 404 URL form field with the value it contained when the control panel page was displayed.)

Add Redirect URLs

{top}

If you know or anticipate certain URLs will be 404s, you can list them here along with their redirect URLs. See Wild Cards for example URLs using asterisk wild cards that can make batch redirections easy and quick.

Simply type the URLs into the relevant form fields, check a radio button and, if applicable, select from the dropdown list.

Export Databases

{top}

Three files can be exported:

1. The log file, a record of all 404 handling Master 404-Terminator™ performs.
2. The redirection URLs database.
3. The unassigned 404 URLs database, those 404 URLs that have not yet been managed.

Check the database you wish to export.

The export will be a flat file database and can be suitable for importing into desktop spreadsheet/database programs. The delimiters can be

1. Tab-delimited, a format most spreadsheet/database programs can import.
2. Comma-delimited (also called "CSV" for "comma separated values"), a format most spreadsheet/database programs can import.
3. Other, allowing you to specify which character or set of characters to separate the fields with.

Check the delimiter you prefer.

The exported file can be saved anywhere on your server.

If you specify only a file name, the file will be saved in the m404Tdata directory. If you prefer the file to be saved somewhere else, specify the server directory path with the file name.

Once the file has been exported and saved on your server, it can be downloaded with an FTP program (download as plain text), imported into a spreadsheet/database program, and manipulated according to the abilities of the program it was imported into.

Change Password

{top}

When you need to change the password, use this control panel page.

Log Off

{top}

When you're ready to log off, use this control panel page. Logging off will cancel the log-on cookie.

Wild Cards

{top}

The primary reason for using asterisk wild cards is to make less work for yourself.

The asterisk in 404/Location URLs means "zero or more of any characters". Thus,
/books/*
will match any of these:

/books/
/books/any.gif
/books/any.html
/books/anything

Now, if
/books*
were used, instead, it would match all of the above, plus

/books
/books.html
/booksale.html
/bookstuff/
/books/and/such/index.php

Whatever characters are before and/or after the asterisk must match exactly. At the spot where the asterisk is, any or no characters will match.

How to use it, a directory move scenario —

Let's suppose you moved everything in the /books/ directory to the /mybooks/ directory, and all browser requests to that directory should be pointed to the new directory.

For that, use
/books/*
as the 404 URL and
/mybooks/
as the Location.

Consideration — pointing to another domain:

The Location URL could also be to the allmybooks.com domain, with URL
http://allmybooks.com/
instead of a URL to another directory on the same domain.

Consideration — pointing to the same file in the new directory:

If the file names in the new /mybooks/ directory have all the same files that the previous /books/ directory did, an asterisk can be used in the Location URL, too.

When an asterisk is used in the Location, the asterisk will be replaced with whatever characters (if any) were matched with the asterisk in the 404 URL. In other words, what's matched in the 404 URL is mapped into the Location.

If this 404 URL and Location were used

 404 URL: /books/*
Location: /mybooks/*

then 404 URLs like
/books/uniquepage.html
will be redirected to
/mybooks/uniquepage.html.

Consideration — when there are many types of files:

Let's suppose your previous /books/ directory, in addition to HTML web page files also contained image, JavaScript, and CSS files. Let's further suppose the browser should be pointed to the same file in the new /mybooks/ directory.

The web page files will need a browser redirect. The image, JavaScript, and CSS files will need to have the file contents sent directly to the browser instead of redirecting. (Sending the file directly to the browser allows the browser to use the image, etc. to render the web page being loaded.)

To accomplish the objective, let's use an asterisk as the first part of the file name for the 404 URL and map the matched characters into the same position of the Location. This way, the different file types can be managed separately.

 404 URL: /books/*.html
          redirect with 301 Moved Permanently
Location: /mybooks/*.html

 404 URL: /books/*.gif
          replace
Location: /mybooks/*.gif

 404 URL: /books/*.js
          replace
Location: /mybooks/*.js

 404 URL: /books/*.css
          replace
Location: /mybooks/*.css

and so forth. In all cases, the characters matched with the asterisk in the 404 URL are mapped into the Location.

How to use it, other examples —

The "directory move" example, above, contains several examples within it. Here are other examples, which didn't easily fit into the above scenario as presented.

Example — site-wide .html to .shtml web page file name extension change:

If you're changing all web page file name extensions from .html to .shtml and want to let visitors and search engine robots know about it when they try to access the now absent old pages, use

 404 URL: *.html
          redirect with 301 Moved Permanently
Location: *.shtml

Example — missing favicon.ico file:

If you don't have a favicon.ico file in the document root directory, calls for it can be replaced with another image file or even a favicon.ico file on one of your other domains. (This example doesn't really belong in this section about wild cards, but I didn't immediately see where else to put it.)

To replace with another image file (which should be the format expected by the browser):

 404 URL: /favicon.ico
          replace
Location: /images/other.gif

To replace with a favicon.ico file located on one of your other domains:

 404 URL: /favicon.ico
          replace
Location: http://myotherdomain.com/favicon.ico

Example — missing robots.txt file:

If you don't have a robots.txt file in the document root directory, call for it can be replaced with a properly formatted text file, with a blank text file, or with a robots.txt file on one of your other domains. (This example doesn't really belong in this section about wild cards, but I didn't immediately see where else to put it.)

To replace with another text file, either blank or properly formatted:

 404 URL: /robots.txt
          replace
Location: /other.txt

To replace with a robots.txt file located on one of your other domains:

 404 URL: /robots.txt
          replace
Location: http://myotherdomain.com/robots.txt

Example — missing image (and other non-HTML) files:

Commonly, it is more appropriate to replace missing non-web page files by sending the replacement directly to the browser than it is to redirect browsers to a new URL. Exceptions would be files meant to be viewed directly, such as text or image files not meant for display from within an HTML coded web page.

These examples all assume replacement, although redirection can be specified instead.

If you've replaced all *.gif files with *.png, retaining the same file location and name (except the file name extension), this will work:

 404 URL: *.gif
          replace
Location: *.png

If you've moved all your images to a different domain (maybe to balance server loads), in the same directory paths, this could handle the 404 URLs until all your web pages can be updated:

 404 URL: *.gif
          replace
Location: http://imagedomain.com/*.gif

 404 URL: *.jpeg
          replace
Location: http://imagedomain.com/*.jpg

 404 URL: *.jpg
          replace
Location: http://imagedomain.com/*.jpeg

 404 URL: *.png
          replace
Location: http://imagedomain.com/*.png

If you've moved all the JavaScript files that used to be in the root directory to a new directory just for those files, this can help:

 404 URL: /*.js
          replace
Location: /jsdir/*.js

Example — guessing games:

Sometimes surfers will guess the location of pages rather than take the time to follow a navigation thread. Here are a few examples of how these might be handled.

These examples all assume replacement, although redirection can be specified instead.

If you've replaced all *.gif files with *.png, retaining the same file name (except the file name extension), this will work:

 404 URL: *contact*
          redirect with 301 Moved Permanently
Location: /contactus.html

 404 URL: *feedback*
          redirect with 301 Moved Permanently
Location: /contactus.html

 404 URL: *about*
          redirect with 301 Moved Permanently
Location: /aboutus.html

Backing Up Data Files

{top}

For backups, it is suggested that the databases be exported and downloaded to your computer. See Export Databases.

A data import function is in the works for the next upgrade of Master 404-Terminator™. The exported databases could then be used to restore database files if the need arose.

It is suggested that backups be made regularly, especially the redirection database, which could contain hundreds or even thousands of redirection URLs.

Technical Support

{top}

For technical support, please contact us at the technical support URL in the README.txt file.

Use the same URL to inquire about professional Master 404-Terminator™ installation services.

Thank you for your purchase of Master 404-Terminator™.

Copyright 2004 Bontrager Connection, LLC