Recently I have been working on a PHP script for tracking our 404 error page so that we may better direct users who request pages or blog posts that may no longer exist to relevant existing content. The reason for this experiment was because during the development and launch of the new Response Marketing site I found a large number of requests coming through for nonexistent content. Also a couple months ago one of our client’s Facebook tab was down during a critical point in a negotiation for new work and none of the pre-formatted alerts systems like www.pingdom.com would work for individual 404 requests. Most of the services offered by sites like www.pingdom.com only offer alerts on errors like “HTTP Error 503 – Service unavailable” and “HTTP Error 500 Internal server error“, basically only the 500s.
What About the 400s?
me may not know what I’m talking about, so let me explain.
A 5xx error (better known as a server error) means that the client side has made a valid request and the server cannot fulfill it or the server is inaccessible. A 4xx error (you may have heard of the 404 error before) means that the server is accesible and the client side could have been the cause of the error. I say “could” because it is more than likely but not absolute.
It is best SEO practice to have an error document for all 400 errors, or a dynamic error page showing the user a custom error message based on the bad request. Having this readily available will show Big Brother (Google, Yahoo, etc…) you realize there may be something missing and you’re working on it. That said, you must constantly monitor your 400 errors to know what needs fixing–thus the reason for building a script to track the requests.
The Experiment
Initially I started by just adding a php mail () function to the 404 document to send me an email whenever someone, something (bots), some site (hot linkers) requested something that wasn’t there. The email contained the HTTP_REFERER and the REQUEST_URI so that I could see where the link was accessed from and the file extension trying to be accessed. The cool part about it was that based on the day of the week it would either email me or text me.
Needless to say that due to the length of time thepowertoprovoke.com has been our domain there were a lot of missing files and documents being requested. Within the first week I had over 900 emails come in for 404 errors, each of which had to be fixed with 301 redirects so that the request would be directed to a corresponding document.
Almost Hacked
Last Saturday my email exploded–I had over 200 emails come in from the 404 alert in 30 minutes. Before the emails stopped I added some info to the email alerts that would display the client request IP so I could really see who was doing this. It ended up being a spider looking for Javascript files with loose permissions, and was also sending so many requests that it risked crashing the server. I did some research and found that the IP was owned by Adaptive Spider Technologies and that they had quite a range of IP addresses being used to exploit servers. I got as many IP addresses as I could find for them and blocked them using the “order allow, deny” Access Control command.
Geo Location
The idea then came to add some geo location functionality to the 404 tracker and integrate it with a database so that we could display a map of all the requests coming in. The map displays pins for each 404 document request and in the info bubble displays the IP, latitude, longitude, country, state, city, area code, referring URL, requested URI, and a link to Google Maps querying all the businesses in that location. I have also created a script to build a KML document based on a MySQL database query of the stored results, which can then be imported to Google Earth and will display colored pins based on the number of visits by that IP.
(p.s. Here is some of your info:)
Your ip: [geoip]
Your ISP: [geoisp]
Your City, State and Area Code: [geocity][geostate][geoareacode]
[geogooglemap width=”500″ height=”300″ results=”8″ miles=”50″]
More Than Just Error Reporting
This system has more potential than just error reporting. It could be a great addition to an analytics system if implemented on main pages or conversion pages of a website. If you have a business IP database it could be used as a tool for a proactive sales team–they could use this information to actively engage someone that has already shown interest.
More than just Error Reporting
In closing I think I have come to the resolution that this could be more than just error reporting. It could be a great addition to an analytics system if implemented on main pages or conversion pages of a website. If you have a business ip database it could be used as a great to for a proactive sales team, using this information to actively engage someone that has already shown interest could lead to more sales.