If you are tired of referrer spam filling up your logs with complete crap here is a very simple way to isolate the spam bots so you can later remove the information from your log files and then only pull out the correct information.

Somthing that I realized while reading another post is a spam bot trying to post garbage on your web site must have to ignore redirects. Otherwise people could do all sorts of horrible tricks like redirecting the spam bot to a tar pit or very large files or particullary nasty html pages aimed to crash them. Its probably worth pointing out that this only works for some of the spam bots out there. Not all of them.

This works in a really simple way. The majority of spam bot's since they are making request to post comments also attempt to post referrer spam along with their first get request to the web server. This can be exploited since it will not be followed.

The method to exploit this is simple. A browser with a real user will follow the redirect. So why not use the referrer url and the ip address the request is coming from to redirect the request to the same url again that the browser will follow. This will cause the spam bot to stop processing but allow normal users to continue on the path. Once a single user has made it past this barrier you have then identifiyed a valid referrer url and can skip the checking on other users. The browsers help with this somewhat because they will carry the correct referrer url across the redirect onto the 2nd request (the request after the redirect)

When processing the log files on the web server all that is now required to to remove all references to the redirects and the log files now have the spammer information removed.

I have also considered the impact on a few other thigns while writing this.

• Search engines don't provide a referrer references when making requests.
• People who disable the browser referrer also will not have a problem.

I put the following asp.net httpmodule together to exploit this weakness in the spammers bots. You will of course have to modify it to ignore your own site url

public class RefAntiSpam : IHttpModule
{
	//Use Cache To Form AN IP + Refferer to perform a redirect
	private Dictionary<string, DateTime> Cache = new Dictionary<string, DateTime>();

	public void Init(HttpApplication App)
	{
		App.BeginRequest += new EventHandler(App_BeginRequest);
	}

	public void App_BeginRequest(object sender, EventArgs e)
	{
		HttpRequest Request = HttpContext.Current.Request;
		HttpResponse Response = HttpContext.Current.Response;

		if (Request.HttpMethod == "GET")
		{
			if (Request.UrlReferrer != null)
			{
				if (Cache.ContainsKey(Request.UserHostAddress + "-" + Request.UrlReferrer.OriginalString) == false &&
					Request.UrlReferrer.OriginalString.Contains("stev.org") == false &&
					Request.UrlReferrer.OriginalString.Contains("localhost") == false)
				{
					Cache[Request.UserHostAddress + "-" + Request.UrlReferrer.OriginalString] = DateTime.Now;
					Response.Redirect(Request.Url.OriginalString, true);
				}
			}
		}
	}

	public void Dispose()
	{
		
	}
}

I was previously trying to prevent comment spam by blocking access by ip address. However it does appear that this really isn't very suitable. The amount of comment spam did drop from around 500-600 items per day to around 40 - 60. So it just is not effective enough and you also run the risk of blocking valid users from accessing the content on the site.

I have now come up with a new method which is a much safer and does not require the overhead of using a database. I decided to look into the web server logs and noticed that the clients are bots and not really web browsers they only request then postback a valid page in an attempt to get the comments to appear on the site. So this new approach uses this to an big advantage.

More...

Recently I seem to be running into a little bit of a spamming problem with backlink's being submitted to the blog. It looks like the way .net blog engine does post is really easy for the spammer to be able to post lots and lots of comments with bots (I guess the spammers are getting good at this now). This sounds quite bad but everything is being caught by the spam filters so its really not so bad. So this is a bit of a guide to attempt to protect a website from such action coming from abusive computers around the internet. More...

It looks like Dell has been a little dirty with their mailing lists. Using my old email address. Over the recent months since about chrismas (2010) dell started sending spam in my general direction. Under the UK data protection laws this should not be possible since they have to assure that personal information is accurate. Which of course it is.

However it gets worse. Since I hit an a unsubscribe link. The offical dell emails appeared to stop and then more dell emails started to suddenly come from somebody calling them selves Digital UK in the email header. Of course the unsubscribe from these emails does not appear to work are being sent from random servers and are also being sent from rotated domain names. Which in my terms means they are a straght up spammer since they are attempting to get around filters.

The people calling them selves Digital UK appear to also now be sending more spam for other companies. This only started sometime after the first dell email was recived. So I am adding dell to my list of dirty companies who are incapable of dealing with data in a legal and correct matter and of course adding them to the list of companies I would prefere never to deal with again.

I am posting this because it is probable the most stupid spam I have ever seen. do people not check their bulk email for usability before they send it out these days?

There is no no text no rendered images no links. Really there is nothing I can do with the email!