Blocking Referrer Spam

1. October 2011 09:10

If you are tired of referrer spam filling up your logs with complete crap here is a very simple way to isolate the spam bots so you can later remove the information from your log files and then only pull out the correct information.


Somthing that I realized while reading another post is a spam bot trying to post garbage on your web site must have to ignore redirects. Otherwise people could do all sorts of horrible tricks like redirecting the spam bot to a tar pit or very large files or particullary nasty html pages aimed to crash them. Its probably worth pointing out that this only works for some of the spam bots out there. Not all of them.


This works in a really simple way. The majority of spam bot's since they are making request to post comments also attempt to post referrer spam along with their first get request to the web server. This can be exploited since it will not be followed.


The method to exploit this is simple. A browser with a real user will follow the redirect. So why not use the referrer url and the ip address the request is coming from to redirect the request to the same url again that the browser will follow. This will cause the spam bot to stop processing but allow normal users to continue on the path. Once a single user has made it past this barrier you have then identifiyed a valid referrer url and can skip the checking on other users. The browsers help with this somewhat because they will carry the correct referrer url across the redirect onto the 2nd request (the request after the redirect)


When processing the log files on the web server all that is now required to to remove all references to the redirects and the log files now have the spammer information removed.


I have also considered the impact on a few other thigns while writing this.


  • Search engines don't provide a referrer references when making requests.
  • People who disable the browser referrer also will not have a problem.


I put the following httpmodule together to exploit this weakness in the spammers bots. You will of course have to modify it to ignore your own site url



public class RefAntiSpam : IHttpModule
	//Use Cache To Form AN IP + Refferer to perform a redirect
	private Dictionary<string, DateTime> Cache = new Dictionary<string, DateTime>();

	public void Init(HttpApplication App)
		App.BeginRequest += new EventHandler(App_BeginRequest);

	public void App_BeginRequest(object sender, EventArgs e)
		HttpRequest Request = HttpContext.Current.Request;
		HttpResponse Response = HttpContext.Current.Response;

		if (Request.HttpMethod == "GET")
			if (Request.UrlReferrer != null)
				if (Cache.ContainsKey(Request.UserHostAddress + "-" + Request.UrlReferrer.OriginalString) == false &&
					Request.UrlReferrer.OriginalString.Contains("") == false &&
					Request.UrlReferrer.OriginalString.Contains("localhost") == false)
					Cache[Request.UserHostAddress + "-" + Request.UrlReferrer.OriginalString] = DateTime.Now;
					Response.Redirect(Request.Url.OriginalString, true);

	public void Dispose()
E-mail Kick it! DZone it! Permalink

ASP.NET - Cannot find System.Web.Razor

1. September 2011 15:08


If you have downloaded a new mvc3 project or a application using razor views and you have recived an error being able to find System.Web.Razor



Line 78:         <add assembly="System.Xml, Version=, Culture=neutral, PublicKeyToken=B77A5C561934E089"/>
Line 79:         <add assembly="Microsoft.Web.Infrastructure, Version=, Culture=neutral, PublicKeyToken=31BF3856AD364E35"/>
Line 80:         <add assembly="System.Web.Razor, Version=, Culture=neutral, PublicKeyToken=31BF3856AD364E35"/>
Line 81:         <add assembly="System.Web.WebPages, Version=, Culture=neutral, PublicKeyToken=31BF3856AD364E35"/>
Line 82:         <add assembly="System.Web.WebPages.Razor, Version=, Culture=neutral, PublicKeyToken=31BF3856AD364E35"/>



If you have the error above it is because you are missing the new razor views from microsoft. Which is avilable to download and install at the following location.


Note: Something that really annoyed me while installing the above is that it decided to crash then attempt to restart windows without any confirmation!

E-mail Kick it! DZone it! Permalink

C# - IsGuid

29. August 2011 08:00
Another simple function in c# which is usefull for validating a guid passed in a url string which may have been modified to try to invoke a bug / issues in an application.
public static bool IsGUID(string expression)
    Regex guidRegEx = new Regex(@"^(\{{0,1}([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}\}{0,1})$");
    return guidRegEx.IsMatch(expression);
E-mail Kick it! DZone it! Permalink

ASP.NET Login by username or email

13. June 2011 19:21

Something that isn't support by default in is the ability to have end users login using either their username or their email address. Though it is somewhat easy to add this support using this little trick to override the normal login procedure on the login control.


To get started we will need our aspx page with a login control as follows.


<asp:Login runat="server" ID="Login2" DisplayRememberMe="false" 
    DestinationPageUrl="/Users/Default.aspx" onloggingin="Login2_LoggingIn" >


We can then override the default action in the event handler if we cannot find the current user trying to login and the username contains an '@' symbol.



public partial class Login : AppPage
    protected void Page_Load(object sender, EventArgs e)


    protected void Login2_LoggingIn(object sender, LoginCancelEventArgs e)
        if (Login2.UserName.Contains("@") && Membership.GetUser(Login2.UserName) == null)
            string Username = Membership.GetUserNameByEmail(Login2.UserName);
            if (Username == null)

            if (Membership.ValidateUser(Username, Login2.Password) == true)
                FormsAuthentication.SetAuthCookie(Username, false);
                FormsAuthentication.RedirectFromLoginPage(Username, false);
                e.Cancel = true;


Note: if you are going to use the above you will need to make sure that your configured provider has a unique set of email addresses per user.

E-mail Kick it! DZone it! Permalink

MSSQL Removing the aspnet membership database

22. May 2011 22:01

If you have ever attempted to remove the aspnet_* tables, views and stored procedures from a database you will know that it is not simply stright forward. It takes a little bit of time but before you go disabling any constrints or various things it is possible to remove it from a database without too much effort. It is possible by just reordering the tables before they are deleted. The following will get the job done.


Before running this I would strongly suggest that you take some sort of backup of the database.



DROP VIEW vw_aspnet_Applications
DROP VIEW vw_aspnet_MembershipUsers
DROP VIEW vw_aspnet_Profiles
DROP VIEW vw_aspnet_Roles
DROP VIEW vw_aspnet_Users
DROP VIEW vw_aspnet_UsersInRoles
DROP VIEW vw_aspnet_WebPartState_Paths
DROP VIEW vw_aspnet_WebPartState_Shared
DROP VIEW vw_aspnet_WebPartState_User

DROP PROCEDURE aspnet_AnyDataInTables
DROP PROCEDURE aspnet_Applications_CreateApplication
DROP PROCEDURE aspnet_CheckSchemaVersion
DROP PROCEDURE aspnet_Membership_ChangePasswordQuestionAndAnswer
DROP PROCEDURE aspnet_Membership_CreateUser
DROP PROCEDURE aspnet_Membership_FindUsersByEmail
DROP PROCEDURE aspnet_Membership_FindUsersByName
DROP PROCEDURE aspnet_Membership_GetAllUsers
DROP PROCEDURE aspnet_Membership_GetNumberOfUsersOnline
DROP PROCEDURE aspnet_Membership_GetPassword
DROP PROCEDURE aspnet_Membership_GetPasswordWithFormat
DROP PROCEDURE aspnet_Membership_GetUserByEmail
DROP PROCEDURE aspnet_Membership_GetUserByName
DROP PROCEDURE aspnet_Membership_GetUserByUserId
DROP PROCEDURE aspnet_Membership_ResetPassword
DROP PROCEDURE aspnet_Membership_SetPassword
DROP PROCEDURE aspnet_Membership_UnlockUser
DROP PROCEDURE aspnet_Membership_UpdateUser
DROP PROCEDURE aspnet_Membership_UpdateUserInfo
DROP PROCEDURE aspnet_Paths_CreatePath
DROP PROCEDURE aspnet_Personalization_GetApplicationId
DROP PROCEDURE aspnet_PersonalizationAdministration_DeleteAllState
DROP PROCEDURE aspnet_PersonalizationAdministration_FindState
DROP PROCEDURE aspnet_PersonalizationAdministration_GetCountOfState
DROP PROCEDURE aspnet_PersonalizationAdministration_ResetSharedState
DROP PROCEDURE aspnet_PersonalizationAdministration_ResetUserState
DROP PROCEDURE aspnet_PersonalizationAllUsers_GetPageSettings
DROP PROCEDURE aspnet_PersonalizationAllUsers_ResetPageSettings
DROP PROCEDURE aspnet_PersonalizationAllUsers_SetPageSettings
DROP PROCEDURE aspnet_PersonalizationPerUser_GetPageSettings
DROP PROCEDURE aspnet_PersonalizationPerUser_ResetPageSettings
DROP PROCEDURE aspnet_PersonalizationPerUser_SetPageSettings
DROP PROCEDURE aspnet_Profile_DeleteInactiveProfiles
DROP PROCEDURE aspnet_Profile_DeleteProfiles
DROP PROCEDURE aspnet_Profile_GetNumberOfInactiveProfiles
DROP PROCEDURE aspnet_Profile_GetProfiles
DROP PROCEDURE aspnet_Profile_GetProperties
DROP PROCEDURE aspnet_Profile_SetProperties
DROP PROCEDURE aspnet_RegisterSchemaVersion
DROP PROCEDURE aspnet_Roles_CreateRole
DROP PROCEDURE aspnet_Roles_DeleteRole
DROP PROCEDURE aspnet_Roles_GetAllRoles
DROP PROCEDURE aspnet_Roles_RoleExists
DROP PROCEDURE aspnet_Setup_RemoveAllRoleMembers
DROP PROCEDURE aspnet_Setup_RestorePermissions
DROP PROCEDURE aspnet_UnRegisterSchemaVersion
DROP PROCEDURE aspnet_Users_CreateUser
DROP PROCEDURE aspnet_Users_DeleteUser
DROP PROCEDURE aspnet_UsersInRoles_AddUsersToRoles
DROP PROCEDURE aspnet_UsersInRoles_FindUsersInRole
DROP PROCEDURE aspnet_UsersInRoles_GetRolesForUser
DROP PROCEDURE aspnet_UsersInRoles_GetUsersInRoles
DROP PROCEDURE aspnet_UsersInRoles_IsUserInRole
DROP PROCEDURE aspnet_UsersInRoles_RemoveUsersFromRoles
DROP PROCEDURE aspnet_WebEvent_LogEvent

DROP TABLE aspnet_WebEvent_Events
DROP TABLE aspnet_SchemaVersions
DROP TABLE aspnet_Profile
DROP TABLE aspnet_UsersInRoles
DROP TABLE aspnet_Roles
DROP TABLE aspnet_PersonalizationPerUser
DROP TABLE aspnet_PersonalizationAllUsers
DROP TABLE aspnet_Paths
DROP TABLE aspnet_Membership
DROP TABLE aspnet_Users
DROP TABLE aspnet_Applications

DROP SCHEMA aspnet_Membership_FullAccess
DROP SCHEMA aspnet_Membership_ReportingAccess
DROP SCHEMA aspnet_Personalization_FullAccess
DROP SCHEMA aspnet_Membership_BasicAccess
DROP SCHEMA aspnet_Personalization_BasicAccess
DROP SCHEMA aspnet_Personalization_ReportingAccess
DROP SCHEMA aspnet_Profile_BasicAccess
DROP SCHEMA aspnet_Profile_FullAccess
DROP SCHEMA aspnet_Profile_ReportingAccess
DROP SCHEMA aspnet_Roles_BasicAccess
DROP SCHEMA aspnet_Roles_FullAccess
DROP SCHEMA aspnet_Roles_ReportingAccess
DROP SCHEMA aspnet_WebEvent_FullAccess

DROP ROLE aspnet_Membership_FullAccess
DROP ROLE aspnet_Membership_ReportingAccess
DROP ROLE aspnet_Personalization_FullAccess
DROP ROLE aspnet_Membership_BasicAccess
DROP ROLE aspnet_Personalization_BasicAccess
DROP ROLE aspnet_Personalization_ReportingAccess
DROP ROLE aspnet_Profile_FullAccess
DROP ROLE aspnet_Profile_ReportingAccess
DROP ROLE aspnet_Roles_FullAccess
DROP ROLE aspnet_Roles_BasicAccess
DROP ROLE aspnet_Roles_ReportingAccess
DROP ROLE aspnet_WebEvent_FullAccess
DROP ROLE aspnet_Profile_BasicAccess
E-mail Kick it! DZone it! Permalink