DNN Tips & Tricks #7: DotNetNuke Menu and SEO

Nov 18

Written by: Joe Brinkman
11/18/2008 12:28 AM  RssIcon

During the OpenForce Conference, I heard from several designers that the DNNMenu had problems with Search Engine Optimization (SEO) because it rendered menu items that used JavaScript for navigation.  While I assured them that this was not the case and that the menu rendered a down-level version for search bots, I thought I would perform further testing and document the exact behavior.  In the process, I found that while the DNNMenu performed as expected, ASP.Net did not.

To observe the behavior of the menu, I used Firefox 3 with the User Agent Switcher plug-in.  This plug-in allows you to simulate any user agent string you wish.  For testing purposes you can find a complete list of user agent strings at useragentstring.com.

Below is the DNNMenu as rendered in the standard Firefox browser.

MozillaMenu

To change user agents, create a set of new user-agent using the Tools>>User Agent Switcher>>Options>>Options screen and add the string for the browser or web crawler you wish to emulate.  If you are emulating a web crawler, you should also disable JavaScript.  With the GoogleBot user-agent enabled and JavaScript disabled, Firefox will render the menus as seen by Google.  You can see this behavior in the picture below.

GoogleMenu

 

Opening Firebug, you can easily verify that the menu is rendered as a series of hyperlinks as shown below.

GoogleMenuRendering

Everything appeared to work as I expected.  Unfortunately, when I tested the user-agents for Yahoo and ask.com, I found that the menu did not behave the same as for the Google and MSN/Live bots. 

The DNNMenu IsDownLevel property determines whether to render the full JavaScript menu or to render a down-level version that doesn’t require JavaScript.  If the “browser” is a web crawler, then the menu should always render as down-level.  Looking at the IsDownLevel code below we see that the key is the IsCrawler property which further delegates the crawler check to the ASP.Net framework.

Public ReadOnly Property IsDownLevel() As Boolean
    Get
        If ForceDownLevel OrElse Me.IsCrawler OrElse _
         DotNetNuke.UI.Utilities.ClientAPI.BrowserSupportsFunctionality(Utilities.ClientAPI.ClientFunctionality.DHTML) = False OrElse _
         DotNetNuke.UI.Utilities.ClientAPI.BrowserSupportsFunctionality(Utilities.ClientAPI.ClientFunctionality.XML) = False _
        Then
            Return True
        Else
            Return False
        End If
    End Get
End Property
 
Public Property IsCrawler() As Boolean
    Get
        If Len(ViewState("IsCrawler")) = 0 Then
            Return System.Web.HttpContext.Current.Request.Browser.Crawler
        Else
            Return CBool(ViewState("IsCrawler"))
        End If
 
    End Get
    Set(ByVal Value As Boolean)
        ViewState("IsCrawler") = Value
    End Set
End Property

Looking at this code, I see that for some reason ASP.Net is not properly detecting any bots other than GoogleBot and MSN.  After doing further research I found that browser detection is controlled by the .Net 2.0 .browser files that are located at %FrameworkDir%\v2.0.50727\CONFIG\Browsers.  A quick WinGrep determined that the only browser file that sets the Crawler property is the gateway.browser file.  The relevant portion of the file is:

<gateway id="Crawler" parentID="Default">
    <identification>
        <userAgent match="crawler|Crawler|Googlebot|msnbot" />
    identification>
 
    <capture>
    capture>
 
    <capabilities>
        <capability name="crawler" value="true" />
    capabilities>
gateway>

Clearly, this is only detecting Google and MSN.  Further research indicated that this is a known problem with the default ASP.net behavior.  ASP.Net 2.0 added the ability to define .browser files in the App_Browsers folder of your web application.  By defining custom .browser files we should be able to correctly identify targeted search bots.

The below slurp.browser file will detect the Yahoo crawler.

<browsers>
    <browser id="Slurp" parentID="Mozilla">
        <identification>
            <userAgent match="Slurp" />
        identification>
        <capabilities>
            <capability name="browser" value="Yahoo!Slurp" />
            <capability name="crawler" value="true" />
            <capability name="cookies" value="false" />
            <capability name="css1" value="true" />
            <capability name="css2" value="true" />
            <capability name="javascript" value="false" />
            <capability name="tables" value="true" />
            <capability name="w3cdomversion" value="1.0" />
            <capability name="xml" value="true" />
            <capability name="tagwriter" value="System.Web.UI.HtmlTextWriter" />
        capabilities>
    browser>
browsers>

Most of the crawlers use a Mozilla compatible string like Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) and so you need to make sure to set the parentID to “Mozilla” so that the browser detection works correctly.

As you can see, DotNetNuke has good support for search engines in the default DNNMenu.  There are still optimizations you must make if you wish to cater to search engines beyond Google and Windows Live/MSN.  One of the great aspects of DotNetNuke is that if you don’t care for the tradeoffs that the DNNMenu makes, then you can choose to use a third-party CSS menu like the HouseMenu or Snapsis CSS NavMenu

All of this works just fine in DotNetNuke 4.4 and beyond, however, in DotNetNuke 5.0 SEO will get a slight boost.  The down-level rendering in previous versions of the DNNMenu was not very lightweight nor was did it use semantic markup.  The next version of DNNMenu in DotNetNuke 5.0 allows the menu to be rendered as an unordered list which should help search engines to better understand the menu structure.  I will have a separate post to discuss this version of the menu.

EDIT:  Updated DotNetNuke 4.9.1 and DotNetNuke 5.0.0 to include the Crawler and Webkit browser files from http://owenbrady.net/ 

21 comment(s) so far...


Gravatar

re: DNN Tips & Tricks #7: DotNetNuke Menu and SEO

Can we expect future versions of DNN to also include the updated browser file(s) in the App_Browser directory?

By Will Strohl on   11/18/2008 8:34 AM
Gravatar

re: DNN Tips & Tricks #7: DotNetNuke Menu and SEO

Great find Joe... What would be great is to provide a .browser already populate with a list of the most popular crawlers. Just a thought. Thanks for the post.

By Antonio Chagoury on   11/18/2008 8:36 AM
Gravatar

re: DNN Tips & Tricks #7: DotNetNuke Menu and SEO

Great post Joe. Thanks for getting all of this out in the open! Take care, Ian

By Ian Robinson on   11/18/2008 8:42 AM
Gravatar

re: DNN Tips & Tricks #7: DotNetNuke Menu and SEO

Very useful post. Few questions; 1. Can you put the additional browser files into the %FrameworkDir%\v2.0.50727\CONFIG\Browsers folder so that every app on the server gains the additional detection? 2. You can create a single browser file containing multiple browser definitions can't you? 3. The downlevel URL's generated are they formatted using the configuration of the FriendlyURL provider? Thanks antony

By Antony on   11/18/2008 8:53 AM
Gravatar

re: DNN Tips & Tricks #7: DotNetNuke Menu and SEO

@Will, @Antonio - I'll discuss this with Jon Henning to see if we can come up with a comprehensive file that covers the major bots. @Antony - 1. I don't know if adding files to that folder will work as those files are compiled and stored in the GAC. You can make changes and compile the new browser files. 2. See above 3. Not sure how the downlevel URLs are generated. I assume that they are normal page links.

By jbrinkman on   11/18/2008 9:17 AM
Gravatar

re: DNN Tips & Tricks #7: DotNetNuke Menu and SEO

Thanks for digging into this Joe. Why does it have to be so damn complicated though? If we had a menu that emits semantic markup (unordered list), then we would not have to play any kind of user agent tricks at all.

By Tom Kraak on   11/18/2008 10:47 AM
Gravatar

re: DNN Tips & Tricks #7: DotNetNuke Menu and SEO

@Tom - I personally agree about the need for a nice simple, CSS based menu. However, as the menu is Jon Henning's baby, he tends to lean more towards providing a richer user experience than what is possible with a pure CSS approach. The good news is that there are already good CSS solutions available, so everyone can choose the menu that they prefer.

By jbrinkman on   11/18/2008 11:11 AM
Gravatar

re: DNN Tips & Tricks #7: DotNetNuke Menu and SEO

Personally, I'm not asking for a pure CSS-driven menu. There is nothing wrong with JavaScript when used intelligently. Telerik's RadMenu is a perfect example ... feature rich AND renders as an unordered list. Also, you and I and all the geeks above know that the DNN menu follows the provider module and can easily be switched, but that should not be used as an excuse for providing the best possible solutions "out of the DNN box."

By Tom Kraak on   11/18/2008 12:07 PM
Gravatar

re: DNN Tips & Tricks #7: DotNetNuke Menu and SEO

Joe, thanks for putting this together to further clarify the menu. It was great meeting you at the conference!

By Cuong Dang on   11/18/2008 12:40 PM
Gravatar

re: DNN Tips & Tricks #7: DotNetNuke Menu and SEO

Answered my own question re the downlevel URLs. They are generated with the friendly URL provider. Tested with 4.9 with the FriendlyURL's set to 'HumanFriendly' and the downlevel menu displays links like the following:

By Antony on   11/18/2008 3:44 PM
Gravatar

re: DNN Tips & Tricks #7: DotNetNuke Menu and SEO

Joe, thanks for the update, I already looked for ways to get the unordered list working but I did not succeed (it worked in the alpha release way back 2007). Given the fact that we have default access to jQuery in DNN, we have ample possibilities to skin the menu, including some very nice behaviors. A clean unordered list without additional (menu related) javascript would work fine for me.

By Peter Schotman on   11/19/2008 3:54 AM
Gravatar

re: DNN Tips & Tricks #7: DotNetNuke Menu and SEO

@Peter - I tend to agree, but most people using DotNetNuke just want it to work out of the box. I am also sympathetic to Jon's position as he has spent countless hours in development of the existing menu.

By jbrinkman on   11/19/2008 4:34 AM
Gravatar

re: DNN Tips & Tricks #7: DotNetNuke Menu and SEO

I hear you Tom. I certainly appreciate the time that Jon has put into the existing menu, however I would prefer to see the default be a simple unordered list controlled by CSS / JS if you want. Starting with the son of suckerfish code or equivalent would be a good start. But, Snapsis is a fabulous menu that I'd recommend to anyone looking for a cleaner, friendly, highly accessible output.

By Ryan Doom on   11/19/2008 6:00 PM
Gravatar

re: DNN Tips & Tricks #7: DotNetNuke Menu and SEO

@Joe, don't get me wrong, I truly appreciate the work Jon is doing, without Jon's work, DNN would not be, where it is today. Furthermore, the menu should work out of the box and also the new capabilities for styling are great, there is not much you can't do with nav 2.0. See the great online tutorial at: http://www.dotnetnuke.com/Community/Blogs/tabid/825/EntryId/1874/New-Video-Available-Demonstrating-New-CSS-Capabilites-in-DNNMenu-v2-0.aspx I suppose only us "paranoid and UL addicts"-techies (NOM) want to have full control over the menu... So for those of us who want power, simplicity and easy styling nav 2.0 is clearly the way to go. And for the rest of us, an UnorderListMenuProvider would be a great addition.

By Peter Schotman on   11/24/2008 10:15 PM
Gravatar

re: DNN Tips & Tricks #7: DotNetNuke Menu and SEO

Thanks for the mention of the Snapsis CSS Menu guys. For anyone interested, I have created a discount code. Enter [b]AGEEK[/b] on the product page for 25% off.

By John Mitchell on   11/26/2008 9:01 AM
Gravatar

re: DNN Tips & Tricks #7: DotNetNuke Menu and SEO

@John - That is very generous. Thank you.

By jbrinkman on   11/26/2008 9:59 AM
Gravatar

re: DNN Tips & Tricks #7: DotNetNuke Menu and SEO

Joe, thank you so much for the piece of information. There is a question in my mind after reading your last paragraph. If DNN team switch to unordered list for menu what happens for the menu graphic. As you know, each item would be a simple

  • tag in which can have only one CSS style. What happens if there are left, right or top images in the design? As far as I know the CSS can only apply a background image to the item. How would they support complex designs? will the additional contents be injected via CSS styles? is it an standard way and efficient? Thanks, Aboo

  • By Aboo on   11/26/2008 10:45 AM
    Gravatar

    re: DNN Tips & Tricks #7: DotNetNuke Menu and SEO

    Aboo, There are many ways to achieve tabs with rounded corners, etc. with simple UL/LI elements. One popular way is to use the sliding-doors technique by Douglas Bowman. http://www.alistapart.com/articles/slidingdoors/ The technique uses the anchor tag background for the right side, and the surrounding li tag for the left side.

    By John Mitchell on   11/27/2008 7:02 AM
    Gravatar

    re: DNN Tips & Tricks #7: DotNetNuke Menu and SEO

    For those interested, I posted on my reasoning for not emitting a UL/LI structure over a year ago here. http://www.dotnetnuke.com/Community/Blogs/tabid/825/EntryId/1511/Next-Version-of-DNNMenu.aspx It has to do with the amount of markup that gets sent down from the server (raw data will be smaller than data + markup). However, as the blog mentions, it will be possible for the new dnnmenu to emit data + markup by setting the RenderingMode property. Please review that blog entry for more info. Also, to understand the new css capabilities see the entry video mentioned by Peter earlier.

    By Jon Henning on   12/1/2008 10:02 AM
    Gravatar

    re: DNN Tips & Tricks #7: DotNetNuke Menu and SEO

    Brendan Kowitz reported a hole in the browser detection in ASP.NET 2.0 back in 2006 and was noted in a recent blog post on Google Webmasters Blog as well. Hard to imagine something so important slippled through the cracks for so long. You can read his original post about it at http://www.kowitz.net/archive/2006/12/11/asp.net-2.0-mozilla-browser-detection-hole.aspx

    By Al on   12/4/2008 3:27 AM
    Gravatar

    re: DNN Tips & Tricks #7: DotNetNuke Menu and SEO

    @Al - I had noticed that post while working on and researching this post but forgot to mention it. Thanks for bringing it up.

    By jbrinkman on   12/6/2008 2:06 AM
    dummy