Week 5: ResearchXL Method

For this week of the CXL growth marketing mini-degree, I was able to go through the ‘Identifying and amplifying growth channels.

This week I didn’t have enough time to go through the course too much. Client responsibilities and projects had taken a lot more time.

I was able to take some notes about the ResearchXL method and definitely will factor this into my testing process. Here are my notes:

ResearchXL Method

  • What kind of data do we need?
    • Step 1: Heuristic Analysis
      • A heuristic analysis is an experience-based analysis of the website through every page and device types
      • 4 features to consider when auditing a website for experience
        • Clarity
        • Friction
        • Anxiety
        • Distraction
    • Step 2: Technical Analysis
      • Use cross-browser testing
      • Cross-device testing
      • Speed analysis
    • Step 3: Digital Analytics
      • This will help you figure out where you have problems on the site
      • Where are the drop-off points
      • Correlate behaviors with outcomes
        • You need to make sure everything the user can do on your site is being measured
    • Step 4: Qualitative Research
      • Surveys and user testing
        • Surveys
          • On-site polls
            • You can put polls on key pages
            • or pages that saw a large drop-off
          • Follow up surveys
            • Survey people that just made a purchase that was recent ( a couple of hours or a couple days)
            • Ask them open-ended questions and get more information about their shopping experience
    • Step 5: User testing
      • Recruit people who represent your target audience
      • Have these people use your website and do a task on the site
      • Watch what they do
      • Let them talk out loud and share their thoughts
      • But watch what they’re actually doing
    • Step 6: Mouse tracking analysis
      • You want to know where people’s mouse cursor is going
      • Click maps
      • Heat maps
      • Scroll maps
  • What to do after research?
    • After you write down all your observations and findings, you need to prioritize which should be handled.
      • You’ve just identified a bunch of issues
    • Use a 5 point ranking scale
      • 5 – sever issue
        • You’re losing a lot of money and affecting a large portion of users
      • 1 – minor issue
        • Minor usability issue that should be fixed eventually and it’s not affecting a whole lot of people
    • You also want to categorize the issue
      • Instrumentation issue – not measuring data that needs to be measured
      • Tests – You know the problem area, now you just need to test different ideas to solve the issue
      • Investigate – Figure out what which would be the best way to solve the problem
      • No-brainer issues – issues that just need to be fixed
        • Text that is unreadable and too small
    • A bit more about star ratings:
    • ★★★★★
      • This rating is for critical usability, conversion, or persuasion issue that will be encountered by many visitors to the site or has high impact. Implementing fixes or testing is likely to drive significant change in conversion and revenue.
    • ★★★★
      • This rating is for a critical issue that will be encountered by many visitors to the site or has a high impact.
    • ★★★
      • This rating is for a major usability or conversion issue that may not be viewed by all visitors or has a lesser impact.
    • ★★
      • This rating is for a lesser usability or conversion issue that may not be viewed by all visitors or has a lesser impact.
      • This rating is for a minor usability or conversion issue and although is low for potential revenue or conversion value, it is still worth fixing at lower priority.
    • There are 2 criteria that are more important than others when giving a score:
      • Ease of implementation (time/complexity/risk). Sometimes the data tells you to build a feature, but it takes months to do it. So it’s not something you’d start with.
      • Opportunity score (subjective opinion on how big of a lift you might get). This depends on how many users are exposed to the issue, and how close to the money the issue is. Let’s say you see that the completion rate on the checkout page is 65%. That’s a clear indicator that there’s lots of room for growth, and because this is a money page (payments taken here), any relative growth in percentages will be a lot of absolute dollars.
    • Essentially: Follow the money. You want to start with things that will make a positive impact on your bottom line right away.
      • Give more weight to issues that affect a large portion of your visitors
        • problems in high traffic pages or in the checkout funnel
      • Most conversion projects will have 15-30 pages full of issues
  • Measuring the effectiveness of testing a program
    • Three key metrics to measure effectiveness of a testing program
      • Testing velocity – How many tests are you running per month? per year?
      • Percentage of tests that provide a win
        • Most tests end in failure or no significant difference
        • But thats because most don’t know what they’re doing
        • ResearchXL framework allows you to use data
      • Impact per successful experiment
        • Most tests are in between minus 15 and plus 15 percent relative increase or decrease

Leave a Reply