Stairwellblog

Recently, I viewed an insightful webinar which benchmarked 5 major brand hotel websites by giving 250 users 2 tasks and evaluating the results. It should be mentioned that the agenda for the webinar was not just purely for purposes of academic enlightenment. The two companies responsible for conducting the broadcast probably hoped to get some lift by the exposure and there was some detailed explanation about their services/capabilities throughout the course of the presentation. So, I’ll give them a quick shout out here as well.

1. Userzoom (Sunnyvale, CA) provided the technology. They offer UX pros the ability to run online, task-based studies with geographically dispersed participants.
2. KeyLime Interactive (Miami, FL) moderated the sessions. They specialize in online web-based usability testing as well as traditional lab-based testing.

A little background about the study before diving into the results. Each Hotel site tested (Hilton, Marriot, Intercontinental, Starwood, Hyatt) were randomly assigned 50 different users to perform 2 tasks seperatly. Tasks took roughly 10 min in length to complete. Users were selected regardless of their prior affiliation with any hotel rewards program or familiarity with the site.

Tasks:

1. Finding a room on a beach at a particular time of year, particular distance from a landmark, within a set budget of $250.
2. Finding a hotel that provided parking to it’s guests

The web sites were evaluated on several metrics based on the tasks which users attempted to cary out (Overall success, Ease of use, Ease of Navigation, Clear where to start, Satisfaction with Time) and then are given an overall score (KLI score) by the moderating agency (Keylime interactive)

Data gathered from open questioning and Task 1 Results overview:

• Marriot, Hilton had largest groups of rewards program participants – and considered ‘most popular’ by users as well
• Hilton ranks best from a search perspective
• Hyatt ranks worst from a search perspective
• Marriot and Hilton brands scored well on ease of use and accessibility
• Hilton & Marriot score highest on success rate, Ease of Use, and Satisfaction. Hyatt scored lowest
• User satisfaction related directly to the time it takes to complete tasks

Data gathered from open questioning and Task 2 Results overview:

• Marriot scored highest on search, but most users had a hard time figuring out where to start clicking
• Hyatt & Starwood rank consistently low in all categories of evaluation within the task result

The KLI results from the 2 tasks were the aggregated for an overall KLI score

Executive takeaways

• All brands had varying strengths and weaknesses
• All brands have room for improvement
• All brands had a ‘negative’ score of with regards to the brand meeting end-users expectations (Hilton however was the ‘best’)
• No single brand or site should be picked on as being ‘bad’ in strict black and white terms. These results were simply the outcome of 2 tasks performed online. But it starts to paint a picture of the overall User Experience the various brands provide

Last 5 posts by tyler

• Show how it works - December 6th, 2010
• Rah! Rah! Team! - November 9th, 2010
• Show how it works - November 4th, 2010
• Dark Patterns in designing the User Experience - September 28th, 2010
• Flow & the User Experience - September 13th, 2010