I recently had the experience of designing and managing three rounds of tree testing using Treejack. Treejack is an online usability evaluation tool by Optimal Workshop that tests information architecture (IA) through scenario based tasks without visual distractions.
A few of the guys I work with had used it before, but this was my first experience using this tool. It was an awesome experience and it was really easy to use. Treejack walks you through all the steps you need to take to build, launch and complete the activity.
The steps involved in creating the online testing activity are:
Draft your IA
I like using paper and sticking things to the wall/desk in the first instance because it gives me the freedom to step back and rearrange items until I’m happy. From here, I recommend building it out electronically in a simple table or spreadsheet.
Write your test scenarios
Think about what you would like to test. The number of tasks should be limited to no more than ten, so make them count and don’t double up. Scenarios work best if they are short, sweet and directional. An example would be: You want to read the latest Annual Report. Where would you go to do this?
Ask yourself – do you need to ask any demographic based questions? If yes, what are they?
Treejack provides the option to include questions before or after the activity and this is a good opportunity to ask your users demographic based questions. It is recommended that the survey is set to record responses anonymously.
Build your activity in Treejack
This online tool is pretty awesome in the way it walks you through the creation of the activity.
After you’ve built and launched the activity, all you have to do is email the link to users and sit back and wait for the results to start appearing in the dashboard. The results are recorded automatically and Treejack presents them in a visual, easy to understand format that makes analysis easier. It does this in two ways; by surfacing meaningful and useful statistical results and by presenting all pathways followed by users in the form of a pie tree diagram.
A pie tree diagram is essentially a tree diagram where each joint in the tree is a pie chart. Imagine gathering up several of those pie shaped tokens that Trivial Pursuit uses to represent the players and then imagine building a tree diagram out of them and joining them up with string.
This is very useful because each pie is divided with colour coding and statistics showing:
Grey: How many people skipped the question at that point
Green: How many people were on the correct path when they passed through it,
Red: How many people were on the incorrect path when they passed through it,
Blue: How many people turned around and went back from that point
Yellow: How many people nominated it as the correct answer
Treejack also provides an overview at each joint of the pie tree. This outlines; how many people clicked on it and the breakdown in numbers of the colour coded pie.
Tips for interpreting the results:
Don’t just look at the stats
The stats will definitely give you quantitative findings but the pie tree diagrams will tell you much more of the story. Ask yourself questions like; How many times has Home been clicked on? How often did users turn around and go back? Where did they turn back? If the failure rate is high, where did they go instead?
All data is good data
Don’t be discouraged if the skip rate or the failure rate is high. This is a good thing – your IA may have failed in some places in this activity but the results will clearly show where the issue is and recognising that means you’re already on your way to fixing it.
Print all the results pages out, including the pie trees, and pin them up
I’m a huge fan of sticking things on walls. It might be my Industrial Design background, but I see a lot of value in pinning it all up, stepping back and absorbing it. This gives an overall picture of what you’re dealing with and may make it easier to spot patterns in the results.
Lastly, here are my general recommendations to keep in mind when using Treejack:
Build the tree in an Excel spread sheet first
Get it right in Excel (or something similar) first before attempting to move the tree into Treejack. It’s an easy copy and paste and it will save you time. Especially if you’re working within a team and changes are likely to be needed.
Choose the background colour wisely
Try not to choose anything that is; too bright, too distracting for the user or anything that might be hard to look at for more than a few seconds. Don’t forget they’ll see it through the entire activity. I once saw a Treejack activity done by someone else that had a bright yellow background and it was really hard to look at while I worked through the tasks.
Limit your questions to no more than 10
Previously mentioned nameless yellow background offender also required users to work through more than 20 scenarios. I personally wouldn’t have more than 10 scenarios per test. You run the risk of boring and annoying people and then you’ll be left wondering why your skip rate is so high.
Use the instructions it comes with
The instructions Treejack provides are actually pretty good, so don’t make work for yourself by trying to make your own unless you really think it will add value.
Always do a dry-run
No matter how careful I am, I always find mistakes during the dry run and I never launch the real one without doing a test run with my colleagues first.
Overall, I really enjoyed using this tool and will definitely be using it again.