Teachers Field Test The Top 5 Search Engines
Teachers Say: Google Do Your Homework!
Testing Results From May 2011.
For years, we have been recommending Google as our top choice for teachers, without having actual data specific to teachers to go on . Time to change that! We were challenged to field test search engines and find out who really has the best search results. The challenge came from a teacher who's school district switched their default search engine from Google to Bing.
We chose the top 4 search engines (Ask, Bing, Dogpile, and Google) by search volume. Please note that Dogpile does not maintain their own index. They are a meta-search of Bing and Google. We also included an up-and-comer Blekko due to the buzz it has been getting for being unique. Many teachers recommended testing out Sweet Search as well. We will include Sweet Search in our next testing session.
To do this we decided to have 7 groups of 30 teachers field test the top five search engines and determine which results were the best. We conducted seven field test rating sessions to try to determine who provides the best results for K-12 teachers. As you can see in our test notes teachers were not aware of what search engine produced the result they were reviewing. We did this to insure that bias towards a favorite search engine or interface had nothing to do with the outcome. We were solely interested in identifying the web pages they find helpful. Teachers were just asked to visit the links we provided and rate that web site using a 0 (useless) to 100 (extremely useful). The rating system ran in increments of 10. After we completed the testing, teachers were asked to review the interface of each site to try to see which search engine was easiest for them to use.
Many of you just want to read the results, so we will not slow you down and throw tons of ads at you. If you would like to do your homework before reading the results here is the nitty gritty of what we did:
Teachers' Ranking of the Top 5 Search Engines
Average Place: 4.14 (1 Being Best, 5 Being Worst)
Average Results Score: 70.50 (Out of 100)
Pros: Teachers did find Ask easy to use in the post-testing interface portion.
Cons: In our testing we found that Ask normally has 2-3 top 10 results that either do not relate to the search, are dead links, or are just spammy.
Average Place: 4.00 (1 Being Best, 5 Being Worst)
Average Results Score: 59.50 (Out of 100)
Pros: Offers many sites that the other search engines look over. Many times they were quality results. The interface was nice and easy to use.
Cons: In some areas Blekko was way off. On one of our tests Blekko produced 7 of 10 results that were in no way what so ever related to our search. Not only were some of results useless, but we had a single search yield three Malware installing sites.
Average Place: 3.14 (1 Being Best, 5 Being Worst)
Average Results Score: 76.71 (Out of 100)
Pros: On 6 of the 7 tests Google identified the single best result somewhere in the top 10. The interface appeal to teachers were the best.
Cons: We started this experiment to really validate our claim that Google produces the best and most relevant results. We were very disappointed with Google's results.
In one of our tests, not only did they come in last, it appeared as if the other search engine understood what we were looking for and Google was completely lost. Hopefully someone over in Mountain View catches a glimpse at this and they retool a bit to help teachers out. They are usually very responsive. We will expect better results on our next testing session.
Average Place: 2.00 (1 Being Best, 5 Being Worst)
Average Results Score: 79.93 (Out of 100)
Pros: Bing always seemed to just find 1-2 better sites than their counterparts. Whereas Google would usually have 2 out of 10 results that were stellar web sites, Bing would provide 3 stellar results. Bing definitely out performed our expectations. On 7 of the 7 tests Bing identified the single best result somewhere in the top 10. It's probably time to start giving them a bigger look.
Cons: Teachers were not happy with the layout of their site. It seemed pretty busy especially if you are using the low resolution screens found in most classrooms.
Average Place: 1.71 (1 Being Best, 5 Being Worst)
Average Results Score: 81.57 (Out of 100)
Dogpile Is a Meta-Search Engine. It returns the results from Bing and Google. So common sense would tell us that their ranking should be near both Google's and Bing's rating.
Please caution yourself on the results of our test. We wouldn't run out and "Dogpile" everything right away. You need to remember that we provided teachers with the organic results, they never saw the search engines until after the test. So the interface was never taken into account with this site. As you will see below, Dogpile's results are difficult to sort through. It almost makes it feel like work when you are searching.
Pros: Dogpile's organic results had the uncanny ability to pick the absolute best results from Google and Bing. Best organic results in our test. Out of our 7 tests Dogpile always came in first or second.
Cons: A HUGE word of caution here, while Dogpile's organic results were rated the highest consistently; their actual results are overloaded with ads. Take a look at the basic search below. While you can see 7 results, only 1 of those is an actual organic result. Look below at a typical query, 1 organic result and 6 ads, not a great ratio.
How Did We Get These Results?