Search Engine Field Test: How Was This Conducted?
It is only responsible to disclose everything about our testing procedure.
Who did the field test?
We found seven separate groups of thirty veteran teachers. We also had at least five alternate teachers available to make sure that we would have data from a full thirty teachers. Each group had teachers with at least three years experience on the topic they were reviewing the search results for. We were able to locate teachers from geographically diverse areas for each group. Many were excited just to take part in the experiment after we told them what we were trying to test. The teachers were compensated for their time with a free three-year Teachnology Supreme Membership.
What Did The Teachers Review?
Each group evaluated the top 10 organic search results that were generated from the top 5 search engines. The web sites that hold the top 10 position all came from searches that were performed on the same day, using the same browser, from the same geographic location. Teachers rated each set of search engine results from Ask, Bing, Blekko, Dogpile, and Google. The teachers did not have any idea which search engine produced the web site they were viewing. We provided the results and links to the results using the same bland background. We also provided each reviewer with the results in a random order. For example, Reviewer #1 might see the results from Bing (result #7), while Reviewer #2 is viewing the result #1 from Google. The next site they reviewed would be from another search engine and position number. We did this so that raters would not influence each other.
To determine the keyword phrases that we would rate, we surveyed the teachers taking part. One week prior to each experiment, we asked our 210 teachers to tell us the last ten searches they performed in school. We then chose commonalties we could find in the phrases. We cross referenced those with their global popularity. This allowed us to determine the keyword phrases that the majority of teachers were looking for worldwide; that related to our group of participants. We also wanted to insure a good level of diversity between the phrases that we tested. We only allowed one search phrase for the typical teacher search interests (i.e. lesson plans, worksheets, crafts, templates).
The keyword phrases tested were:
Group 1 - water cycle lesson plan
Group 2 - writing worksheets
Group 3 - Thanksgiving Kids Crafts
Group 4 - Lesson Plan Template
Group 5 - Establishing Class Rules
Group 6 - Parent Teacher Conference Forms
Group 7 - Middle School Report Card Comments
|
When Was This Done? We retrieved the search results in late April 2011. Each set of results was pulled from the search engines approximately one week prior to the review date. The date of each pull of search results is noted with each field test URL set. The testing took place 1-2 days after each data pull. As soon as we had a group of teachers organized and identified keyword phrases for that group, we pulled the top 10 organic results from each engine. |
Where Were The Field Tests Conducted?
It was actually very easy for us to recruit teachers; they jumped at the opportunity to help us out with this experiment. We identified county learning centers and recruited teachers from those counties. After we contacted administrators in each county, we would normally receive over a hundred willing and ready teachers.
How Did Teachers Rate the Quality of the Search Engine Results?
Even though hypothetically there should be fifty web sites, there was a great deal of crossover. Some search engines will generate the same results. Teachers rated each web site with a 0 to 100 rating for usefulness. We included increments of 10 in the scoring. A rating of 100 represents an incredibly useful website and a rating of 0 represents a web site that has no value for the teachers.
Here is what teachers saw when reviewing the web sites:
It would be irresponsible of us not to disclose that this experiment was purely about data. Since we provided the organic results to teachers, the user experience of the search engine was not included in this experiment. After each field test was complete, we had the teachers perform one search on each of the search engines web sites. We then asked them to rank the search engines 1 being the easiest to use and 5 being the most difficult to use. In addition, we asked them to write a one sentence explanation as to why they chose each position in their ranking system.
How Do Teachers Define an Incredibly Useful Web Site?
Prior to each test we spent 2 minutes dialoguing with teachers about this. Here are the qualities of an incredibly useful website:
1. Clear and easy to follow. Ads don't make the site difficult to use.
2. Provides teachers with unique activities, ideas, or tips. Things that you could add to your lesson on the subject you are searching for.
3. Saves you time in helping you prepare for your lesson.
We created a simple rubric for teachers to follow when rating the web sites.
Let's Take A Look At The Basic Procedures By Examining the Basics of Our First Field Test:
We pulled the top ten results from our five search engines, assembled our thirty teachers, and got to work. We found that it took the teachers approximately 3 hours to review these web sites. Let's look at the scores and the results that come with them.
How Did We Get These Results?
Part 2: Which Search Engine Did Teachers Say Was The Easiest To Use?