Wagon Ruts of the Web

Imagine that Google had discovered the ideal search algorithm and every search returned the best content from the web.  This ideal algorithm would be based on a  larger number of factors including back links, how long people stayed on a page, how long the page has existed and the relevancy of the content.  Under the ideal model, it would seem that each time a page was visited, the page would become even more solidified as the proper results for a particular search term.  This action would be equivalent to creating wagon ruts in the internet.  People continue to travel down the same path and that path becomes self fulfilling forcing people to travel in that path.

An example of wagon ruts worn into rock.

Does search fall into the same trap?

Source: http://www.txaisi.com/Texas_History.html

Unfortunately, while the algorithm was ideal and gave the best results for your search query, it would be very hard for new sites to get a better ranking since rankings would be based partly on historic data.  In some cases, the wagon ruts would be accepteble.  Suppose you searched for "official IRS website".  In this case, the first result should be www.IRS.gov.  However, if you searched for, "favorite song in Canada", you might be taken to a page that lists the most popular songs but would this be really relevant to you?  The context would determine how relevant the results were.  It is very likely, many different people have an opinion about the best song in the country and it is up to the searcher to further sift through the different pages.

It seems then, that search is a mix of exact matches and some not so exact matches.  The results for, "restaurants within 1 block of 1800 Union St" would fall into a wagon rut only changing when a restaurant physically went out of business or a new one opened. However, searches for, "best restaurant within 1 block of 1800 Union St", should return a a more open path result.  The campus map below shows many paths across a campus.  It is very likely, that people often leave the path and look for a direct route that meets their immediate needs.

There are many paths that let you cross this campus.

Source: http://www.ohio.edu/athens/greens/cgreen.html

YouTube shows evidence of wagon ruts.  Here is an example of a search for, "golf swing" taken two days apart.  Are these the absolute best results in the world for golf swing?  The only difference between the two results is the second ad.  The first ad is the same in both results.

YouTube search results on 4/9/12 and 4/10/12.  

The only change is the second add.

When searching YouTube, people are not usually looking for a wagon rut result.  They usually are looking for something new and interesting.  The one exception to this is when people are trying to find the original version of a video that has many parodies (http://www.youtube.com/watch?v=WrfUXuBaJ1E&feature=youtube_gdata).

Research has been done that shows that wagon ruts are not only a result of technology but also are the result of social behavior.  In a study titled "Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market", http://www.filosofitis.com.ar/archivos/experimentalmarket.pdf  Matthew J. Salganik, Peter Sheridan Dodds and Duncan J. Watts showed that wagon ruts can also be created when people are influenced by other peoples choices.  When people were asked to rank different songs, people gave a higher ranking to songs that other people liked.  This causes some songs to rise to the top of the charts only because they happened to get some sort of initial push.

So how could Google or Bing improve their search results?  To improve the results, search engines should combine a mix of wagon ruts and open paths in their search results.  The wagon ruts would be helpful when people are looking for exact information that has a defined source.  The open paths would be used to introduce new possible results that might meet the users' requirements.  Over time, if the open path results are useful, they would upset the wagon ruts and become wagon ruts themselves.  This continuous testing would allow new ideas to be introduced without creating too much chaos in the results.

By mixing up wagon rut results with open path results, Google would present a broad result set that would help a rational person find the specific information that the person is looking for.