Research:Evaluating RelatedArticles recommendations
Created
23:16, 14 November 2016 (UTC) Duration: 2016-November – 2017-March
This project's code is open-source
This project's data is available for download and reuse.
The RelatedArticles feature adds links to related articles after the current displayed article content. RelatedArticles currently uses the 'morelike' feature of CirrusSearch to select which articles to recommend. Ellery Wulczyn has recently developed an alternate model for identifying related articles, called Wikipedia Navigation Vectors, which is based on an entirely different set of features.
In this project, we will ask people to evaluate the output of these two different approaches to article recommendation, in order to determine whether the new model should be adopted.
Goals
edit
The goal of this study is to determine which model delivers the most relevant/useful suggestions, from an reader's perspective, in the context of the RelatedPages list at the bottom of mobile Wikipedia articles.
Depending on the results of this study, comparative evaluations may also be performed in additional contexts such as the recommended article feed on the front page of the Wikipedia Mobile app, on additional languages of Wikipedia, or on new recommendation-driven features (e.g. displaying general "recommended reading" suggestions for a particular wiki).
Timeline
edit- Jan 2017: build initial study
- Feb-Mar 2017: recruit participants, run study, and analyze results
Methods
editThe study will use an approach similar to the one used in a 2006 study that compared the results of several recommendation algorithms in an academic paper recommendation task[1] and a 2014 study that compared lists of movie recommendations on MovieLens website[2].
The basic design of the study involves mocking up a preview of a Wikipedia article (the lead sentences and image, plus the section headings), followed by two sets of 3 related articles each—one generated by CirrusSearch, the other generated by WikiVectors (mobile view of example study page). The Morelike-based recommendations will be generated using the classic_noboostlinks profile, which is the currently-implemented model for RelatedPages. Wikivectors currently delivers Morelike-based recommendations as a fallback if it is unable to generate its own recommendations for an article; for this study, we will only compare articles for which Wikivectors was able to generate at least 10 endogenous recommendations.
Study participants will be asked to read the preview and then answer a series of survey questions about which set of article recommendations they prefer, and why. Study participants will be shown the mobile version of the articles article preview, even though they will be viewing the test articles in a desktop browser. This is done to reduce the amount of irrelevant content the participant sees in their browser window, keeping them focused on their primary task, and also because the Related Articles feature is currently only enabled on mobile web and apps.
Two versions of each article preview are generated: in one the Wikivectors recommendations are listed first, in the other second. Study participants do not know which algorithm generated which set of recommendations.
Sampling
editTo make sure we evaluate the performance of the two models across different kinds of pages, we ran the study with both low- and high-traffic articles.
- Set 1. 24 low-traffic articles pulled from this list.
- Set 2. 25 high traffic articles pulled from among the top 100 articles by pageviews (mobile app only) for the month of February 2017.
Each set of articles was evaluated by 60 US-based Amazon Mechanical Turk workers. Each worker evaluated recommendations for 3 articles.
Survey questions
editThe survey will be delivered via the Qualtrics survey platform. The first five questions are Likert scales of the form "Much more [list] A than B"/"More A than B"/Neither A or B"/"More B than A"/"Much more B than A". The sixth question accepts a free text response.
- Which list has more articles that you would be interested in reading?
- Which list has more articles that are similar to each other?
- Which list has more articles that are NOT clearly related to the source article?
- Which list contains the article that you would be most likely to read next? (set 2 only)
- Based on these two lists, which recommender would you trust more to provide you with article recommendations that match your own interests?
- In a sentence or two, please describe why you do (or do not) prefer one list over the other?
Results
editThe responses to the five Likert scale questions show a strong overall preference for MoreLike-based recommendations over WikiVectors. All results are consistent across both low and high-traffic articles, suggesting that these two algorithms behave in noticeably different ways.
The bar charts below show the total normalized responses per question, grouped by rank. Colored bars present counts from set 2 (high-traffic articles); grey bars present the counts from set 1. Question #4 was only asked during set 2.
-
Q1. Which list has more articles that you would be interested in reading? -
Q2. Which list has more articles that are similar to each other? -
Q3. Which list has more articles that are NOT clearly related to the source article? -
Q4. Which list contains the article that you would be most likely to read next? (set 2 only) -
Q5. Based on these two lists, which recommender would you trust more to provide you with article recommendations that match your own interests?
For each question, the differences between response category counts ("prefer Morelike", "prefer neither", and "prefer Wikivectors") are significant (chi square, df=2, p<0.05). This just means the differences shown in the charts are unlikely to be the result of random variation, so we can be fairly confident that the patterns we see here are real.
Primary take-aways
edit- lists of Morelike-based recommendations tend to contain more articles that raters found interesting.
- lists of Morelike-based recommendations are more likely to contain the article that the rater most wants to read next.
- lists of Morelike-based recommendations tend to be more internally coherent: the articles in the list were more obviously related to one another.
- based on the lists they were asked to compare, raters trust that Morelike would deliver them better recommendations than WikiVectors in future
- lists of Wikivectors-based recommendations are more likely to contain articles that are not clearly related to the article the recommendation was based on.
Based on the results of this study, there is no reason to replace Morelike with Wikivectors as the recommender model for the Related Pages feature. Future studies may compare the performance of these two algorithms again if there are updates to how Wikivectors generates recommendations or if new features that use Morelike-based recommendations are developed.
Rater preferences and rationales
editMost respondents provided succinct rationales for their list preferences (question #6 above).
| article and condition | choice | In a sentence or two, please describe why you do (or do not) prefer one list over the other? |
|---|---|---|
| A Cure for Wellness 1 | strongly prefer wikivectors | Because List B has more general film info, and that's what I'm interested in rather than the info on the specific actors, director, etc. |
| A Cure for Wellness 1 | strongly prefer morelike | The links in list A are directly related to the movie. |
| A Cure for Wellness 1 | strongly prefer morelike | Here, list A clearly is more to the point than list B. List B has all generalized topics. |
| A Cure for Wellness 1 | somewhat prefer morelike | A is just more about the people in the movie - nice to read. |
| A Cure for Wellness 2 | strongly prefer wikivectors | I'm more interested in upcoming films to be released than actors. |
| A Cure for Wellness 2 | strongly prefer wikivectors | I think the topics in A are more the type of articles that I would read. |
| A Cure for Wellness 2 | strongly prefer wikivectors | A has a coherent themed list full of movies that would be relevant to my interest if I looked up the horror A Cure for Wellness 2. B has articles about people that I has literally nothing to do with horror. |
| A Cure for Wellness 2 | strongly prefer wikivectors | List B is a disaster, none of those things are relevant to the topic of horror or movies. A at least has movie list for me to discover new films/ |
| Adele 1 | no preference | I think list B is slightly more relevant, and has more interesting content overall. |
| Adele 1 | somewhat prefer morelike | All links are about Adele in List B. Interesting articles. Something I would Click. |
| Adele 1 | strongly prefer morelike | It is more focused on Adele. |
| Adele 1 | strongly prefer morelike | B has more articles related to the subject and her songs. |
| Adele 2 | somewhat prefer morelike | I love Adele and anything having to do with music |
| Adele 2 | somewhat prefer wikivectors | List B has the discography, so is probably better |
| Adele 2 | somewhat prefer wikivectors | It has the ability to keep you interested. |
| Adele 2 | somewhat prefer wikivectors | I prefer list B because list A didn't have the proper picture. |
| Ben Rhodes (White House staffer) 1 | strongly prefer morelike | Because List A had articles that were much more likely to be relevant to someone who had looked up the initial article. |
| Ben Rhodes (White House staffer) 1 | somewhat prefer morelike | The list A seems to have more information in it and is more relevant. |
| Ben Rhodes (White House staffer) 1 | somewhat prefer morelike | because of my choice is slightly favour for choice a because the choice is good at relevant and perfect to be the example |
| Ben Rhodes (White House staffer) 2 | strongly prefer morelike | List B listed different Staffers and other positions in the White House which was more similar to the article. |
| Ben Rhodes (White House staffer) 2 | strongly prefer morelike | I am more interested in learning about the people in list b |
| Ben Rhodes (White House staffer) 2 | somewhat prefer morelike | B has more about the obama administration which is who the guy is in this article |
| Bruno Mars 1 | strongly prefer morelike | List B contains more information about Bruno Mars than the other. I am much more interested in him than an actress or producer. |
| Bruno Mars 1 | somewhat prefer morelike | List B had the information related to the source topic. |
| Bruno Mars 1 | somewhat prefer morelike | B seems to be more related than A, which seems kind of random |
| Bruno Mars 1 | strongly prefer morelike | if i was looking into bruno mars list b is much more relevant list a seem to go off topic |
| Bruno Mars 2 | strongly prefer wikivectors | I prefer B because it will allow me to find more information about similar people. The items in A are all things that I would expect to be located somewhere in the Bruno Mars article so I don't find it necessary to list them below everything else in the article. |
| Bruno Mars 2 | strongly prefer morelike | List B has articles that are not on topic. List A is clearly the better choice as it has articles that you would naturally want to see next. |
| Bruno Mars 2 | somewhat prefer morelike | I prefer this list (A) because it is related to Bruno Mars unlike list B, which seems to veer off topic. |
| Bruno Mars 2 | strongly prefer morelike | B does not interest me |
| Daft Punk 1 | strongly prefer wikivectors | List B has more details about the work of Daft Punk |
| Daft Punk 1 | strongly prefer morelike | It seems to have the correct information. |
| Daft Punk 1 | strongly prefer morelike | I don't know much about Draft Punk but I wouldn't mind reading these in A. |
| Daft Punk 2 | no preference | I do not prefer any list over the other. |
| Daft Punk 2 | somewhat prefer morelike | I would be interested more in concert information from a previous show, than the awards listed in A. |
| Daft Punk 2 | no preference | Both have some pretty relevant information to daft punk but if I were to look up daft punk I would be highly interested in learning some new artists similar to daft punk to listen to. That is how music discovery works. Both list honestly have such similar content it is so hard to choose. |
| David Miscavige 1 | strongly prefer morelike | I prefer list A because it suggests a timeline of the church and more information on hubbard's church of spirituality which is related to scientology. It also has one individual that is related to this topic with a photo that is known as a whistleblower. This is preferable to the one individual mentioned in List B without a picture. |
| David Miscavige 1 | somewhat prefer wikivectors | I like list A because it includes an American whistle blower that is trying to end something that he's passionate about. |
| David Miscavige 1 | no preference | List B seems to be more interesting to me. |
| David Miscavige 2 | no preference | They are very similar and I don't think I would like any of the articles. I don't see much difference in the articles. |
| David Miscavige 2 | no preference | They both have related information to public figures. |
| David Miscavige 2 | no preference | List B has a link to an article of a Scientology whistleblower which sounds interesting, but list A has the history of Scientology and celebrities which also sounds interesting. |
| David Miscavige 2 | strongly prefer morelike | I choose B simply because it appears to be more relevant, there are pictures that represent the organization and a little about the back ground. |
| Dementia with Lewy bodies 1 | strongly prefer morelike | List b is more related to the disease |
| Dementia with Lewy bodies 1 | strongly prefer morelike | List B has more articles releating to the disease. |
| Dementia with Lewy bodies 1 | somewhat prefer morelike | In this case, list B definitely did a better job. I didn't have to guess if the articles were related to the source article - they were straight-forward. |
| Dementia with Lewy bodies 2 | no preference | I would like to read about Robin Williams disease involved with Lewy Bodies. |
| Dementia with Lewy bodies 2 | no preference | Like science |
| Dementia with Lewy bodies 2 | strongly prefer morelike | List B includes articles that are only tangentially related to the topic, so list A looks better |
| Dementia with Lewy bodies 2 | strongly prefer morelike | I prefer list A over B because the information is more relevant. |
| Elizabeth II 1 | somewhat prefer wikivectors | Because I don't know that much about the monarchy, and would like some general background info to give context to the main article. |
| Elizabeth II 1 | strongly prefer wikivectors | I'm more interested in reading about the institutions than the people. |
| Elizabeth II 1 | somewhat prefer wikivectors | I found List A has more relevancy to the source article. It simply has more interesting articles. |
| Elizabeth II 2 | strongly prefer morelike | List A seems to have things that are way more similar to the article. If I were to group them together they would overlap. |
| Elizabeth II 2 | somewhat prefer wikivectors | List B has more variance than list A. |
| Elizabeth II 2 | somewhat prefer wikivectors | I like the overall history of England rather than a bio of a king |
| George Charles Boldt Jr. 1 | strongly prefer wikivectors | List B was much more related to the topic. |
| George Charles Boldt Jr. 1 | somewhat prefer morelike | I would like to know more about the Waldorf-Astoria hotel. |
| George Charles Boldt Jr. 1 | somewhat prefer morelike | I prefer list A because it expands on information about the hotel, that what was mentioned in the preview. |
| George Charles Boldt Jr. 1 | somewhat prefer morelike | I prefer list A because it has more to do with the Walforf-Astoria. However, list B actually contains the article which deals directly with Mr. Boldt. |
| George Charles Boldt Jr. 2 | strongly prefer wikivectors | There are several articles about that man who is is need of |
| George Charles Boldt Jr. 2 | no preference | Na |
| George Charles Boldt Jr. 2 | strongly prefer morelike | A has like one hotel relevant article, but B has 2, be is better for research by default. |
| Kidnapping of Shannon Matthews 1 | strongly prefer wikivectors | The b list contains more articles about missing people similar to the story in the header. |
| Kidnapping of Shannon Matthews 1 | somewhat prefer wikivectors | There was a similar case I would like to read about involving another missing girl. |
| Kidnapping of Shannon Matthews 1 | strongly prefer wikivectors | All material in list b is more related |
| Kidnapping of Shannon Matthews 1 | somewhat prefer wikivectors | I prefer list B because it provides information that might be similar to the topic and that was mentioned in the article preview. |
| Kidnapping of Shannon Matthews 1 | strongly prefer wikivectors | List B has similar stories to what the article is about. List A is about the area of England. |
| Kidnapping of Shannon Matthews 2 | somewhat prefer wikivectors | I prefer A because it gives me information about other events similar to the one in the article I am reading. With B, I am not quite sure why those items are recommended. |
| Kidnapping of Shannon Matthews 2 | strongly prefer wikivectors | The article mentions Madeline McCann and A had at least two articles that were similar. |
| Kidnapping of Shannon Matthews 2 | strongly prefer wikivectors | I think reading about other cases would be more interesting to read about - than the lists provided in B. |
| Matt Ryan (American football) 1 | strongly prefer morelike | List A has more info about football over the other list. I am more interested in the players vs an actor. |
| Matt Ryan (American football) 1 | no preference | I have no real interest in football players. |
| Matt Ryan (American football) 1 | somewhat prefer wikivectors | i love matt ryan |
| Matt Ryan (American football) 1 | somewhat prefer morelike | The articles in A and B are very similar and I would go with either one, however column B has an article about an actor with the same name as the featured article. Therefore I would be more inclined to read from A because it contains similar information to my core search. |
| Matt Ryan (American football) 2 | somewhat prefer morelike | I prefer List B over List A because List B seems to have more information than List A. |
| Matt Ryan (American football) 2 | somewhat prefer morelike | because it words it better. It says what I am thinking. |
| Matt Ryan (American football) 2 | somewhat prefer wikivectors | I prefer list A to list B because it relates more to Matt Ryan. |
| Michael T. Flynn 1 | somewhat prefer wikivectors | I would not pick list B because it seemed unrelated a little bit and it seemed to have less about the things they were recommending |
| Michael T. Flynn 1 | strongly prefer morelike | List B links to articles that give more general information and I could learn more from. |
| Michael T. Flynn 1 | no preference | I like the clarity of information in Article A than B. |
| Michael T. Flynn 1 | somewhat prefer wikivectors | List a was all relatively the same material on different people. |
| Michael T. Flynn 1 | strongly prefer wikivectors | The articles in column A give more of a preview so I know I will be interested in the articles. |
| Michael T. Flynn 2 | somewhat prefer morelike | The things surrounding Flynn are of more interest than those who held similar rank to him. If I were interested merely in the military, I may prefer list B, but I would want to know about the things surrounding Flynn. |
| Michael T. Flynn 2 | somewhat prefer morelike | One has better views than the other. I think it says more to me. |
| Michael T. Flynn 2 | somewhat prefer morelike | While B has more similar content, A is of an example of intelligently generated list. If anybody looks up flynn they are looking him up based on the current news. List A correctly generates the list based on the news which is far superior to list B. |
| Moana (2016 film) 1 | strongly prefer wikivectors | I am more interested in the mythology than I am with the animators. |
| Moana (2016 film) 1 | no preference | I have no preference towards either recommenders at all. |
| Moana (2016 film) 2 | somewhat prefer wikivectors | I prefer list B because it talks about the Hawaiian mythology from the movie. |
| Moana (2016 film) 2 | somewhat prefer morelike | I think that List B is really interesting because it focuses more on the mythology behind the movie, but List A seems a better option for movie buffs because it has Zootopia, which is a very relevant movie in the same category, and the two directors involved. If I was looking for tangible information about the movie production and similar movies to this one, I would choose List A, although the mythology is more interesting to me personally, but it might not be that way for everyone. |
| Moana (2016 film) 2 | strongly prefer morelike | I feel like I would prefer more topics on animation if I were looking into this movie, as opposed to history thats kind of related |
| Moana (2016 film) 2 | strongly prefer morelike | List A is more related to the main article as well as having data in the same category that would be connected to the main article. |
| Moonlight (2016 film) 1 | somewhat prefer morelike | List A had more information connected to the source topic. |
| Moonlight (2016 film) 1 | strongly prefer morelike | I think A is better because it leads to some more general categories about movies. |
| Moonlight (2016 film) 1 | somewhat prefer morelike | A is all related to the article, while B is a little more random. |
| Moonlight (2016 film) 1 | somewhat prefer morelike | A has articles directly related to the film and B has only a couple articles related and is not interesting. |
| Moonlight (2016 film) 2 | no preference | I prefer list B because it has three articles directly pertaining to the source article. |
| Moonlight (2016 film) 2 | somewhat prefer wikivectors | I enjoy reading about the actors associated with the film, though some may enjoy reading about the film itself. I would rather branch out. |
| Moonlight (2016 film) 2 | strongly prefer morelike | B has more information |
| Neil Gorsuch 1 | somewhat prefer wikivectors | List B seems more relevant and consistent. Both are not very interesting tnbough. |
| Neil Gorsuch 1 | strongly prefer morelike | Again, the presentation of the articles in column A are better. The photos make the the articles more appealing. |
| Neil Gorsuch 1 | somewhat prefer morelike | I prefer list A because it seems more relevant to the topic and offers information that one might be interested in after reading the topic. |
| Neil Gorsuch 1 | strongly prefer morelike | List A contains articles that are germane to the topic. List B is comprised soley of biographical sketches of judges (none of whom are Judge Gorsuch). |
| Neil Gorsuch 2 | no preference | I think that list A is better because if you were looking for more information about supreme court judges, you would want the list that has the names of other judges so that you can get a full picture of who else is serving. The information in list B seemed almost disjointed and a bit unrelated to each other as a whole. |
| Neil Gorsuch 2 | strongly prefer morelike | One is about a topic I'd find boring while one has varied interesting topics. |
| Neil Gorsuch 2 | strongly prefer morelike | List B is more likely to be of interest, as Gorsuch is of interest currently because of his involvement with the Supreme Court and Trump's presidency. Reading about other judges is not that interesting. |
| Neil Gorsuch 2 | somewhat prefer morelike | The B list clearly has data related to the main article and it looks to be connected by what is presented. |
| Neil Gorsuch 2 | somewhat prefer wikivectors | List B is more eye catching and seems more interesting. |
| Oroville Dam 1 | somewhat prefer morelike | I prefer B because it is related to Oroville and has links around the Oroville crisis. |
| Oroville Dam 1 | somewhat prefer morelike | In this example, it really would depend on why I search for oroville dam in the first place. Was I searching for California Dams? Or, was I searching for articles about Oroville itself? I made the assumption I was looking only for info on Oroville. |
| Oroville Dam 1 | strongly prefer morelike | I prefer B because it has more articles related directly to the dam in the article instead of descriptions of other dams. |
| Oroville Dam 1 | somewhat prefer morelike | List B had more relevant articles. |
| Oroville Dam 2 | no preference | I don't prefer either one as they both provide links to useful information, the first about items near the Oroville Dam, and the second giving me information about other dams. |
| Oroville Dam 2 | strongly prefer morelike | it has info on a crisis that is currently happening |
| Oroville Dam 2 | no preference | I prefer List A over List B because List A seems to have more information that List B. |
| Oroville Dam 2 | strongly prefer morelike | list a has more things related to orovollie dam |
| Oroville Dam 2 | strongly prefer morelike | I prefer A, because it expands on the existing article in an interesting way. I feel like the other will contain much of the same information. |
| Riverdale (2017 TV series) 1 | strongly prefer morelike | I think list A is better because it provides a more logical movement from this article in that it provides a plot overview for the pilot episode in the series and also provides two individual actresses pages that are in this series. List B provides some more information but it is mostly rooted in the history of the series rather than the current going-ons of the series now. |
| Riverdale (2017 TV series) 1 | somewhat prefer morelike | I would prefer to see more information about the actors and the pilot episode which would be list A |
| Riverdale (2017 TV series) 1 | somewhat prefer wikivectors | List B sounds more believable to me. |
| Riverdale (2017 TV series) 1 | somewhat prefer wikivectors | I would read either the articles in both lists |
| Riverdale (2017 TV series) 2 | no preference | B has more related to the current tv series of this |
| Riverdale (2017 TV series) 2 | no preference | They both have a lot of random entries. Neither really stands out to me. |
| Riverdale (2017 TV series) 2 | strongly prefer wikivectors | I would be interested in knowing more about this show. I felt like the first list provide such information. |
| Sexuality after spinal cord injury 1 | somewhat prefer morelike | The articles are more related to what would be lost in the situation described in the header. |
| Sexuality after spinal cord injury 1 | somewhat prefer wikivectors | Some good links that are on topic. And some general information. (List B) |
| Sexuality after spinal cord injury 1 | somewhat prefer wikivectors | If I were researching various reasons for sexual dysfunction, List A is better. I made the assumption that I was researching sexuality and disabilities which made list B better. |
| Sexuality after spinal cord injury 1 | somewhat prefer morelike | It seems to be more relevant. |
| Sexuality after spinal cord injury 1 | somewhat prefer wikivectors | Although list A does have articles dealing with sexual dysfunction, the articles do not specifically related to post-trauma sexual issues. The articles on list B are more closely related to the topic. |
| Sexuality after spinal cord injury 2 | somewhat prefer wikivectors | There are articles about sexuality and disabilities. So Assuming that those articles talk about what its like to have a sex life when injured. |
| Sexuality after spinal cord injury 2 | strongly prefer wikivectors | List A has a broader range of articles that are less clinical than list B |
| Sexuality after spinal cord injury 2 | no preference | I think both A and B are similar with regards to the topic on hand (sexuality after spinal cord injury 2). |
| Sexuality after spinal cord injury 2 | strongly prefer wikivectors | List A is much related it with the article |
| Split (2016 American film) 1 | no preference | Neither of these two lists has additional content that are super compelling; although, I'd probably read List B next to learn more about the leading actress in this movie. |
| Split (2016 American film) 1 | no preference | Because of my choice b is more slightly than slightly a because they are good at in wiki search |
| Split (2016 American film) 1 | somewhat prefer morelike | List A refers to the actual movie director |
| Split (2016 American film) 1 | no preference | They seem to be the same and I have no preference towards either. |
| Split (2016 American film) 2 | no preference | Both of the list has content both relevant to the main topic and relevant to readers who might be interested in the main topic. Both list has information on important people to the film Split. |
| Split (2016 American film) 2 | strongly prefer morelike | Both lists have kind of random entries but at least list B features the director. |
| Super Bowl LI 1 | no preference | I don't prefer one list over the other they both seemed thorough and on topic |
| Super Bowl LI 1 | somewhat prefer wikivectors | At least I could read about the stadium used instead of another game. |
| Super Bowl LI 1 | no preference | list a has some info on future super bowl that would interest me |
| Super Bowl LI 1 | strongly prefer wikivectors | List A is more varied about the topic at hand. |
| Super Bowl LI 1 | strongly prefer morelike | List B was focused just on Super Bowl games themselves. That appears to be more consistent than list A which has one item about the stadium and two about other superbowls. |
| Super Bowl LI 2 | no preference | I like to read |
| Super Bowl LI 2 | somewhat prefer morelike | I did not like that B had an article about the stadium. It did not fit in line with what I was reading, and I did not think I would be interested |
| Taboo (2017 TV series) 1 | somewhat prefer wikivectors | List B has more drama series and their details. |
| Taboo (2017 TV series) 1 | strongly prefer morelike | The actors are more related to the movie |
| Taboo (2017 TV series) 2 | somewhat prefer morelike | The list in B at least as an article about Tom hardy, which he is the creator. |
| Taboo (2017 TV series) 2 | somewhat prefer wikivectors | Na |
| Taboo (2017 TV series) 2 | no preference | They both share the same type of information. |
| Taboo (2017 TV series) 2 | strongly prefer morelike | I am interested in the writer, in this case, and possible other things he has had a part in. The other list were other British series that I am not interested in |
| The Lego Batman Movie 1 | strongly prefer morelike | I prefer list B over list A because it concentrates on lego related things that are tied into DC comics universe. I think this ties in with the movie better than list A, which suggests a completely different lego movie. |
| The Lego Batman Movie 1 | somewhat prefer morelike | B provides more information of batman DC |
| The Lego Batman Movie 1 | somewhat prefer wikivectors | List A was a little better and referred to the movie prequel |
| The Lego Batman Movie 1 | no preference | they were both pretty on point but if I went with one it would be a |
| The Lego Batman Movie 2 | strongly prefer morelike | the first list has info about the joker |
| The Lego Batman Movie 2 | somewhat prefer wikivectors | I feel like List A has more "random" information, like including information about the Joker which I don't know how applicable that really is to the film itself other than he may be a character in it since he's well known in the Batman franchise. I feel like list B is better because it focuses more on the director and the movies that came before it. |
| The Lego Batman Movie 2 | no preference | I personally think that both lists are very similar. Therefore I dont really prefer either. |
| The Lego Batman Movie 2 | somewhat prefer wikivectors | I am not really interested in DC comics, so I chose list B. |
| Tom Brady 1 | somewhat prefer wikivectors | B has more pictures making it more interesting to view |
| Tom Brady 1 | no preference | Because the over two articles are good at same line i would chose to prefer not in the correct order is is exact one also i prefer to either a nor b |
| Tom Brady 1 | strongly prefer morelike | list a is more on topic of quaterbacks |
| Tom Brady 1 | strongly prefer wikivectors | I would want to ready more about Tom Brady - not former quarterbacks. |
| Tom Brady 2 | no preference | I liked the diversity of the articles in List A |
| Tom Brady 2 | strongly prefer morelike | List B is comprised strictly of football related articles whereas list A is not (entirely) |
| VX (nerve agent) 1 | strongly prefer morelike | It contains details on the article ans some related things same as article |
| VX (nerve agent) 1 | strongly prefer morelike | List b has more information related to the article |
| VX (nerve agent) 1 | somewhat prefer morelike | It seems to make the most sense. |
| VX (nerve agent) 2 | somewhat prefer wikivectors | One has articles that I would be more likely to remember + pay attention to. |
| VX (nerve agent) 2 | somewhat prefer morelike | The A list has more data and articles related to the main article in this set. I would choose on behalf of those reasons. |
| VX (nerve agent) 2 | strongly prefer morelike | I chose A because it has the compound break down, shows diagrams and explains in Scientific terms the properties of the agent. |
| VX (nerve agent) 2 | strongly prefer morelike | B has things im not interesting in reading about |
| Valentine's Day 1 | somewhat prefer morelike | List B has a couple much more compelling articles I'd like to read next. |
| Valentine's Day 1 | somewhat prefer morelike | List B because i want to read about lupercalla. |
| Valentine's Day 1 | no preference | I think that the articles in either list are interesting. |
| Valentine's Day 2 | strongly prefer morelike | Na |
| Valentine's Day 2 | strongly prefer morelike | I think list A is more relevant to the topic and is much more to my taste, it has articles on st. valentine and the history which i like. |
| Valentine's Day 2 | strongly prefer morelike | I prefer A over B because it has more of an explanation appears to be more concrete and has more pictures. |
| Valentine's Day 2 | strongly prefer morelike | This list was definitely more related to the topic and would have provided me more information linked to it. the second list was a variety of other things |
Ad hoc examination of these rationales suggests that they are informative and potentially useful. Thematic analysis of these responses would likely yield additional context on why MoreLike recommendations are preferred. However, given the results presented above, that analysis is not planned at this time.
Limitations
editIn this study, Amazon Mechanical Turk workers are functioning as 'stand ins' for Wikipedia readers. So consider the following potential limitations/confounds when interpreting these results.
- Motivation: Turkers are motivated by payment for completing the task, and possibly also by how interesting the task is. This is different from the motivation of Wikipedia readers. In a natural context, a reader who encounters Related Articles recommendations would choose to click on a recommended article because they were genuinely interested in reading that article next.
- Level of topic knowledge: readers encounter Related Pages recommendations after (presumably) reading the majority of the source article. In the current study, Turkers were asked to evaluate the recommendations after reading a short preview of the source article, so they probably don't know as much about the subject as a Wikipedia reader would, at the point when they're asked to evaluate the recommendations. The set 2 sample (high-traffic articles) was used to partially correct for this source of error, because US-based Turkers are much more likely to be at least passingly familiar with those topics than they were with the topics presented in set 2 (low-traffic articles).
- Context of the recommendations: this study only compares these two algorithms in the context of the Related Pages feature. This feature is implicitly geared towards readers who are in "deep dive" mode: it's presumed that most people who make it all the way to the bottom of a mobile article are reading for detail, not skimming or checking on a quick fact. In other product contexts (for example, recommendations on the mobile app homepage), the results may be different because people's needs and interests change based on what they're doing, and they may respond to information differently based on the context in which that information is being presented. See Research:Characterizing Wikipedia Reader Behaviour for more information on reader motivations and use cases.
- Demographics: For practical reasons, all turk workers who participated in this study were based in the United States. However, people from different cultures and/or geographies may have different preferences when it comes to Related Articles.
- Study site: this study was conducted on English Wikipedia. Morelike and WikiVectors may perform differently in different languages, or on larger or smaller Wikipedias.
- Choice of articles: This study looked at a set of very high traffic articles, and a set of relatively low-traffic articles. But there are a lot of "middling" articles out there that get a good amount of traffic, but aren't among the top 100. WikiVectors and Morelike may perform differently for these mid-tier articles.
See also
edit- Wikipedia Navigation Vectors
- ReadMore web application (based on WNV model)
- CirrusSearch "morelike" query documentation
- Extension:RelatedArticles
- mw:Wikimedia_Apps/Feeds
- Research:Task recommendations/Qualitative evaluation of morelike
- Mw:Extension:RelatedArticles/CirrusSearchComparison
Subpages
editPages with the prefix 'Evaluating RelatedArticles recommendations' in the 'Research' and 'Research talk' namespaces:
Research:
Research talk:
References
edit- ↑ McNee, S. M., Kapoor, N., & Konstan, J. A. (2006, November). Don't look stupid: avoiding pitfalls when recommending research papers. In Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work (pp. 171-180). ACM. (PDF)
- ↑ Michael D. Ekstrand, F. Maxwell Harper, Martijn C. Willemsen, and Joseph A. Konstan. 2014. User perception of differences in recommender algorithms. In Proceedings of the 8th ACM Conference on Recommender systems (RecSys '14). ACM, New York, NY, USA, 161-168. DOI=http://dx.doi.org/10.1145/2645710.2645737 (PDF)