Grants talk:Project/Stephane/Kiwix/Final

Thanks for the enthralling read :)

Couple of questions/comments:

You say 30% retention - is that by installs, usage, or...? I'm curious to learn more about how the app is being used and what metrics you're tracking.
For your next steps, a search by symptoms sounds like it could be an interesting and useful project!
It sounds like this grant was of majority benefit to Kiwix internally (which is great) and included improvements/ideas for end users. I'm not sure if that's because you're reporting more on that subject or that's where the bigger investment went. It would be great in the next grant to learn more about what you learned/accomplished for your users through this work, and how you know you did. We get that a little bit with downloads and retention, but a deeper dive would be great.

AGomez (WMF) (talk) 22:47, 12 January 2018 (UTC)Reply

Hello Anne

1. The 30% retention rate is that given by the Google Store interface, and by install. It is apparently a standar industry metric and the Foundation should have these somewhere as well for its own app (couldn't figure out where/if it's been published but it would be a great comparison). As it is explained on the Google interface, retention means that for 100 people installing the Wikimed app, a little more than 30 of them still use it after a month. Here are the actual numbers for our largest languages (android only):

App	Conversion	Retention D+1	Retention D+30
Kiwix	30%	63%%	37%
English Wikimed	13%	89.6%	71.2%
Wikimed Mini (English)	25%	79%	50%
Arabic Wikimed	14%	86%	59%
French Wikimed	20%	93%	75%
German Wikimed	12.5%	86%	70%
Spanish Wikimed	17%	81%	64%

A few comments:

"Conversion" measures people coming to the App's description page and actually downloading the app afterwards. Why so much variation, from 30% for Kiwix to 12-20% for Wikimed? I don't know: maybe the Kiwix description text is more appealing; maybe people are more excited at the idea of having the whole wikipedia on their phone rather than a bit of it, and then change their mind when they realize how big that still is; maybe our French is better written than our German; interestingly, reducing the app's size seems to both favor downloads and uninstalls; suggestions on how to improve all of this are more than welcome!
We actually do a lot better than 30% retention at D+30 (can't remember how I got it so off). Based on this article ("we can see that the average app loses 77% of its users within the first 3 days after the install. Within 30 days, it’s lost 90%") we seem to be at the top in terms of retention, which is nice (to be fair, the same site indicates somewhere else that medical apps in general tend to fare better than average);
Ideally, we should try to collect data at D+90: unfortunately the Play store doesn't seem to offer this metric, or I couldn't find it.

2. Indeed!

3. Re: feedback, the sad truth is that it's easier to report on what we did than to go out and ask people how they feel about the app (e.g. similar to the user feedback report the Foundation did last year): we simply do not have that kind of budget. The retention metrics above seem to indicate that users are overall pretty happy with it, and the rating hasn't really moved compared to when we started the project (and it already was pretty high). However any suggestion on what we should/could look for is also greatly welcome. Stephane (talk) 09:48, 15 January 2018 (UTC)Reply

Hey Stephane! Thanks for the follow up. We do track the 7- and 30-day retention through the Play store as well. I can dig up those numbers at some point for comparison's sake.

For the conversion being comparatively lower in the Wikimed app, if you want to experiment with that I believe that you can a/b test store info in the Play store. Your screenshots and text for Kiwix are in a really different tone than those of the Wikimed app and it might be something to consider.

For point 3, it would be great for you to include analysis on retention (like you did above) in future reports so that we can see how you're looking at end-user benefit. I totally understand that it's expensive to do qualitative research and a new work stream to integrate, but I imagine there are some light-weight ways to think about connecting more with your users and what they want/need that we could support you in. Let's keep in touch about it.

AGomez (WMF) (talk) 20:50, 16 January 2018 (UTC)Reply

Add topic