Wikimedia monthly activities meetings/Quarterly reviews/Research, Design Research, Analytics, and Performance, July 2016

Notes from the Quarterly Review meeting with the Wikimedia Foundation's Technology I: Research, Design Research, Analytics Engineering, Performance teams, July 14, 10:00 - 11:30 AM PT.

Please keep in mind that these minutes are mostly a rough paraphrase of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases and other official material

Present (in the office): Dario, Ellery, Leila, Michelle, Ori, Tilman, RobLa, Jaime, Madhu, Grace, Katherine, ; participating remotely: aotto, Dan, Darian, Faidon, Gilles, Greg, halfak, Jonathan, Maggie Dennis, Marcel Ruiz Forns, Mark, nuria, Samantha, Wes

Backup Datacenter

edit
 
  • Wes: Great rollout. We had to make adjustments, moved out a quarter do be prepared and do this well. Good learning and improvements.
  • Katherine: reiterating last session; huge accomplishment for the team, community members were very positive; confidence in the results; board appreciated the work

Research and Data

edit
 

Objective: Revscoring in production

edit
 
  • Dario: first goal, ORES

most important milestone for this project. Both service (API) and client interface in the production cluster and available to users. Big thanks to Amir Sarabadani and others from Release Engineering, Operations and our volunteers.

 

Objective: Revscoring in production - Successes and Misses

edit
  • Dario: related updates on ORES. Wikimania session to socialize the changes in ORES by Amir.
  • Dario: 14 new models, substantial performance improvements
 

Objective: Discussion modeling

edit
  • Dario: Also related to the application of machine learning methods. Discussion modeling project (Detox) has the goal of providing insights into talk page discussions when it comes to interactions that may drive people away. This is a collaboration with different stakeholders, the community department, research-and-data, and Jigsaw. We have the first results of it available this quarter. Data-set released, for now there has not been an announcement. We will make an announcement one all diffs are scored by the algorithm. Ellery also did an analysis of users that are blocked to understand why users are blocked.
 

Other successes and misses

edit
  • Dario: It divides in 3 steps. Understanding harassment, designing interventions, [what is the third step? --Leila]
 

Objective: Research outreach

edit
  • Dario: hosted 2 pretty major outreach events. Co-organized by Stanford University, Wiki Research in WWW and ICWSM conferences. We also cohosted WikiCite with WMDE. A lot of effort for doing these outreach. The results have paid off for all the effort that we put in for organizing them.
  • Learnings: how to organize funding we get for events such us this one? We had no network for the events we organized.
 

Other successes and misses

edit
  • Dario: Wiki workshop largest concentration to date of Wikimedia research scientist. A report for WikiCite for the community and the funders in the coming quarter.
 

Objective: Reader segmentation

edit
  • Dario: Carry-over goal. Concluded that this was a miss, as not yet completed.
  • figure out next steps, still a lot of interesting results that could be drawn from this data
  • Katherine: partnership with Reading team?
  • (Dario:) yes, started as collaboration on technical level, in past 2 quarters shared findings with them
 

Other successes and misses

edit
  • Dario: Wikistats 2.0. Had been maintained quietly by Erik Z, but he'll focus on other tasks now. As of this quarter, the official maintenance of wikistats is handed over to the Analytics team.
  • Dario: Wikipedia navigation vectors data release which we expect to increase the research on understanding Wikipedia readers and logs.
  • Dario: published Research FAQ after request by FDC
 

Core workflows and metrics

edit
  • Dario: we did not host a showcase in the last quarter. You should see them coming back and happening on the regular basis starting this month.
 

Appendix

edit
  • Katherine: workshop (about harassment at Wikimania[?]) went really well. q: other than attendees, what do you see as outcomes from that? conversations, research, projects?
  • Leila: conversations around three projects. empowers when we want to start other projects, people get together and they may start something that comes up a year from now. we can announce maybe 6mos year from now. hard to quantify concrete outcomes of such workshops, but the relationship built are really important, enable collaboration
  • Dario: we org dev summit every year. research community is as important as developer community. tighter integration between researchers' priorities and community needs
  • Maggie: we heard from German community just last week that harassment research project is helpful work
  • Katherine: appreciated documentation, so people know who is doing what in research. I know that's not nearly as fun as working on projects, but it's important
 

Design Research

edit
  • Jonathan: Filling in for Abbey who is in a workshop.
 

Objective: Personas

edit
  • Jonathan: not completed for several reasons: most importantly loss of partners in org, in particular Kaity (design) also, had to make room for New Readers research (Mexico etc.)
 

Personas

edit
  • Jonathan: going forward, emphasis on integrating these personas into product research and design (rather than creating new personas or refining existing ones)
 

Objective: Evaluative Design Research

edit
 

Evaluative Research

edit
  • Screenshot credit: Pau Giner
 

Objective: Deep dive / contextual inquiries

edit
  • Collaboration with Reboot; we're prioritizing getting the information published
 

New Readers Contextual inquiries

edit
  • Each deep dive took approx 2 weeks
 

Objective: Collaboration with UW on Survey

edit
  • research prompted by questoin asked by Trevor; how do people learn? collaboration deployed to students to find how they find informatino. summary provided at last month's metrics. this was design research's first
 

Collaboration with UW on Survey

edit
  • noting UW's publication of data
 

Objective: Benchmarking / Tooling

edit
 

Benchmarking / Tooling

edit
  • new user testing platform.
  • Jonathan: relationship with previous vendor was not working for several reasons. primary goal was to get something in place to meet product team's needs right now. Re-scoped to just securing a contract; meets our current needs and we think our future needs.
  • Wes: thanks to Legal team
  • Jonathan: Manprit and (?)
  • Jonathan: we have 3 seats for 3 researchers. we can use this for any research, not just 3 verticals. our previous contract
  • Jonathan: we're phasing out some of my old responsibilities; Chris Schilling taking over some of them
  • Katherine: thank you
 

slide 25

edit
 

Analytics Engineering

edit
  • Nuria presenting
  • our team : Madhu is transitioning to Ops. We work quite a lot with ops
  • we use velocity as KPI, went down during this q - several of team took vacation, Wikimania
 

Objective: Public by default

edit

also track data by country, but only release aggregate publicly, for privacy reason

 

(data)

edit
  • Nuria: WIkistats transition (old version has been up for the better part of a decade) so far relied on XML dumps
 

Objective: Wikistats 2.0

edit
 

Public events stream

edit
 

Druid (screenshot)

edit
  • Nuria: tool for e.g. some people in Reading (JonK, Tilman), much easier way to consume our data, queries takes seconds in Druid instead of minutes in Hive. WMDE loves this too
  • Dario: for Wikidata?
  • Nuria: yes
  • Dario: e.g. Russia is top traffic source for Wikidata
 

Objective: Better Data Access

edit
 

Objective: Operational Excellence

edit

our team has big operational (component), e.g. Cassandra scaling issues

 

Other successes and misses

edit
  • Nuria: Varnish/Varnishkafka upgrades
  • Nuria: Launched http://analytics.wikimedia.org : one place for our tools
  • Michelle: I'm so impressed with your emphasis on privacy. curious about API response rate. Practical impact on API users?
  • Nuria: impacted by cache. only heavy users of the tools. those people see the impact. overall if you request data, you aren't hitting storage
  • Dan: most people accessing via PageView tool by MusikAnimal, he's handling from community view
  • Dario: it was all really great this quarter. please demo Druid
  • Katherine: making unique dataset public is fantastic. have it for all projects?
  • Nuria: yes, did it originally for enwiki, but then for all. Also projects that are too small to yield
  • Katherine: overall de-duplicated?
  • Nuria: yes, it's something Reading/Tilman have requested. investigate performance/traffic considerations of additional cookie https://phabricator.wikimedia.org/T138027
  • Katherine: is that a goal currently?
  • Nuria: not a goal this q, perhaps next q, depends on ops/perf feedback
  • Katherine: recognize it's not something we would use in product decisions, but eg important for board (important vs urgent vs informational)
  • Nuria: banner consultation[?]
  • Katherine: interest to external audiences
 

Dashboards and Data analysis

edit
 

Performance team

edit
 

KPI: First paint time

edit

Ori: theory: traffic moving to mobile, but don't know

 

KPI: Page save time

edit

Ori: regressed during quarter. one major regression (Authmanager)

 

KPI: Page save time

edit

Ori: graph that goes to today; we're actually better now

 

Objective: Thumbor

edit

Ori: tight coupling to mw. goal was prod deploy. we have VM deployment. team decided to package for Debian. coming along quite well. Filipo provided enormous time and assistance

 

Objective: Performance Inspector

edit

Ori: not deployed; asked Peter to help Reading team with lazy loading. looking good for next q

 

Multi-DC

edit

Ori: talked a lot about that earlier.

 

other successes and misses

edit
  • Ori: Running MediaWiki from secondary datacenter without writing. reducing slave lag (read slide)
  • Ori: Aaron Schulz made optimistic saving work for majority of edits
  • Ori: expect substantial impact of inlining CSS on first paint time
  • Katherine: so optimistic saving is live?
  • Ori: yes
  • Katherine: slave lag: that's in addition to multi-DC work?
  • Ori: yes. don't want to serve stale content, this imposes a lag tax
  • Katherine: does this (metric) mean team will work further on this
  • Ori: it's a huge win already, but.. we've made a lot of progress in past 3 mos. 5 secs of slave lag still a problem for modern web application
  • Dario: as top KPI?
  • Ori: it's a longer bet, switchover to Dallas was one of biggest payoffs of continuously serving from both DCs is that caches remain warm in future, should allow automatic swtichover improve performance e.g. in south America (closer to Dallas) but bigger payoff: can open more DCs in this way
  • Katherine: interested in progression on lag time
  • Ori: mostly social rather than technical. a lot of stuff exclusively Aaron Schulz. this is going to be problematic. needs to be widely understood in org, otherwise this will be difficult to work on. Aaron has been collaborating with Stas, documenting the work on mw.org; ideally we would reallocate
  • Katherine: when we first started; paint time increases may be transition to mobile. in next q, do you feel like you have resources to do what you want/tradeoffs?
  • Ori: work that Timo is doing will make substantial impact. but even if metrics substantially improve by eg. including CSS next q: it will be great to have compensated for that, but will still be not fully satisfactory; doesn't answer the question why regression happened in the first place. don't feel adequately resourced
  • Katherine: what is your purpose for next quarter?
  • Ori: jumping between projects; drop in project management
  • Wes: Ori, thanks for pushing forward performance and making the site more efficient; heard lots of compliments from other teams about your help supporting their teams.
  • Katherine: I really appreciate time we have together. helps me understand arc of the org, appreciate learning since this isn't my core experience, so thank you very much

General

edit
  • Katherine: comment about presentation structure: makes sense to rotate order so VE doesn't always get smushed at the end.