IRC office hours/Office hours 2013-04-30

56 PM — Coren waves.

56 PM <Lydia_WMDE> hey Coren

58 PM  → Silke_WMDE_, petan and sumanah joined  

00 PM <Lydia_WMDE> alright peoples :)

00 PM <Lydia_WMDE> who is here for the office hour?

00 PM <Silke_WMDE_> o/

00 PM <Silke_WMDE_> :)

00 PM <sumanah> Hi

00 PM <Silke_WMDE_> hi sumanah!

00 PM → Nettrom joined (6641@wikipedia/Nettrom)

01 PM <sumanah> Hi Silke_WMDE_!

01 PM  → mindspillage joined  

01 PM <Matthew_> Lydia_WMDE: Meeeeeee!

01 PM <Lydia_WMDE> wohoooo!

01 PM → auduwage joined (uduwage@ceylon.cs.umn.edu)

01 PM <Lydia_WMDE> someone with questions i hope :P

01 PM <Matthew_> Yeah, a couple :)

01 PM <Coren> Not me!  I'm just here to lurk...  oh.  :-)

02 PM <Lydia_WMDE> awesome

02 PM — Lydia_WMDE pokes Coren :P

02 PM <Lydia_WMDE> Matthew_: shoot then

02 PM <halfak> Lydia_WMDE: Me too.  I was helping auduwage get set up.

03 PM <Lydia_WMDE> halfak: cool

03 PM <JohannesK_WMDE> hi all!

03 PM <Lydia_WMDE> so here's how we usually do this:

03 PM <auduwage> I see issues with toolserver lately, I have send a job and its at qw stage for last 12 hours.

03 PM <Lydia_WMDE> i collect questions and then Silke_WMDE_ and Coren will answer them in the order i give them to them

03 PM <Matthew_> ... Huh, OK.  That took me by surprise...

04 PM <halfak> Lydia_WMDE: Do we PM you?

04 PM <Lydia_WMDE> halfak: that works but here is fine too

04 PM <Coren> Hello auduwage, strictly speaking, we can't really give support for the toolserver - but Petan here is an admin here and might be able to help later.  :-)

04 PM sankarshan → sankarshan_away

05 PM <Lydia_WMDE> (silke is typing)

05 PM <Matthew_> So, my tools are a little interesting.  I'm building a system that requires both a bot and web-based tools to have access to the same database.  I've been told there's going to be database sharing between the web-based tools project and the labs project.  When can we expect that?  or is there some way I have to specifically request that?  Sorry if it's a stupid question, but it's kinda important for my tools.

05 PM <Silke_WMDE_> auduwage: Yeah, we are rather here to talk about issues and questions concerning the migration from toolserver to Tool Labs. We are not toolserver admins

05 PM <auduwage> Thanks Coren

05 PM <halfak> Shall we start asking questions now or is there more into?

06 PM <Lydia_WMDE> halfak: just give questions to me, yes

06 PM <Coren> Matthew_: In the tools labs, there isn't an actual distinction between bots and web-tools; every tool is allowed to implement both and they have access to a DB.  So yes, what you are discussing (bot and web tool speaking to the same DB) is actually the default.

06 PM <halfak> My only blocker for moving from toolserver to labs is mysql access.   What's the ETA on that feature?

07 PM <halfak> Lydia_WMDE: ^^

07 PM <Lydia_WMDE> halfak: noted

07 PM <Coren> Matthew_: That said, there is also nothing that prevent different tools from giving each other accesses to their databases; they have control with grant option.

07 PM <Platonides> auduwage, 5 queues were in error state, I just reseted them

07 PM → K4-713 joined (~khorn@wikimedia/katiehorn)

07 PM <Matthew_> Coren: OK, but I thought they were in separate projects?

07 PM <Matthew_> Whoa, that'll be helpful!

08 PM <Lydia_WMDE> alright

08 PM <Lydia_WMDE> next one:

08 PM <Lydia_WMDE> [18:07:02] <halfak> My only blocker for moving from toolserver to labs is mysql access.   What's the ETA on that feature?

08 PM <Matthew_> ^

08 PM <Coren> That depends on what you mean by 'mysql access'.  If you you mean databases for your own tools (i.e.: not the replicas from the WMF projects) then it's already there.

08 PM ⇐ Alchimista quit (~alchimist@wikipedia/Alchimista) Quit: Konversation terminated!

09 PM <halfak> I need a replica of some tables from enwiki.

09 PM → multichill joined (~multichil@zembla.brewery.verymad.net)

09 PM <Coren> halfak: Ah, that will come in the next month or so; we expect before the Amsterdam Hackaton.  We're at the "select what columns/tables we can replicate" stage, right now, with is nearly the last step.

10 PM → qgil joined (~qgil@c-69-181-248-201.hsd1.ca.comcast.net)

10 PM <multichill> Is the log somewhere online? I didn't notice this until just now

10 PM <Lydia_WMDE> multichill: i'll post it later

10 PM → Ryan_Lane joined (~Ryan_Lane@wikimedia/Ryan-lane)

10 PM <sumanah> multichill: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/20130430.txt

11 PM <halfak> Thanks Coren

11 PM <Lydia_WMDE> next one:

11 PM <auduwage> Thanks Platonides, I guess its better kill the job and re-run again.

11 PM <Lydia_WMDE> [18:09:40] <Nettrom> Some bots/tools, e.g. SuggestBot which I run, is in a production state, running as a service to the community 24/7 (on four Wikipedias).  It’s also being actively developed, so it requires a development environment.  How does Tool Labs support this kind of setup?

11 PM <sumanah> halfak: have you already seen https://wikitech.wikimedia.org/wiki/ToolLabsDatabasePlan  ?

12 PM <Coren> Nettrom: We have, in fact, two project:  "tools", which is the stable environment where 'live' tools are expected to run, and "bots" which is meant to be a more experimental environment for development and prototypes.

12 PM <multichill> Ah, summertime, right :-)

12 PM <halfak> sumanah: I hadn't.  Thanks!

12 PM <sumanah> Coren: hey, you ought to update https://wikitech.wikimedia.org/wiki/ToolLabsDatabasePlan because right now it says the Redactatron is not done yet.

12 PM <Coren> Nettrom: Both projects offer the same facilities, but "tools" is a bit more restrictive in its setup to ensure stability.

12 PM <Coren> sumanah: I though binasher updated it.  :-)  I will.

12 PM <Silke_WMDE_> BTW here is the draft for a roadmap for migration, in case you don't know it already: http://www.mediawiki.org/wiki/Tool_Labs/Roadmap_en

13 PM <Nettrom> Coren: so I'd be a member of both projects a migrate over to "tools" once development becomes stable?

13 PM <Nettrom> *and migrate

13 PM <multichill> Silke_WMDE_: Based on DAB's email. What are you going to do to improve the Toolserver cluster?

13 PM <Ryan_Lane> Nettrom: not necessarily migrate

13 PM <scfc_de> Nettrom: You can develop very well in the Tools project alone.

14 PM <Coren> Nettrom: You can have both running in parallel, really.

14 PM <Nettrom> oh, ok

14 PM <Ryan_Lane> Nettrom: one would be your dev environment, the other would be production

14 PM <Silke_WMDE_> multichill give me a moment to type...

14 PM → giftpfla1ze joined (~GVMBot@p4FDCB3ED.dip0.t-ipconnect.de)

14 PM addenergyless → addshore

15 PM <Nettrom> sounds like I've been a bit confused about how projects and members and the whole thingamajig works... will go read some documentation, thanks!

16 PM  → PPena_ joined  ← Ironholds left  

16 PM <Ryan_Lane> Nettrom: yw. a brief overview: a project has members and admins. it holds resources like virtual machines, storage, public IP addresses, firewall rules. etc. it's mostly a security separation that allows a community to manage their own infrastructure.

17 PM ⇐ jorm quit (~bharris@wikimedia/jorm) Quit: jorm

17 PM <Ryan_Lane> so, you join two "communities". one of those is meant for development, the other is meant for production

17 PM ⇐ giftpflanze quit (~GVMBot@wikipedia/Giftpflanze) Ping timeout: 248 seconds

17 PM <Ryan_Lane> Labs is comprised of a bunch of these communities, each aimed at a specific real world project

17 PM <scfc_de> Ryan_Lane: You *can* join two communities, you're not required to :-).

18 PM <Ryan_Lane> indeed

18 PM <Ryan_Lane> you can join as many or as few as you'd like

18 PM  ⇐ PPena quit  •  PPena_ → PPena  

18 PM <MF-W> so one tool = one labs project?

18 PM <Silke_WMDE_> multichill One thing is: We dont' have any more rack space nor electricity. DaB asked if it's possible to add disk to virtualize. For me it's hard to find out about the exact situation...

19 PM <Silke_WMDE_> ... of electricity because none of us is on-site.

20 PM <Silke_WMDE_> multichill: It's NOT that we would invest some more money.

20 PM <multichill> Silke_WMDE_: And how are you going this problem?

20 PM Raylton → Raylton_Away

20 PM <sumanah> https://wikitech.wikimedia.org/wiki/Help:Move_your_bot_to_Labs and https://wikitech.wikimedia.org/wiki/Help:Terminology may help people out here

20 PM <addshore> Or even http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Help

21 PM <varnent> James_F: ping

21 PM <scfc_de> MF-W: No, "project" = (for example) "Tools" or "Bots" or ...  Tools (= plural of tool) then are directories, code, data, etc. on Tools.

21 PM <multichill> Silke_WMDE_: If you want to know the current usage I can just drop by and check the racks. If that's keeping you from improving things.....

21 PM <Lydia_WMDE> (can we please discuss one question after another?)

22 PM <Silke_WMDE_> multichill: I hope the members' assembly will give me some direction here. they ought to say if we still want to start contracts with a data centre we never had a contract with a year before leaving etc.

22 PM <Platonides> I expect that the way the racks are used is fully documented...

22 PM <scfc_de> Silke_WMDE_: So currently the contract is with WMF?  Don't they have documentation?

23 PM <multichill> afaik everything is in racktables

23 PM <Silke_WMDE_> scfc_de: The contract is with WMF.

23 PM <multichill> Just throw out some old servers saving power and replace them with newer ones

25 PM <multichill> What is your power capacity per rack? 2 * 16A? How much are using? What is the power usage per server? etc

25 PM <Lydia_WMDE> multichill: assuming the toolserver will go away - what do you need from labs to move there?

26 PM <multichill> Lydia_WMDE: That's a distant future, I would like to see the Toolserver fixed in the meantime

26 PM <multichill> The new product isn't ready yet and the old one is falling apart

26 PM <Lydia_WMDE> but you still have not said what is missing to make it ready

27 PM <Lydia_WMDE> that would really help

27 PM <Lydia_WMDE> then maybe we can give you a timeline

27 PM <multichill> Fully replicated databases with user databases

27 PM <multichill> That's Wikipedia + Commons + Wikidata + user databases on one server for joins

28 PM <scfc_de> multichill: Seconded.

28 PM <addshore> As far as I knew this was planned for May.?

28 PM <Lydia_WMDE> (coren is answering)

29 PM ⇐ Matthew_ quit (~matthew_@wikipedia/matthewrbowker) Read error: Connection reset by peer

29 PM <multichill> addshore: WMF and planning ;-)

29 PM  → eyoung and Matthew_ joined  

29 PM <Ryan_Lane> yes, it was planned for May. eqiad migration took priority

30 PM <Coren> multichill: The long story short; replicating databases is happening soon (Within the month)  Replicating multiple copies of commons and wikidata isn't going to happen that way; it needs to be built into application logic or using federated tables.  Almost /all/ of the problems the TS have had with replication were caused by that redundancy and trying to keep it synced.

30 PM <multichill> Coren: So you're basically saying Toollabs is useless for me

30 PM <Platonides> Template:Ref needed

30 PM <Coren> We're all more than happy to help you (and any other maintainer) with adapting your tools to work in that setup.

30 PM <scfc_de> Coren: In that case, Tools will not be able to replace Toolserver.

31 PM <giftpfla1ze> what are federated tables btw?

31 PM <Ryan_Lane> AFAIK toolserver will also have this limitation at some point

31 PM <scfc_de> Ryan_Lane: What do you mean?

31 PM <multichill> Coren / Ryan_Lane : You want to replace the product. We give specifications of the product. You change the specs and say we're complaining

32 PM <Coren> giftpfla1ze: Basically, a "symlink" between databases.  You can join with it, provided you are a little careful about the kind of joins you do (so that you don't copy the database at every query!)

33 PM <scfc_de> Coren: Have you looked at the queries that are typically run on Toolserver?

33 PM <multichill> Platonides:  +1. Toolserver databases are running on ancient hardware

33 PM <giftpfla1ze> Coren: can you give examples for joins that can/can't be done?

34 PM <Coren> scfc_de: Those I can see publically (through the query service).  Almost all would work with little to no adaptation.

34 PM sankarshan_away → sankarshan

35 PM <JohannesK_WMDE> giftpla1ze: +1. i am still not entirely sure what the differences will exist in db replication toolserver vs labs, could you please go into some more detail Coren?

35 PM <Coren> giftpfla1ze: They can all /be/ done, but some will be horribly slow.  The easy way to know if it'll work well is this:  if you are picking rows in the federated table from a 1:1 match over an indexed column, it'll work very well.  SELECT fed.a, fed.b, local.a, local.b from local, fed where fed.a=local.a; would be 100% supported at speed.

37 PM <halfak> Would all replicated tables be federated?

37 PM <scfc_de> Coren: You can see some more for example at https://github.com/mzmcbride/database-reports.

38 PM <Lydia_WMDE> ok how about Coren has a look at this in detail after this meeting and we move on to the next question?

38 PM <Lydia_WMDE> we have only 20 mins left

39 PM <Lydia_WMDE> the next question i have is:

39 PM <Lydia_WMDE> [18:30:35] <scfc_de> What's the status of the additional Toolserver admin?

40 PM <Silke_WMDE_> scfc_de: This should be clear and finished in the next 10 days or so.

40 PM <Silke_WMDE_> This is going to be a paid admin with 15-20 hours per week

40 PM <Silke_WMDE_> on the toolserver

40 PM <multichill> What is going to be the availability of tool labs? Is it production grade? When is the maintance window? How long in advance are you announcing it? Is it supported 24x7 or less?

41 PM <scfc_de> Silke_WMDE_: "Clear and finished" = "hired"?  (Taking no chances here :-).)

41 PM  → jvandavier and lwelling joined  ↔ Matthew_ nipped out  

42 PM <Silke_WMDE_> scfc_de: Hopefully, yes. The thing is that I am the person here to make the ultimate decision on my own. (This is why it has taken a while.)

42 PM <Silke_WMDE_> ^^not

42 PM awjr_away → awjr

42 PM <sumanah> multichill: ok, that's several questions, it's taking a moment for Coren & Ryan_Lane & me to answer :)

42 PM <Silke_WMDE_> as in I can't hire people on my own.

43 PM <multichill> sumanah: It's basically what SLA to expect ;-)

43 PM <Coren> multichill: It's production grade.  It's been set up so that it's even possible to do rolling restarts of the entire cluster with migration of jobs to keep them up, for instance.  Maintenance windows should be announced at least a week in advance, and would generally not cause interruption of tools over very brief periods while they move from one node to another.

43 PM <sumanah> yup

43 PM <Ryan_Lane> multichill: It's semi-production

43 PM <sumanah> multichill: Tool Labs, as part of Wikimedia Labs, is not production grade, but then again we have really really high standards for what "production grade" means!

43 PM <sumanah> Ryan_Lane: would you feel comfortable talking about predicted uptime numbers now?

45 PM <Matthew_> Please?

45 PM <multichill> Did you guys document what we can expect in terms of availability somewhere?

45 PM <halfak> Wait a second.  Coren says Labs is production grade, Ryan says it is semi-production and Sumanah says it is not production grade.  Huh?

45 PM <sumanah> multichill: also, when you say "supported 24x7" you're asking "is there going to be a person pingable and online 24x7"?

45 PM <Coren> Terminology confusion.  :-)

45 PM <sumanah> halfak: I understand your confusion

45 PM <sumanah> halfak: yeah, it's all about what "production grade" means

45 PM <Silke_WMDE_> This is a question of definitions I think

46 PM <sumanah> compared to the vast majority of services available on the internet - and certainly compared to what the Toolserver is now -- Tool Labs is and is going to be reliable.

46 PM <multichill> It's all about the expectations. If shit hits the fan this should be clear. What I'm reading here is that you don't even know it for yourself

46 PM <sumanah> Compared to en.wikipedia.org - no it's not going to be held to THAT standard of reliability.

46 PM mwalker|away → mwalker

46 PM <sumanah> So I'm with Ryan_Lane, as I suspect Coren is, that "semi-production" is the right term to use. Hope that helps halfak

46 PM <multichill> If something breaks in on Sunday morning, is someone getting out of bed to fix it?

46 PM <Ryan_Lane> multichill: yes

47 PM <Coren> I'm talking about extectation of uptime, Ryan is talking about SLA and I'm not.  :-)  Semi-production is better indeed, and matches WMF terminology.

47 PM <Ryan_Lane> we aren't going to provide support to the level of other wikimedia projects, but yes, we have people that will fix labs when issues occur

48 PM <multichill> *please* document this somewhere. This sounds like what we used to call the "Super Best-Effort SLA": We'll do our best, but we're not promising anything to you

48 PM <Ryan_Lane> when wikipedia breaks half of the operations teams shows up on IRC

48 PM <Ryan_Lane> that's not going to happen for labs

48 PM ⇐ andre__ quit (~andre@wikimedia/aklapper) Quit: andre__

48 PM <Matthew_> multichill: AKA what Toolserver does now :P

48 PM <Ryan_Lane> assume the level of support will be better than TS

49 PM <Silke_WMDE_> I propose to document this:  what level of production grade do Labs/ Tool Labs have and what does that exactly mean. Should be written down somewhere in the docs.

49 PM <Coren> Matthew_: The difference is that *someone* will show up, maybe just not half of opsen from around the world at once.  :-)

49 PM <Lydia_WMDE> (fyi: 10 mins left)

49 PM <Matthew_> Coren Ryan_Lane: OK, makes sense.

49 PM → lizzard joined (~lizzard@173-228-80-237.dsl.static.sonic.net)

50 PM <sumanah> Ryan_Lane: If Tool Labs breaks, someone will be woken up and will work on fixing it and WMF will not pause till it's fixed (barring pathological catastrophe) -- is that fair to say?

50 PM  → Kolossos joined  ↔ Matthew_ nipped out  

51 PM <Ryan_Lane> sumanah: ideally. yes.

51 PM ⇐ Matthew_ quit (~matthew_@wikipedia/matthewrbowker) Remote host closed the connection

52 PM <sumanah> also Ryan_Lane if we can point to the Tiers of Support (which WMF services are wake-everyone-up things and which are high-priority and which are lower-priority) that would be great.

52 PM <Ryan_Lane> historically this has been the case

52 PM <multichill> Do you keep backups of the homedirectories etc?

52 PM <sumanah> I understand if it's too complicated to be broken out this way

53 PM <multichill> When will ipv6 be supported?

54 PM ⇐ HaeB quit (~quassel@wikipedia/HochaufeinemBaum) Ping timeout: 256 seconds

54 PM <Silke_WMDE_> (Ryan and Coren are about to answer)

54 PM <Ryan_Lane> multichill: we'll be bringing up a zone in eqiad, which will be a new openstack deployment. that zone will start with having IPv6. Once that is working correctly, we'll look at bringing it into the pmtpa zone

54 PM <Coren> multichill: We have automatic snapshots of the past 3 hours, 3 days, and 2 weeks on the new fileserver.  There are no off-site backups of maintainer data, however.  Good practice would have you store important code and config in git or other source control.

55 PM <multichill> Block based or file based backups?

55 PM → brion joined (~brion@wikipedia/pdpc.professional.brion)

55 PM <Ryan_Lane> eqiad = ashburn datacenter, pmtpa = tampa datacenter

55 PM <Ryan_Lane> multichill: LVM snapshots

55 PM <Ryan_Lane> so, it's on the same fileserver

55 PM <Coren> multichill: They are timetravel snapshots; implemented at the block and filesystem level.

56 PM <multichill> ok

57 PM <Silke_WMDE_> OK, one more question!

58 PM <Silke_WMDE_> multichill: Come and meet us at the Hackathon!

58 PM <Silke_WMDE_> Who else will be there?

58 PM <addshore> I will :)

58 PM <Lydia_WMDE> \o/

58 PM <sumanah> http://blog.opengeo.org/2013/04/03/why-we-sprint/ why it's nice to meet in person :)

58 PM <multichill> Silke_WMDE_: You that I'm organizing the hackathon?

58 PM <JohannesK_WMDE> i will be there

58 PM <Silke_WMDE_> multichill: I just learned that's you from Lydia

59 PM <Lydia_WMDE> alright - our hour is unfortunately over

59 PM <Lydia_WMDE> if you have more questions please do send them to the mailing list

59 PM <halfak> Thanks for your time folks!

59 PM <Lydia_WMDE> i will publish the log but probably tomorrow unless someone beats me to it

59 PM <Silke_WMDE_> thanks for coming!

59 PM — Nettrom seconds halfak, thanks!

00 PM <halfak> How can I best keep tabs on new developments?

00 PM <halfak> Sorry.  Meta-question.

00 PM <Silke_WMDE_> halfak: Coren sends updates on toolserver-l and labs-l

01 PM <Silke_WMDE_> halfak: I'll gather a list of wiki pages that are regularly updated

01 PM <Silke_WMDE_> I don't have all of them here, now

01 PM  → everton137 joined  ↔ jorm popped in  

02 PM <sumanah> halfak: labs-l is good

02 PM <sumanah> https://lists.wikimedia.org/mailman/listinfo/labs-l

02 PM → jorm joined (~bharris@wikimedia/jorm)

02 PM <sumanah> halfak: https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs is good