Abstract Wikipedia/Design/Wikifunctions usability tests 2022

This research report is based on interviews with seven technical users from outside the Wikimedia community. They were not experienced with editing Wikipedia. Participants reviewed a Figma prototype of the Wikifunctions UI.

A shorter overview of some highlighted excerpts is available in this PDF:

Excerpted comments

Highlights

Wikifunctions mental model

• Definition vs description
• Language agnostic functions
• Implementations come later
• Functions have a default implementation
• Search and general navigation
• Expert mode

Language translation

• English is dominant
• Don't use English-specific functions to test translation
• Location of translation buttons

Function naming conventions

• Programming syntax
• Plain language for names
• Name can connote the output type
• Clarifying the intent

Function description

• Feature request

Aliases

• Purpose is unclear

Input type

• Programming language specific
• Primitive types are limiting
• List (function)

Input label

• Variable or argument name
• Labels help to distinguish multiple inputs
• Ridicule for unnecessary labels

Output type

• Some functions don’t have an output
• Multiple outputs

Publishing is premature

• Function is incomplete without implementation
• Form design

Work summary diagram

• Simple to understand
• Well-liked
• Overkill for this audience

Tab bar navigation

• Edit vs Edit Source is unclear
• Article alternatives
• Star option

Subtabs

• Rename about and details
• Split details into implementations and tests

Examples for using this function

• Scaling concerns

Placeholder for using function

• Popular feature
• Prototype needs a better depiction

Z-Identifier

• No recognition of Wikidata
• Assume it’s an ID

List of implementations

• Status flags
• One implementation per language
• Language as a filter

Implementation naming

• Unnecessary
• Focus on distinct attributes

How to assess code quality

• Readability
• Algorithm comments
• Big-O notation
• Featured implementation
• Too many clicks to get the details
• Test cases in the list

Test case table

• Scaling concerns
• Prioritize failing tests
• Tests don’t apply to all implementations

Wikifunctions mental model

Definition vs description

Joao: So you're calling this "function definition." When you say function definition, I'm expecting that you're asking me to write code.

Joao: The only thing I would change is I would make this model explicit. When you say "define" I'm already expecting to write code. You're not really doing that. You're just creating a page for the function. Describe it, add the inputs and outputs. But don't worry about the implementation. You'll be able to add your implementation later.

Joao: Now it's starting to make sense. I would call this "entry definition" that would be enough to clear up the question.

Joao: If this is your expected workflow then I'm fine with having "publish" here because I'm creating the new entry for my new function and this is just an abstract function that checks the string for a number.

Alex: Yeah, "publish" seems like an interesting choice depending on what exactly this is for. If people are building a huge program with lots of functions themselves then maybe something like "add to project." But if each function is more stand alone and is going on somewhere open and wiki-ish then maybe "publish" makes sense.

Language agnostic functions

Mark: Now I understand why your input types are from a dropdown list. You are expecting that you can have a repository of test cases that are input and expected output entries. And you are expecting to be able to do that language agnostically.

Mark: That leads to subtle differences because not all languages think of the same simple type in the same way.

Mark: Number in Python and number in JavaScript are different. They're subtle. They're close but they have different corner cases which might end up passing in some cases and not in others. Maybe that's okay. Maybe it's not.

Mark: Strings are a good example. Did you mean a unicode string, which would cover Python 3 and JavaScript or did you mean string as "encoding, not yet specified" which would be Python 2 or C++ or C. You're going to end up with those corner cases with almost every type.

Mark: So I'm not saying this is a bad idea. I'm just warning you that there is going to be difficulty when you try to abstract away from the language for anything other than the simplest cases.

Implementations come later

Mark: I'm imagining that the schema is that, yes, you are in fact publishing the function and the next step would give you the option of adding an implementation.

Mark: The implication is that you're hoping to classify functions as a whole like: "oh, you want a function to find a number in a string? Here are fifteen people's implementations of it in seven different languages.

Joao: I'm not going to worry about specific algorithms, I'm not going to worry about implementations or any of those things. I'm just creating this page where anyone will be able to add functions that check strings for a number.

Tiffany: Maybe like if someone wants help making that function then it could be implemented later. I guess that would make sense too.

Functions have a default implementation

Alex: I think maybe some way of assigning a default implementation. So for most users the default is fine and then for advanced users they can select or maybe the default is something that's readable but slow. And the other ones are either random other implementations or faster or not easy to read or something like that.

Alex: How does it assign the default? People would just try to game any metric. Maybe there could be voting like on Stack Overflow.

Alex: It's probably not necessarily to give a name to the implementations but maybe just a radio button for which implementation is in use or something would be fine.

Search and general navigation

Mark: So it's interesting that I have both "search" which makes me think "documentation" but I also have "define new function" which makes it sound like "interactive exploration." So, that's kind of a strange pairing.

Koosha: I'm curious about how the search bar works. Would you type the name associated with different kinds of function? Or would you actually type in the code that you might find in a function?

Koosha: I guess it's easier to search for a question than to search for code, especially when you don't know what the code is yet.

Alex: Search makes sense.

Alex: I didn't know if there's some other general navigation page that we'll look at but there may be better ways of organizing functions. There may be important search functions to have like, search by input or output. Maybe some functions sort animals.

Alex: I think Wikipedia has already a bunch of structures that could be useful to have categories. There's all those Wikipedia lists and stuff so maybe those would have their own category.

Expert mode

Alex: Is the idea that I have this type of interface and then also have a more text-based interface? I was thinking maybe this current interface reduces to some low level set of API calls or queries and maybe having some way of exposing that lower level to people might be interesting.

Alex: Maybe it would let you do things that are at a larger scale or on mass downloaded data on your own computer may be more flexible.

Language translation

English is dominant

Joao: I'm from Portugal but the vast majority of the code I interact with is in English. At times you will find code written in other languages especially in intro to programming classes that are taught in countries other than the US or the UK or I remember writing a lot of code in Portuguese that included variable names, comments, function names.

Joao: The language keywords you cannot change but everything that's user defined like variables and function all of that you can use other languages and people will do that.

Joao: I've never seen code in Cyrillic or Arabic and I don't know if it would run if it's not in the Latin alphabet. With compilers and interpreters.

Roshan: And I think the language is not needed because like most of the language, as per my knowledge, all of the programming works in English. So I didn't get this one.

Roshan: I haven't seen anyone programming in Russian. It can be possible, but more than ninety percent use English as programming language.

Mark: Programmers almost never encounter a language besides English. I mean, I'm only gonna consider the last ten years of professional coding experience. Prior to that, of course, obviously no. You never got ANYTHING except English. It's extraordinarily rare for you to find a library that has a definition or documentation in anything other than English.You very rarely find actual program names in languages other than English.

Mark: The short answer is that it's de facto. I feel chauvinistic for saying that because it's my native language but sadly it's true.

Koosha: I've heard that usually people who speak other languages are just expected to learn English to code but I've also heard that there's some push back to that. People are trying to make a programming language where that isn't the default. But English is overwhelmingly the most popular language.

Tiffany: The movies that are based in non-English speaking countries and when they do code they use English terms. Maybe when they tried to do a looping thing, they still use keywords like "for" in the loop. So I guess, yeah, the implementation and the function name would probably be in English but maybe with their input label would be in their own language.

Tiffany: I think variable names still have to accept English letters. (Latin alphabet) Just like the label of what to tell the person putting in or the input type.

Kenneth: I've seen code with Chinese variable names but they still have to use "for" loops and all that, they still have to use the English version of that for compatibility. It's a weird crossover I guess. I also can't read Chinese so I do most of my programming in English. I can see how a language toggle would help. I mean this is in plain English, this summary is in plain English. I mean if you're a programmer from a different language this would be very helpful.

Don't use English-specific functions to test translation

Kenneth: Translating the function definition makes less sense to me because it's an English function. Why would you translate it? Do I really need to do this in French?

Alex: Translation? Like for the function name?

Alex: Well this one is a bit funny. Because it's very English-centric but I think in general it makes sense. (translating the English singular to plural function)

Location of translation buttons

Tiffany: Maybe if the whole website other than this placeholder were translated itself there wouldn't be a need for these two buttons.

Joao: The only thing I'm finding weird is the button for showing in other languages. One weird thing is the location of the button, I mean, I'm used to Wikipedia where the other languages are here (in the sidebar). Pick the language I'm interested in.

Joao: Why are there two different buttons?

Function naming conventions

Programming syntax

Alex: Is the audience non programmers?

Joao: Is this name like a normal English sentence or is it the name of the function as I was writing code?

Mark: The first thing I'm going to begin to think about is... it's not clear to me whether these names and aliases are supposed to be within the programming language that I intend to write the code or whether these are just English narratives.

Mark: Am I gonna call this function stringContainsNumber all smushed together with camel case, or am I gonna write something like "test if a string contains a number."

Kenneth: When you look for “name” here… is that like a name in plain English? Or is it a name like as you would reference in a function? Because for example Java uses camel case, and then there's Python underscores.

Kenneth: Public functions have an upper case in the first letter while it's convention from many other languages to use lower case for the first character.

Kenneth: There definitely are guidelines but the style changes across languages and people are very opinionated.

Kenneth: Function names typically can't have a space and then I can't think of any language I use that allows that.

Alex: I guess Python would be short. So maybe the Python name would be int_InString or something like that.

Plain language for names

Mark: But if you write them in human language, and I call it "test if a string contains a number" and someone else writes "see if there are numbers in a string" How do you de-duplicate those two things. How do you merge them?

Mark: Even if it matches the language I intend to code it in, there's still the problems of de-duplication and finding other people's duplicate functions but at least the name fits a known pattern of that programming language.

Kenneth: It makes a lot more sense to define it in plain English. Some languages won't let you write a function unless you give it a name first.

Koosha: I guess this is a common name. So "find number in string" or something.

Name can connote the output type

Tiffany: Usually when we're trying to return a boolean we write it as a question, like "is number in string" or something like that.

Alex: I think "Is number in string" is actually better because it's a question and implies the data type of the output.

Alex: Some of these are very good function names for this because they say when it does. And you get a sense of the input and output. For "English plural" I think I would actually expect to output a boolean whether or not it's plural rather than a string with the plural form.

Clarifying the intent

Alex: I don't think that's a problem unless the function name is unclear in which case I think there's a little bit of clarity lost. Imprecise names are fine as a search term but less good as a function name.

Joao: Instead of "enter a descriptive function name" a better placeholder text would be to use an actual example.

Function description

Feature request

Alex: Maybe there should be an optional description.

Alex: Because I think maybe if people were doing something really complicated then they could write all the description and the name first if they were doing a complicated implementation.

Joao: It could be helpful to have here in the function definition a box called function description, where the person would write one or two sentences just explaining what the function is supposed to do.

Mark: Descriptions are difficult. You can see this in a lot of documentation for a lot of libraries. If you expect people to "please write a complete description of the function here." Then for simple functions people will just end up being wordy. It's essentially repeating the name. Why bother writing that?

Mark: On the other hand, if the thing is more complex like finding the largest bucket in a histogram then you really do expect a summary. Given a histogram, find the bucket that contains the largest count. If there are multiple buckets that have the same counts, choose the first one. Then you expect there to be more details. Maybe call it "additional details, if needed."

Mark: But if you always demand an expanded declaration then you just end up with verbiage. You can see that with a lot of libraries. Some always require an expanded statement text block and you just get the name again.

Aliases

Purpose is unclear

Alex: I don't know, what's the point of the alias? I haven't heard of it for functions. I guess I'd never needed to call a function by an alias. From a programming perspective, I just call functions by what I name them.

Alex: So if a function can be called by the alias, I think it's a different matter then if it can be searched by the alias. So maybe that's something to add to the earlier page you could add like a note like "aliases are just for searching" or something.

Joao: I have no idea what that might be.

Joao: I would have to click to see a lot of alias examples, to see what you're wanting me to write in here.

Mark: I'm not sure what the intention there is. If the name is already descriptive. It's truly human language descriptive then I'm not sure what an alias is.

Mark: Maybe it's something like "short name." But if the intention was to actually have a program language acceptable name, something you'd actually put in a library, then “alias” could easily be the two, three, four or five alternate ways of naming this thing.

Mark: A descriptive function name sounds like it's in English. In which case I'm not sure what an alias would be.

Koosha: Yeah, I've never heard of "alias" before.

Koosha: I guess "alias" is just an alternate name for the function. Alternatively, it could be what the function is actually called in your code.

Koosha: It would make more sense to have a name and then just click the button to add another name if you want. (the same way you add multiple inputs)

Tiffany: Maybe this is the descriptive name and the alias is like the everyday name or something. I haven't heard the term "alias" used.

Tiffany: Descriptive means it would have like a lot of words describing it but this one will just be a shorter version of the name.

Kenneth: My interpretation is that alias is just another way to name the function so like I'd default to the number in string.

Observation: The list of aliases on the function page does not actually include the word "alias" in the heading to describe that section.

Input type

Mark: You have "type" as a drop-down list. But type is a constructed expression. I mean, sure, it could be an integer or a string or number, more generically, but it could also be an array of numbers which is different than an array of strings so just having like an array as the type is not very specific. Or maybe this thing takes a histogram. Type is a complicated construct.

Mark: But if I saw this as a senior programmer I'd be like, "this is the toy," because I can't express any type other than the simplest of things.

Mark: That's the problem, it's very program language specific as to what constitutes a "type." A type in C++ and a type in Haskell, notably have different expressiveness.

Mark: In Python, an array is good enough because Python doesn't distinguish between an array of int and an array of string. But if this were in C++ absolutely. An array of int is completely different than an array of string. Completely incompatible. It's a complicated question.

Mark: Sadly there's not an easy answer. But it begs the question of whether you can really express anything of detailed interest with this approach.

Joao: Is this just a few examples? Does it present a filtered list of input type options based on the English name? Or are there more types?

Joao: For "number in string" I would expect to see any type that encodes a number because I want to check a string for a given number so I would expect to see some numeric types.

Joao: It depends on the programming language. In C it's a set list of a dozen different types. I don't know exactly. In Python and other object oriented programming languages you can define your own type. Even in C you can define your own types.

Joao: This is a design decision you have to make. You can make a survey of what are the most common abstract types. I'm talking about numbers, strings, a collection — that includes lists and sets and tuples. Hash tables. It would definitely be more than three types.

Joao: The other alternative is enabling people to define their own types if they need to, that might be a little more complex in language dependence.

Kenneth: I hope this drop down will open up to like int, string, lists etc. I can see that becoming constrained if your scope goes beyond primitive types. So like, if I had a function, I was working with say graphs which are common or just objects which are even more common.

Kenneth: Graphs are the most basic example of something that's slightly above the data type that this would support.

Koosha: Would there be a scroll bar to scroll down and see more because I don't see integer or number anywhere in the list. You could have floats for non-integers or even different types of ints like signed or unsigned. Boolean is another example. You could have true or false or just define it as a number.

Koosha: The type of input is dependent on the language. But stuff like boolean and array and string are common so it's fine to have them.

Roshan: There should be multiple input types. Because in many functions like, there will be not only a single input. There'll be multiple inputs of different types.

Tiffany: Depending on languages they could have different objects, maybe like a dictionary. Maybe they have something like tree objects in there or a try which is like a tree but before words it has letters or branches off.

Alex: I think input "data type" would be more clear.

List (function)

Koosha: Not sure why it says "function" in parentheses after “list.” We haven't specified the programming language yet so it would make sense to have an array in here because they're more common than list.

Kenneth: What does the “list (function)” mean?

Kenneth: I can see how a list would be a good input, and see how a function would be a good input. I just don't understand how a list, with the qualifier "function" here makes sense to me.

Kenneth: Lists and arrays are pretty interchangeable. But why "function"?

Input label

Variable or argument name

Kenneth: Alright, "give me any string" for type. Not too sure what to make of this. I thought about this as a variable name. This made very little sense. In terms of label, like, if it was more clear what it wanted…

Kenneth: Actually if it asked for both I think it'd be great to ask for a variable name and then a descriptor of what the variable did. So, we name a variable, assign a type, and what is it that variable meant to do?

Alex: This would be helped by having an example beforehand. It's all a little abstract.

Alex: I think a variable name would make more sense to me.

Alex: I think it makes sense to fit in at least somewhat with the structure of general programming languages in this way because when people say "string," they're expecting that sort of "programming language speak" now. That would probably be a variable.

Alex: I think you probably need to pick a how "programming languagey" it is and then make everything the same amount of programming language. I think it would be improved by being more consistent.

Joao: For input label, if you want an alternative name I would use the word “argument” instead of “variable.” You pass variables to a function but you pass them as arguments. For the output side I would use “variable.”

Joao: As a coder only having English descriptions and not having variable names that really doesn't work because I'm used to thinking of variable name as "this variable does this."

Joao: “Label” I find is problematic because people are used to labels being very short. So maybe “description” is better. Input name and description.

Labels help to distinguish multiple inputs

Joao: You especially need variable or argument names when you have multiple inputs or outputs.

Roshan: What is this input label? I don't know what that is.

Roshan: It makes sense when there are multiple inputs. Some of the functions may take a string and an integer. So, I think it should be over here an input label. Is it the name of the parameter?

Ridicule for unnecessary labels

Mark: This is a fabulous example. Notice that when you require an explanation, but no explanation makes any sense we end up getting pointless verbiage.

Mark: If the input is a string and the function only takes a string and the string has no external other meaning like, if it had to be a place name, or something. Then "give me a string" is kind of pointless text.

Mark: Just be careful about asking for text when none is needed.

Mark: (joking) how come you didn't ask me for a string to describe the output? Huh? Huh? Because it's ridiculous.

Output type

Alex: Even from a programming language perspective, some of this is a little bit interesting because I was like, oh, output type, like, I mainly program in Python and so you know, that's the sort of thing that I wouldn't normally define explicitly.

Koosha: Defining a variable name is a good practice. I'm surprised there's not a label for the output like there is for the input.

Joao: Treat input type and output type the same. So, also have an output label. And allow for adding other outputs. This depends on the programming language. In C, for example, you can only return one object, but in Python or Matlab you can return several of them.

Joao: I've never worked with Java but in Python for sure you can return both a string and a number at the same time.

Koosha: It's interesting that you have the option for multiple inputs but only one output. So I'm assuming that if the function was supposed to output two numbers it would just use an array or a list for the output type.

Koosha: Some functions don't have an output, they just change something. Like a global variable. The description should explain what other effects the function has.

Roshan: Output type is okay. There'll be only a one time value most probably.

Roshan: And also for output type there should be more things, it might not be a boolean, string or anything like that. There should be more types.

Observation: Functions can return multiple outputs (that is, an array of outputs) or might return no output at all if the function calls another function or accesses a global variable. Having an output is not guaranteed if a function is working in concert with other functions.

Publishing is premature

Joao: So what I'm finding strange is that I'm publishing something without defining any implementation. I suppose that is coming down the line but if that is so then I would use the word "next" instead of "publish" because I feel that the word publish means, "oh, you're about to commit something that should be publicly available for everyone to see."

Joao: At this point I feel: "but wait, I'm still not done." I'm just giving the very basics. I still haven't done the bulk of my contribution.

Mark: Publish is a surprising button there. I feel like "wow, I've barely entered anything about defining this function and you already want me to publish it to other people. So, I'm assuming there's a next step after I hit that publish button.

Mark: But I would be surprised that I can publish it without the initial implementation because the reality is that it would be really, really hard to know what this function totally is just from these entries. Simple function? sure. Someone who did a really good job on their first pass of editing the details and the name? sure. But for many things you're going to want to eyeball the code and go, "oh, that's what it's doing."

Koosha: I'm surprised it says "publish." I would assume that after you finish entering these details you would be typing the implementation. So I think you'd want to do that before publishing the function.

Tiffany: After you fill out this page you just click the button. If we haven't implemented the function yet then maybe not use the word "publish." Is output the last thing because I think of “publish” as the final step.

Tiffany: So when do you do the implementation?

Tiffany: I was just thinking: if the implementation isn't happening now, this button should be "next" to make more sense.

Form design

Mark: First off, graphic design wise, these are inside the light gray rectangle area of "edit summary." So it's completely, completely unclear whether or not those apply to the page as a whole or just to the summary.

Mark: If I've filled out a form and I haven't submitted it then I'm not sure what "cancel" would do. Can't I just hit the back button or close the window?

Work summary diagram

Tiffany: First, I like how there's this work summary section on the right side. I think that that's a good visualization.

Kenneth: I mean, it looks like a function flow like oh, I have my inputs the code will do something and output something. I don't think it's completely necessary. If you want programmers to be filling this out and my interpretation is correct.

Kenneth: I feel like a work summary here doesn't make sense. That makes more sense if it diagrammed the implementation. So this is probably redundant.

Alex: Sort of programming parameters but in text boxes and then you sort of get a visual output. I like how you get an output. Is that sort of Scratch-inspired or something like that?

Joao: So my guess is that as you enter the details in this form, it will start populating like input, input arguments and output variables.

Joao: I think that would be a useful way of helping you understand what you're doing. Shows you in a structured way more or less as you would write the function in a program.

Joao: So, I type a string and it appears in here, this is nice. It's a double-check that I'm doing what I'm expecting.

Joao: Instead of just "implementation" it should add something like TBD or to be defined later. To clear up that you're not doing it now.

Mark: It's a little unclear whether by work summary, it means "this is what you need to do to get this done" versus the other possibility is you're trying to come up with some auto generated example or some sort of summarized pictographic depiction of the function.

Koosha: Is the idea that this is just pseudo code and then you turn it into whatever language you program in? The implementation looks like it might be pseudo code in the diagram.

Koosha: When you type the name it changes up in the title and over on the diagram.

Roshan: That's a simple function overview. Like it'll be an input type. Then there'll be an implementation; the body of the function which will be the steps we're going through from the input to the output.

Tab bar navigation

Edit vs Edit Source is unclear

Koosha: You've got both "edit source" and "edit." I imagine that "edit source" edits the source code obviously. Then "edit" would edit the page we were just looking at with the function definition. [Note: this is wrong].

Mark: I'm not sure what "edit source" vs "edit" is but I guess one of them is using whatever markup language this is. Maybe the other is an HTML editor? I'm surprised that they're separate tabs.

Tiffany: What does it mean for "edit source" and "edit"? What is the difference?

Kenneth: Edit Source. Yeah, if I want to change any of this, I assume that's where I go, wait, there's an "edit source" and an "edit." Okay. I don't know what "edit source" is going to do. I assume “edit” will let me change this stuff (function definition).

Kenneth: (editing the function) Oh, "edit source" does this. I wonder what "edit" does then?

Alex: Well I would sort of expect in this context that "edit source" would be like editing something lower level and the function and then the "edit" would be like editing the high level description that we saw earlier. But I think in the context that it's like editing the web page I assume, I think it doesn't make much sense.

Observation: The inline editing button is labeled "edit" but the tab bar option for this same functionality is "edit source." It's unclear how the "edit" option in the tab bar works (presumably it edits the wiki source of the page).

Article alternatives

Alex: You could just write "function" there. I mean, it's clearly not an article, right?

Joao: Maybe you could replace “article” with "function" and remove the word function from the title. Another alternative is to use "entry," because it doesn't really feel like an article.

Joao: Function also sounds a little bit too specific so maybe entry.

Mark: If this is a new entry then the name of the function should be in human language.

Observation: Consider using "entry" for the article tab.

Mark: If your tone is to create an encyclopedia of functions with implementations that are actually live, active and editable then "article" is not bad. I can see why you don't want to reuse the term "function" because that's gonna get heavily overloaded and crazy.

Tiffany: I think if you're trying to model the Wikipedia page, I think that using "article" is fine for this. I also think it's fine if I'm just trying to go for the idea.

Koosha: Using "article" for the tab would imply that it kinda talks about the function. Like the history of a sorting algorithm. Or why it's a more efficient function.

Observation: If this is an article, maybe there should be a place to link to Wikipedia pages that are relevant. Like a bibliography or reference for a function.

Star option

Tiffany: I'm not sure what the "star" does. I know on social media as far as “favorite.” I'm not sure what it would be for here.

Koosha: Actually I don't know what the "star" does. I'm assuming it’s like a bookmark. I've never used that on normal Wikipedia.

Subtabs

Rename both tabs and disaggregate

Mark: These tabs "about" and "details" seem like totally strange words for the two sides of this. If you're going to lump implementations and test cases together rather than having separate pages, then maybe the real word for this is "code." But then "about" is odd because it's not really about the function. It's a playground to test things and run it. So that seems odd.

Mark: Code would be the first thing I would expect to see. This is about code, show me the code! Not a place to play with it.

Mark: You could call it "playground," or "try it out" or "live demo," "live code," or "live." I don't know what the right word is but "about" doesn't seem... Or "sandbox." People sort of get what "sandbox" is and it has the implication that you can do stuff and it's not going to become part of the wiki. I'm just going to play and it's going to go away.

Tiffany: Actually maybe it'd be better to split the details into maybe two different pages and just link to the test cases.

Examples for using this function

Mark: Be careful. This is good but the presentation is going to get hard for complex functions. What if the input to the function has three arguments and one of them is an array of values? This is going to be impossible to read. Which doesn't mean that it isn't important to see. It's just hard to see it this way.

Mark: A possible way to handle this is to not show it as tabular results. But to actually show it as code entered and printed response.

Placeholder for using function

Mark: So wait a minute. On this page I haven't actually implemented the function yet, I've really just described it. How can I run the function?

Tiffany: Having an interactive example would be really good for all users.

Tiffany: For this site, is this when you would be able to run your test or what does that mean exactly “placeholder for using the function”?

Observation: Where do you go to run a new test?

Kenneth: I'm really curious what this looks like. Now that you told me it sounds really cool but that didn't come through for me.

Roshan: Okay. So, I can also execute those functions on the website itself.

Observation: I think that people still don't quite get this, even with it obviously labeled as a "placeholder."  They can't distinguish it as commentary on the design. They think it's the actual intended final design. Like "placeholder" is some programming term of art that they just don't recognize. This has to look like a usable widget for testing.

Z-Identifier

Kenneth: I assume it's just how you'll be uniquely identifying each one of these functions. Yeah, looks like an ID to me.

Alex: The Z thing? No, I assume it's some identifier but I don't remember seeing it on Wikipedia before.

Alex: Okay. So it's sort of an alias.

Alex: It might make sense to have that ID auto fill in the little alias box on that other screen if you're going to have that in the creation. If it's actually useful for something then people might as well know about it.

Tiffany: Is that like the string that we're putting in? (the string to evaluate for this function; search for number in string) I didn't get that.

Tiffany: I think that will help to have a label that says "unique id."

Koosha: I was about to ask about that. I've never seen that before. Is that going to be decimal or hexadecimal? Hex would look more program-y. More of an aesthetic thing. It's used in programming because it's easy to convert back and forth from binary to hexadecimal.

List of implementations

Status flags

Tiffany: For these "proposed" ones, what's the difference like "available," "proposed"? I think "verified" means that it's been reviewed.

Tiffany: These have been given to the people reviewing it but it hasn't been approved yet. I'm not sure what they're wanting, but also for the proposed one. If it's not been approved yet, will it still show up on, on the page if it's not been approved?

Tiffany: For failed implementations I think it makes more sense to just disappear, like maybe if someone was just going to see the function. They wouldn't need to know that certain implementations were rejected.

Tiffany: And then for "available," I'm not quite sure where that is in the process. How does “verified” fit into this timeline?

Tiffany: For verified and available, I think maybe keeping it consistent like "implementation is verified." That'd be good to just use the same word.

Kenneth: So "available" means someone's already approved it. “Propose,” it looks like, “there's a test running.” So it's probably waiting for community review.

Kenneth: These flag names aren't that consistent.

Koosha: So "available" means that someone has made a Python implementation already and "proposed" means that someone has written a Java and C++ implementation but it hasn't been verified yet. Or it's not ready to have the available tag.

Roshan: The "available" is like all the test cases pass and the code is working on every test case and it also works on some of the edge cases. The "proposed" one is like, if someone is just to propose this but no one has verified or checked it, is it working. It might have some of that is not passed.

Roshan: Is this a mistake? Why is it still "proposed" if it passes all the tests?

Roshan: And “verified.” I think that means the site itself hired some people and they wrote that code that's verified.

One implementation per language

Tiffany: Maybe it would be best to compare all the available implementations and pick the one with the best run time and display that instead. Because maybe it's like there were too many implementations and to display all of them. It would be like a really long webpage.

Roshan: We can test whether the new implementation is better or not. If the new implementation passes all the test cases and takes less time to execute then it is better. It's faster and more efficient. It would replace the old implementation.

Observation: Assuming that the website automatically curates the best function.

Language as a filter

Joao: Adding multiple implementations per language might make this too cluttered. I would just have one entry per programming language. I would add a layer on top of this implementation. First pick the programming language and then pick the implementation.

Kenneth: What if the languages I had were like 30? What kind of table would that look like? The same thing applies here. Whether I have 30 rows.

Kenneth: Would you have more than one implementation per language? If I wrote another Python function that did the same thing, I don't see why I would name it any differently. Even if the code itself differs, it does the same stuff.

Koosha: It'd be nice to have filters. If you had hundreds of implementations you could filter just to show the Python implementations.

Implementation naming

Kenneth: What's the point of having a "name" column if the implementations all do the same thing? Do I really need a name that says "C++" if the language is already there? Do I need Python and Java here in these names?

Joao: For “name,” since the language is already there, it feels redundant and adds unnecessary clutter to the table.

Joao: I think there are other ways to distinguish among implementations. I honestly don't think that differentiating implementations by name is the best way to go because that will lead to very complicated names and I don't want that showing in my code. Probably what would happen is if I get an implementation here and the function has a name I don't like then the first thing I'll do is change the name to something I like.

Mark: Name is kind of funny. I mean why is the name Python and then a repeat of the name of the function? If there are a bunch of names that are just slight variations on the name of the function that doesn't really tell me anything.

Mark: In fact, that whole column tells me nothing.

Focus on distinct attributes

Kenneth: Maybe the name could be more specific about how it works, like: "this implementation uses regex."

Joao: The name should be the same as the function name. But the user should be able to define what makes their implementation different. It will be good to allow the user to specify a natural language description. There's a ton of keywords and descriptors.

Mark: You need a way of calling out the details. For example: "this is a cheap mechanical transformation only" or "this one uses an external dictionary of exceptional pluralizations." So it's really more of a description if the implementation needs clarification from why you would do it.

Mark: Here's an implementation that's very efficient and here's one that's very simple but uses lots of memory. Maybe it's "distinguishing notes."

How to assess code quality

Readability

Kenneth: Let's see. I usually ask, which one's cleaner? Which one do I understand more? Readability is my number one thing. Can I read this code? Can other people?

Kenneth: Two months down the line when I come back to this is it going to make sense? I assume that's something that will be reviewed by people before it gets verified.

Kenneth: If it looks egregiously slow then I'll avoid it.

Alex: Wikipedia is really quite large. So, I think people might want to do things at scale that might need some performance engineering that is not the most readable whereas I think a lot of users will want to have the most readable function even if it doesn't scale as well.

Algorithm comments

Alex: I would assume that people would have some way of commenting like this implementation is especially good for certain conditions.

Alex: It would be nice to have at least some of those comments viewable where people are picking which implementation they're using.

Alex: Some way to expose that within this table. Maybe like a one liner.

Big-O notation

Roshan: One more thing: you should create another slot for time complexity, space complexity using Big-O notation to describe the complexity of the algorithm and its efficiency.

Featured implementation

Joao: Are "available" and "proposed" the only options? Other options might be "failed" or "featured implementation."

Joao: For "featured" I would say selected by the community. The same way Wikipedia has featured pages it would be nice to feature implementations for several reasons. It can be because they're written in a very good code style. And it can serve as an example of how you write high quality code.

Joao: It could be featured because it's ridiculously fast. Or because it's a famous algorithm. It has historical significance. There could be a ton of reasons there.

Too many clicks to get the details

Joao: The runtime is helpful. But having to open each entry to see the individual duration feels like a bit too much work for too little information. Maybe include the runtime in the test results table for ALL the tests.

Tiffany: I think this makes sense if we're just keeping it here and not looking into another web page. But also could get really messy, like, there are a lot of inner arrows to click on and it could be hard to read.

Roshan: Like instead of showing the duration for every single test case just show the average duration for each implementation. That will be more helpful instead of writing it for every single test case, I don't care.

Test cases in the list

Joao: When I expand a row, the tests are not what I would be expecting to find. I would be expecting to find the actual code for the implementation.

Tiffany: I guess my first thought is why do we have this since it's also down here it's like repeating the work.

Kenneth: That's just translating from the lower table.

Test case table

Kenneth: Passing tests kind of reminds me of the way Github does it. Every time you make a change they run a million tests. It's all automated.

Kenneth: I can see this little table becoming very crowded. What if I had instead of four test cases I had a hundred. This is very readable as is. Makes sense. I'm just thinking about how to scale up.

Kenneth: I'm more worried about scrolling horizontally. I mean, you can keep scrolling vertically for sure. But it would be awkward to scroll side to side. Maybe you could lock this column and only scroll it (as a viewport). Horizontal scroll just isn't as intuitive.

Joao: I will remove this from the table. For three implementations this works fine but when you have ten of implementations and well this is going to be a pretty big table.

Alex: This pluralization function would be very hard to test. I think that there would eventually be a test case with the entire Oxford English dictionary in it.

Alex: Maybe these three tests should not have been approved because they don't really test the function and the actual test should pick five random words.

Prioritize failing tests

Joao: Most of it doesn't interest you. If it passes all of the tests, that column isn't interesting. So, I would just show the failing tests. Not passing tests.

Koosha: The way they do it on w3 schools is that if it's an expected output it's in green and they list the output. So instead of "fail" it would have the actual output highlighted in red.

Koosha: Since the Java implementation fails for fisheries, I'd like to see what the output would be, or if it just says that it's an error. If there's an error or an input, in some cases that's what you want it to do. It's easier to catch bugs if a function throws an error for inputs you don't want. How it failed and why it failed.

Roshan: Instead of this "fail" label there should be more information. Was it giving a runtime error? Or was the output wrong? If error, what kind of error? It would be better.

Tests don’t apply to all implementations

Koosha: Technically, the test cases should be tied to a certain implementation. Not all the implementations.

Tiffany: So if we have like multiple implementations, are you planning to have maybe like it's a checkbox to say which one is active to run the test cases?

Tiffany: I was just thinking maybe it would be time expensive to run all the test cases against all the implementations and it'll slow down the website.

END