Learning patterns/Set test tasks for recruiting programmers

Set test tasks for recruiting programmers

problemRecruiting inexperienced newcomers for open source software development projects is valuable, but it is a great risk if you don't know anything about the person yet.

solutionAfter talking to promising candidates, set a small task to test their skills. Make it clear but not too easy. Encourage asking questions. Observe, learn, and then decide whom to work with.

creator• Markus Krötzsch

discuss

endorse

created on8 June, 2014

What problem does this solve?

You need to build a team for your software project. Funding is available, but you still need great people. You want the best – skilled, dedicated, creative – but the candidates who apply for the job are completely unknown to you. Maybe this is because of your project being targetted at newcomers (outreach programs, student grants, etc.); or maybe you just had to learn that all of the brilliant, senior developers you know are already busy with their own projects. In any case, you need fresh people.

This is a moment of great opportunity and great risk. You might discover a new superstar of software development – and steer her path towards the projects you care about. Or you might contract a team member who requires you to spend twice the time on explaining what to do than it would take you to do the work yourself. Your choice might seal the fate of your project.

What is the solution?

Set a test task. You can collect all kinds of information online or in interviews, but a work sample is much more informative (and much more reliable) than most other sources. If done right, it will tell you not only about the technical skill level of a candidate, but also about their values, interests, and communication behaviour.

Step 1: Have a chat

Before handing out tasks, the first step is to talk in person, live or via video conference. You will need to work together and it is vital that you can understand each other (on the language level as well as on the higher level). Personal interviews also set the tone for the interaction that follows. Much more can be said about such interviews, but this would be another learning pattern. For this learning pattern, the most important part of the interview is to explain the test-task process to the candidate. Explain that you use test tasks to ensure that the selection is as fair as possible. Be considerate of the fact that you are asking an advance-investment from the candidate without making any promise that there will be a future reward. Don't give away any information about the details of the task yet.

Step 2: Set a task

Now it remains to set the actual task. If you have many candidates, you could synchronise this and give all of them the same task at the same time. However, you can also devise specific tasks for specific people, customised for the role that the candidate could play in the team. The following advice applies in either case:

Start from a clear goal that every candidate can relate to. The simpler the better.
Choose a complex task with some underspecified parts. "Complex" does not mean "complicated" but "consisting of different parts that need to play together". The candidate should need to ask around or do some Web search. No single Web page should have a model solution to the problem.
Provide a self-contained task description that does not require specific background knowledge. Give pointers on where to find relevant information. Try to choose a task that is somewhat new to everybody.
Encourage interaction. Tell the candidates that this is not an exam. Encourage "cheating". Tell them they can ask you or anyone else if they are unsure or get stuck.
Acknowledge that the task is hard. Tell candidates that perfect, polished solutions are impossible in the given time, and that their approach is more interesting than the final output of their program.
Set a clear time limit. Tell them how much work you think is needed to solve the task and that they should not invest much more than that. After all it is their own time. In many cases, the work of a day should be enough information for you, no matter what gets done in that time.
Set a clear deadline by which the results should be submitted. A whole week is a good time overall.

Example task (used in the Wikidata Toolkit project):

Task: Douglas Adams was born on March 11 1952. Use Java to find out how many people in Wikidata are born on the 11th of March.

Hints:

The data can be downloaded at http://dumps.wikimedia.org/wikidatawiki/20140106/ (you can use this state of the data; don't worry about more recent changes). Use the dump of current revisions to simplify your task.
The XML structure of the dump file is the same as for all MediaWiki dumps. It is described online and relatively easy to process.
The page content in the dump is in JSON (encoded as an XML literal) and encodes the data that you can see on wikidata.org. There might not be a full documentation of this JSON format online (don't be confused by other JSON formats described elsewhere). To understand the format, do the following:
1. Look at the page https://www.wikidata.org/wiki/Q42 to see what the data encodes.
2. Look at the page https://meta.wikimedia.org/wiki/Wikidata/Notes/Data_model_primer to get an overview of the data structure.
3. Find the JSON for Douglas Adams (Q42) in the dump and figure out how it encodes the content you see online at (1).

You do not need to understand all of the encoding to solve the task.

Dates in Wikidata come with a calendar model. You can ignore this and just check the day of birth as it is given in the dump (whatever model is used).
Most programming languages and some text viewers (like "less" in Linux) can read a compressed file without decompressing it first. This saves disk space.

Please submit your answer, the source code you wrote, and some instructions on how to use it.

Step 3: Observe and learn

The result of the test task is not just the final implementation/solution. Moreover, there is usually no single right or best solution. Observe the candidate to learn something about how her or him that can help you in your decision, but also give you valuable information for working together later on.

Questions to watch out for include: "Did the candidate …

… seek support from others?" (post questions on mailing lists or Q/A sites, etc.)
… choose a specific technology to work with?" (this might indicate technical preferences)
… focus on solving the task fast and efficiently, or spend extra time for improving quality and details?"
… acquire skills in an additional topic area to realise a particular solution?"
… communicate the outcomes in a specific way?" (e.g., detailed or concise)
… show specific interest in certain aspects?" (such as runtime performance or software architecture)

There is no "right" way in doing these things, so it is easy to miss this information when viewing tasks in rigidly as "exams" that one can either solve or fail to solve. Indeed, you may find that a candidate who did not solve the task is a good fit for your project. The very point of this learning pattern is that the tasks should not be a one-dimensional test with just two possible outcomes.

Step 4: Give feedback

Whatever the outcome of your decision, take some time to give feedback to the candidate. You asked them to invest a lot of time upfront; constructive feedback is what you can offer in return (even if you don't take a candidate). Tell them what you liked about their answer, and what they should improve in the future. Be constructive. For candidates who did not make it into your project, make suggestions on how one can still be active in open source to improve one's skills for the future.

General considerations

The above guidelines were written for situations where you need people who can work independently in an open environment without close supervision. Hence the recommendation to choose an open-ended, somewhat underspecified task. You can adjust this to meet your actual requirements. However:

Don't make the task too easy. If nobody solves the task, you can still learn something from the efforts; if everybody solves the task easily, there is not much to act on.
Don't specify every step of the task. Small projects need independent thinkers. Don't make tasks that test if somebody can follow a step-by-step manual.
Don't try to make the task useful for your project. Some test tasks may prepare for the actual work you plan, but don't restrict yourself to tasks that are needed right now. Often they are not in the form that makes most sense for a test.
Don't think that impressive interviews or resumes can replace test tasks. Candidates whom you may want to hire on the spot after the interview still might fail completely in the test. Talking ≠ programming.
Don't set trick tasks. Tasks should be solvable and realistic. Don't set impossible tasks to see how the candidates react. Candidates are not objects to experiment with.

When to use

When recruiting students for programs like Google Summer of Code or the Outreach Program for Women
When recruiting student assistants at a university ^[1]
When building a team of professional, full-time developers that are not known to you yet

However, there are also cases where you probably don't want to set test tasks:

If a candidate has references that are so compelling that you don't need any further information; however, if you want a completely fair process, you could still set the same tasks for all.
If you are looking for skills that do not lend themselves to testing. If you need a community manager or a administrative project manager, then it will be hard to find a relevant test task.

Endorsements

Markus Krötzsch
لرتنت

بين المسلمين 197.130.160.131 10:17, 21 January 2017 (UTC)[reply]

SFauconnier (talk)