Take-home interviews

Today we're announcing our second experiment, take-home projects. We're going to try a new way of assessing programming ability by having programmers work on a project on their own time instead of coding during an interview. We know there are benefits and drawbacks to this approach, I'll go into more detail into our thinking behind this below.

Anyone who passes our take-home project assessment will get exactly the same service from us as people who do the regular interviews. We'll work hard to find several YC startups they'd be a great fit for, fast track them through the hiring processes, and handle all logistics of flights/accommodations/scheduling.

The Problem

Several weeks ago, we interviewed a recent college grad. He'd done well on our quiz, had great personal projects, and I was excited to talk to him. As soon as the interview started, however, I could tell that something was wrong. I gave him a programming problem, but he could not get started. He'd start to write one thing, mutter that it was a bad place to start, and go back to something else. He switched languages. His breathing accelerated. He started to shake.

Programming interviews are stressful. Fundamentally, the applicant is being judged. They have to understand the question, produce a working solution in limited time, while explaining everything they are doing with no time to stop and gather their thoughts. At its worst it's adversarial.

Some programmers find that this stress pushes them to do their best in interviews. Others find it debilitating. There are programmers with track records of solving hard problems who simply freeze when subjected to the stress of an interview. They babble. They become unable to program.

This does not mean that they are bad programmers[1]. I gave the fellow in our interview a much harder problem to do on his own time. I assumed that he'd never get back to us. The project was a lot of work. Three days later, however, I had a complete solution in my inbox. We got him back on the phone, and he was able to talk in depth about what he had done, about the underlying algorithms, and about the design trade-offs he'd made. The code was clean. He was clearly a skilled programmer.

The Solution

To solve the problem of interview anxiety, we're adding a second track to our interview process at Triplebyte. Applicants, if they choose, will be able go through our process by completing programming projects on their own time. They'll still do interviews with us, but rather than doing interview problems, they will just talk about the project they already completed. Those who do well will be matched with Y Combinator companies, just like programmers who go through our regular interview.

The project-based track will require a larger time commitment (and we expect lots of people to stick with the standard track for this reason). However, doing a larger project is almost certainly a better measure of actual ability to do a job then a traditional interview is.

Here's how our process works:
  1. When a candidate books a 45-minute interview, they can indicate that they want to do a project.
  2. Three days before the interview, we'll send them a list of projects, and they'll pick one and start to work on it. We expect them to spend about 3 hours on the project (or as long as they want to spend to show us that they're a good programmer).
  3. During the interview, we'll talk about what they've programmed, go over design choices and give feedback.
People who pass the 45-min interview will go though the same process in the 2-hour final interview. Rather than pick a new project, however, they'll take the same project further, incorporating feedback from the 1st interview. Those who pass the 2-hour will talk to Harj, get intro-ed to YC companies, and start new jobs!

I'm particularly excited being able to see iterative improvements to the project between the two interviews (an important part of doing an actual job). It's an experiment, and I have no idea how it will turn out, but giving people the option to do larger projects and avoid stressful interviews just seems like a good idea. In a few months, after we've done a meaningful number of these interviews, I'll write about how their results compare to our other interviews.

1. The stress of interviewing seems to be different than the stress of performing a job. None of the people we've spoken to who do poorly in interviews report problems performing under deadlines at work, or when a website is down and there's pressure to get it back up.

33 responses
It's a good idea. When I interviewed at Uber, I had to complete a project at home before the onsite interview and it was a good way to show my skills without any stress. However, how are you going to handle the fact that the projects may pretty soon be available on the Internet, with solutions and details?
Vincent, To your question: However, how are you going to handle the fact that the projects may pretty soon be available on the Internet, with solutions and details? I think that by going over what they have programmed in the 45 minute interview will be a strong indicator if an applicant actually programmed. Same thing with the 2 hour interview. By no means is this a perfect filter for applicants plagiarizing code but it it feels like a strong way to identify it early on in the interview process.
I've done this before and been accused of trying to get free work out of the applicant. I found him complaining on Twitter about it. Fun times.
Motiejus, I think Vincent is talking about project solutions being released on places like github after an applicant creates them, especially if they don't get the job but have a good, workable solution. Having a large pool of projects with certain key variables (using a different algorithm, different processing of data, different way to display results, etc) may be needed to ensure there is no cheating.
But otherwise, a very cool idea. I usually don't test well but being able to complete a project in a much more relaxed environment would allow me to produce good work.
Anthony, At our company we do give candidates take home projects and offer to compensate them for their time since the take home projects are relative to our business operations and theoretically add value to the business. This is regardless of what role we are hiring for. I can sympathize on both fronts (the interviewee and interviewer). Being asked to perform work that a contractor would be in position to do for a company for free feels wrong and exploitative. Danny, Let me start by saying that I do agree with you that a large pool of projects is a good idea. However, it shouldn't matter if sample code is posted publicly on the internet or distributed via other means. The interview questions and candidates ability to explain their design choices should act as a pass/fail for making it through the first interview. So if they cheat and can't back up their code then they will fail the interview. Maybe Triplebyte will post some findings after a few months.
I like getting people to do both. At home projects show if they can follow instructions. Whiteboard problems demonstrate that they actually are able to write any code at all. How would they perform if they had to pair program? The same way likely. If you're hiring jr it's fine but sr developers should have good communication skills as well.
I guess I just don't understand where this comes from. I mean in what other industry does an employer expect a candidate to perform work before getting hired? If you were a plumbing contractor for instance, you would not send a potential candidate out to do a plumbing job to determine if he did good work. If a candidate has proven experience and is knowledgeable (which can be determined during an interview), why should he/she have to jump through hoops to get a job? Why not just hire a person on a trial basis? Lots of employers impose a 30 - 90 day probationary period where if a candidate does not work out, they can be let go. I have been working as a programmer for about 5 years and honestly, if I were asked to do a "project" in order to get a job, I think I would say good day and make my exit.
I get that people are stressed out, but honestly you want people to show you they can do things. Often, people interviewing would be bigger problem, not knowing things and not understanding answers when they get them. So you need good on both sides. I would say give take home project as initial step in interviewing and continue working on it when they come onsite.
Motiejus, It wasn't "real" work, it was a mockup for a site that would never get published. I wouldn't ask them to do real work that would then get published even if they weren't hired. I'm not that desperate for work to get done. ;)
Looking forward to seeing your analysis on which track is more predictive of success - or is this an alternative Y variable for that?
I am talking from the other side of the equation. I asked HRs of few companies, who had contacted me for an interview, whether it is okay for them to give me a home or on-site project (max duration of 1 day) to demonstrate my skills rather than sitting for an interview. I had an opposite reaction. Everyone refused, saying that it was against the law to assign "work" without pay.
99.9995% of interviews are broken. There are only two fundamental questions to discern about candidates: 0. Are they culturally-compatible? 1. Can they do the job well? Having candidates answer formulaic questions or code up toy projects is not enough, candidates should be tackling corners of real business/code issues occurring right now. Otherwise, it's a waste of time in academic debates and bikeshedding on style. To get to know candidates, perhaps invite them over to hang out. If that goes well, offer a contract. Then, hire the good ones. It takes more management time but it's a flawless filtering process based on reality rather than fickle, snap impressions which are often wrong.
verygood
So you're basically trading the avoidance of onsite technical interviews (that suck) for more hours of unpaid labour... Please consider either take-home same smalls tasks as in in-person interviews, or paid larger tasks (regardless of result).
I've been aggressively looking for work over the past few months (while finishing up my degree), and am happy to say that most of the companies were smart enough to recognize the stress of an interview, and assigned programming assignments instead. I've had a mix of time limited tests/assignments and big projects to be finished in 1-2 weeks. The following interview would be a discussion over my design decisions, scalability, pros/cons, and what I might change had the expectations been modified. Honestly this was the best way for them to evaluate my skills since I didn't have the pressure of somebody physically standing over me or asking me to tell them what I'm thinking as I'm working through it. I've only ever had one interview in the past where I had to code on the spot, and I too started panicking and freaking out; some of us just aren't good in an interview setting.
I had the 45 minute interview with Triplebyte on Tuesday, 28th July. I've extensively worked on large projects previously but when I got a relatively simple problem to solve, I wasn't able to come up with a clear approach. Hitting the wrong lanes repeatedly led to coming up only with an incomplete solution in the given time. I was able to solve the same problem instantly when I looked at it again after the interview was over. I think take home projects are a very great way to assess a candidate. Not sure if I can get another chance.
Great ! Make people work for free !
Or not really, Jean Cile. No one is trying to get free work -- I'm not that desperate. Asking someone to do a coding task at home as a test isn't free work. In my case, the design I asked them to slice and code up was a poor unused design that would never see the light of day as a real project. If I asked someone to design a db to sell widgets, or build a crud app to save a mini blog, or some other entry level thing .. I don't see the issue. If they had a strong portfolio and resume in the first place, this wouldn't even be happening. Entry level stuff. This kind of thing for an advanced position? Better be counting on more than a take home test.
A posthaven user upvoted this post.
Can someone elaborate a bit on what these projects are like? Are they more in the vein of "write a library (in whatever language) that implements the obvious operations for dealing with files in the following format," or more like "Implement a mockup web app in language L using the XWYZ stack and making use of PQR and STUV in the backend. You will also be judged on how idiomatic your jQuery is," or what? (I take it you can answer this question without really "giving anything away" about the projects.)
12 visitors upvoted this post.