12,000 engineers evaluated

We launched Triplebyte with the goal of building the first credentials blind hiring process for engineers. Our mission is to give anyone who has the right skills, the opportunity to work at the best technology companies in the world regardless of what school they went to or which companies they've worked at.

We've now evaluated over 12,000 engineers without using their resumes. We've done this by designing a two step process. The first is an online programming test. If you do well on the test, the next step is a technical interview with our interviewing team where the interviewer knows nothing about your background.  We've now interviewed over 2,000 engineers and 15% made it through to the final step of being introduced to the companies we work with.

To put that into context, companies at the size of Airbnb or Dropbox would expect to do technical interviews with approximately 50 engineers a month. We're already interviewing 3x that number every month and we've built software to track every tiny detail of what happens during these technical interviews. This means we're getting data on how to accurately interview an engineer, faster than anyone. In total we've now done over 3,000 hours of technical interviewing, or 127 full days. 

Once an engineer makes it through our process, we match them with companies they'll be a good technical fit for. As we wrote before companies disagree significantly about the types of engineers they want to hire. We're optimizing our matching process for accuracy so we gather as much data about the technical preferences of the current engineering team and use that to match engineers with them. We get this by having the current engineering team complete a technical questionnaire that gives us a fingerprint of their hiring preferences. This matching model is working really well. At our partner companies like Dropbox and Cruise, we're seeing offer rates on our candidates of over 60%. That's more than 2x better than the average they see on their own candidates (the industry average is about 25-30% onsite to offer rate i.e. about 1 in 4 engineers who make it to onsite will receive an offer) and our candidates are going straight to an onsite interview, skipping recruiter and phone screens.

What's really exciting about such a high offer rate is that we're achieving it without doing any culture fit screening. Our process *only* looks at technical skills and that's the data we use for matching engineers to companies. That shows the way for companies to hire more engineers is to get better at identifying candidates with the right skills early on, not doing more culture fit screening early on.

We can also beat companies on the most important metric of all - the number of internal engineering hours they have to spend per new engineering hires. Sequoia recently estimated that it takes a company 82.5 total hours to hire an engineer. Around 30-35 of these are engineering hours. We're able to deliver an engineering hire at an average “cost” of 15 internal engineering hours. 

We're only able to interview so many engineers each month because our programming test can accurately identify good engineers. After many iterations our test now has 70% precision (precision means of the engineers our test identifies as being good, how many of them actually are). This means we can identify great programmers with much higher accuracy than a resume screen and only interview the good ones. 

What's unique about the programming test is that we've iterated on the questions we ask using actual data from interview outcomes at the companies we work with (i.e. we correlate performance on specific questions with performance on the onsite interviews at companies like Dropbox). We only keep the questions that have high signal and continually replace low signal questions with new ones. This process is the only way you can build a test that is actually accurate at screening for what companies want. 

The data set we're building is unique and we use it to improve the accuracy of our process over time. The data set is unique because:

  1. we track a lot more data from each interview than a typical company would. We've built specialized software to track data points like how long it takes to complete each section of a problem and what the interviewer is thinking every 5 minutes throughout the interview.
  2. We also get data on how our engineers perform on the onsite interviews at the companies we work with. We use this data to build a predictive model specific to each company, based on our engineering genome, which we continually update over time.

It's really exciting for us to see that a credentials blind technical evaluation can work at identifying good engineers and matching them to the right companies.

Triplebyte Engineer Genome Project

We launched Triplebyte last year with the goal of building a hiring process focused on evaluating skills and not credentials. Since then we've evaluated over 10,000 engineers without using their resumes, and helped them join companies ranging from three person startups up to Dropbox. Doing thousands of blind technical interviews has forced us to get really good at identifying programming skills directly, and allowed us to work with engineers from a great diversity of backgrounds.

One of the most surprising things we've learned through this process is just how much companies differ in which programming skills they value most. A deliberate, academic programmer, for example, may do extremely well at one company, which thinks that she will be able to tackle tough problems. Other companies instead want fast, intuitive thinkers, and will reject this same engineer on the grounds that she may not be productive enough. Some companies want all their engineers to understand deeply how a computer works. Candidates have no way to know what specific companies prefer, and this results in a large amount of wasted time.  Rejection is also demoralizing and we've seen many engineers, especially those working outside Silicon Valley, start questioning their own abilities after a few failed technical interviews.   

To make the process of finding the right company better for engineers, we're announcing the Triplebyte Engineer Genome project. Using the data we've gathered through our technical interviews, we've mapped out the engineering attributes that technology companies care most, and measured how the companies we work with weigh these attributes differently. We've used this data to build software to intelligently match engineers with the companies where they're the best technical fit, and we're using this software with engineers who go through our process.

Intelligent matching with software is how hiring should work. Failed technical interviews are a big loss for both sides. They cost companies their most valuable resource, engineering time. Applicants lose time they could have spent interviewing with another company that would have been a better fit. 

Moving skills assessment and company matching into software also has another important consequence. It increases diversity in the hiring pool. If companies can trust that an applicant has the technical skills they're looking for, it gives them confidence to speak with candidates who lack the usual credentials of attending a top school or working at a prestigious company. 

We've built up the list of engineering attributes in the Triplebyte Engineer Genome by collecting a large amount of data from the thousands of blind technical interviews we've completed ourselves. We've then tracked how the data we've collected about each engineer matches with their interview performance at top technology companies we work with. This has been a huge schlep but it's the only way to build a matching system that actually works.

By evaluating this many engineers and working with over a hundred companies, we've seen how little consensus there is on what a "great engineer" means to any single company.  We’ve calculated statistically the extent to which interviewers at different companies agree about which candidates are good and which are bad (for the statistics nerds, we calculated the inter-rater reliability), and found it to be about the same as the extent to which people agree on which movies on Netflix are best.

Recruiting services today avoid tacking this problem altogether.  Mapping what companies actually want is a much harder problem than scaling the traditional recruiting agency model of spamming companies and candidates. There are so many different engineering attributes you could conceivably look for it's hard to narrow it down to a concise list. Even if you could cleanly identify these attributes, assigning the right weight to each one adds another layer of complexity. It's too much to expect a single person or team making hiring decisions to do this well.

Zooming out, it's too much to expect a single company to be capable of solving this problem either. Google arguably does the best job and their data is still limited to (1) engineers who applied to Google (2) finding which attributes are most important for success at Google. Most companies are bad at identifying what a great engineer looks like. Even the famous ones get it badly wrong, like Facebook rejecting WhatsApp founder Jan Koum (they did eventually hire him but the price went up a bit).

Triplebyte is uniquely positioned to fix this. Over the past year, we've collected both quantitative (e.g. time to complete milestones within programming problems) and qualitative (e.g. problem solving approaches or code quality) data from several thousand technical interviews and have use it to create a list of the engineering skills most important to technology companies - the Triplebyte Engineer Genome. These are:
  1. Applied problem solving
  2. Algorithms knowledge
  3. Professional code
  4. Communication skill
  5. Architecture Skill
  6. Low-level systems understanding
  7. Back-end web understanding
By scoring engineers on these criteria and then assigning weights to each companies based on empirical observations of their hiring decisions, we can use software to better identify engineering skill than humans. We're excited about this because it moves us towards removing human biases from the hiring process altogether. Humans making judgement calls about objectively measurable skills introduces bias and hurts diversity. If we want more diversity in tech, this needs to be done with machines crunching objective data.

We expect the list of attributes in our Engineer Genome to continue evolving over time as we gather more data on what companies are looking for. We'd welcome your thoughts or feedback on the project and thanks to everyone who has completed the Triplebyte technical evaluation.

Fixing the Inequity of Startup Equity

tl;dr Short stock option exercise windows suck. They force startup employees to make hard decisions, and often rob them of fairly earned compensation. We’ve created docs that companies can use to give their employees 10 years to exercise their options. YC will recommend all their startups use these documents going forward. We’re advising Triplebyte candidates to favor companies making this change, and we’ve already convinced 12 companies to pledge to do this.

Stock options are valuable compensation for startup employees. The high potential upside of these options motivates employees to turn down larger salaries at bigger companies and work at startups. It seems obvious, then, to expect that employees should own their vested options outright, even if they leave the company. Stock options are compensation for work that’s already been done. Returning them to the company when you leave would be inequitable.

Unfortunately, this is exactly what often happens. The industry standard stock option agreement gives employees 90 days after leaving a company to exercise their vested options, or they are returned to the company. Many employees don’t have the money to exercise their options within such a short window and lose them. To fix this we’re announcing three things:

  1. We’ve worked with the team at Ironclad (YC S15) and Nancy Chen at Orrick to create the Triplebyte extended window stock option plan. This is standardized paperwork any company can use to give their employees 10 years from grant date to exercise their options. You can download these directly here or use Ironclad to set up and manage them here.
  2. We’ve been encouraging startups to make this change and are publishing a list of YC companies who offer an extended exercise window here. Currently 14 have already implemented an extended window option plan, and 9 have pledged to do so using our plan. More companies are in the process of deciding and we’ll be updating the list as they do. We’ll be encouraging Triplebyte candidates to weigh this heavily when choosing companies.
  3. Y Combinator has agreed to recommend that its companies use the Triplebyte extended window option plan documents when they form an option plan, beginning with the current Winter 2016 batch.

If you’ve been an employee at a startup, you’ll know this issue is important and causes stress in two ways.

(1) To exercise the options, you need enough money to cover both the exercise price and the taxes you now owe. In the case where the company has performed well and the valuation has increased (i.e. when people care most about their options), this will be more money than you can afford, unless you’re already wealthy.

(2) How do you know if now is the right time to exercise the options? The company may still be years away from a liquidity event, with some uncertainty remaining over its future outcome. Investors in the company are diversified and can absorb this uncertainty. You can only work for one company at a time, with all your eggs likely in this one basket. The stock is also likely illiquid right now so you can’t sell some to recoup your cost of exercising.

So now you’re left with three choices. Give up your options, stay at the company longer or scramble to find a financial solution quickly so you can afford to exercise the options. As Sam Altman wrote, this is an unfair situation and needs to be fixed.

At Triplebyte, we spend a lot of time talking to engineers who are thinking through startup job offers. They’re becoming increasingly savvy about how stock options work, and ask thoughtful questions about the mechanics of options. We also see an outsider’s perspective on this, as we talk to many engineers from non-traditional backgrounds who are based outside of the Valley. Invariably when we get into discussing the details of options, they are surprised by how quickly they’d be forced to exercise their options if they left a company. We agree with them. It’s not a fair situation to put someone in. People should stay at a company because they want to, not because they feel locked in by fear of loss.

There is a growing trend to fix this inequity by increasing the post-termination option exercise window for employees. This “option extension” gives you more time to exercise your options, which increases the likelihood there will be a liquidity event to help you pay the exercise price and the tax triggered upon exercise. Quora, Palantir, Pinterest, Asana and Coinbase have all increased the post-termination option exercise window for their employees. This is the future and we’re going to accelerate this trend to make it the industry standard to give employees 10 years from getting their options, to decide whether to exercise.

As a company, if you haven’t already implemented a stock option plan, adopting the increased exercise window is simple. You can just use our documents.

If you have an existing option plan in place, amending it requires thought and analysis of the tradeoffs. We’ve created a summary of the gory details on both the business and legal aspects here. Our goal is for founders to use this to have an informed discussion with their counsel and make a decision. What we want to make clear is that it is possible to do this for your existing employees by amending their outstanding options and adding a longer exercise window to them.

Much has been written about this issue in the past but not enough has changed. Most companies continue issuing options with a 90 day window. Employees are often either not sophisticated enough to ask about this issue, or are reluctant to ask a company to incur the expense of paying lawyers to draft new and complicated paperwork.

We’re applying the three forces we believe will make a real change (1) standardized paperwork accessible to all, (2) public recognition for companies who have made the change, (3) educating employees about the issue. We expect increasing the exercise window to become a necessary condition for startups who want to hire the best people, which is ultimately what their success depends on.

Thanks to Sam Altman, Carolynn Levy, Jonathan Levy, Jason Boehming and Nancy Chen for reading drafts of this post.

Extending Stock Option Exercise Window Guide

We wrote here why we believe giving employees 10 years (i.e. the full term of their options) to exercise their stock options is the future of startup employee compensation.

If you haven’t already approved a stock option plan and would like to set this up, you can just use our documents to set up your plan, or Ironclad to both set up and manage them.

If you have an existing option plan in place, amending it requires thought and analysis of the tradeoffs. We encourage every company to think about this deeply and make an informed decision. What we want to make clear is that it is possible to amend outstanding options held by existing employees and add a longer post-termination exercise window to them.

There has been a lot of discussion on this topic, with arguments made both for and against implementing the extension. When talking with founders who agreed with making the change in principle, but were deterred by speaking with lawyers, we found there to be a lot of misinformation about the key issues in circulation.

Here is a summary of the issues, both business and legal, we most commonly hear brought up by founders thinking through this. Our goal here is to provide the information needed to raise this topic for internal discussion with your team and make a decision.

Business Issues

1) This matters only to the “wrong” kind of employee

Issue: The exercise window matters most to the mercenaries who want to move onto the hot new company each year. The company will lose an important retention mechanism that comes with options.

Thoughts: Having people stay at your company because they feel locked in isn’t good for employee morale. Companies want motivated people working for them, and should find positive ways to incentivize employees to remain in employment. Companies can also implement the option extension in a way to encourage retention by requiring employees remain in employment for e.g. 2 years to be eligible for the option extension. This is how Coinbase and Pinterest implemented it.

2) This is worse for employees from a tax perspective

Issue: There are two types of stock options, incentive stock options (ISOs) and nonstatutory stock options (NSOs). Only NSOs can have this extended exercise window but they also have less favorable tax treatment than ISOs.

Thoughts: It’s true that ISOs have better tax treatment than NSOs but the difference is not as great as commonly thought because of AMT (Alternative Minimum Tax). To understand this, let’s compare how the tax treatments differ at the two most important events, exercising your options and selling your stock.

1. Employee exercises stock option

ISO: Employee now owes AMT (Alternative Minimum Tax) on the difference between the amount they paid to exercise their options (the exercise price) and the fair market value of that stock today. Calculating exactly AMT can be tricky, most likely you’ll pay 28% on the difference.

NSO: Employee owes Ordinary Income Tax (38%) on the difference between the exercise price and fair market value of the stock.

2. Employee sells stock

ISO: Employees owes capital gains tax on difference between the sale price and fair market value of the stock at exercise. If this sale happens within a year, you’ll owe short term capital gains (38%). If it’s been over a year and two years since the options were granted, you’ll owe long term capital gains (20%).

NSO: The capital gains treatment is the same as above.

You can see the tax difference between ISOs and NSOs is most important at the time of exercise. What’s important is companies don’t have to decide whether employees get the flexibility of an NSO or the preferential tax treatment of an ISO. Employees can make this decision themselves. If an employee exercises their options, post leaving the company, within 90 days they’ll still get the ISO tax treatment. Otherwise they’ll get NSO tax treatment. They can decide whether to tradeoff better tax treatment against the flexibility of having more time to decide if they’d like to exercise.

3) Cap table management

Issue: Increasing the post-termination exercise window means having shareholders who haven’t been actively involved with the company for years. This equity could have been “re-invested” to incentivize new or current employees. This also makes cap table management difficult, putting a burden on the company to keep updated contact information for them all.

Thoughts: Stock options are a reward for work that has already been done. Once vested, employees deserve to own their options outright and have the opportunity to exercise the rest of their vested options. The additional administrative overhead is small in comparison to protecting this right.

4) Dual class of employees

Issue: Extending the exercise window is not possible for existing employees and will only be applicable to new hires. This creates two classes of employees, penalizing the loyal long-time employees whose options did not originally have the option extension.

Thoughts: This is simply not true. It is possible to amend outstanding options held by existing employees to add an option extension. This amendment may convert an incentive stock option (ISO) into a nonstatutory stock option (NSO), and the company may have to comply with the tender offer rules to offer employees this option extension. However, this is completely feasible, and many companies have complied with the tender offer rules to implement this.

5) Better handled on a case by case basis

Issue: This issue is better handled on a case by case basis with each individual employee. Companies should have a personal conversation and find the best solution for each person, rather than offering this to everyone by default.

Thoughts: Option extensions require board and optionee consent. Handling option extensions on a case by case basis is administratively burdensome, because the company has to remember to seek board approval of an option extension each time someone leaves. In addition, a case by case analysis may make the company vulnerable to a claim of discrimination if employees and their options are not treated in the same manner. Finally, if there are multiple case by case option extensions, there comes a point when the tender offer rules will get triggered anyway, because of the number of option extensions that have been offered. This isn’t something a company wants sprung upon them without time to prepare for it.

Legal and Accounting Issues

1) Tender Offer

Giving current employees the decision whether to keep their ISOs without the option extension or have NSOs with the extension is an investment decision, which may require the company to engage in a tender offer when offering the option extension. This tender offer process is relatively easy to implement, but the company has to give its employees at least 20 business days to think it over and decide.

2) Accounting Charge

An option extension will likely result in a higher accounting charge with respect to the options, although many companies have not found this additional charge to be material. The company needs to check with its outside auditors on the accounting implications of an option extension, which will vary depending on the number of options affected by the option extension and the length of the option extension.

3) Taxes

With an option extension, a company can anticipate that most options will be NSOs when exercised. If NSOs are exercised when there is a gain on the exercise date, the company will incur an additional tax cost, because the company has to pay the employer portion of the employment tax on such gain. At the same time, the company gets a deduction equal to the gain recognized on the exercise of an NSO, so the company has to balance the cost of the employment tax against the benefit of a tax deduction. There is no comparable tax cost with the exercise of an ISO.

4) Acquisitions

If the company is acquired, occasionally, a buyer may require that the company track down all former employee optionees to obtain their consent to the option treatment, as a closing condition. However, the form of the Triplebyte stock plan is drafted to give maximum flexibility in the treatment of options in an acquisition, so optionee consent to the option treatment shouldn’t be required in most acquisitions.

5) IPO

Due to the option extension, there may be more options outstanding at an IPO, which may result in a larger overhang (i.e. larger number of outstanding equity). However, it’s possible that the company will have a similar overhang without the option extension, if most optionees exercise options and become shareholders.

If you have any questions about these, please get in touch. More companies are starting to follow the trend set by Pinterest, Quora, Coinbase, Amplitude and others and we’d like to see this become the standard.

Other Resources

A collection of thoughts discussing this issue we’d recommend reading:

Sam Altman: Employee Equity

Adam D’Angelo Quora Post

George Grellas (Founder of Grellas Shah LLP) Hacker News Post

Extending the Option Exercise Period — A Tactical Guide

Thanks to Nancy Chen at Orrick for providing the legal details in this post.

Gaming the H-1B system (for good)

A recent article in the NY Times exposed how flawed the H-1B lottery process is. A handful of giant outsourcing companies flood the system with applications, making it near impossible for startups to hire international engineers. 

These companies are gaming the system. But there is a way to turn this game against them, by exploiting the Achilles heel in their plan - the H-1B transfer. Getting a H-1B is tough because regardless of your personal merits, you're in a lottery with thousands of other candidates. Your choice of employer is limited by those willing to play the lottery.  There's no lottery for transferring a H-1B though. The process is straightforward with no quota, you just have to find an employer willing to file the paperwork. This gave us an idea. 

We're announcing the Triplebyte H-1b transfer program. If you're working on a H-1B at one of these outsourcing companies, apply to Triplebyte and we'll cover all the costs of transferring your H-1B. We'll help you find a startup doing work you're excited about and walk them through the H-1B transfer process, making it a no brainer for them. We'll also provide you with an immigration lawyer, to answer any questions you have, and we'll cover the cost of that too.   

We're going to expand the pool of startups doing H-1B transfers so you have the same choice as anyone else.  We recently placed an engineer using a H-1B transfer, at a startup who wouldn't have considered doing this without our help. Many founders mistakenly assume that applying for and transferring a H-1B are synonymous. 

Helping great people move here is something that's personally important to us. My life was changed by moving out here to work on my first startup (after a year of struggling with trying various approaches to getting a visa). My co-founder Guilllaume moved here from France to work at Justin.tv and then found his own startup, Socialcam. We want to see more talented people coming here to work on building the future, not being cheap labor for giant corporations.  

Thanks to Theo Negri and Buildzoom for shining a light on this issue in the original story.

Improving the technical hiring process

Guillaume, Ammon and I are excited to announce the launch of our new company, Triplebyte. Our goal is to build a consistent and data-driven process for hiring programmers.

Most companies make up their hiring process as they go along. We certainly did that when hiring at our own startups. This has problems. Resumes are relied on heavily as the first screen, but many great programmers have really bad resumes. Technical interviews are typically run by an interviewer who is unsure which questions to ask or how to evaluate answers. Final hiring decisions are based on gut feeling, which is rarely (i.e. never) measured for accuracy.

This is a manifesto of how we believe technical hiring should work. We want to build a company that specializes in assessing the ability of engineers without relying on the prestige of their resume credentials. Once we've identified them, we're going to help them find great places to work. We'll use the latter to measure how well we're doing at the former.

We're going to do two things differently. First, track decisions as quantitatively as possible. Second, run experiments with our own process. We expect it to change completely over time. Frankly, we'd love to get rid of interviews entirely.

We're starting our first experiment today - blind phone screens. First, we ask a few questions to verify you're a programmer. It's our version of an online FizzBuzz. Once you pass those, we ask you to schedule a 15-minute technical phone call. We only want to talk about one thing: code you've written in the past. That's literally the only thing we'll ask you about. Our hypothesis is that's enough to help good programmers stand out.  After that we'll go deeper into code you've written before over a couple of 45 minute technical interviews via screen share.   

Humans are complicated and making decisions about their ability is difficult. We're excited about trying because the potential reward is so large. A better hiring process can significantly reduce bias. It'll open up the opportunity for anyone, from anywhere, to be assessed on their ability. It'll help startups find the programmers they need to build great products. We think this would be a great thing for the world and we're excited to build it!

If you have ideas for other ways we could experiment with our process, or if you think there's a better approach than the one we're taking, we'd love to hear from you. founders@triplebyte.com.