Bootcamps vs. College

Programming bootcamps seem to make an impossible claim. Instead of spending four years in university, they say, you can learn how to be a software engineer in a three month program. On the face of it, this sounds more like an ad for Trump University than a plausible educational model.

But this is not what we’ve found at Triplebyte. We do interviews with engineers, and match them with startups where they’ll be a good fit. Companies vary widely in what skills they look for, and by mapping these differences, we’re able to help engineers pass more interviews and find jobs they would not have found on their own. Over the last year, we’ve worked with about 100 bootcamp grads, and many have gone on to get jobs at great companies. We do our interviews blind, without knowing a candidate's background, and we regularly get through an interview and give a candidate very positive scores, only to be surprised at the end when we learn that the candidate has only been programming for 6 months.

Bootcamp grads are junior programmers. They have a lot to learn, and represent an investment on the part of a company that hires them. That said, this is also true of recent college graduates. We’ve found bootcamp grads as a group to be better than college grads at web programming and writing clean, modular code, and worse at algorithms and understanding how computers work. All in all, we’ve had roughly equivalent success working with the two groups.

In this post, I'm going to try to shed some light on how this can be true. I’ll dig more into the differences that we see between the two groups, and hopefully explain how some people can become competitive junior programmers in under a year.

The Analysis

Our technical interview at Triplebyte is about two and a half hours long, and is broken into four main parts, focusing on practical programming, web architecture, low-level system understanding, and algorithmic understanding. Not every engineer completes every question (we let programmers focus on their strengths), but the process gives us a good measure of the relative strengths of each engineer in each of these areas.

To get a better idea of how bootcamp grads and college grads compare, I graphed the two groups’ average performance in each of these areas. The y axis is the score on each problem (where 1 = strong no, 2 = weak no, 3 = weak yes, 4 = strong yes). For reference, I also included the entire population of applicants, and also all engineers who pass our interview. 
The first thing to note about this graph is that bootcamp grads do as well as or better than college grads on practical programming and web system design, and do worse on algorithms and low-level systems. Our practical programming questions are not easy. They require understanding a problem, coming up with abstraction to solve the problem, and rendering this in code. If anything, our practical programming questions require more on-the-spot thinking than our algorithm problems do. They do not, however, require academic CS or math, or any specific knowledge. This is the crux of the issue. Bootcamp grads match or beat college grads on practical skills, and lose on deep knowledge.

A similar pattern holds on the design questions. Bootcamp grads do better on web questions involving web servers, databases and load balancers. College grads do better on low-level design questions involving bit/bytes, threading, memory allocation, and understanding how a computer actually works.

Triplebyte sees a biased sample of both bootcamp grads and college grads. We do background-blind screening via an online programming test, and only interview engineers who pass this test. Thus we have no way to know what percentage of bootcamp grads and college grads fail early in our process, and the graph above reflects only people who pass our test. Still, a significant number of bootcamp grads pass our test and go on to do as well as college grads on our interviews.

I want to specifically draw attention to the performance of college grads on algorithm problems. They are not only better than bootcamp grads, they are a lot better. They are significantly better than the average programmer making it to our interview (most of whom have 2+ years of experience), and almost as good at the average engineers who we pass. This is interesting. It backs up the assertion that algorithm skills are not used on the job by most programmers, and atrophy over time.

How is this possible?

Our data aside, it still just seems hard to believe that 3 months can compete with a 4-year university degree. The time scales just seem off. The first thing to note is that the difference in instructional time is not as large as it seems. Bootcamps, are intense. Students complete 8 hours of work daily, and many stay late and work on the weekends (one popular bootcamp runs 6 days per week). TAs are working with the students during this entire time. What bootcamps lack in duration they perhaps make up in intensity. 

The second point is that bootcamps teach practical skills. Traditional CS programmers spend significant amounts of time on concepts like NP-completeness and programming in Scheme. Now, I in no sense mean to belittle this material. I love academic CS (and the Cook–Levin theorem). It’s beautiful, fascinating stuff, and I got a lot out of learning it. But it is not directly applicable to what most programmers do most of the time. Bootcamps are able to show outsized results by relentlessly focusing on practical skills. Bootcamp TAs continually critique each student's coding style. They teach testing. They teach their students how to use their editors. How to use an editor is something that a traditional CS degree program would never think of teaching.

This does not leave bootcamp grads equivalently skilled to university grads. If you want to do hard algorithmic or low-level programming, you’re still better served by a traditional CS eduction. But it does leave the best of them surprisingly competitive for many entry-level development positions.

Conclusion

There are two ways to interpret the results in this blog post. One way is to say that bootcamps are window dressing. They teach inexperienced programers what they need to know to look like good programmers, but skimp on the heart of the discipline. However, I think this view is too cynical. The other way to view this post is as evidence that bootcamps focus on totally different areas than CS programs. They focus intensely on the practical skills required to be a productive programmer. These are skills that CS programs expect students to pick up around the edges of their course work. By being this pragmatic and giving students an intense workload, bootcamps are able to match the practical skills of CS grads.

Bootcamp grads don’t make sense for all companies. Just like recent college grads, they are an investment for a company that hires them. They have much to learn. And they are clearly worse at algorithms and low level systems than engineers with academic training. A database or self-driving car company should probably stick to folks with CS degrees. But the significant majority of companies need programers to solve practical problems on the web. On this axis, we’ve found bootcamp grads totally competitive.

Triplebyte is one year old. In that time, we’ve both placed bootcamp grads at top companies, and also watched them grow. We’ve watched them learn some of the CS skills that they lack on graduation. We’ve watched them learn about large-scale production systems. We’ve watched them take on leadership positions. It’s really incredible how quickly and how well the best bootcamp grads learn. It’s been a pleasure to work with them, and we’ll definitely keep working with bootcamp grads.

If you’re a bootcamp grad (or a college grad, or anyone else), and are interested in a way to find companies where you’re a strong technical match, give our process a try. I'm also interested in your thoughts on this post! Send me an email at ammon@triplebyte.com.

Thanks to Jared Friedman and Daniel Gackle for reading drafts of this, and Buck Shlegeris for major help writing it.
37 responses
"Bootcamp grads match or beat college grads on practical skills, and lose on deep knowledge. ... There are two ways to interpret the results in this blog post. One way is to say that bootcamps are window dressing. They teach inexperienced programers what they need to know to look like good programmers, but skimp on the heart of the discipline. However, I think this view is too cynical.": After many years, even from the beginning of my 'career', this lack of "deep knowledge" is far more frustrating than dealing with "practicality". In fact, "practicality" is greatly informed by "deep knowledge" and I have found when people are being "practical" it generally means taking the easy path because they simply cant take any other path! I dont understand these (meaning often pop-up on HN) numerous posts that intimate, if not outright state, that college is obsolete.
I think the real problem here is that the software space has evolved enough in the past 20 years that those four skills aren't universally relevant for all development roles. We've refactored the environment enough that many components can be safely hidden away from the average developer. Consider the following: Practical Programming: Probably the closest to a universally useful skill of the four BUT - there's a universe of point-and-click tools for data analysis (R, SAS, SPSS), business intelligence (Microstrategy), and rapid application development (Force.com, app builders) for folks who want to focus on the algorithmic side of the space without really learning to code Algorithms & Data Structures: Can be safely evaded on low-volume systems (startups with few users) or when most of the detail is abstracted away via your ORM or framework. Low Level System Design: Another topic which can be safely evaded by many developers working on virtual platform. Web System Design: Vital if you're running a website. Merely nice to have for other applications (yes, its a quick UI option). What we're really seeing here is the labor force re-mixing the set of skills to more efficiently meet the needs of different roles. The typical junior dev isn't going to work on algorithms (much). The really serious algorithms work will likely be sent to a super-high level specialist who is retained as a consultant or in-house expert. Thus...why teach all people all the things?
I never went through a bootcamp but the process described here sounds exactly like my first six months in a Software Engineer position. I'm not surprised that being taught in the same way you're going to work is so beneficial to new developers. For reference I'm a SW Engineer I in Devop and my primary duties are building out automation for existing applications and maintaining the developer build pipeline.
We've seen the same kind of shifts in market demand and our curriculum at Code Fellows (www.codefellows.org) in Seattle. Three years ago when the company started you could place a code school grad as a front end developer without full stack skills. The market needed people and programs like this were new. Today, full stack skills reflect the minimum bar and we continue to add more basics on the CS skills. To be clear though, this is skills based training. The majority of students have a degree already and are retooling their career to an industry where jobs are available. The recent Stack Overflow survey also validates some of the data around percentage of developers (50,000) that were self taught as well as compensation based on experience and training.
Does your list of "bootcamp grads" include both people who went to college and those who didn't? I expect that a decent fraction of the bootcamp grads DID go to college. If so, then the bootcamp grads are people who did college + bootcamp, which, on priors, should make them better than college grads. I don't know how many of them majored in CS though. I'm expecting bootcamps to be used mostly by people who didn't major in CS, though I'd expect some who majored in CS but didn't cover enough practical stuff to also use bootcamps.
Some more things to consider: Bootcamps frequently require pre-work so basically before you enter a bootcamp you've shown yourself to be able to learn on your own, be a self starter etc - basically, if you're accepted into such a bootcamp, you're already identified as a success story. Then, you have to be able to afford to go to a bootcamp - that's the bootcamp fee and no other income for 2 to 3 months so we're talking about basically an affluent population. So, most bootcamps are taking young people who, had they thought of studying CS in college would likely have been at or near the top of their classes. So, to be a fair comparison, you should only consider college CS majors from programs that cream from the high school population - the most exclusive CS programs in the country. Next, one year isn't long enough. It would be interesting to track careers many years out. On the one hand, the bootcamp people were pre-selected as people who can learn on their own (pre-work) but they don't have the foundations of the college cs majors -- I think it would make for interesting results.
I think the data makes sense. Despite working in more theoretical fields I would say that most of my computer science degree was never applicable in practical work. While many of the important skills for programming was never learnt in college. E.g. after my BSci I knew about machine learning algorithms like neural networks, I knew signal processing, reverse kinematics and a whole bunch of other stuff I never used. Yet I knew nothing about how to structure and organize a program. I knew nothing about debugging. I knew nothing about practical usage of compilers. I definitely think there is a need for computer science, but I think there is a gross overuse and demand for computer science background in industry when practical programming knowledge is what most jobs require. In the years saved doing something like bootcamp, you can learn a lot of practical programming. If 6 months of bootcamp beats a CS graduate in practical problem solving. Then what does 6 months + 2.5 years of experience do then?
Boot Camp looks like a great way to jumpstart your skills. I would suggest both college and Boot Camp.
Here's where I beg to disagree that all students are affluent. I'm a hearing impaired queer woman who has been in the tech industry in one way or other for 20 years. I'm also a single parent who provides 100% support for my child. I literally can not afford to be the kind of poor or have the kind of debt that goes into converting my associates degree into a 4 year CS degree, nor is it very likely I'd be accepted without spending another 6 months just on taking the required math classes (it's not a lack of understanding, but a lack of exposure combined with adhd which made logic super easy to get a 4.0 in my classes, but made math a struggle). I grew up destitute on a level most Americans only see in save the children commercials. I'm lucky enough to have lucked into a situation where I could afford (via unemployment and buying out my stocks and other hard savings) to spend the next 6 months on a boot camp to learn c# and .net. I am not and by no means ever have been considered affluent at all in my life. Not having a degree is a sucky thing for sure, it gives companies a way to legitimately discriminate against me when most of the time it's the parental status or the minority status that is the true reason; (esp obvious when they don't advertise they wanted a degree in the first place). The boot camp literally is me using what time I have to have one reportable thing behind me so that they can not use that against me. Don't get me wrong, I rock the ever living bjebus out of coursera and every free CS related class I can get my hands on. I just do not have the time or money or credit to pull 2 more years of very expensive ($40,000 vs code school's $5,000) out of my backside. It actually takes someone quite young and privileged to go to college. You can't have any medical (or otherwise) debt to pull off the loans you need to cover what a pell grant won't give you, and even with loans and pell grants you are still going to have to work while going to school. At least with a code school, that amount of time being dirt poor is way reduced, and there are companies that will finance your code school.
There is yet another way to interpret the data. It may be that Triplebyte's interviewing method is simply not very good at distinguishing good candidates from bad candidates, so that whether the candidate went to college or boot camp is not the most relevant reason why some selected candidates succeed and others fail. I have read many articles saying that companies aren't very good at hiring good developers, so this result would not really be a surprise if true. We may have to wait a few years to see how boot campers fare in the job marketplace as the technologies they were originally taught become obsolete. Knowledge of algorithms and computability is more durable than knowledge of specific tools and languages. A college major is about 1,000 hours of instruction and an equal number of hours of homework. A BS degree contains another 1,000 hours of instruction in non-major subjects, and an equal amount of homework. At least some of this instruction is likely to be applicable to the major area of concentration. If a boot camp offers six hours of instruction per day for 3 months of weekdays, that's still only 375 hours of instruction, about a third of what you get in a major field of study with a bachelor's degree. Each hour of boot camp instruction would have to be astonishingly more productive than the equivalent hour of college classes. This might be the case, because college instruction is often not of high quality. But the advocates of boot camp definitely have something to demonstrate here. This article doesn't address the cost-effectiveness of boot camps. I don't personally have good numbers for the cost of boot camps, but I have the impression that they can be quite expensive. It may be that the cost per hour of instruction at a state University is lower than for a boot camp. This would make boot camps a bad deal even if they were more concentrated, unless the instruction was of demonstrably higher quality (whatever that means).
I don't see why bootcamp skills (i.e., the ones that aren't discussed in the formal courses) can't be taught in college as a course (with lab) right after the introductory CS courses.
I think the major point of confusion is that CS grads are not programmers, and programmers are not software engineers. It don't think it's too surprising that you can pick up some programming in a rather short period of time---if you take the same amount of time, a person can also get reasonably fluid in language and culture of a foreign country. It doesn't make him a linguist though.
Problem with most college CS programs is they focus more on theory instead of actual programming which should be actually taught like a math/foreign language class. Because of this deficiency most CS majors are not ready for programming in the job world. In the business world from what I hear the people who hire in the programming field if you can prove your programming skills they really don't care if you have a CS degree or not.
"Focusing on theory" is not a problem. But, let me be a person from the "business world" who values "theory" over "practicality". How does that manifest in my hiring practice? I hire in this order: 1. Theory + proven experience 2. Theory only 3. Proven experience only If Im hiring for a bridge builder I don't want: "Well, I built a number of bridges and they are still standing. Not really sure why tho." I want: "Well, I haven't built any bridges yet but I know how they are supposed to be built." Its then my job to mentor them in the experiential/practical. Please don't hire those who don't know "why" what they do works or not.
This topic is quite fascinating. I neither took a single CS course in college, nor have I been to a bootcamp. I am completely self-taught. But my natural curiosity has forced me into low-level topics. I've soldered my own binary calculator together using basic logic IC's. I've put together my own transistor. I even read through and understand the electro-chemical properties of an NPN circuit. What makes someone good at what they do? In the field of technology, I believe it is a combination of curiosity, determination, and a natural cognitive skill set.
Practical skills can be taught to a CS grad over a matter of weeks. They have the "deep knowledge" to pick up skills required for a particular job on the fly. While it isn't required, a degree in CS (or related field) is a proven way to filter out crap for just about every top tech company in the world
The fields where you need regular use of algorithms or low-level programming are few and far between. The vast majority of current openings in tech will almost never require you to do anything other than google stack-overflow. If you somehow need algorithms and data structures to be known, put that up front in your job description or train them yourself.
20 visitors upvoted this post.