The trouble with technical interviews? They aren't like the job you're interviewing for

Sara Kassabian ·
Mar 19, 2020 · 21 min read

Interviewing for an engineering job in the tech world can mean you’ll be asked all sorts of questions. Sometimes, the job interview questions can be pretty straightforward: “Tell me about a time that you have implemented an effective monitoring solution for a production system.” Other times, the questions are impossible to answer and designed to spark your creativity: “How many windows are in New York City?” After passing the initial interview, the applicant or candidate graduates to the next tier of interviewing: The often-dreaded technical interview.

What is a technical interview?

A technical interview is one that is conducted to gauge a candidate’s skill level for positions in the information technology, engineering, and science fields. It may also determine how much a candidate knows in more niche areas of a company, such as marketing, sales, and HR.

Preparing for technical Interviews

Prospective engineers often face a challenge when it comes to preparing for the technical interview, largely because there is no playbook for how companies set them up technical. It’s unclear whether to prepare by memorizing many different topics, or focusing on specific projects. Is it better to practice with a computer or a peer engineer? There are an overwhelming number of resources available online, but with little clarity as to what the standard is for a technical interview and little guidance from the company on what to expect, most of the time engineers start technical interviews in the dark.

Inconsistencies in the technical interview process isn’t just a job candidate problem. In fact, many companies struggle to set up a technical interview process that is effective, equitable, and allows the hiring manager to compare candidates. The problem with technical interviewing compounds when a company is experiencing rapid growth.

The challenge with conducting technical interviews at a growing company

"Imagine you had a hiring target of doubling your team size and all your interviews are conducted remotely. Welcome to GitLab," says Clement Ho, frontend engineering manager on the Monitor: Health team at GitLab.

Hiring chart shows GitLab more than doubled the number of hires from around 400 in 2019 to roughly 1300 by end of 2020

GitLab more than doubled the number of hires from around 400 in 2019 to roughly 1300 by end of 2020.

We identifed three core challenges with orchestrating technical interviews as GitLab grows.

  1. We didn't have enough interviewers for the pipeline of candidates.
  2. Our technical interviewing process was inconsistent and even a little biased.
  3. It was difficult to measure whether or not we were raising the bar.

"And by raising the bar, I mean making sure each candidate that joins the team makes the team better," says Clement.

These problems are by no means unique to GitLab. Any engineering company that is scaling rapidly will encounter some growing pains when it comes to hiring, and many will end up falling back on some of the typical models for conducting technical interviews.

The typical technical interview methods

During his talk, "Using GitLab to Power our Frontend Technical Interviews" at GitLab Commit San Francisco, Clement explained the four different techniques that are often employed in technical interviews. Each method comes with advantages and disadvantages from the perspective of the hiring manager.

Good technical interview questions

A good technical interview needs to be about more than practical skills – it’s about the whole package.A candidate should possess the ideal coding skills but also be a team and culture fit and be able to discuss developer topics efficiently. A technical interview should include both situational interview questions and a skills assessment to discern a candidate’s potential.

The types of questions to ask can concern a candidate’s technical abilities and background, their career journey so far, and queries specific to the team or company.

Types of questions asked during a technical interview and their purpose

Even though employers have already reviewed your resume and cover letter, they will want you to flesh that out during the interview to learn more about how you attained those skills. In order to assess your level of experience, they will likely also ask you to provide concrete examples from prior jobs. Make sure you are prepared—do your research on the company and the type of questions you may be potentially asked. This will help build your confidence level and reduce any nervousness you might feel. It’s also an opportunity for you to set yourself apart from other candidates by showcasing your knowledge and additional skills you can bring to the job.

It is important to be honest about your skill set because that is something employers value. You may find the company will be willing to hire someone who is transparent about the areas where they need to improve and where they’d like to gain more skills.

Examples of common questions to expect in a technical interview:

Sample technical interview questions and answers

Soft skills and coding skills to highlight in a technical interview

A technical interview assesses your technical expertise, coding skills, and ability to fit into a team. However, soft skills are just as important and often aid in the development of more technical skills – particularly in a team setting.

As the technical interview progresses, be prepared to tackle some questions about soft skills like:

Verbal technical questions

There are countless articles online that try to prepare job candidates for a verbal technical interview, but whether this method truly effective for evaluating the technical competency of a software engineer is debatable.

In the typical scenario, the interviewer asks the candidate to describe a technical concept and tries to measure their fluency in said concept based on the quality of the conversation.

The advantage of this method is that the interviewer can understand how the candidate communicates, which is of particular importance when the engineering team is all-remote, as is the case at GitLab. The drawback? Being a good communicator does not necessarily mean the candidate knows how to code effectively.

"So I've interviewed candidates that could talk the talk, but they couldn't really write the code,” says Clement. "And that's not a great situation for an engineer to join GitLab." Clement’s team has moved away from using verbal technical questions as a method for evaluating candidates.

Live coding exercises

One of the more popular methods for evaluating engineers is through live coding. While it allows the evaluator to see how engineering candidates answer data structure questions, it also has its disadvantages.

A key advantage of live coding data structures is that it offers a fairly consistent measurement and evaluation.

"I can talk to another manager or another interviewer and be able to communicate, 'Hey, this person wasn't able to do a linked list, they got stuck here. They weren't able to understand a runtime efficiency here.' So it's pretty consistent," says Clement.

But the ability to create data structures is not always the best indicator of ability. Oftentimes engineers with a very traditional background or recent graduates will shine here, but someone who is more senior and able to do a lot of great things, but is perhaps not as brushed up on data structures, may struggle.

Live coding interviews probably aren’t going anywhere fast, but the pitfalls of this method are well documented by engineers and hiring managers. Brennan Moore, a product engineer in New York City, explains why he does not conduct live coding interviews when evaluating a prospective candidate:

"Much like the SAT when applying for college, live coding is a structured test. I didn’t go to a school that trained me to do live coding, and so will probably fail the test. As I’ve experienced it, live coding isn’t the meritocratic space that it pretends to be. Live coding interviews weed out the people who are good at live coding interviews," says Brennan in his blog post.

At GitLab, we found that live coding exercises don't accurately represent engineering capability. Oftentimes, a recent computer science graduate will outperform a more senior candidate with a lot of valuable experience. In summary, live coding exercises will often disadvantage more senior candidates, people who are nervous in high-pressure situations (read: everyone), and advantages more junior engineers or people who have practiced live coding.

Digital prompt

A third common method for evaluating candidates is to ask the engineer to code a UI using an online editor while on screen share with the evaluator.

The advantage of this method is that it allows the evaluator to observe how a candidate builds. The drawbacks here are similar to those with live coding. First, the engineer is under pressure to build while the evaluator watches on, making it a nerve-wracking situation. The other drawbacks come from an evaluation perspective: It is challenging to measure the effectiveness of this method and it is hard to compare between candidates.

Take-home project

Any engineer (or writer, for that matter) can tell you, the supplemental take-home project is a very common ask when going through an interview process. The advantage here for us is that this assignment closely mimics the reality of building environments while working remotely at GitLab.

But this task comes with major drawbacks, mainly that it disadvantages candidates who may not have the time or capacity to complete the project.

"… imagine a scenario where you're a single parent and you have kids; you may not have as much opportunity to take dedicated time, a couple of hours after work to really focus on a take-home project compared to someone from a more privileged background," says Clement. "They might be able to dedicate and output something better."

Diversity and inclusion is a core value for GitLab, and anything that disadvantages candidates from underrepresented groups is not inclusive, and therefore suboptimal for evaluating candidates based on their engineering abilities.

What are they looking for during a technical interview?

Companies want candidates who can discuss the industry in the context of the job they are applying for. Be prepared to discuss examples of your work. Many will want to hear about soft skills, too—your ability to communicate and collaborate and work with others to problem-solve issues.

They will also want to see how passionate and enthusiastic you are and whether you have the self-motivation to not only do the job but take the initiative to do more than what you’re tasked with.

Also, interviewers will want to see whether candidates have the desire to increase their technical knowledge.

Online preparation tools and resources for technical interviews

Meaningful questions to ask the interviewer

Candidates will also be given a chance to ask questions they might have to learn more about the company. This is a great opportunity to gain more insight into how the company operates, what its philosophy is, and its vision for the long term.

It’s also a good way to glean how the company views its IT team. If you don’t ask questions, that could give the impression you are unprepared or not terribly interested in the job.

Questions to ask can include:

The new way

While each method for conducting a technical interview comes with advantages, there are also numerous disadvantages when it comes to conducting an effective and measurable evaluation and creating an equitable interview process. Under the guidance of Clement, the Monitor:Health team decided to interview frontend engineers in an entirely new way using GitLab.

Now let's take a deep dive into the nuts and bolts of reinventing the technical interview for frontend engineers at GitLab. Just wondering about the key takeaways? Skip ahead. As we continue to iterate on a more effective and measurable technical interview process, we hope this inspires other engineering organizations to rethink theirs and share learnings with us.

Our first step: Standardize the interview process.

Fixing an MR on a test project

The team standardized the interview process by creating an open source test project, called project-seeder, which seeds projects to different candidates using a GitLab Bot. Candidates are assigned a merge request to troubleshoot in the project created for the technical interview. The project-seeder is powered by the GitLab Bot so the interviewer doesn't have to worry about API keys, and works in four steps:

  1. Exports the template project
  2. Imports template project
  3. Adds users with expiration
  4. Triggers pipeline for candidate to review MR

The candidate is sent an email with a link to the MR the candidate is assigned to fix as part of the technical interview.

Standardize the evaluation rubric

The team also created a standardized rubric for how the candidate's performance on a technical interview is evaluated.

"We don't want to be in a situation where unconscious bias or bias of one candidate over another plays a part because of our preconceived notions," says Clement.

Creating a rubric that looks at multiple categories allows the evaluator to look at the performance of the candidate from a more holistic perspective, as opposed to looking at a candidate's performance on one technology.

The team created a Periscope dashboard to create a feedback loop between the candidates and evaluators to identify opportunities for improvement in the technical interviewing process.

Frontend team used Periscrope to collect feedback from candidates who participate in technical interviews

The frontend engineering team used Periscope to collect feedback from candidates who participate in technical interviews.

Demoing the technical interview

Inside the technical interview project

Clement created a sample project to demonstrate how we use GitLab to power our technical interviews.

In the gl-commit-example group, there is a subgroup with all the interview projects we are seeding to the imaginary candidates, a template, and a project seeder.

A screenshot of the sample project shows the interview project's subgroup, template, and project seeder application

The interview project's subgroup, template, and project seeder application lives inside the sample project for the technical interview.

Inside the template, there are GitLab pages and the interview test merge request.

The assignment here is pretty simple. The candidate needs to update the website to say "Hello GitLab Commit SF," but in order to accomplish this, the candidate will need to fix the failing pipeline.

Powering project-seeder

We use variables from GitLab CI to configure the project-seeder application.

Screenshot of the project for the project-seeder application

Inside the project-seeder application which seeds the interview projects to job candidates.

"I'm creating new-project-example-two, and I'm adding this bot user that I created and the expiration, so I can just easily run this pipeline and it'll seed this project," says Clement.

We use variables from the GitLab CI to configure the project-seeder applications

The next step is to run the setup pipeline, which will create the project, import the project, export the project, and share it with the job candidate.

A look inside the pipeline that will create the test project A look inside the pipeline that will create the test project.

Looking inside example-one, we can see there is a project and broken MR.

"And an example for a candidate – they would probably look at the CI and see, 'Oh there's a failing test. Let's see what that's about. Oh, it looks like it's checking for "hello world". So since we changed the message earlier, we can just change this and get this test passing and then pass this interview,'" says Clement.

Why this new model for technical interviews is better

The new model surpasses the old model because we created realistic scenarios that reflect what it's like to actually work for GitLab, and we established a more consistent method of measurement.

"So we're able to get better candidates overall. Candidates that pass through this technical interview, we're sure that they're going to be successful at GitLab," says Clement.

By designing our technical interviews this way, we can ensure that the interview project matches our actual product architecture at GitLab, which in this case is Ruby on Rails for Vue JS.

We also struggled in the past with finding a good way to check that the candidate knows how to use Git, and can navigate pipelines and testing. By using GitLab for interviews, we're able to confirm a candidate's competency with Git implicitly by evaluating their performance on the technical interviews.

We wanted to mirror the actual experience of troubleshooting a broken MR while working at GitLab, so we allow our candidates to use the internet during their technical interview. This allows the evaluator to see how the candidate solves problems and see their resourcefulness.

"If you're already using GitLab for your tooling, you're just exposing them to what it's like to work at GitLab; it's a more accurate representation," says Clement. "And you can also make sure you're measuring testing proficiency and you make sure they understand how that works before they join your company."

Four key takeaways from our technical interview update

Whether or not a company uses GitLab, there are a few key lessons that we learned by iterating on how we conduct technical interviews for engineers.

  1. Make technical interviews as much like real work as possible: Nine times out of ten, an engineering manager isn't going to sit back and watch an engineer break a sweat in a live coding exercise, any more than they will watch on as an engineer builds in UI. Create realistic scenarios based on the actual work and evaluate based on the candidate's performance.

  2. Make any technical interview process "open-book": Engineering doesn't involve much rote memorization. Instead, allow the engineering candidate to use the internet (and in our case, the GitLab Handbook) to look up their questions. It's better to see how a candidate applies their knowledge and troubleshoots the inevitable problems that may arise. This change will likely improve your candidate experience too.

  3. Standardize your rubric: However the technical interview is done, make sure that the rubric is as objective as possible and that the candidate is evaluated based on various criteria, not on their familiarity with a particular technology. A strong rubric means a stronger, more valid method for evaluating candidate performance.

  4. Create an inclusive process: Think critically about how the technical interviewing process and evaluation is structured so a diverse group of candidates can be recruited and evaluated based on their merits. When in doubt, ask a diversity, inclusion and belonging expert or turn to your human resources team for advice. Still coming up empty? Hire a diversity consultant; it will be worth it.

Interviewing at GitLab? We encourage you to use the resources GitLab creates during your technical interview. We don't publish our evaluation criteria publicly, but we do have the Periscope dashboard which can provide some insight.

Watch Clement's talk from GitLab Commit San Francisco to learn more about how we used GitLab to power our technical interviewing process.

“Technical interviewing is broken. Here are some tips on how to fix it @GitLab” – Sara Kassabian

Click to tweet

Guide to the Cloud

Harness the power of the cloud with microservices, cloud-agnostic DevOps, and workflow portability.

Learn more
Edit this page View source