Introduction

What’s an interview without questions? Obviously, technical interviews are pivotal in evaluating candidates for technical roles, and the quality of questions asked can significantly impact the hiring process (duh). In our experience, asking the “right” questions is a much more nuanced skill than initially thought. In this (mini)guide, we delve into the importance of well-developed technical questions, the process of crafting them effectively, and strategies for ensuring fairness and efficacy throughout the interview process.

Importance of Well-Developed Technical Questions

Well-crafted technical questions are critical in:

• Ensuring a better quality of hire.
  • Technical questions bridge the gap between a candidate’s resume and their actual abilities, providing valuable insights into their suitability for the position.
• Enhancing the candidate experience.
  • Engaging and relevant questions demonstrate the organization’s commitment to a fair and thorough evaluation process, leaving candidates with a positive impression.

Job Analysis and Requirements

Before formulating technical questions, it’s essential to conduct a comprehensive analysis of the job requirements. This involves:

• Reviewing the job description and identifying the required technical skills.
• Identifying core competencies relevant to the position.
• Understanding the level of expertise expected for the role.

❓Defining Question Types

1. Technical Knowledge Questions:

• Assess fundamental knowledge in relevant technologies.
• Cover key concepts and principles essential for the role.

1. Problem-Solving and Coding Questions:

• Evaluate problem-solving abilities.
• Assess coding skills and logical reasoning.
• Align with real-world scenarios and day-to-day tasks of the role.

1. Leadership Questions:

• Survey leadership qualities for growth and increased responsibilities.
• Even if not in a “leadership” position, leadership qualities inspire others and promote ingenuity.

1. Culture Add Questions:

• Shows interest and commitment to inclusivity and diversity.
• Demonstrates commitment to company values.

An example of what not to do:

At a prior company, we ran into a problem of many Mobile Engineering candidates being unable to pass the technical interview. We diagnosed that the questions being asked were focused on algorithmic knowledge and data structures that weren’t relevant to the day-to-day responsibilities of a Mobile Engineer as outlined in the job description. Most of these candidates hadn’t touched these concepts since their degree, so were unequipped for success unless they spent weeks preparing beforehand. When we ran this up the chain, we were told that the company wanted Engineers with “strong fundamentals,” and to continue our current course. It took a well-known iOS Engineer coming directly from Apple to fail the technical interview before Engineering Leadership took notice and reconsidered their interview questions. 

🎉Structuring Questions Effectively

To maximize the effectiveness of technical questions:

• Provide clear context and instructions.
  • Break down complex problems into manageable parts.
• Use a step-by-step approach for coding questions.
• Ask one question at a time to avoid confusion.
• “Bad” vs. “Good” vs. “Great” Interview Questions
  • Bad question
    • Can be answered with “yes” or “no”.
    • Not relevant to the work being performed.
    • “Have you ever built a data platform before?”
  • Good question
    • Open ended, requires short/long answer.
    • Relevant to work.
    • Doesn’t direct candidate to give deep specifics.
    • “Tell me about the time when you built a data platform in Java and what your role was in the development.”
  • Great question
    • Specific and relevant to day-to-day work and responsibilities.
    • Asks candidate to provide in-depth, technical, and specific example.
    • Is narrow focused.
    • Leads to specific follow up questions.
    • Provides insight into rationale behind decision making.
    • “What were some of the major challenges or obstacles you encountered while building the data platform at Company X? How did you overcome them? Why did you pick that solution vs. one of the others?” 

🧑‍⚖️Avoiding Bias and Unfairness

To begin, avoiding bias and unfairness requires a meticulous review of interview questions. Each question should undergo scrutiny to identify and eliminate any biases that may inadvertently disadvantage certain demographics. While this is by no means an exhaustive list, here are some common examples:

• Stereotyping
  • Generalized questions/opinions about a protected class such as race, age, religion, sex, etc.
• Inconsistency
  • Asking different questions to different candidates for the same job. For example, asking a Latino candidate if they’re bilingual while not asking a Caucasian candidate the same.
• Contrast
  • Comparing current candidates to past ones. Candidates should be evaluated against the job description, not each other.
• “Similar to Me”
  • Evaluating the candidate by connecting on a personal level instead of evaluating against the job criteria.

 Additionally, ensuring inclusivity and fairness in both language and content is crucial. Using neutral language and avoiding terms that may alienate specific groups helps create an environment where all candidates feel equally valued and respected.

To maintain fairness and inclusivity:

• Review questions for potential bias.
• Ensure inclusivity and fairness in language and content.
• Avoid questions that may disadvantage certain demographics.

👏Practicality and Feasibility

Practicality and feasibility are also essential considerations when crafting interview questions. Questions should be designed to be answered within a reasonable time frame, avoiding complexities or time-consuming tasks that may overwhelm candidates or lead to unfair assessments of their abilities. Moreover, questions should allow for clear and objective evaluation, free from subjective interpretation. This ensures that candidates are assessed solely on their merits and qualifications, rather than subjective judgments.

• Assess the feasibility of answering questions within a reasonable time frame.
• Ensure questions can be evaluated objectively.
• Consider the resources required for candidates to answer effectively.

🧑‍✈️Pilot Testing

Conducting pilot testing with current employees is a valuable step in refining interview questions. Gathering feedback on question clarity and difficulty enables adjustments to be made to ensure questions are well-suited for the actual hiring process. Similarly, maintaining consistency across interviewers is critical for fair evaluations (check out our Interviewer Training mini-guide!). Providing clear guidelines for asking questions and conducting calibration sessions to align interviewers’ expectations help ensure uniformity in evaluation standards.

• Conduct pilot interviews with current employees.
• Gather feedback on question clarity and difficulty.
• Make adjustments based on pilot test results.

Consistency Across Interviewers

Feedback from both interviewers and candidates is invaluable for continuous improvement. Collecting feedback on interview questions and the overall interview experience helps identify areas for refinement and enhancement. Regularly updating questions based on evolving technologies, candidate responses, and market conditions ensures that interview questions remain relevant and effective in assessing candidates’ suitability for the role.

• Provide clear guidelines for asking questions.
• Ensure uniformity in the evaluation process.
• Conduct calibration sessions to align interviewers’ expectations.

Feedback and Continuous Improvement

Collect feedback from interviewers and candidates.

Update questions based on evolving technologies, candidate responses, and market conditions.

Regularly review and refresh the question bank.

Documentation

Finally, meticulous documentation is essential for accountability and transparency in the hiring process. Documenting the rationale behind each question and maintaining a repository of approved technical questions ensures consistency and standardization across interviews. Regular updates to documentation reflect changes in job requirements or organizational priorities, ensuring interview questions remain aligned with evolving needs.

• Document the rationale behind each question.
• Maintain a repository of approved technical questions.
• Update documentation to reflect changes in job requirements.

Conclusion

In conclusion, crafting fair and effective interview questions is foundational to promoting diversity, equity, and inclusion in the hiring process. Create a more equitable and transparent hiring process that provides equal opportunities for all candidates to succeed and in turn, help your business succeed.