Create Interviewer Feedback Form

[doctorsandy]

Overview

We need to create an interviewer feedback form template so that interviewers can submit information about the quality of interviews they have performed after each research plan.

Details

Excerpt from the GUIDE: How to Conduct Interviews

See GUIDE: How to Conduct Interviews, page 4

To facilitate this process, in the post-interview phase, each researcher should fill out a separate form that is only available to other team members who must have access to this information. See this Interviewer Feedback Form. Researchers filling out this document will not be asked for any personal information and will be further de-identified if necessary This form should include the following:

• Likert Scale. “On a scale of 1-10, how would you rate the quality of this interview?”
• A brief memo that can be a few bullet points. Commentary can be given on anything that the researcher finds significant in helping other researchers analyze the data. For instance:
  • “Participant asked me why I speak English so well. I am a Korean-American who speaks English fluently.”
  • “Participant made a homophobic remark during the interview.”
  • “Participant expressed disdain for international students.”
  • “Male participant was very argumentative with the focus group participants who were women.”
  • “Participant seemed hesitant to discuss…..”

Action Items

• Identify what information needs to be collected.
  • See Guide: How to Conduct Interviews
• Create a comment below with form questions
• Review with research lead
• Review with product
• Create a template issue to use form
• Update relevant WIKI page (See resources)
• When this issue is completed, release the dependency on TWE: Research Ethics Guide: How to Conduct Interviews: Draft #495

Resources/Instructions

• Guide: How to Conduct Interviews Draft TWE: Research Ethics Guide: How to Conduct Interviews: Draft #495
• https://github.com/hackforla/internship/wiki/UX-Research-Guides-Templates
• https://github.com/hackforla/internship/wiki/Design-Process-Overview
• TWE: Template: Research Form

[pandanista]

• For a bit more context, this feedback form will be used to provide feedback to fellow researchers on the team after their interview sessions to help them improve their interview skills, so it is a similar to a peer review format.

@e-khlystova:

• Will you take a stab and draft the questions for the feedback form in a new comment below?
• Please only focus on the first three action items for now, and we can go over them at the next meeting.

Thank you.

[e-khlystova]

Here is my draft of the questions for the feedback form:

Interviewer Feedback Form (Post-Interview):

• On a scale of 1-10 (with 1 being poor and 10 being excellent), how would you rate the quality of this interview?
• Please provide a brief memo about this interview, including any points that may be useful for future interviewers and researchers to better understand the participant.
  • This can include information about an individual's interactions with other participants (e.g., “Male participant was very argumentative with other participants") or a participant's level of comfort with particular questions (e.g., “Participant seemed hesitant to discuss…..”)
• Please provide any other comments that you may have.

@pandanista Ready for review with research lead!

[e-khlystova]

@pandanista Any chance you've had a chance to look over this? :)

[e-khlystova]

Here is another draft of the Post-Interview questions:

Interviewer Feedback Form (Post-Interview):

• How would you rate the quality of this interview on a scale of 1-5, with 1 being "low" and 5 being "high"?

Questions about interviewee

• How would rate the engagement/responsiveness level of the interviewee, on a scale of 1-5?
  • Explain, if needed.
• How would you rate the comfort level of the interviewee, on a scale of 1-5?
  • Explain, if needed.

Questions about interviewer

• How would you rate your comfort level for this interview, on a scale of 1-5?
  • Explain, if needed.
• How would you rate the quality of your interview questions, on a scale of 1-5?
  • Is there anything about the questions you would change for future interviews?

General memos

• Please leave any comments or memos about this participant or interview that could be helpful for later analysis of this interview (e.g., "participant made homophobic remarks", "male participant x consistently spoke over other participants in the focus group, who were female").

@pandanista Is this more in line with the feedback form you were thinking of?

[pandanista]

• The feedback form is for interviewer to get feedback from interviewee, observers and note takers, so the questions for interviewee won't apply here. Please ignore my previous interpretation.
• This form is to collect interviewers' experience, so it could be anything that caught their attention during the interview sessions, such as a comment made by the interviewee, or an observation the interviewer had when the interviewee seemed to have trouble answering. It serves the purpose of collecting interviewers' thoughts and feeling retrospectively.
• Comfort level seems a bit vague. What does it entail?
• Is there any example you can find on rating an interview quality or in the Guide: How to Conduct Interviews?
  • I know we discussed that at the UXR meeting, but wasn't sure if you were able to do that yet. We need to decide what aspects need to be included when it comes to rating an interview.

[e-khlystova]

@pandanista Based on the description in the original issue, it seems like this form is for the intervieweR, not intervieweE:

"We need to create an interviewer feedback form template so that interviewers can submit information about the quality of interviews they have performed after each research plan."

Can you confirm this form is for the interviewee and update the issue above if so?

My original draft was based on examples from the Guide. I have read the relevant sections, and to me, this form seems to be aimed at allowing researchers who have conducted an interview to provide additional information about the interview they performed, to note any additional aspects that may affect the quality or responses the interviewee had given. For example, if an interviewee makes a racist remark during the interview, this would help contextualize any other responses they have given that may otherwise seem like outliers in the dataset. Does my interpretation make sense?

[pandanista]

@e-khlystova You are right on that. I corrected my comment above. Will re-assign you to this issue once you are done with the current one.

[pandanista]

@e-khlystova I am assigning you back to this issue. Would you please update the draft questions? Thank you.

• You were on the right track that this form would be to collect interviewers' stories.
• This comment is a good start.
  • Does engagement and responsiveness mean the same thing here? If not, we should separate them into two questions.
  • The other aspects I could think of were interviewees' willingness to share, usefulness of the conversation (this might be part of the interview quality), and authenticity. But it might be difficult to measure those. Let me know your thoughts.

[pandanista]

@e-khlystova: Hope you had a good summer break. Would you please provide an update?

1. Progress: "What is the current status of your project? What have you completed and what is left to do?" - we know you just came back from break, please just summarize the progress from before then
2. Blockers: "Difficulties or errors encountered."
3. Availability: "How much time will you have this week to work on this issue?"
4. ETA: "When do you expect this issue to be completed?"
5. Pictures or links* (if necessary): "Add any pictures or links that will help illustrate what you are working on."

• remember to add links to the top of the issue if they are going to be needed again.