What makes research "valid"?

While in my Ph.D. program, I learned that academics love using fancy terms to explain pretty straightforward research methods and concepts. Some of those fancy terms have wormed their way out of academia and into everyday conversation like rigor, validity, and significance. In some ways, this is a great thing – information about research methods shouldn't stay locked up behind ivory towers and paywalls. But, like any game of telephone, as these terms move from academia to foundations and nonprofit organizations, their meanings can get a little muddled.

In the next several articles, I will explain these research terms and show nonprofit organizations how to apply them to their surveys, focus groups, and other research projects. This article will cover what validity means, some common research mistakes, and steps nonprofit leaders can take to ensure validity in their organization's work.

What is validity in research?

Validity is essentially about accuracy. Ensuring validity means that the project team has taken steps to avoid making common mistakes in research design, data collection, and data analysis. It also involves transparency, meaning documenting data processes and being transparent about limitations.

Mistake #1: Choosing the wrong groups

One common data collection mistake is choosing the wrong group(s) of people to participate in a project or excluding groups that should be involved. For instance, say a nonprofit leader wants feedback from staff about an upcoming strategic planning retreat. However, the executive director only sends a feedback survey to directors, assuming that the directors can speak for the rest of the staff. That is a data collection error; the ED should have sent the survey to all the staff so everyone could have the opportunity to share their views.

Mistake #2: Choosing the wrong methods

Another common mistake is choosing the wrong tool or method for the job. Take employee wellness as an example. There are many ways to better understand employee wellness in an organization, including surveys, interviews, or focus groups. However, some methods are better than others.

For instance, if a program director decides to do one-on-one interviews with their employees about their mental state, employees may not feel comfortable telling their boss about their personal struggles. Instead, employees may withhold information and only focus on the positives. Instead, nonprofits can ask employees to fill out an anonymous survey to increase the odds of more honest answers.

Mistake #3: Forgetting to define terms

A third common mistake happens when the project team doesn’t clearly define the terms or concepts they want to study. This mistake happens all the time, especially in survey research. Again, take the employee wellness example. There are many definitions of employee wellness and the factors that contribute to it. The executive director may define wellness differently than the directors, human resources department, and program staff. Without establishing a common definition of “wellness,” the project team may end up asking irrelevant or unclear questions that could negatively impact the results.

Four steps to increasing your project's validity

1. Make sure your team thinks through your research questions and comes to a joint agreement on any terms that could be interpreted in multiple ways (e.g., “success,” “vision,” or “impact”).
2. Identify who needs to participate in the project. Participants should be the people who are best suited to answer your team’s research questions. Be exhaustive and specific, listing reasons why each group should or should not be included in a survey, interview process, or focus group.
3. Brainstorm ways to collect data and write down the pros and cons of each. Be mindful of participants’ experiences as you go through this process. What is the best way to make your participants feel comfortable and give open, honest responses?
4. Create a research project plan and document any changes made throughout the process. This plan will help your team explain what you did and how you did it when it’s time to share the results.
   

Morgia Research Services works with organizations to ensure their evaluation and assessment projects ask the right people the right questions to determine their impact. Schedule a free 45-minute consultation call about your project today!