Question

Understand the validity and reliability concerns for experimental research, survey research, and ethnography?

Answer 1

Reliability and Validity

Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways. Unfortunately, being consistent in measurement does not necessarily mean that you have measured something correctly. This is where validity comes into play. Validity refers to the extent to which a given instrument or tool accurately measures what it’s supposed to measure. While any valid measure is by necessity reliable, the reverse is not necessarily true. Researchers strive to use instruments that are both highly reliable and valid.

It can be hard to understand the difference between reliability and validity without looking at some examples. Let's take a look at a few together. Say you're in the market for a new bathroom scale. It's important to you that the scale is both reliable and valid. What do these mean when we're talking about a scale?

The scale is reliable if you weigh yourself several times in a row and it gives you the same weight every time. You wouldn't want a scale that gave you a certain result, but when you weighed yourself again immediately, it gave you a result that was different by several pounds.

The scale is valid if it accurately measures your weight. This is important also, especially if you're trying to use the scale to track your weight over time.

As you can see, it's important that your new scale be both reliable (it gives the same reading if you weigh yourself several times in a row) and valid (it gives you an accurate measurement of your weight). Let's look at another example.

Assume that you're a highschool teacher teaching Psychology, and you're developing a test for your students. What does it mean for the test to be reliable and valid?

The test is reliable if it gives the same result consistently - for example, if you give the test twice to the same group of students, the scores should be very similar.

The test is valid if it measures what it claims to measure. For example, if the test is supposed to be over the entire Psychology course, but it only has questions from the last two weeks, it would not be a valid test.

Reliability and Validity in experimental research:

Reliability:

Repeated use of the measure with identical subjects yields identical and consistent results. It is improved by:

Clear conceptualization

Precise measurement

Multiple indicators

Pilot-testing

Validity:

Internal Validity– design and measurement concerns that reduces chances for internal errors.

Specifically, measurement validity

Measures are valid for a single purpose

Three types of validity:

Face—as judged by others or by logic

Content—captures the entire meaning of the experience

Criterion—agrees with a validates, reliable external source:

Concurrent, agrees with a preexisting measure

Predictive, agrees with a future behavior or outcome

External Validity– describes our ability and intent to generalize to subjects beyond our study sample. Largely an issue of design and sampling.

The degree to which the findings of the study are valid for subjects outside the present study. The degree to which they are generalizable.

Unbiased, complex sampling procedures; many studies, mid-constraint approaches help strengthen external validity.

< external = > internal

The Reliability and Validity of Survey Research:

Validity and reliability in surveys in often assumed, but a lot of work is required to achieve bothThere are many things to consider if we want to write surveys that gather high quality data, including data collection method, respondent effort requested, question wording, order, format, structure, visual layout behaviors to be measured, accuracy of the elicited information, among others. Although all these issues are important, at the end of the day, what we want is to create surveys that yield results that are valid and reliable.

Validity and reliability are often discussed in the field of psychometrics, but not so much in market research, although it is assumed they are present. Validity is concerned with the accuracy of our measurement, and it is often discussed in the context of sample representativeness. However, validity is also affected by survey design since it also depends on asking questions that measure what we are supposed to be measuring.

Most surveys often have what is called face validity, which is a matter of appearances. The questions seem like a reasonable way to obtain the information we are looking for, but are they really? There are other types of validity survey writers should strive for:

Content validity: This is related to our ability to create questions that reflect the issue we are researching and make sure that key related subjects are not excluded. For example, if we are interested in learning how consumers use hair styling products, and only ask how they have used them in the past week, we are likely to miss information about how these products are used under different weather conditions (given that humidity can give you a bad hair day in a blink of an eye) and end up with an incomplete picture of consumers’ behavior in this category.

Internal validity: This asks whether the questions we pose can really explain the outcome we want to research. In our hair styling product example, we need to ask questions that help us identify factors that influence the selection of hair styling products. Here we are looking for a relationship between independent variables (e.g. hair type, desired hair style etc.) and the dependent variable (e.g. likelihood to buy the hair styling products).

External validity: This refers to the extend in which the results can be generalized to the target population the survey sample is representing. As we all know, the way we ask questions will determine the answer we get, so the questions should reflect how the target population talks and think about the issue under research, which often call for the need to conduct exploratory qualitative research. In our example, if we want to estimate the share of preference our hair styling product would get in the hair styling category, we need to include other brands that represent this category, otherwise we can’t extrapolate the results to the category as a whole.

Reliability, on the other hand, is concerned with the consistency of our measurement, that’s the degree to which the questions used in a survey elicit the same type of information each time they are used under the same conditions. This is particularly important in satisfaction and brand tracking studies, as changes in question wording and structure are likely to elicit different responses.

Reliability is also related to internal consistency, which refers to the degree different questions or statements measure the same characteristic. A practical application of this concept can be found in market segmentation studies that try to capture psychographics and construct behavioral or satisfaction segments by asking respondent to rate a list of statements using different rating scales (e.g. agreement/disagreement; likes/dislikes; excellent/poor, etc.). In our example, if we want to identify “lovers of styling products,” the statements used to describe such consumers should provide a consistent description of this group. This can be tested by using correlations, split sample comparisons or methods such as Cronbach’s Alpha.

Validity and reliability are not always aligned. Reliability is needed, but not sufficient to establish validity. We can get high reliability and low validity. This would happen when the wrong questions are asked over and over again, consistently yielding bad information. Also, if the results show large variation, they may be valid, but not reliable. So, don’t forget to think about reliability and validity when writing your next survey and strive for reliable and valid results.

Reliability and Validity of ethnography:

Although problems of reliability and validity have been explored thoroughly by experimenters and other quantitative researchers, their treatment by ethnographers has been sporadic and haphazard. Issues of reliability and validity in ethnographic design are compared to their counterparts in experimental design. Threats to the credibility of ethnographic research are summarized and categorized from field study methodology. Strategies intended to enhance credibility are incorporated throughout the investigative process: study design, data collection, data analysis, and presentation of findings. Common approaches to resolving various categories of contamination are illustrated from the current literature in educational ethnography.