One Old-Schooler’s View of Social Media Sampling in Marketing Research?

“Randomization is the key element in any marketing research enterprise aimed at making projections about a population based upon a sample of that population.”

To many of us who grew up in the world of “traditional” marketing research, this statement is the foundation of our work.  However, a review of recently published marketing research articles – mainly web-based sources – turns-up a number of instances in which research groups are employing sampling strategies which depend upon non-random, social media sources.  More amazing is the claim by some futurists/soothsayers that before we know it, traditional surveying will be replaced by social media.

Simply put, a randomized sample is one in which each member of the universe being studied has an equal chance of being selected for inclusion in the research sample.  It involves both the random selection of potential respondents and the random assignment of those respondents to treatment groups (e.g., pre/post promotional campaigns).

In our day-to-day operations, there are certainly many factors which continually mitigate against our generating a truly random sample. I will address these issues in a coming installment.  However, in this initial blog post I will center my attention on the principles involved in employing random samples versus using social media sampling techniques.

Research samples derived from social media channels (river sampling, samples of convenience, web panels, etc) most often violate the basic assumption of randomness -- each person in the segment/groups under investigation has an equal chance of being sampled.  Therefore, scientifically speaking, they cannot be considered quantitative research projects. Instead, they are essentially qualitative research investigations with findings, analysis, and insights that cannot be reliably projected to the universes under examination. 

The reader might ask, "Why does all of the fuss about randomness matter, anyway?"  Simply put, the results of these studies cannot be reliably applied to any real world situations.  The IT team says it best, “Junk in, junk out.”

Example -- Salt in the Soup

Here's an oldie but goodie. Think about the issues involved in making soup this holiday season. Specifically, Marmee (Gammer, Grammy, Grandmother, Granny, Ma-maw, Nana, Na-naw) wants to see how her cream of broccoli soup will taste after adding some sea salt to the mixture.  Will she taste the whole pot?  Of course not.  Will she consume an entire cup?  Again, probably not.  Marmee will most likely decide if she has added the right amount of salt by tasting a teaspoon full.

Interestingly, Marmee will be following the "scientific method" when making decisions on the effect of adding ingredients to her cream of broccoli soup:  select, add, stir, sample, and taste.

1. Select the ingredient you want to add – sea salt
2. Add the sea salt to the soup mixture
3. Stir the soup thoroughly - to insure that the salt is evenly distributed throughout the soup
4. Sample a small, but representative portion of the soup
5. Taste and analyze the results

Social media sampling is like adding the salt without stirring the mixture.  The result?  If you sample of a part of the soup that has no salt – which is easy to do without stirring, you will make an erroneous judgment about the salt's effect. I.e., "Not enough salt."  Conversely, if you sample the soup from the area where the salt is concentrated (remember before stirring), you will come to a completely different – and equally inaccurate – conclusion, "Too much salt."  So, the analysis of the salt's effect does not depend upon the amount of salt that was added.  Rather it is dependent on the randomness of the sample evaluated. 

If you prefer a different analogy, try applying the same concept to adding color tints to house paint and not shaking the can thoroughly before deciding if the new color matches what you want -- better when used in reference to river sampling.

Most likely, enterprising marketing research professionals will develop workarounds.  Areas for consideration include:

1. Sample selection -- generating a pool of potential respondents via social media sources and randomly selecting potential respondents from that pool.
2.  Analytical schemes -- developing more strict statistical tests which will account for the error inherent in the use of non-random sample.
3. Weighting -- we can always weight the data by the known distribution of key segments. However, all statistical analysis must be based upon the original distributions.

Summary – "That’s My Time"

Randomization is a necessary condition!  Marketing research studies which depend upon social media sources may provide worthwhile insight; but, the results simply cannot be projected to any specific segment – consumer or business.  And, isn’t that the whole point? 

-- Charles L. Montgomery, Ph.D.