I realize now that equating or closely associating Quant with WHATs and Qual with WHYs is not a very useful way to dig deeper into the issue (thanks to all of your excellent input!); but going back to what was originally bothering me, I think there is so much good WHAT in the Quant stuff that is available to us today that it’s a shame that design folk don’t go there first – for fear of numbers, not being aware or just not considering the richness that can be found in this kind of data.

Carrie: Oh wouldn’t it be nice? That is not the reality I see. That’s why I started looking at this from “what is efficient” and hopefully what could yield most value upfront.

Jared: I implied no such a thing sir! I am certain a lot of metrics are NOT going to be helpful. My point was simply that they may be overlooked entirely. And that is a shame. Now, if people did look at it, there would certainly be the problem of UNDERSTANDING it and being able to differentiate the stuff that is useful and the stuff that is not (as your examples clearly illustrate). I would say though, these metrics are still answering some WHAT questions. WHY you are using those metrics are indeed a separate problem – good catch!

Take conversion rate. It’s often used to measure an overall site’s e-commerce performance. Unfortunately, because it’s a ratio of purchases to visitors, that’s a stupid way to measure performance, you can increase conversion by decreasing visitors. The business doesn’t benefit, but the metric improves.

However, when looking at specific marketing campaigns, say ads on particular sites, conversion rate makes a lot of sense, since the ratio normalizes the traffic abnormalities.

So, I’m not sure that you can separate the WHATs and the WHYs. I think good methodology, as Sam points out, is both.

Jared

Design research ought to help in assessing, by means of reflecting on our own premises and subjective understanding of the design problem, Which matter most. And of using data to surface gaps in our knowledge as well as to verify and validate the relevance of the Whys Which we have chosen to address.

The reason quant/qual is always a squirmy problem is that it has two worlds — the subjective world of design and process of designing for solutions to subjectively defined problems. And the objective world in which object problems, processes, and solutions exist.

Resolving the quant/qual matter then is about choosing the Which Whats and Whys that matter. Of constraining the problem area to a specific set of definable design goals against which success can be measured.

So it seems to me — thanks for bringing this up. I struggle with this all the time in social, where our notions of what users like are based entirely it seems on what they’re doing, which is of course limited to what they can do. So Which one matters is a big issue for me — and should be used to reflect on our own assumptions as much as to guide us in what we see happening in the world of user/consumer behaviors.

I was definitely focusing more on the lack of attention to existing WHATs, but I certainly agree that ignoring previous WHYs is also common occurrence. I guess my main hypothesis here is: if all things are equal (which they are not) and designers have the opportunity to start whatever they are investigating by looking at existing WHATs or existing WHYs, they will absolutely go to the WHYs first and not the WHATs. (yeah, most times they are just not going to either, but you get my point).

In short, yes, it’s not about quant or qual necessarily, but whats and whys. And as you put it, it’s not about research methods but methodology. Hmmm.. how do we tackle that problem???

Thank you both for helping me untangle the issue!

You are right; design researchers often reinvent the wheel when embarking on a research project. They repeatedly, incorrigibly ignore mountains of previous insight gathered on their topics. They fail completely in covering the basic findings already published — sometimes for decades! — and instead start from scratch. Again.

Now does that mean they ignore the WHAT? Yes, but they also often ignore the WHY. If there are Web analytics, they may ignore those data. If there are previous ethnographic studies on, say, mobile use among teenagers in public spaces, they ignore that too. Both the WHAT of how many mobile devices ping the current site and WHY teenagers use mobiles differently with their parents than their friends — both of these already established answers are ignored.

Interaction designers may ignore the WHAT more readily than the WHY because numbers are icky. But I would argue that they ignore WHYs found by other researchers as well.

Why does this “start from scratch” phenomenon happen? I believe it because interaction designers are not taught how to do research. Sure, they’re taught a bunch of research methods — usability studies, ethnographic interviews — but they do not take “methodology,” which is the study of methods.

Social scientists must take these courses. They must learn where a particular method comes from, what assumptions it carries with it, and when it is appropriately used. This includes the dreaded “literature review.”
And btw, yes, teenagers use their mobiles differently with their parents, and how do I know that? Because I read a paper about it.

Steve’s point is spot on as well — I think many people dismiss research that isn’t EXACTLY FOCUSED on, say, belly bras (whether it’s qual or quant). Methodology teaches you to find relevance in all sorts of data that may be only tangentially related to your topic. So perhaps you found some metrics on number of bras sold before, during, and after pregnancy. Then you found some qual research that found pregnant women “feel ugly” in most pregnancy bras. Voila! A triangulated insight!

I am saying this in part because not every designer is well versed in all research methods. The most common methods rise to the top and quickly become the go-to tools that people use over and over. These are *generally* research instruments that yield qualitative data. I do think some early quant data can help regardless of the problem to be solve, if nothing else to help scope it.

The question of whether it makes sense to tackle ‘what’ before ‘why’ is an interesting one, but I wonder whether it is dependent on the nature of the design problem. It strikes me that quantitative methods as a starting point are really only useful when you have something concrete to measure beforehand.

This is an issue I touched on last year in a presentation to a group of design students. There is a class of problem wherein identifying the ‘what’ and then the ‘why’ makes a great deal of sense. But there is another class of problem when you need to be asking ‘why not?’ instead, and a metrics-driven approach doesn’t lend itself to this mode of thought.

Cheers
Steve
(lover of numbers)