[Tfug] user group survey

[David Cowell]

Feh! Surveys are almost always poorly designed, unless they are
*extremely* simple. And even then it's easy to get it wrong.

In the survey that Mike's talking about:
<http://www.informit.com/promotions/promotion.aspx?promo=136703>

1) First question: "Do you use some form of social media on a weekly
basis?" Aw, fer chrissakes! What the heck is "social media"?

If a buzzword is not in a 1999 Merriam-Webster, it is not rude to give a
simple explanation of what is meant: Does "social media" (as the
surveyor understands the phrase) include blogging? email? Facebook?
television? YouTube? "People" magazine? telephone? BBSing? personal
webpage? All these can be considered "social media" by a large segment
of our culture... so the term should be defined. But the surveyor
thought we understood him/her clearly.

2) "Rank these sites" involves the assumption that the respondent has
experience with all of them. Otherwise, wouldn't it be wise to include a
"N/A" column?

3) Often, we have similar reactions to many things.

In a list of 10 items, a person may greatly prefer two items (10.
untrammeled lust and 9. rock `n' roll), greatly dislike two items (1.
jail and 2. taxes), and not be horribly opinionated about the other (Big
Macs, Whoppers, milkshakes, Maria's taco salad, everyone else's taco
salad, Milk Duds). How does a respondent rank this middle stuff? Or even
the extremes? (There are times when x is preferable to y, yet there are
times when y is truly preferable to x. [Attributed to Thoreau.])

(Even more interesting to me is how significant are the conclusions you
generate from your rankings... is untrammeled lust 10 times as much fun
as jail, are taxes 22.2% the fun of rock `n' roll, is Maria's taco salad
*really* twice (or more) as good as jail?) (<- This is a crucial
weakness in most Bayesian analyses - the reductio ad absurbum. The
reply, of course, is that we learn a "trend" or "preference"... which
then begs many other questions.)

Looking at a distribution of preferences doesn't always tell you much,
even if it fits a perfect bell curve. Almost certainly, the stuff in the
middle of the curve doesn't even deserve ranking except as "not worthy
of comment".

Life is neither binary nor hierarchical in essence. Please let this
secret out: life is *much* more multifaceted than bifaceted. And
"multifaceted" still doesn't touch it.

==

The major causes of inadequate surveys are inadequate planning and
insufficient investigation of both the participants and the subject
matter. 

I would guess that the person who gets stuck with survey design usually
is either rushed ("Hey! This shouldn't take more than 45 minutes or
so!") or inexperienced ("Terry's here as a summer intern. Let her/him do
it and get some experience."). Or is a consultant. :)

Do *you* make a detailed outline before you begin a project? Do you
rigorously research a problem's fundamentals before you start working on
it? Or do you prefer "Worse is better"? Believe it or not, the world to
a large degree imitates you no matter how you answer these questions.

(Worse is better is a stronger method only within a system that is
capable of rapid adaptation. A one-time-only survey is *not* appropriate
for WIB... masterwork should be done by masters, not apprentices;
battlehardened soldiers are the preferable salients.)*

What I find most amazing, though, is that somehow we feel comforted and
supported in our opinions because some investigator has come up with a
parallel conclusion to ours. This is in spite of our knowledge that we
often do an inadequate job ourselves (so why shouldn't they?).

*This is one of the wonders of our time, that experience to solve many
of our deepest problems is both under-available and under-valued... and
yet we bumble along. We will eventually bungle into something truly bad
if we continue with WIB as our preferred method of solving grave
problems. See the two-month rush into World War I as an example of WIB
at its worst. 


On Mon, 2008-09-08 at 22:22 -0700, Michael Omar Gatto wrote:
> So, I took the survey, but the big array of checkboxes for rating
> social media did this:
> 1.) would not let me select 10 for more than one row
> 2.) would let me, however, select more than one checkbox in the same
> row
>  
> After submitted, it complained about improper data. Soooo, I hit my
> back button and lo and behold all the filled in form data was gone,
> gone, gone. So I clicked that pretty little 'x' in the window bar and
> said "see yah!" 
>  
> Thier developer should be forced to have to use his/her/thier own form
> dozens of times with no successful submissions, and then scrub the
> facility's toilets for a day or two...
>  
> - Mike Gatto