Bad metrics, no biscuit!
Jan. 11th, 2006 08:39 am![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
featherynscaleI think I just took the worst written opinion poll ever created in the history of polling. Now, to back this up, I studied survey and questionnaire design in college as a psych major. Now, I never did do any graduate studies, and I was perhaps not the best of students, but I do like to believe that I know something about data-gathering and creating this sort of instrument. And I would also like to think that the major polling organizations would know something about it, too. And yet.
I got a Zogby poll by email this morning. One part was pretty standard marketing stuff, and they should have stopped there. The next thing that came up was "We would like to ask you some questions about your attitudes toward relationships", and I went, "Oh, good." So they ask your gender (Male, Female, Refuse to Say). Then, they ask you for your sexual orientation (Heterosexual, Homosexual, Bisexual, Transgender, Don't Know, Refuse to Say). Anybody starting to see what's wrong here?
It gets better. I click the button for "Bisexual". I get "Are you currently in a sexual relationship? (Yes, No, Don't Know, Refuse to Say)". I click "Yes", and am given a series of questions about my "husband/boyfriend/partner", who is always referred to as "he". At this point, I'm thinking, well, if you only want to know about women who have relationships with men, why do you ask for sexual orientation? If I'd written this, the orientation question and the relationship question would have been one question, which would have been "Are you currently in a sexual relationship with a man?". But I digress, and I haven't even gotten to the really heart-rendingly bad part yet.
At first, I got questions like "Would you ever cheat on your husband/boyfriend/partner? (Yes, No, I Have Already Cheated, Don't Know, Refuse to Say)" and "Has your husband/boyfriend/partner ever pressured you into sexual activity you didn't want to participate in? (Once, Two-Three Times, Four-Ten Times, Regularly, Never, Don't Know, Refuse to Say)". No big thing.
But then, we transition into a huge list of questions about things you might be afraid to tell your partner: "Are you afraid to tell your husband/boyfriend/partner that he is not very good in bed?" "Are you afraid to tell your husband/boyfriend/partner that you cheated on him?" "Are you afraid to tell your partner that he is not well-endowed?" and so on and so forth. And I go, "Wow!". Not "Would you be afraid to tell your partner that he was not well-endowed?" or "Would you be afraid to tell your partner that you had cheated on him if you had cheated on him?" or anything that in any way acknowledges that the statement might not be true, or that gives the respondent any way to indicate that the question is not relevant to them, or anything.
It's a little like asking someone "Have you stopped beating your wife?" or "Have you told your mother that you are a junkie?". That form sets up a confusing condition, which many people have no idea how to resolve. Am I afraid to tell my partner he's not good in bed? I don't know. If I say "No", am I essentially saying that he's not good in bed? If I say "Yes", am I saying he's not good in bed? Either answer requires a respondent to accept the premise of the question. This makes people freak out, so I'm guessing that whatever data they get from this thing is going to bear little to know resemblance to the "truth" about their respondents and the level of fear they have with respect to their partners.
Other items required equally ridiculous mental gymnastics on the part of the respondent. "Would you be happier if your husband/boyfriend/partner spent less time watching sports?", for example, without first determining if the partner does, in fact, watch sports at all. If he doesn't watch sports at all, he could hardly watch less, right? So the answer there would be "No"? Hard to say.
You would think that as much money as goes into these organizations, that they would at least be able to hire college graduates to create these instruments, but maybe not. I'm somewhat offended by the whole experience. On the other hand, maybe I should try to get a job with a polling organization. Clearly, they need me.
Dear Sir or Madam,
I'm writing to you concerning the design of a Zogby Interactive poll with an ID number of 28951. It asked me about whitening toothpaste and also about relationship issues. Speaking as a person with some knowledge of data-gathering and instrument design, and also as a person with some concern about gender identity issues, there are a few things I would like to bring up about this poll.
First, in the section about relationships with men, the poll asks first for gender, then for sexual orientation. In the sexual orientation question, there is an option for "Transgendered". While I appreciate the inclusion of this option, I believe that it is in the wrong place. To know that a person is transgender tells you about what they think about themselves, not who they are attracted to or who they might form relationships with. A person may be transgender and attracted to either men or women or transgender people or some combination of those possibilities. The transgender option seems to more logically belong in the response set to the previous question, which asked for a response of "Male" or "Female".
Second, when I indicated that I was bisexual in that item, I was taken to a set of questions which were exclusively about current relationships with a male partner. If the interest of the instrument was in women's current relationships with men, then why ask for sexual orientation? Why not replace the orientation question altogether and simply ask something like "Are you currently involved in a sexual relationship with a male partner?" I would think that this would eliminate some potential confusion relating to terms, and also clarify intent without being disrespectful to bisexual women who were currently in a relationship with a female partner.
Finally, many of the items in this section of the instrument were phrased in such a way as to create a double-bind situation for the respondent. A number of the relationship items were written using a form like: "Are you afraid to tell your husband/boyfriend/partner that he is not good in bed?" or "Are you afraid to tell your husband/boyfriend/partner that you have cheated on him?". While these items are somewhat less alarming than the textbook double-bind example of "Have you stopped beating your wife?", they do still require the respondent to assert the truth of the indicated behavior in order to give a response. In order to answer those questions, a respondent has to assert that their partner is not good in bed, and that they have cheated on their partner, respectively. If those things are not true for the respondent, they may either be hesitant to respond to the item, or may give an answer which is not accurate. Had these items been phrased as "Would you be afraid to tell your partner if he were not good in bed?" or "Would you be afraid to tell your partner if you had cheated on him?", the double-bind would be avoided and the data probably cleaner.
Thank you for looking at my comments. I do enjoy participating in these polls, and would like to continue to do so. Most of the polls I have participated in seem to be well-built. However, the issues I see in this instrument give me some concern about the quality of the data being collected. I hope that you will take these comments into consideration in future poll design.
Thank you,
XXXXXXXX XXXXXXXX
I got a Zogby poll by email this morning. One part was pretty standard marketing stuff, and they should have stopped there. The next thing that came up was "We would like to ask you some questions about your attitudes toward relationships", and I went, "Oh, good." So they ask your gender (Male, Female, Refuse to Say). Then, they ask you for your sexual orientation (Heterosexual, Homosexual, Bisexual, Transgender, Don't Know, Refuse to Say). Anybody starting to see what's wrong here?
It gets better. I click the button for "Bisexual". I get "Are you currently in a sexual relationship? (Yes, No, Don't Know, Refuse to Say)". I click "Yes", and am given a series of questions about my "husband/boyfriend/partner", who is always referred to as "he". At this point, I'm thinking, well, if you only want to know about women who have relationships with men, why do you ask for sexual orientation? If I'd written this, the orientation question and the relationship question would have been one question, which would have been "Are you currently in a sexual relationship with a man?". But I digress, and I haven't even gotten to the really heart-rendingly bad part yet.
At first, I got questions like "Would you ever cheat on your husband/boyfriend/partner? (Yes, No, I Have Already Cheated, Don't Know, Refuse to Say)" and "Has your husband/boyfriend/partner ever pressured you into sexual activity you didn't want to participate in? (Once, Two-Three Times, Four-Ten Times, Regularly, Never, Don't Know, Refuse to Say)". No big thing.
But then, we transition into a huge list of questions about things you might be afraid to tell your partner: "Are you afraid to tell your husband/boyfriend/partner that he is not very good in bed?" "Are you afraid to tell your husband/boyfriend/partner that you cheated on him?" "Are you afraid to tell your partner that he is not well-endowed?" and so on and so forth. And I go, "Wow!". Not "Would you be afraid to tell your partner that he was not well-endowed?" or "Would you be afraid to tell your partner that you had cheated on him if you had cheated on him?" or anything that in any way acknowledges that the statement might not be true, or that gives the respondent any way to indicate that the question is not relevant to them, or anything.
It's a little like asking someone "Have you stopped beating your wife?" or "Have you told your mother that you are a junkie?". That form sets up a confusing condition, which many people have no idea how to resolve. Am I afraid to tell my partner he's not good in bed? I don't know. If I say "No", am I essentially saying that he's not good in bed? If I say "Yes", am I saying he's not good in bed? Either answer requires a respondent to accept the premise of the question. This makes people freak out, so I'm guessing that whatever data they get from this thing is going to bear little to know resemblance to the "truth" about their respondents and the level of fear they have with respect to their partners.
Other items required equally ridiculous mental gymnastics on the part of the respondent. "Would you be happier if your husband/boyfriend/partner spent less time watching sports?", for example, without first determining if the partner does, in fact, watch sports at all. If he doesn't watch sports at all, he could hardly watch less, right? So the answer there would be "No"? Hard to say.
You would think that as much money as goes into these organizations, that they would at least be able to hire college graduates to create these instruments, but maybe not. I'm somewhat offended by the whole experience. On the other hand, maybe I should try to get a job with a polling organization. Clearly, they need me.
Dear Sir or Madam,
I'm writing to you concerning the design of a Zogby Interactive poll with an ID number of 28951. It asked me about whitening toothpaste and also about relationship issues. Speaking as a person with some knowledge of data-gathering and instrument design, and also as a person with some concern about gender identity issues, there are a few things I would like to bring up about this poll.
First, in the section about relationships with men, the poll asks first for gender, then for sexual orientation. In the sexual orientation question, there is an option for "Transgendered". While I appreciate the inclusion of this option, I believe that it is in the wrong place. To know that a person is transgender tells you about what they think about themselves, not who they are attracted to or who they might form relationships with. A person may be transgender and attracted to either men or women or transgender people or some combination of those possibilities. The transgender option seems to more logically belong in the response set to the previous question, which asked for a response of "Male" or "Female".
Second, when I indicated that I was bisexual in that item, I was taken to a set of questions which were exclusively about current relationships with a male partner. If the interest of the instrument was in women's current relationships with men, then why ask for sexual orientation? Why not replace the orientation question altogether and simply ask something like "Are you currently involved in a sexual relationship with a male partner?" I would think that this would eliminate some potential confusion relating to terms, and also clarify intent without being disrespectful to bisexual women who were currently in a relationship with a female partner.
Finally, many of the items in this section of the instrument were phrased in such a way as to create a double-bind situation for the respondent. A number of the relationship items were written using a form like: "Are you afraid to tell your husband/boyfriend/partner that he is not good in bed?" or "Are you afraid to tell your husband/boyfriend/partner that you have cheated on him?". While these items are somewhat less alarming than the textbook double-bind example of "Have you stopped beating your wife?", they do still require the respondent to assert the truth of the indicated behavior in order to give a response. In order to answer those questions, a respondent has to assert that their partner is not good in bed, and that they have cheated on their partner, respectively. If those things are not true for the respondent, they may either be hesitant to respond to the item, or may give an answer which is not accurate. Had these items been phrased as "Would you be afraid to tell your partner if he were not good in bed?" or "Would you be afraid to tell your partner if you had cheated on him?", the double-bind would be avoided and the data probably cleaner.
Thank you for looking at my comments. I do enjoy participating in these polls, and would like to continue to do so. Most of the polls I have participated in seem to be well-built. However, the issues I see in this instrument give me some concern about the quality of the data being collected. I hope that you will take these comments into consideration in future poll design.
Thank you,
XXXXXXXX XXXXXXXX
