Friday, December 7, 2007

The results are coming in! (Reading Research)

Many of you were kind enough to help me last month by answering a reading survey I conducted - thank you again. I promised some results. Here they are. You can scroll past the whole intro if you wish and get the cut to the chase version.

It's a tricky thing reporting results from research, especially when conducted by asking people about themselves. You may have noticed that when I write about articles from the Science Times, I critique the experiment. That' s not because I imagine I'm smarter than the creator of the study, it's because no study can be perfect. No study can ask every question from every angel, it would be too long. Some studies inquire not about finite knowledge but about constructs, as in the case of my survey - the construct was susceptibility to the experience created by interaction with a fictional narrative. In that case another researcher could define that construct differently and ask different questions. If a study asks only similar questions about the experience of reading fiction - let's say it only looks at internal experiences like being made to feel sad or aroused - one could then criticize it for omitting behaviors that are more confirmable - like a quickly beating heart or turning on more lights in your home. If a survey includes a wider variety of questions - some about how often they have these internal and external experiences, and others about the degree of that experience - e.g. a question about reading some thing suspenseful or scary had choices ranging from - "It's fun, but I don't literally care..." to "Gets me involved so that I am tense/sweaty, etc..." The survey designer must decide whats more useful - focus or breadth. If breadth - do the different types of questions receive the same number of points each? Is doing something the most frequently numerically equivalent to doing something the most intensely? In any event, designing a survey is fraught with questions, it's not as simple as saying, Gee, I think I'll ask a bunch of people about chewing gum today...

The second tricky part is interpreting the results. What does a survey that asks people to report on their own behaviors actually measure? Articles will tell you it means that they felt the feeling 78% of the time or 15% feel it very deeply, but that's not accurate. What they can say with certainty is that respondents reported feeling the feeling 78% of the time. We don't know for certain that because someone said they had an experience that they did and that they all did so in the same way. Or that one person's most intense degree of feeling is equivalent to another person's. It's not simply a matter of whether a respondent is being truthful. Some people's answers are weighted by what they think they should answer, others by what they think the experimenter would like to hear, some are highly sensitive judges of their own experience, others are relatively insensitive. It's why I frequently talk about the narrative of science, because when we report results from studies we tell a story about them to give them meaning in the context of what we wish to know. With those provisos you should know that there are tools that help create strict standards for saying we are certain about something and the degree of that certainty and those are the dreaded statistics - the class that is the bane of every social science student. I hear the collective groan my 101 students used to give me every time we discussed not the fun concepts but the tests used to assess them. I'll leave out all the complicated stuff here but know that I did use them in order to try to keep my interpretations honest.

My question:
Do reader's vary in their susceptibility to involvement in the experience created by interaction with a fictional narrative? If so, under what circumstances?

I am trying to develop a way to measure that susceptibility. The survey some of you took was my first draft. It was buried among other questions about demographics, other reading behavior, and another already existing survey about openness to experiences in general.


92 of you of whom I could use responses from 88 (20 men/68 women)
Ages: 18 - 63 (men, mean age 41.4/women, mean age 33.6)

From where:
Western Europe, South America, Australia, and Asia - 13%
Canada - 9 %
U.S. - 78%

High School - 6%
Some College - 18%
College - 31%
Some Graduate School - 10%
Graduate - 35%

Number of languages you read in:
1 - 69%
2 - 26%
3 - 4%
4 + - 1%

Reading Frequency:
fewer than 1 book/month - 16%
2-3 books/month -34%
1 book/week -15%
2-4 books/week - 29%
4+ books/week - 6%

What is great about the reading frequency numbers is that there is a good distribution among frequencies, i.e. the survey has no choice but to question people who read, but the results are not skewed towards only those who read a lot either by virtue of speed, free time, or enthusiasm.

I also computed a score I called the "reading affinity score," which was culled from various questions regarding reading frequency, self-identification as a "reader" or "book person," whether participants have a literary blog or belong to a book club. Those scores ranged from 1 to 9 and were almost normally distributed, which means that most people were in the middle of this range with fewer people at the upper and lower ends.

Reading Affinity:
1 - 1%
2 - 3%
3 - 12%
4 - 26%
5 - 9%
6 - 22%
7 - 17%
8 - 9%
9 - 1%

There were also questions about reasons for reading which are complex, but to summarize, way more of you read for reasons like pleasure, education, escape and comfort, than read for obligation.

Some of the patterns:
The possibilities are endless. I'm going to give you a few patterns that I noticed, but if you are curious to know how certain other factors related to each other (e.g. did women read more than men? or Did having completed more schooling effect how susceptible you were to your reading experience?) ask me, and if I can figure it out from the information I collected I will tell you what I observe.

First I tested the reliability of my survey. I found that it seemed to test a unitary construct rather than several distinct constructs when tested with two different measures. For those of you for whom it has any meaning for coefficient alpha r = .85 and for split-half reliability, applying the Spearman Brown correction r = .87.

I tested the validity of my survey - i.e. does it measure what I say it measures - in a bunch of different ways. I compared it to another scale that measures openness in general and absorption related to imaginative involvement. My correlation was not terrific, however it was significant by statistical standards (r = .46, p < .01). I also tried some other measures that you may or may not think are meaningful. I compared the scale to a portion of another scale measuring synesthesia - a condition in which a person experiences an intermingling of two or more senses (i.e. they see specific numbers as specific colors or experience specific musical notes as having odors. By the way scientists can confirm this is actually happening, we're not taking people's word for it, nor is this describing a metaphoric experience - it is actual.). I did this correlation not because I think there should be a relationship between reading susceptibility and synesthesia but because I think there should not. This is called a divergent analysis and my suspicions were confirmed - correlation was very low and demonstrated no significance (r = .17, p = .12).

Then I compared women's and men's susceptibility scores, given that vulnerability and expression of many emotions are seen as socially more acceptable in most of the places in which our respondents reside. The difference was very apparent. Women scored significantly higher than men (F - 5.0, p < .03). Since women outnumbered men 3:1, I took a subset - 20 of each sex and matched them for age, level of education, and how frequently they read. I still saw a difference with women's scores significantly higher than men's (F = 4.3, p < .05). Does that mean that women are actually more susceptible? Not necessarily, it could mean that they are more willing to report that they are. It could also express a difference in men's and women's awareness of their experience rather than whether it was actually physically different. But scientific conclusions are best stated by observing convergences and divergences of many sorts of information versus making grand conclusions based on 1 single statistic or another. So consider this 1 piece of information in an accumulating pattern.

Then I compared susceptibility scores based on the reading affinity score I computed. Any guesses? Does one's affinity for reading (as computed by my imperfect method) effect one's susceptibility to the experience of reading (as assessed by my imperfect survey)? I'll give you some more results after I hear your predictions. Take the poll near the top of my side bar.


Dewey said...

I'm pretty sure my son filled out your survey, but he's under 18, so I assume he's one of the four you couldn't use. Was it because of his age or having the same IP address as an earlier responder (me)? Or maybe he just said he'd do it and forgot. :)

Ted said...

I did have 1 or 2 respondents under 18 and also a few I couldn't use for other reasons. The site did accept multiple responses from one IP address, but I turned off the feature that would show me IP addresses so I wouldn't be able to identify people.

Anonymous said...

What I hear you asking is whether people who like reading are more emotionally involved in their reading that a wrong way to put it? My guess is that they are, otherwise why else spend so much time reading?
I suppose you could flip that, if someone was more susceptible they might not enjoy the reading experience because they can't separate from the fictional reality.
Have I completely misunderstood your project? Feel free to set me straight. Regardless, I think this is fascinating.

Ted said...

My hypothesis is that peoples' responsiveness to reading differs and I'm curious to see if I can learn about anything that might contribute to that difference. I wondered if there had been a difference between the male and female readers who answered my survey, and they did report being differently responsive from each other. I'm curious whether you think readers responsiveness or susceptibility will differ as a function of their affinity for reading (as computed in my study). Sometimes one's suspicions are confirmed and sometimes they aren't. I had an idea about this factor and ran the numbers. Before I report on what I found, I'm wondering what you all hypothesize.

Anne Camille said...

Fascinating research! I find the questions on synthesia interesting. Given the size of the survey, though, was it possible to make the divergent conclusion? I thought synthesia was rare, but maybe I am misinformed?

I am curious also about the level of education; specifically, I was initially surprised by the number of respondents with graduate degrees. But then I started thinking about the bloggers and book lovers I know. Seems like there are many teachers, librarians, and writers in blogland. (I do not know if this was the only source of your survey). But, what I'm wondering about is this: was there a difference in responses between those who had a more 'bookish' type of degree -- whether graduate or not -- vs those who have degrees in science, or law, or business? Did you gather that kind of demographic? I'm curious how the reading affinity differs between a librarian and an engineer, for example.

Ted said...

Cam - synthesthesia is very rare. The questions asked were by no means conclusive - you couldn't make a diagnosis off of them - but there were a group of questions that were only about synthesthesia-like intermixing of senses and, as such, I felt had pretty much nothing to do with reading experience. It's a baby-divergent anlysis, the numbers in my survey preclude making generalizable conclusions about anthything impossible. But they do let us have a little fun.

I wish I had asked a question about course of study so that I could run an analysis on your question, unfortunately... I didn't. The closest I got were the questions about things like "identifying as a book-person," and things like that which I used to compute the reading affinity score.