Wednesday 10 February 2010

Paula Rothermel's research

I think that it is time once again to take a close look at the work of Paula Rothermel. In practically every book, website, newspaper article or debate on home education in this country, sooner or later the research which Paula Rothermel carried out will be cited. It is impossible to overstate the significance of this work. Of course the author of the work cannot be blamed if others exaggerate or distort her findings. For this reason, I propose to limit myself to examining what she herself has to say about her own work. In Home Education by Deborah Durbin, (Hodder Headline, 2009), Paula Rothermel contributed a section about her findings. All that follows will be taken from this account.

The piece begins on page 26 with what one might term the standard story of the thousand responses to the questionnaires which she received and the four hundred and nineteen of these which were examined in greater detail. Dr. Rothermel goes on to say;

"This study remains the largest and most important of its kind, including as it did, a very broad section of the home-educating community. Access is always a problem with home education and my original survey was distributed through a number of home-educating organisations as well as religious groups, Local Authorities and Internet sites."

This is, whether by accident or design, more than a little misleading. Figure 4.1, to be found in Chapter 4 of Dr. Rothermel's thesis, reveals that in fact five thousand, three hundred and eighteen questionnaires were distributed. Of these, five thousand were sent to Education Otherwise. "A very broad section of the home-educating community" and "a number of home-educating organisations" does not really give the impression that 95% of the questionnaires were sent to just one organisation.

There follows an overview of the study. This talks about what the families valued, socialisation and so on. It is understood that this information comes from the four hundred and nineteen families whose responses were examined in detail. Then, without breaking step, Rothermel segues smoothly into the statement;

"Children aged between four and five years old were tested twice over a 'school' year"

Nothing is said to indicate that we are no longer talking about the four hundred and nineteen families. Any reader would naturally assume that this was the case. In fact, Dr. Rothermel is now talking about a mere thirty five assessments. (Chapter 6, 5.1) Without reading the original thesis, it is impossible to know this.

Later on, Rothermel mentions the literacy assessments which were conducted. These results are given as, for example, 94% of six year olds scoring in the top band. This is indeed impressive, until you learn that only seventeen six year olds are involved here; a very small sample indeed. (Chapter 4, Table 4.1) Even more interestingly, these tests were not carried out by the researcher herself; they were posted to the parents. This does not of course invalidate the results, but it is certainly the sort of thing one would wish to know about when evaluating the scores. I have mentioned in previous posts some possible disadvantages of such a method.

On page 31 of Durbin's book, Paula Rothermel talks of "The social and psychological data" We read that;

"The purpose of these assessments was to establish whether home-educated children experienced social or behavioural problems"

There were one hundred and three participants in this part of the research and three standard instruments were used to gather data, including the Children's Assertiveness Scale. In Chapter 4, 8.3 of the thesis, we learn that only four of these assessments were conducted in the presence of the researcher. The rest were posted out to families and done by the parents and children themselves. At this point, anybody who has actually had any dealings with real human beings will probably see a problem.

Parents are in general the worst possible people to be expected to make objective judgements about their own children's behaviour! I am sure that we have all met absolutely vile children whose behaviour is deplorable, yet the parents dote on them and apparently cannot see how badly their kid is acting. We are all a bit like this. I certainly have never been the least bit objective about my own children; their behaviour has always been perfect in my eyes! I think we can safely take the social and psychological data with a large pinch of salt.

As a vignette of British home education in the late nineteen nineties, this thesis is fascinating. It is when one treats it as a piece of serious academic research that the problems begin to emerge. The idea that to look too closely at a piece of research associated with a major university should render one liable to legal action is, quite frankly, preposterous. Durham University evidently know nothing about this and the person to whom I spoke there was a little alarmed to hear that they were about to become embroiled in such an affair. The correct way to deal with criticism of this sort is by calm and reasoned rebuttal, but I have a suspicion that this is not the approach that the author favours.

2 comments:

  1. There's a well-established protocol for critiquing academic work. You submit your critique for peer review and if deemed valid by reviewers, your critique gets published. Journal editors usually welcome lively debate because it increases the impact of the journal.

    There's a reason for the protocol. If it's not adhered to, it means that the reputation of the researcher being critiqued can be trashed by people who might be completely ignorant of the issues in question, but might happen to have a vested interest in rubbishing valid research and the resources to do so. I am not suggesting this is so in your case, Simon, just that those are the reasons for the protocol being in place.

    Whether Paula Rothermel should have responded to your comments about her work in the way she did is a moot point. But if you are that worried about the impact of her work, I would recommend writing a paper and submitting it for publication. Home education research is now attracting funding, so I'm sure educational journals will be interested and you would be able to disseminate your views to the most relevant readership.

    ReplyDelete
  2. This is not really meant to be a critique of an academic work suzyg, the procedure for which is familiar to me. Rather, it is an attempt to draw attention to a few points not immediately apparent to those unfamiliar with the source material.

    This research has, as I have said, been quoted repeatedly by those who are favourably impressed with the conclusions drawn in it. It has been in the public domain for over a decade and people have offered their opinions ufreely upon its merits. I am simply introducing a note of caution and suggesting that those interested in the work examine in detail what was actually undertaken.

    That mentioning the work with anything other than unqualified approval should attract the threat of litigation from the author is, to say the least of it, unusual. I think that you realise this suzieg. That such a threat should be made, combined with the withdrawal of the original thesis from the website, suggests to me that it is worth looking a good deal closer into this matter.

    ReplyDelete