Friday 3 August 2012

The research shows…



No real research has been undertaken on home education in this country. That being so, those who defend the practice are compelled to fall back on some pretty feeble work and hope that nobody will notice how shoddy it is. When a couple of months ago Janine Ainsworth, Chief Education Officer of the Church of England, said that home education was not a good idea, Mike Fortune-Wood was quick to leap in with what he thought was some evidence which discredited Ms Ainsworth’s view. See;

http://www.home-education.org.uk/articles/article-c-of-e-reply.pdf

As usual, Mike Fortune-Wood’s statement contained some unintentionally hilarious passages. When he said, for example, that, ‘Practically all Home educated children take an active roll in their local communities.’, I don’t think that he was really suggesting that they tumble head over heels down hills, like so many Jack and Jills! Perhaps ‘role’ might have been a more felicitous choice of words. He went on to say, 'Home educated children have been shown to be highly social, balanced and to mix well with other children and adults. (Paula Rothermel’s work).' Which brings us neatly to the subject of feeble work and shoddy research.

It is astonishing how many people still trot out Paula Rothermel’s findings, under the apparent impression that they tell us anything useful about home education in this country. Let us see what they tell us about how social and balanced home educated British children are.

Reading about Paula Rothermel’s work, one often comes across large numbers. Thousands of questionnaires sent out, over four hundred families interviewed in depth; that sort of thing. The truth is a somewhat different. In February 1997, Rothermel sent out 2500 questionnaires to members of education Otherwise. A year later, she sent out a similar number, again only to members of Education Otherwise. Two hundred were also sent to religious groups and local authorities. A thousand were returned, of which four hundred and nineteen were chosen for further research. Already, we have run into a several serious problems.

The first difficulty is that this was a self-selected group, almost entirely from one organisation. (Twenty four out of twenty five of the questionnaires were sent only to members of Education Otherwise). Only a fifth of those asked, wished to take part by answering any questions. These are likely to be those whose home education was going well and who wished to tell others of it. In addition to this, they were people with high degrees of ‘document literacy’; those probably of a higher educational level than average. This is also suggested by the fact that a quarter of them were, or had been, school teachers. The final point to consider is that this was all fifteen years ago.

To sum up, the sample group of children tested fifteen years ago were self-selected members of a single organisation, whose educational standards were probably higher than average. Still, testing over four hundred children, even from an atypical group like this might still tell us something about home educated children. We now encounter two more difficulties which render the findings from this research project pretty well useless. The tests relating to social skills and the ability to get on with others were not administered to hundreds of children; not even a hundred children, nor even fifty. In fact the crucial tests about the ability of the children to get on with others and behave in a socialised way were only given to groups of twenty children. Not only that, but the parents did the tests, answering questions about their own children.

Here is one of the three tests used by Rothermel on twenty children:



http://www.sdqinfo.com/py/doc/b3.py?language=Englishqz(UK)



Click on the one marked P4-16 to see it. Has anybody seen the problem with handing a parent a form of this sort and asking her to answer the questions about her own child? I wonder how many parents will admit that their own child often lies and cheats? Or whether the average parent is objective enough to admit that her son often fights with other children and bullies them? Parents are the worst possible people to ask about this sort of thing! We all stick up for our kids and gloss over their faults, even to ourselves. Nobody will get an objective assessment of my daughter from me and I suspect that other parents are the same. A much more reliable way of establishing this sort of thing is for teachers and nursery workers to fill out these forms based upon their observations of children.

As I say, this really important test, which has led many to make such extravagant claims about what research has shown about home educated children in this country, was given to only twenty children. The answers were all those given only by the children’s mothers or fathers. When people like Mike Fortune-Wood say things like the quotation above about research showing that home educated children are highly social, well balanced and mix well with other children, what they are really saying is this; ‘Fifteen years ago twenty parents of home educated children claimed that their children were sociable and well balanced’. That is it, the whole of the evidence for this often repeated assertion.

25 comments:

  1. Isn't all social research self selected? I suppose the census might qualify as compulsory social research, but I've not heard of anything else.

    ReplyDelete
  2. 'Isn't all social research self selected'

    Not really. Rutter famously followed the progress of every child on the Isle of Wight. Other research might involve everybody living in a smaller area, patients using a hospital and so on. If you only look at people who offer to take part, your results are skewed from the start. When, in addition to this initial bias, the sample is only twenty people; it makes it impossible to say anything worthwhile.

    ReplyDelete
  3. Rutter would probably not gain permission for such research today though, and it sounds as though at least some of the research was voluntary, relying on people to return questionnaires, so therefore self selected:

    "The Isle of Wight Study conducted by Rutter in the 1960s was unique at the time due to both the richness of data collected, that was far greater than for comparable studies, and its methods. It was the first study to involve direct assessment of children and depended on a partnership with social and medical services, which could not take place today, for example gaining direct access to their record systems...

    ...For example, for the Isle of Wight Studies and the earlier waves of the NHSD, consent to participate was not governed by any legal principles and, consequently, was assumed by virtue of the questionnaire having been returned."


    http://tinyurl.com/cmp73dv

    It may be possible for basic data can be compiled without permission (for school league tables, for instance), but social research of the type you describe above requires informed consent and voluntary participation. We've taken part in various pieces of research as a family and the voluntary nature has always been made very clear.

    "There are a number of key phrases that describe the system of ethical protections that the contemporary social and medical research establishment have created to try to protect better the rights of their research participants. The principle of voluntary participation requires that people not be coerced into participating in research. This is especially relevant where researchers had previously relied on 'captive audiences' for their subjects -- prisons, universities, and places like that. Closely related to the notion of voluntary participation is the requirement of informed consent."

    http://www.socialresearchmethods.net/kb/ethics.php

    ReplyDelete
  4. "Has anybody seen the problem with handing a parent a form of this sort and asking her to answer the questions about her own child? I wonder how many parents will admit that their own child often lies and cheats?"

    Who answers when these questions are applied to other children in other pieces of research? Hopefully not teachers since, judging by their report writing skills, they are even less likely to be accurate.

    Looking at the web site these questionnaires are designed to be completed by parents and are used by many researchers. It seems to be an accepted practice. Do you really think they haven't taken parental bias into account? These are psychology questionnaires - bias is something they are likely to be aware of and take into account when interpreting the results! The results will also be comparable between different groups if they are collected in a similar way.

    ReplyDelete
  5. Maybe Mike was talking about a different piece of research? The piece I read last included 51 SDQs (not 20) along with similar numbers of two other related questionnaires, for instance.

    ReplyDelete
  6. There was no other research. All Paula Rothermel's work relates to the data gathered from the 419 parents questioned following the questionairres which she sent out in 1997 and 1998. The data relating to sociability were acquired by administering these instruments:

    Goodman Strengths & Difficulties Scale (4-16): 20 children aged 4 to 11 years assessed by their parents.
    Revised Rutter Scale for School Aged Children: 20 children aged 5-11 years assessed.
    Michelson; Children’s Assertiveness Behaviour Scale questionnaire (CABS): 20 children aged 8-10 years old assessed.

    It is unclear how many children were actually involved, that is to say whether or not some the same children were given one, two or three of the tests.

    ReplyDelete
  7. ' Do you really think they haven't taken parental bias into account? These are psychology questionnaires - bias is something they are likely to be aware of and take into account when interpreting the results!'

    Of course bias was taken into account. The scoring borderline for detecting abnormalities is slightly different depending up whether the test was given by a parent or teacher. This does not alter the fact that any claim about Rothermel's research and the sociability of home educated children is based only upon the assertions of twenty parents who were part of a very carefully selected group of home edcuators fifteen years ago.

    Apart from the fact that these were all members of Education Otherwise, there is reason to believe that in any case they were a different bunch from the average modern home educator. As I said, a quarter of them were school teachers and almost without exception, their children had never attended school at all.

    ReplyDelete
  8. 'The results will also be comparable between different groups if they are collected in a similar way.'

    Another problem, which I did not mention above, is that these questionairres were designed to be used for school children. The scoring assumes that the children will be spending much of the day with a group of other children. This too makes it hard to take the scores seriously. The definitions of what constitutes normality relate largely to chidlren's relationships within a class at school.

    ReplyDelete
  9. Simon wrote, "The first difficulty is that this was a self-selected group, almost entirely from one organisation. (Twenty four out of twenty five of the questionnaires were sent only to members of Education Otherwise)."

    I'm surprised you don't own a copy of the research since you wrote a book on HE. If you had the research you would know that 29% of the questionnaires were returned from sources other than EO. All modern social research is self selected - it would be unethical otherwise and would not pass the requirements for university research.

    Simon wrote, "Only a fifth of those asked, wished to take part by answering any questions. These are likely to be those whose home education was going well and who wished to tell others of it."

    They were sent out with the newsletter to all EO members. Many of those members are not home educating yet or their children were the wrong age or no longer HE. Also, it sounds as though the questionnaire was long and quite complex (Rothermel received some feedback from non-responders to this effect). Of course many of them were not returned.

    Simon wrote, "The tests relating to social skills and the ability to get on with others were not administered to hundreds of children; not even a hundred children, nor even fifty. In fact the crucial tests about the ability of the children to get on with others and behave in a socialised way were only given to groups of twenty children."

    Wrong. A total of 136 tests were administered involving 103 different children, a few children taking more than one test. These are large numbers for research such as this. Only someone funded by business or the state can organise studies on the scale of Rutter. As it is, the amount of data produced during this study meant that help from a team at another University was needed to input and collate the data.

    Simon wrote, "click on the one marked P4-16 to see it. Has anybody seen the problem with handing a parent a form of this sort and asking her to answer the questions about her own child? I wonder how many parents will admit that their own child often lies and cheats?"

    These are standard questionnaires and methods used by many researchers in this and related fields. I'm sure then know what they are doing!

    Simon wrote, "A much more reliable way of establishing this sort of thing is for teachers and nursery workers to fill out these forms based upon their observations of children."

    Funnily enough, in research carried out by Shyer in the US, the parent completed questionnaires gave similar results to those of school using parents. But during a blind observation study the home schooled children were found to have fewer behaviour problems than schooled children. It's possible also that some parents are harder on their children than strangers!

    ReplyDelete
  10. 'I'm surprised you don't own a copy of the research since you wrote a book on HE'

    Well of course I do! It was freely available on Paula Rothermel's website until 11/11/09, when she removed it; a few days after I had asked some searching quesions about it on here. I observe that she is now hawking it round on Kindle, priced rather optimistically at £7.20. Is this the first Phd thesis to be released on Kindle?

    One of the problems with this is that the figures do seem a little hard sometimes to pin down. For instance, at Exeter in 1998, Paula Rothermel claimed to have been involved with 53 National Literacy Project Assessments. Four years later, the number had shrunk to 49. One can understand carrying out more assessments in the intervening four years, but not fewer! I think that I can hardly do justice to the questions about this work in a comment like this and so I shall write a longer article, detailing some of the difficulties and explaining why Rothermel's work was not seriously considered during Graham Badman's review of home education.

    It is quite true that the numbers of psycho-social assessments total 136 if you add all the numbers togther;

    Goodman Strengths and Difficulties Scale (SDQ)
    N=44 children aged 4 to 11-years-old (adult informant)
    N=7 children aged 11-years-old (self rated)
    Revised Rutter Scale for School Aged Children
    N=42 children aged 5 to 11-years-old (adult informant)
    Children's Assertive Behaviour Scale (CABS)
    N=43 children aged 8 to 10-years-old (self rated)

    There are two points to bear in mind though. I was speaking specifically about socialising, a point raised by Mike Fortune-Wood. The questions about sharing and playing from the Goodman SDQ were, according to the full thesis, administered by parents to 44 children. Elsewhere though, Rothermel has quoted a figure of 20;

    Goodman Strengths & Difficulties Scale (4-16): 20 children aged 4 to 11 years assessed by their parents.
    Revised Rutter Scale for School Aged Children: 20 children aged 5-11 years assessed.
    Michelson; Children’s Assertiveness Behaviour Scale questionnaire (CABS): 20 children aged 8-10 years old assessed.

    It is of course possible that after she mentioned these data, more of the tests were carried out. What militates against that hypothesis is that the figures for academic testing remain largely unchanged. If additional psycho-social tests were administered, it would be intereting to see the timetable for these. There is a reason why I am a little dubious about this being the case, but I shall go into that in further detail in my next post.

    ReplyDelete
  11. All of the psycho-social assessments relate directly or indirectly to socialising and certainly cover the type of concerns people have when they raise socialisation as an issue. The full report states clearly when this data was collected, it was during the second year of the study. The group was chosen at random from the original respondents and attained a 100% response rate.

    The main discrepancy between 20 tests and 136 is easy to explain. If you read the Exeter paper that included to figure you quote, it states:

    Data collection is current and full analyses will not be completed before April 1999. However, tentative results suggest that the children assessed, demonstrated high levels of ability and good social skills.

    Clearly the Exeter report was an interim. No doubt this also explains the other, smaller, discrepancy you mention. Hopefully you'll read the reports before your next post.

    ReplyDelete
  12. Simon wrote, "A much more reliable way of establishing this sort of thing is for teachers and nursery workers to fill out these forms based upon their observations of children."

    http://onlinelibrary.wiley.com/doi/10.1111/j.1469-7610.1997.tb01545.x/abstract

    "A novel behavioural screening questionnaire, the Strengths and Difficulties Questionnaire (SDQ), was administered along with Rutter questionnaires to parents and teachers of 403 children drawn from dental and psychiatric clinics. Scores derived from the SDQ and Rutter questionnaires were highly correlated; parent-teacher correlations for the two sets of measures were comparable or favoured the SDQ. The two sets of measures did not differ in their ability to discriminate between psychiatric and dental clinic attenders."

    ReplyDelete
  13. ' Hopefully you'll read the reports before your next post.'

    The easiest thing to do before my next post is to give readers a chance to familiarise themselves with the actual dissertation, which may be found here:

    http://pjrothermel.com/phd/Home.htm

    There are quite a few interesting points in it and I shall, as I say, go into the thing in detail in a post soon.

    ReplyDelete
  14. 'Simon wrote, "A much more reliable way of establishing this sort of thing is for teachers and nursery workers to fill out these forms based upon their observations of children."

    http://onlinelibrary.wiley.com/doi/10.1111/j.1469-7610.1997.tb01545.x/abstract

    "A novel behavioural screening questionnaire, the Strengths and Difficulties Questionnaire (SDQ), was administered along with Rutter questionnaires to parents and teachers of 403 children drawn from dental and psychiatric clinics. Scores derived from the SDQ and Rutter questionnaires were highly correlated; parent-teacher correlations for the two sets of measures were comparable or favoured the SDQ. The two sets of measures did not differ in their ability to discriminate between psychiatric and dental clinic attenders."'

    I'm not at all sure that this is good news for home educated children! if we are to place high reliance upon these instruments, then we must acknowledge that the majority of the children tested by Rothermel were abnormal socially! Of which, more soon. As she herself said of her findings;

    'The Goodman Scale diagnosed many of the home-educated children as having peer problems'

    ReplyDelete
  15. The rest of the quote fire anyone interested.

    "The Goodman Scale diagnosed many of the home-educated children as having peer problems. However, this scale represented common expectations that children should prefer the company of other children, should be liked by other children, should have at least one good friend, should prefer to play in groups and should want to share with other children. These are aspects of behaviour expected of schoolchildren. Where home-educated children are concerned, they tend mix with both children and adults, enjoying the company of all ages equally, sometimes other children do not like them, they tend to have a number of friendships from across the age spectrum and very often do not prefer to play in groups. Many home-educating families see sharing as maturational and do not expect their children to specifically share with other children. The research found that children learning at home have very difference experiences and skills from school children and that to expect the same of all children underestimates many valuable aspects of the home-educated children's characters. These children lived a very different life from children at school; their values were different and they were growing and developing on a path that was in so many ways, dissimilar from their schooled peers.

    The Rutter Scale, with twice as many questions as the Goodman Scale and with its broader perspective showed all the children to fall within acceptable social and behavioural parameters."

    ReplyDelete
  16. 'The rest of the quote fire anyone interested'

    Spoiler alert! I was planning on going into this in a post today. When a researcher carefully chooses a certain instrument, it is a bit much if, when the findings are disagreeable, she then blames the test for not being the right one! Of which, more later.

    ReplyDelete
  17. Its like you rеad my mind! Yοu appear to unԁerstand sο much
    аpproximately this, like yοu wrote the е-book in it or something.
    I feеl that уou саn do wіth
    а few percent to fοrce thе mеsѕage hоusе а bit,
    but other than that, this is greаt blog. A fantastic
    reаd. Ι'll certainly be back.
    Here is my web site ; personaltrainingtips.webstarts.com

    ReplyDelete
  18. If you wish for to get a good deal from this paragraph then you have to apply these strategies to your won webpage.



    My web page site here

    ReplyDelete
  19. It's going to be end of mine day, but before ending I am reading this enormous article to improve my knowledge.

    Also visit my webpage: nike shox deliver womens

    ReplyDelete
  20. This piece of writing will help the internet viewers for building up new webpage or
    even a weblog from start to end.

    Also visit my website; top seo companies 2012

    ReplyDelete
  21. You really make it seem so easy with your presentation
    but I find this topic to be really something that I think I would never
    understand. It seems too complicated and extremely
    broad for me. I'm looking forward for your next post, I will try to get the hang of it!

    Also visit my web page; as explained here

    ReplyDelete
  22. Excellent post. I was checking continuously this blog and I am impressed!
    Very useful information specifically the last part :) I care for such information a lot.
    I was seeking this particular info for a long time.

    Thank you and best of luck.

    my webpage: furnished

    ReplyDelete
  23. Great post.

    My weblog; buy a Flat

    ReplyDelete
  24. Very soon this website will be famous amid all blogging visitors,
    due to it's pleasant articles or reviews

    my web blog - www.propertywide.co.uk

    ReplyDelete
  25. Wow, fantastic blog layout! How long have you been blogging for?
    you made blogging look easy. The overall look of your web site is magnificent,
    let alone the content!

    Also visit my site :: v2 cigs coupon code free shipping

    ReplyDelete