Sunday 25 July 2010

Home education and the curse of anecdata

I am grateful to my daughter for teaching me about the word 'anecdata' in recent months. This word could have been coined with home education in mind! Anecdata means presenting various scraps of anecdotal evidence together in such a way as to suggest that it supports some pseudo-scientific idea; in other words presenting random, isolated stories together in such a way as to make them look like proper data.

One of the things that I have noticed time and again when dealing with home educators and talking about home education is that it is very tricky to pin down facts and figures. I am frequently assured, for example, that FE colleges will accept people on to courses without any previous qualifications. The same is said of universities. Whenever I ask for the names of these institutions, they never seem to be forthcoming. This happened during a recent exchange on here. After I had cast doubt upon the chances of studying at university without having any formal qualifications, somebody told me,

'I spoke to a home educator recently whose child has been accepted on a postgraduate course despite not having a degree.'

This is actually quite exciting, but when I tried to find out the details, it turned out that this was another of those cases where it boiled down to the equivalent of, 'A man in the pub told me.....' . No possible way of checking the facts. This is how it is with so many stories about home education; they end up with vague accounts which are impossible to follow up. Now twenty vague stories from unidentified sources do not impress me more than one such story. This is what is meant by anecdata, a collection of anecdotes which together are supposed to be as good as properly collected and verified data. The reason that these sorts of data do not really support any idea or theory is that one unsubstantiated account by itself is worthless. So are two, twenty or a hundred. It is not the case that the more vague and unidentified witnesses one can cite, the more reliable is the thesis being presented. If that were the case then we would all believe in flying saucers or homeopathy!

We see anecdata being used again and again in discussions of home education. This is such a useful word; I am surprised that nobody coined it long ago. The autonomous acquisition of literacy for instance is almost invariably supported by this kind of 'evidence'. people often claim, 'I know that children can learn to read like this because that's how my children learned'. What can one say about this? Without personal knowledge of the family in question, it is hard to judge this claim. is the person making it generally truthful? Does she have a mental health problem? Is she really even a home educator? Did her children learn to read without being taught? It is quite impossible to answer any of these questions. The person is typically anonymous and we are invited to take her word for an extraordinary assertion. Sometimes it is presented in the form of, 'My children and many of those of fellow autonomous educators'. This is no better really, we still don't know how much credence to place upon the word of an unknown stranger.

It is for these reasons that attempts are regularly made to gather real, verifiable data about home education. Such efforts are almost invariably viewed with suspicion and parents urge each other not to take part. The participation rate in such surveys never rises above 20%. This at once rings alarm bells for any objective observer. After all, for years these dramatic claims have been made about the efficacy of this treatment and yet when an attempt is made to examine these claims and verify them; nobody wants to participate. This makes the thing look more like a pseudo-scientific enterprise along the lines of some quack remedy rather than a genuine pedagogic technique. Most practitioners and devotees of proper medical treatments and educational systems are quite happy to allow sceptical researchers in to look at what they are doing. Those who are reluctant to do so are usually followers of some crank ideology who have an inherent fear of opening it up to the gaze of unbelievers. This is fine if what is happening is s religion. For instance in my church, claims are made for the transformation of the host into something different. This is not a scientific claim and no amount of measuring or analysis would reveal the supposed changes. There is nothing wrong with this, it is simply an irrational belief that some of us hold. I recognise this and am quite happy with it. But some home educators make the claim that their ideas produce quantifiable changes in children and their minds. If this is so, then it should be possible to measure and observe these changes. At this pint though, the additional claim is made that to measure these changes would be to destroy them and wreck the whole process. If we are to take them only on trust and not allowed to look too closely for fear of harming the activity, then we can be pretty sure that we are not dealing with a scientific or rational enterprise at all, but something as peculiar as the transubstantiation of the host into the body of Christ. In other words, it makes autonomous education into a religion or faith rather than an educational method.

9 comments:

  1. Simon wrote,
    "The reason that these sorts of data do not really support any idea or theory is that one unsubstantiated account by itself is worthless. So are two, twenty or a hundred."

    The example you quote above will appear unsubstantiated to you. But I spoke to the mother and the child in question. They sounded knowledgeable and trustworthy so I believed that they were telling the truth. I had no reason to suspect them of lying to me. I accept that this is not true for you, it's impossible for you to judge the veracity of my report because you know nothing about me, but this is how people tend to gain information - most people don't believe everything they hear, it depends on the source and context and then a judgement is made. Even on internet lists people tend to build up relationships, sometimes meeting in real life or at least knowing someone in real life who has met the other person. Relationships build up over time, information is exchanged and a picture of that persons accuracy and truthfulness builds up. This background knowledge can then be used to weigh up any such reports that they bring to the list. I don't think you can compare acceptance or non-acceptance to a course of study with the effects of a medication! One is objective and the other subjective, they are either on the course or not!

    Simon wrote,
    "The autonomous acquisition of literacy for instance is almost invariably supported by this kind of 'evidence'. people often claim, 'I know that children can learn to read like this because that's how my children learned'."

    It is also supported by research such as that carried out by Alan Thomas and also US research that found no significant differences in outcomes according to methods followed. BTW, do you have any research into the effectiveness of structured home education in the UK? Or are your beliefs in this method mainly based on the successful and effective education your daughter gained via this method (as mine with AE are)?

    ReplyDelete
  2. Simon wrote,
    "But some home educators make the claim that their ideas produce quantifiable changes in children and their minds. If this is so, then it should be possible to measure and observe these changes. At this pint though, the additional claim is made that to measure these changes would be to destroy them and wreck the whole process."

    Do you agree that removing a plant from the ground to measure it's root growth repeatedly would affect the growth rate of those roots? How would you prove that attempts to measure or monitor change will not affect the results in other areas such as learning and education? Once you have proved beyond doubt that attempting to measure my child's progress would not change their attitude and approach to learning I would be perfectly happy to ask them if they wish to take part in research. Isn't it part of the ethics of research that any likely or suspected risks are known and communicated to the participants so that informed consent can be gained?

    As an example of how research can change the behaviour being studied: I listened to a radio report of a study in Bradford (I think). They are following a cohort of babies in an attempt to observe the effects of various inputs, including diet, on children as they grow in an attempt to find out why health outcomes in that area are so poor. They played a recording of part of the study where parents were asked what they fed their babies. One mother commented afterwards that she found the process very interesting and intended to try some of the foods they had listed with her baby. This baby's diet was changed by the study that was attempting to observe her natural diet but the process itself changed that diet.

    Obviously the significance of this will depend partly on how well the study is designed. If the next round of questions picks up on this change in diet then all may not be lost as at least they will have an accurate picture of a diet that led to particular outcomes even though those outcomes may be different from the ones the research set out to study, but this may not happen and other changes in behaviour are more difficult to measure. How would you measure the child's attitudes to learning and education, their motivations for the choices they make and the affect this has on their ability to internalise and integrate knowledge, before and after the study process so that you are able to judge if you have studied 'true' AE or 'research affected/changed' AE?

    ReplyDelete
  3. I expect all home educators would find it easy to agree with yesterday's post - this one will of course be far more controversial!

    Few thoughts...
    a) yes, anecdotal evidence abounds in the HE worlds and in all of life; frustrating if you want to verify the facts, but just because you can't do that in some cases, doesn't make the facts wrong.

    b) The example you give (about post grad courses etc) does seem odd; logically you can't do a post grad course without being a graduate in the first place. However that may be for a number of odd reasons - it may be exactly true; because strange exemptions do happen. It also seems possible to me that there is a misunderstanding - the constant naming and renaming of quailifications muddles me, and I spend most of my time dealing with "education". For example, once upon a time a "diploma" was usually the title given to a qualification taken post 18 and about equivalent to the first 2 years of a degree, but now of course it also the title of a range of qualifications that a school child can take at 14 over 2 years and at the lowest level only equivalent to GCSEs belore a grade C. So "my child is at college doing a diploma" can mean lots of different things. A "post graduate course" may not be what they even meant.

    c) As much as you would love some objective research into what works in home ed... it won't happen. There is no way that the majority who adhere to the very methods that you want testing
    wil ever cooperate, so I think it isn't even worth time suggesting that they should!

    ReplyDelete
  4. "It also seems possible to me that there is a misunderstanding - the constant naming and renaming of quailifications muddles me, and I spend most of my time dealing with "education"."

    The course in question is normally only available to degree holders, that's what postgraduate usually means isn't it? I must admit that I could be wrong as I've had very little to do with universities. The young person dropped out of the relevant degree during their first year but went on to work in a related occupation before applying for this course a year or two later.

    ReplyDelete
  5. BTW, I've just remembered (about the postgraduate course entry for a non-graduate case) that the university's initial response was to say no, but the young person persevered and talked to the tutor directly. This led to an interview and the offer of a place on the course. I suspect that this initial response is typical of the response Simon usually receives when he enquires randomly about atypical entry qualifications to degree courses.

    ReplyDelete
  6. "I suspect that this initial response is typical of the response Simon usually receives when he enquires randomly about atypical entry qualifications to degree courses."

    Actually, I am usually a little more cunning than that! I devise a student on whose behalf I am enquiring and bend over backwards to find ways that she might gain access without formal qualifications. You are right that a straightforward email enquiry might produce an automatic response of 'nothing doing'. Despite all this and being a very persuasive person, I have never had any luck. Anybody who knows of a university which will take students without qualifications or an FE college who will accept students for A level courses without previous GCSEs; I would be very keen to hear of them.

    ReplyDelete
  7. "But I spoke to the mother and the child in question. They sounded knowledgeable and trustworthy so I believed that they were telling the truth. I had no reason to suspect them of lying to me."

    This post was not directed against you! I gave this as a recent example of stories that circulate; another recent instance was the idea that home education was illegal in France until 1998. We hear so many tales about higher education, but whenever I look into them and follow them up, they seem to vanish like mirages.

    ReplyDelete
  8. "How would you prove that attempts to measure or monitor change will not affect the results in other areas such as learning and education? "

    By this reasoning, we would know nothing about education at all because even to study what was happening would destroy the process. I don't believe that children are in general such fagile creatures that they are unable to tolerate a few questions and a little sympathetic curiosity about their lives and mental activity. If a child were so delicate, it would be a cause for concern; this would not be a normally robust child.

    ReplyDelete
  9. "This post was not directed against you! I gave this as a recent example of stories that circulate;"

    I realised this. My point is that people judge how much they believe something like this using their background knowledge of the person giving the information - it's not just a case of believing everything you hear indiscriminately.

    "We hear so many tales about higher education, but whenever I look into them and follow them up, they seem to vanish like mirages. "

    I would only take information about access to higher education seriously if I had good reason to trust the person giving the information. Other information I would take with a pinch of salt.

    "By this reasoning, we would know nothing about education at all because even to study what was happening would destroy the process."

    Possibly except that we can see outcomes when the child or young person autonomously chooses to take qualifications, apply for jobs, start businesses, etc.

    "I don't believe that children are in general such fagile creatures that they are unable to tolerate a few questions and a little sympathetic curiosity about their lives and mental activity."

    I don't think children are especially fragile either. I don't think they will be upset by the questions, but I do think it could affect how they view their education and the reasons for their learning. They may begin to make choices in order to impress and please others, or to avoid hassle with the LA, rather than because of intrinsic motivation, for instance.

    ReplyDelete