Thursday, 15 December 2016

The BTO Reply: Dear Gavin, re 'Lies, Damned Lies...etc'

On 11 December I wrote the post Lies, Damned Lies...and Statistics, in which I took issue with a tweet published by the BTO back on 3 December. The BTO has kindly responded.

Here's the original tweet...


My criticism focused on that statistic: '...[cats] only bring 23% of their prey home'

I'll try and summarise my criticism by breaking it into three main points:
  1. I questioned the motive for the tweet, suggesting it was inflammatory.
  2. After reading the research paper which produced the 23% figure I felt that including this statistic in the tweet was not valid or appropriate; disingenuous even.
  3. On first reading the tweet I thought its context implied reference to avian prey items, i.e. that for every single bird killed by a cat and brought home, another three were killed and not brought home. I also thought it was specifically referring to the state of affairs in the UK. I was wrong! It was soon evident that I had read more into the tweet than it actually said. So now I felt misled!
In other words I felt the venerable BTO had rather strayed into tabloidism, with a sensationalist tweet embellished by spurious data. And the BTO is an organisation I want to trust.

Below my original post are several comments, some sympathising with aspects of my view, one vigorously not. The BTO got in touch, wishing to add a comment but finding themselves constrained by the character limit on Blogger's 'comments' facility.

So here is the BTO comment in full, mostly written by Dr. Viola Ross-Smith, BTO Science Communications Manager, with input from four others...


________________________________________


We were interested to see your blog post, prompted by our tweet about a talk given at the recent BTO conference. The BTO is an impartial, evidence-based, non-campaigning organisation. We have no position or agenda when it comes to cats, but we do share relevant peer reviewed science on this topic, as it is of interest to our supporters and social media followers.

We hold an annual conference for our members each December, and our programme always includes a range of speakers from within and outside the BTO. The tweet you are referring to was about a talk given by an invited external speaker, Dr Becky Thomas. Dr Thomas’s research is wider than traditional ornithology, but our conference programme always includes topics that encompass broader issues, as our members tend to have a general interest in wildlife. Incidentally, despite our name, our own BTO research incorporates topics other than birds, and recent publications have included work on invertebrates and mammals. We also work internationally, typified by our recent success tracking Cuckoos breeding in China.

Our BTO tweeters are trained scientists, able to think critically about other peoples’ research and disseminate it appropriately. We aim to make science accessible, but it is of course very difficult to capture the complexity and caveats of a scientific study in 140 characters, which is why we try to include links to further relevant information where possible, and also to address questions any about our tweets. The photo we used in this particular tweet was one Dr Thomas herself presented to accompany the 23% statistic, which featured in the introduction to her talk.

The response to this tweet exposes a wider issue being debated in society at the moment about the validity of the scientific method itself, and whether experts should be taken seriously or indeed trusted at all. All published science goes through a rigorous peer review process. This has been refined over hundreds of years, and although criticism may be levelled at it, peer review is widely recognized as being the best procedure we have for publishing science that seeks to understand the true nature of the universe – whether that be interactions between cats and other wildlife, clinical drug trials or experiments on particle acceleration.

Peer review is the scientific gold standard and, therefore, studies like the one Dr Thomas quoted should not be dismissed out of hand. Having spent several years working on her topic, Dr Thomas is an expert in her field and we invited her to speak at our conference on this basis. She has published several peer reviewed articles on the subject she spoke about, one of which was linked to in the tweet. She herself takes part in the peer review process to critically assess others’ work, and if she considers that another study is worth reporting at our annual conference, we in turn trust her expert judgement and tweet it.

When it comes to sample size, it is important to remember that science is incremental. It’s true that 39 prey items is a relatively small sample, but it doesn't make the research worthless or wrong, it just means we should be careful with extrapolating up to larger samples. If we always waited until we had a sample size of 500 or 5000 before publishing, then scientific knowledge would progress at a much slower rate than it does currently. It's important to publish with the data we have available to help define future questions and hypotheses. The peer review process ensures studies do not appear too often or with too little data, safeguarding against people extrapolating their results or drawing conclusions that aren't justified. By the by, in the study in question, the comments on the blog calculating 2.8 kills a year per cat are not quite correct – the study found an average of 2.4 prey items per 7 days (not over 3 months), so the total number of kills for an average cat in a year would be estimated at 124.8, of which 16 would be birds if the ratios in the overall population are the same as the published sample. Of course, we agree with you that these ratios may vary for cats in different parts of the USA and in other countries such as the UK, where the faunal community is different. There will also likely be variation in prey items for cats in different seasons and habitats. These would all be interesting avenues for future research.

These are some of the reasons why we stand by our tweet, our guest speaker, and by work published by our fellow scientists in general; we hope it underlines both our impartial stance and the thought we put into how we compose our social media content. The scientific method is central to the work that we do and we place huge importance in presenting statistics that stand up to scientific scrutiny. We’d be disappointed to see people dismiss our wider work purely because they take issue with a statistic from peer reviewed research, within the public domain, and presented through one of our communications channels.

Our reputation as an independent and impartial organisation enables us to provide the evidence base that supports conservation and other decision making processes that shape the natural world in which we live. We hope that this reassures you about our intentions in promoting this particular piece of scientific research, taken from a longer presentation, and please do get in touch if you have any further queries about this or, indeed, our other work.

________________________________________


So, had that comment appeared beneath my original post, this is more or less how I would have replied:

Dear BTO,

Many thanks for taking the time to comment in response to this post. I do appreciate it.

First off, thanks for correcting my arithmetic in relation to Karen Woolley's comment. I had overlooked the fact that the numbers were based on one week rather than three months. Mercifully though, that basic error doesn't torpedo my criticism.

The tweet touches on what is an emotive issue for many. In view of the fact that I saw the tweet as somewhat inflammatory I was interested to see what you might have to say about the motive behind it. I note that you say the BTO is 'impartial' and 'non-campaigning' and has 'no position or agenda when it comes to cats' and that the tweet 'was about a talk given by...Dr Becky Thomas.' Fair enough, and I take all that at face value. However, if you were to show that tweet to a few hundred random people on the street and ask them what they thought it was all about I would be interested to know the outcome. I am confident that a fair percentage would see it as I did.

Now I'd like you to imagine those few hundred random people asking you a simple question...

“Twenty-three percent? Where did that come from?”
So you explain there was a study carried out. Cats were fitted with little video cameras so that the researchers could see exactly what happened to anything the cats caught. And that the cameras revealed they only brought home 23% of their prey, roughly a quarter. Which means that three out of every four prey items is either eaten on the spot or just left there.
“Really? That's terrible! So all our cats are out there killing things and we only get to see a quarter of it?”
“Well, no. Not all our cats; we can't apply it to all cats.”
“What do you mean?”
“Well, the study only involved 55 cats in total.”
“Ah, right, fair enough. But those 55, you're saying that between them they all brought home only a quarter of what they caught?”
“Er...no, not exactly. Only 16 of them actually caught anything, and 31 didn't even try. But, on average, each of those 16 brought home only 23% of their prey."
Man-in-the-street is now a bit confused...
“But it says in this tweet: 'Cats hunt no matter how much they are fed by their owners & they only bring 23% of their prey home'. It doesn't say some cats, it doesn't say 16 out of 55...I thought it meant cats generally, you know, all of them! That's what it implies!”
“But it doesn't actually say 'all of them', does it.”
“Well, I suppose not, but I just thought...okay, so you're saying I'm just a bit dense and read it wrong? Ha ha! It's all right, relax – I'm joking! Anyway, 16 cats you say? Just 16? Was that round here was it? Local?”
“Er...no. It was in the state of Georgia, USA.”

Personally I reckon man-in-the-street is now giving you the wry look that he usually reserves for numbers he reads in the Daily Mail. How do you think he views your use of that 23% statistic?

And finally, I wonder how many of those random citizens would look at the pigeon in the photo, learn you were from an organisation focused on birds and mistakenly conclude that this must therefore be all about birds.
“Oh dear! So for every little Robin my Tiddles brings home she leaves another three out there to rot, poor things?”
“No, no,” you explain, “There were only five birds killed in this study. The other 34 prey items were mammals, dragonflies, worms, lizards and so on, and the 23% figure was calculated from all of them, not just the birds.”
“Oh, I see. So this tweet's not just about birds? I thought...oh...Anyway, did you say 'lizards'?”
“Yes. You see, lizards are common where this research was carried out.”
“Oh, okay. Where was that then?”
“Er...in Georgia, USA...”

Hopefully my point is clear.

In the penultimate paragraph you state: “These are some of the reasons why we stand by our tweet...”

Sadly I think this statement underlines a fundamental problem.

Let me illustrate by referring to Bob Vaughan's hearty criticism of my original post. He states that in my post I “...attempt to belittle what is a perfectly honest piece of research...” The fact is, I had absolutely no intention of belittling that research; on the contrary, I found it very interesting (I questioned its application in your tweet, but that's different). I have read over my post several times and cannot see how Bob interpreted it the way he did, but nevertheless that is how he saw it, that I was knocking the research. In the light of his comment, if I was to wind the clock back and write it all again I would write it differently. I would make strenuous efforts to word it less ambiguously, to minimise the possibility of anyone being misled. In other words, because of the feedback I would change. Why? Because I wouldn't want the same thing to happen again.

Do you think that would be a wise and sensible course?

Or do you think I should simply write it exactly as before, word for word, because I know what I mean and if he doesn't get it, well then, that's his fault?

Possibly you are correct in paragraph four, in that my response to your tweet exposes the wider issue being debated in society re "the validity of the scientific method itself, and whether experts should be taken seriously or even trusted at all". That is all wa-a-a-ay beyond the scope of this particular post, but I suppose I could say this: if the public felt it had been misled by scientists, and said so, and the scientists' response was "That's how we do things, and we think it's okay", well, trust will struggle to thrive...

That your reply to my post contains only defense of your tweet, no suggestion that you might review your tweet output in future, and conveys only a deep conviction that scientists intrinsically have the high ground, all makes me wonder if you actually heard what my post was trying to say.

So there we are. Once again, my thanks to you all at the BTO for taking the time to respond to what was little more than a "rantette" really, as Bob Vaughan put it. It's been fascinating and I've learned a lot. Clearly there were other points in your reply which I haven't addressed, but this post is more than long enough already. Cheers, Gavin.

22 comments:

  1. It is excellent that the BTO have responded, good on them. Their reply unfortunately is very 2016 in nature, in that it is a perfect illustration of the vast gulf that exists between the man on the street (even the intelligent man on the street) and experts in whatever field they may have expertise in.

    In this case it is scientists, and whilst they and their aims are frequently worthy and deserving of time and respect, the response really only goes to show how much of a ‘bubble’ they exist in. The problem here, 140 characters notwithstanding, is one of communication. How do you go about making scientific research truly accessible to the man on the street?

    Currently I am not feeling very much in love with democracy – look at where it has led us recently – but ultimately, and whether we like it or not, it is the man on the street that dictates the future, and given his vote cannot be taken away upon completion of an IQ test, getting us back on track will involve real engagement and a breaking down of barriers such that valid scientific research isn’t trusted or is simply ignored. I am no fan of Michael Gove, not by a long shot, but the original tweet and this lengthy response go firmly and unfortunately into the “we have had enough of experts” bucket. That’s what needs fixing, not the science itself (which was fascinating by the way), but the way results are communicated, and really going back to basics here, how scientists interact with people who are not scientists.

    ReplyDelete
    Replies
    1. Thanks Jono, your comment echoes one on my Twitter feed which said "This is about communication, not science". I thought that put it very nicely, and wished I could have been as pithy.

      Delete
    2. I am incapable of pithy, but yes that's spot on.

      Delete
  2. “These are some of the reasons why we stand by our tweet...”

    This is the same mentality by which the law keeps innocent people in prison when it has been proved; away from the courts, that the original verdict was incorrect.
    Their ego, professional pride and not an unsubstantial amount of arrogance leads them to stances such as this.

    Gav, I doubt if your post on this subject actually had them thinking they may have made a mistake somewhere at all. To them it's impossible. They cannot make mistakes, they never make mistakes, how could they have always passed every exam and interview ever, if they had ever got something/anything wrong.

    No, scientists are the modern high priests of rightness and cor-rectitude.

    All that reply of theirs indicated was a futile attempt to cover their arses having screwed up.

    And they got it wrong.


    ReplyDelete
    Replies
    1. Ric, for me the BTO response was quite an eye-opener, being my first ever candid exchange of this kind with a professional scientific body. To be honest it was not what I was expecting, to the extent that I do wonder if my original point even got through.

      Anyway, it is interesting to read how others interpret the BTO reply. Including one or two off-blog comments I've received, there does seem to be some consensus. Your own mildly uncompromising comment here (Sandra enjoyed it by the way!) reflects that.

      It's a shame really. I almost felt I was committing an act of birding heresy by criticising the BTO, however trivially. And the communication chasm thus revealed I find quite depressing...

      Delete
  3. Most of the response is simply them telling you why they are great. An appeal to their authority

    They made a hash of it and presented the information in a misleading way, albeit it with good intentions.

    They ought to apologise and move on. There is some dignity in that



    ReplyDelete
  4. Come on- there are too many people and as a result, too many pets, be they cats or dogs. Both are a nuisance. When you have a cat taking residence in your garden and terrorising the wildlife and have watched someone walking 10 dogs on the beach at N Gare you loose patience.......

    ReplyDelete
    Replies
    1. As I said in the post, David, it is an emotive issue. Whatever one's position on the debate...

      Delete
  5. I wont be making anymore observations on anything as I didn't exactly cover myself with glory last time.

    ReplyDelete
    Replies
    1. Ha ha! That was my fault K. Sorry about that!

      Delete
  6. Some one pushes me into a corner, there's a better than middling chance that I'll come out fighting, whatever the odds. The BTO needing to offer such an extrapolated response to a simple post on NQS speaks volumes? One - NQS is a very popular, bird biased, blog and, thus, any views expressed will have sway on the opinions of other like-minded souls. Two, and probably more importantly, is the Institution is fast becoming out of step with where birding is headed and needs to reinstate it's marker as the central hub for all things ornithological? Straight away we've got a problem - we're birders, not ornithologists - that's science, what we do is fun! Alienation of the masses, by educated elitist organisations, is exactly why we're in the shit at present. The BTO hasn't done anything wrong, apart from demonstrate an inability to adapt to the modern world. It uses social media, but to purvey information which is beyond the comprehension of the audience it's aimed at. Not because that audience are stupid (far from it?) but, instead, not party to the science and that infrastructure which holds court in these higher echelons of excellence. Base line communication has fallen by the wayside, so this mighty bastion has sought to regain credibility, after a blooded nose! The followers of this blog are far more forgiving than the guys on the shop floor of my factory. If I were to take that original tweet and ask for a gut reaction, it would be far more extreme than any offered here. The BTO, on this occasion, screwed up - end of! Cats are a problem, not up for discussion, but 23% and research in the USA are, at the minimum, misleading if not down right fraudulent when offered in these circumstances. Lies, damn lies & statistics - Winston knew what he was talking about! Great post Gav - take care - Dyl

    ReplyDelete
    Replies
    1. Thanks Dyl. I must confess, I would not have foreseen NQS becoming a pop-up forum for a debate like this! It does kind of demonstrate the power of social media. In a positive way, I think. At the very least it facilitates constructive engagement of parties who would certainly not have been able to exchange opinions so easily in the pre-digital age.

      Delete
  7. Dear BTO, just on the off chance you get as far as reading all the way down to here, I'd just like to add something...

    In your reply you state that, based on the study data, you calculate "the average cat" would kill 16 birds per year. But isn't that figure derived only from the cats that make a successful kill? To quote the paper: "2.4 [prey] items/successfully hunting cat/week of footage". If you include those cats that failed to successfully hunt, plus the 31 that didn't even try (surely these too make up what you have termed "the average cat") wouldn't the calculation be as follows:

    5/55 x 52 = 4.7 birds per cat per year? Not 16.

    [Based 55 cats being monitored for 7 days each and catching between them a total of 5 birds]

    I appreciate this might appear somewhat petty, but it does illustrate how easy it is to get things wrong and, when it comes to a subject like this, there could easily be unwelcome negative implications.

    Imagine if your tweet had read "56% of cats do not hunt" or, even better: "At least 91% of cats do not hunt birds". Both statements true I think, based purely on the research data. How would your Twitter audience interpret such tweets? Very, very, very differently from the tweet you did publish. Of course, there are a few caveats which it might be helpful to include...but,hey-ho...

    ReplyDelete
    Replies
    1. Amendment to the above, which should read:

      'At least 91% of cats do not catch birds'

      Delete
  8. Personally speaking Gav, I feel you have won this argument with the BTO, somewhere between points, and a knock out.
    I feel like taking that rubbish response of theirs and re-writing it to say what they really mean.
    Their PR is total pony.

    ReplyDelete
    Replies
    1. Ric, no winners or losers here I think. At least, I don't really look at it that way. I'll be interested to see if there's any further dialogue, and certainly wouldn't want to say anything to discourage that. Possibly this whole episode was just a momentary distraction for some busy professional folk who have now moved on to more important matters. We'll see...

      Delete
  9. Do you think they are reading all of these comments Gav?

    ReplyDelete
    Replies
    1. Not sure Ric, but a comment in the covering email that came with the BTO reply tells me it's quite possible.

      Delete
  10. Hi Gavin, I was a trained scientist! But that doesn't mean I don't tweet rubbish on occasions. I've seen rubbish written by scientists and good stuff written by un-trained scientists - better stop there!

    ReplyDelete
    Replies
    1. Ha ha! Thanks Mark, that's a good reminder that we all risk showing ourselves up every time we open our mouth or put finger to keyboard! The only way to maintain a totally unblemished record is to keep silent and, as you know, I've never really been a fan of that approach!

      Delete
  11. Maybe the BTO should have stuck to ringing birds instead of telephones!

    ReplyDelete