The BTO Reply: Dear Gavin, re 'Lies, Damned Lies...etc'

On 11 December I wrote the post Lies, Damned Lies...and Statistics, in which I took issue with a tweet published by the BTO back on 3 December. The BTO has kindly responded.

Here's the original tweet...

My criticism focused on that statistic: '...[cats] only bring 23% of their prey home'

I'll try and summarise my criticism by breaking it into three main points:

1. I questioned the motive for the tweet, suggesting it was inflammatory.
2. After reading the research paper which produced the 23% figure I felt that including this statistic in the tweet was not valid or appropriate; disingenuous even.
3. On first reading the tweet I thought its context implied reference to avian prey items, i.e. that for every single bird killed by a cat and brought home, another three were killed and not brought home. I also thought it was specifically referring to the state of affairs in the UK. I was wrong! It was soon evident that I had read more into the tweet than it actually said. So now I felt misled!

In other words I felt the venerable BTO had rather strayed into tabloidism, with a sensationalist tweet embellished by spurious data. And the BTO is an organisation I want to trust.

Below my original post are several comments, some sympathising with aspects of my view, one vigorously not. The BTO got in touch, wishing to add a comment but finding themselves constrained by the character limit on Blogger's 'comments' facility.

So here is the BTO comment in full, mostly written by Dr. Viola Ross-Smith, BTO Science Communications Manager, with input from four others...

________________________________________

We were interested to see your blog post, prompted by our tweet about a talk given at the recent BTO conference. The BTO is an impartial, evidence-based, non-campaigning organisation. We have no position or agenda when it comes to cats, but we do share relevant peer reviewed science on this topic, as it is of interest to our supporters and social media followers.

We hold an annual conference for our members each December, and our programme always includes a range of speakers from within and outside the BTO. The tweet you are referring to was about a talk given by an invited external speaker, Dr Becky Thomas. Dr Thomas’s research is wider than traditional ornithology, but our conference programme always includes topics that encompass broader issues, as our members tend to have a general interest in wildlife. Incidentally, despite our name, our own BTO research incorporates topics other than birds, and recent publications have included work on invertebrates and mammals. We also work internationally, typified by our recent success tracking Cuckoos breeding in China.

Our BTO tweeters are trained scientists, able to think critically about other peoples’ research and disseminate it appropriately. We aim to make science accessible, but it is of course very difficult to capture the complexity and caveats of a scientific study in 140 characters, which is why we try to include links to further relevant information where possible, and also to address questions any about our tweets. The photo we used in this particular tweet was one Dr Thomas herself presented to accompany the 23% statistic, which featured in the introduction to her talk.

The response to this tweet exposes a wider issue being debated in society at the moment about the validity of the scientific method itself, and whether experts should be taken seriously or indeed trusted at all. All published science goes through a rigorous peer review process. This has been refined over hundreds of years, and although criticism may be levelled at it, peer review is widely recognized as being the best procedure we have for publishing science that seeks to understand the true nature of the universe – whether that be interactions between cats and other wildlife, clinical drug trials or experiments on particle acceleration.

Peer review is the scientific gold standard and, therefore, studies like the one Dr Thomas quoted should not be dismissed out of hand. Having spent several years working on her topic, Dr Thomas is an expert in her field and we invited her to speak at our conference on this basis. She has published several peer reviewed articles on the subject she spoke about, one of which was linked to in the tweet. She herself takes part in the peer review process to critically assess others’ work, and if she considers that another study is worth reporting at our annual conference, we in turn trust her expert judgement and tweet it.

When it comes to sample size, it is important to remember that science is incremental. It’s true that 39 prey items is a relatively small sample, but it doesn't make the research worthless or wrong, it just means we should be careful with extrapolating up to larger samples. If we always waited until we had a sample size of 500 or 5000 before publishing, then scientific knowledge would progress at a much slower rate than it does currently. It's important to publish with the data we have available to help define future questions and hypotheses. The peer review process ensures studies do not appear too often or with too little data, safeguarding against people extrapolating their results or drawing conclusions that aren't justified. By the by, in the study in question, the comments on the blog calculating 2.8 kills a year per cat are not quite correct – the study found an average of 2.4 prey items per 7 days (not over 3 months), so the total number of kills for an average cat in a year would be estimated at 124.8, of which 16 would be birds if the ratios in the overall population are the same as the published sample. Of course, we agree with you that these ratios may vary for cats in different parts of the USA and in other countries such as the UK, where the faunal community is different. There will also likely be variation in prey items for cats in different seasons and habitats. These would all be interesting avenues for future research.

These are some of the reasons why we stand by our tweet, our guest speaker, and by work published by our fellow scientists in general; we hope it underlines both our impartial stance and the thought we put into how we compose our social media content. The scientific method is central to the work that we do and we place huge importance in presenting statistics that stand up to scientific scrutiny. We’d be disappointed to see people dismiss our wider work purely because they take issue with a statistic from peer reviewed research, within the public domain, and presented through one of our communications channels.

Our reputation as an independent and impartial organisation enables us to provide the evidence base that supports conservation and other decision making processes that shape the natural world in which we live. We hope that this reassures you about our intentions in promoting this particular piece of scientific research, taken from a longer presentation, and please do get in touch if you have any further queries about this or, indeed, our other work.

________________________________________

So, had that comment appeared beneath my original post, this is more or less how I would have replied:

Dear BTO,

Many thanks for taking the time to comment in response to this post. I do appreciate it.

First off, thanks for correcting my arithmetic in relation to Karen Woolley's comment. I had overlooked the fact that the numbers were based on one week rather than three months. Mercifully though, that basic error doesn't torpedo my criticism.

The tweet touches on what is an emotive issue for many. In view of the fact that I saw the tweet as somewhat inflammatory I was interested to see what you might have to say about the motive behind it. I note that you say the BTO is 'impartial' and 'non-campaigning' and has 'no position or agenda when it comes to cats' and that the tweet 'was about a talk given by...Dr Becky Thomas.' Fair enough, and I take all that at face value. However, if you were to show that tweet to a few hundred random people on the street and ask them what they thought it was all about I would be interested to know the outcome. I am confident that a fair percentage would see it as I did.

Now I'd like you to imagine those few hundred random people asking you a simple question...

“Twenty-three percent? Where did that come from?”

So you explain there was a study carried out. Cats were fitted with little video cameras so that the researchers could see exactly what happened to anything the cats caught. And that the cameras revealed they only brought home 23% of their prey, roughly a quarter. Which means that three out of every four prey items is either eaten on the spot or just left there.

“Really? That's terrible! So all our cats are out there killing things and we only get to see a quarter of it?”

“Well, no. Not all our cats; we can't apply it to all cats.”

“What do you mean?”

“Well, the study only involved 55 cats in total.”

“Ah, right, fair enough. But those 55, you're saying that between them they all brought home only a quarter of what they caught?”

“Er...no, not exactly. Only 16 of them actually caught anything, and 31 didn't even try. But, on average, each of those 16 brought home only 23% of their prey."

Man-in-the-street is now a bit confused...

“But it says in this tweet: 'Cats hunt no matter how much they are fed by their owners & they only bring 23% of their prey home'. It doesn't say some cats, it doesn't say 16 out of 55...I thought it meant cats generally, you know, all of them! That's what it implies!”

“But it doesn't actually say 'all of them', does it.”

“Well, I suppose not, but I just thought...okay, so you're saying I'm just a bit dense and read it wrong? Ha ha! It's all right, relax – I'm joking! Anyway, 16 cats you say? Just 16? Was that round here was it? Local?”

“Er...no. It was in the state of Georgia, USA.”

Personally I reckon man-in-the-street is now giving you the wry look that he usually reserves for numbers he reads in the Daily Mail. How do you think he views your use of that 23% statistic?

And finally, I wonder how many of those random citizens would look at the pigeon in the photo, learn you were from an organisation focused on birds and mistakenly conclude that this must therefore be all about birds.

“Oh dear! So for every little Robin my Tiddles brings home she leaves another three out there to rot, poor things?”

“No, no,” you explain, “There were only five birds killed in this study. The other 34 prey items were mammals, dragonflies, worms, lizards and so on, and the 23% figure was calculated from all of them, not just the birds.”

“Oh, I see. So this tweet's not just about birds? I thought...oh...Anyway, did you say 'lizards'?”

“Yes. You see, lizards are common where this research was carried out.”

“Oh, okay. Where was that then?”

“Er...in Georgia, USA...”

Hopefully my point is clear.

In the penultimate paragraph you state: “These are some of the reasons why we stand by our tweet...”

Sadly I think this statement underlines a fundamental problem.

Let me illustrate by referring to Bob Vaughan's hearty criticism of my original post. He states that in my post I “...attempt to belittle what is a perfectly honest piece of research...” The fact is, I had absolutely no intention of belittling that research; on the contrary, I found it very interesting (I questioned its application in your tweet, but that's different). I have read over my post several times and cannot see how Bob interpreted it the way he did, but nevertheless that is how he saw it, that I was knocking the research. In the light of his comment, if I was to wind the clock back and write it all again I would write it differently. I would make strenuous efforts to word it less ambiguously, to minimise the possibility of anyone being misled. In other words, because of the feedback I would change. Why? Because I wouldn't want the same thing to happen again.

Do you think that would be a wise and sensible course?

Or do you think I should simply write it exactly as before, word for word, because I know what I mean and if he doesn't get it, well then, that's his fault?

Possibly you are correct in paragraph four, in that my response to your tweet exposes the wider issue being debated in society re "the validity of the scientific method itself, and whether experts should be taken seriously or even trusted at all". That is all wa-a-a-ay beyond the scope of this particular post, but I suppose I could say this: if the public felt it had been misled by scientists, and said so, and the scientists' response was "That's how we do things, and we think it's okay", well, trust will struggle to thrive...

That your reply to my post contains only defense of your tweet, no suggestion that you might review your tweet output in future, and conveys only a deep conviction that scientists intrinsically have the high ground, all makes me wonder if you actually heard what my post was trying to say.

So there we are. Once again, my thanks to you all at the BTO for taking the time to respond to what was little more than a "rantette" really, as Bob Vaughan put it. It's been fascinating and I've learned a lot. Clearly there were other points in your reply which I haven't addressed, but this post is more than long enough already. Cheers, Gavin.