On 11 December I wrote the post
Lies, Damned Lies...and Statistics, in which I took issue with a tweet published by the BTO back on 3 December. The BTO has kindly responded.
Here's the original tweet...
My criticism focused on that statistic:
'...[cats] only bring 23% of their prey home'
I'll try and summarise my criticism by breaking it into three main points:
- I questioned the motive for the tweet, suggesting it was inflammatory.
- After reading the research paper which produced the 23% figure I felt that including this statistic in the tweet was not valid or appropriate; disingenuous even.
- On first reading the tweet I thought its context implied reference to avian prey items, i.e. that for every single bird killed by a cat and brought home, another three were killed and not brought home. I also thought it was specifically referring to the state of affairs in the UK. I was wrong! It was soon evident that I had read more into the tweet than it actually said. So now I felt misled!
In other words I felt the venerable BTO had rather strayed into tabloidism, with a sensationalist tweet embellished by spurious data. And the BTO is an organisation I want to trust.
Below my original post are several comments, some sympathising with aspects of my view, one vigorously not. The BTO got in touch, wishing to add a comment but finding themselves constrained by the character limit on Blogger's 'comments' facility.
So here is the BTO comment in full, mostly written by Dr. Viola Ross-Smith, BTO Science Communications Manager, with input from four others...
________________________________________
We were interested to
see your blog post, prompted by our tweet about a talk given at the
recent BTO conference. The BTO is an impartial, evidence-based,
non-campaigning organisation. We have no position or agenda when it
comes to cats, but we do share relevant peer reviewed science on this
topic, as it is of interest to our supporters and social media
followers.
We hold an annual
conference for our members each December, and our programme always
includes a range of speakers from within and outside the BTO. The
tweet you are referring to was about a talk given by an invited
external speaker, Dr Becky Thomas. Dr Thomas’s research is wider
than traditional ornithology, but our conference programme always
includes topics that encompass broader issues, as our members tend to
have a general interest in wildlife. Incidentally, despite our name,
our own BTO research incorporates topics other than birds, and recent
publications have included work on invertebrates and mammals. We also
work internationally, typified by our recent success tracking Cuckoos
breeding in China.
Our BTO tweeters are
trained scientists, able to think critically about other peoples’
research and disseminate it appropriately. We aim to make science
accessible, but it is of course very difficult to capture the
complexity and caveats of a scientific study in 140 characters, which
is why we try to include links to further relevant information where
possible, and also to address questions any about our tweets. The
photo we used in this particular tweet was one Dr Thomas herself
presented to accompany the 23% statistic, which featured in the
introduction to her talk.
The response to this
tweet exposes a wider issue being debated in society at the moment
about the validity of the scientific method itself, and whether
experts should be taken seriously or indeed trusted at all. All
published science goes through a rigorous peer review process. This
has been refined over hundreds of years, and although criticism may
be levelled at it, peer review is widely recognized as being the best
procedure we have for publishing science that seeks to understand the
true nature of the universe – whether that be interactions between
cats and other wildlife, clinical drug trials or experiments on
particle acceleration.
Peer review is the
scientific gold standard and, therefore, studies like the one Dr
Thomas quoted should not be dismissed out of hand. Having spent
several years working on her topic, Dr Thomas is an expert in her
field and we invited her to speak at our conference on this basis.
She has published several peer reviewed articles on the subject she
spoke about, one of which was linked to in the tweet. She herself
takes part in the peer review process to critically assess others’
work, and if she considers that another study is worth reporting at
our annual conference, we in turn trust her expert judgement and
tweet it.
When
it comes to sample size, it is important to remember that science is
incremental. It’s true that 39 prey items is a relatively small
sample, but it doesn't make the research worthless or wrong, it just
means we should be careful with extrapolating up to larger samples.
If we always waited until we had a sample size of 500 or 5000 before
publishing, then scientific knowledge would progress at a much slower
rate than it does currently. It's important to publish with the data
we have available to help define future questions and hypotheses. The
peer review process ensures studies do not appear too often or with
too little data, safeguarding against people extrapolating their
results or drawing conclusions that aren't justified. By the by, in
the study in question, the comments on the blog calculating 2.8 kills
a year per cat are not quite correct – the study found an average
of 2.4 prey items per 7 days (not over 3 months), so the total number
of kills for an average cat in a year would be estimated at 124.8, of
which 16 would be birds if the ratios in the overall population are
the same as the published sample. Of course, we agree with you that
these ratios may vary for cats in different parts of the USA and in
other countries such as the UK, where the faunal community is
different. There will also likely be variation in prey items for cats
in different seasons and habitats. These would all be interesting
avenues for future research.
These are some of the
reasons why we stand by our tweet, our guest speaker, and by work
published by our fellow scientists in general; we hope it underlines
both our impartial stance and the thought we put into how we compose
our social media content. The scientific method is central to the
work that we do and we place huge importance in presenting statistics
that stand up to scientific scrutiny. We’d be disappointed to see
people dismiss our wider work purely because they take issue with a
statistic from peer reviewed research, within the public domain, and
presented through one of our communications channels.
Our reputation as an
independent and impartial organisation enables us to provide the
evidence base that supports conservation and other decision making
processes that shape the natural world in which we live. We hope that
this reassures you about our intentions in promoting this particular
piece of scientific research, taken from a longer presentation, and
please do get in touch if you have any further queries about this or,
indeed, our other work.
________________________________________
So, had that comment appeared beneath my original post, this is more or less how I would have replied:
Dear BTO,
Many thanks for taking
the time to comment in response to this post. I do appreciate it.
First off, thanks for
correcting my arithmetic in relation to Karen Woolley's comment. I
had overlooked the fact that the numbers were based on one week
rather than three months. Mercifully though, that basic error doesn't
torpedo my criticism.
The tweet touches on
what is an emotive issue for many. In view of the fact that I saw the
tweet as somewhat inflammatory I was interested to see what you might
have to say about the motive behind it. I note that you say the BTO
is 'impartial' and 'non-campaigning' and has 'no position or agenda
when it comes to cats' and that the tweet 'was about a talk given
by...Dr Becky Thomas.' Fair enough, and I take all that at face value.
However, if you were to show that tweet to a few hundred random
people on the street and ask them what they thought it was all about
I would be interested to know the outcome. I am confident that a fair
percentage would see it as I did.
Now I'd like you to
imagine those few hundred random people asking you a simple
question...
“Twenty-three
percent? Where did that come from?”
So you explain there
was a study carried out. Cats were fitted with little video cameras
so that the researchers could see exactly what happened to anything
the cats caught. And that the cameras revealed they only brought home
23% of their prey, roughly a quarter. Which means that three out of
every four prey items is either eaten on the spot or just left there.
“Really? That's
terrible! So all our cats are out there killing things and we only
get to see a quarter of it?”
“Well, no. Not all
our cats; we can't apply it to all cats.”
“What do you mean?”
“Well, the study only
involved 55 cats in total.”
“Ah, right, fair enough. But those
55, you're saying that between them they all brought home only a quarter of what they
caught?”
“Er...no, not exactly. Only 16 of
them actually caught anything, and 31 didn't even try. But, on
average, each of those 16 brought home only 23% of their prey."
Man-in-the-street is
now a bit confused...
“But it says in this
tweet: 'Cats hunt no matter how much they are fed by their owners &
they only bring 23% of their prey home'. It doesn't say some cats, it
doesn't say 16 out of 55...I thought it meant cats generally, you know, all of them! That's what it implies!”
“But it doesn't
actually say 'all of them', does it.”
“Well, I suppose not,
but I just thought...okay, so you're saying I'm just a bit dense and read
it wrong? Ha ha! It's all right, relax – I'm joking! Anyway, 16 cats you say? Just
16? Was that round here was it? Local?”
“Er...no. It was in
the state of Georgia, USA.”
Personally I reckon
man-in-the-street is now giving you the wry look that he usually
reserves for numbers he reads in the Daily Mail. How do you think he views your use of that 23% statistic?
And finally, I wonder
how many of those random citizens would look at the pigeon in the
photo, learn you were from an organisation focused on birds and
mistakenly conclude that this must therefore be all about birds.
“Oh dear! So for
every little Robin my Tiddles brings home she leaves another three
out there to rot, poor things?”
“No, no,” you
explain, “There were only five birds killed in this study. The
other 34 prey items were mammals, dragonflies, worms, lizards and so
on, and the 23% figure was calculated from all of them, not just the birds.”
“Oh, I see. So this tweet's not just
about birds? I thought...oh...Anyway, did you say 'lizards'?”
“Yes. You see,
lizards are common where this research was carried out.”
“Oh, okay. Where was that
then?”
“Er...in Georgia,
USA...”
Hopefully my point is clear.
In the penultimate
paragraph you state: “These are some of the reasons why we stand by our
tweet...”
Sadly I think this
statement underlines a fundamental problem.
Let me illustrate by
referring to Bob Vaughan's hearty criticism of my original post. He states that in my post I “...attempt to belittle what is a perfectly
honest piece of research...” The fact is, I had absolutely no intention of
belittling that research; on the contrary, I found it very
interesting (I questioned its application in your tweet, but that's different). I have read over my post several times and cannot see
how Bob interpreted it the way he did, but nevertheless that is how
he saw it, that I was knocking the research. In the light of his comment, if I was to wind the clock
back and write it all again I would write it differently. I would make strenuous efforts to word it
less ambiguously, to minimise the possibility of anyone being misled. In other words, because of the feedback I would change. Why? Because I wouldn't want the same thing to happen again.
Do you think that would
be a wise and sensible course?
Or do you think I
should simply write it exactly as before, word for word, because I know what I mean
and if he doesn't get it, well then, that's his fault?
Possibly you are
correct in paragraph four, in that my response to your tweet exposes
the wider issue being debated in society re "the validity of the
scientific method itself, and whether experts should be taken
seriously or even trusted at all". That is all wa-a-a-ay beyond the scope of
this particular post, but I suppose I could say this: if the public felt it had been misled by scientists, and said so, and the scientists' response was "That's how we do things, and we think it's okay", well, trust will struggle to thrive...
That your reply to my post contains only defense of your
tweet, no suggestion that you might review your tweet output in
future, and conveys only a deep conviction that scientists
intrinsically have the high ground, all makes me wonder if you actually
heard what my post was trying to say.
So there we are. Once again, my thanks to you all at the BTO for taking the time to respond to what was little more than a "rantette" really, as Bob Vaughan put it. It's been fascinating and I've learned a lot. Clearly there were other points in your reply which I haven't addressed, but this post is more than long enough already. Cheers, Gavin.