This is a fascinating share, if not for the results—"None of the measures of intellectual engagement was linked to the speed of decline of people's mental abilities over time"—but for the study itself.
"Many people also dropped out of the study. Only 96 of the 498 people recruited took part in the last round of testing. The researchers lost touch with 13 people, while 57 people died and 332 declined to take part."
Do studies like this often reach the front page of HN?
This is a cohort study conducted over 15 years. You can't really criticize it only based on those numbers, at least not without looking at similar numbers from other cohort studies with the same length.
I tend to ascribe some value to "It was published in the peer-reviewed British Medical Journal."
Additional thought: Especially because it is essentially supporting the null hypothesis.
You can't really criticize it only based on those numbers, at least not without looking at similar numbers from other cohort studies with the same length.
Well, one can't criticize the quality of the study without looking how well other studies did. But even if all studies involve a massive drop-out rate, the drop-out rate still seems like something to look at in evaluating the accuracy of the study.
I mean my concern, looking from the outside, isn't relative accuracy, but absolute accuracy. Not "is it as good as other studies?" but "it is good enough that I can rely on it?"
15 -years- is a LONG time for a study. I'm not sure about the dropout rates, but it is a real statistical issue. I'm just not sure there's anything that can reasonably be done about it. You'd have to have something like federally mandated participation / continued participation. That's also very unlikely to pass ethics boards.
BTW, it would also be an issue if the subjects were compensated to the point that they differed from the actual population at large.
Fundamentally having more respect for doing things right, the right way, over the /long/ term, including the STEM fields, would be a 'good thing' for any society, but particularly western ones (where often the bad/evil antagonist is based on science or a cold lack of emotions; but the protagonist is based mostly on emotion and feeling instead of logic).
15 -years- is a LONG time for a study. I'm not sure about the dropout rates, but it is a real statistical issue. I'm just not sure there's anything that can reasonably be done about it.
Everyone keeps replying as if my comments imply a demand that "something be done about this". That's not my point. Maybe nothing can be done about it. The situation is still significant or at least appears significant.
Similarly, it may be that running studies like this is really difficult and that massively reduced sample sizes are something we must accept for practical reasons... but that doesn’t mean it permits us to ignore statistical ramifications
There is nothing more frustrating than doing things you are bad at, regularly, especially if you are bound to get somewhat worse as time goes on. To your point, even if that were the reason for a significant number of departures, it would not go against the conclusion.
> This is a cohort study conducted over 15 years. You can't really criticize it only based on those numbers
You absolutely can.
Scientific studies should be judged on their potential to produce interesting results. This study has too small a sample size to provide results that merit making even the weakest claims about. Even the initial sample size was, frankly, just barely large enough to maybe provide suggestions for future research.
If a meaningful drop-out rate is expected, I would argue that the researchers running these sorts of studies are mostly wasting everyone's time and money for the sake of publishing a few papers.
a rate of drop-out like that would certainly make me doubtful of the results of the study. It seems reasonably likely that people dropped-out in a fashion that would be weighted in one way or another.
There are some ways to adjust and counteract these effects (it's an issue with every clinical trial on serious illnesses and older populations though).
for sure.. but I expect the real results to be worse (though this is purely speculative on my part), as I can only imagine people with steeper intellectual decline being more likely to drop out
Its mentioned in the paper: "However, recruitment to and
retention in cognitive ageing studies is also affected
by self selection to volunteer among better educated
individuals who are ageing well, which is also true of
our study. Thus, the findings might be biased towards
those individuals with higher cognitive performance."
I immediately thought the same. Then I thought about for a bit longer and quickly came up with an equal number of reasons why participants experiencing more rapid mental decline might stick it out. Really no way to know.
Maybe, but given the huge dropout rate I don't see how we can conclude anything from this study.
Of course, that would also be true if it had reached the opposite conclusion.
Have you ever taken these sorts of tests? They are not fun. Many questions are childish. Others make you feel stupid for not getting them right. Both extremes are necessary to the study.
I'm reminded of a study using the Nintendo Wii to help keep elderly people mobile. 90+% of them stopped playing immediately, within days, of the researchers leaving.
Have you ever taken these sorts of tests? They are not fun. Many questions are childish. Others make you feel stupid for not getting them right. Both extremes are necessary to the study.
So you're saying it's logical that a lot of people dropped out? Seems reasonable.
But that doesn't mean that you wouldn't have the problem of the high drop-out rate implying the final people remaining are biased sample. When you are filtering a large number of people down to a small number, making sure that small number is representative of the large number is a hard problem.
Maybe there's no solution or no easy solution. Doesn't change the problem.
> They are not fun. Many questions are childish. Others make you feel stupid for not getting them right. Both extremes are necessary to the study.
If that's the case, then the test should say this in the instructions... (It's a classic example of setting expectations, introducing objections up front..)
> This test may not feel exciting. Some questions could seem childish. Other questions could make people feel silly for marking the incorrect answer. These extremes are required for the study.
Maybe they dropped out because they were failing the tests of mental abilities, and were too frustrated and embarrassed to continue. But one would think that the authors had explored that.
I wonder if we'll see that change with the proliferation of communication across so many mediums. Between mail, email, texting, voice, social media, etc. staying in touch with a cohort might be much easier and more sustainable.
"Many people also dropped out of the study. Only 96 of the 498 people recruited took part in the last round of testing. The researchers lost touch with 13 people, while 57 people died and 332 declined to take part."
Do studies like this often reach the front page of HN?