HN Theater @HNTheaterMonth

No. It's called p-hacking, and it's a sleazy way to get published.

https://youtu.be/42QuXLucH3Q

⬐ bpodgursky

This is not p-hacking. p-hacking is when you don't pre-register your hypothesis and you hunt for ones which hit .05 after the fact.

There was a single hypothesis being tested here.

⬐ scotty79

Also this was >0.60 not around 0.05 If I read correctly.

⬐ darkerside

It was an adaptive clinical trial with multiple arms, so it wasn't a single hypothesis. Apparently this is considered legitimate in medical trials, but I have some reservations about the concept.

Please watch this video. It neatly shows why should you treat small and less rigorous studies with suspicion, even if there are many giving some results:

https://www.youtube.com/watch?v=42QuXLucH3Q

⬐ belter

"Why Most Published Research Findings Are False"

https://en.wikipedia.org/wiki/Why_Most_Published_Research_Fi...

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/

Negative Results:

https://www.negative-results.org/

Veritasium made a video about this: https://youtu.be/42QuXLucH3Q

⬐ koheripbal

They make a great point that modern media is about generating addiction.

...Outrage --> Hate --> Gratification --> Outrage --> Hate...

Step 1. Outrage: Strike the audience with some gross injustice - true or not - it's always useful to exaggerate. Make sure to use an echo chamber so no one can diffuse the emotion with a different perspective.

Step 2. Hate: Define a target. Demonize this target to create an emotional investment by the reader. Pile on ancillary "facts" to give the illusion of rigor. Truth is irrelevant as few will research independently.

Step 3. Follow up with some form of "justice" (usually better to do this at a later update). It doesn't matter if the subject was really hurt, as long as you can make it appear that the "bad guys" actually won in some way.

repeat.

Yeah, sure. Just don't eat, and that'll prevent alzheimer's.

If you study history, you have to reconcile these views with the fact that we've been wrong about every medical development until recent history. At one point we were killing cats to curtail the black plague, when in fact it was the fleas that were the cause.

We just don't know. And it's fine not to know. https://www.youtube.com/watch?v=42QuXLucH3Q

⬐ pmiller2

"There's evidence that..." does not equal "To prevent Alzheimer's, do...." Science does not work that way. We start from a hypothesis ("Alzheimer's is a type of metabolic dysfunction."), to test that hypothesis ("Intermittent fasting may delay or prevent the onset of Alzheimer's."), to finally saying "There's evidence that...." Eventually, once there's enough evidence, we can say with some certainty "To prevent Alzheimer's, do....", but we're not there yet.

⬐ sillysaurusx

Yes, and it's extremely unsettling that we have absolutely no idea what causes it or how to treat it. But jumping straight to "There is evidence in mice" with the implication of "try this yourself" is not justifiable.

⬐ peatmoss

On the other hand, even if it’s highly speculative, the cons for most people are “feeling hungry” and “a handful of other benefits that are supported by a growing body of evidence.”

I’m all about rigorously testing the hypothesis over time, but the safety / risks of the intervention has been well studied for a very, very long time.

“Might help and extremely unlikely to hurt” is not a bad basis on which to decide to take up IF.

⬐ hans1729

Replying to an article that lists a variety of studies with "we just don't know" doesn't really do your audience here justice tbh

⬐ curryst

You're being downvoted without anyone responding, which seems unfair to me. I disagree with you, but your point is valid (if more certainly phrased than I would have gone with).

You're right. As a species, we are wrong nearly all the time. For all the things we know, there are dozens of theories that were contemporaries or predecessors of the correct answer as we understand it now. You are entirely ignoring that our understanding does generally get progressively better until we fully understand the subject in question.

> At one point we were killing cats to curtail the black plague, when in fact it was the fleas that were the cause.

You're right, the cats were not the direct cause, and doing so may have even increased the spread of the Black Plague by allowing rats to spread more quickly.

On the other hand, blaming the cats is much closer to the actual answer than what we would have assumed 500 years before that. Namely that some deity was angry at us, and the only way to stop it was to offer gifts to that deity. And treating it through "humors" was a bad approach, but again, much better than guessing that the person had been cursed by a deity and that their survival hinged upon making good with that deity.

Our understanding is progressive, not binary. Our correctness is binary (i.e. we're either right, or we're not), but our understanding usually comes in steps where we slowly get closer and closer to the right answer. For fasting and dementia, who knows? We might be right, and we're at the end of the tunnel. Or we might be wrong, but hopefully one step closer to figuring out what's actually going on.

> We just don't know.

This part I really don't like. There's a difference between "don't know" and "can't know". Assuming you place some level of inherent value on knowledge, it would be optimal if we did know all the things that can be known. Even if you don't place inherent value on knowledge, some knowledge is useful, and surely we should seek to know those things? It's just difficult to determine whether a particular piece of knowledge has use or not. The Planck constant doesn't strike me as particular useful, but I'm almost positive it is to other people.

> And it's fine not to know.

I also heavily disagree with this part. People are, quite literally, dying in a way that is unimaginable to most of us. Getting dementia is a terrifying thing; it's one of the few things in humanity that can make you entirely lose "you". I've lost several relatives to dementia, and it's heartbreaking to watch. Just alternating periods of dementia where they're practically not there, and periods of clarity where they're terrified about what's happening to them. Death truly is a blessing to someone in advanced stages of dementia.

"We just don't know. And it's fine not to know." reads like some late-stage depression that has manifested itself into nihilism. There are tons of corollaries that are probably a lot less comfortable, but with the same implications. "Famine has always existed. And it's fine to not help.", "Racism has always existed. And it's fine to be racist.", "We've tried to stop child abuse and it hasn't worked, so it's fine to stop trying.". Just because something is currently true does not mean it will always be true, nor does it mean that trying to change it is futile.

I'm not particularly religious, but your response does remind me of the beginning of the serenity prayer:

> God grant me the serenity > To accept the things I cannot change; > Courage to change the things I can; > And wisdom to know the difference.

The most important part of that to me is the wisdom to know the difference. I fundamentally believe that dementia is something we can change, given enough time and resources. That makes it something worthwhile to pursue.

⬐ SpikeDad

Except of course there is no deity which can assist us in this decision. That means our society has to have this difficult discussion and determine if or how we can gain the wisdom to change things. There is no question this is an issue where can change things but can we gain the courage to determine the outcome?

So far the answer is no considering how unavailable death with dignity resources are in most states. Ironically it's religion that is blocking the fruitful discussions.

⬐ masterMiller_

It’s pretty glib to just reduce the idea of intermittent fasting to “you don’t eat.” Intermittent fasting shouldn’t even be seen as some kind of innovation, since it’s essentially a return to how human beings have managed their metabolic processes for tens of thousands of years.

It’s actually the idea of eating all day everyday that’s the innovation. We evolved to be hunter-gatherers; we had to chase down and kill animals for sustenance, and there was no reliable way to preserve that food. Which means that we are adapted to eating once or twice during a day, probably during the daytime, when our bodies are meant to be active, and then abstaining from food. It’s the relatively recent innovations that changed the way we handle our metabolic processes, like having food available all day and filling it with sugar, that are responsible for the proliferation of things like obesity and diabetes, which our ancient ancestor just didn’t have to deal with to the extent that we do, because their diets were more “natural” and less maladaptive compared to ours.

All this is to say that when we’re talking about our diets from a normative point of view, we should think of intermittent fasting sort of as the null hypothesis, necessitating a rebuttal before it can be refuted, and our modern way of stuffing our faces all the time as the alternative hypothesis, requiring justification before it can be accepted.

For a healthy adult it’s probably more sensible to consume one’s diet within a reasonable window, say 8 hours per day, than to be constantly digesting the hyper processed, over sugared nonsense we put into our bodies nowadays, to the point where it overlaps with things like our sleep cycle and when we get our exercise in.

Of course, I’m not a physician, so don’t take my word for it. But you should consider the fact that formal research is increasingly coming to understand the benefits of IF; OP’s link is only one example. We don’t understand the full range of benefits that IF provides because modern medicine, in the grand scheme of things, is still pretty much in its infancy; we’ve existed for approximately 200,000 years, but we’ve only had modern antibiotics for less than 100. But it can’t be denied that so far, things are looking pretty good for intermittent fasting, at least when it comes to its suitability for “healthy” adults.

⬐ cosmiccatnap

You lost us at glib

⬐ inglor_cz

I am an intermittent faster as well and IF got my blood pressure back to normal after 17 years of medication.

But I have my doubts regarding usefulness of IF against dementias. These are strange, autodestructive diseases. Maybe IF can lower their incidence or push them to higher age, but unlike diabetes 2nd type, I cannot believe it can reverse them.

⬐ masterMiller_

>I cannot believe it can reverse them

Yeah definitely, I wasn’t trying to claim it was some sort of panacea. I’d be surprised too if it helped cure something like Alzheimer’s.

Still, we’re learning surprising things about how our bodies work every day, and how seemingly unrelated parts are actually very interdependent. For example, we’re now seeing that the bacteria in our guts, our microbiome there, has a range of ways it maintains our health beyond just serving our digestive tracts.

⬐ novok

I think it is more like avoiding large amount of carbs to avoid developing type 2 diabetes. IF, low carb diets and more all contribute to helping avoid damage. I've even heard of Alzheimer's being called 'type 3 diabetes'. Also looking at the victim, he didn't look to be metabolically healthy, along with his wife. Households tend to eat similar.

Also not eating for 4 hours before you sleep and after you wake up is a very low risk, cheap & safe thing to try out compared to many other things. Same with avoid cheap carby things and eating more meat, eggs, vegetables and more non-processed carby foods.

⬐ t-writescode

The plant-based diet is absolutely full of carbs and apparently is good for diabetes, too.

Of course, I’m not a doctor, I just watch YouTube videos xD

⬐ novok

Yes they have carbs, but usually not to the amount and density that most pure, cheap carby foods like rice, potatoes, yams, bread, etc tend to have. Vegetables also have higher satiation.

It's actually fairly hard to get enough protein with a vegan diet unless you're eating beans, soy and vegetable protein extract and it's easy to fall into eating a lot of carbs and vegetable oil with some vegetables for flavor.

⬐ kortex

> Also not eating for 4 hours before you sleep and after you wake up

Wait, does that count as IF? I mean yeah, I guess it is called break-fast, but does sleep time count as fasting time? Somehow in my mind it felt like it would "pause" the clock.

⬐ novok

Yup sleep time counts! And you can drink water and coffee and other zero calories drinks. Some say you should avoid zero-calorie sweeteners.

So 4 hours no eating before sleep, 8hrs sleep & 4hrs after sleep is a 16 hour / 8 hour IF fast. Basically don't snack after dinner and skip breakfast.

You can do longer fasts for different reasons, but that is the basic one.

Cue “p = 0.05 is a random choice” and a big thread about p-hacking. Any takers?

It feels like everything to say has already been said about the bias of small sample sizes and the dangers of p-hacking. But it also seems like the discussion is inevitable here.

I wonder if there’s a new insight to glean, somehow...

For a background on why p-values are misleading, see https://youtu.be/42QuXLucH3Q which is excellent. He points out that 0.05 was selected by Ronald Fischer in 1925, and people have gone with it since.

⬐ im3w1l

The reason we have stuck with it ever since it s because it's a decent cutoff. I think in physics they sometimes use a much higher threshold of 5 sigma. Presumably because having much more data means they can afford to.

⬐ outlace

A 5% false positive rate can be pretty high depending on the situation, like whether or not a drug is effective.

Veritasium did a great piece on this.

https://www.youtube.com/watch?v=42QuXLucH3Q

⬐ igravious

Superb link. The end is humbling,

“What gets me is the thought that even trying our best to figure out what's

true, using our most sophisticated and rigorous mathematical tools: peer review,

and the standards of practice, we still get it wrong so often; so how frequently

do we delude ourselves when we're not using the scientific method? As flawed as

our science may be, it is far away more reliable than any other way of knowing

that we have.”

Veritasium has a good overview of the replication issues in much pf modern science: https://www.youtube.com/watch?v=42QuXLucH3Q

All I can say... "Is Most Published Research Wrong?" https://www.youtube.com/watch?v=42QuXLucH3Q

TL;DR: Likely yes. Spinning statistics is not that hard, even with the best of intentions.

My dude. Please go get yourself some education on how science works. I feel embarrassed for you.

The reproducibility crisis is about the prevalence of false positive results in the scientific literature. This video is a good primer: https://www.youtube.com/watch?v=42QuXLucH3Q

A false positive is when a paper says "this works" when it actually doesn't, because of a statistical fluke in their sample.

Even if citing the reproducibility crisis as an excuse to believe something that science has repeatedly found to be false isn't a really bad reason for belief (which it definitely is)...

it doesn't even apply because there can't be false positive results regarding neurolinguistic programming because there aren't any positive results regarding neurolinguistic programming. Even the paper put out by neurolinguistic programming practicioners themselves wasn't able to produce a detectable effect over a placebo.

I'm sorry, my friend, but this is an intervention. You're in a cult. You're making the exact same arguments as homeopaths and faith healers and fortune tellers.

⬐ carapace

i notice that you didn't answer my question. Do you have any actual experience with NLP at all?

In any event, it really seems to me like you're more interested in insulting me than in learning about NLP, so I don't think there's any reason to continue talking about it, eh?

> You're making the exact same arguments as homeopaths and faith healers and fortune tellers.

I understand that there's a superficial resemblance and I hate that, but as I said, I don't know what to tell you.

NLP is really easy, so if those "scientists" weren't able to measure something it's got to be "user error". I mean, I've personally done lots of experiments that show obvious results, so... I could go on but what's the point, eh? You've got your reality and I've got mine.

If the test data does not have a 50/50 split (or something around that ratio), the headline is straight up lies.

Skewed test data is the most common problem with research, reminds me of this great video by Veritasium (Is most published research wrong?): https://www.youtube.com/watch?v=42QuXLucH3Q

I mean, doesn't the west have the exact same issues? p-Hacking, lack of repeat studies, focusing on experiments that will make the biggest headlines, etc.

https://www.youtube.com/watch?v=42QuXLucH3Q

https://www.youtube.com/watch?v=0Rnq1NpHdmw

⬐ krona

The issue is as much a cultural one. I used to work with an editor of one of the worlds most prestigious chemistry journals, and she oft remarked how Chinese scientists didn't think accusations of plagiarism were sufficient grounds for rejection of a paper.

⬐ everybodyknows

In some cultures, cheaters try to keep it secret from the people they care about, because ostracism might follow. In other cultures, successful cheating is praised.

⬐ whooshee

I cannot think of a country which values cheating.

⬐ frozenport

And much of it is committed by Chinease origin scientists.

⬐ sctb

We've already asked you to please not make any nationalistic swipes like this, so we've banned the account.

⬐ robotresearcher

The issues are the same, but the frequency of them is very important. Academic fraud is like crime: it's always going to happen, but a little is very different to a lot.

⬐ boomboomsubban

The frequency is very important. Where is any evidence that the Chinese have a higher rate?

⬐ None

None

⬐ robotresearcher

From the fine article that we are discussing:

“Since 2012, the country has retracted more scientific papers because of faked peer reviews than all other countries and territories put together, according to Retraction Watch”

Along with the entire rest of the article filled with quotes and stats.

⬐ None

None

⬐ boomboomsubban

Number of faked papers isn't a frequency.

⬐ robotresearcher

Do you believe that China published more papers than everyone else put together in the same period?

I helped run a large international conference recently, with 900 published papers. Of these 66 (7.3%) were from China, and 280 (31%) from the US. Germany had 104 (12%). The numbers will vary by field, of course, but I've no reason to believe that this meeting was an outlier.

⬐ boomboomsubban

>Do you believe that China published more papers than everyone else put together in the same period?

I don't hold a belief, I haven't seen data that you seemed to be using to make conclusions. Or you're making somewhat large assumptions based on a few pieces of information. A little is different than a lot, both could have a little or a lot though.

> I've no reason to believe that this meeting was an outlier

Here's one, it was presumably run across the Pacific ocean from China.

⬐ robotresearcher

You’re trying too hard. There is no reason to believe that China is publishing papers faster than the rest of the world put together, unless you have data to that effect.

China is big, but not bigger than everyone else put together. Less than one person in six is Chinese.

And your presumption happens to be accurate, but 1 in 3 editions of the conference happen in Asia including China and China is still out-published by the USA by 4 to 1 or so even when the event is held in China. Chinese outnumber Americans five to one. Americans outpublish Chinese people many times over. As an aside, Swiss people outpublish Americans 8 to 1.

So, while skepticism is noble, when a country of 1/6 the people in the world has a retraction rate higher than everyone else put together, you need extraordinary data to decide that they are not in a mess. And “I don’t have data” is no such argument.

⬐ boomboomsubban

So in a discussion about research fraud, you're chastising me for wanting data you call important, putting complete trust in the data of a blog that you haven't seen, and posted random data you claim to have helped collect without giving me any reason to believe it's accurate.

Looking for retraction watch's numbers gave me a study that shows the US having the most retractions.[1] So again, if rates are what is important, you need data showing the rates.

http://www.pnas.org/content/109/42/17028.full

There are many, many BS papers because of the mass-production death-march researchers are told to produce, low p-value and sample size standards and

”Is Most Published Reseach Wrong?” (Veritasium, 2016)

https://youtu.be/42QuXLucH3Q

Which is about an essay from 2005: http://journals.plos.org/plosmedicine/article?id=10.1371%2Fj...

This doesn’t mean adversarial and critical feedback are bad, it means dis/incentives, politics and standards need adjustments.

The vast majority of the studies mentioned in those videos are old, where most researchers could get away with hacking the P values [0]. A lot of studies done in the last decade contradict the older ones (the article I linked earlier is full of sources). Also correlation is not causation. In how many of those studies controlled the amount of carbs? Because the combination is very important [1]

[0] https://www.youtube.com/watch?v=42QuXLucH3Q

[1] https://news.ycombinator.com/item?id=14831224

Edit: Looking at the last link of your first video, it acknowledges low carb diets are beneficial short and medium term, but there's very little literature for long term. It concludes low carbs are bad long term... but you must see the table on page 13 to see how they reached that conclusion. The "low carb score" group still had a crazily high 37-42% of carbs (compared to LCHF diets which are usually 5% carbs), it had more smokers and all of them had trans fats (very dangerous but can be easily avoided altogether).

⬐ cageface

There are plenty of studies from the last few years that back all this up in detail. If you go to nutritionfacts.org you'll find more references than you can handle.

⬐ DiThi

Did you search enough within those sources to refute the ones of the article I linked? I barely find any in nutritionfacts made in the last 10-15 years, and of those I can't find even a single one that points to fat and cholesterol as the main cause of heart disease.

It's not 'these days' it's the intrinsic nature of false hypotheses outnumbering true hypotheses by an enormous ratio. Even if everything were working perfectly you'd expect false results to outnumber true results. It's like the common example of disease screening, where even if you only get a false positive 1% of the time, but there are only 1% of true positives in your sample population, you're going get many more inaccurate results than accurate results. This video walks through an example calculation and what kinds of steps are being taken to improve on things.

https://www.youtube.com/watch?v=42QuXLucH3Q&t=218s

⬐ None

None

There's an interesting video by Veratasium "Is Most Published Research Wrong?" (Clickbaity title to be sure.)

It shows, on the overall statistics of valid/invalid vs. published/unpublished papers, the arbitrary p-value threshold is wholy inadequate to prevent a sizable fraction of published papers from being false positives (invalid but published).

https://youtu.be/42QuXLucH3Q

⬐ ReligiousFlames

Perhaps a more human-understandible criterion threshold for publishing would be logarithmic inverse p-value... the closer p gets to 0 (stronger correlation), the bigger log(1/p) gets. This of course is just one aspect of greater issues.

⬐ wyager

Worth noting that the particle physics community requires five sigma of confidence to consider a claim "legitimate", which is vastly higher than almost all other fields and has saved particle physics from making mistakes several times (like with the diphoton excess in 2015).

Definitely.

Another good idea is to show the difference between reality and data can be overwhelmingly significant.

Videos demonstrating this with examples:

- "The Bayesian Trap" https://youtu.be/R13BD8qKeTg

- "Is Most Research Wrong?" https://youtu.be/42QuXLucH3Q&

⬐ r_singh

I like to call myself a science enthusiast and like to know the answers to questions as much as my time and cognitive abilities allow me.

Being an average engineering graduate, I confess to taking published research seriously and spending energy thinking about it, assuming it to be true (even if counterintuitive obviously) all along.

Would be cool to read about some personal stories/experiences anyone on HN might have seen on data dredging (p hacking) by a scientist for advancement in their career.