p-values and p-hacking

In short

• Right-tail test: $p = P(T \ge t_{obs} \mid H_0)$. Left-tail: $p = P(T \le t_{obs} \mid H_0)$. Two-tailed: $p = 2P(T \ge |t_{obs}| \mid H_0)$.
• Reject $H_0$ if p < $\alpha$; the smaller the p-value, the stronger the evidence against $H_0$.
• p-hacking: changing the analysis (period, benchmark, excluded data) until a result becomes significant, then reporting only that version.
• It inflates the probability of Type I errors (false positives).
• Three investment arenas: selective performance reporting, data mining for strategies, and manipulating performance metrics.
• Defences: repeated confirmatory testing, out-of-sample validation, transparent methods and ethical standards.