What does the p-value mean?

WARNING: this a long and 2 part post. the first part addresses the question asked. The second part addresses the 'tyranny of the null hypothesis ritual' on the quality profession. (I paraphrased two others in that phrase...)

There are several issues with the p value and how it is used in the "null hypothesis" construct.

First the .05 limit is arbitrary and was never really intended to be used without replication of results. (Fisher*).

Second, it is important to understand what the response factor is in this - or any – scenario as well as the sampling and test structure. Particularly when dealing with sociological and other areas where replication and controlled experiments are difficult or impossible. To address the exact question that you asked, let’s assume that your response is the results of various polls (well written and from truly random and representative samples of those who are likely to vote OR from some a priori selected balanced group of likely voters who were known to be undecided or likely to vote for one or the other candidates at the top of the ticket before the VP debate – this will give you direct before and after results) ; in other words NOT from some voluntary survey monkey post on facebook). A comparison of the poll results of the 3 categories, undecided, candidate A or Candidate B, will result in a p value that will indicate how likely the magnitude of the change is to occur if there were no real change. In the world of the null hypothesis, a high p value would indicate that the magnitude of the change in the sample, was likely even though there was no real change in the ‘population’. A low p value would indicate that the change was large enough to be unlikely to have occurred if there really were no statistically significant change. One can then conclude that the change was statistically significant. But one cannot conclude that the debate was causal merely from the p value.


Now two things to remember:

1. Statistically significant doesn’t mean large or even of any practical importance.
2. Statistical significance of the change doesn’t mean that your theory of the cause of the change is the cause of the change.

Unfortunately, the second point is incredibly important for sociological studies that cannot be planned or replicated. The results are often confounded by many other things that are not controlled for. One might be able to discern cause through interviews of opinions and thought processes of those polled, in other old fashioned scientific research instead of blind application of statistical tests. So many more things than the statistical math are critical to the reliability of the statistical test: sample size, sample selection criteria, structure of the samples (two random samples or matched pairs), choice of statistical test to match the data type (in this case categorical data) and structure of the test (random groups or matched pairs and 3 comparison categories), what other factors are confounded with the event of interest. This is why one day coffee is bad for you and the next day it is good for you…

* Fisher said “He should only claim that a phenomenon is experimentally demonstrable when he knows how to design an experiment so that it will rarely fail to give a significant result. Consequently, isolated significant results which he does not know how to reproduce are left in suspense pending further investigation.” (from “Proceedings of the Society for Psychical Research” in 1929 as quoted by David Salsburg in his book: The Lady Tasting Tea: How Statistics Revolutionized Science in the Twentieth Century, (p. 100). Henry Holt and Co.)

=========================================
Warning Rant Ahead: Interestingly, the way in which we use the p value and the null hypothesis ‘ritual’ was not endorsed by either of the two (opposing) forces that are often touted as championing them: Fisher and the Neyman-Pearson duo. (See “Bad statistical practice in pharmacology (and other basic biomedical disciplines): you probably don’t know P”, Michael J Lew, British Journal of Pharmacology, 2012, pp. 1959-1967 and “Mindless Statistics” by Gerd Gigerenzer, The Journal of Socio-Economics 33 (2004) pp. 587–606).


This approach was popularized by text book writers and ingrained into the common quality tool box by second generation 'six sigma' instructors. It was not endorsed by most of the Quality profession's founders and 'heroes' such as Shewhart, Deming, et al. (See “On Probablility as Basis for Action” by W. Edwards Deming, American Statistician, November 1975, Vol. 29, No. 4, pp. 146-152 https://www.deming.org/media/pdf/145.pdf ; “Shewhart-Deming Critique of Classical Statistics”, by Jonathon Siegel originally posted on the Deming Electronic Network Web Site available in the "Practical Quality Engineering Resources area of this forum; Statistical Method from the Viewpoint of Quality Control by Walter Shewhart, Dover Press

Well I suppose that is enough to start a debate that certainly shouldn't be 'dull'

 

1), 3) & 6) NO. The p-value says absolutely nothing about the quality of the sample. Look up SAMPLING BIAS. Remember the polls that overestimated Mitt Romney's ratings during his presidential run? A poor sample can bias the results in either direction.

2) Very close. The statistical difference could be 1) real due to theorized cause, 2) random chance, 3) sampling bias, 4) real, but due to another cause. I'm sure others can come up with additional possibilities.

4) & 5) True. The impact may have been real, but the noise drowned out the signal.

 

Closer, but you can only say that it is unlikely that the observed difference occurred simply by chance. (the likelihood of the observed difference occurring simply by chance is the alpha risk of the sample size)

The observed difference was not statistically significant. (this is where power comes in, but power must be established prior to selecting the samples) If there is a difference it is likely less than the detectable difference of your sample size.

the dilemma you are facing is not in understanding what you can and cannot claim with the p value and the null hypothesis approach. Your dilemma is that you want it to say more than it can. by itself it is a VERY weak 'tool'...

 

this is a slippery slope. it traps many many people.
If the study is important enough to do the first time, it's important enough to do it well. a well designed study involves a lot more than taking some data and putting it through some statistical software.
the whole issue with Mitt Romney was people's biased view of the voting process and their almost complete misunderstanding of statistics. Mitt was always within the 'margin of error' of Obama. (in p value terms the p value was high) So people who wanted him to win kept believing that he was going to win. the key here was replication (it's always replication...and homogeneity). EVERY credible poll had him on the losing side of Obama. true it was always within the margin of error of the poll so it was 'close'. Even Mitt thought he was going to win on election night. He - and others thought that if they just a bigger sample (everyone who was going to vote) the 'sampling error' would be eliminated. but it was always on the losing side, always. when you replicate the same result over and over and over, the margin error doesn't apply.

think of it this way. there is a 50% chance you will flip a head on a fair coin with a fair flip. every time you flip that coin there is 50% chance that you will flip a head. that is always true. but it isn't the whole truth. If you flip a coin 5 times and get a head, what do you think the odds REALLY are that you will flip a head again on the 6th flip? If you have a two headed coin the odds are 100%. If you have a fair coin, the probability of getting 5 heads in a row is only 3.1%, so getting a 6th head in a row is vanishingly small...this is conditional probability. A quote from Deming: Levels of significance furnish no measure of belief in a prediction. Probability has use; tests of significance do not"

so when you get a result you don't like and it looks 'close' the answer isn't to do a different study. the answer is to design a better study for your next question. otherwise you keep chasing your bias.

 

Part of the problem is that today we are trying to stretch the null hypothesis test way beyond what it was originally designed to do, which was to deal with small samples. It is part of the iterative inductive-deductive experimentation process and cannot be separated from that without profound risk. In that iterative process, you develop a hypothesis and run an experiment to test that hypothesis. If the results of the experiment are not significant, you modify your hypothesis. If they are significant you run a series of follow up experiments to validate the hypothesis. Many of the studies today that end up debunked fail to go beyond the "statistically significant" result and declare early victory only to be shamed in Andrew Gelman's blog. Bottom line, the hypothesis test was never intended to be the final say. It was just more information on which to base further experiments.

 

Why do we bother with an alternative hypothesis? Well, the null hypothesis ritual stems from old Greek logic. But there is NO law that says you have to follow it. Many thought leaders are now advocating for what Deming and Shewhart advocated for in the last century. There are MANY alternative theories and all of them matter.
I stopped teaching the Null hypothesis thing years ago. None of my students ever understood it and it isn't necessary. I've solved hundreds of complex problems, validated process changes, designed new processes and products, established process capability and reduced variation and never used the p value or the null hypothesis thingy.

Face it: the reason it doesn't make sense to you is that it isn't a very useful model. Miner's description of the iterative approach (famously explained by George Box) is useful model. And you can use p values for it - or not. But you don't need to think about the null and alternate hypothesis at all...
You might find two presentations I've posted in the resources section somewhat helpful: A Scientific Approach to determining root cause and Power Tools for Six Sigma. They involve an approach that doesn't require the null or p values...

 

Several reasons: First, it helps clarify your thinking. Second, it helps to formulate whether you need a 1-tailed or 2-tailed test. If you are very careful in how you word the null hypothesis, it might not be necessary, but many times the alternative is worded more clearly.

 

I would add another step to this. If it looks favorable from a time and cost perspective, run a confirmation pilot to validate before committing.

 

let's try this: read "On Probability as a basis for Action" then let's see what questions you have.

I think Fisher and Deming said it pretty well. Unfortunately too many people have fallen for the old "bright line" requirement. a p-value of .049 is no different than a p value of .051. the 3 of us keep circling around the same thing: p values, RPNs, Cpk values, R&R numbers. it's really all the same. the bright line is a seductive concept but "the calculation of mathematical formulas is no substitute for thinking". Replication, iteration, understanding the homogeneity of the process, study design, science, understanding simple probability and plotting your data: these are the critical elements we need to understand.

I understand that people keep hitting you with the p value and the NH thingy, but the only way to understand those weaknesses and be able to articulate them to others is to study and understand the alternate approaches.

 

Bev, Your link has an extra %20 in front that prevents it from working. When I strip that off the link works.

 

Thanks Miner - I didn't' notice that...strange

 

I just read the article referenced above. It was an excellent read. While having heard of the disagreement between the two factions, I had never seen it explained so clearly. Fisher's philosophy really resonated with me and closely matches my actual practice, whereas I have always been uncomfortable with the arbitrariness of the 0.05 threshold, particularly if you do not temper it with risk based thinking (i.e., 0.1 vs. 0.05 vs. 0.01) and follow up with a validation experiment.

I agree that the textbook authors popularized the Neyman-Pearson approach (corrupting it in the process by garbling the correct interpretation of the p-value), but this has been going on for much longer than Six Sigma has been around. I think it was already firmly implanted in the psyche before Six Sigma came along. Look at its prevalence in the soft sciences. In defense of "reputable" Six Sigma training organizations, they do emphasize running confirmation experiments, which safeguards against a lot of mistakes. However, there are a lot of schlocky Six Sigma organizations out there , which do not.