Let's say I run a website, the background of which is this white color and I know the average amount of time, what people spend on my website, let's say it's 20 minutes, and I want to make a change, which will make people spend more time on my website. My idea is to make a background of your website yellow. But after I make that change, how can i be sure this will happen the intended consequences? This is where significance tests come into play. I will first make some hypotheses, null hypothesis and alternative hypothesis. The null hypothesis is generally a statement that: "Hey, your change had no effect, there is nothing new here "and it will be that the average is still 20 minutes after changing the background, in this case, to yellow. And we will have an alternative hypothesis. Our alternative hypothesis is that our average is now higher because of the change, that people spend more time on my site. Our average is greater than 20 minutes after the change. The next thing is to put a limit known as the significance level.
And in a little while you will see how this enters the game. Your level of significance usually marked with the Greek letter alpha and significance levels such as 1/100 or 5/100, or 1/10, or 1%, 5% or 10%. You can see others, but we will set a level of significance for this case in particular … Let's say it's going to be 0.05. And now it will take a sample of the people visiting this site with the new yellow background and we will calculate some statistical characteristics. The average of the sample, the standard deviation of the sample and we will say, "Assuming that the null hypothesis is true, what is the probability of having a sample with the statistical characteristics we received? " And if the probability is lower than our level of significance, if this probability is less than 5/100, if it is less than 5%, then we will reject the null hypothesis and we will say that we have proof of the alternative. But if the probability of obtaining statistical characteristics for this sample the level of significance is higher or higher, then we will say that we cannot reject the null hypothesis and we have no proof of the alternative.
I will call this step 3. In step 3 we will take a sample. Let's say we take a sample size … let's say we take 100 people who have visited our new website, the website with a yellow background, and we measure the same statistical characteristics for the sample. We measure the average of the sample here, let's say for this sample the average is 25 minutes. And it is also likely to … if we do not know the real standard deviation of the population, which we don't usually know we will also calculate the standard deviation of the sample. Then our next step is to calculate the p-value. p-value, which means probabilistic value, this is the probability of getting a characteristic at least as far from the average, if we assume that the null hypothesis is true. One way to think about it is that this is a conditional probability. This is the probability of the average value of our sample, when we take a sample of size n = 100, is greater than or equal to 25 minutes, provided that our null hypothesis is true.
In other videos we talked about how to do this. Assuming the sample distribution the average values of the sample is approximately normal, we can use the average value of the sample, we can use the sample size, we can use the standard deviation of the sample, we probably use the t-test (Student's test), to find how much this probability will be. And then we decide if we can reject the null hypothesis. Let me call this step 5. Step 5, there are two situations. If my p-value, if less than alpha, then I can reject the null hypothesis and to say that I have proof of my alternative hypothesis. If we have the other situation, if my p-value is greater than or equal to, in this case 0.05, if it is greater than or equal to my significance level, then I cannot reject the null hypothesis. I'm not going to say I accept the null hypothesis, I'll just say that we do not reject the null hypothesis. Let's say that when I do all these calculations, i get a p-value that will put me in this scenario right here.
Let's say I get a p-value of 0.03. 0.03 is really less than 0.05 and therefore I will reject the null hypothesis and I will say that I have proof of the alternative. And that must make sense, because we say that we took a sample and if we accept the null hypothesis, the probability of getting this sample is 3%, it is 3/100, and after this probability is less than our probability threshold here, then we will reject it and say that we have proof for the alternative hypothesis. On the other hand, there may be a scenario in which we perform all calculations here and the p-value we get, is equal to 0.5 which you can interpret by saying that if we assume that the null hypothesis is true, then making the background yellow does not lead to any change, I will have a 50% chance of getting this result.
And in this situation, since it is higher than my level of significance, I will not reject the null hypothesis. In case the null hypothesis is true and I get this result, this seems logical and plausible. This is the basis for significance tests in general and, as you will see, can be applied in almost any field, in which you find yourself. There is one last explanation, which I want to make very, very, very clear. Our p-value, what we use, to decide whether to reject the null hypothesis or not, this is the probability of obtaining the statistical characteristics of the sample, provided that the null hypothesis is true. Sometimes people confuse this and say: "Is this the probability of the null hypothesis is it correct according to the statistical characteristics we get? " And I will say that this is not the case at all.
We are not trying to measure probability the null hypothesis is true or false. We are trying to say that if we accept the null hypothesis as true, what is the probability of getting the result, which we received for our sample. And if that probability is low, if it is below any threshold we have previously set, then we decide to reject the null hypothesis and to say that we have proof of the alternative.