We often invoke the Central Limit Theorem to model the sampling distribution of the mean as a normal distribution, and in doing so usually calculate the standard error of the mean (SEM) using the formula
Here s is the sample standard deviation and n is the sample size. The SEM is then used as the standard deviation of the normally distributed model of the sampling distribution of the mean, and gives a measure of how precisely the true mean is estimated.
As the sample size is increased, SEM approaches zero. However, the rate at which it approaches zero slows as sample size increases.
Consider the following gamma-distributed population:
Suppose we sample between one and 1,500 values from this population, and plot the computed SEM as a function of the sample size:
Here we see that the rate at which the SEM approaches zero slows significantly as sample size increases. The percent decrease in the SEM value is 28% between sample sizes 500 and 1,000, but only 14% between sample sizes 1,500 and 2,000. The percent change in SEM continues to decline at higher sample size values.
From this analysis, we see that the payoff in increased precision due to increasing sample size falls off at high sample sizes. It follows then that collecting more data during an experiment has diminishing returns once the intended precision is reached.