In statistical inference and data science, we rarely have access to an entire population. Instead, we observe random samples and must infer unknown population parameters, denoted by symbols such as μ (mean), σ² (variance), or θ (a generic parameter). When a single numerical value computed from the sample is used to estimate a parameter, it is called a point estimate. The function or rule that produces this value is the point estimator.
Because a point estimator is computed from random data, it is itself a random variable. Therefore, it has a probability distribution (called the sampling distribution), an expected value, and a variance.
A natural and important question then arises:
What qualities should a “good” point estimator possess?
Classically, four properties form the backbone of estimation theory:
- Unbiasedness – the estimator is correct on average.
- Consistency – the estimator converges to the true value as the sample grows.
- Efficiency – among unbiased estimators, it has the smallest variance.
- Sufficiency – it extracts all the information available in the data about the parameter.
These properties are central in mathematical statistics, econometrics, and data science.
1. What Makes a Good Estimator?
Before turning to formal mathematics, it helps to build intuition. Think of the true parameter θ as a target on a dartboard. Each time we draw a sample and compute an estimate, we are throwing a dart. Some estimators:
- hit the centre on average but are widely scattered (high variance),
- cluster tightly but miss the centre slightly (biased but low variance),
- or miss the centre completely and inconsistently (biased and unreliable).
An ideal estimator should, as far as possible:
- not miss the target on average (unbiasedness),
- get closer to the target with more data (consistency),
- fluctuate as little as possible among unbiased competitors (efficiency), and
- use all relevant information contained in the sample (sufficiency).
In practice these aims sometimes conflict, which is why understanding the trade‑offs is essential.
2. Sampling Distributions and Their Role
Let a random variable X have mean μ and variance σ². Suppose X follows a normal distribution. If we compute the sample mean X̄ (that is, the arithmetic mean of n independent observations):
- the expected value of X̄ equals μ, and
- the variance of X̄ equals σ² divided by n.
Two essential consequences follow:
- the centre of the sampling distribution of X̄ equals the population mean μ, and
- the spread shrinks as the sample size n increases.
Graphically, as n grows large, the bell-shaped curve describing the distribution of X̄ becomes increasingly narrow and peaked around μ. This shrinking reflects the Law of Large Numbers and plays a key role in understanding consistency and efficiency.
3. Unbiasedness
3.1 Formal definition
Let θ be a population parameter and let θ̂ be an estimator of θ. The estimator θ̂ is said to be unbiased if the expected value of θ̂ equals θ. In symbols, E(θ̂) = θ.
This means that, in the long run over many hypothetical samples of the same size, the average value of the estimator equals the true parameter.
3.2 Intuitive explanation
Imagine repeatedly drawing samples from the same population and calculating θ̂ from each sample. If you could average all these calculated values, and this average equals θ exactly, then θ̂ is unbiased.
3.3 Examples
Example 1: Sample mean.
If the expected value of X is μ, then the expected value of the sample mean X̄ is also μ. Hence, the sample mean is an unbiased estimator of the population mean.
Example 2: Sample variance.
The usual estimator of σ² used in statistics is
S² = (1/(n − 1)) × Σ (Xi − X̄)².
This estimator is unbiased. The version with denominator n, sometimes written as the “population variance formula”, is slightly biased when used as a sample estimator, although it is still consistent.
3.4 Why unbiasedness is not enough
An estimator may be unbiased but still undesirable if it varies too much from sample to sample. In other words, among several unbiased estimators, we prefer the one with the smallest variance. This brings us naturally to the concept of efficiency.
4. Efficiency and the Bias–Variance Perspective
4.1 Relative efficiency
Suppose E1 and E2 are two unbiased estimators of θ. We say that E1 is more efficient than E2 if the variance of E1 is less than the variance of E2. That means E1 typically stays closer to the true parameter value than E2 does.
The relative efficiency can be written as the ratio
relative efficiency = Var(E1) / Var(E2).
If this ratio is greater than or equal to 1, then E1 is at least as efficient as E2.
4.2 Geometric intuition
Visualise the probability distributions of two unbiased estimators. Both are centred at θ. However, one bell curve is wide and flat, while the other is tall and narrow. The narrow one represents the more efficient estimator, because most of its probability mass lies close to θ.
4.3 Cramér–Rao lower bound
Under certain regularity conditions, there is a theoretical minimum possible variance for any unbiased estimator of θ. This minimum is the Cramér–Rao lower bound. Any unbiased estimator that actually achieves this bound is called a minimum variance unbiased estimator (often abbreviated MVUE).
4.4 Mean squared error and the bias–variance trade‑off
The mean squared error (MSE) of an estimator θ̂ is the expected value of the squared difference between θ̂ and θ:
MSE(θ̂) = E[(θ̂ − θ)²].
This decomposes into
MSE(θ̂) = Var(θ̂) + (Bias(θ̂))².
This formula shows clearly how bias and variance interact. An estimator with a very small variance but a tiny bias may have a smaller MSE than a perfectly unbiased estimator with large variance. Many modern statistical regularisation methods (for example, ridge regression) deliberately introduce a small and controlled bias in order to obtain a marked reduction in sampling variability, thereby lowering the overall mean squared error.
5. Consistency
5.1 Formal definition
An estimator θ̂ is said to be consistent if it converges in probability to θ as the sample size n tends to infinity. Symbolically, the probability that θ̂ lies close to θ becomes arbitrarily close to 1 when n becomes very large.
5.2 Intuition
A consistent estimator is one that “learns” from data. With small samples, it may fluctuate. But as we gather more and more information, it becomes almost indistinguishable from the true parameter. Consistency is therefore an asymptotic property.
5.3 Links to LLN and CLT
The Weak Law of Large Numbers guarantees that the sample mean is a consistent estimator of the population mean. The Central Limit Theorem goes further and states that, for large n, the distribution of the sample mean becomes approximately normal, centred at μ with variance σ²/n.
5.4 Visual story
Imagine plotting the sampling distributions of the sample mean for n = 1, 4, 25, 100, 1000, and 5000. You would observe the curves becoming increasingly narrow and spiky around μ. Two conditions characterise consistency:
- the sampling distribution of θ̂ collapses toward a single point as n increases; and
- this point is the true parameter θ.
6. Sufficiency
6.1 Definition via conditional distributions
Consider a sample X whose probability density or mass function depends on θ. A statistic S = S(X) is called sufficient for θ if, once S is known, the conditional distribution of the sample does not depend on θ. In other words, S contains all the information in the sample that is relevant to θ.
6.2 Neyman–Fisher factorization theorem
A powerful theorem states that S is sufficient for θ if and only if the joint density of the sample can be written as
f(x1, …, xn | θ) = g(S(x), θ) × h(x),
where the function h(x) does not depend on θ. This factorization shows that all dependence on θ enters only through the statistic S.
6.3 Illustration for Bernoulli trials
Suppose we have three Bernoulli observations, each taking value 1 for success and 0 for failure, with success probability θ. Then the statistic
S = X1 + X2 + X3
is sufficient for θ. Although different sequences of successes and failures are possible, once the total number of successes is known, the probability of any specific arrangement of those successes does not depend on θ. Therefore, S captures all available information about θ.
6.4 Why sufficiency matters
Sufficient statistics play a key role in:
- reducing data without losing information about parameters,
- maximum likelihood estimation, and
- constructing minimum variance unbiased estimators.
7. Relationships Between the Four Properties
The four properties are related, but one does not imply the others:
- An estimator may be unbiased but inefficient.
- An estimator may be biased yet still consistent.
- A sufficient statistic need not have minimum variance.
The ideal estimator is unbiased, consistent, efficient, and sufficient. In many real‑world problems, however, compromises are necessary.
8. Practical Worked Example: Estimating a Mean
Let X1, X2, …, Xn be independent observations from a population with mean μ and variance σ². Compare the following two estimators of μ:
- X̄ = (1/n) × Σ Xi
- T = X1 (the first observation only)
Then:
- both are unbiased because their expected value is μ,
- the variance of X̄ equals σ² divided by n, while the variance of T equals σ²,
- therefore X̄ is more efficient than T,
- and X̄ is also consistent (by the Law of Large Numbers), whereas T is not.
Hence, X̄ is clearly the better estimator.
9. Maximum Likelihood Estimation and the Four Properties
In many important models, maximum likelihood estimators (MLEs):
- are consistent,
- are asymptotically efficient (that is, they achieve the theoretical minimum variance when the sample becomes very large), and
- are functions of sufficient statistics.
However, MLEs are not always unbiased in small samples. This is a reminder that no single property tells the full story.
10. Summary and Key Takeaways
- Unbiasedness means the estimator is correct on average.
- Efficiency means the estimator has the least variance among unbiased estimators.
- Consistency means the estimator converges toward the true parameter as the sample grows.
- Sufficiency means the estimator or statistic captures all the information about the parameter contained in the sample.
Together, these properties define what it means for an estimator to be theoretically sound and practically reliable.
11. Suggested Figure Set for Your Website
To support learning and accessibility, consider including diagrams showing:
- the sampling distribution of X versus the sampling distribution of X̄,
- two unbiased estimators with different spreads,
- shrinking sampling distributions as n increases,
- a table illustrating sufficiency in Bernoulli trials.
12. Conclusion
The properties of point estimators form the conceptual foundation of estimation theory. They tell us when an estimator is reliable, efficient, and information‑rich. Whether you work in theoretical statistics, applied statistics, econometrics, or mathematical modelling, understanding these properties helps ensure that your numerical results carry rigorous scientific meaning.
By balancing unbiasedness, variance control, asymptotic behaviour, and information sufficiency, we move from mere computation to sound inferential reasoning—the essence of statistics as a scientific discipline.