The likelihood function is not a probability function

Some people always misinterpret the likelihood function as probability function. They’re similar and related but distinct.

The likelihood function is a measure of an intensity of likelihood for a particular value for the parameter that varies as the “parameter” (here we consider a single parameter) changes. This measure cannot be interpret as probability. The main reason is that we do not assume any probabilistic structure for the parameters (note*). Parameters are fixed but unknown quantities.

For instance, the mass of sun is a parameter that influence the sunlight intensity. Based on data, we can infer the mass of sun by the likelihood function. So we can say the case that the sun has mass 1.9891 × 1030 kilograms has the likelihood 0.9 (numbers are made by arbitrary). This does not mean that the mass of sun is a random quantity (and it shouldn’t); it just states that the intensity of sun’s mass being 1.9891 × 1030 kilograms by the likelihood measure is 0.9.

For another example in the daily life, assuming you want to know somebody’s age. However, you cannot directly ask him/her (this may be impolite). All you can do is to infer the age by asking him/her some other questions. Based on the responses, you can make some inference. So after a short chat, you will make a conclusion in your mind like “there’s 0.3 likelihood that his/her age is 25”. However, this doesn’t mean that his/her age has probability 30% being 25; the age is just an unknown value for you and has no probability.

It is true that the likelihood function is related to the probability density function (note**). For probability density function, we fixed the parameters and consider the probability density for different observations. For likelihood function, we use the same form of probability density function but we fixed the observation and consider different parameters. A critical difference is that if we sum over all possible observations for the probability density function for any fixed parameters, we will get 1. But the sum over all possible parameters under a specific observation is usually not 1 and even infinity. This makes the likelihood function differs from the probability density function.

Note*: This is called the Frequentist’s point of view. In the view of frequentist, the parameters are just unknown quantities that have no probability structure. In statistics, there’re another school called the Bayesian. In Bayesian’s perspective, the parameters can have probabilistic structure.

Note**: The probability density functions is in fact the joint probability density function for continuous random variables and is the joint probability mass function for discrete random variables.