Topic 3

Nonlinear Methods in Econometrics

Nonlinear Model Estimation
Statistical Inferences in Nonlinear Models

Readings and References:

R. F. Engle, "Wald, Likelihood Ratio, and Lagrange Multiplier Tests in Econometrics", Handbook of Econometrics, Vol. 1, Chapter 13, ed. by Z. Griliches and M. D. Intriligator, North-Holland, 1983, 775-826 (Paper).
W. H. Greene, Econometric Analysis, 5th Ed., Chapter 10: Nonlinear Regression Models, Prentice-Hall, 2002.
G. G. Judge, R. C. Hill, W. E. Griffiths, H. Lutkepohl, T.-C. Lee, Introduction to the Theory and Practice of Econometrics, 2nd Ed., Chapter 12: Nonlinear Least Squares and Nonlinear Maximum Likelihood Estimation, John Wiley, 1988, 497-570.
G. G. Judge, W. E. Griffiths, R. C. Hill, H. Lutkepohl, T.-C. Lee, The Theory and Practice of Econometrics, 2nd Ed., Chapter 6: Nonlinear Statistical Models, John Wiley, 1988, 195-223.
K.-P. Lin, Computational Econometrics: GAUSS Programming for Econometricians and Financial Analysts, Chapter 7: Nonlinear Regression Models.
A. Zellner and N. Revankar, "Generalized Production Functions," Review of Economic Studies, 1970, 241-250. (Paper)

Nonlinear Model Estimation

Nonlinear least squares and maximum log-likelihood are the most common methods for parameter estimation of nonlinear models in econometrics. Assume that the model, linear or nonlinear, is expressed as:

F(z,b) = e

F(z,b) defines the functional form of the model, where z = [y x] is the data matrix which includes both dependent (endogenous) y and independent (exogenous) x variables, b is the vector of unknown parameters and e is the model error. A typical nonlinear model in econometrics takes a separable form (between y and x) like this:

F(z,b) = F(y,x,b) = y - f(x,b)

Nonlinear Least Squares

The model is estimated by minimizing the sum of squared errors:

S(b|y,x) = e'e = F(y,x,b)' F(y,x,b)

Least squares estimates of the parameters are computed from the first-order condition for minimization (zero gradient):

¶S/¶b = 2e'(¶e/¶b) = 0.

Finally, the following Hessian matrix must be checked for the positive definiteness:

¶²S/¶b¶b' = 2[(¶e/¶b)'(¶e/¶b) + S_i=1,2,...,Ne_i (¶²e_i/¶b'¶b)]

In addition to the parameter estimates, the estimated variance-covariance matrix of the parameters is derived from the positive definite Hessian matrix as follows:

Var(b) = s² [½E(¶²S/¶b¶b)']^-1 = s² [(¶e/¶b)'(¶e/¶b)]^-1

where the estimated model variance is s² = e'e/N, and N is the sample size used for estimation.

If there are inequality parameter constraints (e.g., non-negativity) expressed in terms of a continuous transformation b = f(a) where a is an unconstrained parameter vector. Then from the estimator of a and Var(a), we have

b = f(a)
Var(b) = (¶f/¶a) [Var(a)] (¶f/¶a)'

Nonlinear Weighted Least Squares

The technique of nonlinear least squares can be generalized straightforwardly to consider the weighted model errors. Denote the weighting scheme w = w(b|y,x), a scalar or a vector, which may be linearly or nonlinearly dependent on part or all of the parameters. Define the weighted error terms as e^* = we. The model can be estimated by minimizing the sum of weighted squared errors:

S^*(b|y,x) = e^*'e^*

Maximum Log-Likelihood

Assuming the normal probability distribution of the model error, or e ~ normal(0,s²I), then the log-likelihood function for each data observation i is

ll(b,s²|y_i,x_i) = -½ [ln(2p)+ln(s²) + e_i²/s²] + ln(J_i(b))

where e_i = F(y_i,x_i,b), and J_i(b) = |¶e_i/¶y_i| is the Jacobian of transformation from e_i to y_i. The model is estimated by maximizing the sum of log-likelihood over a sample of N observations as follows:

ll(b,s²|y,x) = -½N [ln(2p)+ln(s²)] -½ (e'e/s²) + S_i=1,2,...,Nln(J_i(b))

The solution is obtained from the system of first-order condition as follows:

¶ll/¶b = - e'/s² (¶e/¶b) = 0.
¶ll/¶s² = - N/(2s²) - e'e/(2s⁴) = 0.

Usually the maximum likelihood estimation is performed by substituting out the asymptotic variance estimate s². That is, s² = e'e/N. Then the following concentrated log-likelihood function is maximized to find the parameter estimates b:

ll^*(b|y,x) = -½N [1+ln(2p)-ln(N)] -½N ln(e'e) + S_i=1,2,...,Nln(J_i(b))

Let e^* = e/[(J₁...J_N)^1/N]. Then the last two terms of the above concentrated log-likelihood function can be combined and the function is re-written as

ll^*(b|y,x) = -½N [1+ln(2p)-ln(N)] -½N ln(e^*'e^*)

Therefore, maximizing the concentrated log-likelihood function ll^*(b|y,x) is equivalent to minimizing the sum of squared weighted errors:

S^*(b|y,x) = e^*'e^*

where e^* = we, with the weight w = 1/[(J₁...J_N)^1/N] (inverse of the geometric mean of Jacobians) applied to each observation of the error terms e.

Solving from the first-order condition or zero-gradient condition:

¶ll^*/¶b = -½N ¶ln(S^*)/¶b = -½(N/S^*)(¶S^*/¶b) = -(N/S^*)[e^*'(¶e^*/¶b)] = 0,

the solution must be checked for the negative definiteness of the Hessian matrix (the second-order condition):

¶²ll^*/¶b¶b' = ½N ¶²ln(S^*)/¶b¶b'
= ½(N/S^*)[(1/S^*)(¶S^*/¶b)'(¶S^*/¶b) -(¶²S^*/¶b¶b')]

Since ¶S^*/¶b = 0 from the first-order condition for the maximum likelihood solution, the corresponding negative definite Hessian matrix is simply

¶²ll^*/¶b¶b' = -(N/S^*)[½(¶²S^*/¶b¶b')]
= -(N/S^*)[(¶e^*/¶b)'(¶e^*/¶b) + S_i=1,2,...,Ne_i^* (¶²e_i^*/¶b¶b')]

Thus the estimated variance-covariance matrix for the maximum log-likelihood estimator is:

Var(b) = [-E(¶²ll^*/¶b¶b')]^-1

= s²^*[½E(¶²S^*/¶b¶b')]^-1 = s²^*[(¶e^*/¶b)'(¶e^*/¶b)]^-1

Given that s²^* = S^*/N. It is ready to see that the solution derived from the maximum log-likelihood procedure is identical as that derived from the model formulated as a weighted least squares problem, in which the weight is defined as the inverse of the geometric mean of Jacobians or

1/[(J₁...J_N)^1/N].

Further, as in the case of nonlinear least squares, if there are inequality parameter constraints (e.g., non-negativity) expressed in terms of a continuous transformation b = f(a) where a is an unconstrained parameter vector. Then from the estimator of a and Var(a), we have

b = f(a)
Var(b) = (¶f/¶a) [Var(a)] (¶f/¶a)'

A Special Case

If the Jacobian J_i(b) = |¶e_i/¶y_i| = 1 for all observation i, then the log-Jacobian terms in the above concentrated log-likelihood function vanish. Therefore,

ll^*(b|y,x) = -½N [1+ln(2p)-ln(N)] -½N ln(e'e)

This is exactly the special case of classical nonlinear model in which e = F(y,x,b) = y - f(x,b). For this special case, maximizing the concentrated log-likelihood function ll^*(b|y,x) is the same as minimizing the sum of squared errors S(b|y,x).

Example 1: Generalized Production Functions

Based on the least squares and maximum likelihood criteria, estimate the CES production in the previous example using NLSQ and MAXLIK methods in GPE package, respectively (Program):

ln(Q) = b₁ + b₄ln(b₂L^b₃ + (1-b₂)K^b₃) + e

Based on Zellner and Revankar [1970], the CES production function may be generalized to consider variable rate of returns to scale as follows:

ln(Q) + q Q = b₁ + b₄ln(b₂L^b₃ + (1-b₂)K^b₃) + e

Modify and estimate the Generalized CES production function (Program).

Statistical Inferences in Nonlinear Models

The fundamental assumption of statistical inferences is the normal probability distribution of the model error:

e = F(y,x,b) ~ normal(0,s²I).

Thus the estimated least squares or maximum likelihood parameters b^* is normally distributed:

b^* ~ normal(b, Var(b^*))

where the estimated variance-covariance matrix is

Var(b^*) = [-E(¶²ll(b^*)/¶b¶b')]^-1

= s²^* [½ E(¶²S(b^*)/¶b¶b')]^-1

and the estimated asymptotic variance of the model is s²^* = S(b^*)/N.

Confidence location of the true parameter vector b is derived from the estimated b^* based on the following familiar asymptotic F statistic:

F = (S(b)-S(b^*))/J/s²^*

where J is the degrees of freedom associated with the testing hypotheses. By approximating the sum of squares function S(b) at b^* up to the second order and set ¶S(b^*)/¶b = 0,

S(b) - S(b^*) = ½ (b-b^*)' [¶²S(b^*)/¶b¶b'] (b-b^*)

Therefore, the test statistic for testing b = b^*:

JF = (b-b^*)' [Var(b^*)]^-1 (b-b^*)

follows a Chi-Square distribution with J degrees of freedom.

Wald Test

Consider J active constraints of parameters, linear or nonlinear (continuous and differentiable), expressed as the equation:

c(b) = 0.

If the constraints were true, without estimating the constrained model, the unconstrained parameter estimator b^* is expected to satisfy the constraint equation closely. That is, c(b^*) = 0. The test statistic

W = c(b^*)'[Var(c(b^*)]^-1c(b^*)

has a Chi-square distribution with J degrees of freedom. With the first-order linear approximation of the constraint function c(b) at b^*,

W = c(b^*)' {(¶c(b^*)/¶b) [Var(b^*)] (¶c(b^*)/¶b)'}^-1 c(b^*)

Note that this test statistic does not require the computation of the constrained parameter estimator.

Lagrangian Multiplier Test

Given the J-element constraint equation c(b) = 0, let b^* denote the constrained maximum likelihood estimator of the parameters b. The Lagrangian multiplier test is based on the score vector ¶ll(b^*)/¶b of the original parameterization. If the constraints hold, then ¶ll(b^*)/¶b should be close to ¶ll(b^*)/¶b for the unconstrained parameter estimater b^*, which is of course zero.

The test statistic is written as:

LM = (¶ll(b^*)/¶b) [Var(b^*)] (¶ll(b^*)/¶b)'

The estimated variance-covariance matrix of the constrained estimator b^* is computed as follows:

Var(b^*) = H^-1 [I - G'(G H^-1G')^-1GH^-1]

where H = [-¶ll(b^*)²/¶b¶b'] and G = [¶c(b^*)/¶b].

LM test statistic is easily approximated with the following formula:

LM = {[e(b^*)'(¶e(b^*)/¶b)] [(¶e(b^*)/¶b)'(¶e(b^*)/¶b)]^-1 [e(b^*)'(¶e(b^*)/¶b)]'}/(e(b^*)'e(b^*)/N)

Note that the maximum likelihood estimates of errors e(b^*) may be properly weighted, and this test statistic is based on the constrained parameters alone.

Likelihood Ratio Test

If both the constrained and unconstrained maximum likelihood solutions are available, then the test statistic

LR = -2(ll(b^*)-ll(b^*))

follows a Chi-square distribution with J degrees of freedom, in which there are J constraints in the equation c(b) = 0. In terms of sum of squares, it is

LR = N ln(S(b^*)/S(b^*)).

Example 2

Returning to the previous example of CES production function: