Systems of Regression Equations

Table of Contents

The General Model

Seemingly Unrelated Regression Model

Random Coefficents Model

Vector Autocorrelation Model: An Application

The General Model

Y_i = X_iβ_i + ε_i (i=1,2,...,G)

where Y_i = [Y_i1,Y_i2,...,Y_iN]', X_i = [X_i1,X_i2,...,X_iN]', and ε_i = [ε_i1,ε_i2,...,ε_iN]'. The model satisfies the following assumptions:

E(ε_i) = 0_Nx1
Cov(X_i,ε_i) = E(X_i'ε_i) = 0_Kx1
Cov(ε_i,ε_j) = E(ε_iε_j') = σ_ijΩ_ij

• σ_ii = σ²_i > 0.
• σ_ij = 0, if no cross equation correlation (i≠j).
• Ω_ij = I_NxN, if no serial correlation.

Y = Xβ + ε, or

⌈ | | ⌊

Y1 Y2 : YG

⌉  |  | ⌋

=

⌈ | | ⌊

X1 0 .. 0 0 X2 .. 0 : : : : 0 0 .. XG

⌉  |  | ⌋

⌈ | | ⌊

β1 β2 : βG

⌉  |  | ⌋

+

⌈ | | ⌊

ε1 ε2 : εG

⌉  |  | ⌋

This model is also known as the Seemingly Unrelated Regression Equations.

Special Cases

• Common Parameters: β_i = β for all i, or
  Y_i = X_iβ + ε_i

  ⌈ | | ⌊

  Y1 Y2 : YG

  ⌉  |  | ⌋

  =

  ⌈ | | ⌊

  X1 X2 : XG

  ⌉  |  | ⌋

  β

  +

  ⌈ | | ⌊

  ε1 ε2 : εG

  ⌉  |  | ⌋

• More restricted cases may include common regressors (X_i = X for all i) and restrictions on part or all of the paramters.

Seemingly Unrelated Regression Model

E(ε) = 0_NGx1
E(X'ε) = 0_Kx1
Var(ε) = E(εε') = V_NGxNG = Σ_GxG⊗I_NxN, where

Σ =

⌈ | | ⌊

σ11 σ12 .. σ1G σ21 σ22 .. σ2G : : : : σG1 σG2 .. σGG

⌉  |  | ⌋

Notice that contemporary correlation across equations is assumed although there is no serial correlation for each equation. GLS (Generalized Least Squares) estimation of the model parameters β follows:

b = (X'V^-1X)^-1X'V^-1Y
Var(b) = (X'V^-1X)^-1

The estimator of the elements σ_ij of V = Σ⊗I is obtained from s_ij = e_i'e_j/N, where e_i = Y_i - X_ib is the residual vector for equation i obtained from the OLS estimation (assuming no cross equation correlation). The above GLS estimation may be iterated to update the residuals e and variance-covariance matrix V.

Random Coefficients Model

Y_i = X_iβ_i + ε_i
β_i = β + υ_i

We note that not only the intercept but also the slope parameters are random across equations. This model generalizes from the system of regression equations with common (but random) parameters. The assumptions of the model are:

• E(ε_i) = 0_Nx1
  
• Var(ε_i) = E(ε_iε_i') = σ_i²I_NxN
  
• Cov(ε_i,ε_j) = 0_NxN, i≠j

• E(υ_i) = 0_Kx1
  
• Var(υ_i) = E(υ_iυ_i') = Γ_KxK (independent of i)
  
• Cov(υ_i,υ_j) = 0_KxK, i≠j
  
• Cov(υ_i,ε_i) = 0_Kx1

The model for estimation is

Y_i = X_iβ + (X_iυ_i + ε_i), or
Y_i = X_iβ + ω_i where ω_i = X_iυ_i + ε_i, and

• E(ω_i) = 0_Nx1
• Var(ω_i) = E(ω_iω_i') = Π_i
  = E(X_iυυ'X_i'+X_iυ_iε+ε_iυ_i'X_i+ε_iε_i') = σ_i²I_NxN + X_iΓX_i'

Then the stacked model is

Y = Xβ + ω

where ω = [ω₁,...,ω_G]', and

E(ω) = 0_GNx1

Var(ω) = E(ωω') = V =

⌈ | | ⌊

Π1 0 .. 0 0 Π2 .. 0 : : : : 0 0 .. ΠG

⌉  |  | ⌋

GLS is used to estimate the model. That is,

b^* = (X'V^-1X)^-1X'V^-1Y
Var(b^*) = (X'V^-1X)^-1

A special case of random coefficient model is to assume fixed slop coefficents but the intercept is random:

Y_ij = X_ijβ + ω_ij
ω_ij = υ_j + ε_ij

where i = 1,2,...G (equations) and j = 1,2,...,N (observations). For each observation j, υ_j is the random component of the intercept. We assume

• E(υ_j) = 0, Var(υ_j) = σ_u²
• E(ε_ij) = 0, Var(ε_ij) = σ_e²
• Cov(υ_j,ε_ij) = 0

The stacked form of the model is written as

Y = Xβ + ω
ω = ι⊗υ + ε

where ι is the unit vector of size G (the number of equations). Then
Var(ω) = Var(ι⊗υ+ε) = ιι'⊗Var(υ)+Var(ε) = ιι'⊗[σ_u²I_N]+σ_e²I_NG = V

The model can be estimated with GLS.

The computation of random coefficient model is based on the following steps (Swamy, 1971):

1. For each regression equation i, Y_i = X_iβ_i + ε_i, obtain the OLS estimator of β_i:
   b_i = (X_i'X_i)^-1X_i'Y_i
   Var(b_i) = (X_i'X_i)^-1(X_i'Π_iX_i)(X_i'X_i)^-1 = σ_i²(X_i'X_i)^-1+Γ = V_i+Γ
   (Taking account of heteroscedasticity, where V_i = σ_i²(X_i'X_i)^-1)
   Note that σ_i² is estimated by s²_i = e_i'e_i/(N-K), where e_i = Y_i - X_ib_i.
   Then, the estimate of V_i = s_i²(X_i'X_i)^-1.

2. For the random coeffcients equation, β_i = β + υ_i, the variance of b_i (estimator of β_i) is estimated by
   ∑_i=1,...,G(b_i-b^m)(b_i-b^m)'/(G-1) = ∑_i=1,...,G(b_ib_i'-G b^mb^m')/(G-1), where b^m = ∑_i=1,...,Gb_i/G.
   Therefore, Γ = ∑_i=1,...,G(b_ib_i'-G b^mb^m')/(G-1) - ∑_i=1,...,GV_i/G
   Concerning the possibility that Γ may be nonpositive definite, we use
   Γ = ∑_i=1,...,G(b_ib_i'-G b^mb^m')/(G-1).

3. Write the GLS estimator of β as:
   b^* = (X'V^-1X)^-1X'V^-1Y
   = [∑_i=1,...,GX_i'Π_iX_i]^-1 [∑_i=1,...,GX_i'Π_iY_i]
   = [∑_i=1,...,GX_i'Π_iX_i]^-1 [∑_i=1,...,GX_i'Π_iX_ib_i]
   = [∑_i=1,...,G(Γ+V_i)^-1]^-1 [(Γ+V_i)^-1b_i]
   = ∑_i=1,...,GW_ib_i, where W_i = [∑_i=1,...,G(Γ+V_i)^-1]^-1 [(Γ+V_i)^-1].
   Similarly,
   Var(b^*) = (X'V^-1X)^-1 = [∑_i=1,...,G(Γ+V_i)^-1]^-1

The individual parameter vectors may be predicted as follows:

b_i^* = (Γ+V_i)^-1[Γ^-1b^*+V_i^-1b_i] = A_ib^* + (I-A_i)b_i,
where A_i = (Γ+V_i)^-1Γ^-1.

Var(bi*) = [Ai

I-Ai]

⌈ ⌊

∑i=1,2,...,GWi(Γ+Vi)Wi'    Wi(Γ+Vi) (Γ+Vi)Wi'    (Γ+Vi)

⌉ ⌋

⌈ ⌊

Ai I-Ai

⌉ ⌋

Vector Autocorrelation Model

Y_t = μ + ρY_t-1 + ε_t

the mutivariate system of G variables can be written as follows:

Y_it = μ_i + ∑_j=1,2,...,G ρ_ijY_j,t-1 + ε_it (i=1,2,...,G)

This is called Vector Autocorrelation of order 1, or VAR(1). The matrix representation of the model as a simultaneous linear equations system looks like this:

[Y1t,Y2t,...,YGt] = [μ1,μ2,...,μG] + [Y1,t-1,Y2,t-1,...,YG,t-1]

⌈ | | ⌊

ρ11 ρ21 .. ρG1 ρ12 ρ22 .. ρG2 : : : : ρ1G ρ2G .. ρGG

⌉  |  | ⌋

+

[ε1,ε2,...,εG]

The alternative is the stacked form suitable for estimation as a system of regression equations:

⌈ | | ⌊

Y1t Y2t .. YGt

⌉  |  | ⌋

=

⌈ | | ⌊

μ1 μ2 .. μG

⌉  |  | ⌋

+

⌈ | | ⌊

ρ11 ρ12 .. ρ1G ρ21 ρ22 .. ρ2G : : : : ρG1 ρG2 .. ρGG

⌉  |  | ⌋

⌈ | | ⌊

Y1,t-1 Y2,t-1 .. YG,t-1

⌉  |  | ⌋

+

⌈ | | ⌊

ε1t ε2t : εGt

⌉  |  | ⌋

In a shorthand notation,

Y_t = μ + ρ Y_t-1 + ε_t

Extension: VAR(p)

Y_t = μ + ρ₁Y_t-1 + ρ₂Y_t-2 + ... +ρ_pY_t-p + ε_t
Y_t-1 = Y_t-1
Y_t-2 = Y_t-2
:
Y_t-p+1 = Y_t-p+1

Or,

⌈ | | ⌊

Yt Yt-1 : Yt-p+1

⌉  |  | ⌋

=

⌈ | | ⌊

μ 0 : 0

⌉  |  | ⌋

+

⌈ | | ⌊

ρ1 ρ2 .. ρp 1 0 .. 0 : : : : 0 .. 1 0

⌉  |  | ⌋

⌈ | | ⌊

Yt-1 Yt-2 : Yt-p

⌉  |  | ⌋

+

⌈ | | ⌊

εt 0 : 0

⌉  |  | ⌋

That is,

Y_t = μ + ρ Y_t-1 + ε_t

This is a system of p equations with restricted parameters matrix. The usable time series observations are from p+1 to N (N-p in total).

Similarly, for the multivariate VAR(p) system, the model can be expressed in terms of the stacked G endogenous variables. Therefore, Y_t, Y_t-1, ..., and Y_t-p are Gx1 vectors. The size of the problem is (N-p)Gp. Then the parameter matrix ρ of the lag variable Y_t-1 is

ρ =

⌈ | | ⌊

ρ1 ρ2 .. .. ρp I 0 .. .. 0 0 I : : 0 0 0 .. I 0

⌉  |  | ⌋

where, for each k = 1,2,...,p, ρ_k = [ρ_ij,k (i,j=1,2...,G)]. Furthermore, I is GxG identity matrix, and 0 is GxG zeros matrix.

Impulse Response Functions

[I-ρ(B)]Y_t = μ + ε_t

where B is the backshift operator. Then,

Y_t = [I-ρ]^-1μ + ∑_i=0,2...,∞ ρⁱε_t-i

= Y^* + (ε_t + ρ¹ε_t-1 + ρ²ε_t-2 + ...)

Y^* is the equilibrium and ε_t is the innovation. By shocking one element of ε_t, says ε_jt, Y_t will move away from the equilibrium Y^*. Note that the effect of Y_t due to change of ε_jt is not just on the jth variable alone but also on other variables in the system. The path whereby the variables returns to equilibirum is called the Impulse Responses of a stable VAR system. The Impulse Response Function traces the effects of a one-time innovation ε_jt on the k-th variable over time (i=0,1,2,...) as ρⁱ_kj (k,j = 1,2,...,G).