Forecast combinations

Содержание

Слайд 2

Lecture Objectives Introduce the idea and rationale for forecast averaging Identify

Lecture Objectives

Introduce the idea and rationale for forecast averaging
Identify

forecast averaging implementation issues
Become familiar with a number of forecast averaging schemes
Слайд 3

Introduction Usually, multiple forecasts are available to decision makers Differences in

Introduction

Usually, multiple forecasts are available to decision makers
Differences in

forecasts reflect:
differences in subjective priors
differences in modeling approaches
differences in private information
It is hard to indentify the true DGP
should we use a single forecast or an “average” of forecasts?
Слайд 4

Introduction Disadvantages of using a single forecasting model: may contain misspecifications

Introduction

Disadvantages of using a single forecasting model:
may contain misspecifications of

an unknown form
e.g., some variables are missing
one statistical model is unlikely to dominate all its rivals at all points of the forecast horizon
Combining separate forecasts offers :
a simple way of building a complex, more flexible forecasting model to explain the data
some insurance against “breaks” or other non-stationarities that may occur in the future
Слайд 5

Outline of the lecture What is a combination of forecasts? The

Outline of the lecture

What is a combination of forecasts?
The theoretical problem

and implementation issues
Methods to assign weights
Improving the estimates of the theoretical model performance
Conclusion – Key Takeaways
Слайд 6

Part I. What is a combination of forecasts? General framework and

Part I. What is a combination of forecasts?
General framework and notation
The forecast

combination problem
Issues and clarifications
Слайд 7

General framework Today (at time T) we want to forecast the

General framework

Today (at time T) we want to forecast the

value of (at T+h)
We have M different forecasts:
model-based (econometric model, or DSGE), or judgmental (consensus forecasts)
the model(s) or judgment(s) are our own or of others
some models or information sets might be unknown: only the end product – forecasts – are available
How to combine M forecasts into one forecast?
Is there any advantage in combining vs. selecting the “best” among the M forecasts?
Слайд 8

Notation is the value of Y at time t (today is

Notation

is the value of Y at time t (today is

T )
h is the forecasting horizon
is an unbiased (point) forecast of at time T
m= 1,…,M the indices of the available forecasts/models
is the forecast error of model m
is the forecast error variance
covariance of forecast errors
is a vector of weights
L(et+h) is the loss from making a forecast error
E{L(et+h)} is the risk associated with a forecast
Слайд 9

Interpretation of loss function L(e) Squared error loss (mean squared forecasting

Interpretation of loss function L(e)

Squared error loss (mean squared forecasting error:

MSFE)
equal loss from over/under prediction
loss increases quadratically with the error size
Absolute error loss (mean absolute forecasting error: MAFE)
equal loss from over/under prediction
proportional to the error size
Linex loss (γ>0 controls the aversion against positive errors, γ<0 controls the aversion against negative errors)
Слайд 10

A combined forecast is a weighted average of M forecasts: The

A combined forecast is a weighted average of M forecasts:
The

forecast combination problem can be formally stated as:
Note: Here we assume MSFE-loss, but it could be any other

Problem 1: Choose weights wT,h,i to minimize the loss function subject to

The forecast combination problem

See Appendix 1 for generalization

Слайд 11

Clarification: combining forecasting errors Notice that since then Hence, if weights

Clarification: combining forecasting errors

Notice that since then

Hence, if weights sum to

one, then the expected loss from the combined forecast error is
Слайд 12

Summary: what is the problem all about? (II) We want to

Summary: what is the problem all about? (II)
We want to find

optimal weights (the theoretical solution to Problem 1)
How can we estimate optimal weights from a sample of data?
Are these estimates good?

Problem 1: Choose weights wT,h,i to minimize the loss function subject to

Слайд 13

General problem of finding optimal forecast combination Let: u an (M

General problem of finding optimal forecast combination

Let:
u an (M

x 1) vector of 1’s,
and Σ the (M x M) covariance matrix of the forecast errors

It follows that
For the MSFE loss, the optimal w’s are the solution to the problem:
To find optimal weights it is therefore important to know (or have a “good” estimate) of Σ

Слайд 14

Issues and clarifications Do weights have to sum to one? If

Issues and clarifications

Do weights have to sum to one?
If forecasts

are unbiased, this guarantees unbiased combination forecast
Is there a difference between averaging across forecasts and across forecasting models?
If you know the models and the models are linear in parameters, there is no difference
Is it better to combine forecasts rather than information sets?
Combining information sets is theoretically better*
practically difficult’/impossible: if sets are different, then the joint set may include so many variables that it will not be possible to construct a model that includes all of them
* Clemen (1987) shows that this depends on the extent to which information is common to forecasters
Слайд 15

Summary: what is the problem all about? (I) Observations of a

Summary: what is the problem all about? (I)

Observations of a variable

Y
Forecast observations of Y:
forecast 1

forecast M
Forecasting errors
Question: how much weight to assign to each of forecasts, given past performance and knowing that there will be a forecasting error?

?

Слайд 16

Part II. The theoretical problem and implementation issues A simple example

Part II. The theoretical problem and implementation issues
A simple example with only

2 forecasts
The general N forecast framework
Issue 1: do weights sum to 1?
Issue 2: are weights constant over time?
Issue 3: are estimates of weights good?
Слайд 17

Optimal weights in population (M = 2) Result 1: The solution

Optimal weights in population (M = 2)

Result 1: The solution to

Problem 1 is

weight of

weight of

Assume we have 2 unbiased forecasts (E(eT+h,m) = 0) and combine:

Слайд 18

Interpreting the optimal weights in population Consider the ratio of weights

Interpreting the optimal weights in population

Consider the ratio of weights
A

larger weight is assigned to a more precise forecast
If the covariance of the two forecasts increases, a greater weight goes to a more precise forecast
The weights are the same (w = 0.5) if and only if
This is similar to building a minimum-variance-portfolio (finance)
See Appendix 2: a generalization to M>2
Слайд 19

Result: Forecast combination reduces error variance Compute the expected MSFE with

Result: Forecast combination reduces
error variance

Compute the expected MSFE with the

optimal weights:

|ρ| ≤ 1 Is the correlation coefficient


Result 2:
The combined forecast error variance is lower than the smallest of the forecasting error variances of any single model

Suppose (forecast 1 is more precise), then:

(see Appendix 3)

Слайд 20

Estimating Σ The key ingredient for finding the optimal weights is

Estimating Σ

The key ingredient for finding the optimal weights is

the forecast error covariance matrix, e.g. for M=2:
In reality, we do not know the exact Σ:
we can only estimate (and then the weights) using past record of forecasting errors
Слайд 21

Issues with estimating Σ Is the estimate of based on the

Issues with estimating Σ

Is the estimate of based on the

past forecasting errors “good”?
If forecasting history is short, then may be biased
may or may not depend on t (e.g., a model/forecaster m may become better than others over time – smaller )
If not, converges to as forecasting record lengthens
If it does, different issues: heteroskedasticity of any sort, serial correlation, etc.
If such issues are there, the seemingly “optimal” forecast based on the estimated might become inferior to other (simpler) combination schemes…
Слайд 22

Optimality of equal weights The simplest possible averaging scheme uses equal

Optimality of equal weights

The simplest possible averaging scheme uses equal

weights
The equal weights are also optimal weights if:
the variances of the forecast errors are the same
the pair-wise covariances of forecast errors are the same and equal to zero for M > 2
the loss function is symmetric, e.g. MSFE:
we are not concerned about the sign or the size of forecast errors

Empirical observation: Equal weights tend to perform better than many estimates of the optimal weights (Stock and Watson 2004, Smith and Wallis 2009)

Слайд 23

Part III. Methods to estimate the weights: M is small relative to T (M

Part III. Methods to estimate the weights:
M is small relative to

T (M<
Слайд 24

To combine or not to combine? Assess if one forecast encompasses

To combine or not to combine?

Assess if one forecast encompasses

information in other forecasts
For MSFE loss, this involves using forecast encompassing tests
Example: for 2 forecasts, estimate the regression
If you cannot reject…
… there is no point in combining – use one of the models
Rejection of H0 implies that there is information in both forecasts that can be combined to get a better forecast

→ forecast 1 encompasses 2

→ forecast 2 encompasses 1

Слайд 25

OLS estimates of the optimal weights Recall the general problem of

OLS estimates of the optimal weights

Recall the general problem of

estimating wm for m forecasts (slide 12)
We can use OLS to estimate the wm‘s that minimize the MSFE (Granger and Ramanathan -1984):
we use history of past forecasts over t = 1,…,T–h and m=1,…,M to estimate
or
including intercept w0 takes care of a bias of individual forecasts
Слайд 26

Reducing the dependency on sampling errors Assume that estimate is affected

Reducing the dependency on sampling errors

Assume that estimate is affected

by a sampling error (e.g., is biased due to a short forecast record)
It makes sense to reduce the dependence of the weights on such a (biased) estimate
Can achieve this by “shrinking” the optimal weights w’s towards equal weights 1/M (Stock and Watson 2004)

Notice:
the parameter k determines the strength of the shrinkage
as T increases relative to M, the estimated (e.g., OLS) weights become more important:
Can you explain why?

Слайд 27

Part IV. Methods to estimate the weights: when M is large relative to T

Part IV. Methods to estimate the weights: when M is large relative

to T
Слайд 28

Premise: problems with OLS weights The problem with OLS weights: If

Premise: problems with OLS weights

The problem with OLS weights:
If M

is large relative to T–h the OLS estimates loose precision and may not even be feasible (if M > T–h)
Even if M is low relative to T–h, the OLS estimates of weights may be subject to a sampling error
the estimate may depend on the sample used
A number of other methods can be used when M is large relative to T
Слайд 29

MSFE weights (or relative performance weights) Relative performance weights An alternative

MSFE weights (or relative performance weights)

Relative performance weights

An alternative

to the of OLS weights:
ignore the covariance across forecast errors
compute weights based on past forecast performance

For each forecast compute

Слайд 30

Emphasizing recent performance Compute: where is the number of periods with

Emphasizing recent performance

Compute:

where is the number of periods with δ(t)>0

and δ(t) can be either

Such MSFE weights emphasize the recent forecasting performance

Using only a part of forecasting history
for forecast evaluation

Discounted MSFE

or

Слайд 31

Shrinking relative performance Consider instead As parameter k 0 the relative

Shrinking relative performance

Consider instead

As parameter k 0 the relative performance

of a particular model becomes less important

If k=1 we obtain standard MSFE weights
If k=0 we obtain equal weights 1/M

Слайд 32

MSFE weights ignore correlations between forecasting errors Ignoring it, when it

MSFE weights ignore correlations between forecasting errors
Ignoring it, when

it is present decreases efficiency – larger forecasting variance from the combined forecast
Consider instead
Note: this weighting scheme may be computationally intensive. For M models we need to calculate M(M+1)/2 different

The relative performance weights adjusted for covariance:

Performance weights with correlations

Слайд 33

Rank-based forecast combination Aiolfi and Timmerman (2006) allow the weights to

Rank-based forecast combination

Aiolfi and Timmerman (2006) allow the weights to

be inversely related to the rank of the forecast
The better is the forecast (e.g., according to MSFE) the higher is the rank rm
After all models are ranked form best to worst, the weights are:
Слайд 34

Trimming In forecast combination, it is often advantageous to discard models

Trimming

In forecast combination, it is often advantageous to discard models

with the worst and best performance (i.e., trimming)
This is because simple averages are easily distorted by extreme forecasts/forecast errors
Trimming justifies the use of the median forecast
Aiolfi and Favero (2003) recommend ranking the individual models by R2 and discarding the bottom and top 10 percent.
Слайд 35

Example Stock and Watson (2003): relative forecasting performance of various forecast

Example

Stock and Watson (2003): relative forecasting performance of various forecast

combination schemes versus the AR (benchmark)
Слайд 36

Part V. Improving the Estimates of the Theoretical Model Performance: Knowing the parameters in the model

Part V. Improving the Estimates of the Theoretical Model Performance: Knowing the

parameters in the model
Слайд 37

Question So far we assumed that we do not know models

Question

So far we assumed that we do not know models

from which forecasts originate
Would our estimates of the weights improve if we knew something about these models
e.g., if we knew the number of parameters?
Слайд 38

Hansen (2007) approach For a process yt there may be an

Hansen (2007) approach

For a process yt there may be an

infinite number of potential explanatory variables (x1t,x2t,…)
In reality we deal with only a finite subset (x1t,x2t,…,xNt)
Consider a sequence of linear forecasting models where model m uses the first km variables (x1t,x2t,…,xkt):
with bt,m the approximation error of model m:
and the forecast given by
Слайд 39

Hansen (2007) approach (2) Let be the vector of T-h (in-sample!)

Hansen (2007) approach (2)

Let be the vector of T-h (in-sample!) residuals

of model m
The {(T-h)xM} matrix collecting these residuals:
K = (k1,…, kM) is an Mx1 vector of the number of parameters in each model

The Mallow criterion is minimized with respect to w
where s2 is the largest of all models sample error variance estimator
The Mallow criterion is an unbiased approximation of the combined forecast MSFE:
Minimizing CT-h(w) delivers optimal weights w
It is a quadratic optimization problem: numerical algorithms are available (e.g., in GAUSS, QPROG; in Excel, SOLVER)

Слайд 40

Example of Hansen’s approach (M = 2) We need to find

Example of Hansen’s approach (M = 2)

We need to find

w that minimizes the Mallow criterion:
Minimizing gives:

The optimal weights
depend on the Var and Cov of residuals
penalize the larger model: the weight on the (first) smaller model increases with the size of the “larger” second model k2>k1
See appendix 7 for further details

Слайд 41

Conclusions – Key Takeaways Combined forecasts imply diversification of risk (provided

Conclusions – Key Takeaways

Combined forecasts imply diversification of risk (provided not

all the models suffer from the same misspecification problem)
Numerous schemes are available to formulate combined forecasts
For a standard MSFE loss, the payoff from using covariances of errors to derive weights is small
Simple combination schemes are difficult to beat
Слайд 42

Thank You!


Thank You!

Слайд 43

References Aiolfi, Capistran and Timmerman, 2010, “Forecast Combinations“, in Forecast Handbook,

References

Aiolfi, Capistran and Timmerman, 2010, “Forecast Combinations“, in Forecast Handbook, Oxford,

Edited by Michael Clements and David Hendry.
Clemen, Robert, 1985, “Combining Forecasts: A Review and Annotated Bibliography,” International Journal of Forecasting, Vol. 5, No. 4, pp. 559–583.
Stock, James H., and Mark W. Watson, 2004, “Combination Forecasts of Output Growth in a Seven-Country Data Set,” Journal of Forecasting, Vol. 23, No. 6, pp. 405–430.
Timmermann, Allan, 2006. "Forecast Combinations," Handbook of Economic Forecasting, Elsevier.
Слайд 44

Appendix

Appendix

Слайд 45

Appendix 1: generalization of problem 1 Let w be the (M

Appendix 1: generalization of problem 1

Let w be the (M x

1) vector of weights, e the (M x 1) vector of forecast errors, u an (M x 1) vector of 1s’, and Σ the (M x M) variance covariance matrix of the errors

It follows that

Problem 1: Choose w to minimize w’Σ w subject to u’w = 1.

Слайд 46

Result 1: Let u be an (M x 1) vector of

Result 1: Let u be an (M x 1) vector of

1s’ and ΣT,h the variance-covariance matrix of the forecast errors eT,h,i. The vector of optimal weights w’ with M forecasts is

Appendix 2: generalization of result 1

For the proof and to see how this applies when M = 2 see Appendix 1

Слайд 47

Appendix 2: generalization of result 1 Let e be the (M

Appendix 2: generalization of result 1

Let e be the (M x

1) vector of the forecast errors. Problem 1: choose the vector w to minimize E[w’ee’w] subject to u’w = 1.
Notice that E[w’ee’w] = w’E[ee’]w = w’Σw. The Lagrangean is

and the FOC is

Using u’w = 1 one can obtain λ

Substituting λ back one gives

Слайд 48

Appendix 2: generalization of result 1 (M = 2) Let Σt,h

Appendix 2: generalization of result 1 (M = 2)

Let Σt,h be

the variance-covariance matrix of the forecasting errors

Consider the inverse of this matrix

Let u’ = [1, 1]. The two weights w* and (1 - w*) can be written as

Слайд 49

Optimal weights in population (M = 2) Result 1: The solution

Optimal weights in population (M = 2)

Result 1: The solution to

Problem 1 is

weight of

weight of

Assume we have 2 unbiased forecasts (E(eT+h,m) = 0) and combine:

Слайд 50

Appendix 3 Notice that Need to show that the following inequality

Appendix 3

Notice that

Need to show that the following inequality holds

and that

Rearrange

the above
Слайд 51

Appendix 4: trading-off bias vs. variance The MSFE loss function of

Appendix 4: trading-off bias vs. variance

The MSFE loss function of

a forecast has two components:
the squared bias of the forecast
the (ex-ante) forecast variance
Combining forecasts offers a tradeoff: increased overall bias vs. lower (ex-ante) forecast variance
Слайд 52

Appendix 4 The MSFE loss function of a forecast has two

Appendix 4

The MSFE loss function of a forecast has two components:
the

squared bias of the forecast
the (ex-ante) forecast variance
Слайд 53

Appendix 5 Suppose that where P is an (m x T)

Appendix 5

Suppose that where P is an (m x T) matrix,

y is a (T x 1) vector with all yt , t = 1,…T. Consider:
Слайд 54

Appendix 5 Consider:

Appendix 5

Consider:

Слайд 55

Appendix 6: Adaptive weights Relative performance weights may be sensitive to

Appendix 6: Adaptive weights

Relative performance weights may be sensitive to

adding new forecast errors (may vary wildly)
We can use an adaptive scheme that updates previous weights by the most recently computed weights
E.g., for the MSFE weights (can use other weighting too):
The update parameter α controls the degree of weights update from period T-1 to period T