Keenan on McKitrick

Sep 29, 2014
Bishop Hill in Climate: Statistics, Climate: Surface

Doug Keenan has posted a strong critique of Ross McKitrick's recent paper on the duration of the pause at his own website. I am reproducing it here.

McKitrick [2014] performs calculations on series of global surface temperatures, and claims to thereby determine the duration of the current apparent stall in global warming. Herein, the basis for those calculations is considered.

---

Much of McKitrick [2014] deals with a concept known as a “time series”. A time series is any series of measurements taken at regular time intervals. Examples include the following: the maximum temperature in London each day; prices on the New York Stock Exchange at the close of each business day; the total wheat harvest in Canada each year. Another example is the average global temperature each year.

The techniques required to analyze time series are generally different from those required to analyze other types of data. The techniques are usually taught only in specialized statistics courses.

Assumptions
The calculations of McKitrick [2014] rely on certain assumptions. In principle, that is fine: some assumptions must always be made, when attempting to analyze data. A vital question is almost always this: what assumptions should be relied upon? The question is vital because the conclusions of the data analysis commonly depend upon the assumptions. That is, the conclusions of the analysis can vary greatly, depending upon the assumptions.

The problem with McKitrick [2014] is that it relies on assumptions that are wholly unjustified—and, worse, not even explicitly stated. Hence, I e-mailed McKitrick, saying the following.

The analysis in your paper is based on certain assumptions (as all statistical analyses must be). One problem is that your paper does not attempt to justify its assumptions. Indeed, my suspicion is that the assumptions are unjustifiable. In any case, without some justification for the assumptions, there is no reason to accept your paper's conclusion.

The issue here is not specific to statistics. Rather, it pertains to research generally: in any analysis, whatever assumptions are made need to be justified.

McKitrick replied, claiming that “The only assumption necessary is that the series [of temperatures] is trend stationary”. The term “trend stationary” here is technical, and is discussed further below.

There are two problems with McKitrick's claim. The first problem is that trend stationarity is not the only assumption made by his paper. The second problem is that the assumption of trend stationarity is unjustified and seemingly unjustifiable. The next sections consider those problems in more detail.

The assumption of linearity
McKitrick claimed that his paper only made one assumption, about trend stationarity. In fact, the paper also assumes that all the relevant equations (for the noise) should be linear. Hence, I e-mailed McKitrick back, saying the following.

Stationarity is not the only assumption. Your paper also includes some assumptions about linearity … I do not see how [linearity] can be justified….

McKitrick did not respond. Five days later, I sent another e-mail, again raising the problem of assuming linearity. This time McKitrick replied at length. His reply, however, did not mention linearity.

The climate system is nonlinear. This is accepted by virtually everyone who has done research in climatology. For example, the IPCC has previously noted that “we are dealing with a coupled non-linear chaotic system” [AR3, Volume I: §14.2.2.2]. Hence the assumption of linearity is very dubious. There might be occasions where it is suspected that a linear approximation is appropriate, but if so, then some argument for the appropriateness must be given.

The assumption of trend stationarity
For technical details of what it means for a time series to be trend stationary, see the Wikipedia article. This section considers issues that do not require those details.

McKitrick's first e-mail to me acknowledged that trend stationarity “makes an enormous difference for defining and interpreting trend terms”. Simply put, if the trend in global temperatures is not assumed to be trend stationary, then the calculations of McKitrick [2014] are not valid.

The abstract of McKitrick [2014] states that the calculations used in the paper are “valid as long as the underlying series is trend stationary, which is the case for the data used herein” (emphasis added). The emphasized claim seems to imply that trend stationarity of the temperature data is an established fact.

The body of the paper says that the temperature data is “assumed to be trend-stationary”. The paper makes no attempt to justify the assumption. At least, though, the body of the paper acknowledges that trend stationarity is an assumption, rather than a fact.

McKitrick's first e-mail to me said that “decisive tests [for trend stationarity] are difficult to construct”. Thus, McKitrick seems to be acknowledging that he has no decisive statistical tests to justify the assumption of trend stationarity.

McKitrick's first e-mail also referred to a workshop, held in 2013, at which “there were extended discussions on whether global temperature series are stationary or not”. Thus, this effectively acknowledges that McKitrick knows trend stationarity is nowhere near being an established fact.

McKitrick's second e-mail attempted some justification for assuming trend stationarity. It said this: “The reason I do not accept the nonstationarity model for temperature is that it implies an infinitely large variance, which is physically impossible, and also that the climate mean state can wander arbitrarily far in any direction, which does not accord with life on Earth”. The first claim, about “an infinitely large variance”, is false; so it will not be discussed further here. The second claim, about how “the climate mean state can wander arbitrarily far in any direction”, is true in principle.

To understand McKitrick's second claim, first note that for “climate mean state” it is enough to consider simply “global temperature”. If the global temperature were truly non-stationary, then it could indeed wander arbitrarily far, up and down; i.e. it could become arbitrarily hot and arbitrarily cold. We know that global temperatures do not vary that much. Hence, global temperatures cannot be non-stationary—this is McKitrick's argument.

McKitrick's argument is easily seen to invalid. Consider a straight line (that is not perfectly horizontal). The straight line goes arbitrarily far up and arbitrarily far down—i.e. arbitrarily far in both directions. A straight line, though, is the basis for the calculations of McKitrick [2014]. Thus, if McKitrick's argument were correct, it would invalidate the basis for McKitrick's own paper.

McKitrick's argument against non-stationarity was raised earlier, by someone else, on the Bishop Hill blog. In response, an anonymous commenter (Nullius in Verba) left a perspicacious comment. The comment is excerpted below.

… everyone agrees that a non-stationary … process is not physically possible for temperature, in exactly the same way as they agree that a non-zero linear trend isn't physically possible. If you extend a non-zero trend forwards or backwards in time far enough, you'll eventually wind up with temperatures below absolute zero in one direction, and temperatures hotter than the sun's core in the other. For the *actual* underlying process to be a linear trend is physically and logically impossible.

However, nobody objects on this basis because everybody knows it is only being used as an approximation that is only considered valid over a short time interval. ….

In exactly the same way, a non-stationary … process is being used as an approximation to a stationary one, and is only considered valid over a short time interval. It arises for exactly the same reason….

Statisticians use non-stationary [models] routinely for variables that are known to be bounded, for very good reason. They're not stupid.

Additionally, McKitrick's argument is an appeal to physics. Yet using physics to exclude a statistical assumption is inherently very dubious. For some elaboration on this, see the Excursus below.

As noted above, several researchers have contended that non-trend-stationarity might be an appropriate assumption for global temperatures. An early paper making that contention is by Woodward & Gray [1995]. That paper currently has 68 citations on Google Scholar, including several since 2013. (One of the latter even presents a physics-based rationale for non-trend-stationarity: Kaufmann et al. [2013].)

There are other papers that do not cite Woodward & Gray, but which also contend for considering non-trend-stationarity; e.g. the paper of Breusch & Vahid [2011]—which is part of the Australian Garnaut Review. Such contending has even appeared in an introductory textbook on time series: Time Series Analysis and Its Applications [Shumway & Stoffer, 2011: Example 2.5; see too set problems 3.33 and 5.3]. Contentions for non-trend-stationarity would not appear in so many respected sources, over so many years, if McKitrick's appeal to a simple physical argument had merit.

It is worth reviewing how McKitrick's story on trend stationarity of the global temperature series changed. First, the abstract of the paper claimed that the temperatures are trend stationary—seemingly an established fact. Second, the body of the paper mentions, in one sentence, that trend stationarity is actually an assumption, rather than a fact—but it gives no justification for the assumption. Third, McKitrick's first e-mail acknowledged that there have been no tests to justify the assumption and also that the validity of the assumption is debated. Fourth, McKitrick's second e-mail, in response to my criticisms of the foregoing, attempted some justifying of the assumption—but with a justification that is easily seen to be invalid, as well as not supported by many other researchers who have studied the issue.

Statistical models
Whenever data is analyzed, we must make some assumptions. In statistics, the assumptions, collectively, are called a “statistical model”. There has been much written about how to select a statistical model—i.e. about how to choose the assumptions.

This issue is noted by the book Principles of Statistical Inference (2006). The book's author is one of the most esteemed statisticians in the U.K., Sir David Cox. The book states this: “How [the] translation from subject-matter problem to statistical model is done is often the most critical part of an analysis”. In other words, choosing the assumptions is often the difficult part of a statistical analysis.

Another book that is relevant is Model Selection [Burnham & Anderson, 2002]. This book currently has about 25 000 citations on Google Scholar—which seems to make it the most-cited statistical research work published during the past quarter century. The book states the following (§8.3).

Statistical inference from a data set, given a model, is well advanced and supported by a very large amount of theory. Theorists and practitioners are routinely employing this theory … in the solution of problems in the applied sciences. The most compelling question is, “what model to use?” Valid inference must usually be based on a good approximating model, but which one?

The book also refers to the question “What is the best model to use?” as the critical issue (§1.2.3).

The selection of a statistical model tends to be especially difficult for time series. Indeed, one of the world's leading specialists in time series, Howell Tong, stated the following, in his book Non-linear Time Series (§5.4).

A fundamental difficulty in statistical analysis is the choice of an appropriate model. This is particularly pronounced in time series analysis.

Note that, in making the statement, Tong does not assume that time series are linear—as the title of his book makes clear.

Concluding remarks
What McKitrick [2014] has done is skip the difficult part of statistical analysis. That is, McKitrick does not genuinely consider the choice of statistical assumptions. Instead, he just picks some assumptions, with negligible justification, and then does calculations.

Realistically, then, McKitrick [2014] does not present a statistical analysis—because the paper is missing a required part. If McKitrick had been forthcoming about this, that would have been fine. For example, suppose McKitrick had included a disclaimer like the following.

The calculations in this work rely on assumptions: about linearity and trend stationarity (and normality). Those assumptions are unjustified and might well be unjustifiable. Relying on different assumptions might well lead to conclusions that are very different from the conclusions of this work. Hence, the conclusions of this work should be regarded as highly tentative.

Such a disclaimer would have been fair and honest. Instead, the paper, especially the abstract, greatly misleads: and McKitrick must have known that it does so.

Finally, methods to detect trends in global temperatures have been studied by the Met Office. A consequence of the study is that “the Met Office does not use a linear trend model to detect changes in global mean temperature” [HL969, Hansard U.K., 2013–2014].

Excursus: Realistic models?
A statistical model does not need to be physically realistic. An example will illustrate this. Suppose that we have a coin. We toss the coin a few times, with the outcome of each toss being either Heads or Tails. We might then make two assumptions. First, the probability of the coin coming up Heads is ½. Second, the result of one toss is unaffected by the other tosses.

The two assumptions comprise our statistical model. The assumptions obviously elide many physical details: they do not tell us what type of coin was used, how long each toss took, the path of the coin through the air, etc. The assumptions, though, should be enough to allow us to analyze the data statistically.

The set of assumptions—i.e. the model—also differs from reality. For instance, our assumption that a coin comes up Heads with probability ½ is only an approximation. In reality, the two sides of a coin are not exactly the same, and so the chances that they come up will not be the same. It might really be, for instance, that the probability that a coin comes up Heads is 0.500001 and the probability that it comes up Tails is 0.499999. Of course, in almost all practical applications, this difference will not matter, and our assumption of a probability of ½ will be fine.

There is also a second way in which our model of a coin toss differs from reality. We can predetermine the outcome of a toss by measuring the position of the coin prior to the toss, measuring the forces exerted on the coin at the start of the toss, and determining the air resistances as the coin was about to go through the air (all this is in principle; in practice, it might not be feasible [Strzalko et al., 2010]). Thus, a real toss is deterministic: it is not random at all. Yet we modelled the outcome of the toss as being random.

This second way in which our model differs from reality—incorporating randomness where the actual process is deterministic—is fundamental. Yet, by modelling the outcome of a coin toss as random, our model is vastly more useful than it would be if we modelled the toss with realistic determinism (i.e. with all the physical forces, etc., that control the outcome of the toss). Indeed, statistics textbooks commonly model a coin toss as being random. Moreover, people have probably been treating a coin toss as random for as long as there have been coins.

To summarize, we model a coin toss as a random event with probability ½, even though we know that the model is physically unrealistic. This exemplifies a maxim of statistics: “all models are wrong, but some are useful”. The maxim seems to be accepted by all statisticians (as well as being intuitively clear). McKitrick, by appealing to a supposed lack of physical realism of non-stationary models, ignores that.

❧  A draft of this Comment was sent to Ross McKitrick; McKitrick acknowledged receipt, but had nothing to say on the technical issues.

 

See also

•	Is a line trending upward?

---

 

Article originally appeared on (http://www.bishop-hill.net/).