Note first that what we are discussing is an idealized theoretical construct which relies on a postulate called the "ergodic hypothesis". What that is will be left for later. Most importantly, however, just because you have a theoretical model in your hands it is not self-evident that this model can be applied to any specific problem in the real world. Physicist seems to be especially vulnerable to this kind of confusion between model and reality, not least when it comes to thermodynamics; It is not only in climate science that researchers complain about the arrogant thermometers who never seem to measure the correct temperature.
Ludwig Wittgenstein said that the purpose of philosophy was to clear out linguistic misunderstandings. Karl Popper, on the other hand, was somewhat more skeptic about the usefulness of this principle in science where he instead advocated that a scientific theory is nothing else than the set of its predictions. He saw a danger in the possible infinite regression of constantly dwelling over definitions. In this case, however, I think that we can indeed resolve matters with the Wittgensteinian approach.
Consider an arbitrary energetically isolated (adiabatic) thermodynamic system divided into two parts. Moreover, we impose the two measures entropy (S) and temperature (T) on this system: (S1, T1) for the first half and (S2, T2) for the second half. The entropy is supposed to be a so called "extensive" thermodynamic variable which means that the total entropy S is the sum of the entropies of the two subsystems respectively:
S = S1 + S2
The same does not hold for the temperature which is a so called "intensive" thermodynamic variable. Indeed, unless we are in equilibrium it is not even defined for the system as a whole. Now we define the temperature for each subsystem by means of the entropy as follows:
1/T = dS/dE
Or put into words, the inverse of the temperature is the (infinitesimal) rate of change of entropy per unit change of energy. Now suppose the following:
1. The system as a whole has reached a maximum possible entropy given the available energy.
2. T1 != T2 (Arbitrarily we may assume that T1 > T2)
Now imagine that subsystem 1 looses a small amount of energy \Delta E to subsystem 2. The total change in entropy can then be calculated as follows
In other words: Without adding any external energy to the system as a whole we have increased the total amount of entropy thus contradicting the first assumption.
Notice the very limited number of prerequisites. We didn't say anything about an ideal gas nor anything about a gravitational field. We didn't even define the entropy other than assuming that it was extensive. This is in fact so banal that we immediately realize the following: The modern definition of temperature is constructed in such a way that the two statements
1. The (energetically isolated) system has reached a maximum possible entropy given the fixed amount of energy
2. The temperature is the same everywhere in the system
are logically equivalent.
The paper of Coombes and Laue and related articles
According to the above analysis a "paradox" can only arise in the minds of physicists who tacitly introduce another definition of temperature (in this case for an ideal gas) and then assume that by some logical necessity this new definition must be the same as the old one. This is what I believe has happened here, that is why I gave this other definition of temperature the name "kinetic temperature" which is the ensemble average of the kinetic energy per constituent particle (omitting constant factors). If one performs the calculations one can indeed show that for the canonical ensemble of an ideal gas in the absence of a gravitational field the temperature (as defined by the Gibbs entropy) is indeed the kinetic temperature. The question you may now ask is the following
Do the distributions maximizing the Gibbs entropy for the canonical ensemble of an ideal gas in a gravitational field imply a uniform (constant) kinetic temperature?
This question has been analyzed rigorously and the answer appears to be yes. Nowhere does Cotton present any calculation or reference to show the opposite. The key to intuitively grasp this is to realize that the gravity make both the density and the pressure decline with height according to the barometric formula from which it follows that the temperature must be uniform. One should remember though that the isothermal column isn't the only hydrostatically stable column, but it is the one that maximizes the entropy.
There is a small mistake in the Coombes and Laue paper. They talk about an adiabatically closed system when, in fact, the derivation they are referring to pertains to the canonical ensemble. The canonical ensemble is not energetically closed but may exchange energy with an external reservoir kept at a constant temperature. The statistical ensemble corresponding to an energetically isolated system is instead called the "microcanonical ensemble". In fact, the microcanonical ensemble of an ideal gas in a gravitational field was dealt with relatively recently by S. Velasco et al. They concluded that in this case the kinetic temperature was not uniform for finite systems. The practical implications of this result is almost zero though. First of all there is no reason to assume that the atmosphere is an energetically isolated system, moreover the distributions in the microcanonical ensemble approaches those of the canonical ensemble very quickly in the thermodynamic limit. There is, however, an important didactic value in the sense that it shows that the "absolute" temperature need not be the same as the kinetic temperature for all systems.
The experiments of Graeff
These experiments pose a serious challenge to the standard wisdom if one assumes that the thermometer does indeed measure the kinetic temperature of the gas rather than simply "its own temperature". Since this needs to be verified separately I introduced another "definition" of temperature called the "empirical temperature" which is simply the reading of some particular thermometer. One of the most difficult problems with these kind of experiments is the question of how you verify that your system has reached equilibrium (that there is no tiny heat transport from the warmer to the colder parts). This must be taken on faith. In any case, the only thing these experiments can possibly show is that the ideal gas model with maximum Gibbs entropy is not valid for this experimental setup. Perhaps this is the case for the atmosphere as well.
Other "derivations" of the lapse rate
The "ergodic hypothesis" upon which the Gibbs entropy is based can be formulated like this
All microstates with equal energy are equally probable.
Somewhere on the path this statement seems to have got confused with something like
The total energy density is the same everywhere
leading to derivations based on the following assumption
potential energy + kinetic energy = constant
which (almost) leads to the adiabatic lapse rate. A simple thought experiment tells us that this cannot possibly hold for a finite atmosphere in an infinite space since that would imply an infinite amount of energy in the system. Also here we touch the problem of how to treat the atmospheric boundary.
The paper of Hans Jelbring and related articles
I and many other owe a great deal to Hans Jelbring for having brought to our attention the issue of the temperature gradient and its relevance to the climate debate. His various statements do however come in a rather incoherent form and do not, as yet, in my mind constitute a physical model of the atmosphere. The theory seems to come in two separate parts, on the one hand an assumption about a "static" gravitationally induced lapse rate and on the other hand a conjecture that the atmospheric mass is the single most important parameter in determining the elevation in surface temperature of the planets, as expressed in the title of his paper "Greenhouse Effect as a function of Atmospheric Mass". In hindsight one can see that there is something, perhaps unwittingly, catchy about this title. Notice in particular that it doesn't say "Pressure induced Greenhouse Effect" or anything like that, more about this soon. People who cannot tolerate any dissent to the greenhouse gas dogma immediately assume though that what is implied is some kind of pressure induced effect against which they can use a plethora of arguments including "the second law of thermodynamics" and "static air pressure cannot create heat" etc. There might be some justification for these arguments but it is somewhat ironic to see the same people embrace a theory that states that instead of gravity some magic gases create a temperature gradient in a system which would be isothermal in their absence What appears to be missing though in the Jelbring theory is an incorporation of the solar forcing (F) and a theory of the atmospheric boundary layer to produce a formula with which one can calculate the temperature field.
"Pressure/Gravity effect" versus "Blanket effect"
As I have argued on several posts there is another approach to the atmospheric mass conjecture which I here call the "blanket effect". Put in simple terms the atmosphere acts as a blanket whose effective "thickness" (L) is determined by its mass. If we treat the incoming sunlight as an energy source and assume, at a certain pressure, there is some effective boundary layer (whose temperature we put to zero for simplicity) then we can derive the following simple but illuminating formula:
T = F*L/k
where T is the temperature, F the solar forcing, L the effective length or thickness of the atmospheric blanket above that altitude and k is the thermal conductivity. The concept of effective atmospheric thickness can be made more rigorous which is done for example in the post "A Discrete Model Atmosphere" where the boundary layer is also taken care of. Since both the pressure and the "effective blanket thickness" are both proportional to the mass it is very easy to mix these up and confuse correlation with causation.
I am not completely alien to the concept of a pressure induced effect though. If we take the sun as an extreme example, I guess no physicist would consider treating the core plasma as an ideal gas.
Some answers to Doug Cotton's comments
"(a) There is no issue about the top of atmosphere or the stratosphere or thermosphere. These are regions where new absorption of incident Solar radiation dominates the much slower process of diffusion of kinetic energy. Also, the thermal gradient (aka "lapse rate') is -g/Cp where Cp is specific heat. But specific heat is only constant at a constant pressure and temperature. So the gradient approaches a limit of zero and never goes negative."
The latter part of this comment is the most confusing since it seems to assume that Cp approaches infinity. I cannot imagine the physical conditions under which it would do that.
"(b) The paper by Coombes et al simply is not based on the Second Law of Thermodynamics. They quite incorrect assume thermal equilibrium is implied by that law. It is not, and nothing in the law implies that it must be. The law states the thermodynamic equilibrium will evolve in a state of maximum accessible entropy."
In order to understand this comment I assume that Cotton advocates a second law of thermodynamics stating that
The spontaneous tendency of any thermodynamic system is to evolve towards an equilibrium characterized by a maximum accessible entropy
As I argued in the very beginning there exists a definition of temperature which makes the conditions "maximum entropy" and "constant temperature" logically equivalent. The quantity which I guess that Cotton is more interested in is the "kinetic" temperature. I state that
Cotton is correct in asserting that the second law he assumes valid does not imply a constant kinetic temperature for all systems.
Analysis has shown that for the canonical ensemble of an ideal gas in a gravity field it does.
What appears to be missing in Cottons arguments is some kind of physical model, especially one that incorporates the solar forcing and takes care of the atmospheric boundary level. Moreover I conclude that his claim that the adiabatic lapse rate maximizes the Gibbs entropy of a column of air in a gravity field are unsubstantiated.