3. The Many Body Problem and Density Functional Theory
In this chapter we shall take a look at perhaps the most fundamental difficulty in condensed matter theory (the Many Body problem) and at a particularly successful way of avoiding it (Density Functional Theory, or DFT). We shall see in what situations DFT can give meaningful results and also in which situations it fails. The purpose of this work as a whole is to look at ways of moving beyond DFT, but a thorough understanding of the starting point is, of course, essential....
So what is a many-body situation, and why is it such a problem? Let us start with a classical example. Consider a hard ball (ball A) moving in some kind of force field which may be spatially and temporally varying, but which is uncoupled to the motion of the ball (i.e. unaltered by the presence of the ball). If we know the force on the ball at every point in space and time then we can easily calculate the trajectory of the ball by using Newton's laws. This situation is that of a single particle moving in an external field. It doesn't matter that the field is varying; as long as the variation does not depend upon the position or velocity of the ball the problem remains easy.
Now add a second hard ball (ball B) to the problem, and furthermore let us attach it to the first by means of a spring. At a ball separation equal to the natural length of the spring there will be no force between the balls, but in general the spring will either be stretched or compressed and so ball A will exert a force on ball B and vice versa. The motion of the balls is still described perfectly well by Newton's classical laws of motion, but now the motion of ball A is intimately linked with the motion of ball B. That is, one cannot solve the trajectories of ball A and ball B separately: their equations of motion are coupled and must be solved simultaneously. Fortunately, with two balls this is not too much of a problem. However, if there are many balls then the problem rapidly becomes insoluble. It should be stressed that the balls are still subject to Newton's laws, but the difficulty of solving the equations which result from those laws increases rapidly because of the coupling. This is the essence of the Many Body Problem.
However, all is not doom and gloom. Imagine that the balls are not connected by springs, what happens then? The answer is that the balls move precisely as single particles until they collide with each other. Now the problem has become easier again, because we can use single particle theory to describe the trajectories of individual balls, until such times as they instantaneously scatter off each other. Thus we don't have to solve any equations simultaneously. Instead we have to know a bit about scattering theory. Now, it would not be correct to describe these balls simply as single particles because that would be to ignore the effect of the scattering, but we can call them single-particle-like particles or, to use the accepted term, quasiparticles.
Somewhere between these last two cases there is a grey area in which the balls interact with each other only when they are within a certain range of each other. That is, when the balls are well-separated, they do not significantly interact, but when they come close together they exert a force on each other. In the limit when the critical range is very small this situation reduces to the single-particle case, and in the opposite limit of a very large critical range the situation is that of a true many body problem. Thus the applicability of the quasiparticle approach to a given many body problem is largely dependent on the range of the inter-particle forces involved.
The electrons in a solid interact strongly both with the ion cores making up the crystal structure and with each other. The electron-ion interaction does not constitute a many body problem, however, because the ions are essentially stationary on the time-scale of the motion of the electrons (i.e. the electronic and ionic degrees of freedom are not coupled). On the other hand, the electron equations of motion are most certainly strongly coupled to each other by the electrostatic interaction. The long range nature of the electrostatic interaction does not bode well for a quasiparticle description. Nevertheless, the most successful current method for electronic structure calculations in solids is based on the even more radical approximation of single-particle behaviour. In order to understand the problem better let us look at some of the different approaches taken by different workers over the years....
To describe completely the quantum mechanical behaviour of electrons in solids it is strictly necessary to calculate the many-electron wavefunction for the system. In principle this may be obtained from the time-independent Schrodinger equation, but in practice the potential experienced by each electron is dictated by the behaviour of all the other electrons in the solid. Of course, the influence of nearby electrons will be much stronger than that of far-away electrons since the interaction is electrostatic in nature, but the fact remains that the motion of any one electron is strongly coupled to the motion of the other electrons in the system. To solve the Schrodinger equation directly for all these electrons would thus require us to solve a system of around 10^23 simultaneous differential equations. Such a calculation is beyond the capabilities of present-day computers, and is likely to remain so for the foreseeable future.
One of the earliest attempts to solve the problem was made by Hartree. He simplified the problem by making an assumption about the form of the many-electron wavefunction, namely that it was just the product of a set of single-electron wavefunctions. In a uniform system these wavefunctions would take the form of simple plane waves. Having made this assumption it was possible to proceed using the variational principle.
This principle is a very powerful concept in mathematics. In the form most commonly applied to theoretical physics it states that if a given system may be described by a set of unknown parameters then the set of parameter values which most correctly describes the ground state of the system (i.e. the state in which the system exists when not perturbed by outside influences) is just that set of values which minimises the total energy. A simple example of this principle is the case of a ball sitting in a valley in a gravitational field. The system may be described by a single parameter: the height of the ball above the bottom of the valley. The ground state of the system may be determined by finding the value of this parameter which minimises the total energy. Clearly this value of the parameter corresponds to the ball sitting at the very bottom of the valley.
By using the variational method Hartree found the Hamiltonian equation of the many-electron system (just a fancy name for the equation of motion). In fact, for an N-electron system there are N equations; one for each of the N single-electron wavefunctions which made up the many-electron product wavefunction. These equations turned out to look very much like the time-independent Schrodinger equation, except the potential (the Hartree potential) was no longer coupled to the individual motions of all the other electrons, but instead depended simply upon the time-averaged electron distribution of the system. This important fact meant that it was possible to treat each electron separately as a single-particle. Consequently the Hartree approximation allows us to calculate approximate single-particle wavefunctions for the electrons in crystals, and hence calculate other related properties. Unfortunately, the Hartree approximation does not provide us with particularly good results. For example, it predicts that in a neutral uniform system there will be no binding energy holding the electrons in the solid. This, of course, is in direct contradiction to the experimental evidence that electrons must be given a finite amount of energy before they can be liberated from solids.
The most obvious reason for the failure of the Hartree approach lies in the initial assumption of a product wavefunction. The famous Pauli exclusion principle states that it is not possible for two fermions (the class of particles to which electrons belong) to exist at the same point in space with the same set of quantum numbers (the parameters which define a particle's quantum mechanical state). This principle is manifest as an effective repulsion between any pair of identical fermions possessing the same set of quantum numbers. Mathematically, the Pauli exclusion principle can be accounted for by ensuring that the wavefunction of a set of identical fermions is antisymmetric under exchange of any pair of particles. That is to say that the process of swapping any one of the fermions for any other of the fermions should leave the wavefunction unaltered except for a change of sign. Any wavefunction possessing that property will tend to zero (indicating zero probability) as any pair of fermions with the same quantum numbers approach each other. The Hartree product wavefunction is symmetric (i.e. stays precisely the same after interchange of two fermions) rather than antisymmetric, so the Hartree approach effectively ignores the Pauli exclusion principle!
The Hartree-Fock approach is an improvement over the Hartree theory in that the many-electron wavefunction is specially constructed out of single-electron wavefunctions in such a way as to be antisymmetric. The wavefunction has to be much more complicated than the Hartree product wavefunction, but it can be written in a compact way as a so-called Slater determinant (for those who know what a determinant is).
Starting from this assumption it is once again possible to derive the Hamiltonian equation for the system through the variational principle. Just as before, this results in a simple equation for each single-electron wavefunction. However, this time in addition to the Hartree potential (which described the direct Coulomb interaction between an electron and the average electron distribution) there is now a second type of potential influencing the electrons, namely the so-called exchange potential. The exchange potential arises as a direct consequence of including the Pauli exclusion principle through the use of an antisymmetrised wavefunction.
We can get a visual impression of the effect of exchange by considering the region surrounding a given electron with a particular quantum mechanical spin. In this context spin is a quantum mechanical property (in some ways analogous to mechanical spin) which is related to magnetism. Electrons can be either in the spin-up state or the spin-down state (loosely you could think of spin-up electrons spinning clockwise, say, with spin-down electrons spinning anticlockwise) . In a non-magnetic sample half of the electrons will be spin-up while the other half will be spin-down. If we look at an electron with spin-up, then the Pauli exclusion principle means that other nearby spin-up electrons will be repelled. Spin-down electrons will not be affected since they have a different spin quantum number. Thus our spin-up electron is surrounded by a region which has been depleted of other spin-up electrons. Thus this region is positively charged (remember that the average electron distribution exactly balances the positive charge of the ion cores, and that this region is relatively depleted of electrons). Similarly, if we had considered a spin-down electron from the start, then we would have found a region depleted of other spin-down electrons. The edge of the electron depleted region is not clearly defined, but nevertheless we call this region the exchange hole.
Notably, the exchange potential contributes a binding energy for electrons in a neutral uniform system, so correcting one of the major failings of the Hartree theory. However, in calculating many other properties the Hartree-Fock approach is actually worse than the Hartree approach. How is it that improving the physics which went into the theory can actually give us worse answers?
The reason the Hartree-Fock theory gives worse answers than the Hartree theory is simply that there is another piece of physics which we are still ignoring. To some extent it cancels out with the exchange effect and so when we use the Hartree approach (i.e. we ignore both effects) we get reasonable results. On the other hand the Hartree-Fock approach includes the exchange effect but ignores the other effect, which balances it somewhat, completely. This new effect is the electrostatic correlation of electrons....
Ignoring the Pauli exclusion principle generated exchange hole for the moment, we can also visualise a second type of hole in the electron distribution caused by simple electrostatic processes. If we consider the region immediately surrounding any electron (spin is now immaterial) then we should expect to see fewer electrons than the average, simply because of their electrostatic repulsion. Consequently each electron is surrounded by an electron-depleted region known either as the Coulomb hole (because of its origin in the electrostatic interaction) or the correlation hole (because of it origin in the correlated motion of the electrons). Just as in the case of the exchange hole the electron depleted region is slightly positively charged. The effect of the correlation hole is twofold. The first is obviously that the negatively charged electron and its positively charged hole experience a binding force due to simple electrostatics. The second effect is more subtle and arises because any entities which interact with the electron over a length scale larger than the size of the correlation hole will not interact with the bare electron but rather with the electron+correlation hole (which of course has a smaller magnitude charge than the electron alone). Thus any other interaction effects, such as exchange, will tend to be reduced (i.e. screened) by the correlation hole.
Clearly we can now see why the Hartree-Fock approach fails for solids: firstly the exchange interaction should be screened by the correlation hole rather than acting in full, and secondly the binding between the correlation hole and electron has been ignored. At this point I should mention that the Hartree-Fock approach gives quite creditable results for small molecules. This is because there are far fewer electrons involved than in a solid, and so correlation effects are minimal compared to exchange effects.
Although neither of the above methods succeeded in solving the many-body problem of electrons in solids they did at least elucidate the important physical processes which we must describe: namely exchange and correlation. The breakthrough which revolutionised the field came in 1964, and we shall meet these two concepts again when we examine this watershed in the next section....
The methods of the previous section were both essentially based upon the variational principle. As already mentioned, this is an extremely powerful approach, but it depends for its success upon a good parametric description of the problem in the first instance. To go back to the "ball in the valley" example from the previous section it is clear that choosing the height of the ball as our adjustable parameter is a good idea, but that choosing the ambient temperature instead is not. We can vary the temperature all we like, but still get no nearer to finding a solution. Similarly, in quantum mechanics, the better the form of wavefunction we take (i.e. the more suited it is to describing the problem at hand) then the better results we will get from the variational approach. Thus the Hartree-Fock antisymmetric wavefunction allowed us correctly to describe the exchange interaction, whereas the Hartree product wavefunction did not. Accordingly we can view the ultimate failure of both of these models as being due to not having good enough approximate forms for the wavefunction to start with. This is hardly surprising, since we assumed that the many-electron wavefunction was expressible in terms of many single-electron wavefunctions, which is not necessarily possible at all.
Hohenberg and Kohn, in 1964, suggested that the problem really was that the many-electron wavefunction was too complicated an entity to deal with as the fundamental variable in a variational approach. Firstly, it cannot adequately be described without ~10^23 parameters, and secondly it has the complication of possessing a phase as well as a magnitude. They chose instead to use the electron density as their fundamental variable. That is, they considered the ground state of the system to be defined by that electron density distribution which minimises the total energy. Furthermore, they showed that all other ground state properties of the system (e.g. lattice constant, cohesive energy, etc) are functionals of the ground state electron density. That is, that once the ground state electron density is known all other ground state properties follow (in principle, at least).
In 1965, Kohn and Sham showed that the Hamiltonian equation derived from this variational approach took a very simple form. The so-called Kohn-Sham equation is similar in form to the time-independent Schrodinger equation, except that the potential experienced by the electrons is formally expressed as a functional of the electron density. Again it is effectively a single-particle equation. In addition to the contribution from the electron-ion interaction, the electron-electron interaction potential is split for convenience into two parts: the Hartree potential, which we have met before, and an exchange-correlation potential, whose form is, in general, unknown. Application of this theory to real-life situations involves heavy computational effort.
For many years now, density functional theory has been used with great success to investigate the ground state properties of solids, both in the bulk form and at surfaces and interfaces (see, for example, my recent work on silicon surfaces and gallium arsenide surfaces). However, as a variational approach, it cannot reliably be used to provide information about excited states of the system (i.e. states other than the ground state). In particular, density functional calculations in semiconductors are well known to place the excited states of electrons (i.e. conduction band states) too close in energy to the excited states of holes (i.e. valence band states), resulting in predicted band gaps which are 50-100% too small.
In order to rectify this problem it is necessary to go beyond the single-particle approach epitomised by density funtional theory and instead utilise a quasiparticle approach, as described in the next chapter.