Barometric altitude is determined by an outside pressure reading, a standard model of the atmosphere, and the barometric correction to the standard sea level pressure (29.92 inches of mercury, or 1013 millibars). Altitude calculated by this standard model (see article on this site) becomes more and more inaccurate as you rise above the location where the baro-correction is reported because the atmosphere is never "standard". If we all agree nonetheless to use that standard then we will be separated from other traffic by flying an assigned baro-altitude on IFR flights. But this does not guarantee separation from rocks in the sky, so we have generous margins in the minimum safe altitudes (MSA) published in our area.
With the advent of WAAS corrections to our GPS position, our 3D location is accurate to a few meters. But what is the reference for that altitude? It would be useless to refer your vertical position to the center of the earth, so instead a two step process is used to first give your GPS altitude above a reference ellipsoid that approximates the MSL surface, then apply a correction for your specific lat/lon location to get your local MSL altitude. So what is that ellipsoid surface, and what do we mean by the MSL surface of the earth? Obviously it is different from the actual surface which has mountains, valleys, and oceans. First let's talk about the gross shape of earth, and also define what is called the Geoid surface of the earth.
Since the earth spins, centrifugal force makes it bulge at the equator. It bulges less at higher latitudes and not at all at the poles (centrifugal force is zero there). Perhaps it's no surprise its basic shape is (very closely) an ellipse of revolution. So for GPS purposes an ellipsoid is defined for altitude reference, known as the WGS-84 reference ellipsoid. GPS altitude is first calculated from your 3D position solution as a height above the ellipsoid (HAE altitude). The major radius of this ellipsoid is 6,378.137 kilometers (note the typo in the drawing), and the minor radius is 6,356.752 kilometers, a difference of 21.4 kilometers between the earth's radius at the poles and the equator.
In contrast to the WGS-84 surface, the Geoid is a surface of equal gravitational potential. Gravitational force is a vector defined as the gradient of that potential. Contours of constant gravitational potential are analogous to constant pressure contours, and the gravitational force on mass is analogous to the pressure gradient force on air. Just as you can make a 2-D map of pressure contours, you can define a 3-D nested set of gravitional potential surfaces surrounding the center of the earth, starting near the center of the earth and extending past the MSL surface and continuing into outer space. But only one of these surfaces is the MSL surface. That one is defined by the fact that mean level of all the seas in the world lie on that surface even though different seas are at different levels relative to the center of the earth. These concepts are illustrated in the picture below.
It turns out that this geoid surface is roughly the same as the WGS-84 surface. The difference between HAE and MSL is known as the geoid separation or geoid height, and world maps of that separation can be found by searching on the web. The geoid has been very accurately mapped, most recently with the satellite called GOCE (Gravity Field and Steady-State Ocean Circulation Explorer). The Geoid height ranges from 105 meters below the ellipsoid in the oceans off the southern tip of India, to 80 meters above in the ocean between Scotland and Iceland and in the oceans around Indonesia and the Philippines, north of Australia. The map below shows contours of constant geoid height in the US, where the separation is typically 20 to 30 meters below the ellipsoid. These variations in MSL from the ellipsoid surface are the result of variations in density of the earth's crust and mantle, which locally affect gravity. Plumb bobs don't always point straight "down"; instead they are perpendicular to the Geoid surface, and the Geoid surface is irregular. The fact that plumb bobs don't point straight down was discovered during the survey of India that began in 1800 and led to the naming of Mt. Everest. A book called The Great Arc tells this fascinating story.
This data of Geoid height is stored in a lookup table in your GPS receiver, and by extrapolation from data for nearby points, an appropriate correction is added or subtracted from HAE to get MSL at your position. This is how your GPS determines your MSL altitude to within a few meters. Add a terrain database of MSL altitude on a world wide grid of small dimensions, and GPS altitude can be used for Terrain maps and displayed on your GPS. Areas 100 ft below you and higher are colored red to warn you of these sky rocks, and yellow areas are 100 - 1000' below you.
So, in the immortal words of Paul Harvey, " now you know the rest of the story".