Creating a Map from Digital Terrain Data and Google Maps

[Gumboots]

Woohoo!

Nailed it. Took a bit of head scratching, because it's not the way I usually think of doing numbers, but it turns out that the formula behind the blue-green height scale is really simple. Stunningly doofus simple. Your cat could do it.

All it does is start at 0, of course, with an RGB value of 0 0 0.
It then proceeds one step at a time to 0 0 1, then 0 0 2, etc, all the way up to 0 0 255.
Then it starts again, with 0 1 0 taking the place of what would have been 0 0 256 (if the hex scale went that far).
It then goes 0 1 1, 0 1 2, all the way up to 0 1 255, at which point it starts again, at 0 2 0.

So, if you take the position on the scale, which for 0 1 0 is 256, then all you need to do to get the RT3 height value is multiply that by 0.7 and round it down to the nearest integer.

The upper limit is 10,000. That has a hex code of 0 55 206. So its position on the scale is 55(256)+206 = 14,286.
Multiply that by 0.7 and you get 10,000.2 and if you round that down as you normally would anyway you get 10,000.

Taking another example, there's hex code 0 3 127. Its position on the scale is 3(256)+127 = 895.
Multiply that by 0.7 and you get 626.5. I would normally round that up, but RT3 rounds it down, giving a height of 626 units.

This works perfectly for any value between 0 0 0 and 0 55 206. Which is a vast relief, because now I won't have to waste any more time screwing around making patches of colour in Photoshop and loading them into RT3. I can now do the entire humungous beastie via basic arithmetic, which is a lot faster and easier.

Edit: Lolz. Maybe the cat is smarter. I just twigged to how it could be written succinctly.
If you take the green channel value as G, and the blue channel value as B, the formula for finding height value for any value of G or B is:

ROUNDDOWN(0.7*((256*G)+B);0)

in standard spreadsheet syntax.

And another one. Got the formula for greyscale too. That one is even simpler.

Height = ROUNDDOWN(0.7*32*V;0)

where V equals the RGB value (which for greyscale is the same number for all three channels).

Note that this one tops out at 5712 height units, instead of the 10,000 units allowed by the Green-Blue scale.
Also note that all game maps imported from a greyscale heightmap will have a highest point of 5,712 units.

That's just the way greyscale works, and it means importing from a greyscale heightmap will require massively different height modifiers compared to importing from a green-blue scale heightmap. Also note that if you accidentally mix a red ocean with a greyscale heightmap it will multiply all heights by a factor of 8. Which tends to be rather surprising if you're not expecting it.

---

Just found out something. The first series of the blue scale, from 0 0 1 to 0 0 255, work perfectly as long as there is at least one pixel of pure red somewhere on the heightmap. They need that as a reference. If you make a patch of any of those 255 colours, without at least one square of red for a reference of 0, it will result in a height of 0. Not only that, but in the case of pure blue (0 0 255) it will automatically be made into an ocean! Just like as if it was pure red. Go figure.

All the other green-blue colours, from 0 1 0 up to 0 55 206, work perfectly with or without a pixel of red for reference. As soon as there is at least one unit of green, the red is no longer necessary. The obvious catch is that without that reference pixel the heights from 0 to 178 won't be available. This wouldn't matter for a map of an inland area that had a minimum altitude up over 178 anyway, but could be a consideration for low-lying but dry areas. If necessary it would be easy to incorporate just one pixel of red somewhere inconspicuous, and just fill in the divot later (lake tool reset would pull it into place easily).

Anyway, I made a proof of concept tile to see what this stuff is capable of. The TGA was 321x193 and looks like the attached .jpg. The result it generates in RT3 is shown in the screenshot. It's a perfect ramp, on an even gradient of about 1 in 85. No terracing at all. Even slope from top to bottom.

321_ramp_test.jpg

321_ramp_result.jpg

[Gumboots]

Ok, tried a test on an actual map.I started by making a custom elevation table with 5 metre resolution. Cropped a DEM, and exported it at its default resolution (ie: no resizing losses). Turned it into a suitable TGA (513x833) and imported into RT3 editor with an overall height modifier of 4.5. Hey presto, one map. This is an area .231 degrees north to south and 0.207 degree east to west, with a scale of about 52 pixels per mile. IOW, each pixel is about 30 metres (100 feet) square.

New_test_10m_res.jpg

This appears to be meshed at around 10 metre height resolution. This is a lot better than the 40 metre resolution we had before, but still not as good as it should be. Since I did the ramp test I know RT3 is capable of utilising all 14,286 possible height values, with height resolution down to 0.7 metres, so I checked the number of colours in the TGA. There are 32 colours.

The maximum elevation in the cropped area is around 280 metres, so there theoretically should be around 55 colours (give or take a few). IOW, the image as exported from MicroDEM only has about half the colour range that it should have. Not sure why at the moment, but will keep investigating. It's obviously some sort of problem with the custom elevation table, and hopefully will not be too hard to track down.

I did check exporting the image from MicroDEM with a few of its standard elevation colour options. These come out with 48 colours, which is about where they should be if resolution is 5 metres.

The game map is attached if anyone wants to check it out.

[Gumboots]

Yay! Got it working.

Turns out that when the MicroDEM help pages helpfully say that elevation .dbf files can have 1,024 values, what that really means in English (or any other language, come to think of it) is that when viewed in OpenOffice or a similar app, the .dbf can have a total of 1,024 cells used for data, and that includes the headings of each of the four columns.

Edit: This is not quite right, because the elevation tables have 5 columns instead of 4 columns. I blame lack of sleep at the time.
However, the key point is the next one, about the limit of 255 heights. Once the header row is taken into account, 255 heights require a total of 256 rows. This is the actual limitation, and equates to 1,280 cells.

IOW, you can have up to 255 heights. If you attempt to enter more heights, what MicroDEM does is drop every second one and do a recount. This is why the previous test turned out to have 10 metre res: that one had 256 heights, so every second value was being dropped. For this test I knocked the upper limit back to 1,250 metres, which reduced the number of heights to 252 (250 for 0 to 1250, plus one for below 0 and one for above 1250) and suddenly they all work just like they should.

This also explains why the old 0-9999.dbf only achieved 40 metre height resolution: it had 1,003 heights plus the one extra line for headers, so 5,020 cells in total. MicroDEM ditched half of the height cells and found there were still 2,013 cells, so ditched half of them again and then found there were only 1,255 cells left. It was happy with 1,255 cells, since that was less than 1,280, but of course by that time it only had 1/4 the resolution it was meant to have. Result: 130 foot tall Lego blocks.

The good news is that it is now clear that my cunning plan of splitting things into a range of sheets, to handle every elevation from 0 up to the peak of Everest in 5 metre slices, will work. And, if you import a heightmap to RT3 and want to tweak some part of it slightly, you will be able to do that with an image editor and a colour picker if the game's editor is making things difficult for you.

Screenshot of the new map:

2nd_test_5m_res_yay.jpg

The exported image has 56 colours. One of those is red for the ocean, leaving 55 for the heights. 55 heights times 5 metres = 275 metres maximum elevation, plus up to another 4.99 before it would bring in another colour, so highest point on the map is somewhere between 275 and 279.99 metres, which is spot on where it should be.

That was made with the same cropped area as the previous test and at the same native DEM resolution, but with an extra 10% on the RT3 overall height modifier. It's still as smooth as silk, even with 0 smoothing applied. Zip is attached, for comparison with the previous one. Doubling the height resolution makes a big difference. I'll get onto making the rest of the .dbf tables so we can cover the rest of the world.

[RulerofRails]

Good work. That one looks very nice. The foot of the hills especially look far better to me than on any typical map.

[Gumboots]

Yup, I think we're on a roll now.

I gave it some more thought before that last test. The RGB scale is based on multiples of 8 anyway (256 possibly values for each channel) so it all works really well if I assign the 5 metre elevation contours in steps of 8 RGB height units. This allows using 8 elevation tables to cover everything from sea level to just past the peak of Mount Everest, with a little bit left over before hitting the game's hard-coded height limit. If some nutter wants to map Everest and then build a hotel on top, this scale would just allow them to get away with it.

It's easy on the brain too. I've attached a spreadsheet to show how it works.

BTW, I'm kicking myself that I didn't figure this stuff out before doing that Latvia map. I could easily have had the heightmap done to 2 metre resolution instead of 40. That would have made those !%@$@ rivers and lakes a lot easier.

[Gumboots]

I made the rest of the tables, and gave them a full testing. They work.

Obviously, the way to test them is with Mount Everest. Aint nothing taller than that. So, I grabbed the DEM for that area and exported 8 BMP images, using one elevation table after another. Stacked the BMP's in Photoshop, knocked out the green pixels, and saved as a TGA heightmap.

Imported that into RT3, and it looks like this...

Everest_import_testing.jpg

I've attached the TGA in a zip, so anyone can play with it themselves.

Ok, so how did I make this beastie? I used the new elevation tables which are in the other zip.
They come with a readme file, and you are advised to read it, but I'll describe the procedure in the next post.

Edit: Just for fun, I threw a satellite shot on it.

Everest_satshot.jpg

[Gumboots]

The procedure goes like this:

Due to MicroDEM being limited to 255 heights in every elevation table, each table covers a range of 1,250 metres.
They are all clearly named, so even I can tell which one I want when my brain isn't working.

All of the elevations are done with the usual RT3 colour scale, which looks like blue and black but is actually blue and green.
Red is for oceans, as is also usual for RT3. To get your oceans red:
- right click on the image,
- select Display parameter > Reflectance > Water > Set to red > Ok.

Now switch back to elevation view by:
- right click on the image,
- select Display parameter > Elevation > Color from table > Pick your table >
- then, in the same pop-up > Enable Sea Check > Leave Lake Check disabled > Ok.

The colours to watch out for are yellow, lime green and pure white. These are warnings that you need to do something.
These colours should not be visible in the finished heightmap, or RT3 will do weird things when you import it.
If you want to see RT3 do weird things, go ahead and import it with some of those colours (those pixels will be 10,000 high).

Taking the warning colours in order:

Yellow means the DEM thinks that area is below sea level. It might be, or it might just be an artifact in the DEM. Satellite data is not perfect, and the sea goes up and down, so sometimes a DEM will have pixels registering as below sea level when the area in question is at or above sea level. If you see any yellow pixels when using the 0-1250 table, you should check the area out on a map or a satellite shot and decide what you want to do with it. If you want it as ocean, fill it with red. If you want it as land at sea level, fill it with black.

White means those pixels are at an elevation that is higher than the current elevation table will cover. If you see white pixels, you'll need to also save another BMP with the next table up. You don't have to remove white pixels, because the next layer will cover them.

Green means areas that are below the range covered by the current table. If you see green pixels, you'll also need the next table down (and another BMP).

Once you have all the BMP's you need, stack them up as layers in your image editor and knock out the green pixels with the Magic Wand tool. Make sure that anti-aliasing is not enabled, because if it is the cuts will be furry. No anti-aliasing means you get clean cuts exactly around each pixel. That's what you need for this job.

Once all layers are free of green pixels, you can save the whole thing as your TGA heightmap.

This is the first layer for the Everest heightmap:

Everest_0_1250.jpg

Lotsa white there. This means Everest is taller than this (in case anyone didn't know).
So, next layer up...

Everest_1250_2500_a.jpg

Lotsa green there, so knock it out with the magic wand and the two layers together will look like this:

Everest_1250_2500_b.jpg

Pretty straightforward. Only takes few seconds. Keep going like that until there's no white left. Easy.

[Wolverine@MSU]

Greaat work Gumboots . I especially like the way you've color-coded places that are below the range of the elevation table (green) and those above (white). Makes touching up the stacked BMPs really easy as long as the layers are in the correct order.

[Gumboots]

Glad you like it. I wanted it easy. I don't like playing guessing games.

White means up to most people, since terrain maps usually have white on top of high mountains (for obvious reasons). Green means land, so down. Both are nice and clear against the rest of the heightmap, without burning your eyes out. Yellow stands out well against red, so is good for sea level artifacts. And, most importantly, none of these colours will be used anywhere else in the heightmap, so you can't accidentally knock out pixels you want to keep by selecting any of these colours.

I've run a few other tests with this scale and I think it's a good general purpose one, but I can see specific cases where someone might still want a different scale. Now that we know how to do them, that should still be easy. One detail I did notice is that apparently the elevation .dbf's won't accept fractions of a metre. I did experiment with setting a range for 0-0.1 metres and then 0.1-5 metres (in the hope it would be handy for small artifacts around sea level) but after saving the file and running MicroDEM those decimal values were reverted to 0. So it looks like the smallest range possible is one metre, which is still ludicrously precise for RT3 maps given that DEM data is often out by more than that.

[Gumboots]

I came up with an idea last night. It's one of those ideas that's so obvious in retrospect that I'm wondering why it only just occurred to me.

MciroDEM doesn't just do TGA heightmaps. It's capable of a lot more than that. One of the easy things it will do is make reflectance maps, which are the basis of the terrain shading that is commonly used on terrain maps. These turn out to be very useful, because since they are generated directly from the DEM they match it exactly. This means that if you export one with it zoomed in, so that the exported size is greater than 1024x1024, you can then use it as a guide for positioning and scaling a large composite satellite shot. The advantages here are:

a/ You don't have to guess how it will fit your game map. You can see it right in front of you. This is a very welcome change.

b/ There's no need to worry about exact coordinates to get your satellite shot overlay fitting your game map. As long as the image(s) you start with cover more area than you need to cover, that's good enough. Cut the edges anywhere that's easy.

The current test case is a map of the Darjeeling Himalayan Railway. This was chosen because it's a possible good scenario at some stage, the area has highly complex terrain that provides a good test case, and I already had the DEM for that region anyway. The game map is 448x576 pixels and the area it covers is from 26.60042 North to 27.20042 North, and from 88.09958 East to 88.62042 East.

Why the funny decimal values? Because that's the closest that MicroDEM would get to simple values while giving me the finished map size I wanted. MicroDEM does the cropping by using values of pixels on the DEM, and you just put these at the most convenient values. A basic reflectance map of the area, with standard IHS colours enabled, looks like this:

DHR_reflectance_2_dir_IHS.jpg

That's a handy starting point, but it can be tweaked to make it more useful. Desaturating it and then colorizing gives this:

DHR_reflectance_desat_and_colorize.jpg

When overlaid on top of the previous image, with layer blending set to Color Burn, you get this:

DHR_reflectance_combined_color_burn.jpg

Which looks a bit lurid, but turns out to be useful. Use the Magic Wand, on a suitable tolerance level, to select the river valleys (blue areas) and ridges (reddish areas). Then "Copy merged" and paste those selections as a new layer. This makes a very good guide for resizing your satellite image. You can just stick the satellite shot underneath the guide layer and then manually resize it, without worrying about any coordinates or other numbers, until it fits the way you want it to fit.

DHR_reflectance_satshot_resizing.jpg

Once you have that sorted, resize the thing to 1024x1024 and save as a BMP. Apply to the game map with bmp2gmp, and you get this:

DHR_bmp2gmp_result.jpg

Which is pretty good for a first stab at it, given the complexity of the area. It's not pixel perfect everywhere. You can find small anomalies if you look for them, but it's close enough to be a very good starting point for further tweaking, and it's a process that is a lot easier on the brain than either guesstimating or trying to calculate it all accurately and hoping it works.

Now for the next step, which I haven't looked into yet but has great potential. MicroDEM will also import stuff other than DEM's, like shapefiles. These can be used for things like this: Polylines of 687 rivers, associated to the basins from the river mouths.

The idea here is that if you can directly overlay accurate river courses onto your DEM, you can obviously export that as an image to your image editor. This will then allow you to colour the rivers with whatever colour fits the elevation you want for the river bed, using the values from the RGB elevations spreadsheet back here.

This method should ensure that you can cut rivers through any terrain at all, at the correct heights, regardless of the game map resolution. It should all be done automatically when you import the heightmap into the RT3 editor.