Author Topic: Large Tiff Heightfield Issues  (Read 390 times)

Offline D.A. Bentley

  • Member
  • *
  • Posts: 62
    • D.A. Bentley
Large Tiff Heightfield Issues
« on: November 01, 2017, 02:34:03 AM »
I had some issues loading in very large 8-bit Grayscale Tiff files as heightfields.  I was loading in 40k x 40k maps, but then cropped down several times to get rid of some abnormalities forming at the bottom of my image/terrain.  Where there should have been just a flat black area (ocean) there was data forming in the area in a perfect horizontal bar.  I could crop my tiff image/heightmap smaller and eventually the bar went away.

Is this a known bug/limitation?
« Last Edit: November 03, 2017, 02:32:30 AM by D.A. Bentley »

Offline Oshyan

  • Planetside Staff
  • *
  • Posts: 11746
  • Holy snagging ducks!
Re: Large Tiff Heightfield Issues
« Reply #1 on: November 04, 2017, 01:53:08 AM »
Strange, we're not aware of this problem. What is the source of the heightfields?

- Oshyan

Offline D.A. Bentley

  • Member
  • *
  • Posts: 62
    • D.A. Bentley
Re: Large Tiff Heightfield Issues
« Reply #2 on: November 04, 2017, 10:31:29 PM »
Hi Oshyan,

I will assemble a test package for you to try out soon.  I can duplicate the issue, and take screen shots.  I can also share the tgd & tiff files with you to test.  It will be quite large though because of the 40x Tiff.  Should I send you a Dropbox link by email, or post the link here?  I'm on the latest "frontier build" by the way.

The heightfield is a single Tiff I assembled from 3 DEMs I downloaded from the USGS site.  I converted the IMG files using GDAL to 32-bit tiff files, and then in Photoshop downsampled to 16-Bit using HDR Toning.  I toned each of the three tiff files exactly the same way so that they would blend together nicely.  I combined them into one file which at this point was about 20k square, then I doubled the resolution to 40k before bringing the heightfield into World Machine for erosion.  So the heightfield went through many steps, but the final result was an export of a Tiff from World Machine. 

Thanks,

Derek
« Last Edit: November 04, 2017, 10:43:44 PM by D.A. Bentley »

Offline Oshyan

  • Planetside Staff
  • *
  • Posts: 11746
  • Holy snagging ducks!
Re: Large Tiff Heightfield Issues
« Reply #3 on: November 05, 2017, 02:02:53 AM »
Thanks, that would be helpful. Dropbox link to support AT planetside.co.uk would be ideal.

It's unfortunate that the heightfield had to go through that much processing. It makes me wonder A: can't World Machine import at least GeoTIFF/32 bit TIFF? If so, use Global Mapper (or even Terragen) to export a single 16/32 bit heightfield and erode in Terragen. And B: if erosion is the main thing you want added to these heightfields, maybe now that Daniil Kamperov's "Classic Erosion" plugin is available you could use that instead and avoid all the resampling, etc. which is inevitably bad for height data. You could just import the DEM tiles directly into TG, they should align automatically if they're georeferenced (they will be from the USGS), and then just do the erosion on it and you're good. Just a new option to consider. :)

- Oshyan

Offline D.A. Bentley

  • Member
  • *
  • Posts: 62
    • D.A. Bentley
Re: Large Tiff Heightfield Issues
« Reply #4 on: November 05, 2017, 03:14:44 AM »
The reason I was going through that processing is because I manually added in some "fake bathymetry" to my merged DEM working file (in Photoshop) and wanted to use erosion to blend the terrain into the underwater area.  I couldn't find any bathymetry DEM for the area I'm working on (LA Coastline / Santa Monica Mountains), and i just needed some variation in the underwater terrain so the water would render more realistically. 

Also the USGS DEM I downloaded was in 1/3 arc-second DEM in IMG format from here: https://viewer.nationalmap.gov/basic/.  I tried using MicroDEM to convert to tiff with no luck, and then tried QGIS with Grass with some luck, but ended up using a command line batch convert in OSGeo4W Shell using GDAL.  I will look into Global Mapper, and I am planning on buying Daniil Kamperov's "Classic Erosion" soon.  I've just been scrambling this last week to finish a project that could land me a job so haven't had time to do everything properly.  ;)

I have attached two screenshots of what I am seeing, and I will send support a dropbox link to download the tgd scene and tiff heightfield.

OK thanks for the tips, and I'll try to learn as much as I can from the Wiki & Forum so I don't have to bug the support crew much.

Derek
« Last Edit: November 05, 2017, 05:23:16 PM by D.A. Bentley »

Offline Oshyan

  • Planetside Staff
  • *
  • Posts: 11746
  • Holy snagging ducks!
Re: Large Tiff Heightfield Issues
« Reply #5 on: November 06, 2017, 01:04:09 AM »
Thanks for the files, we'll take a look.

- Oshyan

Offline sboerner

  • Member
  • *
  • Posts: 64
    • Steve Boerner Typography & Design
Re: Large Tiff Heightfield Issues
« Reply #6 on: November 12, 2017, 01:19:31 AM »
Derek, have you seen this resource? NOAA's bathymetric data viewer:

https://maps.ngdc.noaa.gov/viewers/bathymetry/

It includes bathymetric DEMs for much of the North American coastline. The Los Angeles area is well covered and includes terrain along the coast. Maybe enough for what you need?

I used a few of the DEMs for an illustration a couple of years ago. Catalina Island is on the right, and I believe that those are the Santa Monica mountains on the left, behind the ship.

77084-0

Steve

Offline D.A. Bentley

  • Member
  • *
  • Posts: 62
    • D.A. Bentley
Re: Large Tiff Heightfield Issues
« Reply #7 on: November 12, 2017, 01:59:58 AM »
Hi Steve,

I did end up finding some DEM that extended into the Ocean contiguously (Integrated Coastal Relief Models).  I ended up downloading all the formats from the Grid Extract Tool, and settled on XYZ format.  It loads into Terragen without converting to anything, and it just works (although huge and slow).   

Grid Extract Tool:
https://maps.ngdc.noaa.gov/viewers/wcs-client/
OR big chunks:
https://www.ngdc.noaa.gov/mgg/coastal/grddas06/grddas06v2.htm

Actually I was just telling Oshyan about this resource, and thought the GeoTiff downloads didn't contain bathymetric data, but I was wrong.  Photoshop just has trouble with GeoTiff files.  Usually I can load a GeoTiff into Photoshop and use HDR Toning to Equalize the histogram to see all the data (in crunched form), but with the GeoTiff's with bathymetric data that doesn't work.  Everything below sea level just appears to be clipped (looks like a mask/all white).

I figured out that the bathymetric data was indeed in the GeoTiff by loading them into Global Mapper.  See attached images.

Offline sboerner

  • Member
  • *
  • Posts: 64
    • Steve Boerner Typography & Design
Re: Large Tiff Heightfield Issues
« Reply #8 on: November 12, 2017, 02:02:36 AM »
Had to take another look at that website. It's much improved since I last used it. You can now download seamless GeoTIFFs in 1- and 3-arcsecond scale, at least around L.A. The download tool is here: https://maps.ngdc.noaa.gov/viewers/wcs-client/

Here is a very quick rendering of your area, color coded above and below sea level.


Offline sboerner

  • Member
  • *
  • Posts: 64
    • Steve Boerner Typography & Design
Re: Large Tiff Heightfield Issues
« Reply #9 on: November 12, 2017, 02:03:53 AM »
Just saw your post! No need to tweak the GeoTiffs in Photoshop. Best not to touch them. Just load them into Terragen. They'll render just fine.

Offline Matt

  • Planetside Staff
  • *
  • Posts: 2849
  • I'm the crazy one
Re: Large Tiff Heightfield Issues
« Reply #10 on: November 12, 2017, 03:30:55 AM »
I had some issues loading in very large 8-bit Grayscale Tiff files as heightfields.  I was loading in 40k x 40k maps, but then cropped down several times to get rid of some abnormalities forming at the bottom of my image/terrain.

This is fixed in v4.1.17. Available now for Windows and Linux (check for updates in the app). The Mac version will follow shortly.

Matt
Just because milk is white doesn't mean that clouds are made of milk.

Offline Oshyan

  • Planetside Staff
  • *
  • Posts: 11746
  • Holy snagging ducks!
Re: Large Tiff Heightfield Issues
« Reply #11 on: November 12, 2017, 06:39:29 PM »
It's not totally surprising that Photoshop has trouble handling a GeoTIFF with bathymetry, I would guess the values underwater are negative and I don't think Photoshop can probably deal with negative values as its entire purpose is as an image editor for visible images, and in that context negative values don't really make sense.

- Oshyan

Offline D.A. Bentley

  • Member
  • *
  • Posts: 62
    • D.A. Bentley
Re: Large Tiff Heightfield Issues
« Reply #12 on: November 16, 2017, 06:03:27 PM »
Thanks Matt!  I'm not sure I will be using 8-bit heightfields anymore now that I know better, but it's nice to know this has been fixed.  I tested a 16-bit 40k x 40k Grayscale Tiff as a heightfield, and it worked flawlessly.  :)

Thanks again everyone,

Derek