Planetside Software Forums

General => Terragen Discussion => Topic started by: bigben on September 01, 2008, 05:39:58 PM

Title: River/Lake model revisited (solved)
Post by: bigben on September 01, 2008, 05:39:58 PM
Hi All

As mentioned in another post I'm revisiting my river/lake model with the latest version

Terrain -> Masked opacity of default shader -> water shader
http://forums.planetside.co.uk/index.php?topic=1301.msg12986 (http://forums.planetside.co.uk/index.php?topic=1301.msg12986)
This doesn't work anymore as the water shader appears across the entire terrain, poking out through the overlying surfacing.
[attachthumb=#1]

http://forums.planetside.co.uk/index.php?topic=1636.0 (http://forums.planetside.co.uk/index.php?topic=1636.0)
This approach is the one I'm testing at the moment.  The main problem with this at the moment is that at some camera angles there are triangular gaps in the river surface.  I'm investigating the cause of these to see if I can find a workaround.  My current suspicions are with the colour adjust node used to adjust the extent of the river surface. Dropping the white level to really small values (making the mask area wider... and sharper) seems to be a part of the problem. I had also suspected image resolution may play a part as I was really pushing the limits of resolution, but I don't think this is to blame.
[attachthumb=#2]

Here's a partial render showing the river layer rendering before the terrain.  As you can see the drop off at the edge of the river is very severe, and this combined with the contrast of the edge of the river appear to be causing the problem.
[attachthumb=#3]

I ran a series of renders using different white level values for the river mask (1, 0.75, 0.5, 0.25, 0.1, 0.05, 0.01)
[attachthumb=#4]

You can see how the width of the river surface is affected by this change, casting a shadow in the river bed when it doesn't extend beyond the river bank.  No glitches at this camera angle. Repeating at a lower camera angle today.  PS, the water transparency makes this approach very worth while... there's a lot of potential here.

Title: Re: River/Lake model revisited
Post by: bigben on September 01, 2008, 05:52:55 PM
One last screengrab. Lowering the camera angle gave me this glimpse below the terrain before the preview render kicked in. This gives you a good idea of how this model works.
[attachimg=#1]
Title: Re: River/Lake model revisited
Post by: rcallicotte on September 01, 2008, 08:51:41 PM
Cool  I want to spend some time on this.  So many lessons, so little time.

Thanks BigBen.  Welcome back, again.
Title: Re: River/Lake model revisited
Post by: bigben on September 02, 2008, 04:46:49 PM
Restarted the render last night. I'd increased the quality settings yesterday, but the render times were a little higher with the lower camera angle... it was still on frame 1 when I got home.  Here's frame 1 showing the river protruding through the terrain but not reaching the bank. Note the shadow on the surface and river bed.

The second image shows the increase in width from dropping the white level of the mask to 0.75. The defects in the surface have started to appear (triangular chunk missing on the right)
Title: Re: River/Lake model revisited
Post by: bigben on September 03, 2008, 05:14:14 AM
Nearly fininshed the series..  Here are images for white values of 0.5, 0.25, 0.1 respectively.  The defects are  less severe than in earlier versions of TG2 which rendered more like the preview render above. What is interesting is that the last image looks fine while I was expecting it to get worse with lower white values. The next image is looking OK too which is promising.

Next will be exploring different render quality settings and checking different locations. These renders are Qualtiy 0.35, AA 3 which is well below my normal settings. I'll take suggestions as to which setting is most appropriate in this instance.... anyone?
Title: Re: River/Lake model revisited
Post by: jo on September 03, 2008, 08:54:51 AM
Hi Ben,

Just a random thought, but it seems to me you are doing the river surface by pushing one heightfield just slightly through another, right? In this case the dark artefacts you're seeing might be caused by something like "z fighting (http://en.wikipedia.org/wiki/Z-fighting)". Perhaps the renderer is having trouble with the two surfaces very close together.  This could be borne out by the two example images in the third post. If you increase the spacing between the two surfaces the problem may go away.

Regards,

Jo
Title: Re: River/Lake model revisited
Post by: bigben on September 03, 2008, 05:23:44 PM
Hi Jo

Thanks for the feedback. Yes you're right. The river bed is a masked negative displacement of the terrain and the river is a copy of the terrain lowered by a smaller amount and then masked as well, dropping areas away from the river back to the planet surface.

I hope this is not the case, because that would make it very difficult to have shallow water.  My initial suspicions regarding the increase in contrast of the mask image were wrong, as the three lowest white levels set had no problems.

Given that the problem looks worse in the preview render (large triangular chunks missing across the top of the surface), and that the mask is based on an antialiased image, my first retry involved increasing the AA.

I first repeated the image with the worst defects, and while the small defects were still present the larger one was largely gone, making the original glitch look more like an incompletely rendered section of the terrain. Increased AA from 3 to 6 and while the render stalled about 1/3 of the way through (my home PC isn't the greatest) these errors were not present.  I'll take the files to work and try a repeat there.

The other thing I noted with this render is that the CLI output only had 3 numbers for the number of microtriangles rendered, changing every 1 - 1.5 hours even though the render was obviously progressing in between times. (piping the CLI output to a text file didn't work???   %terragen_path%\tgdcli.exe > %terragen_path%\log.txt ).

At least increasing AA to a value I normally render at anyway is making this look very promising.
Title: Re: River/Lake model revisited
Post by: bigben on September 04, 2008, 03:11:47 AM
[attachimg=#1]

Here's the partial render with AA=6. This time there are two large chunks missing. across the entire river.  I'll have to play around a bit more to see if I can find a way past this.
Title: Re: River/Lake model revisited
Post by: bigben on September 04, 2008, 08:11:17 AM
Fighting back the Alzheimers... ;)

I have previously run tests with just the river terrain with the same glitches, so it's not a competition between the two surfaces.  From the quick test with <http://forums.planetside.co.uk/index.php?topic=4716.0 (http://forums.planetside.co.uk/index.php?topic=4716.0)> it would appear to be more related to the use of TER files than procedural terrains.

In my model I used a blending shader to mask the terrain. Trying a different approach now by displacing the area away from the river by a larger negative amount than the river. Crossing fingers.  ;)

Title: Re: River/Lake model revisited
Post by: rcallicotte on September 04, 2008, 01:17:18 PM
This is like reading a blog.

Thanks for the information.  Braincells have begun to acclimate to your visitations.   :P
Title: Re: River/Lake model revisited
Post by: bigben on September 04, 2008, 04:49:23 PM
I do tend to blog my posts when I'm working on something a) it helps me sort things out (thinking out loud) and b) you never know when someone chips in with a gem that gets you a lot further along.

And the next entry ...  ;)

[attachimg=#1]

This image has uses an elevation adjustment of the river terrain followed by a large negative displacement of the terrain away from the river.  This means that the entire surface consists of the original TER whereas the previous model masked the TER.  This looks a lot more promising. There may be 2 minor glitches (far right and middle right) but the image is a bit too small to tell for sure. The edges don't quite look sharp enough.  I also changed the lighting to use a soft shadow. it's at least starting to look the way I wanted it to.

Time for  a larger render with higher quality and perhaps a smaller displacement of the terrain away from the river (-500m in this image).  Something for my work computer to play with over the weekend.  Here's the reference pic without the river and a bit of extra candy.
Title: Re: River/Lake model revisited
Post by: rcallicotte on September 04, 2008, 05:50:52 PM
This looks good.  I can't wait to hash over this and then to see your final.
Title: Re: River/Lake model revisited
Post by: bigben on September 05, 2008, 02:33:16 AM
Quote from: bigben on September 03, 2008, 05:23:44 PM
... piping the CLI output to a text file didn't work???   %terragen_path%\tgdcli.exe > %terragen_path%\log.txt...

Sorted this one out. Different computer, this was set up in a shortcut. Works OK in a batch file.

The low quality render was progressing nicely at home when I left. The higher quality render kept crashing at work with odd C++ errors, but that'll learn me for trying to do some work at the same time. It's progressing further now that I've shut down some hungry Adobe apps  ;)
Title: Re: River/Lake model revisited
Post by: bigben on September 05, 2008, 05:43:54 AM
Here's the low res render. Detail 0.35, AA 4, GI 1,1

The water surface is glitch free (that's an island near the centre of the image) but you can see why I wanted to use more extreme negative displacements for the areas away from the river.  The displacement in this one is only -50m, but there are cliffs that are higher than this and you can see some of the water on the cliffs on the right, and you also have to take into consideration additional displacements to the terrain which may expose a nearby water layer ... but I'm very happy with the river.  ;D

Rendering again with a displacement of -500m.

The clip file contains the basics of the river model.
Title: Re: River/Lake model revisited
Post by: bigben on September 05, 2008, 04:26:07 PM
Here's another test from a higher camera position... still no glitches. The larger displacement here is -500m.

Another benfit of this approach is that it is MUCH faster to render than the masked terrain model. This image (640x480) took just under 5 hours to render, while the first Grand Canyon test above (320x240) using the masked terrain took just over 6 hours.

Rendering one last test from head high before I call this one a success.
Title: Re: River/Lake model revisited
Post by: rcallicotte on September 05, 2008, 10:10:52 PM
These are very good.  Congratulations on discovering and clarifying how this works. 
Title: Re: River/Lake model revisited
Post by: Oshyan on September 06, 2008, 01:28:42 AM
Nice work Ben, it's good to have a workflow developing on this. I remember some earlier work you did on the same subject - do you find it better or worse in the more recent TP versions?

- Oshyan
Title: Re: River/Lake model revisited
Post by: bigben on September 06, 2008, 03:06:13 AM
Quote from: Oshyan on September 06, 2008, 01:28:42 AM
Nice work Ben, it's good to have a workflow developing on this. I remember some earlier work you did on the same subject - do you find it better or worse in the more recent TP versions?

- Oshyan

The earlier model you suggested to me of using a default shader with masked opacity doesn't seem to work now. The visibility of the lake object doesn't seem to be affected at all. (very first example in this post). I haven't looked too closely into it yet, but that model may be more efficient from a performance point of view... explained later.

The behaviour of the masked terrain model (missing chunks on the top surface) was the same in the earlier versions of TG. The latest model described here is a bit simpler in that the water surface consists of modified data from just one source (the TER file).  With the addition of transparency, this makes creating rivers and lakes on real data like this so much more worthwhile. There is plenty of scope for roughing things up a little with some additional displacements of the river bed so it should be possible to create some very realistic bodies of water over a large area with relatively simple masks.

The main drawback to both the masked terrain and double terrain displacement models is that the render is effectively doubled, with one terrain/water shader and then the normal terrain.  In both models the vertical faces where the water surface drops below the terrain seem to take a very long time to render, possibly because of the complexity of the reflections. This is where the masked opacity model may be more useful, as you simply wouldn't be rendering those areas of water outside of the mask. I'll have to do some benchmarking on this, but it may be possible that displacing the areas outside of the visible water surface by even larger amounts so that they are out of the frame may improve render times.

The extra render time is particularly exacerbated with more horizontal camera angles, where you have to wait for the water terrain to render all the way to the horizon, and then again for the normal terrain. In the case of my full Grand Canyon terrain set this can be a long wait indeed... but I'm just glad that it's working and looking really good.

The last render is still going but it is looking very promising. Just waiting for the bottom half to see how sexy the water transparency looks in the foreground.
Title: Re: River/Lake model revisited
Post by: bigben on September 06, 2008, 10:55:43 AM
Here's the rest of that render.  The fake stones are nothing special.. just providing some extra detail to test the water transparency and the use of the river mask for controlling fake stone distribution.  Final render time was around 17 hours, about 5 of which was waiting for the last remaining bit of the lower right bucket to be rendered before the overlying surface could be rendered. I'm not quite sure why this particular bit took so long, but it makes further benchmark testing of different displacement settings potentially worthwhile.

While you can't see it here, the water surface extending below the bank of the river is much better than the masked terrain model which tended to have a more rounded edge,  giving the river a slight bulge.  This is much flatter, and with the surface glitches solved is going to make for some very interesting work.  I used the Grand Canyon  as an example partly because it has been rendered so often. In most cases people simply flood the floor of the canyon, but the river in my scene was traced from Landsat imagery providing more accurate changes in width, course, islands/branching etc...
Title: Re: River/Lake model revisited
Post by: dandelO on September 06, 2008, 07:33:08 PM
This last one I think has nailed it! How far lateraly into/under the banks does the river extend? One of the main problems when rendering water is waiting while the sub-terrain portion of a default waterplane is rendered, unnecessary time. I know you will still need to wait on the verticaly displaced portion of the river's 'walls, which you say take an age to render, but could you mask the water shader distribution of these vertical parts with an altitude/slope constraint, so as to only make the water shader render just slightly lower than the height of the riverbed and, not on vertical surfaces at all? that should still be significantly quicker, I'd think. That way, the reflections/transparency do not need to be calculated beneath the riverbed, only the 'terrain' displacement, which you could maybe also completely mask beneath the 'real' terrain awell?

INSTANT EDIT:

QuoteThis is where the masked opacity model may be more useful, as you simply wouldn't be rendering those areas of water outside of the mask.

Lol, sorry, I missed that.  ::) :D
Title: Re: River/Lake model revisited
Post by: bigben on September 06, 2008, 09:43:17 PM
I have a render running at the moment so it's a little hard to check, but from memory the model mentioned in the very first post is supposed to do just that.  [edit] no warning of the edit while I was posting ;) [/edit] The adjusted terrain is applied to a lake "object" which also has a default shader using the river mask as an opacity mask, so that only the masked portions of the lake object are visible. I will be looking at this more closely later on, as it would be the most efficient model in terms of minimising the amount of subterrain rendering.

Another workaround may be to lay another surface above the water shader so that it covers the vertical faces and regions outside of the exposed water so that there are no complex depth/reflection calculations required for these areas... see, this is why I blog my experiments here... ideas come from the feedback  ;)
Title: Re: River/Lake model revisited
Post by: bigben on September 06, 2008, 10:22:55 PM
Quote from: bigben on September 06, 2008, 09:43:17 PM
Another workaround may be to lay another surface above the water shader so that it covers the vertical faces and regions outside of the exposed water so that there are no complex depth/reflection calculations required for these areas... see, this is why I blog my experiments here... ideas come from the feedback  ;)

Cancelled the render, as this was more interesting  ;)  A few carefully selected numbers for the colour adjust shaders and I've added a grey surface above the water shader that only covers the smallest areas of the upper water surface (but is still well below the terrain). The precision of the colour adjust shader is for creating multiple masks from images is fantastic.  I'm using differences invalues in the range of 0.01 and it's working exactly as I expected.

Thanks for making me think some more dandelO (and thanks for the mixed rocks that that are about to go on the riverbed  ;D)

The screengrab shows the adjusted node network and a preview of just the water surface. A quick test showed an improvement in speed. The calculation of the vertical faces is still relatively slow but greatly improved.

Also added an additional displacement of the riverbed to rough it up a bit to start making things look a little more natural.
Title: Re: River/Lake model revisited
Post by: dandelO on September 06, 2008, 10:53:21 PM
I noticed when I made this: http://forums.planetside.co.uk/index.php?topic=2992.0
That the large vertical displacements do take a bit of time to render. This is simply terrain displacements with lots of crater shaders and minimal texturing(which doesn't react correctly to the displacement, displacements come BEFORE colour so, why don't the textures overlay the pre calculated displacements at their correct scales/resolutions? Instead, the textures are stretched in the same way as the terrain. verticaly displaced walls do take longer than they should, in my opinion.

Why not try a TG plane object for the river? It's not connected to the planet surface like the terrain method and, could be more easily masked from the riverbed, down, than the default lake disk.
A TG plane can be masked via black/white channels so that it will only render portions of the plane designated to 'white'. The whole ground portion of the river could be removed from rendering completely by using the invert of the river mask, and vertical constraints on the water shader would stop the need for rendering reflections on the already slow vertical displacements.
I dunno... Great tests though.
Title: Re: River/Lake model revisited
Post by: bigben on September 07, 2008, 02:44:51 AM
I had contemplated using a plane as well, and have constructed a model based on that as well.  Getting the displacement and water surfacing applied to both is easy, but I can't seem to use the river mask as an opacity function with either object type. Can someone confirm whether this should work or not?... and if it should work, some suggestions on how to get it working?

Applying the colour adjust node from the image mask to the plane's opacity function results in a black object instead of invisible.


[update]
NVM, almost there.
1: Add plane, set size to image mask
2: Connect second terrain (minus fractal noise) to input of plane's default shader.
2b: Connect river mask to opacity function
3: Insert surface shader between default shader and plane's shader input
3b: Connect water shader to surface layer child input
3c: Connect river mask to surface layer blending shader
4: (Kinda weird ???) Edit vertical adjustment of second terrain to position water level. In my test this is -16m even though the water surface is only 20cm or so below the unadjusted terrain level.

This results in just the top surface of the water. The preview render is looking OK. Running a test render... very promising  :D
Back to low detail (Detail 0.35, AA 3, GI 1,1) but as you can see by the subsurface render in the second bucket there are no vertical faces, and the time is reasonable.
[attachimg=#2]
Title: Re: River/Lake model revisited
Post by: bigben on September 07, 2008, 10:22:36 AM
Quote from: bigben on September 07, 2008, 02:44:51 AM
4: (Kinda weird ???) Edit vertical adjustment of second terrain to position water level. In my test this is -16m even though the water surface is only 20cm or so below the unadjusted terrain level.

Ummm duh!!!  A plane, of course is flat, which is why you need the lake object on larger terrains.  I'll redo this with the lake object tomorrow.  Off to bed now while a larger render of the last attempt completes.  This is about as efficient as you can get, looking at about 4 hours for 1000x500, Detail 0.5, AA 6, GI 1,4
Title: Re: River/Lake model revisited
Post by: bigben on September 07, 2008, 05:33:53 PM
Here's the render of the plane model described above. Final render time 3h 23m.

Not a bad results given that the waves need a bit of work and there are no other shaders for additional shore effects (wet sand etc...)  Had a quick play to apply this to other object types, but so far the lake, sphere and planet objects all result in vertical faces. Perhaps an imported sphere?
Title: Re: River/Lake model revisited
Post by: Matt on September 07, 2008, 10:43:16 PM
Hi Ben,

The Water Shader ignores any shaders that come before it, so opacity masking of any object that uses the Water Shader isn't currently possible. (And even if you can get a shader further down the chain to mask away parts of the object it will be too late to prevent the Water Shader from being evaluated, unfortunately.) I will do something about this or maybe add a separate mask shader input to the Lake and other objects.

The good news is that you can use other shaders to control where the Water Shader is applied, e.g. by making it a child layer of a Surface Layer, and use your mask as the Surface Layer's blend shader. That should save you a lot of render time. When you applied a Default Layer to cover up the water shader I don't see how it could have saved render time, because the shading system isn't quite clever enough to prevent the Water Shader from being calculated when the later shader completely covers it up. Some shaders work that way, but many don't.

Maybe you could combine the Water Shader masking with the water object masking (if you can find a way to apply an opacity mask in a later shader without blotting out the reflections - I can't think of a solution at the moment) to mask away the black parts of the water object.

(Edit: Use the inverse of the mask as the blend shader for another Surface Layer that drives a shader with zero opacity, to mask away parts of the water object. You'd still want to combine this with masking of the Water Shader as explained above to get the shortest render time.)

Matt
Title: Re: River/Lake model revisited
Post by: Matt on September 07, 2008, 10:51:44 PM
Quote from: bigben on September 07, 2008, 05:33:53 PM
Had a quick play to apply this to other object types, but so far the lake, sphere and planet objects all result in vertical faces. Perhaps an imported sphere?

The vertical surfaces are caused by your displacement shader, so the choice of object won't change this. Or do you specifically mean that you don't like the way the surface is tessellated into long triangles? All the objects will suffer from a similar problem with very steep faces, although they might give slightly different results.

Matt
Title: Re: River/Lake model revisited
Post by: bigben on September 08, 2008, 04:27:35 AM
Thanks for the feedback Matt.... and once again, getting me to think through things a little differently. Firstly, here is a screengrab of the river surface from the previous render using a plane. This is what I'm trying to achieve using the lake object. A distorted plane for the surface, with 0 opacity where there is no water.
[attachthumb=#1]

I've had a quick play again, rearranging nodes and have solved it  ;D It all depends on how you connect the water shader as you metioed.  Put it in the wrong place and you get the steep sides, extending back down to the planet (with the possibility of gaps in the upper surface).  But now I have it working properly..
[attachthumb=#2]

[Edit] Attached a clip of the basic river model. Insert your own TER/river mask. Posted the clip file in file sharing http://forums.planetside.co.uk/index.php?topic=4459.msg49935#msg49935 (http://forums.planetside.co.uk/index.php?topic=4459.msg49935#msg49935). The river bed consists of two displacements, a small uniform displacement to provide a minimum depth and a variable displacement to break up the smoothness resulting from the  antialiased image map. The rest is relatively simple, and pretty efficient in terms of rendering speed. 

The speed of rendering water (especially given that it now has transparency) is noticeably faster than earlier versions of TG2TP.  Now for some image based depth maps of lakes based on Landsat imagery... and other possibilities...  :)
Title: Re: River/Lake model revisited (solved)
Post by: bigben on September 10, 2008, 05:10:30 AM
Just giving this a bump to say that I checked the clip I uploaded and found half of the nodes were missing. Fixed the file and posted in File sharing.
http://forums.planetside.co.uk/index.php?topic=4459.msg49935#msg49935 (http://forums.planetside.co.uk/index.php?topic=4459.msg49935#msg49935)
Title: Re: River/Lake model revisited (solved)
Post by: rcallicotte on September 10, 2008, 06:53:55 AM
I love this, Ben.  It's like watching over your shoulder and I can come back any time I need / want to.  Thank you.
Title: Re: River/Lake model revisited (solved)
Post by: dandelO on September 10, 2008, 11:37:23 PM
This isn't exactly what you're trying to do because it relies on an image map to distribute the water.
To keep it all TG, you could still use a black/white displacement function to apply the same terrain displacement/river opacity effect but I don't know how to make a river from functions...

[attachimg=#]
The river is a TG plane object with shadow casting disabled.
The water renders only where the riverbed is(and will always intersect the shore if the water plane is set to a lower 'Y' altitude than the top 'Y' altitude of the riverbanks).

This is the river with the planet surface disabled, that's just the atmosphere behind it...
[attachimg=#]


Sadly, it doesn't rely on any complex functions or technical terrain mapping to course a river like your terrain method but it will serve as an easy way to put water only where it's wanted, with no sub-terranean render time. And, as it's just a plane, there are absolutely no vertical faces. The water-level will always just be a case of changing the 'Y' altitude in the river-plane to a point between the highest/lowest 'Y' displacements of the banks.

River tgd.RAR attatched. All you need to look at are the terrain and water groups for the general idea, the rest is just cosmetic.

Title: Re: River/Lake model revisited (solved)
Post by: bigben on September 11, 2008, 05:59:44 AM
Actually I do use an image to distribute the water  ;) You could connect any terrain into my clip file, replacing the vertical adjust node on the water terrain with a displacement node for fractal terrains.

There are several possible ways to use an object as the surface of water. The key to avoiding the vertical faces is not to have anything connecting to the water shader's input, so one you do that there are then other things to consider.

The main limitation I found with using a plane is that it's not very practical for very large terrains due to the curvature of the planet.

While a plane doesn't appear to involve any rendering calculations where it goes below the terrain, when it is above the terrain you have to wait for the terrain to render first before the water.

With the lake object, areas below the terrain surface are rendered as well, but masking can be used to minimise this. When the water is above the terrain, the terrain is not rendered, only the water. Whether there is any difference in the overall render speed between these two methods may be worth testing.

Using the masked lake/duplicate terrain, you don't have to calculate the placement of the water. It's all taken care of by simply displacing the river bed more than the water surface, giving you precise control over the entire terrain, but if using a plane renders faster it may be more practical for more localised scenes.
Title: Re: River/Lake model revisited (solved)
Post by: max_thehitman on April 21, 2011, 10:37:40 AM


WOW! I am amazed!  8) Great posts!

Thanks for all the valuable information! I will give this a try soon!
Cheers
MAX