Author Topic: Terragen GI cache blending and Multithreading  (Read 551 times)

Offline D.A. Bentley

  • Member
  • *
  • Posts: 89
    • D.A. Bentley
Terragen GI cache blending and Multithreading
« on: November 28, 2017, 01:23:49 AM »
I am monitoring a large 8k x 4k single frame/still render and only about 20% - 30% of my 12-Core cpu is being used.  The only difference from other renders I have done is this one is doing some GI cache frame blending between 7 GI cache files using the "Equal Blend" option.

Is this normal CPU behavior when rendering with GI cache blending?  I can't bear to watch my 12-core CPU only use 1/4 of it's power.  :(

UPDATE:  I restarted my render and chose to only blend 3 GI cache files and now I am seeing 100% CPU utilization.

Derek
« Last Edit: November 28, 2017, 02:13:26 AM by D.A. Bentley »

Offline bobbystahr

  • Member
  • *
  • Posts: 8204
  • Turn, and face the Strange Ch Ch Changes...D Bowie
Re: Terragen GI cache blending and Multithreading
« Reply #1 on: November 28, 2017, 02:57:57 PM »
Just curious as to why you are using multiple GI cache files blended for a single frame image...is it from an animation you looking to do at some point?
something borrowed,
something Blue.
Ring out the Old.
Bring in the New

bobbystahr

Offline D.A. Bentley

  • Member
  • *
  • Posts: 89
    • D.A. Bentley
Re: Terragen GI cache blending and Multithreading
« Reply #2 on: November 28, 2017, 03:09:14 PM »
I was just doing an experiment to see if I could create a smoother lighting effect in my clouds.  I animated the sun moving horizontally about 10 degrees, and rendered out the GI cache files for those 10 or so frames and then rendered the final still frame from a frame at about the middle of the Sun's animation using the set of previously rendered GI cache files.  The idea was to get a smoother result of light penetration through the clouds.  You could think of it as doing a super long exposure, or maybe a time lapse photo merge, but of just GI.  :)  Terragen has so many possibilities.

Offline bobbystahr

  • Member
  • *
  • Posts: 8204
  • Turn, and face the Strange Ch Ch Changes...D Bowie
Re: Terragen GI cache blending and Multithreading
« Reply #3 on: November 28, 2017, 03:16:41 PM »
I was just doing an experiment to see if I could create a smoother lighting effect in my clouds.  I animated the sun moving horizontally about 10 degrees, and rendered out the GI cache files for those 10 or so frames and then rendered the final still frame from a frame at about the middle of the Sun's animation using the set of previously rendered GI cache files.  The idea was to get a smoother result of light penetration through the clouds.  You could think of it as doing a super long exposure, or maybe a time lapse photo merge, but of just GI.  :)  Terragen has so many possibilities.

cool, be interested in seeing the results. never thought of that approach..carry on and please post results as you've piqued my interest.
« Last Edit: November 28, 2017, 03:30:43 PM by bobbystahr »
something borrowed,
something Blue.
Ring out the Old.
Bring in the New

bobbystahr

Offline D.A. Bentley

  • Member
  • *
  • Posts: 89
    • D.A. Bentley
Re: Terragen GI cache blending and Multithreading
« Reply #4 on: November 28, 2017, 04:04:01 PM »
Sure thing.  I will definitely post an example soon.  :)

Offline Oshyan

  • Planetside Staff
  • *
  • Posts: 11796
  • Holy snagging ducks!
Re: Terragen GI cache blending and Multithreading
« Reply #5 on: November 29, 2017, 02:33:36 AM »
GI cache blending has some overhead, mostly in memory use as far as I'm aware, but it shouldn't have *that* dramatic an effect on CPU utilization. That being said we don't generally recommend blending more than 5 GI caches, and often 3 is sufficient.

Also if you're using v3 clouds, be aware that their internal GI calculation is not stored in a GI cache file, unlike v2 clouds and the rest of the environment. A GI cache will have an effect for v3 clouds because some light does come from the environment, but their primary GI scattering is not cacheable at this time so you need to increase the Cloud GI Quality if you want to reduce flicker or increase GI quality/stability in animation (it's in the GI in Clouds tab of the GI settings panel inside the Render node).

- Oshyan

Offline D.A. Bentley

  • Member
  • *
  • Posts: 89
    • D.A. Bentley
Re: Terragen GI cache blending and Multithreading
« Reply #6 on: November 29, 2017, 04:36:36 AM »
Hey Oshyan,

I am actually using all v2 Clouds in my scene, but that's good to know.  So what about Easy Clouds?  Do Easy Clouds use the GI cache, or are they like v3 clouds and use the "GI in Clouds" tab for GI Quality?

Also I discovered the CPU utilization doesn't seem to a problem caused by frame blending in itself.  I have the same issue just using one super high quality GI cache file.  For example, when I render out a GI cache file for my 8k x 4k scene using 8-8-4 (8 GI cache detail, 8 GI sample Quality, and 4 GI blur radius) the end result is a GI cache file of about 13GB in file size.  This causes my single frame render to crawl at only about 1/4 CPU core utilization.  When I use a lower quality setting like 4-4-8 I have no issues with my CPU being under utilized.  I also have plenty of RAM (64GB), and I'm only seeing about 28GB of memory use when doing these 8k renders.

I guess I'll just have to not go so extreme with my GI caching.  ;)

Derek

Offline Dune

  • Member
  • *
  • Posts: 12826
  • Corkscrew Bird
    • www.ulco-art.nl
Re: Terragen GI cache blending and Multithreading
« Reply #7 on: November 29, 2017, 06:43:41 AM »
Easy clouds are V3 too. You can tell by the lighting tab, which is different from V2 clouds.
In case you still haven't seen enough of my work: www.ulco-art.nl

Offline D.A. Bentley

  • Member
  • *
  • Posts: 89
    • D.A. Bentley
Re: Terragen GI cache blending and Multithreading
« Reply #8 on: November 29, 2017, 02:25:54 PM »
I have something interesting to report.  So this render i am doing is using a spherical camera, and as the render proceeds further into the render (away from the stretched pole) my CPU utilization went back up to 100%.  Could the Spherical Camera be the reason for the underutilization of my CPU I wonder?


Offline Oshyan

  • Planetside Staff
  • *
  • Posts: 11796
  • Holy snagging ducks!
Re: Terragen GI cache blending and Multithreading
« Reply #9 on: November 30, 2017, 12:15:53 AM »
As Dune notes, Easy Clouds use the "v3" cloud shading model (which is why they look so good :D).

Your findings regarding larger/more detailed GI caches are interesting. I honestly don't know if it's just the size/detail of the cache that would be causing a problem, especially since you have enough memory for it, as you reported. I wonder if you have tested yet how that might interact with your most recent finding of possible correlation between spherical/pole area rendering and underutilization?

In any case I can tell you that such high GI settings are very seldom necessary, especially if clouds are majority of your scene (as in the case of a skybox/environment map rendering, for example). I would say that anything above 4/4 is probably unnecessary except for situations where a majority of light is going to be bounced/indirect (like near/after sunset), or with detailed objects in the scene (e.g. architecture). Otherwise, with any normal large-scale Terragen scene, you will see minimal improvements past 4/4 under most lighting conditions.

Regarding your preliminary finding of possible correlation with underutilization and spherical/pole area rendering, that's quite interesting. Can you do a test or two to try to confirm?

- Oshyan

Offline D.A. Bentley

  • Member
  • *
  • Posts: 89
    • D.A. Bentley
Re: Terragen GI cache blending and Multithreading
« Reply #10 on: November 30, 2017, 12:30:08 AM »
Thanks for the info Dune, and Oshyan.  I still am learning, so I will likely keep doing things that are unnecessary until I know better.  ;)  Sometimes users like me are good at finding bugs too because we do uncommon things in the App.

Yes, I will do some more render tests to see if I can track down exactly what is causing this underutilization.  I'll report what I find of course.

Derek

Offline Oshyan

  • Planetside Staff
  • *
  • Posts: 11796
  • Holy snagging ducks!
Re: Terragen GI cache blending and Multithreading
« Reply #11 on: November 30, 2017, 12:45:35 AM »
Sounds great, and yes we always appreciate people pushing the boundaries in unusual ways and potentially uncovering hidden issues. :D

Btw you might try rendering a crop of an area of your spherical that is not near the poles, to see if the underutilization still happens. Obviously you'd have to use all other settings the same, and you'd also want to make sure the crop area is big enough that it has enough buckets in it to fully utilize your CPU anyway. Perhaps just do a crop that cuts out the top and bottom 1/3 or 1/4 of of your image (i.e. render the middle half or third).

- Oshyan

Offline D.A. Bentley

  • Member
  • *
  • Posts: 89
    • D.A. Bentley
Re: Terragen GI cache blending and Multithreading
« Reply #12 on: December 01, 2017, 02:20:55 AM »
I'm making some progress on this, and have made an interesting discovery I am trying to verify at the moment.  I'll update this will all the finding soon.

Offline D.A. Bentley

  • Member
  • *
  • Posts: 89
    • D.A. Bentley
Re: Terragen GI cache blending and Multithreading
« Reply #13 on: December 01, 2017, 03:03:23 PM »
OK, this may sound weird, but the solution to the underutilization of my CPU while rendering was to rotate the camera 180 degrees to do an "upside down" render.

After doing multiple tests, I have verified that the underutilization is related to GI cache complexity (there is a certain threshold), and so I was able to get my CPU back up to 100% utilization by decreasing GI quality, or decreasing render resolution, or as stated above, rendering upside down.  Weird huh!?

To give you an idea of how this impacted render times, I did two renders of the exact same scene with exact settings.  The only difference was one was rendered upside down, and the first rendered in 8 hours 57 min, while the second rendered in 4hrs 58 minutes.  So rendering upside down saved 4 hours in this case.  Rendering was done with a 12-core Xeon E5-2697 v2 CPU, with 64GB of memory.  I did a render test on a 6-Core Core i7-4930K with 64GB memory as well with the same odd underutilization behavior.

The test scene was rendered at 8000x4000 using a Spherical Camera, and a Crop Region was used to render 60% of the image.  The GI Prepass settings were 4-4-8, with Supersampling On.  The scene itself was made up of a couple layers of Cloud Layer v2, Atmosphere, but no Terrain.  I have screen captures to document all of this as well.

One interesting note is the slow down only occurs at the top of the rendering (think North Pole).  The GI cache itself renders at 85% - 100% CPU utilization, but once the rendering starts I saw about 20% utilization for the first 6 hours of the render.  At 6hrs 41min I was seeing 25-30% utilization, at 7hrs 34%, at 7hrs 19min 42%, at 7hrs 26min 59%, at 7hrs 31min 74%, at 7hrs 33min 79%, at 7hrs 34min 86%, at 7hrs 35min 90%, and finally at 7hrs 41min I saw 100% utilization until the end of the render.

Render testing at other resolutions has shown that the CPU slow down / underutilization starts to happen is at about 5000x2500 for my test scene.  I also discovered that removing one of my cloud layers (Cloud Layer v2) helped in getting my CPU back up near 100%.

Next thing to try is rendering my test scene on a Linux render node.

Derek



Offline Dune

  • Member
  • *
  • Posts: 12826
  • Corkscrew Bird
    • www.ulco-art.nl
Re: Terragen GI cache blending and Multithreading
« Reply #14 on: December 01, 2017, 03:06:43 PM »
That's most interesting! I am tempted to try an ordinary scene and render upside down versus normal, see what happens.
In case you still haven't seen enough of my work: www.ulco-art.nl