I am monitoring a large 8k x 4k single frame/still render and only about 20% - 30% of my 12-Core cpu is being used. The only difference from other renders I have done is this one is doing some GI cache frame blending between 7 GI cache files using the "Equal Blend" option.
Is this normal CPU behavior when rendering with GI cache blending? I can't bear to watch my 12-core CPU only use 1/4 of it's power. :(
UPDATE: I restarted my render and chose to only blend 3 GI cache files and now I am seeing 100% CPU utilization.
Derek
Just curious as to why you are using multiple GI cache files blended for a single frame image...is it from an animation you looking to do at some point?
I was just doing an experiment to see if I could create a smoother lighting effect in my clouds. I animated the sun moving horizontally about 10 degrees, and rendered out the GI cache files for those 10 or so frames and then rendered the final still frame from a frame at about the middle of the Sun's animation using the set of previously rendered GI cache files. The idea was to get a smoother result of light penetration through the clouds. You could think of it as doing a super long exposure, or maybe a time lapse photo merge, but of just GI. :) Terragen has so many possibilities.
Quote from: D.A. Bentley on November 28, 2017, 11:09:14 AM
I was just doing an experiment to see if I could create a smoother lighting effect in my clouds. I animated the sun moving horizontally about 10 degrees, and rendered out the GI cache files for those 10 or so frames and then rendered the final still frame from a frame at about the middle of the Sun's animation using the set of previously rendered GI cache files. The idea was to get a smoother result of light penetration through the clouds. You could think of it as doing a super long exposure, or maybe a time lapse photo merge, but of just GI. :) Terragen has so many possibilities.
cool, be interested in seeing the results. never thought of that approach..carry on and please post results as you've piqued my interest.
Sure thing. I will definitely post an example soon. :)
GI cache blending has some overhead, mostly in memory use as far as I'm aware, but it shouldn't have *that* dramatic an effect on CPU utilization. That being said we don't generally recommend blending more than 5 GI caches, and often 3 is sufficient.
Also if you're using v3 clouds, be aware that their internal GI calculation is not stored in a GI cache file, unlike v2 clouds and the rest of the environment. A GI cache will have an effect for v3 clouds because some light does come from the environment, but their primary GI scattering is not cacheable at this time so you need to increase the Cloud GI Quality if you want to reduce flicker or increase GI quality/stability in animation (it's in the GI in Clouds tab of the GI settings panel inside the Render node).
- Oshyan
Hey Oshyan,
I am actually using all v2 Clouds in my scene, but that's good to know. So what about Easy Clouds? Do Easy Clouds use the GI cache, or are they like v3 clouds and use the "GI in Clouds" tab for GI Quality?
Also I discovered the CPU utilization doesn't seem to a problem caused by frame blending in itself. I have the same issue just using one super high quality GI cache file. For example, when I render out a GI cache file for my 8k x 4k scene using 8-8-4 (8 GI cache detail, 8 GI sample Quality, and 4 GI blur radius) the end result is a GI cache file of about 13GB in file size. This causes my single frame render to crawl at only about 1/4 CPU core utilization. When I use a lower quality setting like 4-4-8 I have no issues with my CPU being under utilized. I also have plenty of RAM (64GB), and I'm only seeing about 28GB of memory use when doing these 8k renders.
I guess I'll just have to not go so extreme with my GI caching. ;)
Derek
Easy clouds are V3 too. You can tell by the lighting tab, which is different from V2 clouds.
I have something interesting to report. So this render i am doing is using a spherical camera, and as the render proceeds further into the render (away from the stretched pole) my CPU utilization went back up to 100%. Could the Spherical Camera be the reason for the underutilization of my CPU I wonder?
As Dune notes, Easy Clouds use the "v3" cloud shading model (which is why they look so good :D).
Your findings regarding larger/more detailed GI caches are interesting. I honestly don't know if it's just the size/detail of the cache that would be causing a problem, especially since you have enough memory for it, as you reported. I wonder if you have tested yet how that might interact with your most recent finding of possible correlation between spherical/pole area rendering and underutilization?
In any case I can tell you that such high GI settings are very seldom necessary, especially if clouds are majority of your scene (as in the case of a skybox/environment map rendering, for example). I would say that anything above 4/4 is probably unnecessary except for situations where a majority of light is going to be bounced/indirect (like near/after sunset), or with detailed objects in the scene (e.g. architecture). Otherwise, with any normal large-scale Terragen scene, you will see minimal improvements past 4/4 under most lighting conditions.
Regarding your preliminary finding of possible correlation with underutilization and spherical/pole area rendering, that's quite interesting. Can you do a test or two to try to confirm?
- Oshyan
Thanks for the info Dune, and Oshyan. I still am learning, so I will likely keep doing things that are unnecessary until I know better. ;) Sometimes users like me are good at finding bugs too because we do uncommon things in the App.
Yes, I will do some more render tests to see if I can track down exactly what is causing this underutilization. I'll report what I find of course.
Derek
Sounds great, and yes we always appreciate people pushing the boundaries in unusual ways and potentially uncovering hidden issues. :D
Btw you might try rendering a crop of an area of your spherical that is not near the poles, to see if the underutilization still happens. Obviously you'd have to use all other settings the same, and you'd also want to make sure the crop area is big enough that it has enough buckets in it to fully utilize your CPU anyway. Perhaps just do a crop that cuts out the top and bottom 1/3 or 1/4 of of your image (i.e. render the middle half or third).
- Oshyan
I'm making some progress on this, and have made an interesting discovery I am trying to verify at the moment. I'll update this will all the finding soon.
OK, this may sound weird, but the solution to the underutilization of my CPU while rendering was to rotate the camera 180 degrees to do an "upside down" render.
After doing multiple tests, I have verified that the underutilization is related to GI cache complexity (there is a certain threshold), and so I was able to get my CPU back up to 100% utilization by decreasing GI quality, or decreasing render resolution, or as stated above, rendering upside down. Weird huh!?
To give you an idea of how this impacted render times, I did two renders of the exact same scene with exact settings. The only difference was one was rendered upside down, and the first rendered in 8 hours 57 min, while the second rendered in 4hrs 58 minutes. So rendering upside down saved 4 hours in this case. Rendering was done with a 12-core Xeon E5-2697 v2 CPU, with 64GB of memory. I did a render test on a 6-Core Core i7-4930K with 64GB memory as well with the same odd underutilization behavior.
The test scene was rendered at 8000x4000 using a Spherical Camera, and a Crop Region was used to render 60% of the image. The GI Prepass settings were 4-4-8, with Supersampling On. The scene itself was made up of a couple layers of Cloud Layer v2, Atmosphere, but no Terrain. I have screen captures to document all of this as well.
One interesting note is the slow down only occurs at the top of the rendering (think North Pole). The GI cache itself renders at 85% - 100% CPU utilization, but once the rendering starts I saw about 20% utilization for the first 6 hours of the render. At 6hrs 41min I was seeing 25-30% utilization, at 7hrs 34%, at 7hrs 19min 42%, at 7hrs 26min 59%, at 7hrs 31min 74%, at 7hrs 33min 79%, at 7hrs 34min 86%, at 7hrs 35min 90%, and finally at 7hrs 41min I saw 100% utilization until the end of the render.
Render testing at other resolutions has shown that the CPU slow down / underutilization starts to happen is at about 5000x2500 for my test scene. I also discovered that removing one of my cloud layers (Cloud Layer v2) helped in getting my CPU back up near 100%.
Next thing to try is rendering my test scene on a Linux render node.
Derek
That's most interesting! I am tempted to try an ordinary scene and render upside down versus normal, see what happens.
That is incredibly strange! My first thought when you mentioned flipping the render upside down was that the hardest work for the renderer to do was at the bottom of the image, but by the time the render process gets down there, it has fewer areas of work for render buckets than the amount of render work you have left to do. In other words you end up with a few render buckets - assigned to a single CPU thread each - that take proportionally longer, and thus the total render time is longer. This does happen on occasion and I have seen similar situations where flipping the camera fixes it because what then happen is these difficult buckets start rendering earlier on and can be crunching away while the rest of the image renders, and they have more time to finish before the rest are done, basically. So you put the hardest to render stuff at the beginning of the render process.
But... your situation isn't that way at all! Somehow and for some reason the *beginning* of the render is slower (lower CPU utilization) than later. Which is really... odd. This is definitely worth looking into further, and thank you for all the in-depth testing and reporting back.
- Oshyan
Another interesting comparison was I rendered the same exact scene twice; once with Hyper-Threading on, and a second time with it off. When Hyper-Threading was off my CPU utilization (according to windows 10) was 100% the entire render, but ironically it took about 3-4 hours longer to finish compared to the Hyper-Threading On render which was showing underutilization at times
Render with HT on took about 19 hours, and with HT Off it took 23 hours.
Not sure what to conclude from that, but that's what happened.
That's quite a difference. HT should add at most 20% of additional performance, and that's under ideal conditions. Typical is more like 10-15%. So that's on the edge of possibility. ;)
Remind me, are you Mac or Windows? And what OS version are you running?
- Oshyan
Windows 10 Pro x64
OK, good to know. Let us know the results of any other tests. Is the scene file you're testing with the same as the one you sent us?
- Oshyan
It's been awhile, and I can't remember what scene file I sent you. I haven't had time to do any further testing, but the scene I was using to do these tests were related to the spherical camera "Peachy Clouds" scenes. I can't remember for sure, but it might have been only happening with Cloud Layer v2. When I get back to doing more renderings I'll try and re-visit this.