to be honest, I tryed 16bit grayscale, but realized that for some uncertain reason my Photoshop ignores 16bit mode.. I mean, I set it up to RGB 16bit (tryed also Grayscale 16bit), also I have checked "Image -> Mode -> 16bit" and still my Photoshop document acts like 8bit. I bet Im overlooking some attribute..
Meanwhile I think I found a decent solution - still using 8 bit heightmap (little bit tweaked) and applyed Heightfield Smooth Operator to it, and on top of it I placed Power Fractal v3 with some low values, to add imperfections.
This is looking much closer to the desired result, than previous versions:
Tomorrow I will see what textures and shaders will do to it, maybe I will adjust the lighting slightly, to make it a bit lighter