Digital Art Zone

DimensionTheory - 08 November 2012 01:19 PM

Some things of note...

When converting to .TIFF with the script included in UberEnv it's going to 16-Bit Floating Point. For all intents and purposes, this file along with the full resolution backdrop provide the same information as the HDR would at around 5% of the file size.

http://www.openexr.com/about.html

ILM decided to develop a new HDR file format with 16-bit floating-point color component values. Since the IEEE-754 floating-point specification does not define a 16-bit format, ILM created the "half" format. Half values have 1 sign bit, 5 exponent bits, and 10 mantissa bits. For linear images, this format provides 1024 (210) values per color component per f-stop, and 30 f-stops (25 - 2), with an additional 10 f-stops with reduced precision at the low end (denormals).

You can convert any .HDR into these separate parts without losing any of the functionality, so it's not a matter of what it produces. Just whether or not you want to bother with the act of converting files.

Are you quite sure about this?

1 The hdri tiffs supplied by omifreaker are 32-bit hdri tiff images, according to Picturenaut.

2 Omnifreaker explicitly states that the environment map tiffs for use with UberEnvironment produced by his SkyGen script are 32-bit

3 the 3Delight docs say that tdlmake supports 32-bit floating point tiffs as input files, but only signed and unsigned integer 16-bit, not floating point 16-bit tiffs. They also refer to the support of the logluv 32-bit tiff. tdlmake DOES support OpenEXR files - but OpenEXR files have filetype .exr, not .tiff. And while tdlmake supports .exr as an input filetype to convert to 32-bit tiff, the DS interface does NOT support .exr.

4 According to Picturenaut YOUR UrbanRecreation tiffs are 32-bit tiffs, not 16-bit. And they clearly are tiffs, not .exr files.

5 I just used HDRConverter.dse on an .hdr file, and it produced a 32-bit tiff.

I am therefore presuming that in fact HDRConverter is producing the 32-bit floating point logluv TIFFS mentioned in the 3Delight tdlmake docs, not 16-bit ILM-type files, which it can read as a source, but only to convert to what it uses.

I am probably recalling old information when things have changed. When becoming a PA my first set was a pack of HDRIs and they wanted me to support the "soon to be released" Daz Studio 3, that was UberEnv1. Getting it to 16-bit Float was a big deal as I remember it, because I had a hard time finding things that would handle it.

Well, I've rechecked to make sure I wasn't assuming anything unwarranted. Using Omnifreaker's HDRConverter.dse on some 32-bit per channel .hdr images definitely actually does the conversion with tdlmake.exe (rather than the hdr2tif.exe from 3Delight that also comes with DS), and produces 32-bit per channel floating point .tif images that are still HDRI images.

That is as opposed to running e.g. hdr2tif.exe on the .hdr images, which also produces .tif image files, but simple 8 bit per channel LDR TIFFs.

And the .tif images in this Urban Recreation set are full HDRI 32-bit per channel TIFFs, even if you didn't realise that's what HDRConverter and tdlmake were producing for you. :} Well, the file format is ... so the images are presuming you started with HDRI data.

Well that makes sense I guess, I'd been under the wrong impression with what it'd been doing. Don't think that it would have made much difference in how I used it though... Those HDRIs are 400MB a piece and UberEnv really only spits out lighting which doesn't need to use 16k maps, the .TIFFs here are greatly reduced and modified. If those original 16k 400MB .HDR files are something you guys want I can't do it for this set without completely restitching panoramas, but I could get them from my upcoming Monterey set depending on DAZ's say about files that big.

My understanding is that normally with HDRI lighting one uses small, convolved (spherically blurred) HDRI versions of the original images as the environment map for the lighting (as at the bottom of the page http://omnifreaker.com/index.php?title=UberEnvironment. Then the large, detailed (i.e. before the convolvinng and shrinking) HDRI as the environment map for reflections etc. And the flattened but very large, LDR on the skydome/backdrop.

If you don't have/haven't seen, Picturenaut is quite a handy donation-ware tool, especially with it running HDRShop plugins. And that site has quite a lot of good stuff, and links to other software and information sources. The Smart IBL section, while the software doesn't work with DS, still provides good examples and samples of the sets of 3 versions of the images to create together for best result (for backgound, diffuse light and reflection maps), and Picturenaut itself can do all the appropriate manipulation to create the HDRI and then produce the derived versions.

...OK that all went by me [makes gesture of hand passing over head].

Can this be explained in terms a semi-luddite artist is able to understand?

Kyoto Kid - 08 November 2012 10:44 PM

...OK that all went by me [makes gesture of hand passing over head].

Can this be explained in terms a semi-luddite artist is able to understand?

What - my previous post? Well ...

I'm not an expert at this stuff by any means, but AIUI -

A basic problem with using big HDRI images as the environment maps for Global Illumination, IBL etc is that when you combine the shear amount of data in them and hi-rez detail, you have to use very VERY high quality settings not to get odd artefacts appearing in the render where often abrupt transitions from one detail to another in the environment map affect the lighting in the scene. This means (a) needing a high poweer computer, and (b) humungous render times. (I'm not talking DS specifically here, but all the assorted rendering solutions like Lightwave, Maya and whatnot.)

So the usual solution is to shrink the HDRI image and apply a 'spherical convolutoion' to it - basically blurring the image a lot, but through a specific form of blurring that retains the optical properties of the light. This small, convolved image HDRI image is then used as the environment map for the GI, IBL, IDR. The blurring loses the transitions that cause the artefacts, and as the details have been blurred the size isn't necessary, but you still have the full dynamic range that makes it HDRI. So using the small, convolved map you can use much lower quality settings and get relatively quick results with a much lower power computer.

If you are doing proper ray-traced reflections, though, you do need reasonable detail in the map for that. So as the map for that you want an HDRI image, NOT convolved, but still doesn't need to be as big as what you use e.g. on the skydome as a background. And then you want a really big image for the skydome, but a regular LDR image.

So the general solution, as used with Lightwave, Maya etc, and also with the UberEnvironment products as with the files supplied by Ominfreaker, is that for an environment light set up, you start with a BIG HDRI map, and from it produce:

i) A big LDR JPEG or PNG version to texture the skydome.
2) A smaller HDRI version to use as the environment map for reflections.
3) A very small convolved HDRI version to use as the environment map for GI/IBL.
4) A standard distant light, lined up appropriately with the maps and appropraitely coloured to act as the sun, to give sharp shadows and controllable intensity for the sun light.

If you go to the Smart IBL section of the website I linked above - http://www.hdrlabs.com/sibl/index.html , then read that Overview page and the How it Works one in that section you'll see this discussed and explained, again noting this is general for all CGI applications, If youi then follow the sIBL archive link, you'll find a page with lots of lighting sets arranged for the sIBL software. Just download one. It's a zip. Open it and you'll ffind a set of files, as discussed above, whch are actually perfectly useable in DS.

I'm looking in the 'desert highway' set at the moment, and in the zip, for that one lighting set are:

An 8000*4000 jpeg (LDR) to use on the skydome.
A 1600*800 hdr to use for reflections
A 360*180 convolved hdr for the environment map for IBL/GI

All created from the one original 8000*4000 hdr, and together making up the set needed for a remder of that setting.

There's also a 600*300 jpg preview, and a 128*128 thumbnail reeady for use packaging for a particular app like DS or whatever, and an .ibl file. If you load the 'ibl file into a text editor, you will see that it's in fact a perfetly readable set of metadata (in the Smart IBL metadata format) which includes the colour (in RGB) and position for a matching sunlight, as a normal distant light of the application being used, like Lightwave, Maya ... or DS.

Note that these image files mare in the correct longlat format used by DS, so all the sets for download on that page are quite usable with DS. All you need to do is run the HDRConverter.dse in the Uberenvironment2 folder in DS on the two .hdr files (the reflection one ending _Ref.hdr and the environment map one ending _Env.hdr) to change them into tiffs, and you have the set of jpgs and tiffs to use with UE2 for the IBL/GI, reflections and skydome.

So you can use the sIBL sets in the archive as examples of what image sets to create for use in DS. And you cna also use them to stock up your library.

Final thing - go to the menu on the right side of those sIBL pages and Click on Software, and in the expanded menu click on sIBL-Edit, Have a read, and then I'd recommend downloading it along with some of the sIBL sets from the archive. Note two things:

1 pick a folder, say in your DS My library runtime Textures and call it sIBL, and unzip the sample sIBL sets into folders in that. Start sIBL-Edit and select that folder with the sIBL sets in in the preferences, and you will find that sIBL-Edit gives a neat interface to look through all the sIBL sets you've donwloaded, see the images and details on them, the sun colours etc. Neat, but even more ...

2 In sIBL, if you click to create a new sIBL set and then choose one large .hdr you've downloaded from somewhere else, sIBL-Edit will, in about 1 second, automatically create from the large .hdr (a) the fullsize jpeg for the skydome (b) an appropraitely reduced size .hdr for the reflection map (c) the appropraitely even smaller AND convolved hdr for the environment map, and (d) even a preview image and thumbnail. As I say, all you do is click to start a new sIBL set, select the source large hdr, and BAM - it makes thee whole set in a second. It offers the set of images made with the default settings, but in a UI so you can, if you want, alter sizes, change which bit is cropped for the thumbnail, apply all sorts of transforms. AND it will convert any HDRI projection to any other. So you can start by selecting a large mirrorball HDRI, then when the images are made, just change the 'projection conversion setting' to mirrorball to longlat and not only do you have the set of versions, bbut they've all been converted to the format DS uses.

Worth having a look, a browse and a play, IMO.

...hmmm, think I'll stick to LDP for the time being...

Now if only DreamLight will update it for 4.5

None of this is required knowledge to use the product, it's above and beyond usage. Everything being discussed by David has to do with using UberEnvironment in general, which I had to do to make this product. It's not something you need to worry about if you're using the presets provided, only if you're planning on making your own presets with UberEnvironment to use with this shadow catcher.

Kyoto Kid - 09 November 2012 09:41 PM

...hmmm, think I'll stick to LDP for the time being...

Now if only DreamLight will update it for 4.5

I believe he's said he's not going to.

Personally I still use LDP2 resources with DS4/4.5. Just sat down one evening and converted into a usable form. Started DS3, loaded up LDP2. Exported just the skydome and turned it into a prop. Back with LDP2, worked through each LDP2 environment. For each, switched to it and saved (1) Material preset for skydome, and (2) Light preset for the sun light (ONLY sunlight), naming the presets so they are in matching pairs for each environment: one light preset and one material preset. I now use in DS4/4.5 by simply: (i) load skydome and select; (ii) apply material preset for the sky I want; (iii) load matching light preset, which produces the matching sun light in position and colour for that sky; (iv) load UE2 GI (Bounce) light. Takes less time than it took me to write that, is actually a sensible, simpler, modern way to light the scene with the GI IDL rather than over a hundred sky and ambient regular lights trying to fake sky and ambient light as LDP2 does - and also produces better lit pictures than using LDP2 how it was. So that gives all the LDP2 skies with matching light, but in a converted form that doesn't need any plugin updating for current or future.

DimensionTheory - 09 November 2012 09:48 PM

None of this is required knowledge to use the product, it's above and beyond usage. Everything being discussed by David has to do with using UberEnvironment in general, which I had to do to make this product. It's not something you need to worry about if you're using the presets provided, only if you're planning on making your own presets with UberEnvironment to use with this shadow catcher.

I don't know if you've already seen that sIBL-Edit I mentioned - if not, you might find it a handy tool when making more sets.

I tried the Desert Highway set from the sIBL archive: ran HDRConverter.dse on the two hdrs to get them as tiffs, then loaded one of your Urban Recreation sets in DS, swapped the texture on the SkyDomeMesh to the JPEG in the sIBL set, switched the Environment map in the UberEnvironment to the small convolved environment tiff, and popped the (midsize, unconvolved) reflection tiff into the reflection environment map setting of everything I'd used ubersurface on. Altered the distant light 1 to the colour for the sunlight in the .ibl metadata file for the set, lined the distant light up with the sun position on the skydome ... and it works pretty well.

DavidGB, how do you line up the distant light correctly?

DavidGB - 10 November 2012 03:44 PM

DimensionTheory - 09 November 2012 09:48 PM

None of this is required knowledge to use the product, it's above and beyond usage. Everything being discussed by David has to do with using UberEnvironment in general, which I had to do to make this product. It's not something you need to worry about if you're using the presets provided, only if you're planning on making your own presets with UberEnvironment to use with this shadow catcher.

I don't know if you've already seen that sIBL-Edit I mentioned - if not, you might find it a handy tool when making more sets.

I tried the Desert Highway set from the sIBL archive: ran HDRConverter.dse on the two hdrs to get them as tiffs, then loaded one of your Urban Recreation sets in DS, swapped the texture on the SkyDomeMesh to the JPEG in the sIBL set, switched the Environment map in the UberEnvironment to the small convolved environment tiff, and popped the (midsize, unconvolved) reflection tiff into the reflection environment map setting of everything I'd used ubersurface on. Altered the distant light 1 to the colour for the sunlight in the .ibl metadata file for the set, lined the distant light up with the sun position on the skydome ... and it works pretty well.

Yes, but by doing so you caused a tear in the space/time continuum of the virtual universe.
Dimension Theory is not the content creator's name for nothing.

barbult - 10 November 2012 04:00 PM

DavidGB, how do you line up the distant light correctly?

If my head ever clears so I can think, I'm sure there's a simple way of working out the rotations of the light from the UV coordinates for it in the .ibl metadata file for the set (under the RGB values for the sunlight colour).

As I mostly can't think straight (combination of permanent pain, brain fog from permanent high-power prescription pain medication, and lack of sleep due to the pain) I just cheat. Created a primitive cube, scaled it into a thin pole long enough to go through the skydome, zeroed the rotation and trans of the distant light, rotated the pole onto the 'back' (non-pointy end) of the distant light, in line with it, parented the pole to the distant light .... then just rotated the distant light until the pole was poking through the middle of the sun on the skydome, at which point the distant light is also aligned with the sun on the skydome. That's another of those things that took longer to type than actualy to do. (Saved the pole as a prop in the appropriate folder so can just load, parent to light, rotate to pont through sun in future.)

It strikes me that the sIBL system is supposed to be software independent, with other people then creating scripts or plugins to use the sIBL sets (including the metadata) to load them into particular CGI applications, such as the loaders already existing for Lightwave, Maya and a few other apps. Now, I don't write script for DS, let alone plugins. But it seems to me that it should certainly be possible, if not relatively easy, for someone to write a script to load the sIBL sets in DS. Just needs a skydome of the right mapping, like DTs ... or is there a default one that comes with the Omnifreaker stuff that's in all DS installs? An sIBL loader script (or perhaps plugin, but I'd prefer a script if one would do) is something I'd certainly pay a bit for.

DavidGB - 10 November 2012 07:50 PM

barbult - 10 November 2012 04:00 PM

DavidGB, how do you line up the distant light correctly?

If my head ever clears so I can think, I'm sure there's a simple way of working out the rotations of the light from the UV coordinates for it in the .ibl metadata file for the set (under the RGB values for the sunlight colour).

As I mostly can't think straight (combination of permanent pain, brain fog from permanent high-power prescription pain medication, and lack of sleep due to the pain) I just cheat. Created a primitive cube, scaled it into a thin pole long enough to go through the skydome, zeroed the rotation and trans of the distant light, rotated the pole onto the 'back' (non-pointy end) of the distant light, in line with it, parented the pole to the distant light .... then just rotated the distant light until the pole was poking through the middle of the sun on the skydome, at which point the distant light is also aligned with the sun on the skydome. That's another of those things that took longer to type than actualy to do. (Saved the pole as a prop in the appropriate folder so can just load, parent to light, rotate to pont through sun in future.)

It strikes me that the sIBL system is supposed to be software independent, with other people then creating scripts or plugins to use the sIBL sets (including the metadata) to load them into particular CGI applications, such as the loaders already existing for Lightwave, Maya and a few other apps. Now, I don't write script for DS, let alone plugins. But it seems to me that it should certainly be possible, if not relatively easy, for someone to write a script to load the sIBL sets in DS. Just needs a skydome of the right mapping, like DTs ... or is there a default one that comes with the Omnifreaker stuff that's in all DS installs? An sIBL loader script (or perhaps plugin, but I'd prefer a script if one would do) is something I'd certainly pay a bit for.

David, thanks for the tips on the cube. That is pretty clever, and it works! This uses Tropical Beach from http://hdrlabs.com/sibl/archive.html
Edit: Swapped out image, because I forgot to turn on the shadow catcher in the other one. Even though her shadow falls on the towel, it made a small difference in the image.