Lens Flares Revisited

I was just watching the excellent behind-the-scenes material on the Blu-Ray release of the new Star Trek film. J. J. Abrams discussed his choice to use lens flares and fake dirt/moisture elements to enhance the realism of CGI space shots. While he may have gone a little too far in this particular instance, I believe lens effects are a very helpful technique to make CG imagery appear less artificial.

I developed a custom lens flare tool for my Mars Phoenix animation project in 2006. Although lens flares are traditionally a feature of 3D rendering packages, I chose instead to handle them as a 2D compositing element. I implemented a simple “renderer” that draws radial glows and discs based on the projected location of a light source in screen space. My custom pipeline integration package (interp) already had the ability to color each pixel of a 2D layer based on an arbitrary function of x,y position. Using this feature I was able to implement the flare renderer entirely in script, without adding a single line of C++ code to the core interpreter.

The original script takes as input the screen-space location of a single light source (as determined by sampling the baked motion data and transforming it through the camera-to-screen projection). Then it generates a list of radial glows and lens reflection rings, and finally goes down the image line by line, iterating through the list and adding up the contribution from each element.

This system works great for single point light sources like the sun. But today I wanted to push the technique further, to handle many sources and large area glows. I imagine the ultimate lens-effect system would not require you to specify the light source locations numerically; instead you should be able to give it an HDR image and then automatically generate lens flare for each sufficiently bright pixel. This way you could design the flare effect just once, and then apply it to any scenario like multiple small point sources or large glows and trails.

It was not too difficult to implement this within interp. I just had to add one C++ function to scan through an image and return a list of all the non-zero pixels. I used script functions to darken and clamp the source image so only very bright pixels would be identified. Then I instantiated a glow-and-ring lens flare for each pixel in the list.

Here are the results:

I was particularly surprised by the smooth appearance of anamorphic radial glows. In the past I had always made these using a 2D blur filter. But the new flare renders looked a lot better. It’s probably because they use a true radially-symmetric blur, as compared to the blobby Gaussian effect you get with a separable 2D filter.

Note the superior appearance of the horizontal glow bands in the “radial glow” image compared to the glow that was based on a 2D Gaussian blur.

Unfortunately, the renderer slows down from a few seconds per frame to a minute or more when it has to draw more than a handful of flares. Despite interp’s efficient line-by-line SIMD compositing architecture, there is still too much interpreter overhead for this operation. At first I thought about implementing more of the glow-and-ring drawing in C++, which would involve finding efficient bounding boxes for each element and sorting them in screen space. But given that I have RenderMan handy, it would be easier just to translate the necessary operations into a RIB scene and let PRMan draw it.

I will probably use this technique for my next 3D production. It can produce beautiful area glows and will save me the effort of specifying individual flare elements for the sun and specular glints.

The New Client Application Landscape

Back in the early 2000s, selecting a development platform for client software applications was easy. You wrote C++ code using the native OS APIs (Win32, Carbon, X11) or thin wrappers on top of them. It also became practical to write clients in higher-level languages like Python or Java, provided you were willing to put up with some rough edges in the implementations of GUI toolkits and cross-platform issues.

There’s nothing fundamentally wrong with these platforms, and despite all the hype about web applications taking over the world, I think it’s perfectly reasonable to develop new client software with them (provided you “pay your taxes” to keep up with modern OS requirements like packaging systems and user data persistence). In fact, good cross-platform toolkits like Qt and WxWidgets have made OS-native client development easier over time.

However, as I consider developing a new, graphically-intensive client program intended for a large user base, I feel obligated to look at browser-based platforms, specifically Flash and JavaScript.

Flash

The Flash Player has already become popular as a game client platform, notably for Zynga‘s Facebook games, and even some MMOs. It has no standards-compliance issues because there is only one canonical implementation. Its sound and graphics features are solid and well-optimized, although the 3D API is still pretty limited. There are some GUI controls libraries for Flash, but they are not as well-developed as those in a desktop OS or a good JavaScript framework.

JavaScript

From a desktop developer’s point of view, JavaScript is essentially a bare-bones scripting language interpreted by the browser, coupled with a small “standard library” including an API for manipulating the browser’s layout and display engine (the HTML DOM). JavaScript applications work by pretending to be a web page with various elements that appear, disappear, or move around according to the script. There are many “widget libraries” for JavaScript that emulate desktop GUI controls to some degree of success. Although JavaScript-based controls sometimes feel clunky, it’s important to remember that you get the full benefit of the web browser’s page-layout engine and its HTML controls (buttons, sliders, etc) for very little code. This could be a powerful advantage for writing GUIs that make heavy use of traditional desktop-style controls.

Future browsers will (hopefully) also include WebGL, a complete JavaScript interface to OpenGL which renders into an HTML canvas. In theory, you could port most sophisticated rendering code, like a modern 3D game engine, to this interface. Unfortunately, WebGL is just an emerging standard and there is a risk that browsers won’t ship it widely or support it well. It could join the long history of widely-promoted but seldom-implemented standards like VRML or the non-audiovisual parts of MPEG.

Limitations of the browser as a platform

Flash and JavaScript both have some important limitations:

  • You can’t communicate over arbitrary TCP sockets. In both platforms, client/server communication has to use XML-RPC features.
  • You have no control over the “event loop.” This is a potential source of problems with input polling and timing control. In particular, the WebGL code I’ve seen uses a wonky technique for frame timing: the script schedules an “update” function to run at periodic intervals (or with a zero timeout, which tells the browser to handle inputs and then return immediately to the script). This feels clunky compared to the high degree of control you get with a native client, which can explicitly select on file descriptors with precise timeouts and/or pause for graphics buffer swaps.
  • The programming language will be different from the server side, making it difficult to share libraries and move code back and forth. There are a couple of “language translators” that target JavaScript (Pyjamas for Python and GWT for Java) but this seems like a fragile approach.
  • Client-side storage options are limited. There are ways to store small stuff, but typically not hundreds of megabytes of 3D models and textures. I suppose you could do something crazy like install a small native web server on the client and access it via HTTP to localhost.

How to Decide

In the end, the decision of which platform to use will depend on a few factors:

  • How much does the application resemble typical desktop GUI software?
  • Are you continuously rendering a changing scene, or does the app sit and wait for input?
  • How much client-side storage is needed, and is client-side software installation going to be a problem?

Server Platforms

On the server side, not much has changed over the last few years. The first tool I’m going to reach for will still be Python, writing directly to the Linux POSIX APIs. Python has a rather nasty (though constant) performance penalty versus compiled code, but I can usually get around that by selectively translating critical sections into C.

(The only situation where this approach might fail would be a system that had to run arbitrary, hard to vectorize scripts very very fast – something like a game engine simulating a large number of independent AI agents. In this case you might need to look into a more JIT-oriented platform like Java. However, without going this far I think it’s possible one could get the performance penalty low enough to make it reasonable just to throw more hardware at the problem.)