NPR of Videos

Artistic rendering of Images

Image used as input
result from npr
Since videos have large no of frames manual specification of brush strokes will be very tedious. So I decided to use techniques which can do this automatically. One of such approaches is to assume that a brush stroke approximates image around each pixel and calculate the parameters of brush stroke. To calculate we chose a neighbourhood of square shape around pixel and calculate image moments of zeroth, first and second order. Using image moments we find the orientation, size etc of stroke. The ratio of first order and zeroth order moments give the approximate location of brush stroke in the neighborhood. The second order moments are needed for angle calculation. These formulae are easy to understand if we relate drawing a brush stroke to fitting an ellipse to same colored points. The axes can be calculated using information about variance and covariance. If we look closely first order moment is average and second order moment is variance and covariance. This approach was inspired from An algorithm for automatic painterly rendering based on local source image approximation" by Michio Shiraishi and Yasushi Yamaguchi.

Artists and painters normally follow a coherent theme in their painting. May be we too can improvise if we can orient each brush stroke based on the complete image rather than just the local neighbourhood. One of the ways to do this can be to use gradient information to draw strokes. But gradient information is meaningless in interior of objects. Using the values of gradients at the edges we can create hypothetical "edges" inside the objects. To do this we can use neural networks in conjunction with radial basis functions. Neural networks are widely used in AI applications to learn outputs based on given inputs. The nice thing about these networks is that they can learn anything! just as human brain is capable of adapting to varied inputs same set of perceptrons can learn different data sets. We exploited this idea to interpolate gradient values in the interior of objects and then paint the stroke perpendicular to them.(Taken from image and video based painterly animation by Irfan Essa). See the images below for results.

Image used as input
result from npr
Have you noticed than till now I completely overlooked the smaller details like canvas, the medium and brushes used in painting. But if we want to create effects similar to watercolor and paintbrush we have to account for the things like capillary effects on canvas. Watercolor over canvas can be viewed as three layers - over the paper is the part where shallow water dynamics is in action. On the paper-water interface the pigments get adsorbed and desorbed. Just beneath this layer the capillary effects occur. OF course we dont need to emulate the exact laws of physics involved. All we need is to produce effects similar to ones produced in real world paintings. This video was produced using simulation techniques discussed in Computer genrated watercolor by curtis. Notice how the stroke changes with time.

Now comes the part where we simulate the brush strokes and try to make them just as an artists brush is moving on the canvas. On the side you can see some brush strokes that I modeled using plain physics laws. Since fibers are made up of same material we can assume they have same Young's modulus. Based on this we can calculate the extent of their compression when we place a stroke with certain amount of pressure on paper. Since paper is uneven pigment from stroke gets transferred selectively. In a flat brush small variations in fiber lengths produce striated effect in brush stroke. This can be modeled by adding a small amount of pressure where brush stroke is longer than the average. The nice "meandering" can be modeled as a shift in center of stroke along its length. Canvas can be modeled as a height field and we can use pseudo random number generator to initialize it. All these effects can be combined to produce nice effects shown in the image on right.

Application over videos >>