NUIPaint 0.1 Screenshots

Here are few of the screenshots of the application.

Initial Screen

Initial Screen

Collaboration Windows

Collaboration Windows

Fullscreen Mode

Fullscreen Mode

Windows in Fullscreen mode

Windows in Fullscreen mode

Layers within Canvas

Layers within Canvas

Filter Application

Filter Application

Comments, Suggestions and Criticisms welcome 😀

Advertisements

NUIPaint Pre Final Eval Report

Being selected as a GSOC student is one of the best thing that has happened to me in life. Not only being selected, but selected as a student learning under the two most admired people by me i.e. Mathieu Virbel and Thomas Hansen was the reason for my double joy. I thought that being a GSOC student will benefit me in getting a better job in the future. Man was I surprised as I progressed through the programme. So in this blogpost I will be summarizing how I learnt new things, altered my previous plans, made new plans to adjust to the schedule.

Phase 1 (Till Midterms)

This OO based programming was pretty new to me. Plus NUIPaint is my first actual application that I have coded. So I had very little idea how to start and where to start? So I set out building the UI as I provided the Mock-ups. But little did I know starting with UI is not a good way of software engineering. Mathieu suggested that I start building pymt widgets first and then integrate them one after another. So this was lesson number one for me “Modular Coding => Better Structure and better to handle”.

In my first phase I built several UI widgets, they are as follows

  • Circular Slider Widget
  • Circular Menu Widget
  • Quarter Circle Colour Picker Widget
  • FileBrowser Widget
  • Icon Widget
  • Tiny Colour Picker
  • Toolbars with Background
All Widgets

All Widgets

Descriptions of some of these and how I created them, I have already posted in my previous blogs.

With these widgets plus a simple implementation of layering system, I closed the Phase 1 of the project. So in the next phase I decided to integrate all these and focus on building the application aspect of the project.

Phase 2 (Post Midterms):

So I passed the midterms with the blessings from my mentorsJ, in this phase I wanted to focus more on the application aspects like building the layering system, GLSL filters, Natural Colour Mixing,   smudging, windowing, cut/copy/paste and others.

  • Layering System: Layering is one of the most important parts of any painting application. So I had to carefully plan how to build this, instead of directly jumping into coding, so I discussed this with Mathieu, he showed me how a pattern called Facade can be used to achieve this framework. I have finally implemented the following layering structure with Layer Manager being the Facade.  Using layer list manager it’s possible to merge layers, create new layers, multiple layer merge, multiple layer delete etc.  Layer list manager provides an interface for accessing the layer manager.
Layer Manager

Layer Manager

One more important feature I have implemented in layering is Layer ordering using gestures. Double tapping on a Layer takes it one level higher than its neighbour. Similarly holding one finger on the layer and double tapping on the same layer with another finger takes it one level lower.

  • Multiuser Windowing: There is all provision for multiuser collaborative windows.

    Collaborative Windows

    Collaborative Windows

  • Observer Architecture: Two days back it struck me that instead of passing parameters and parameters of current canvas, current layer manager, current layer etc. It’s much better if there is some entity which always keeps tracks of these things. So I implemented a new Observer Factory. This Observer is global throughout a program. It holds all latest information of various active objects in the system. So if some part of the application wants to know who the current canvas is, he just needs to query the Observer. This really simplified the entire programming routine.
  • Copy Cut Paste Architecture: The copy paste system I implemented is a very simple one its basically one class in the each instance of NUIPaint window. It consists of one Data store which holds the copied or cut texture. When pasting this data store is queried for the texture and a new layer is created in the target window. Similarly for cutting, the source layer is cleared after copying the texture of the source to the data store
  • GLSL Filters: This part of the project took a really long time to materialize, because I had no idea what GLSL means, how it works. My goal was to implement a separate Filter class, so that any texture can be altered using simple method calls like blur(texture, value), sharpen(texture,  value). So it took 10days+ to understand the basics of glsl and implement 7 simple image processing filters. Following are the filters that I implemented Gaussian, Blur, Sharpness, Saturation, Brightness, Contrast, Black and White, Sepia.

first fourother filters

  • Natural Colour Mixing and Smudging: I’m still trying to figure out this one, I have a vague implementation now based on Gaussian Blur. Need a better algorithm for this one.

The sheer size of this project was overlooked by me during the initial proposal writing phase and there are several other aspects like focusing on development of reusable widgets for PyMT lead me to have shortage of time. But I’m very keen on continuing this project even after the GSOC, and take it to the next iteration and make it a truly complete pymt app.

As for my satisfaction about this Project. It’s been a really great experience for me, I have learned some really high technical stuff, I feel that now I have a better understanding about how the software engineering process works. And of course I really thank my Gurus(mentors) for teaching me and always being there to answer my questions and clear my doubt.

In my next blog post i’ll post a series of Screenshot of how NUIPaint looks as of now. Cheers!

Pixel Shader Magic

One thing which really fascinated me from really long is use of Pixel shaders to do interesting things in OpenGL. I’ve been trying to decipher the mystery of GLSL from a long time now, this interest in GPU based computation was sparked especially from Thomas’s Example apps in the PyMT trunk( Mandelbrot and Shadowing in Pictures example), although I never understood how it works back then (or might be because I never put much effort in understanding it).

GL Shading Language is a extension to OpenGL using which we can directly compute pixel and graphics on the GPU instead of the CPU. This provides us with tremendous performance benifits since pixel transformations run in parallel. The scope of Pixel shaders is amazing, some of us sometimes dont know how they achieve those stunning effects in latest computer games. Most of the effects like

  • Motion blur
Motion Blur

Motion Blur

  • HDR Lighting

HDR Lighting

HDR Lighting

are all done utilizing the pixel shader capabilities of your graphics card. So I started out by searching some  tutorials on the net and there are several good references that i found out.

GLSL is all about pixel manipulation, usually involving usage of neighboring pixel colors, distance etc etc. Its mathematics. But there are a lot of free GLSL code available online using which you can learn and implement your won shaders.

Once I learnt the basics of GLSL, I decided to put effort to further learn this technology and implement it in NUIPaint as filters. I found out that some common filters like Gaussian Blur, Sharpening, Embossing etc belong to a common group of shaders called as Convolutions Filters. These vary only by a single matrix called as Convolution Matrix. Its very interesting, for more details please check this link http://www.ozone3d.net/tutorials/image_filtering.php .

I implemented a separate class called Filters which will later be added into PyMT core. This class has several functions which take in texture as input, apply GLSL filters to it and return the transformed textures back. In total i implement 7 kinds of filters

  • Gaussian Blur
  • Sharpness
  • Saturation
  • Brightness
  • Contrast
  • Black and White
  • Sepia

Here is a set of screenshots of the Filtered images.

first four

other filters

It works much faster than regular CPU based filters like the ones in Python Imaging Library. If you want to use the glsl filters then here is the link to the source code, each filters vertex and fragment shaders are in separate functions, so you can easily use it in your programs. If you want to use the Filter class into your software then wait for the induction of the class into PyMT core soon 🙂

A Color Picker for PyMT

It’s been long since I last posted about my GSOC work, so today I’ll be writing about how I achieved the color picker. There are several points that comes to the mind when we thing about the colorpicker.

  • How to achieve the gradient color selection area
    • Use a pre-rendered image ?
    • Generate using computer algorithm ? If what are the appropriate opengl calls?
  • What is the algorithm for picking the color from the location ?
    • Use opengl methods to pick the pixel from the point where the user touches ?
    • Or use some algorithmic technique to determine where the user is touching local to the gradient and then calculate the appropriate RGB values

There is a humongous list of resources available for Color pickers especially AJAX based, but my problem was different, I wanted a Quarter Circle Color Picker 🙂 , nowhere in the internet did i find one single open-source quarter circle color pickers, but hey no pain no gain :), so I set out to create a Opengl rendered quarter circle color picker with algorithmic based color picking.

1. Rendering the Gradient Circle

So my first goal was to create a opengl rendered quarter circle, well its very easy to generate a single color circles in opengl like the ones that I have used in my previous widgets, Rendering a gradient one is a very hard task. I know that opengl allows to blend colors by specifying different colors at different vertices of a polygon, but how can one do the same in a circle ?!!, circle has no vertex edges.

A Color Blended Triangle

A Color Blended Triangle

GL_TRIANGLE_FAN

I did not find any way to get around this problem using Circle render, so the alternative solution was to render thin triangles and arrange it in the form of a fan, and when i looked around the net for this one, to my surprise there was one opening call to make such a polygon using triangle fan.

def drawPartialCircle(pos=(0,0), radius=100):
    with gx_begin(GL_TRIANGLE_FAN):
        glColor3f(0,0,1)
        glVertex2f(0,0)
        for angle in range (90,185,5):
            glColor3f(sin(radians(angle-90))*sqrt(2),cos(radians(angle-90))*sqrt(2),0)
            glVertex2f(int(cos(radians(angle))*radius),int(sin(radians(angle))*radius))

So in the above algorithm i’ve chosen the angle range from 90 to 185 because i want it to appear at the bottom right corner. So what the algoritm does it,

  • It generates 5 degree triangles to form a quater circle.
  • The center of the circle is blue so the other two edges of the two radii will be red and green respectively.
  • The mid way of the circumference must be yellow that is RGB =>(1,1,0) , so to we know that only at 45 degrees sin 45 = cos 45.
  • sin 45 = 1/sqrt(2), so to obtain 1 at the center we multiply it by sqrt(2), so we get (1,1,0) at the center.
Opengl rendered Quarter Circle

Opengl rendered Quarter Circle

If you have any queries regarding this algo, feel free to comment.

2. Calculating color at the touch point

I used the similar sine cos formula to calculate the RGB values, The R and G values varies according to the angle made by the line joining the center and the touch point with the horizontal line. Whereas the blue value is a function of the distance touch point from the center. So here goes the algorithm

def calculate_color(self):
        b = 1-self.point_distance/self.size[0]
        r = (sin(radians(self.point_angle))-b)*sqrt(2)
        g = (cos(radians(self.point_angle))-b)*sqrt(2)

where self.size[0] is the radius of the circle.

3. Hue and Saturation

As you can see in the rendered quarter circle above there is no range of black nor white, so this posed a new challenge to use a slider to make a hue saturation variation, HSV[Hue, Saturation, Value]  is an alernative representation of RGB color space, something similar to Rectangular to Polar coordinates conversion.

HSV Representation

HSV Representation

Since I need to vary the color from White>>Color>>Black, the variation has to be done in two steps

  • White >> Color variation, which is nothing but variation of the Saturation, here saturation = 0 implies White
  • Color >> Black which is variation of the Value field

Thomas (My mentor) pointed me to this really nifty function in core python rgb_to_hsv and hsv_to_rgb , which converts between the colorspaces with ease. So I divide the slider into two portions lower limit to middle value for saturation, middle value to upper limit for Value. I chose to take 2 as the maximum range of the slider, so it varies from 0 to 2, and middle i get 1. I developed the following algorithm.

value = rgb_to_hsv(self.slider.slider_color[0],self.slider.slider_color[1],self.slider.slider_color[2])
h,s,v = value[0],value[1],value[2]
if self.slider._value <= 1.0: s = self.slider._value else: v = 2-self.slider._value self.slider.slider_color = hsv_to_rgb(h,s,v) [/sourcecode] Overall this entire widget was very complex to build, but I got to learn alot from this tiny project. If you have queries regarding any of the algorithm that I developed, or if you have suggestions of better algorithm, or anything at all feel free to comment :), it will definitely help me make this project better

A Circular Slider

Sliders are very nifty UI control tool, they provide some sort of natural feedback to the user visually when they interact with it. Especially in a touch user interface sliders are very useful. In PyMT we already have Horizontal and Vertical rectangular sliders.

PyMT rectangular sliders

PyMT rectangular sliders

But I needed something different, a circular slider for the next widget that I wanted to develop a Quarter Circle color picker. So I sat down with the sketchbook again and noted down the following pointers

  • The Slider must provide a way to limit the angle of the slider i.e not full 360 always, meaning it should provide me a way to select 60 Degree Circular sliders too.
  • Must be rotatable so that it can be set in any direction.
  • Must provide a way for customizations like color, size etc.
  • Must provide all features of a regular rectangular slider.

The three main things that i had to take care when designing the algorithm was that

  1. Drawing the Ring in Opengl
  2. Angle Calculation to fill in the slider as you touch it
  3. Collision detection, as the existing collision detection code was for rectangular widgets

Let start with each part one at a time

1. OPENGL Ring

I was wondering how I can make a ring like structure in opengl, I thought maybe make two circles one of the outer radius another of the inner radius, with the difference between the two being the thickness of the ring. But a bit of browsing on the internet revealed that there is a OPENGL call to draw the exact same thing, and its very customizable too.

 void gluPartialDisk( GLUquadric* quad,
			       GLdouble	inner,
			       GLdouble	outer,
			       GLint slices,
			       GLint loops,
			       GLdouble	start,
			       GLdouble	sweep )

you can read more about gluPartialDisk here. This really simplified the drawing part of the slider.

2. Angle Calculation

Slider Angle Calculation

Slider Angle Calculation

The angle calculation was easy once i found out how to do it,

  • First Make a vector of the one edge of the slider w.r.t the center.
  • Now as the touch is moved on the slider, take one more vector at the current touch location w.r.t to the center
  • Find the angle between the two vector, this is the fill angle for the slider

3.  Collision Detection

Next problem was collision detection algorithm, since the sliders where in a arc form, the collision detection has two boundary conditions and which itself formed the algorithm for collision detection.

  • The distance of touch from the center of the slider must be lesser than the outer radius and lesser than the outer radius-thickness of the slider
  • The Angle created by the current touch location and the vertical axis must be greater than the angle of the start edge of the slider with the vertical axis and lesser than the angle of the end edge of the slider with the vertical axis

Using the above two conditions I wrote the following code which works very nicely

def collide_point(self, x, y):
    #A algorithm to find the whether a touch is within a semi ring
    point_dist = Vector(self.pos).distance((x, y))
    point_angle = Vector(self.radius_line).angle((x - self.pos[0], y - self.pos[1]))
    if point_angle < 0:
       point_angle=360+point_angle
    if point_angle <= self.sweep_angle and point_angle >=0:
       return  point_dist<= self.radius and point_dist > self.radius-self.thickness

Here is a screenshot of the circular sliders, this was taken by my other mentor Thomas, see how they can be stacked over one another and still the collision detection algorithm can work flawlessly

Stacked Circular Sliders

Stacked Circular Sliders

You can read more about the widget here. In the next blog i’ll be posting about Color Wheel, and that involves alot of trigometric equations 🙂 .

PyMT update

I’ve been working with python from the time i lasted posted, i love the language it simple and quick to program and powerful too. I’ve been working with Thomas Hansen and Mathieu Virbel to develop PyMT( A python multitouch framework started by Thomas),I’ts a widget based framework which use opengl acceleration to draw graphics on the screen, very easy to develop apps because of the widgets, there are several widgets available using which you can created rotateable scaleable objects on the screen.

I did committ some code to it, some of the things that i did with the help of them are

Python Particle System

Python Particle System

Python Multitouch Photo app

Python Multitouch Photo app

Video Widget PyMT

Video Widget PyMT

I’ll soon upload a video demoing the working of them if you cant wait download the latest svn from http://pymt.googlecode.com/svn/trunk/

We'll be soon releasing a beta version of PyMT.