A Graphical User Interface Library for Processing
Proposal for Google Summer of Code 2013
Abstract.
The goal of the project outlined in this proposal is to create a functional prototype for a new Graphical User Interface Library for Processing. There is no core GUI system built into Processing as of date but some third-party GUI solutions exist. This project attempts to learn from and build on the existing libraries while keeping in mind the design principles that lie at the heart of Processing: Minimalism, accessibility and extensibility. Ultimately the aim is to spark discussion on what a core GUI library for Processing might encompass.
Motivation.
As of yet there is no 'core' GUI library built into Processing. There are a few existing third-party ones, with different sets of features, and more or less popular. They range in size from a few hundred lines to over a megabyte of source code. Some are extremely feature rich (like controlP5) but knowledge of method signatures is needed. Others seem easier to set up but lack even some of the most basic controls. I started wondering what a core GUI library for Processing could look like and the first key word that came to my mind was accessibility. In the end, it should be pretty easy to throw together a quick UI for controlling some parameters of your sketch.
So I tested four of the GUI libraries linked at http://processing.org/reference/libraries/#interface for how quickly I was able to make a simple sketch controllable via a GUI. The sketch was just a circle with two parameters: Its radius and whether to fill it or not. It took me from 7 to almost 20 minutes to get this working, one way or another, for each of the libraries. While this may not be indicative of the quality or ease of use of any of these libraries, I still learnt a few things about what I expect from a GUI system. I don't want having to remember all the method signatures of all the different widgets. To begin with, I just want to select the type of control and hook it up to a variable of my sketch (→ Accessibility). This might be enough for many sketches, only when they get bigger and more complex I might want to start investing more time into the UI, tuning and refining it. So when the need arises, the library should support ways to customize appearance and functionality of its controls (→ Extensibility).
Regarding design (visually & functionally), I think it's best to keep things minimal and simple. This should put the focus on clarity, functionality and ease of use. In my opinion this is another integral principle of Processing itself (→ Minimalism).
My aim is to build on existing UI solutions by combining some of the best ideas already out there while specifically keeping in mind what I think are some of the core design principles of Processing.
Design Principles & Goals.
* Accessibility. The library should be easy to set up and use. It should be easy and painless to quickly throw together a UI to control some parameters of a sketch. This is in spirit of the 'sketching metaphor' of processing.
* Minimalism. Graphic elements should be clean in design, easily readable and understandable. A basic but usable set of user interface elements should be provided.
* Extensibility. Allow for customizing the display of UI elements and support creating completely new ones regardless of the type of data they represent.
Features.
* A basic set of UI elements.
Slider. Horizontal or vertical. Has a range (minimum and maximum value) and a step size.
Button. Click to trigger some action (a.k.a 'bang').
Toggle. Like a button but toggles between 'on' and 'off' states.
Label. Displays textual info.
* Possible extra elements.
Number Box. A Numeric Value. Can be entered manually or altered by dragging. Has a range and a step size.
Radio Buttons. A Set of Toggles. Exactly one of them selected i.e. 'on'.
Range Slider. Select a range out of a given, bigger range. Useful for setting ranges for random values, for example.
Rotary/Knob. Like a slider but with circular knob-like design.
XY-Pad. Two-dimensional control. Has two ranges and step sizes.
Text Input. Enter textual information. Might be single or multi line.
Graph Display. Display continuous graph data (e.g. waveforms). Might advance automatically.
Fig.1. Basic User Interface Elements. Sliders and Buttons
* Sensible Defaults. For the sake of accessibility, it should be possible to create UI elements without needing to know the constructors signature. Ideally, all UI elements feature an empty constructor which will employ default values for size, position and other parameters. More specifically, position can be derived from a column based grid, and sizes will be defined in relation to this underlying grid as well. Default values for sizes and proportions need to be chosen carefully in order to have a usable GUI even when nothing is customized.
* Grid based Layout. A simple square grid will be employed to enable automatic (and manual) layout of UI elements. When using automatic layout, elements will be placed next to each other along the grid, subsequently filling the space and adding layout columns as needed. Also, default element sizes will be defined in multiples of one grid square, thus allowing for quick customization of the UIs size by changing the grid spacing.
Fig.2. Grid and Column Based Layout
* Graphical Layout Adjustment. After adding UI elements by creating them within the source code, they can be adjusted during runtime. This is accomplished by entering 'UI edit mode' (via a hotkey) and dragging them around and possibly resizing them by dragging from their borders. The underlying grid is shown in this mode and assists with placing elements (snap-to-grid). The new layout created in this manner needs to be saved to file and restored when the sketch is run again to make it persistent.
* Minimal 'Theme'. A Theme will consist of 4 Colors together with a typeface (for labels and other textual information). This should facilitate clarity and also help reduce overhead added by the UI. By changing these colors and the grid used for sizing components the basic look of the UI can be customized quickly.
Overlay Color. Used as the background color for the overlay the UI is drawn on. Can be semi-transparent.
Background Color. Interface Element Background/Inactive Color
Foreground Color. Interface Element Foreground/Active Color
Highlight Color. Used for hovered, selected, or active elements and outlines. Might be a lighter shade of the foreground color.
Fig.3. Example Themes
* Auto connect UI to variables. By providing a name of a variable or method to any UI element, it can be bound to that variable or callback function. In case of a variable, using the UI element will update the variable accordingly. In case of a function, it will act as callback function and will be called when the value of the UI element is changed, with the new value as parameter.
* GUI Overlay. The GUI is drawn on top of the sketch. This overlay can be show and hidden with hotkeys. Later, support for separate control windows can be added.
* Save and load parameters. The current state (i.e. the represented values) of all UI elements can be saved to disk and restored using hotkeys. Additionally a persistent UI feature will be added, where all changes are saved and restored automatically upon restarting the sketch. Taking this idea further, state can be saved to different files, thus creating presets for the collection of parameters represented by the GUI.
Project Outline.
* Survey of UI Libraries. At first I want to do a quick survey of existing UI solutions. An obvious candidate to learn from is the popular controlP5 library by Andreas Schlegel. I also want to look into UI frameworks for other coding platforms like openFrameworks and Cinder. Investigate how they are doing things while keeping in mind the main goals formulated above.
* Base Setup. Resources available on the Processing Wiki will be used to set up the basic structure of the library. This should be done pretty quickly.
https://github.com/processing/processing/wiki/Library-Basics
https://github.com/processing/processing/wiki/Library-Guidelines
* Build the core system. At the core of the UI Library is an event handling system. An interface element lies within a rectangular area and needs to respond to user events such as mouse movements, clicks and drags. The event system takes note of these events and forwards it to the appropriate UI element so it can respond to it. The library can receive mouse events by using PApplet.registerMethod('mouseEvent', …); I will use GUIDO by Florian Jenett as a base for this stage.
* Add basic UI components. Once the event handling is in place, the first basic UI widgets can be added. For this stage a Button/Toggle and a Slider will be enough. I want to work out the interplay of core features before I start adding more UI elements. In this stage I work on connecting controls to variables and methods using Java's Reflection API (java.lang.reflect). I also need to think about how to deal with different kinds of data a - possibly user-provided - interface element might represent. A major challenge will be to find a way to capture different data types while still keeping things as simple as possible. For the sake of flexibility, arbitrary data should be allowed.
* Advanced Features. As soon as the fundamentals are laid out, work can begin on Automatic Layout, Saving state and Theming as described above.
* Add more UI components and Refine API. Adding more widgets obviously adds value for users but it's also a good test for the API developed so far. It needs to be flexible enough to allow for different, maybe even 'exotic' UI elements.
Notes.
* Documentation. This is a very important thing. There are some things you need to know, like the class names of the available controls. I aim to provide concise but thorough documentation of all the features in the library. Source code will be documented continuously to facilitate further development in the future.
* Platforms. For now, I will focus on the Desktop variant of Processing (i.e. Java Mode). There is some desire of having a GUI Library working in JavaScript Mode as well (http://forum.processing.org/topic/a-new-gui-library#25080000002106253). Android compatibility is another possible direction for further development, but will require more substantial work, since it needs to respond to touch events.
* Technical notes. Development will be done using Netbeans or Eclipse on OS X. All code will be publicly available on a Github repository and licensed under an applicable open source License, as required by Googles Terms and Conditions (most likely either GPL or MIT). Ant build scripts will be provided as well as precompiled bundles ready to use with Processing.
* Availability. During the timeframe of Summer of Code (June to September) I will be available full time (40 hours per week) for at least a month. The rest of the time I will be available for at least 20 hours per week. I am pretty flexible in this regard, so other arrangements can be made as needed.
Personal Background.
I started programming when I was a kid with GW-BASIC, Logo and Redcode. I was fascinated by A.K. Dewdneys ‘Armchair Universe’ and ‘The Magic Machine’. In High School I got interested in Fractals and used Visual Basic for exploring them and generating videos.
So, in 2003 I moved to Vienna to study Media Informatics at the University of Technology. We worked mainly with Java and C++, but I also used Csound and Pure Data to work on music and video.
During that time I began working as freelance web developer, getting into JavaScript & PHP, doing visual effects, custom user interfaces, graphics and plain web coding.
By 2006 I felt a lack of a more immediate way of expression through coding and that’s when I came across Processing. It was a great feeling to have the immediacy of visual output paired with the familiar and powerful Java. So I did visual experiments producing graphics and animation using simple geometrical building blocks and images. I also found Processing's sketching metaphor great for building small tools that I needed for web work.
In 2009 I began studying Art & Science at the University of Applied Arts where I got into Visualization tools like Maya and Rhino and experimented with their scripting capabilities (mainly using Python). I also started teaching Processing Basics for lower semesters of Art & Science and Graphics Design classes. Lately I got into physical computing using Arduino building some installations for school and extra-curricular projects.
In 2012 I participated in Google Summer of Code writing a prototype debugging mode for Processing, my first major FLOSS contribution. Later that year I began working for a small graphic design agency (http://100und1.com) where I am mainly coding websites in PHP and JavaScript. Lately we are starting to explore the possibilities of generative tools and methods for commercial graphic design work.
Concluding Remarks.
I am excited to participate in the Processing project again this summer and I'm looking forward to hearing from you. In case there are any questions I am happy to provide more details on this proposal.
Contact.