DevDays lessons: how to use a state machine to drive a QML UI

Posted on by

One of the more interesting presentations at DevDays 2011 was given by Marius from Cutehacks. After hearing this presentation I had a bit of an epiphany when it comes to state machines. I had been using them in Qt for quite a while, but I was never happy with how they meshed with the rest of the components. Indeed, there was a simpler and clearer way to do it, and that’s what I’m going to talk about in this article – the Cutehacks state machine design and how I used it in a real app.

The theory

The actual presentation slides are here and they have some sample code if you’re interested, but off the top of my head, here’s how Cutehacks does it:

  1. First of all, they noticed that implementing a state machine in pure QML was not the best idea. To tell you the truth, I was also surprised when first encountering QML states – sure they were very flexible, but the result wasn’t really a formal state machine. In QML you can switch from any state to any other state and even to ones that don’t exist, because the current state is just a name stored in a string property.
  2. One solution to the above problem is implementing the state machine in C++. Cutehacks inherited their state machine from the QStateMachine class and declared all the states as private members. That state machine had a signal corresponding to each state that shared that state’s name. e.g: state mFoo matched signal foo() inside the state machine.
  3. Each state had an isActive property that was exposed together with the state to QML. In order to make isActive work, they inherited from QState and reimplemented the onEntry and onExit events to toggle the property.

So far we have our own state machine with states and transitions neatly defined in C++ in the constructor, we have signals that can be used to transition between states and each state has an isActive property. How is this connected to QML? Each C++ state was matched with a QML state having the “when” property set to “cppState.isActive”.

It turns out that in QML you can call the signals of any C++ object that has been exposed to QML! So in the UI files when pressing a button you could say something like cppStateMachine.mainState() and then the mainState() signal of the cppStateMachine C++ object would be called. If the current state had a transition to mainState, then the transition would happen, mainState.isActive would become true and the corresponding QML state for mainState would be entered.

Finally, the entire point is that in the QML state you could then change some random QML properties on entry or even run some JavaScript code.

The practice

When coming back from DevDays, I attempted to use this state machine design in a C++/QML app that I was working on. Previously I was using only QML states, but it was reasonably easy to refactor the code to use a C++ state machine in the background. The implementation felt more robust, as I could just take a look in the C++ state machine implementation to see how the app behaved, instead of hunting for the elusive assignments to the “state” property. It was also easier to make changes, since the UI changes were well defined in one place now.

Soon after that, I ran into quite a big issue though… In the app I was working on, transitions were important – for instance when going from A -> B something was supposed to happen, but when going from C -> B, or even B -> A something different had to happen. My first thought was to use the Transition QML element. After adding a few transitions for testing, I noticed that the Transition animations and code were not being executed.

Immediately I suspected that “when” was the problem. If you’ll remember, state transitions were made in a specific way: each QML state had “when” set to the C++ state’s isActive property. In a C++ transition from A -> B, the QML transitions were A -> “”-> B instead of A -> B. The empty string is the default QML state. Since A.isActive became false, the state machine transitioned to the default state automatically. Then it transitioned to B when B.isActive became true.

As one can see QML state machines are quite peculiar…

The way I solved this was by integrating the C++ state machine more tightly with the QML state machine:

  1. I removed the isActive property for the C++ states and the “when” property for all QML states.
  2. For each qmlState I set its name to cppState.objectName, which is just the the QState’s QObject name. 
    State {
           name: cppStateMachine.initState.objectName [...]
  3. In MyState::onEntered I emitted a signal called enteredWithObjectName(QString) and passed the objectName() to it. 
    void MyState::onEntry(QEvent *event)
{
    [...]
    emit enteredWithObjectName(objectName());
  4. When setting up the state machine, I connected the enteredWithObjectName for each state to the C++ state machine’s stateChanged(QString) signal. 
    connect(mInit, SIGNAL(enteredWithObjectName(QString)), SIGNAL(stateChanged(QString)));
  5. In QML I wrote a signal handler called onStateChanged and in there I set the QML “state” property to the name parameter passed to the signal. 
    Connections {
        target: cppStateMachine
        onStateChanged: {
            console.debug("changing state to " + objectName);
            rootWindow.state = objectName;
        }
    }

The end result was that now transitions were done directly from on state to another without going through the default “” state. When a QML object called cppStateMachine.mainState(), a C++ state transition happened, the C++ state mainState was entered and it emitted its QObject.objectName. The state machine then passed that on to QML and the QML set the state property to that name, effectively triggering a QML state transition. Since each QML state had the same name as the C++ QState object, it worked.

This implementation allowed for a lot of flexibility with state changes. If some transitions were similar, they could be grouped in a QML Transition with from set to ‘*’(star/any state). If there was a special case, from could be set to specialState.objectName, and the QML transition would only execute for that state, while ‘*’ would work for the rest.

The advantage with C++ is that the state machine API is already implemented, but my assumption is that a similar state machine API can be written in JavaScript, so that you could have a pure QML/JS project if that’s what you want. The important ideas when building such an API are to respect the definition of a FSM in your design and to encapsulate operations such as state and transition set up and transition triggers.

The state of Qt on Symbian

Posted on by

I’ve meant to write this for a long time, because there’s not a lot of information about what it’s like to use Qt on Symbian. There is the Nokia marketing and there are some short developer testimonials, but the situation in the trenches can be different from what gets out in the news.

This blog post is structured in a Q&A format. The questions are those that I would have liked to have had an answer to when I started Qt Symbian development + some others that I have found interesting to think about.

Q: How hard is it for a Qt desktop developer to make the transition to mobile?

A: Qt is almost the same on Symbian as on other platforms. The Symbian-specific idioms (e.g the dreaded cleanup stack) are hidden from the programmer, but there are new UI concepts and new .pro statements and options. The Symbian build and deployment processes are notable differences that normally won’t give you trouble, but when they do give you trouble you will have to interact with the Symbian side.

Q: I have experience with Symbian C++, how does it compare to Qt?

A: Qt is much easier, you can avoid the cleanup stack, two-phase constructors, descriptors and active objects and the native UI classes. I have talked to people that did Symbian C++, Android and iOS dev. According to them, Symbian C++ is ~2x more time-consuming than the other two, Qt is about the same.

Q: What books / tutorials should I read?

A: Unfortunately, there is no one place where you will find all you need, the info is split between Forum Nokia, the (ex)Symbian Foundation, the Qt bug tracker, the Qt docs and books.

Let’s discuss them one by one:

* Forum Nokia hosts a Wiki and the Qt for Symbian section of the discussion boards. This is the best place to ask questions, because Qt Symbian devs and other experienced people read the boards. Some Wiki articles are part of the Knowledge base and contain info that’s not available elsewhere.

* The Symbian foundation websites had quite a few tidbits of info, but AFAIK those aren’t easily accessible any more because the Symbian Foundation has been merged into Nokia. You had the option of asking more technical Symbian questions on the forums there.

* It’s useful to browse the bug tracker to look out for Symbian bugs and their potential workarounds. Once again, some of this info is not available elsewhere (like a known bugs document).

* The Qt docs contain Symbian-specific sections that you must read. Some classes/functions contain platform notes.

* The “Porting to the Symbian platform” book is useful for a Qt on Symbian newbie since it explains differences between platforms, Symbian stuff such as the security model or PIPS and even contains example ports.

Q: How do I set up my development environment?

A: Just download the latest version of Nokia Qt SDK. There’s a naming problem here, because there’s also a “Qt for Symbian” download package, and a Qt SDK for [your OS]. You need the “Nokia” Qt SDK. Silly, I know, but it has gotten much simpler since the beginning where you had to watch a video to get a dev environment running.

Make sure you use the Qt Creator from the SDK, it has some special Symbian enhancements.

Q: What are the biggest problems that I should expect?

A: The look & feel is not completely adapted to Symbian, and in 4.6.3 there are UI bugs that you have to work around. The Symbian .pro options are not documented enough, and you might have to resort to trial and error.

I think that despite all the available comm channels, Nokia still has a communication problem when it comes to developers. I have gotten good results with the Qt bug tracker and by talking to Nokia devs. On the other hand Ovi store support is only done by e-mail for example and you don’t get a case number for support inquiries or a case history.

Q: How cross-platform is Symbian Qt?

A: Right now it’s probably the most cross-platform framework. I have done Symbian apps that run on Windows or Linux, and this was a major selling point for me. The UI must be carefully designed and will probably be different for each platform, but it’s still Qt and you still use QWidgets.

Q: What phones should I target?

A: Nokia has a lot of smart phone models on the market and most of them support Qt. The important difference is whether they use touch screens or not. You can and should target all the touch screen devices without much trouble. Check the phone stats report published in the Ovi newsletter for hints on popular phones.

Non-touch phones are a different story, and as far as I’ve seen there are few Qt apps targeting them. A potential opportunity, but my own experience indicates that Qt is significantly harder to use on those phones. The UI in particular is troublesome.

Q: Is Symbian Qt ready for commercial use?

A: Yes, but there are a few nuissances.

* You have to use a “Smart installer” when shipping on Ovi. This installer will download Qt if it’s not already available on the target devices (a 6-13MB download). The Smart installer isn’t very user friendly and your Ovi app description must contain a warning that the app might download 13MB of data. Needless to say, the customers can be confused by the message and they can even become upset if it’s a simple app.

* Ovi only allows Qt 4.6.3 which has some annoying bugs on Symbian. Use the bugtracker to see if anything you need is not working and check for workarounds.

Q: Can I ship software with Qt now?

A: You can ship either on Ovi store, on 3rd party stores or on your own website. Ovi supports only Qt 4.6.3 at the moment. If you want to publish outside of Ovi, it is very likely that you will need a Publisher ID and you will have to sign your app. The ID costs about $200 and signing costs $10.

Nokia will take care of signing if you publish on Ovi, but you will not get the signed setup package, it is published directly to the store.

Qt Creator, sln files and pro files

Posted on by

The sln and vcproj file formats can change between Visual Studio versions, so the only reliable way to do conversions is with Visual Studio and the Qt Add-In for Visual Studio.

Once the Add-In is installed, open your Qt solution with Visual Studio and select the “Create basic pro file” option from the “Qt” menu item. The Add-In will generate pro and pri files with all the relevant settings from the sln and vcproj files.

If you want to open pro files from Visual Studio instead, just select “Open Qt pro file” from the “Qt” menu and the sln and vcproj files will be created automatically.

Do expect to make minor adjustments to the generated files, as the conversion is not perfect.

Note: This doesn’t work with the VS Express edition, since that version doesn’t support plugins.

Kinetic scrolling with Qt – the what and the how

Posted on by

Last month I wrote a kinetic scroller component after struggling to find a ready-to-use and reasonably licensed one. Developers that wanted to create Qt apps on Symbian didn’t have much choice when it came to this type of scrolling. With the exception of Ariya’s kinetic model, all the available options that I had found either were not usable with Qt 4.6.3 on Symbian or were GPL.

In this article I’ve tried to explain how kinetic scrolling works and have also linked to a working implementation.

Introduction

Kinetic scrolling is the combination of regular, drag-finger-on-screen scrolling with an additional movement after the finger is lifted off the screen. Based on how fast the finger was dragged on the screen, the duration, speed and deceleration of the additional movement can vary. This video shows kinetic scrolling in action on a Nokia Symbian touch screen phone. A notable difference from using a scrollbar is that when tapping and dragging, the list moves in the same direction that the user is dragging in, which is the opposite of how a scrollbar works. For a scrollbar, when one drags down the list moves up, for kinetic scrolling when one drags down, the list also moves down – it feels more natural this way.

Advanced kinetic scrolling implementations also feature „bounce” and „overshoot”: in the first case the scrollable elements bounce off the scroll area’s edge once the scroll bar reaches the end, while in the second case one can scroll past the end of the scroll area and it will exhibit drag resistance and snap back into place when released.

Recent Symbian OS touch screen phones feature kinetic scrolling prominently in the user interface, but surprisingly Qt apps do not have it enabled by default. A fully featured implementation is still in the works for Qt version 4.8, with the possibility of a back-port to versions 4.7 and 4.6. In the mean time, a proof of concept implementation is available in the Qt labs under the name Flickable. The ideas in the latter example were used as a starting point for the simple kinetic scroller described in this article – QsKineticScroller. It can be used for vertical scrolling on any descendant of QAbstractScrollArea, including QScrollArea, QListView, QListWidget and QTreeView.

Algorithm

Note: the algorithm is designed for touch screen use with fingers, but Qt works with mouse cursors and mouse events. The algorithm is presented with Qt’s terminology, with a list playing the role of scroll area.

Click & drag scrolling

Since kinetic scrolling can be viewed as the sum of two features, it can be implemented in two steps.

The first step is click & drag scrolling. It can be achieved by installing an event filter and intercepting mouse press, move and release events. When a press event is received the scrolling starts, when a move event is received the list is scrolled, and finally when a release event is received the scrolling stops. To avoid accidental clicks, all the events are blocked inside the filter function.

Consuming the mouse events is a necessary step that leads to an unpleasant problem: regular clicks are no longer registered by the target list. This issue can be avoided by making the algorithm guess when the user is clicking and dragging as opposed to when they’re just clicking to select an item in the list. In the QsKineticScroller implementation a press & release event sequence is considered a click when it has less than five move events in between. Personal experiments have shown that a finger tap is much less precise than a pointer click, with one to four move events received between the time the finger is pressed on the screen and then lifted.

By the time the scroller has figured out that the user wanted to tap the screen, the events have already been consumed. To get around this last obstacle, the scroller records the screen position of the last press event and simulates a mouse click at that position.

Finally, by tracking mouse move events and updating the scrollbar position, the scroller makes the item list follow the user’s finger. Implementing this first part of the algorithm allows Symbian Qt application users to scroll much easier than with scrollbars. The second step makes scrolling more visually interesting and easier to do, especially on longer lists.

Kinetic scrolling

For step two, the scroller continues to scroll the list automatically after the user has lifted their finger off the screen, gradually slows down and then stops. To display a pleasing effect, the scroller must decide how fast to scroll, how far to scroll and how fast to slow down.

A good starting point is „how fast to scroll”.  In physics velocity represents the direction in which and magnitude by which an object changes its position. Speed is another word for magnitude in this context. The „how fast to scroll” question can be answered by recording the cursor’s drag velocity on the screen. A simple but imprecise way to do this is to poll the cursor position at specific time intervals; the difference in positions represents the speed (measured in pixels / timer interval) and the mathematical sign of the difference represents the direction. This algorithm will give a good enough idea on whether the cursors is moving fast or slow and it is popular enough, since it can be found in other implementations such as Sacha Barber’s Scrollable canvas.

Next up is „how far to scroll”. How far is actually connected to how fast to slow down because the list is scrolled with a certain velocity and then it decelerates until it stops. Since the velocity has previously been established, the only thing left is to calculate the deceleration based on friction. In physics, kinetic friction is the resistance encountered when one body is moved in contact with another. Of course, there can be no friction between pixels, but kinetic scrolling is a simulation and one can pretend that the list items are moving over the list container and that this movement generates friction.
In reality friction is calculated based on the nature of the materials, mass, gravitational force and so on. In the simulation a numeric value is used to alter the speed of scrolling. QsKineticScroller reduces the speed by a value of 1 at certain time intervals – a very simplified model indeed, but it works.

Having determined the deceleration, „how far” the list scrolls kinetically is simply a function of the time that it needs to reach a speed of zero.

Implementation

Note: comments have been removed and some of the code has been truncated and replaced with a [...] marker.

Step-by-step source code description

QsKineticScroller is implemented as a stand-alone class and most of the implementation is hidden behind a  d-pointer. The event filter function makes QObject inheritance necessary, but with a bit of work the event machinery can be moved inside the d-pointer to make the class implementation truly opaque.

class QsKineticScroller: public QObject
{
[...]
protected:
   bool eventFilter(QObject* object, QEvent* event);
[...]
private:
   QScopedPointer<QsKineticScrollerImpl> d;
};

Moving on to the cpp file, a few variables of interest can be seen at the top. The class user can experiment with these variables to influence the scrolling behavior. Indeed, the default values have also been chosen based on experimentation. As an example, changing the timer interval will affect the scrolling speed and smoothness, while changing the friction will influence the deceleration.

static const int gMaxIgnoredMouseMoves = 4;
static const int gTimerInterval = 30;
static const int gMaxDecelerationSpeed = 30;
static const int gFriction = 1;

The private implementation is aimed at hiding unneeded information from the compiler and component users.
The isMoving and isPressed variables are the simplest way to keep track of what state the scroller is in. e.g: not moving, scrolling by finger and so on. Some implementations assign an explicit state and one can go as far as using the Qt state machine implementation. The rest of the variables are described at the point of use.

class QsKineticScrollerImpl
{
[...]
   bool isPressed;
   bool isMoving;
   QPoint lastPressPoint;
   int lastMouseYPos;
   int lastScrollBarPosition;
   int velocity;
   int ignoredMouseMoves;
   int ignoredMouseActions;
   QTimer kineticTimer;
};

When installing the event filter, the important thing to notice is that it has to be installed for both the scroll area and its viewport. A scroll area is not a single item, and failing to install the filter on the viewport will result in not getting any mouse events at all.

void QsKineticScroller::enableKineticScrollFor(QAbstractScrollArea* scrollArea)
{
[...]
   scrollArea->installEventFilter(this);
   scrollArea->viewport()->installEventFilter(this);
   d->scrollArea = scrollArea;
}

The largest part of the implementation lies inside the event filter. It is described in multiple blocks.

The first job of the filter is to make sure that it only works on mouse events, since the scroll area will also receive paint events, resize events and so on. When a mouse press is registered, the press point is stored in case it is needed later for a simulated click and the scroll bar position is stored so that it can be used when calculating by how much to scroll the list.

bool QsKineticScroller::eventFilter(QObject* object, QEvent* event)
{
   const QEvent::Type eventType = event->type();
   const bool isMouseAction = QEvent::MouseButtonPress == eventType
      || QEvent::MouseButtonRelease == eventType;
   const bool isMouseEvent = isMouseAction || QEvent::MouseMove == eventType;
   if( !isMouseEvent || !d->scrollArea )
      return false;
   [...]
   switch( eventType )
   {
      case QEvent::MouseButtonPress:
      {
         d->isPressed = true;
         d->lastPressPoint = mouseEvent->pos();
         d->lastScrollBarPosition = d->scrollArea->verticalScrollBar()->value();
   [...]

This code block does the click versus click & drag differentiation. If it weren’t for it, the scroller would ignore legitimate clicks. Once the scroller has established that the user is indeed dragging on the screen, it starts a timer that calculates the drag speed. lastMouseYPos will be used later in the speed calculation.

case QEvent::MouseMove:
{
   if( !d->isMoving )
   {
      if( d->ignoredMouseMoves < gMaxIgnoredMouseMoves )
         ++d->ignoredMouseMoves;
      else
      {
         d->ignoredMouseMoves = 0;
         d->isMoving = true;
         d->lastMouseYPos = mouseEvent->pos().y();
         if( !d->kineticTimer.isActive() )
         d->kineticTimer.start(gTimerInterval);
      }
   }
   [...]

When the user lifts their finger off the screen a mouse release event shall be received. This is where the click versus drag differentiation makes a difference: d->isMoving will be false for clicks, but true for drags. The simulated click will be swallowed by the next filter call unless the filter is told to ignore it.

case QEvent::MouseButtonRelease:
{
[...]
   if( !d->isMoving )
   {
      QMouseEvent* mousePress = new QMouseEvent(QEvent::MouseButtonPress,
      d->lastPressPoint, Qt::LeftButton, Qt::LeftButton, Qt::NoModifier);
      QMouseEvent* mouseRelease = new QMouseEvent(QEvent::MouseButtonRelease,
      d->lastPressPoint, Qt::LeftButton, Qt::LeftButton, Qt::NoModifier);

      d->ignoredMouseActions = 2;
      QApplication::postEvent(object, mousePress);
      QApplication::postEvent(object, mouseRelease);
      [...]
}

As mentioned in the algorithm description, both speed calculation and deceleration are done at certain time intervals. This implementation uses a single timer for both.
The kinetic scrolling happens in the else branch: since the speed measurement is imprecise it is restricted to a more reasonable value at first. Then it is adjusted by the friction value until it reaches a minimum boundary, which means that kinetic scrolling should stop.

void QsKineticScroller::onKineticTimerElapsed()
{
   if( d->isPressed && d->isMoving )
   {
      const int cursorYPos = d->scrollArea->mapFromGlobal(QCursor::pos()).y();
      d->velocity= cursorYPos - d->lastMouseYPos;
      d->lastMouseYPos = cursorYPos;
   }
   else if( !d->isPressed && d->isMoving )
   {
      d->velocity = qBound(-gMaxDecelerationSpeed, d->velocity,    gMaxDecelerationSpeed);
      if( d->velocity> 0 )
         d->velocity -= gFriction;
      else if( d->velocity < 0 )
         d->velocity += gFriction;
      if( qAbs(d->velocity) < qAbs(gFriction) )
         d->stopMotion();

      const int scrollBarYPos = d->scrollArea->verticalScrollBar()->value();
      d->scrollArea->verticalScrollBar()->setValue(scrollBarYPos - d->velocity);
   }
   else
      d->stopMotion();
}

Where to get it, how to use it

The source code can be downloaded from my BitBucket repo, together with all the other Qt classes.

It’s licensed under a BSD license, so use it however you like to:

  • Unzip the downloaded file.
  • Add the unzipped cpp and h files to a Qt pro file.
  • Create a scroller object instance. It’s a good idea to make the scroller a child of the dialog or window that also contains the target scroll area.
  • Call the enableKineticScrollFor function with the target scroll area as a parameter.
  • If the scroll area is a list view or list widget, it must have the scroll mode set to ScrollPerPixel.

That’s it, the scroller will take care of everything else.

Virtual machines on telephones: Google didn’t have much choice

Posted on by

An interesting discussion was sparked on HN by a blog entry on apenwarr.ca. I’ve also been a little… surprised by Google’s choice of technology, but when you think about it, they didn’t have much choice.

What were the options? They needed a widely-used, performant, reasonably easy to understand programming language.

Web apps? No, Apple tried at first to use HTML + JS, but it didn’t work so they moved to Objective-C. Ruby, Python? Ruby is memory hungry, on the slow side and web-focused. Python is a little faster, which might work if one or two, or even six mobile Python apps were running at the same time. But what about 20 apps?

Objective-C might as well be owned by Apple. C is much too low level and other languages are either niche languages or too domain specific. The only realistic choices for a mobile platform were and still are C++, C# and Java.

Should Google have picked C++? They didn’t have a programming library: by the time Android 1.1 was released, Trolltech had already been bought by Nokia. It’s really hard to create a quality C++ library from scratch, and they probably didn’t care to deal with a third party like Nokia.

What about C#? Google (and pretty much any company) prefers to own the core technologies and decide their fates. If they would have picked C#, Microsoft would have had an important word to say when it came to Android’s future. That left them with Java. It’s popular. It’s also easy to learn, has good enough performance and a massive class library. The only problem were the licensing fees that they had to pay to Sun, but they got around them – or so they thought before Oracle – with Dalvik.

In the end, was Java a good choice? Before the Oracle lawsuit, it seemed so, but Google couldn’t know that Oracle would buy the friendly Sun. They will bitterly fight Oracle, since they have too much of an investment in Java and Dalvik now.

In the mean time, I wonder if anyone inside Google is looking at Java alternatives. There’s already a Qt port for Android, and some official Google love wouldn’t hurt at all.