Example: Qt Drumkit

Qt Drumkit is a virtual drumkit which lets you play percussion sounds by tapping the screen. You can record your beats and play them back later. It is also possible to play on top of your latest recording.

The Qt Drumkit application contains two views for playing:

• simple view with 2D pads
• 3D drumkit view

Drumkit has been also implemented for ​Series 40, ​Windows Phone 7. If you are interested about the differences and similarities of these implementations, take a look at the comparison article.

Prerequisites

You need the following to develop and test this example:

• ​Nokia Qt SDK 1.1.4 for development
• A Symbian^3 device (or newer) or a MeeGo 1.2 Harmattan device (N9 or N950)^

© Nokia 2011.

Design

The application is a Qt version of the ​JME Drumkit application. As the smartphones with Qt are generally more powerful and have a better screen resolution, more drum pads, effects, and animations can be used.

A QML-based user interface offers a pleasant user experience. The sound engine and playback logic have been implemented with C++, which is accessed from the QML UI.

When designing a touch application, it is important to consider how it reacts to touch events. As the user touches a drum pad, the corresponding sample should be played without delay. Low latency is crucial in a music application like this, as high latency leads to a poor playing experience. Multipoint touch support is also necessary, so that hitting more than one pad simultaneously is possible.

Implementation

QML user interface

In QML UI, there are platform-dependent QML files for Symbian and MeeGo, and a set of common QML files that are used on both platforms. The QtDrumkit.pro file selects which ones to deploy.

Platform-specific QML components, located inqml/symbian and qml/harmattan directories:

• main.qml: Sets up the main display.
• Pads2d.qml: 2D pad view, positions Pad components on the screen.
• Pads3d.qml: 3D drumkit view, positions Cymbal3d and Pad3d components on the screen.

Common QML files, located in theqml/common directory :

• AnimButton.qml: A button component that is able to display an animation using a PNG file. Used for the record button.
• Button.qml: A simple button component that can be toggled. Used for all the other buttons.
• Info.qml: A component displaying the information text.

• Pad.qml: Instrument pad component. Plays a specific sample when tapped, and opens the instrument selector on a long tap.
• Pad3d.qml: Instrument pad component for the 3D view. In addition, contains a bouncing animation.
• Cymbal3d.qml: Cymbal pad component for the 3D view. In addition, contains a rotation animation.

• InstrumentSelector.qml: Instrument selector component. Arranges InstrumentButton components in a circle and performs transition animations. Used by the Pad component for configuring the sample for the pad.
• InstrumentButton.qml: A component for selecting instruments within the InstrumentSelector. Displays a selection highlight and plays the corresponding sample.

The InstrumentSelector component activated:

Engine

In main.cpp, the DrumEngine class is exported to QML. It provides methods for playing samples, recording and playback of a drum track.

Registration in main.cpp:

    // DrumEngine QML bindings
    qmlRegisterType<DrumEngine>("DrumEngine", 1,0, "DrumEngine");

Accessing engine from the QML, in main.qml:

    // DrumEngine instance for sample playing and drum track recording and playback.
    DrumEngine {
        id: engine
    }

Sample playback in Pad.qml is quite straightforward. The play() function is called when the pad receives a touch event:

Item {
    // Sample to be played by this pad.
    property string  sample

    function play() {
        engine.playSample(sample)
    }

The following is a part of the DrumEngine class declaration which shows the QML bindings. These can be used to implement the application features in the QML UI:

class DrumEngine : public QObject
{
    Q_OBJECT
public:
    explicit DrumEngine(QObject *parent = 0);
    virtual ~DrumEngine();

    Q_PROPERTY(bool canPlay READ canPlay NOTIFY canPlayChanged);
    Q_PROPERTY(bool canRecord READ canRecord NOTIFY canRecordChanged);
    Q_PROPERTY(bool isPlaying READ isPlaying NOTIFY isPlayingChanged);
    Q_PROPERTY(bool isRecording READ isRecording NOTIFY isRecordingChanged);

signals:
    void isPlayingChanged();
    void isRecordingChanged();
    void canPlayChanged();
    void canRecordChanged();

public slots:
    // Play a sample with a name.
    void playSample(QString name);

    // Start playback.
    void play();

    // Start recording.
    void record();

    // Stop playback or recording.
    void stop();
    .
    .
    .

Drum track recording and playback

When the user starts recording a drum track, the engine stores the start time and all subsequent pad hits using a time stamp and a sample name for each hit. This information is enough to reconstruct and playback the drum track after the user has stopped recording. Playback is done using a QTimer event in DrumEngine::playbackTimerEvent() function.

Multipoint touch support

The MouseArea element in QML does not provide multipoint touch, so C++ is needed. This is achieved by subclassing the QmlApplicationViewer class and implementing the event() function.

Enable touch events in QmlViewer class implementation:

setAttribute(Qt::WA_AcceptTouchEvents);

Touch event handling in QmlViewer class. A signal is emitted for each pressed event.

bool QmlViewer::event(QEvent *event) {
    if (event->type() == QEvent::TouchBegin || event->type() == QEvent::TouchUpdate) {
        QTouchEvent *te = static_cast<QTouchEvent*>(event);
        foreach (QTouchEvent::TouchPoint tp, te->touchPoints()) {
            if (tp.state() == Qt::TouchPointPressed) {
                emit touchEventReceived(tp.screenPos().x(), tp.screenPos().y());
            }
        }
        return true;
    }
    return QmlApplicationViewer::event(event);
}

A component that can be instantiated from QML and relays the touch signal to QML is needed. The TouchEvents class connects to the QmlViewer touch event signal and passes it on. On the QML side, the QML element corresponding to the touch event coordinate is checked. If it has a play() function, the function is called.

    TouchEvents {
        onTouchEventReceived: {
            if (info.show || selector.show) {
                // Ignore events when info view or instrument selector visible.
                return
            }
            // See which pad was hit, if any. Check which view is visible:
            // The 3d view contains pads in item 'pads', whereas the 2d pad item have
            // them as top level children.
            var item = !flipable.flipped ? flipable.front.childAt(x,y) : flipable.back.pads.childAt(x,y)
            // See if there is a function called play().
            if (item && item.play) {
               item.play()
            }
        }
    }

Sample playback

The application uses Qt GameEnabler for handling the WAV sample files and mixing them together for creating an audio stream. The audio stream is then played with native methods offered by the platform. These native methods are:

• ​PulseAudio: This is a native audio library available on linux and MeeGo/Harmattan, providing low latency audio.
  • Source file: audiopulseaudio.cpp
• ​DevSound: This is the native Symbian audio library. With some adjustments, DevSound produces an acceptably low latency output.
  • Source file: audiodevsound.cpp
• ​Qt GameEnabler: This method is not currently used in the application. However, it can be enabled in the pro-file. QtGameEnabler uses the QAudioOutput class for audio output. This works on both Symbian and MeeGo, but the two native methods above produce a better result.
  • Source file: audiogameenabler.cpp

Sample playback is encapsulated in the SamplePlayer class. An audio back end is selected by compiler flags during compilation: either PulseAudio or DevSound. In the class constructor, all sample files are preloaded to memory for fast access. As a requirement of QtGameEnabler, the sample format is:

• WAV
• stereo
• 6-bit signed
• 22050 Hz sampling rate

Initialization of the samples in SamplePlayer, and creation of the audio back end:

    QStringList samples;
    samples << "cowbell" << "crash" << "hihat1" << "hihat2" 
            << "kick" << "ride1" << "ride2" << "snare" 
            << "splash" << "tom1" << "tom2" << "tom3" << "china";

    foreach (QString name, samples) {
        m_samples[name] = GE::AudioBuffer::loadWav(":/samples/"+name+".wav", this);
    }

#ifdef USE_GAMEENABLER
    m_audioIf = new AudioGameEnabler(m_audioMixer, this);
#endif
#ifdef USE_DEVSOUND
    m_audioIf = new AudioDevSound(m_audioMixer, this);
#endif
#ifdef USE_PULSEAUDIO
    m_audioIf = new AudioPulseAudio(m_audioMixer, this);
#endif

After this, an audio back end is started. It constantly pulls data from the audio source, which in this application is GE::AudioMixer. When one or more samples are played, the mixer mixes them together and the back end receives the resulting data for playback. If no samples are being played, the mixer provides silent data.

The PulseAudio implementation requires a bit of C code, which is based on ​this example. The PulseAudio issues stream_request_cb callbacks when new data is required. The callback implementation pulls data from the GE::AudioMixer instance and passes it on. Latency can be configured in pa_buffer_attr structure.

PulseAudio callback implementation, with the userdata parameter as GE::!!AudioMixer:

static void stream_request_cb(pa_stream *s, size_t length, void *userdata) {

    AudioSource *source = reinterpret_cast<AudioSource*>(userdata);
    if (bufferSize < length*sizeof(short)) {
        bufferSize = length*sizeof(short)*2;
        buffer = (short*) malloc(bufferSize*sizeof(short));
    }
    memset(buffer, 0, length*sizeof(short));
    source->pullAudio(buffer, length/2);

    // Write all the requested bytes. If sample has ended, the
    // rest are empty bytes.
    pa_stream_write(s, buffer, length, NULL, 0LL, PA_SEEK_RELATIVE);
}

The Symbian DevSound implementation is similar: BufferToBeFilled() callback is called when new data is needed. DevSound default buffer size is 4096 bytes, which results in a significant latency in the audio. Latency can be configured by providing less data in the callback. The relevant parts of DevSoundaudio output from audiodevsound.cpp are as follows:

AudioDevSound::AudioDevSound(GE::AudioMixer &audioMixer, QObject *parent)
    : QObject(parent),
      m_audioMixer(audioMixer)
{
    m_devSound = CMMFDevSound::NewL();
    m_devSound->InitializeL(*this, KMMFFourCCCodePCM16, EMMFStatePlaying);
}

AudioDevSound::~AudioDevSound()
{
    m_devSound->Stop();
    delete m_devSound;
}

void AudioDevSound::InitializeComplete(TInt aError)
{
    qDebug() << "InitializeComplete " << aError;

    // Configure sample format.

    TMMFCapabilities caps = m_devSound->Config();
    caps.iChannels = EMMFStereo;
    caps.iEncoding = EMMFSoundEncoding16BitPCM;
    caps.iRate = EMMFSampleRate22050Hz;

    // Possible errors are ignored below:

    TRAPD(err, m_devSound->SetConfigL(caps));
    qDebug() << "SetConfigL()" << err;

    TRAP(err, m_devSound->PlayInitL());
    qDebug() << "PlayInitL()" << err;
}

void AudioDevSound::BufferToBeFilled(CMMFBuffer *aBuffer) 
{
    CMMFDataBuffer *buf = static_cast<CMMFDataBuffer*>(aBuffer);
    TDes8 &output = buf->Data();

    // The default buffer size is 4096.
    // To improve latency, only part of the requested bytes are passed to DevSound.
    const TInt reqSize = 3*256;

    output.SetLength(reqSize);
    short *ptr = (short*)output.Ptr();
    Mem::FillZ(ptr, reqSize);
    m_audioMixer.pullAudio(ptr, reqSize/2);
    m_devSound->PlayData();
}

Conclusion

A QML based UI is a good choice for these kind of applications. The performance is good and animations work smoothly.

The challenging aspect in the application is the audio latency issue. Implementing the engine using C++ enables testing and experimentation with different audio output methods and parameters, and selecting the optimal ones for the target device. PulseAudio on MeeGo and DevSound on Symbian both provide good results. With low latency audio and multipoint touch support, the application guarantees a fun multimedia experience on the current Symbian and MeeGo devices.