Using the Android uiautomator to Record a Promo Video

Posted on 12 April 2014

I don’t usually do something manually when I can write a program to automate it, so I made the promo video for Simple Tasks & Notes using the Android uiautomator tool and ADB screenrecord. Below I’ll share some snippets of code and an Ant script that helped to make it painless.

Why?

It was reliable text entry that initially led me down this path, but that wasn’t the only benefit:

Downsides:

How?

The first thing to note is that a uiautomator project is not a conventional Android project. It’s not even an Android Unit Test project. You must build a plain JAR file (not an APK). This is all spelled out quite clearly in the developer docs, but an important bit is buried below the code examples.

We can do better than typing adb commands, though. The build.xml file that the android create uitest-project command generates is an Ant build script. We can add all our extra adb commands as tasks in build.xml and easily trigger them from Eclipse, or any other IDE.

My build.xml has four extra targets: copyjar, run, record, and fetchvideo. The record target is interesting, as it runs the adb screenrecord command in parallel with the run task. The use of <daemon> means that the recording task gets stopped when the run task finishes. Here it is:

<?xml version="1.0" encoding="UTF-8"?>
<project name="UiTests" default="run">

    <property file="local.properties" />
    <property file="ant.properties" />
    <property environment="env" />
    <condition property="sdk.dir" value="${env.ANDROID_HOME}">
        <isset property="env.ANDROID_HOME" />
    </condition>
    <loadproperties srcFile="project.properties" />
    <fail
        message="sdk.dir is missing. Make sure to generate local.properties using 'android update project' or to inject it through the ANDROID_HOME environment variable."
        unless="sdk.dir" />
    <import file="custom_rules.xml" optional="true" />
    <import file="${sdk.dir}/tools/ant/uibuild.xml" />

    <!-- ...End of standard setup -->

    <!-- Begin our customisations... -->

    <target name="copyjar" depends="build">
        <exec executable="${sdk.dir}/platform-tools/adb">
            <arg value="push" />
            <arg value="bin/UiTests.jar" />
            <arg value="/data/local/tmp/" />
        </exec>
    </target>

    <target name="run" depends="copyjar">
        <exec executable="${sdk.dir}/platform-tools/adb">
            <arg value="shell" />
            <arg value="uiautomator" />
            <arg value="runtest" />
            <arg value="UiTests.jar" />
            <arg value="-c" />
            <arg value="UiTests" />
        </exec>
    </target>

    <target name="record" depends="copyjar">
        <parallel>
            <daemons>
                <exec executable="${sdk.dir}/platform-tools/adb">
                    <arg value="shell" />
                    <arg value="screenrecord" />
                    <arg value="--verbose" />
                    <arg value="/sdcard/demo.mp4" />
                </exec>
            </daemons>
            <antcall target="run" />
        </parallel>
    </target>

    <target name="fetchvideo">
        <exec executable="${sdk.dir}/platform-tools/adb">
            <arg value="pull" />
            <arg value="/sdcard/demo.mp4" />
            <arg value="../submission/demo.mp4" />
        </exec>
    </target>
</project>

In Eclipse you can run a build by right-clicking build.xml in the Package Explorer and selecting “Run As / Ant Build…” . This lets you choose which target to build. I disabled “Build / Build before launch” on the run configuration dialog, as my custom targets already depend on the build step due to the depends=“build” in my run target.

Here’s my UiAutomatorTestCase. It includes code to drive the soft keyboard on my test phone. This generally won’t be directly transferrable to other phones and keyboards, but should be easily tweakable. (To determine the coordinates of the buttons, take a screenshot of the keyboard and read off the coordinates in an image editing program.)

import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.TimeZone;

import android.graphics.Point;
import android.util.Log;

import com.android.uiautomator.core.UiObject;
import com.android.uiautomator.core.UiObjectNotFoundException;
import com.android.uiautomator.core.UiScrollable;
import com.android.uiautomator.core.UiSelector;
import com.android.uiautomator.testrunner.UiAutomatorTestCase;

public class UiTests extends UiAutomatorTestCase {
    UiObject quickAddBox = new UiObject(new UiSelector().resourceId("com.rarepebble.tasklist:id/newTaskTitle"));
    UiObject addButton = new UiObject(new UiSelector().resourceId("com.rarepebble.tasklist:id/newTaskButton"));
    UiObject tagActionButton = new UiObject(
            new UiSelector().resourceId("com.rarepebble.tasklist:id/context_tag_selection"));
    UiObject appWorkTagCheck = new UiObject(new UiSelector().text("App Work"));
    UiObject doNextTagCheck = new UiObject(new UiSelector().text("Do next"));
    UiObject okButton = new UiObject(new UiSelector().text("OK"));
    UiObject menuButton = new UiObject(new UiSelector().description("More options"));
    UiObject manageTagsItem = new UiObject(new UiSelector().text("Manage tags"));
    UiObject actionCloseButton = new UiObject(new UiSelector().resourceId("android:id/action_mode_close_button"));
    UiObject actionHomeButton = new UiObject(new UiSelector().resourceId("android:id/home"));
    UiObject search = new UiObject(new UiSelector().resourceId("com.rarepebble.tasklist:id/search"));
    UiObject searchResultItem = new UiObject(new UiSelector().textStartsWith("Shelves for hall"));
    UiScrollable pager = new UiScrollable(new UiSelector().resourceId("com.rarepebble.tasklist:id/pager"));

    final String taskTitle = "Make app video";
    UiObject addedItem = new UiObject(new UiSelector().text(taskTitle));

    long startTime;
    final int captionTime = 3000;

    public UiTests() {
        pager.setAsHorizontalList();
    }

    public void testDemo() throws UiObjectNotFoundException {
        startTime = System.currentTimeMillis();

        launchApp();

        dumpCaption("Adding items is easy.");
        wait(captionTime);

        quickAddBox.click();
        typeOnSoftKeyboard(taskTitle);
        wait(100);
        addButton.click();
        getUiDevice().pressBack();

        dumpCaption("Drag items to reorder.");
        wait(captionTime); //
        dragListItemInFrontOf("Make app screenshots", taskTitle);
        wait(100);

        dumpCaption("Select items to tag them.");
        wait(captionTime);

        // Select item, tag with "App Work", select tag.
        addedItem.click();
        tagActionButton.click();
        appWorkTagCheck.click();
        okButton.click();

        dumpCaption("They now appear in the list for that tag.");
        // Scroll across to "App Work" .
        pager.flingToEnd(3);

        dumpCaption("You can change the order of the lists.");
        wait(captionTime);
        menuButton.click();
        manageTagsItem.clickAndWaitForNewWindow();
        dragListItemInFrontOf("DIY", "App Work");
        wait(500);
        actionHomeButton.clickAndWaitForNewWindow();
        pager.flingForward();
        wait(1000);

        dumpCaption("When lists get long, you can easily search them.");
        // Swipe to All Items.
        pager.flingToEnd(2);
        search.click();
        wait(500);
        typeOnSoftKeyboard("sh");
        wait(500);
        searchResultItem.click();

        tagActionButton.click();
        doNextTagCheck.click();
        okButton.click();
        actionHomeButton.click(); // dismiss search
        pager.flingToBeginning(3);

        sleep(captionTime * 2); //<Music credit>
    }

    private void dumpCaption(String text) {
        // .sbv format captions
        long showTime = System.currentTimeMillis() - startTime;
        long hideTime = showTime + captionTime;
        System.out.println(String.format(
                "%s,%s\n%s\n\n",
                formatTime(showTime),
                formatTime(hideTime),
                text));
    }

    private String formatTime(long timeMs) {
        Date date = new Date(timeMs);
        DateFormat formatter = new SimpleDateFormat("HH:mm:ss.SSS");
        formatter.setTimeZone(TimeZone.getTimeZone("UTC"));
        return formatter.format(date);
    }

    private void dragListItemInFrontOf(String beforeText, String itemText) throws UiObjectNotFoundException {
        int dragStartY = (new UiObject(new UiSelector().text(itemText))).getBounds().centerY();
        int dragEndY = (new UiObject(new UiSelector().text(beforeText))).getBounds().centerY();
        getUiDevice().drag(10, dragStartY, 10, dragEndY, 10);
    }

    private void wait(int msec) {
        try {
            Thread.sleep(msec);
        }
        catch (InterruptedException e) {
            Log.e("UiTests", e.toString());
        }
    }

    private void typeOnSoftKeyboard(String string) {
        getUiDevice().waitForIdle();
        for (char ch : string.toCharArray()) {
            if (Character.isUpperCase(ch)) {
                // Shift key position for CM11 stock keyboard on 480x800 screen:
                final int shiftX = 32;
                final int shiftY = 680;
                getUiDevice().click(shiftX, shiftY);
                ch = Character.toLowerCase(ch);
            }
            Point pos = keyPositionForCharacter(ch);
            getUiDevice().click(pos.x, pos.y);
        }

    }

    private Point keyPositionForCharacter(char ch) {
        // Alpha and space are sufficient:
        final String[] rows = {
                "qwertyuiop",
                "asdfghjkl",
                "zxcvbnm",
                " ."
        };
        // Key positions for CM11 stock keyboard on 480x800 screen: 
        final int rowSpacing = 75;
        final int colSpacing = 47;
        final int[] rowOffsets = { 0, 24, 70, 364 - colSpacing };
        // The middle of the Q key:
        final int qx = 24;
        final int qy = 528;

        int row = 0;
        int col = -1;
        for (; row < rows.length; ++row) {
            col = rows[row].indexOf(ch);
            if (col > -1) break;
        }
        if (col == -1) {
            return null;
        }
        else {
            Point pos = new Point();
            pos.x = qx + rowOffsets[row] + colSpacing * col;
            pos.y = qy + rowSpacing * row;
            return pos;
        }
    }

    private void launchApp() throws UiObjectNotFoundException {
        getUiDevice().pressHome();
        UiObject allAppsButton = new UiObject(new UiSelector().description("Apps"));
        allAppsButton.clickAndWaitForNewWindow();

        UiScrollable appViews = new UiScrollable(new UiSelector().scrollable(true));
        // Set the swiping mode to horizontal (the default is vertical)
        appViews.setAsHorizontalList();

        UiObject app = appViews.getChildByText(new UiSelector()
                .className(android.widget.TextView.class.getName()),
                "Tasks & Notes");
        app.clickAndWaitForNewWindow();
    }
}

The code outputs timestamped captions in .sbv format, which can be uploaded and added to a video on YouTube. Unfortunately the Google Play app on phones doesn’t display these captions, so in the end I added them manually. There are ways of automatically baking these captions into the video with (e.g.) ffmpeg, but I couldn’t stop them being rendered sideways on my portrait video.

I used Windows Movie Maker to trim excess footage from either end of the video, add captions, music, and fade to black at the end. All easy tasks for a novice video editor like me. You can see the final video on the Simple Tasks & Notes page on Google Play.