My San Jose Mobile App Usability Testing

The Challenge

Use real users to test-drive a clickable wireframe for My San Jose, a mobile app soon to be released by the City of San Jose that allows residents to report issues throughout the city ranging from potholes to graffiti.

The Objective

Identify strengths and weaknesses of the My San Jose app and offer recommendations on how to improve the app’s user experience.

My Role

Lead on usability test procedure design and script. Assisted with test moderation, documentation, and synthesis.
TEAM MEMBERS: Marina Broido, UX Lead; Julie Kim, UX Assistance

WHAT IS A CITIZEN USER TESTING GROUP?

My San Jose’s usability test served as the pilot session for Code for San Jose’s newly established Civic User Testing Group (CUT Group). The City of Chicago pioneered the CUT Group concept in 2013 as a form of “grassroots user testing” in which residents are paid to test-drive government and non-profit technology. The movement has since spread to cities throughout the United States.

“It is rare for government technology projects to include the budget, schedule or subject matter expertise to test products and services with real citizen users before going live. Unfortunately, people’s experience with government digital services often reflect the lack of a user-centered approach to development.” - Michelle Thong, co-founder of Code for San Jose

Both citizens and developers stand to gain with CUT Groups - citizens earn a small incentive, improve their digital literacy and feel more engaged with their communities; developers improve the usability of their products and services. Read more about CUT Groups here.

Limited budgets often result in creative solutions

My San Jose Usability Study Partners:

Methodology

Our team arranged two separate testing sessions for a clickable wireframe of the My San Jose mobile app. The first session tested the app internally with City employees and the second session recruited members of the public through a Craigslist ad. In exchange, they were offered an Amazon gift certificate. Each testing session comprised of an orientation, a pre-test interview, a task performance using an Android or iOS mobile device, and a post-test interview.

Our second testing session with the public took place at a local branch library
We tested five users total while moderating, observing and documenting them performing the following five fundamental tasks:

  • TASK 1: Report an abandoned vehicle.
  • TASK 2: Check the status of your past requests.
  • TASK 3: Read the comments added by others about your past requests.
  • TASK 4: View a map of all requests submitted by other users.
  • TASK 5: Locate the number to call for assistance.

Findings from Usability Test

Strengths:

  • Most testers were able to accomplish all tasks.
  • Testers found the app useful and expressed interest in using it.
  • Testers liked the following features: viewing reports from other residents, tracking statuses of reports, using app as a source of important information about their neighborhoods.

Areas for Improvement:

The table below details common issues that our five testers encountered while attempting to accomplish each task. The first column describes the issue, the second column provides actual quotes said by testers and the final column represents the number of testers that encountered each issue.
COMMON ISSUES USER QUOTES #
Confusion about home page menu features “I really wouldn’t even know where to start.” 4/5
Long and incomplete form fields; not all car colors featured “My [car] color’s not there.” “I don’t know which fields are required.” 4/5
Inability to find status of request “No, there is no indication of statuses.” 4/5
Lack of back button forced tester to click elsewhere and leave the app “That’s how I messed up. I don’t know how to go back. See I did it again.” 4/5
Confusion about the “My Home Services” feature “My Home Services does not tell us what it is. And I still don’t know what it is.” 3/5
Unsure whether to register an account in order to submit a request “Don’t I need to register an account in order to use this app?” 2/5
Inability to understand certain language and icons used throughout the app “Vehicle blocking? What does that mean.” One tester did not comprehend the map icon. 2/5
4 out of 5 testers struggled to find status of requests. “No, there is no indication of statuses.”
4 out of 5 testers struggled with long and incomplete forms. “My [car] color’s not there.”

LANDING PAGE

This is the first screen that users encounter upon opening the app. Many testers were unsure whether to register an account in order to submit a request.

“Don’t I need to register an account in order to use this app?”

HOME PAGE

4 out of 5 testers were confused about the home page menu features. They had difficulty distinguishing between “New Request”, “My Requests” and “All Request” buttons.

“I really wouldn’t even know where to start.”

The most critical task was a challenge for most testers

The most critical task - being able to easily report an issue - was a challenge for four out of the five testers. In order to report an abandoned vehicle from the home page, tester 1 took ten minutes, tester 2 took nine minutes, while tester 3 and 4 both took about four minutes. The following diagram details the actual path taken by tester one in order to report an abandoned vehicle.

MOST CRITICAL ISSUES:

  • The fundamental task of being able to easily report an issue was a challenge for four out of the five testers.
  • Many testers were unclear whether they needed to register in order to submit a report.
  • Forms required a long time to fill out, were incomplete and included cryptic language.
  • Absence of a back button lead to inefficient navigation.

RECOMMENDATIONS:

  • Revise the home page with a clearer “call to action”.
  • Feature back button in a consistent location on all pages.
  • Revisit onboarding experience for first-time users:
    • Change landing page for first-time users; potentially eliminate login page.
    • Describe the app’s purpose on the landing page.
  • Provide a confirmation when user submits a report.
  • Simplify new request form; possibly rename to the more colloquial “new report".
  • Revisit “My Home Services” content and inclusion.

Outcome

Our team presented findings to the owners, innovation leads with the City of San Jose, along with other city staff in March 2017. The owners were so excited to observe real people interacting with their app that they watched all the usability test footage prior to the presentation. As a result, they immediately decided to modify the UI of the "add photo" feature since many testers missed it.

The presentation was well received and the owners are taking our recommendations into consideration for the next phase of development. My San Jose app is scheduled for an alpha launch in May 2017.