Rwandan User Experience Testing
Goal: make SafeMotos app easier to use for both tech savvy and untech savvy Rwandans
Summary of Recommendations
- Test 1: App Comprehension
- All buttons should be visually large and obvious to be clicked.
- Every button should include its function in written or visual form
- A minimum number of options should be displayed to reduce confusion
- No icons should be used unless tested with a large number of target users for recognition (this is especially true of the current icons we use for customer service, GPS and scan)
- Have app visualization and flow mimic WhatsApp or Facebook as users have high knowledge of how these apps work
- Test 2: Less is More
- On screens where we communicate location of drivers we should not communicate exact locations
- For home screen there should be a simple message “There are 10 drivers within 2 kilometers of your location.” The 10 and 2 should be visually made to look dynamic.
- For driver reaching customers see Test 6
- Test 3: Customisation
- Test 4: Map Understanding
- Maps need to either be removed or buried in the SafeMotos app experience.
- A simple button (eg Moto Driver Pick Me Here), choosing landmarks, or choosing past verified locations would be a more effective way of ordering a driver
- Test 5: Map vs Task
- Have very forceful, ideally integrated, ability to turn accurate GPS on. After pairing / end of trip have option to turn accurate GPS off to avoid fear of GPS high battery usage.
- Prod users to do some tasks to help get an accurate GPS. Eg: Turn on Accurate GPS, put hand out window, walk outside, login to a WiFi.
- Suggestion: have accurate location determined after being paired with a close driver. Testers showed a willingness to to use idle time waiting for a driver productively.
- Test 6: Communicate Trust
- Driver trip to customer should either be a countdown, a progress bar or a WhatsApp style form of verification, or a mixture of the three. There should be no map shown. A potential idea brainstormed was also a countdown (where maybe the trip would have a promotion applied to it if it exceeded the time). There needs to be a language that increases trust
- Test 7: Ease of Registration
- Make the solution the exact same as WhatsApp: Auto verify via SIM
- Give a rationale for why we are asking for details (even if expands size of form). Eg: Can we have your number in case the driver is lost and needs to call you? Can we have your email so we can send you details from each trip? We promise to not share the data!
- Additional information could be asked later (eg during wait for driver pick up). The first registration screen should ask for the minimum amount of information to avoid fraud.
Methodology
Design Sprint: (from Wikipedia) A Design sprint is a time-constrained, five-phase process that uses design thinking to reduce the risk when bringing a new product, service or a feature to the market.
With more than 500 new apps entering the market every day, what does it take to build a successful digital product?[1]
This process helps the team in clearly defining goals, validating assumptions and deciding on a product roadmap before one line of code is written. Main pillars of this process: Business strategy, interdisciplinary collaboration, rapid prototyping, and user testing. This design process is similar to Sprints in an agile development cycle[2] that incorporates the same principles of learning early. Design sprints typically last one week.
Tester Profile
Age 30+, Rwandan, has a smartphone but uses it largely for WhatsApp and Facebook
Test 1: App Comprehension
Context
Users are not tech savvy with a very limited exposure to other applications. They have a high fear of looking foolish which makes them sensitive to new experiences.
Hypothesis
Our hypothesis is that users do not respond to industry best practices and need a simpler approach
What we tested was a print out of three different apps (Uber home screen, a simple game home menu, Hailo app driver side) then asked each tester to point out every clickable section they saw on the app and explain what they believed it would do.
Results
Not a single tester was able to identify and give explanatory information on a single button that didn’t have a ‘word’ relating to its functionality.
The clear winner was the game: with four big buttons saying obvious functionality (play, share, high score, etc) users were able to easily identify and guess function of the buttons. However, there was a sub menu with icons that many testers did not see or if they did were unable to guess the functionality (question mark being identified as talking to a live rep instead of a FAQ).
On Uber app, customers were able to identify that the main ‘order’ button was the most important to press, but gave incorrect functionality or failed to identify other buttons. The Hailo app is text heavy in its functionality which allowed drivers identify buttons and their functionality reasonably well but many on screen buttons were identified wrong or not identified if there was not an explicit word on it.
Many of them assumed at a high level that their answers were the only logical answer when in fact it was wrong.
Tentative conclusion
Users respond best to a low number of options that are clearly visible as buttons and have their function communicated is a simple a way as possible, ideally in text.
Following best UX best practices is not fully logical in Rwanda since the metaphors they rely on is not recognized.
Users were extremely unaware of their comparative ignorance to button functionality.
Tentative suggestions to be implemented
- All buttons should be visually large and obvious to be clicked.
- Every button should include its function in written or visual form
- A minimum number of options should be displayed to reduce confusion
- No icons should be used unless tested with a large number of target users for recognition (this is especially true of the current icons we use for customer service, GPS and scan)
- Have app visualization and flow mimic WhatsApp or Facebook as users have high knowledge of how these apps work
Test 2: Less is More
Context
Users often are frustated with SafeMotos when information is communicated to them that they commit to then the reality doesn’t live up to their expectations. One concrete examples would be a customer following a driver on a map and determining they are not taking an effective route. Another would be a customer seeing drivers on a map that are close to them yet they are paired with a farther driver. Customers respond to issues like this emotionally, getting to the point of declaring SafeMotos as liars at an extremely fast speed.
Hypothesis
It is our hypotheses that people will react better to less information rather than more information.
Our test was to ask testers to make a series of “choose one or the other” for a fictional health clinic. The only difference was how the number of clinics was represented: as a number, as placement on a map (a la Uber) and as a qualitative word (many)
Results
Users would prefer the higher number of clinics to the lower and would choose a number qualifier (10 clinics) to a word qualifier (many clinics). Map style showing clinics (like Uber) was rated the most highly, but had a serious number of questions attached to it (there is no clinic here! Where am I on the map? How am I travelling. Are all the clinics the same)
Tentative conclusion
The serious amount of time and confusion from that participants would state that even thought they said they preferred the maps, would suggest that this is not the right way to go. People responded extremely well to having a number: they liked the dynamic nature (it built trust) and then they moved on to the next task.
Tentative suggestions to be implemented
- On screens where we communicate location of drivers we should not communicate exact locations
- For home screen there should be a simple message “There are 10 drivers within 2 kilometers of your location.” The 10 and 2 should be visually made to look dynamic.
- For driver reaching customers see Test 6
Test 3: Customisation
Context
We want our app to feel special to users, something they take pride in and want to show off. With many tech services (WhatsApp, Facebook, Android, Windows) users modify them heavily in a way that is different than in the west. For example, customized launchers are highly adopted and it is normal to change the background screen image on a phone weekly, if not daily.
Hypothesis
Participants would respond better to a customised version of an app instead of a stock version.
What we did was show participants three different version of the WhatsApp homescreen: a stock version, a lightly customised version and a a heavily customised version. Then asked them to rank them by how they would want their own phone to look like and asked them for their justification.
Results
We were surprised that testers did not even see the visual customisations that were the focus of the tests. Where people did respond to customisation it was to make the experience more simple (eg. text larger)
Tentative conclusion.
We believe we made a mistake and were making the hypothesis from our own friendship circles which is an early adopter crowd and likes to show off their tech savvy. High level of user customisation is not something being asked for or something that positively changes experience.
Tentative suggestions to be implemented
n/a
Test 4: Map Understanding
Context
Every user experience testing we have conducted has shown the step where a user has to identify their location on a map as a critical failure point.
Hypothesis
Our hypothesis is that users don’t know how to read maps.
What we did was show a user a map with location services turned off with the map centered on one section of town. We then asked the user to show us on the map where the SafeMotos office (where they currently were) on the map.
Results
Two users had the smart idea of just entering the text in the search box “SafeMotos” and were brought immediately to our location. The other three users were unable to identify SafeMotos, or the area SafeMotos was in on the map, with one user giving up midway over another country.
Tentative conclusion.
Users are able to type addresses, however, this is not very helpful when A) many locations have no formal name (houses / streets not having numbers) and B) the culture of acceptance in multiple spelling choices.
No user was able to understand the map, the location of a major landmark on a map, or how to move to another location that they know.
Tentative suggestions to be implemented
- Maps need to either be removed or buried in the SafeMotos app experience.
- A simple button (eg Moto Driver Pick Me Here), choosing landmarks, or choosing past verified locations would be a more effective way of ordering a driver
Test 5: Map vs Task
Context
Related to the above question, we have known users have major challenges with maps since the earliest day of SafeMotos. The suggested conclusion for the previous test has long been assumed as a potential solution. However, the solution of a button ‘click me here’ is heavily reliant on a phone having an accurate current location, which is challenging in an environment with low quality GPS chips in phones, users propensity to select buttons rapidly without understanding their function and the general challenges of GPS / location technology.
Hypothesis
Our hypothesis is that the frustrations of using maps would be bigger than the frustrations of making users get a more accurate GPS lock. GPS is better if you have location services turned on, wait some time for an accurate lock and are outside rather than inside. To do this test we did proxy tests where we would ask a user to either locate a location on a map (A landmark, their primary school, the Eiffel Tower, etc) or do some arbitrary task (count to ten, complete 5 + 3 - 2, walk up and down the stairs. We were hoping to see people choose to do mildly painful tasks because they were less painful than choosing in a map.
Results
The results were incredibly interesting. Users would choose the options where it demanded mental energy (5 + 3 - 2) but not physical energy (go down the stairs). Capacity of map use was close to zero, with one user choosing to type in locations but all others were unable to find a single location on the map. What was most interesting though was the frustration, sadness and self disappointment users had while engaging with the map. Many users would just wander around the map, visually searching for meaningful text, continuing for upwards of 5 minutes when the facilitator would halt that individual map search. Watching via video feed I (Nash) wanted to jump in and stop the testing, it was as if we were torturing them. One man, who was passionate and thoughtful, looked out the window for minutes at the end of the entire interview period then asked the facilitator to show him how maps worked. There was a sense of their ignorance being captured, embarrassment at their mistake, a feeling that they had failed.
Tentative conclusion
Maps give Rwandans a terrible user experience. It plays against some of the strongest psychological constraints in the region (fear of failure, wanting to impress, over confidence, wanting to avoid embarassment); maps need to be avoided in the experience at all costs.
Users are willing to do semi painful tasks (sitting tasks) but not painful tasks (movement) rather than select a location on a map.
Tentative suggestions to be implemented
- Have very forceful, ideally integrated, ability to turn accurate GPS on. After pairing / end of trip have option to turn accurate GPS off to avoid fear of GPS high battery usage.
- Prod users to do some tasks to help get an accurate GPS. Eg: Turn on Accurate GPS, put hand out window, walk outside, login to a WiFi.
- Suggestion: have accurate location determined after being paired with a close driver. Testers showed a willingness to to use idle time waiting for a driver productively.
Test 6: Communicate Trust
Context
Many SafeMotos users watch the entire experience of the driver coming to them, which makes them angry at every road selection the driver uses, confused by GPS jumps and in a state of anxious engagement as they watch the driver reach them.
Hypothesis
Our hypothesis is that a form of communicating how long until pickup that gives less information and a clearer sign of progress would resonate better with customers.
What we did was ask a tester a series of questions on how they would communicate to a 12 year old child what time they would be home. Then, we shared with them 4 visual examples (a clock countdown, a WhatsApp 2 blue checkmark of a message sent, a progress bar and an Uber style map of a driver coming to a location) then asked testers to say which one they think would be best for the 12yo
Results
In every single case the Uber style map was deemed to be the least effective way to communicate time to arrival. The other examples were scattered in terms of effectiveness.
Tentative conclusion.
Users prefer a non-map real time UI communicating time to pick up by the driver. They would prefer something clear, but that they don’t have to engage with and tells a clear story.
Tentative suggestions to be implemented
- Driver trip to customer should either be a countdown, a progress bar or a WhatsApp style form of verification, or a mixture of the three. There should be no map shown. A potential idea brainstormed was also a countdown (where maybe the trip would have a promotion applied to it if it exceeded the time). There needs to be a language that increases trust
Test 7: Ease of Registration
Context
Registration is an extremely high drop off in our funnel. We have noticed that the culture of being casual with with spelling mistakes means there is a huge amount of irritations with registration from a customer perspective and that the boredom of data entry and asking for two step verification is a major barrier for customers.
Hypothesis
Our hypothesis was that from industry best practice there would be a clear favourite for registration.
We had users register new accounts from a form based service (WhatsApp), auto 2 step verification (WhatsApp) and the option to login with Facebook / Gmail (Trello).
Results
As expected, the form based input (Gmail) was extremely negative from testers. They found it boring, difficult, and asked questions at every stage about why the information was needed.
Logging in by Google/Facebook was very smooth with good reception, but no one actually noticed the ‘login with Google/Facebook’ button, instead they responded to there being only three fields to complete on the form. When asked after about the Google/Facebook button the testers did not realize it was an option. WhatsApp was the obvious favourite, with users entering it so fast that the team watching via the video feed missed it.
Tentative conclusion.
We had been betting on ‘login with Facebook’ to answer registration challenges but it doesn’t look like customers really get it.
Users are fine to fill in forms if short
Users are very cautious of the data they give out, a reminder that for many users this may be the first time they have been asked these questions (WhatsApp / Facebook are often installed by professionals)
An interesting point was that users were very free to give their number, less than any other piece of information (including first name)
Users responded well to what was familiar. WhatsApp was the clear winner
Tentative suggestions to be implemented
- Make the solution the exact same as WhatsApp: Auto verify via SIM
- Give a rationale for why we are asking for details (even if expands size of form). Eg: Can we have your number in case the driver is lost and needs to call you? Can we have your email so we can send you details from each trip? We promise to not share the data!
- Additional information could be asked later (eg during wait for driver pick up). The first registration screen should ask for the minimum amount of information to avoid fraud.