Phone Sensor Unlock: Rock Paper Scissors Recognition

Project Description

In this assignment, we were asked to design a sensor-based interface for a mobile phone's unlock mechanism that utilized one of the phone's sensors. Important details that we needed to consider were making sure that the phone was secure, and that there was no chance of accidental activation. We also needed to consider how the system provides feedback to the user while they're in the process of unlocking the phone, when the unlock fails, and when the unlock succeeds. As such, my team made an unlock mechanism that fulfilled all of these requirements in a creative and fun way.

Concept Sketches

For our initial sketches, I wanted to cover as big of a variety of ideas and sensors that I could come up with so that when I met with my team there would be a lot of inspiration to draw from. I tried to imagine using these sensors in a simple but unique way when combined with the touch screen, even if some of my ideas ended up being similar to ideas from the rest of the class. They were strong enough to inform what I wanted from the project, and get my brain used to using tools for programs and design that I am not familiar with.
When we met, my group's favorite idea from my sketches was sketch 9, my flashlight/camera shape detection sketch.

Refined Sketches

Iterating upon our chosen sketch, the main idea that stood out was using the camera to detect a specific item. We also found through our sketching that we wanted to use either the accelerometer or the gyroscope as a way to prevent accidental activation. We wanted to use the proximity sensor, but it did not have enough support for our needs. Ultimately, our main goal through refinement was to find an idea that could use the camera and machine learning detection to make some sort of password while using other sensors to ensure that this could not happen without the user knowing.

Final Product Description & Demo

Our final product is an unlock screen with three phases. For the first phase, the user must satisfy the conditions of keeping the phone at a certain orientation and having a certain light level so that the camera works as intended. The next phase is to win a best two out of three rock paper scissors match that is against the phone, which leads to the next screen upon the phone's defeat. For the last step the user must input a passcode by making rock paper scissors symbols in their pre-set pattern. I am happy with our final design, because it feels very secure and cohesive. There is also practically no way for a user to accidentally unlock their phone, because of our various measures to prevent that from happening. If there was anything I would change, it would be to iterate more on this idea to find a way to simplify the unlock process and connect the different screens more organically.

How to run:
Use an android phone and google chrome browser to navigate to https://cspc-581-a2.glitch.me/
Enable the ambient light sensor with the following steps:
 a) go to chrome://flags
 b) search for: Generic Sensor Extra Classes
 c) enable this
 d) phone needs to have ~50 lux of light to work, and the reason for this limit is because the camera needs light to capture gestures
Information on how to unlock the phone scrolls along the bottom of the screen, there is an icon on main page if you need more help on how to unlock the phone.
Source code: https://glitch.com/~cspc-581-a2