I. INTRODUCTION
Wearables market is continuously on the rise, paving the way for wearable devices to become a strong candidate as a future computing platform in the post-smartphone era. Wearables market is promising due to the increasing demand for various wearable applications covering users’ changing needs in medical, infotainment and fitness tracking domains. Especially wearable devices such as smartwatches and hearables are demonstrating commercial and consumer-level success. A market analysis forecasts that wearables technology market will reach $57.49 billion by 2025 with a CAGR (Compound Annual Growth Rate) of 9.9% during 2020 to 2025 [1]. Another forecast reports that wearable device user penetration is 5.9% in 2020 and will remain around 4.8% by 2024 even with the negative impact of COVID-19 [2]. Global smartwatch growth YoY (Year-over-Year) is 20.2% led by global brands such as Apple, Samsung and Garmin [3]. According to the International Data Corporation [4], demand for hearables helped the worldwide wearable markets to surge 94.6% during the third quarter of 2019. According to this source, top 5 wearable units include earwear (48.1%), wrist band (22.7%), smartwatch (20.9%), basic watch (7.1%) and other (1.3%).
Still, there is a skeptical view concerning what could be the ultimate killer application for wearables. Currently, various academic and industrial attempts are exploring stand-alone wearable applications [5] as well as cross-device interactive applications [6] for eXtended Reality (XR) that cover Mixed Reality (MR), Virtual Reality (VR) and Augmented Reality (AR). In the light of COVID-19, wearables are deployed to a new use case of tracking social distancing and employees at work [7].
In this paper, we propose a wearable audio book application based on collocated interaction using a combination of smartwatch and hearables.
Our contributions on this paper are as follows.
-
We describe our rationale behind this application and propose our approach as a promising killer application on wearables.
-
We identify 7 scenarios for wearable audio book applications that justifies use of our application over other platforms (i.e., smartphones or tables).
-
We demonstrate feasibility of our approach by developing a working prototype and demonstrating performance of interaction modalities in terms of interaction latency.
II. RELATED WORK
We review related work in emerging technology for innovative wearables and audio-book applications.
Recently cross-device interaction with wearables and other smart devices is studied in many areas. Weiss et al. used smartphone and smartwatch to measure biometrics using activities of daily living (ADL) [8]. Bi et al. studied identifying pen-holding gestures with a smartwatch [11]. Turner et al. compared input methods for typing on a smartwatch [12]. Buddhika et al. used smartwatch PPG sensor for gesture interactions [13]. Others are focusing on potential of hearables to augment new capability [15], physiological in-ear sensing [16] and in-ear health monitoring [17]. There are movements to provide gaming experience using smartwatch [9], applying smartwatch design fundamentals [10], and distributing smartwatch components for various scenarios [14].
There are several subscription-based audio book applications and services to be used in mobile and PC platforms. A partial list include Welaaa (www.welaaa.com), Storytel (www.storytel.com), Millie (www.millie.co.kr), Audible (www.audible.com), Google Play Book and Audiobooks.com. Some of these services provide a wearable application, yet most services operate and target on smartphone applications.
III. DESIGN OF THE WABA
We design and implement a Wearable Audio Book Application (WABA) based on a concept of collocated interaction [20] with a smartwatch with an earwear (i.e., hearables). In this section, we present our rationale for design considerations and analyze requirements for the WABA.
We have elicited criteria for what could be the ultimate killer application for wearables. To do so, we have studied and reviewed previous wearable applications such as text-entry applications [18][19] and input modalities for wearables [21][22][23]. Furthermore, we conceived the idea of transforming popular smartphone apps to counterpart wearable apps.
We have observed following characteristics that formed the basis of our design on the WABA.
-
An application that requires highly expressive input is not suitable for wearables. For example, messenger apps need a user to perform text entry on a small touchscreen. This kind of applications require expressive input capability and there exists a better alternative input modality such as voice-based input.
-
An application that requires a user’s intentional and focused attention consistently is burdensome. Unlike smartphones, wearable devices such as smartwatches should be used as “glanceable standalone devices” for completing quick tasks [24].
-
Video-based media applications may work well with head-mounted displays (HMD) for virtual reality (VR) experience. However, wrist-worn wearables are not fit for such use and HMD will also face high competition from portable smartphones and TVs with much spacious screen.
-
We have observed a momentum is building up on medical and fitness applications for wearables. Indeed, this is a promising area where the wearables will excel. This type of applications does not require high expressivity in inputs and work well with glanceable interactions.
-
We have identified a trendy and plausible approach using collocated interaction and cross-device computation offloading [25]. Notably, sales of hearables have surged recently.
To reflect aforementioned observations, we designed the WABA to provide simple expressions in inputs, less attentive (i.e., glanceable), audio-based media use case and exploit the cross-interaction and collocated interaction with a smartwatch with a hearables.
In this section, we describe how previous design considerations are reflected on the WABA in various forms of functional and non-functional requirements.
First, the WABA operates on simple expressions in wearable inputs. We simplified inputs on the WABA that uses traditional touchscreen inputs and multi-function side buttons. An audio book application can be designed to provide a simple set of functions (i.e., read, pause, volume control) that is mapped to the wearable input modality.
Second, the WABA works in less attentive or glanceable fashion. An audio book can be played as a background service that can continuously persist. Once the WABA is launched on the smartwatch, it does not require the wearer’s full attention, unless the user wants to directly control the app.
Third, we focus on audio-based media use case. The WABA provides media control functions (i.e., play, stop, fast-forward) as well as other audio book related functions (i.e, read in 2x speed, skip 15 seconds forward/backward, previous/next chapter). As discussed earlier, the WABA targets reading audio books using simple and glanceable interactions.
Lastly, the WABA benefits from cross-interaction and collocated interaction with a hearables. We envisioned the WABA to be used with a Bluetooth paired hearables to serve a natural audio book application. When a smartwatch is paired with a hearables, input and output modalities are decoupled and distributed to two devices.
To successfully claim the killer application for wearables, there needs to be designated use cases and scenarios that are distinguished from other devices (i.e., smartphones). We identified several compelling use cases and scenarios for the WABA. The WABA is applicable to people commuting (Figure 1) and jogging/running (Figure 2) as well as accompanying them when they work out (Figure 3). In these scenarios, people can concentrate on their main task (i.e., keeping eyes on walking, running and exercising) while keeping their hands free of smartphones and opening their ears for the WABA.
Additionally, the WABA is useful for employees working from home (WFH) or freelancers with a designated workspace as shown in Figure 4. For example, people can still use their laptops while using the WABA without any interference. Moreover, the WABA could replace radio and in-vehicle infotainment when driving a vehicle or doing housework in the home as shown in Figure 5 and Figure 6 respectively.
Figure 7 demonstrates a multi-user scenario. For parents, an audio book for children can be played by the parents’ smartwatch while the audio book is played on the child’s hearables. All these scenarios make wearables (i.e., smartwatch and hearables) a better choice than smartphones or tablets or PCs.
IV. PROTOTYPE IMPLEMENTATION
We designed and implemented a proof-of-concept (PoC) of the WABA using Android development platform. Android Studio was used to design the layout of the WABA and we programmatically implemented functions of the WABA using Android SDK APIs. The PoC application was tested on an Android Virtual Device (AVD) as well as on a commercial smartwatch running WearOS.
Figure 8 shows the WABA when it is first launched. It has 3 main touch-based control areas as highlighted in Figure 8. Android ImageView widgets are implemented as touch-based controls. Two arrows highlighted in two red dotted boxes are used to navigate between chapters or between different books. The speaker icon highlighted in the yellow dotted box shows mute on and off toggle control. The green dotted box shows an area designated for audio book controls such as play, pause, stop and play in 2x speed. Two functions not shown in the home screen are skipping 15 seconds forward and backward. These two functions will be triggered by other means of input modalities such as using multi-function side button or using gestures. In the center of the home screen, a book cover for currently playing audio book is displayed.
The WABA uses four different wearable interactions modalities. First modality is traditional touchscreen-based interaction used on the touchscreen. This modality is useful when the user is attentive to look down on the smartwatch. Second modality is the multi-function side buttons on the smartwatch as shown in Figure 9. These multi-function buttons are programmable and provide tactile feedback when clicked. Third modality is contactless modality such as gesture-based inputs. Lastly, cross-device input modality is supported by receiving touch-input from the hearables to the smartwatch by Bluetooth communication. All these four input modalities summarized in Table 1, have low input expressivity, yet they are expressive enough to operate an audio book application when their multi-modal inputs are enabled.
| Wearable Input Modality | Contact | Expressivity |
|---|---|---|
| Touchscreen | Screen | Low |
| Multi-function Button | Side Button | Low |
| Gestures | Contactless | Low |
| Cross-device Input Modality | Hearables | Low |
We implemented the WABA prototype on Android emulator and commercial smartwatch Fossil Q Explorist as shown in Figure 10. We paired the smartwatch with a Bluetooth earphone (QCY QY19-YL) to listen audio books through the connected hearables. To implement audio book controls, we used Android MediaPlayer class to play, pause, stop and speed-play audio books in mp3 format. For audio books, we downloaded free audio books from IEEE-USA audio books (https://ieeeusa.org/shop/audiobooks/) and details are shown in Table 2.
| Audio Book Title | Size (MB) | Length (minute:seconds) |
|---|---|---|
| Why STEM is important | 22 | 23:38 |
| Secrets to Being a World-Changer-Part 2 | 50 | 35:24 |
| Women in Engineering – Book 4 | 80 | 55:15 |
To capture the user’s touch inputs, we set a ClickListener for each ImageView widgets. To capture multi-function side button events, we implemented KeyEvent.Callback function onKeyDown() to check for click events of two multi-function buttons. On our tested smartwatch Fossil Q Explorist, it has two multi-function buttons. When they are clicked, they generate KeyEvent.KEYCODE_STEM_1 and KeyEvent.KEYCODE_STEM_2 respectively. To use wrist gestures such as flick wrist out and flick wrist in, we also check for KEYCODE_NAVIGATE_NEXT and KEYCODE_NAVIGATE_PREVIOUS in onKeyDown(). Bluetooth hearables also generated KeyEvent when its hardware buttons (i.e., Media Button in Android). KeyEvents accepted in the WABA are summarized in Table 3. Our current implementation does not allow direct controls from the hearables. Since different Bluetooth hearables have various Media Buttons, we need to explore further on generic KeyEvents that can be widely used in the WABA regardless the brands of hearables.
V. EXPERIMENTS AND DISCUSSION
We measured different wearable interaction latency on the WABA. The four tested wearable interaction modalities were 2 multi-function buttons and 2 gestures. In the experiment, 20 consecutive same inputs (i.e., events) are generated and recorded. The result is shown in Figure 11. We calculated an average for each event to measure the interaction latency. Hardware buttons (i.e., side multi-function buttons) were much faster than gestures. This can be explained by the fact “flick wrist out” and “flick wrist in” gestures are composed of two directional movements (i.e., move wrist outward and inward). If a user’s hands are full then gestures-based interaction could be used. Otherwise, a faster option is to use multi-function buttons when there is a free hand to spare. For the discussed 7 scenarios, gesture-based interaction is more suitable for running/jogging, work out and vehicle-driving scenarios.
VI. CONCLUSION
In this paper, we designed and implemented a wearable audio book application (WABA). As a possible killer application for wearables, we proposed using a smartwatch with a hearables for listening to audio books on wearables. We presented 7 scenarios for the WABA that are more realistic to be used with wearables than smartphones or tablets. We also designed the layout of the WABA and explored several different wearable interaction modalities expanded by collocated wearable devices (i.e., a pair of smartwatch and hearables). Through the implemented WABA prototype, we found that the proposed interaction modalities such as multi-function buttons and gestures demonstrate modest latency. Therefore, our presented WABA is expressive enough as an audio book application while providing alternative and multi-modal wearable interaction modalities.
There are several limitations of our approach that could be further polished in future studies. First, we did not have much programmatic control over the hearables. If a programmable earwear is used, there are many interesting approaches beyond just using the hearables as a mere output device. Second, a usability test on users’ behaviors of the WABA is remained that should help identify important functions and user experiences. We believe the WABA and its multi-modal interaction can be adopted by visually challenged people, but carefully orchestrated user studies are required. Lastly, the WABA uses media playback functions while communicating with a paired Bluetooth device (i.e., hearables). This leads to draining battery unexpectedly fast on wearables. Further studies are needed to investigate these important issues. Nonetheless, our proposed wearable application, WABA has shown promising initial results using collocated wearable devices with multi-modal interaction modalities for wearable audio book applications.
