I. INTRODUCTION
Cardiovascular disease (CVD) is known as the most common cause of death in the world. In 2013, 17.3 million out of 54 million of total deaths, or 31.5% of all global deaths were caused by CVD [1]. There are many different types of CVD such as coronary heart disease where the flow of oxygen-rich blood to the heart muscle is blocked or reduced and strokes where the blood supply to part of the brain is cut off. For this reason, CVD often lead to the heart attack, heart failure, brain damage and even death. With the rapid advance of technology, many researchers and companies have tried to develop wearable devices that have the ability to measure heart rate value with the express purpose of detecting possible cardiac problems [2].
There are currently two kind of sensors that can be utilized for calculating heart rate, namely electrocardiography (ECG) and photoplethysmography (PPG). The main difference between ECG and PPG is that while the former identifying cardiac movements by placing electrodes near the heart of a person [3], the latter monitoring the perfusion of blood to the dermis and subcutaneous tissue of the skin using conventional pulse oximeter to accomplish the same task [4-5]. Due to this hardware design, ECG measurement will obviously give us more robust and detail result than PPG measurement, thereby providing more accurate heart rate calculation.
Regardless of the ECG superiority over PPG, the majority of researchers and companies still rely on PPG as the main sensor in their wearable devices owing to the complexity and inconvenience nature of ECG sensor [6]. According to [7], ECG is actually impractical for long-term monitoring since it normally requires electrodes to be attached to the patient's chest, which is inconvenient and may interfere with day-to-day activities. On the other hand, PPG sensor can be worn on our wrist conveniently as it monitors the perfusion of blood only by transmitting light and calculating its reflection from the skin surface [8]. In addition to this, according to [2], PPG sensor is also a simple and low-cost optical technique as the basic form of PPG technology only requires a few opto-electronic components: a light source to illuminate the tissue, and a photodetector to measure the small variations in light intensity associated with changes in perfusion in the catchment volume.
Despite this convenience use of PPG over ECG, one must remember that PPG sensor measurement is also prone to incorrect readings due to its sensitive nature to noises [9]. Noise caused by people movement or motion artefacts is the main problem that many researchers have tried to solve for decades. Only small amounts of motion will cause PPG to interpret motion as the true signal or obscure the true signal with noise values that can lead to inaccurate readings, false alarms or missed true alarms [10]. The other factors, such as ambient light [11] and skin pigments, can also negatively affect PPG measurement, as has also been thoroughly reported by [12]. However, recent development has successfully dealt with these problems [13].
One of the solutions to increase the accuracy of PPG heart rate calculation is by reducing PPG noises, especially caused by movement artefacts. For achieving this, the common procedure is to utilize another sensor, such as accelerometer, in order to detect motion artefacts. Once we have the signal from the accelerometer, it can be used to reduce PPG noise.
Several techniques for reducing PPG noise caused by movement artefacts have been proposed. A real-time heartbeat estimation system by utilizing two PPG sensors was proposed in [14]. Similar approach, by using two PPG signals to estimate heart rate value, is proposed in [15]. Research on obtaining PPG data that show good correlation with ECG data by replacing PPG noise was conducted by [16].
Although the new approach [14-16] has improved PPG heart rate calculation, it would also increase energy consumption on the wearable device as it require more than one PPG sensors and a more complex signal processing. Moreover, using more than one PPG sensors separated by certain distance, such as 2 cm, is impractical since it would make the wearable device bulky.
The objective of our proposed idea is to improve PPG heart rate calculation while also avoiding the implementation of any expensive signal processing algorithm on the wearable device itself. This system will examine several sensors data to predict its corresponding ECG value for the purpose of extracting more precise heart rate data. Our approach is based on predicting ECG value by considering several parameters, namely PPG value, low-noise 3-axis accelerometer value, wide-range 3-axis accelerometer value and 3-axis gyroscope value. We also evaluate our system by using wrist PPG dataset provided by [17] which is available for public use in Physionet WaveForm DataBase (WFDB). The proposed system manages to make fairly precise ECG value predictions which allow it to produce a significant improvement on estimating heart rate value without increasing energy consumption on wearable device. We also integrate the concept of cloud computing to implement algorithms that requires expensive computation, namely neural network model [18-19].
In summary, this paper makes the following contributions:
-
Proposal of a neural network-based system which can improve PPG heart rate calculation by calculating its corresponding ECG value.
-
Performance evaluation and analysis of the trained neural network system by comparing the prediction result with the real ECG signal.
The remaining of the paper is structured as follows. Section 2 outlines related work on PPG calculation technique. Section 3 provides the design rationale and architecture of our proposed solution. Section 4 shows the performance evaluation through experimental result and analysis. Finally, in Section 5 we conclude our paper and discuss the future work directions.
II. RELATED WORK
One possible solution to improve PPG heart rate calculation is by using 2 PPG sensors, separated by 2 cm on the wrist, in order to find noise patterns as proposed by [14]. This algorithm uses two channels of PPG signals and a mean value of a previously estimated heartbeat rate to check whether there is a significant noise. If unusual amount of noises is detected, it will then discard those signals and select the closest peak frequency to previously estimated heartbeat rate. The main problem with this approach is it requires two PPG sensors separated by a distance of 2 cm, which is fairly too wide for a wearable device. Moreover, there is no enough evidence that can prove if this use of two PPG sensors approach will not consume too much energy in the wearable devices.
A similar method is proposed by [15] where they also use 2 PPG sensors to estimate heart rate value. The two-channel PPG signals obtained from two sensors are placed close to each other, so an average of two-channel PPG signals can be obtained. The averaged PPG signal and three accelerometer signals in the time domain are then converted into frequency domain by means of FFT before it is applied to a band-pass filter. The filtered PPG and accelerometer signals in the frequency domain are transformed back to the time domain by inverse Fast Fourier transform (IFFT) and the time domain signals are considered at the de-noising level. The location of the highest peak obtained from the periodogram of PPG1 and PPG2 are then averaged and verified to estimate the final heart rate value. Since this proposed method is similar to that of [14], it also has the same drawbacks due to its complexity.
[16] proposed the investigation of a noise handling method that provides high correlation between ECG and PPG data so that it can be used to detect driver's drowsiness. Both noise-replacement technique and noise-filtering technique are used to improve PPG measurement and therefore existing ECG-based method can be applied for PPG data. The experiment result showed that the former technique, which substitutes noise instead of filtering it, is more effective. Although this approach can improve PPG measurement, it cannot really reproduce ECG data. The total correlation coefficient for R-R interval is only 0.30±0.139, which is far bellow our result.
In complement to the discussed studies, we propose a neural network-based ECG value prediction system that can minimize the PPG heart rate calculation error by predicting ECG value from its corresponding PPG, accelerometer, and gyroscope values. The whole system will be processed on the cloud since wearable devices are unable to do complex computation such as neural-network algorithm.
III. PROPOSED METHOD
As mentioned in Section 1, neural network-based ECG value prediction is proposed in this paper. This section will cover the general design of our system and neural-network architecture of the ECG value predictor. In addition, we describe our implementation of the proposed system, including evaluation procedures and dataset that were used.
Fig. 1 depicts our proposed system where the relationship between ECG value predictor and the other entities in this system can be seen. RESTful API server, which responsible for controlling every incoming request from the client (see Fig. 1 circle) and maintaining security, receives sensors data from the PPG devices. After checking the credibility of the devices, these data will then be passed to the ECG value predictor where neural network model runs to estimate its corresponding ECG value. After receiving the calculation result from the ECG value predictor, RESTful API server will save the result on the database. This step will ensure that we have every record and are able to retrieve it at a later time when it is needed. Moreover, the tree entities (RESTful API sevrer, ECG value predictor and database) are established inside the public cloud. Our prime objective by implementing the main system on the cloud is simply to avoid running any expensive computation (e.g., neural network) on the PPG devices. Finally, at the client's request, RESTful API server will fetch the data from the database and send it back to the authorized PPG devices or the other authorized devices (e.g personal computer).
This proposed system has two main advantages. First, it will allow us to alleviate hardware limitation problems of the PPG devices by implementing the main system on cloud. Second, and more importantly, this proposed system also offers flexibility in tackling PPG noises problem. For example, as stated in Section 1, PPG measurement is also negatively affected by ambient light. Although this is no longer serious problem for PPG measurement, suppose we want to also consider the effect of ambient light, we can just simply include ambient light measurement as one of our neural network inputs. However, we still need to prove this hypothesis by conducting experiment with more dataset and therefore this can also be important future research direction.
Fig. 2 describes the sequence diagram between PPG / other devices with RESTful API Server and ECG value predictor on the cloud. In order to ensure the security aspect of this system, JSON web token (JWT) is used to differentiate authorized user from unauthorized user. JWT is selected for this system due to the fact that it is perfectly suitable for RESTful API architecture which naturally uses stateless data transfer protocol.
Data communication steps of this system are as follows:
-
Users are required to send their username and password from either PPG devices or other devices such as mobile phone to the server in order to authenticate themselves before being able to use this system.
-
If username and password provided by the user are valid, server will then generate JWT and send it to the user.
-
When user makes any request to the server, the generated JWT from the server is attached to the request header and will be used by the server for the request validation process.
-
In case of post request, sensors data will be sent and processed by the ECG value predictor only if JWT in the request header can be validated by the server.
Fig. 3 shows the neural network model of ECG value predictor. The typical fully-connected neural network is used.
Four sensors are used as input features to predict the corresponding ECG value.
There are 10 input features derived from four different kind of sensors, namely PPG, 3-axis gyroscope, low-noise 3-axis accelerometer, wide-range 3-axis accelerometer, where each 3-axis sensors contributes 3-dimensional values (X, Y, Z). Each input is used inside the hidden layer to compute one output value, the corresponding ECG value, meaning there is no PPG noise filtering applied on this system beyond that built in to the hardware as mentioned by [17]. This system design avoids any computational activity in the PPG device since the device only needs to upload sensor data to the cloud and receive the result back at client request.
Keras library is used for implementing neural network in ECG value predictor. Four hidden layers with 150 neurons each are used in this system. Since ECG value predictor is actually a neural network model for running a regression task, there is only one output neuron which is the corresponding ECG value. Prior to the implementation and evaluation process, the dataset was split into training and test data with the training set being 80% of the data (±4 minutes), and the testing set being the remaining 20% of the data (±1 minute).
For the activation function and gradient optimization technique, we select LeakyRelu [20] and ADAM [21] respectively. LeakyRelu was chosen as our activation function due to its superiority in tackling vanishing gradient problem and dying neurons. ADAM was used as the gradient optimization technique because it requires less memory and is well suited for large problem in term of data and parameters. Moreover, ADAM, which combines the idea of momentum optimization and RMSProp, is an adaptive learning rate algorithm, meaning it requires less tuning of the learning rate hyperparameter, making it easier to use than gradient descent.
In addition, to avoid over-fitting, sensor data is shuffled and two regularization techniques are also implemented, namely early stopping and dropout [22]. Early stopping enables us to stop the training process when our model starts to over-fit our training data by monitoring its performance on the validation set. Dropout will make sure that we have more robust model that can generalize to unseen data by breaks up co-adaptations between neurons. By using random dropout, the presence of any particular hidden unit will be unreliable, and thus forcing each neuron to pay attention to each of its input and being less sensitive to slight changes in its input.
Evaluation steps of our proposed system are as follows:
-
Downloading dataset from Physionet WaveForm DataBase (WFDB);
-
Converting dataset from dat to csv;
-
Training neural network model using the dataset;
-
Evaluating the trained model by using the following four different approaches:
-
Calculation of model accuracy on training and test dataset:
To calculate the accuracy of our neural network model, we are using coefficient of determination or R2, which is formally defined as Equation 1.
where i indicates the index of the sensor data; y is our ECG value target and f is our ECG value prediction.
-
Correlation analysis between the original or ground truth ECG value and our prediction result: We drew scatter plot between the original ECG value and our prediction result on training and test dataset. We also provided the regression line equations and correlation coefficients in both training and test dataset experiment.
-
Bland-altman plot:
In order to analyze the agreement between the original or ground truth ECG measurement and the prediction result, bland-altman plot [23] was used.
-
Signal comparison with the original ECG signal: We also plot our prediction on test data and compare it with its original or ground truth ECG value.
-
As stated in Section 1, ECG value prediction in this system was evaluated by using dataset provided by [17]. In this dataset, 19 records from eight participants are provided. Participants' ages ranged from 22-32 years (mean 26.5) with three participants identified as male and five participants reported as female. Participants were required to perform one or more different types of exercise, namely walking on a treadmill, light jog or run on a treadmill, pedal on an exercise bike set a low resistance and pedal on an exercise bike set at a higher resistance. Each activities would last up to 10 minutes.
A total of 11 signals derived from several different sensors were provided by this dataset. These sensors are as follows:
-
ECG unit
-
PPG sensor
-
±2 g low-noise 3-axis accelerometer
-
Up to ±16 g wide-range 3-axis accelerometer
-
3-axis gyroscope with 0.0481 degrees per second (dps) noise floor
For ECG monitoring, Actiwave (CamNtech, Cambridge, UK) recorder and pre-gelled self-adhesive Silver-Silver Chloride (Ag/AgCl) electrodes were used. During experiment, these sensors were then placed on the participants' upper chest.
Shimmer 3 GSR+ unit (Shimmer Sensing, Dublin, Ireland) with a 510 nm green LED reflective mode PPG sensor connected to it by using 3.5 mm headphone port was used to record PPG and motion data. A single package of Shimmer 3 GSR+ contains a gyroscope, a low-noise accelerometer and a wide-range accelerometer. These sensors were then placed on the left wrist of the participants.
IV. PERFORMANCE EVALUATION
In this section, we will show the performance of our ECG value predictor on one of the participants. As explained in Section 3, the accuracy of our neural network model is calculated by using coefficient of determination or R2. The accuracy of our model to predict ECG value from its corresponding PPG value is 90.9% on the training set and 81.6% on the test set.
Fig. 4 shows the scatter plot between the original or ground truth ECG measurement and the prediction result. The red empty triangle is the scatter plot of the training data while the green empty circle is the scatter plot of the test data. We also calculated the regression line for both training and test data. Equation for regression line is defined as Equation 2:
where m denotes the slope of the line and b denotes the y-intercept. For regression experiment, the ideal best-fitted line equation would have m and b equal to 1 and 0 respectively as it would result in y = x, meaning there was no error in the prediction result. The best-fitted line equations for our experiment result on train and test data are y = 0.94x − 0.96 and y = 0.83x − 2.35 respectively, where x indicates the original or ground truth ECG measurement and y denotes the prediction result. Moreover, the correlation coefficient (r) between the ground truth ECG measurement and the prediction result on training data is 0.95 (p < 0.00001) while the experiment on test data yields a correlation coefficient of 0.904 (p < 0.00001). These results indicate that our prediction on both training and test dataset are strongly correlated with the ground truth value.
In Fig. 5, we also use Bland-Altman plot to analyze the agreement between the original or ground truth ECG value and the ECG prediction result on the test data. Bland-Altman plot is normally use for clinical measurement [23]. The purpose of this technique is to measure by how much the two methods differ from one another. If this will not cause a problem in clinical interpretation, we can use the two methods interchangeably. By utilizing this plot, we aim to assess the relation between our ECG prediction and the original or ground truth ECG measurement. Our ultimate goal will be to conclude whether our ECG prediction result can be used to replace the original ECG measurement or not. The following five steps are the procedure to draw bland-altman plot:
-
Calculating the difference and the means of the two data (ECG prediction value and ground truth ECG measurement)
-
Plotting the difference against the means
-
Calculating mean difference (d̅)
-
Calculating standard deviation of the difference (s)
-
Drawing line for d̅, d̅ − 2s, d̅ + 2s
The average of the difference (d̅) between our ECG prediction value and the ground truth ECG measurement is 2.29 while 199.12 and −194.54 are the limits of agreement (±1.96 standard deviation of the difference). We can also see that most of the differences in this experiment lies between these limits of agreement, which indicate that the differences are normally distributed (Gaussian). By using this plot alone, we can clearly observer by how much our ECG prediction result differ from the ground truth or original ECG measurement. However, it is still difficult to claim whether these results or to be precise, these limits of agreement are acceptable or not. In order to answer this question, we plotted our prediction result and compared it with the original or ground truth ECG measurement, as can be seen in Figs. 6-8.
Fig. 6 depicts the comparison between the original or ground truth ECG measurement and the prediction result on test data. The blue line is the original measurement plot while the orange line is the result of the prediction. It can be seen that, in general, our model's prediction on test data can resemble the shape of the original ECG measurement data. Moreover, it should be noticed that both data have the maximum value of ±2000mV and the minimum value of ± −2000mV. However, by only observing this figure, we cannot clearly see the shape of the signals and where the errors happened; therefore, we also plotted the first 500 data points in Figs. 7-8.
Comparison of the first 500 data points that shows the more vivid resemblance between the original ECG measurement and the ECG prediction result on test data can be seen in Fig. 7. From this figure, it can be seen that the signals of the original ECG measurement and the prediction result overlap one another, meaning our prediction signal resembles the original signals. In this figure, we can also observe where the errors often happened. It usually occurred when the value is between ±250mV and ± −250mV. These errors make some sense since unlike ECG that directly monitors articular and ventricular activities, PPG only detect the volume of the blood injected by the left ventricular that arrive in the extremities or in other words, it monitors ventricular systole. Although this means that by predicting ECG value from PPG value, we will loss the other information, as shown by this error, it is sufficient to use it for calculating heart rate since heart rate calculation is usually derived from ECG's QRS complex, or to be precise, the number of QRS complex within certain period, which basically indicates ventricular contraction. This hypothesis is supported by our result, as shown in Fig. 7, where our model can precisely reproduce the ECG's QRS complex.
By only observing Fig. 7, we cannot really tell the peak value since the two signals overlap one another. Therefore, in Fig. 8, we show the closeness of the ground truth ECG measurement and the prediction result on test data in separate plot so that we can observe every value in more detail. If we refer back to our Bland-Altman plot result, we can conclude that our mean difference and limits of agreement values will unlikely cause serious problem in calculating heart rate as it will not significantly affect the prediction of QRS wave and it also often happen during the other cardiac activities or, indicated by the value ranged from ±250mV and ± −250mV in this experiment. Therefore, this should bring us to our ultimate conclusion that our ECG value prediction can replace the original or ground truth ECG measurement.
Based on the result shown above, ECG value predictor can successfully predict the ECG value based on its corresponding PPG, accelerometer and gyroscope value. This result may further assist PPG sensor on the wearable device to estimate more precise heart rate information, including but not limited to, heart rate value without the need to implement any expensive computation on the device itself. It goes without saying that our proposed neural network and cloud computing-based heart rate calculation from PPG data will improve the efficiency of many existing systems that based on heart rate calculation and internet of things (IoT) such as that of [24].
The main drawback of our system is that it cannot perfectly predict ECG value across different people since blood volume or to be precise, the amount of blood circulating within an individual depends on their size and weight [25-26]. This means that even though two different persons have the exact same electrical activities in their heart at the same time, the perfusion of blood in their wrist at that time is not necessarily the same. This problem can, in fact, be solved by adding more features such as weight and height or even blood volume estimation as our neural network input; however, the lack of available public dataset that related to this prevents us to prove this hypothesis.
V. CONCLUSION
ECG sensor can provide more precise heart rate calculation than PPG sensor as PPG sensor is heavily influenced by motion artefact; however, due to its simplicity, PPG sensor is still widely used for smart devices. Therefore, in this paper, we proposed a neural network-based system that can use PPG, accelerometer and gyroscope sensors data to predict the prediction accuracy to close to ECG value.
We have tested the performance of our proposed system by using dataset provided by [17]. The results have shown that our proposed ECG value predictor can precisely predict the corresponding ECG value from a combination of PPG, accelerometer and gyroscope values. Our deep learning model is also implemented on the cloud, meaning there is no need to run any complex algorithm on the PPG device itself.
There is one important area for future work. This work is to collect more dataset that at least, including size and weight from several different people in order to conduct further research on predicting ECG value across different people. The alternative way to this solution is to develop an algorithm that can calculate the average perfusion of the blood and either adaptively select the neural network model that can suit the results better or add this result to the neural network input features.
