**Blinking Eyes Detection to Monitor Drowsy Drivers Due to Fatigue Using MATLAB Cascade Object Detector**
\*\*Double blind review, please do not include authors information in this version \*\*
Received Date: \*date
Accepted Date: \*date
Published Date: \*date
**HIGHLIGHTS**
- Road accidents can be avoided by detecting fatigue and drowsy faces by looking at the eye area.
- Blinking eye detection can be used to monitor driver’s fitness level.
- Viola-Jones algorithm using MATLAB cascade object detector was used to detect the presence of blinking eyes.
- The results of the Classification Table derived from the algorithm successfully provided output to detect the presence of blinking eyes for the purpose of monitoring drowsy drivers due to fatigue.
ABSTRACT
*Road accidents are incidents that should be avoided. One of the contributing factors of road accidents occurrence is drowsiness while driving due to fatigue. In this project, fatigue and drowsiness of a person can be detected by looking at the eye area. Drowsy situations are dangerous especially when driving a vehicle over long distances. When a person starts to feel drowsy, the eyes will start to blink more frequently. This characteristic can be used to monitor a driver’s fitness level. In this project the Viola-Jones algorithm using MATLAB cascade object detector was used to detect the presence of blinking eyes. The algorithm is consisted of three phases which are, selection of Haar-like characteristics, integrating a picture into the whole, and Classifiers in a cascading fashion. A number of 29 samples images consisting the condition fatigue face and non-fatigue face was used. The obtained results obtained were calculated based on the accuracy value of the detected blinking eye. The result shows that both the total Area Under the Curve (AUC) values for faces with fatigue situations and non-fatigue situations are above acceptable values which is 0.5. This indicates both classifications are acceptable and can be used to detect the presence of blinking eyes which represent drowsiness.*
*Keywords: Blinking Eyes Detection, Cascade Object Detector, MATLAB, Classification Table*
# INTRODUCTION
Road accidents are incidents that should be avoided. One contributing factor to the occurrence of accidents is drowsiness while driving due to fatigue. Drowsiness can be described as a condition of people who are almost asleep and difficult to stay awake due to factors such as body fatigue or due to lack of sleep (Bereshpolova, Stoelzel, Zhuang, Amitai, Alonso, & Swadlow, 2011). Drowsy situations can trigger danger especially when performing tasks that require constant concentration, such as driving a vehicle over long distances. Consequences of being in a fatigue state while driving and experiencing a drowsy state, it is feared may result in accidents (Khan & Panchal, 2015). Therefore, a method to help detect drowsiness while driving is needed to prevent accidents.
This study suggested the detection of fatigue and drowsy faces by looking at the eye area. Changes in eye condition are seen as appropriate for estimating the existence of a person’s fatigue and drowsiness. According to Wolkoff, Nojgaard, Troiano, and Piccoli (2005), the time between eye blinks will increase when the eye has worked under stressful conditions and within a certain period. This situation exists as driving is a challenging activity and needs to be done with full concentration. Therefore, this study proposes to detect blinking eyes as a method of assessing the existence of fatigue and drowsiness experienced by a driver. The Viola-Jones algorithm using MATLAB cascade object detector will be used to detect the presence of blinking eyes.
# LITERATURE REVIEW
Driving over long distances can be dangerous if safety measures are ignored. One of the important safety measures is to keep a driver fresh and energetic. However, fatigue and drowsiness can change the handling style of each driver and increase the risk of accidents to occur. According to a study by Philip et. al. (2005), fatigue and drowsiness will significantly affect the current condition and efficiency of the driver. In another study, Pourabdian, Lotfi, Yazdanirad, Golshiri, and Hassanzadeh (2020) has tested 400 drivers and found that fatigue and unsafe driving habits are closely linked. Furthermore, fatigue and tension tends to exist in a two-way form. In 2011, a study done by Kocalevent, Hinz, Brähler, and Klapp, also shows that fatigue is strongly correlated with perceived stress.
Blinking eye detection can be used to monitor a driver’s fitness level. This is because when the driver's fitness level is at a minimum, the eyes will blink. Thus, the blinking eye is a natural protective system that protects the eyes from environmental exposure. For example, when a driver feels fatigue, the driver's fitness level will directly decrease. This causes the driver’s metabolic rate and mental ability to also decrease. One of the natural reactions that exist is that the eyes will blink due to feeling fatigue and drowsy. Therefore, the condition of spontaneous blinking eyes is considered as an accurate indication to diagnose the existence of fatigue and drowsiness due to diligent driving activities.
Eye blinking detector is a system that could detects blinking eyes that were resulted from fatigue and drowsiness. Peng, Chen, Ruan, and Kukharev (2005) have proposed an eye detection algorithm which was based on gray intensity images. In the algorithm, point template was stored and later will be matched with existing template. The results obtained give a value of 95.2% accuracy. However, the recommended technique was found not to work well if the detection was performed for spectacle wearers although the average implementation time of the proposed algorithm had shown that the approach performed was quite efficient. Another related work is a face detection algorithm proposed by Sirohey and Rosenfeld (2019) to detecting human face based on eye images. They used Gabor filter or also known as linear filter, and wedge-shaped or nonlinear shape filter for feature extraction. Based on the filters used, a detection rate of 90% was achieved without false alarms.
# METHODOLOGY
The method used in this study demonstrates the use of the Viola-Jones algorithm with Cascade Object Detector technique. According to Chopparapu, Seventline, and Beatrice (2020), the efficiency of the Viola-Janes algorithm can be improved by producing inseparable images, thus help reduce errors in analyzing the captured video. This algorithm uses the concept of a rectangular feature that involves the number of pixels in a rectangular area.
In addition, MATLAB software was used as a method to apply the algorithm on the blinking eye detection system. There are three phases to the algorithm which are; selection of Haar-like characteristics, Integrating a picture into the whole, and Classifiers in a cascading fashion. MATLAB has the vision to do this. The computer toolbox contains the CascadeObjectDetector system object, which uses the Viola-Jones method to recognise faces and objects. There are numerous pre-trained classifiers in vision.CascadeObjectDetector that can recognise frontal faces, profile faces, noses and eyes, as well as the upper body. This code is to define the face Detector object
> faceDetector = vision.CascadeObjectDetector
Using one of the attributes from vision as a second step, MergeThreshold is a CascadeObjectDetector. This data will be utilized to improve detection and accuracy of threats.
> faceDetector.MergeThreshold = 4
As a result, the default value is four. Depending on the amount of precision necessary, MergeThreshold can be changed. In other words, face identification accuracy is higher at 3 than it is at 8, and vice versa.
To detect drowsiness, Chopparapu et. al. (2020) recommends measuring the time a person closes his/her eyes, and if the eyes are closed for longer than normal blinking time, the person tends to fall asleep. According to Arceda, Nina, and Fabian (2020), the average human blinking takes around 202.24ms while the drowsy blinking period takes 258.57ms.
The results obtained will be calculated based on the accuracy value of the blinking eye detection. The formula used to calculate the accuracy is based on the following Classification Table as Table 1:
Table 1: Classification Table
| | Fail Observe | Success Observe | |
| --------------- | ------------ | --------------- | ----- |
| Fail Predict | **TN** | **FN** | PN |
| Success Predict | **FP** | **TP** | PP |
| | ON | OP | Total |
where;
> PP = predicted positive = TP + FP, PN = predicted negative = FN + TN,
>
> OP = observed positive = TP + FN, ON = observed negative = FP + TN and
>
> Total = the total sample size = TP + FP + FN + TN.
and;
> TP = True Positive = correctly classified results obtained to be positive.
>
> FP = False Positive = incorrectly obtained result is classified as positive.
>
> TN = True Negative = results obtained that are correctly classified become negative.
>
> FN = False Negative = results obtained that are incorrectly classified as negative.
# RESULTS
The experiments were performed using hardware such as laptops and webcams. A laptop is required to build the programming code for the system, while the webcam required is a High Definition (HD) type used as a tool to capture a picture of the driver’s face. In addition to hardware, software is also required such as MATLAB, Video Cutter, and Third Format Factory Software.
The experiment was conducted on 29 samples of drivers. Each driver was photographed while given a driving assignment. The captured driver's face images were stored in the form of frames that been divided into two, first the driver's with a fatigue face situation and the driver's with non-fatigue face situation.
**Accuracy Values for Detecting Blinking Eyes**
The data obtained were type-in into two different tables, namely Table 2 and Table 3. Table 2 will display the driver's face data for Fatigue situations, while Table 3 displays the driver's facial data for non-Fatigue situations.
Table 2: Driver face data during Fatigue situations
| | Observed | Cumulative | | | | | |
| ------ | --------------------- | ------------------------ | --------------------- | ------------------------ | ------ | ------ | ------ |
| Frames | Detectfail | DetectSuccess | Detectfail | DetectSuccess | FPR | TPR | AUC |
| | | | 0 | 0 | 1.0000 | 1.0000 | 0.0000 |
| 90 | 0 | 30 | 0 | 30 | 1.0000 | 0.9843 | 0.0204 |
| 90 | 6 | 37 | 6 | 67 | 0.9793 | 0.9649 | 0.0166 |
| 102 | 5 | 53 | 11 | 120 | 0.9621 | 0.9371 | 0.0259 |
| 108 | 8 | 47 | 19 | 167 | 0.9345 | 0.9124 | 0.0220 |
| 109 | 7 | 55 | 26 | 222 | 0.9103 | 0.8836 | 0.0274 |
| 112 | 9 | 77 | 35 | 299 | 0.8793 | 0.8432 | 0.0087 |
| 119 | 3 | 56 | 38 | 355 | 0.8690 | 0.8138 | 0.0225 |
| 120 | 8 | 65 | 46 | 420 | 0.8414 | 0.7798 | 0.0269 |
| 124 | 10 | 57 | 56 | 477 | 0.8069 | 0.7499 | 0.0181 |
| 128 | 7 | 43 | 63 | 520 | 0.7828 | 0.7273 | 0.0301 |
| 133 | 12 | 78 | 75 | 598 | 0.7414 | 0.6864 | 0.0284 |
| 137 | 12 | 79 | 87 | 677 | 0.7000 | 0.6450 | 0.0245 |
| 139 | 11 | 62 | 98 | 739 | 0.6621 | 0.6125 | 0.0253 |
| 142 | 12 | 73 | 110 | 812 | 0.6207 | 0.5742 | 0.0436 |
| 144 | 22 | 85 | 132 | 897 | 0.5448 | 0.5296 | 0.0384 |
| 150 | 21 | 86 | 153 | 983 | 0.4724 | 0.4845 | 0.0150 |
| 152 | 9 | 73 | 162 | 1056 | 0.4414 | 0.4463 | 0.0200 |
| 153 | 13 | 72 | 175 | 1128 | 0.3966 | 0.4085 | 0.0085 |
| 153 | 6 | 59 | 181 | 1187 | 0.3759 | 0.3776 | 0.0078 |
| 154 | 6 | 79 | 187 | 1266 | 0.3552 | 0.3361 | 0.0070 |
| 154 | 6 | 37 | 193 | 1303 | 0.3345 | 0.3167 | 0.0229 |
| 158 | 21 | 95 | 214 | 1398 | 0.2621 | 0.2669 | 0.0110 |
| 159 | 12 | 64 | 226 | 1462 | 0.2207 | 0.2334 | 0.0048 |
| 160 | 6 | 40 | 232 | 1502 | 0.2000 | 0.2124 | 0.0081 |
| 160 | 11 | 90 | 243 | 1592 | 0.1621 | 0.1652 | 0.0085 |
| 160 | 15 | 82 | 258 | 1674 | 0.1103 | 0.1222 | 0.0042 |
| 162 | 10 | 63 | 268 | 1737 | 0.0759 | 0.0891 | 0.0015 |
| 163 | 5 | 77 | 273 | 1814 | 0.0586 | 0.0488 | 0.0029 |
| 166 | 17 | 93 | 290 | 1907 | 0.0000 | 0.0000 | 0.0000 |
| | 290 | 1907 | | | | | 0.5010 |
Table 2 displays the data related to the driver's face with the fatigue situation. There are five main columns that need to be observed in Table 2 namely Detectfail and Detectsuccess which is generally a breakdown of the face data captured from observation. In addition, the columns that are important to note are such as False Positive Rate (FPR), True Positive Rate (TPR), and Area Under the Curve (AUC).
Meanwhile, out of 29 driver face samples, the 10th row with 128 frames was selected as the cut-off. Based on the cut-off, the values of FPR, TPR, and Accuracy were obtained as 0.782759, 0.72732, and 0.659991 respectively. This illustration of the relationship of TPR versus FPR is then plotted as a Receiver-Operating Characteristic (ROC) curve as shown in Figure 1.
\[CHART\]
Figure 1: ROC curve for fatigue situation - TPR vs FPR
Figure 1 plots the relationship between TPR and FPR. The curve that exists is known as the ROC curve. Based on the observation of the shape of the curve, it is close to the shape of a straight line directly proportionate from the value of 0 to 1, with only a few insignificant bounces. This situation shows the ROC curve created for this fatigue situation as not good since the value below the curve is far off to reach a higher value approaching 1.0.
Table 3: Driver face data during non-fatigue situations
| | Observed | Cumulative | | | | | |
| ------ | --------------------- | ------------------------ | --------------------- | ------------------------ | ------ | ------ | ------ |
| Frames | Detectfail | DetectSuccess | Detectfail | DetectSuccess | FPR | TPR | AUC |
| | | | 0 | 0 | 1.0000 | 1.0000 | 0.0391 |
| 90 | 7 | 54 | 7 | 54 | 0.9609 | 0.9655 | 0.0108 |
| 90 | 2 | 45 | 9 | 99 | 0.9497 | 0.9367 | 0.0628 |
| 102 | 12 | 34 | 21 | 133 | 0.8827 | 0.9150 | 0.0613 |
| 108 | 12 | 41 | 33 | 174 | 0.8156 | 0.8887 | 0.0695 |
| 109 | 14 | 32 | 47 | 206 | 0.7374 | 0.8683 | 0.0146 |
| 112 | 3 | 23 | 50 | 229 | 0.7207 | 0.8536 | 0.0095 |
| 119 | 2 | 57 | 52 | 286 | 0.7095 | 0.8171 | 0.0137 |
| 120 | 3 | 39 | 55 | 325 | 0.6927 | 0.7922 | 0.0000 |
| 124 | 0 | 50 | 55 | 375 | 0.6927 | 0.7602 | 0.0467 |
| 128 | 11 | 67 | 66 | 442 | 0.6313 | 0.7174 | 0.0281 |
| 133 | 7 | 35 | 73 | 477 | 0.5922 | 0.6950 | 0.0272 |
| 137 | 7 | 37 | 80 | 514 | 0.5531 | 0.6714 | 0.0075 |
| 139 | 2 | 63 | 82 | 577 | 0.5419 | 0.6311 | 0.0423 |
| 142 | 12 | 44 | 94 | 621 | 0.4749 | 0.6029 | 0.0000 |
| 144 | 0 | 37 | 94 | 658 | 0.4749 | 0.5793 | 0.0129 |
| 150 | 4 | 38 | 98 | 696 | 0.4525 | 0.5550 | 0.0341 |
| 152 | 11 | 55 | 109 | 751 | 0.3911 | 0.5198 | 0.0116 |
| 153 | 4 | 61 | 113 | 812 | 0.3687 | 0.4808 | 0.0081 |
| 153 | 3 | 82 | 116 | 894 | 0.3520 | 0.4284 | 0.0215 |
| 154 | 9 | 61 | 125 | 955 | 0.3017 | 0.3894 | 0.0196 |
| 154 | 9 | 93 | 134 | 1048 | 0.2514 | 0.3299 | 0.0000 |
| 158 | 0 | 42 | 134 | 1090 | 0.2514 | 0.3031 | 0.0254 |
| 159 | 15 | 65 | 149 | 1155 | 0.1676 | 0.2615 | 0.0044 |
| 160 | 3 | 102 | 152 | 1257 | 0.1508 | 0.1963 | 0.0110 |
| 160 | 10 | 42 | 162 | 1299 | 0.0950 | 0.1694 | 0.0038 |
| 160 | 4 | 59 | 166 | 1358 | 0.0726 | 0.1317 | 0.0000 |
| 162 | 0 | 87 | 166 | 1445 | 0.0726 | 0.0761 | 0.0013 |
| 163 | 3 | 77 | 169 | 1522 | 0.0559 | 0.0269 | 0.0015 |
| 166 | 10 | 42 | 179 | 1564 | 0.0000 | 0.0000 | 0.0000 |
| | 179 | 1564 | | | | | 0.5882 |
Table 3 displays the data related to the driver's face with the non-fatigue situation. In this Table 3, the values are recorded for TPR is 0.71739, FPR value is equivalent to 0.63128, and Accuracy value is 0.68158. The illustration of the relationship of TPR versus FPR for Table 3 is plotted as shown in Figure 2.
\[CHART\]
Figure 2: ROC curve for non-fatigue situation - TPR vs FPR
Based on the observations in Figure 2, the resulting curve is slightly different from that shown in Figure 1. This is because the shape in Figure 2 has a higher upward curve, which shows that the AUC value on the ROC curve for TPR vs FPR in Figure 2 is larger than the ROC curve for TPR vs FPR in Figure 1. This means that the data in Table 3 are better than the data in Table 2.
**CONCLUSION**
This study is about detecting blinking eyes. The purpose of detecting blinking eyes is to recognize the faces of drivers who drive in a fatigue and drowsy state. Based on the results of the study, two things can be concluded with the first being based on Figure 3.
\[CHART\]
Figure 3: Comparison of ROC curve between fatigue and non-fatigue situations - TPR vs FPR
Through Figure 3, a direct comparison between the two ROC curves based on fatigue situation and non-fatigue situation was performed. Based on the comparison, it is found that the ROC curve for non-fatigue situations is more convex upwards than the ROC curve for fatigue situations. This means that the AUC value for non-fatigue situations is larger, and it directly indicates that classification through non-fatigue situations is better. However, this does not mean that the classification based on the fatigue situation is not good, because the AUC value for the ROC curve resulting from the fatigue situation still exceeds the value of 0.5.
Proving that the AUC values for both ROC curves exceed the values of 0.5, Table 2 and Table 3 will be referred. If observed in both tables, the total AUC values for faces with fatigue situations are 0.5010 and the total AUC values for faces with non-fatigue situations are 0.5882. This indicates both classifications are acceptable.
The conclusion that can be made is that the use of Viola-Jones algorithm through MATLAB cascade object detector has successfully provided output to be used to detect the presence of blinking eyes for the purpose of monitoring drowsy drivers due to fatigue.
**REFERENCES**
Arceda, V.E.M., Nina, J.P.C., & Fabian, K.M. F. (2020, September). *A survey on drowsiness detection techniques*. Paper presented at the Iberoamerican Conference of Computer Human Interaction, Arequipa, Perú. Retrieved July 15, 2021, from http://ceur-ws.org/Vol-2747/paper14.pdf
Bereshpolova, Y., Stoelzel, C.R., Zhuang, J., Amitai, Y., Alonso, J.M., & Swadlow, H.A. (2011), Getting drowsy? Alert/nonalert transitions and visual thalamocortical network dynamics. *J Neurosci*. *31*(48), 17480-17487. doi: 10.1523/JNEUROSCI.2262-11.2011.
Chopparapu, S., Seventline, J., & Beatrice, G., (2020), GUI for object detection using Voila method in Matlab. *International Journal of Electrical Engineering and Technology*, 11(4), 169-174. Retrieved July 17, 2021, from https://ssrn.com/abstract=3657124
Khan, N., & Panchal, D. (2015). Fatigue detection system using image processing on video sequences. *International Journal of Engineering and Technical Research*. *4*(09), 398-400. doi: 10.17577/IJERTV4IS090470
Kocalevent, R. D., Hinz, A., Brähler, E., & Klapp, B. F. (2011). Determinants of fatigue and stress. *BMC Research Notes*, 4(1), 238. doi: 10.1186/1756-0500-4-238
Peng, K., Chen, L., Ruan, S., & Kukharev, G. (2005). A robust algorithm for eye detection on gray intensity face without spectacles. *Journal of Computer Science and Technology*. *5*(3), 127-132. Retrieved July 17, 2021, from https://www.researchgate.net/publication/242272837\_A\_Robust\_Algorithm\_for\_Eye\_Detection\_on\_Gray\_Intensity\_Face\_without\_Spectacles
Philip, P., Sagaspe, P., Taillard, J., Valtat, C., Moore, N., Akerstedt, T., Charles, A., & Bioulac, B. (2005). Fatigue, sleepiness, and performance in simulated versus real driving conditions. Sleep, 28(12), 1511–1516. doi: 10.1093/sleep/28.12.1511
Pourabdian, S., Lotfi, S., Yazdanirad, S., Golshiri, P., & Hassanzadeh, A. (2020). Evaluation of the effect of fatigue on the coping behavior of international truck drivers. *BMC Psychology*. 8(1), 70(2020). doi: 10.1186/s40359-020-00440-2
Sirohey, S., & Rosenfeld, A. (2019). Eye detection in a face image using linear and nonlinear filters*. Pattern Recognition*. *34*(7), 1367-1391. doi: 10.1016/S0031-3203(00)00082-0
Wolkoff, P., Nojgaard, J.K., Troiano, P., & Piccoli, B. (2005). Eye complaints in the office environment: precorneal tear film integrity influenced by eye blinking efficiency. *Occupational and Environmental Medicine*, *62*(1), 4-12. doi: 10.1136/oem.2004.016030