**Blinking Eyes Detection to Monitor Drowsy Drivers Due to Fatigue Using MATLAB Cascade Object Detector**
\*\*Double blind review, please do not include authors information in this version \*\*
Received Date: \*date
Accepted Date: \*date
Published Date: \*date
**HIGHLIGHTS**
- Road accidents can be avoided by detecting fatigue and drowsy faces by looking at the eye area.
- Blinking eye detection can be used to monitor driver’s fitness level.
- Viola-Jones algorithm using MATLAB cascade object detector was used to detect the presence of blinking eyes.
- The results of the Classification Table derived from the algorithm successfully provided output to detect the presence of blinking eyes for the purpose of monitoring drowsy drivers due to fatigue.
ABSTRACT
*Road accidents are incidents that should be avoided. One contributing factor to the occurrence of accidents is drowsiness while driving due to fatigue. This study suggested the detection of fatigue and drowsy faces by looking at the eye area. Blinking eye detection can be used to monitor a driver’s fitness level. This is because when the driver's fitness level is at a minimum, the eyes will blink. Hence, the Viola-Jones algorithm using MATLAB cascade object detector was used to detect the presence of blinking eyes. The method used in this study demonstrates the use of the Viola-Jones algorithm with Cascade Object Detector technique. In addition, MATLAB software was used as a method to apply the algorithm on the blinking eye detection system. There are three phases to the algorithm, there are selection of Haar-like characteristics, Integrating a picture into the whole, and Classifiers in a cascading fashion. The results obtained will be calculated based on the accuracy value of the blinking eye detection. The formula used to calculate the accuracy is based on Classification Table. Results derived from Classification Table shows AUC values obtained exceed the values of 0.5. This concluded that the use of Viola-Jones algorithm through MATLAB cascade object detector has successfully provided output to be used to detect the presence of blinking eyes for the purpose of monitoring drowsy drivers due to fatigue.*
*Keywords: Blinking Eyes Detection, Cascade Object Detector, MATLAB, Classification Table*
# INTRODUCTION
Road accidents are incidents that should be avoided. One contributing factor to the occurrence of accidents is drowsiness while driving due to fatigue. Drowsiness can be described as a condition of people who are almost asleep and difficult to stay awake due to factors such as body fatigue or due to lack of sleep (Bereshpolova, Y., Stoelzel, C.R., Zhuang, J., Amitai, Y., Alonso, J.M., Swadlow, H.A., 2011). Drowsy situations can trigger danger especially when performing tasks that require constant concentration, such as driving a vehicle over long distances. Consequences of being in a fatigue state while driving and experiencing a drowsy state, it is feared may result in accidents (Khan, N., & Panchal, D., 2021). Therefore, a method to help detect drowsiness while driving is needed to prevent accidents.
This study suggested the detection of fatigue and drowsy faces by looking at the eye area. Changes in eye condition are seen as appropriate for estimating the existence of a person’s fatigue and drowsiness. Wolkoff, P., Nojgaard, J. K., Troiano, P., & Piccoli, B., (2005), noted that the time between eye blinks will increase when the eye has worked under stressful conditions and within a certain period. This situation exists as driving is a challenging activity and needs to be done with full concentration. Therefore, this study proposes to detect blinking eyes as a method of assessing the existence of fatigue and drowsiness experienced by a driver. Hence, the Viola-Jones algorithm using MATLAB cascade object detector will be used to detect the presence of blinking eyes.
# LITERATURE REVIEW
Driving over long distances can be dangerous if safety measures are ignored. One of the important safety measures is to keep a driver fresh and energetic. However, fatigue and drowsiness can change the handling style of each driver and increase the risk of accidents to occur. Philip, P., Sagaspe, P., Taillard, J., Valtat, C., Moore, N., Akerstedt, T., Charles, A., & Bioulac, B. (2005), stated in their study that fatigue and drowsiness will significantly affect the current condition and efficiency of the driver. Further on a study by Pourabdian, S., Lotfi, S., Yazdanirad, S., Golshiri, P., & Hassanzadeh, A., (2020), has tested 400 drivers and found that fatigue and unsafe driving habits are closely linked. Studies by Kocalevent, R. D., Hinz, A., Brähler, E., & Klapp, B. F. (2011), also show that fatigue is strongly correlated with perceived stress. This makes the relationship between fatigue and tension tend to exist in a two-way form.
Blinking eye detection can be used to monitor a driver’s fitness level. This is because when the driver's fitness level is at a minimum, the eyes will blink. Thus, the blinking eye is a natural protective system that protects the eyes from environmental exposure. For example, when a driver feels fatigue, the driver's fitness level will directly decrease. This causes the driver’s metabolic rate and mental ability to also decrease. One of the natural reactions that exists is that the eyes will blink due to feeling fatigue and drowsy. Therefore, the condition of spontaneous blinking eyes is considered as an accurate indication to diagnose the existence of fatigue and drowsiness due to diligent driving activities.
Eye blinking detector is a system that detects the blinking eye due to fatigue and drowsiness. Peng, K., Chen, L., Ruan, S., & Kukharev, G., (2015) have presented an eye detection algorithm for gray intensity images. In the study the authors have used an algorithm that stores the point template and matches the existing template. The results obtained give a value of 95.2% accuracy. However, the recommended technique was found not to work well if the detection was performed for spectacle wearers although the average implementation time of the proposed algorithm had shown that the approach performed was quite efficient. Sirohey, S., & Rosenfeld, A., (2019), also have conducted a study related to face detection based on eye images. The study used gabor filter or also known as linear filter, and wedge-shaped or nonlinear shape filter for feature extraction. Based on the filters used, a detection rate of 90% was achieved without false alarms.
# METHODOLOGY
The method used in this study demonstrates the use of the Viola-Jones algorithm with Cascade Object Detector technique. According to Chopparapu, S., Seventline, J., & Beatrice, G., (2020), the efficiency of the Viola-Janes algorithm can be improved by producing inseparable images, thus help reduce errors in analyzing the captured video. This algorithm uses the concept of a rectangular feature that involves the number of pixels in a rectangular area.
In addition, MATLAB software was used as a method to apply the algorithm on the blinking eye detection system. There are three phases to the algorithm, there are selection of Haar-like characteristics, Integrating a picture into the whole, and Classifiers in a cascading fashion. MATLAB has the vision to do this. The computer toolbox contains the CascadeObjectDetector system object, which uses the viola-jones method to recognise faces and objects. There are numerous pre-trained classifiers in vision.CascadeObjectDetector that can recognise frontal faces, profile faces, noses and eyes, as well as the upper body. This code is to define the face Detector object
> faceDetector = vision.CascadeObjectDetector
Using one of the attributes from vision as a second step, MergeThreshold is a CascadeObjectDetector. This data will be utilised to improve detection and accuracy of threats.
> faceDetector.MergeThreshold = 4
As a result, the default value is four. Depending on the amount of precision necessary, MergeThreshold can be changed. In other words, face identification accuracy is higher at 3 than it is at 8, and vice versa.
The experimental methods performed in this study are referenced based on the study of Chopparapu, S., et. al., (2020). The study explains the recommended method for measuring the time a person closes the eyes and if the eyes are closed for longer than normal blinking time, the person tends to fall asleep. Based on studies on human eyelids, it has been known that the average human blinking takes around 202.24ms while the drowsy blinking period takes 258.57ms (Arceda, V. E. M., Nina, J. P. C., & Fabian, K. M. F., 2020).
The results obtained will be calculated based on the accuracy value of the blinking eye detection. The formula used to calculate the accuracy is based on the following Classification Table as Table 1:
Table 1: Classification Table
| | Fail Observe | Success Observe | |
| --------------- | ------------ | --------------- | ----- |
| Fail Predict | **TN** | **FN** | PN |
| Success Predict | **FP** | **TP** | PP |
| | ON | OP | Total |
where;
PP = predicted positive = TP + FP, PN = predicted negative = FN + TN,
OP = observed positive = TP + FN, ON = observed negative = FP + TN and
Total = the total sample size = TP + FP + FN + TN.
and;
TP = True Positive = correctly classified results obtained to be positive.
FP = False Positive = incorrectly obtained result is classified as positive.
TN = True Negative = results obtained that are correctly classified become negative.
FN = False Negative = results obtained that are incorrectly classified as negative.
# RESULTS
The experiments were performed using hardware such as laptops and webcams. A laptop is required to build the programming code for the system, while the webcam required is a High Definition (HD) type used as a tool to capture a picture of the driver’s face. In addition to hardware, software is also required such as MATLAB, Video Cutter, and Third Format Factory Software.
The experiment was conducted on 29 samples of drivers. Each driver was photographed while given a driving assignment. The captured driver's face images were stored in the form of frames that been divided into two, first the driver's with a fatigue face situation and the driver's with non-fatigue face situation.
**Accuracy Values for Detecting Blinking Eyes**
The data obtained were type-in into two different tables, namely Table 2 and Table 3. Table 2 will display the driver's face data for Fatigue situations, while Table 3 display the driver's facial data for non-Fatigue situations.
Table 2: Driver face data during Fatigue situations
| | Observed | Cumulative | | | | | |
| ------ | --------------------- | ------------------------ | --------------------- | ------------------------ | ------ | ------ | ------ |
| Frames | Detectfail | DetectSuccess | Detectfail | DetectSuccess | FPR | TPR | AUC |
| | | | 0 | 0 | 1.0000 | 1.0000 | 0.0000 |
| 90 | 0 | 30 | 0 | 30 | 1.0000 | 0.9843 | 0.0204 |
| 90 | 6 | 37 | 6 | 67 | 0.9793 | 0.9649 | 0.0166 |
| 102 | 5 | 53 | 11 | 120 | 0.9621 | 0.9371 | 0.0259 |
| 108 | 8 | 47 | 19 | 167 | 0.9345 | 0.9124 | 0.0220 |
| 109 | 7 | 55 | 26 | 222 | 0.9103 | 0.8836 | 0.0274 |
| 112 | 9 | 77 | 35 | 299 | 0.8793 | 0.8432 | 0.0087 |
| 119 | 3 | 56 | 38 | 355 | 0.8690 | 0.8138 | 0.0225 |
| 120 | 8 | 65 | 46 | 420 | 0.8414 | 0.7798 | 0.0269 |
| 124 | 10 | 57 | 56 | 477 | 0.8069 | 0.7499 | 0.0181 |
| 128 | 7 | 43 | 63 | 520 | 0.7828 | 0.7273 | 0.0301 |
| 133 | 12 | 78 | 75 | 598 | 0.7414 | 0.6864 | 0.0284 |
| 137 | 12 | 79 | 87 | 677 | 0.7000 | 0.6450 | 0.0245 |
| 139 | 11 | 62 | 98 | 739 | 0.6621 | 0.6125 | 0.0253 |
| 142 | 12 | 73 | 110 | 812 | 0.6207 | 0.5742 | 0.0436 |
| 144 | 22 | 85 | 132 | 897 | 0.5448 | 0.5296 | 0.0384 |
| 150 | 21 | 86 | 153 | 983 | 0.4724 | 0.4845 | 0.0150 |
| 152 | 9 | 73 | 162 | 1056 | 0.4414 | 0.4463 | 0.0200 |
| 153 | 13 | 72 | 175 | 1128 | 0.3966 | 0.4085 | 0.0085 |
| 153 | 6 | 59 | 181 | 1187 | 0.3759 | 0.3776 | 0.0078 |
| 154 | 6 | 79 | 187 | 1266 | 0.3552 | 0.3361 | 0.0070 |
| 154 | 6 | 37 | 193 | 1303 | 0.3345 | 0.3167 | 0.0229 |
| 158 | 21 | 95 | 214 | 1398 | 0.2621 | 0.2669 | 0.0110 |
| 159 | 12 | 64 | 226 | 1462 | 0.2207 | 0.2334 | 0.0048 |
| 160 | 6 | 40 | 232 | 1502 | 0.2000 | 0.2124 | 0.0081 |
| 160 | 11 | 90 | 243 | 1592 | 0.1621 | 0.1652 | 0.0085 |
| 160 | 15 | 82 | 258 | 1674 | 0.1103 | 0.1222 | 0.0042 |
| 162 | 10 | 63 | 268 | 1737 | 0.0759 | 0.0891 | 0.0015 |
| 163 | 5 | 77 | 273 | 1814 | 0.0586 | 0.0488 | 0.0029 |
| 166 | 17 | 93 | 290 | 1907 | 0.0000 | 0.0000 | 0.0000 |
| | 290 | 1907 | | | | | 0.5010 |
Table 2 displays the data related to the driver's face with the fatigue situation. There are five main columns that need to be observed in Table 2 namely Detectfail and Detectsuccess which is generally a breakdown of the face data captured from observation. In addition, the columns that are important to note are such as False Positive Rate (FPR), True Positive Rate (TPR), and Area Under the Curve (AUC).
Meanwhile, out of 29 driver face samples, the 10th row with 128 frames was selected as the cut-off. Based on the cut-off, the values of FPR, TPR, and Accuracy were obtained as 0.782759, 0.72732, and 0.659991 respectively. This illustration of the relationship of TPR versus FPR is then plotted as a Receiver-Operating Characteristic (ROC) curve as shown in Figure 1.
\[CHART\]
Figure 1: ROC curve for fatigue situation - TPR vs FPR
Figure 1 plots the relationship between TPR and FPR. The curve that exists is known as the ROC curve. Based on the observation of the shape of the curve, it is close to the shape of a straight line directly proportionate from the value of 0 to 1, with only a few insignificant bounces. This situation shows the ROC curve created for this fatigue situation as not good since the value below the curve is far off to reach a higher value approaching 1.0.
Table 3: Driver face data during non-fatigue situations
| | Observed | Cumulative | | | | | |
| ------ | --------------------- | ------------------------ | --------------------- | ------------------------ | ------ | ------ | ------ |
| Frames | Detectfail | DetectSuccess | Detectfail | DetectSuccess | FPR | TPR | AUC |
| | | | 0 | 0 | 1.0000 | 1.0000 | 0.0391 |
| 90 | 7 | 54 | 7 | 54 | 0.9609 | 0.9655 | 0.0108 |
| 90 | 2 | 45 | 9 | 99 | 0.9497 | 0.9367 | 0.0628 |
| 102 | 12 | 34 | 21 | 133 | 0.8827 | 0.9150 | 0.0613 |
| 108 | 12 | 41 | 33 | 174 | 0.8156 | 0.8887 | 0.0695 |
| 109 | 14 | 32 | 47 | 206 | 0.7374 | 0.8683 | 0.0146 |
| 112 | 3 | 23 | 50 | 229 | 0.7207 | 0.8536 | 0.0095 |
| 119 | 2 | 57 | 52 | 286 | 0.7095 | 0.8171 | 0.0137 |
| 120 | 3 | 39 | 55 | 325 | 0.6927 | 0.7922 | 0.0000 |
| 124 | 0 | 50 | 55 | 375 | 0.6927 | 0.7602 | 0.0467 |
| 128 | 11 | 67 | 66 | 442 | 0.6313 | 0.7174 | 0.0281 |
| 133 | 7 | 35 | 73 | 477 | 0.5922 | 0.6950 | 0.0272 |
| 137 | 7 | 37 | 80 | 514 | 0.5531 | 0.6714 | 0.0075 |
| 139 | 2 | 63 | 82 | 577 | 0.5419 | 0.6311 | 0.0423 |
| 142 | 12 | 44 | 94 | 621 | 0.4749 | 0.6029 | 0.0000 |
| 144 | 0 | 37 | 94 | 658 | 0.4749 | 0.5793 | 0.0129 |
| 150 | 4 | 38 | 98 | 696 | 0.4525 | 0.5550 | 0.0341 |
| 152 | 11 | 55 | 109 | 751 | 0.3911 | 0.5198 | 0.0116 |
| 153 | 4 | 61 | 113 | 812 | 0.3687 | 0.4808 | 0.0081 |
| 153 | 3 | 82 | 116 | 894 | 0.3520 | 0.4284 | 0.0215 |
| 154 | 9 | 61 | 125 | 955 | 0.3017 | 0.3894 | 0.0196 |
| 154 | 9 | 93 | 134 | 1048 | 0.2514 | 0.3299 | 0.0000 |
| 158 | 0 | 42 | 134 | 1090 | 0.2514 | 0.3031 | 0.0254 |
| 159 | 15 | 65 | 149 | 1155 | 0.1676 | 0.2615 | 0.0044 |
| 160 | 3 | 102 | 152 | 1257 | 0.1508 | 0.1963 | 0.0110 |
| 160 | 10 | 42 | 162 | 1299 | 0.0950 | 0.1694 | 0.0038 |
| 160 | 4 | 59 | 166 | 1358 | 0.0726 | 0.1317 | 0.0000 |
| 162 | 0 | 87 | 166 | 1445 | 0.0726 | 0.0761 | 0.0013 |
| 163 | 3 | 77 | 169 | 1522 | 0.0559 | 0.0269 | 0.0015 |
| 166 | 10 | 42 | 179 | 1564 | 0.0000 | 0.0000 | 0.0000 |
| | 179 | 1564 | | | | | 0.5882 |
Table 3 displays the data related to the driver's face with the non-fatigue situation. In this Table 3, the values are recorded for TPR is 0.71739, FPR value is equivalent to 0.63128, and Accuracy value is 0.68158. The illustration of the relationship of TPR versus FPR for Table 3 is plotted as shown in Figure 2.
\[CHART\]
Figure 2: ROC curve for non-fatigue situation - TPR vs FPR
Based on the observations in Figure 2, the resulting curve is slightly different from that shown in Figure 1. This is because the shape in Figure 2 has a higher upward curve, which shows that the AUC value on the ROC curve for TPR vs FPR in Figure 2 is larger than the ROC curve for TPR vs FPR in Figure 1. This means that the data in Table 3 are better than the data in Table 2.
**CONCLUSION**
This study is about detecting blinking eyes. The purpose of detecting blinking eyes is to recognize the faces of drivers who drive in a fatigue and drowsy state. Based on the results of the study, two things can be concluded with the first being based on Figure 3.
\[CHART\]
Figure 3: Comparison of ROC curve between fatigue and non-fatigue situations - TPR vs FPR
Through Figure 3, a direct comparison between the two ROC curves based on fatigue situation and non-fatigue situation was performed. Based on the comparison, it is found that the ROC curve for non-fatigue situations is more convex upwards than the ROC curve for fatigue situations. This means that the AUC value for non-fatigue situations is larger, and it directly indicates that classification through non-fatigue situations is better. However, this does not mean that the classification based on the fatigue situation is not good, because the AUC value for the ROC curve resulting from the fatigue situation still exceeds the value of 0.5.
Proving that the AUC values for both ROC curves exceed the values of 0.5, Table 2 and Table 3 will be referred. If observed in both tables, the total AUC values for faces with fatigue situations are 0.5010 and the total AUC values for faces with non-fatigue situations are 0.5882. This indicates both classifications are acceptable.
The conclusion that can be made is that the use of Viola-Jones algorithm through MATLAB cascade object detector has successfully provided output to be used to detect the presence of blinking eyes for the purpose of monitoring drowsy drivers due to fatigue.
**REFERENCES**
Arceda, V. E. M., Nina, J. P. C., & Fabian, K. M. F., (2020), A Survey on Drowsiness Detection Techniques. *CEUR Workshop Proceedings*, 2747, 152–161.
Bereshpolova, Y., Stoelzel, C.R., Zhuang, J., Amitai, Y., Alonso, J.M., Swadlow, H.A., (2011), "Getting Drowsy? Alert/Nonalert Transitions and Visual Thalamocortical Network Dynamics". *J Neurosci*. 48, 17480-17487
Chopparapu, S., Seventline, J., & Beatrice, G., (2020), Gui for Object Detection Using Voila Method in Matlab. *International Journal of Electrical Engineering and Technology*, 11(4), 2020, 169-174, Available at SSRN:
Khan, N., & Panchal, D., (2021), Fatigue Detection System using Image Processing on Video Sequences. Retrieved July 17, 2021, from
Kocalevent, R. D., Hinz, A., Brähler, E., & Klapp, B. F. (2011), Determinants of Fatigue and Stress. *BMC research notes*, 4, 238.
Peng, K., Chen, L., Ruan, S., & Kukharev, G., (2015), A Robust Algorithm for Eye Detection on Gray Intensity Face without Spectacles. *Journal of Computer Science and Technology* - JCST. 5.
Philip, P., Sagaspe, P., Taillard, J., Valtat, C., Moore, N., Akerstedt, T., Charles, A., & Bioulac, B. (2005). Fatigue, Sleepiness, and Performance in Simulated Versus Real Driving Conditions. Sleep, 28(12), 1511–1516.
Pourabdian, S., Lotfi, S., Yazdanirad, S., Golshiri, P., & Hassanzadeh, A., (2020), Evaluation of The Effect of Fatigue on The Coping Behavior of International Truck Drivers. *BMC Psychology*. 8. 70. 10.1186/s40359-020-00440-2.
Sirohey, S., & Rosenfeld, A., (2019), Eye Detection in A Face Image Using Linear and Nonlinear Filters*. Pattern Recognition*. 34. 1367-1391. 10.1016/S0031-3203(00)00082-0.
Wolkoff, P., Nøjgaard, J. K., Troiano, P., & Piccoli, B., (2005), Eye Complaints in The Office Environment: Precorneal Tear Film Integrity Influenced by Eye Blinking Efficiency. *Occupational and Environmental Medicine* 2005, 62, 4-12.