Pretraining helped most on the hard cases
Fine-tuning a pretrained model cut errors by about a third with glasses (3.9% to 2.7%) and strong reflections (3.8% to 2.4%), but barely changed them without glasses (3.0% to 2.9%).
Open or closed? Eye-state detection with PyTorch
A computer vision model that tells whether an eye is open or closed from a small infrared image, tested on six people it never saw during training.
- Data
- MRL Eye Dataset: 84,898 infrared eye images from 37 people
- Models
- Small CNN from scratch vs. fine-tuned MobileNetV3-Small
- Tools
- Python, PyTorch, torchvision, scikit-learn, Pandas, ONNX
- Result
- 97.2% accuracy on six new people, 98.5% of closed eyes caught
Problem
Drowsiness and impairment checks often start with one simple question: is the eye open or closed right now? A phone or car camera has to answer it quickly, for people it has never seen, with glasses, glare and bad lighting in the way.
Approach
- Split the data by person, not by image: 26 people for training, 5 for validation and 6 for testing. Photos of the same person look alike, so a random split would let the model memorize faces and inflate the score.
- Ran two experiments on the same split: a small CNN trained from scratch as a baseline, and MobileNetV3-Small, an ImageNet model built for phones, fine-tuned on the eye images.
- Watched validation loss every epoch and kept the best checkpoint, stopping early once it stopped improving, to catch overfitting.
- Measured accuracy, precision, recall, F1 and ROC AUC on the test people, then broke errors down by glasses, reflections, lighting and camera.
- Exported the models to ONNX and timed single-image inference on a laptop CPU as a rough check on phone readiness.
The data
Results on the six test people
| Metric (closed eye = positive) | Small CNN, from scratch | MobileNetV3-Small, fine-tuned |
|---|---|---|
| Accuracy | 96.7% | 97.2% |
| Recall (closed eyes caught) | 97.7% | 98.5% |
| Precision (closed calls that were right) | 95.8% | 96.0% |
| F1 | 96.7% | 97.2% |
| ROC AUC | 0.995 | 0.995 |
| Parameters | 0.29M | 1.52M |
| ONNX file size | 1.2 MB | 6.1 MB |
| One image on a laptop CPU | 1.6 ms | 26 ms |
Where the model goes wrong
The model leans toward "closed"
4.0% of open eyes were called closed, but only 1.5% of closed eyes were called open. For a drowsiness check that means more false alarms than misses. Moving the decision threshold trades one for the other, and the right balance depends on the product.
Some mistakes are in the labels
The most confident mistakes often look mislabelled or half closed. Relabelling a sample of the disagreements would show how much of the last 3% is model error and how much is label noise.
Bigger was not clearly better
The fine-tuned model was 0.5 points more accurate, but 5 times larger and much slower on CPU. On a phone, that trade-off needs measuring on the device with a mobile runtime before choosing a model.
Limitations and next steps
Six test people is a small group, and it matters: the five validation people were easier (99.0% accuracy) than the six test people (97.2%). The per-condition numbers rest on only a few people each and would move with a different split. Repeating the experiment with several person-level splits (grouped cross-validation) would give a more honest range.
These are still, pre-cropped eye images. A real product has to find the eye in a video frame first and then follow it over time, where blinks and eye movements become a time series. Adding temporal smoothing across frames is the natural next step, and the ONNX export is a starting point for testing on an actual phone.
Data: MRL Eye Dataset, Media Research Lab, VSB Technical University of Ostrava.