Semi Supervised YOLOv2

Summary

This project was carried out during an internship at IRIT, Toulouse, with Axel Carlier as tutor.

YOLOv2

The fisrt part of this project was about obtaining a functional and trainable model for image detection based on YOLOv2 ¹. This was done by extracting and adapting code snippets from an existing implementation to a Jupyter Notebook.

Results

I obtained, on the WildLife dataset, the following quantitative results :

And qualitative results :

Semi-supervised Learning for Image Detection

The second part of this project was the implementation of a semi-supervised training for object detection ² on this functional model. This affected the loss of the model ; initially YOLOv2 has the following loss :

$$\begin{array}{rl}{l}_{\text{yolo}}& ={\lambda }_{\text{coord}}\sum _{i=0}^{S^2}\sum _{j=0}^B{\mathbb{1}}_{ij}^{\text{obj}}[{(x_{ij}-{\hat{x}}_{ij})}^2+{(y_{ij}-{\hat{y}}_{ij})}^2]\\ & +{\lambda }_{\text{coord}}\sum _{i=0}^{S^2}\sum _{j=0}^B{\mathbb{1}}_{ij}^{\text{obj}}[{({\sqrt{w}}_{ij}-\sqrt{{\hat{w}}_{ij}})}^2+{({\sqrt{h}}_{ij}-\sqrt{{\hat{h}}_{ij}})}^2]\\ & +{\lambda }_{\text{obj}}\sum _{i=0}^{S^2}\sum _{j=0}^B{\mathbb{1}}_{ij}^{\text{obj}}{(C_{ij}-{\hat{C}}_{ij})}^2\\ & +{\lambda }_{\text{noobj}}\sum _{i=0}^{S^2}\sum _{j=0}^B{\mathbb{1}}_{ij}^{\text{noobj}}{(C_{ij}-{\hat{C}}_{ij})}^2\\ & +{\lambda }_{\text{class}}\sum _{i=0}^{S^2}\sum _{j=0}^B{\mathbb{1}}_{ij}^{\text{obj}}\sum _{c\in \text{classes}}{(p_{ij}(c)-{\hat{p}}_{ij}(c))}^2\end{array}$$ where ${\mathbb{1}}_{ij}^{\text{noobj}}=1-{\mathbb{1}}_{ij}^{\text{obj}}\mathrm{\forall }(i,j)\in S^2\times B$

But to in order take into account the pseudo-labels generated by the teacher, I modified the loss as follows :

$$\begin{array}{rl}{l}_{\text{yolo\_ssl}}& =\sum _{i=0}^{S^2}\sum _{j=0}^B({\mathbb{1}}_{ij}^s{\lambda }_s+{\mathbb{1}}_{ij}^u{\lambda }_u)l_{ij}\\ l_{ij}=& {\lambda }_{\text{coord}}{\mathbb{1}}_{ij}^{\text{obj}}[{(x_{ij}-{\hat{x}}_{ij})}^2+{(y_{ij}-{\hat{y}}_{ij})}^2]\\ +& {\lambda }_{\text{coord}}{\mathbb{1}}_{ij}^{\text{obj}}[{({\sqrt{w}}_{ij}-\sqrt{{\hat{w}}_{ij}})}^2+{({\sqrt{h}}_{ij}-\sqrt{{\hat{h}}_{ij}})}^2]\\ +& [{\lambda }_{\text{obj}}{\mathbb{1}}_{ij}^{\text{obj}}(1-{\mathbb{1}}_{ij}^{\text{noobj}})+{\lambda }_{\text{noobj}}{\mathbb{1}}_{ij}^{\text{noobj}}]{(C_{ij}-{\hat{C}}_{ij})}^2\\ +& {\lambda }_{\text{class}}{\mathbb{1}}_{ij}^{\text{obj}}\sum _{c\in \text{classes}}{(p_{ij}(c)-{\hat{p}}_{ij}(c))}^2\end{array}$$ where ${\mathbb{1}}_{ij}^{\text{obj}}=1$ for all labeled and unlabeled boxes that contain an object, and ${\mathbb{1}}_{ij}^{\text{noobj}}=1$ for all labeled and unlabeled boxes that does not contain an object and all unlabeled boxes for which the teacher has a confidence below a certain threshold.

Results

I obtained the following results :

Showing proof of concept that the student performs better than the teacher, but also that using the student to train a new student (Student² here) does not seem to make any improvement.