Digit Recognizer
Kaggle's handwritten-digit competition: KNN, a linear SVM and a Random Forest compared, then a CNN.
One Jupyter notebook: three scikit-learn models first, then a convolutional network in TensorFlow and Keras, then 28,000 answers sent to Kaggle.
0.99532Kaggle score
4,200 validation digits, 28 wrong
The digits are handwritten, 28 × 28 pixels of grey, and each is a 0 to 9. The notebook held back 10% of Kaggle's training digits, 4,200 of them, as a validation split, and checked all four models on it. Each mark here is one of those digits, stacked by what it really is.
Wrong, out of the same 4,200
- KNN, k = 3about 140
- Linear SVMabout 300
- Random Forest, 100 treesabout 165
- The CNN28
The three baselines' scores, about 0.967, 0.929 and 0.961, are read off the notebook's bar chart. It never printed them, or which digits they missed.
The 28 standing apart are the ones the CNN got wrong. Point at one, or at a digit.
Folded into a confusion matrix
The same 4,200 marks. Each row is a true digit and each column is what the CNN read, so the diagonal holds every digit it read right. Point at or tap a row, a column or a cell.
4,172 of the 4,200 sit on the diagonal. The 28 off it fall in 22 cells; the fullest holds 3, a 5 read as a 6.
The 4,200 are the split training watched to decide when to stop, so 99.33% on them isn't a clean test. Kaggle's 0.99532 is the clean figure.
Totals: 408 true 0s, 407 read as 0; 471 true 1s, 469 read as 1; 420 true 2s, 419 read as 2; 506 true 3s, 505 read as 3; 397 true 4s, 397 read as 4; 339 true 5s, 336 read as 5; 402 true 6s, 405 read as 6; 438 true 7s, 439 read as 7; 403 true 8s, 407 read as 8; 416 true 9s, 416 read as 9.
The network
A convolutional network slides small 3 × 3 filters across the grid. This one has four such layers of 64 filters, in two pairs. After each pair the grid is halved, from 28 to 14 to 7, and a quarter of its values are dropped at random while it trains. Then come 512 units, half of them dropped, and ten outputs, one per digit.
No model file was saved, so nothing on this page runs the network. The marks above are drawn from its confusion matrix.
- 28 × 28the digit
- 28 × 28 × 64
- 14 × 14 × 64
- 7 × 7 × 64
- 512
- 10
Every batch, shaken a little
The network was trained in batches of 256 digits. Before each batch went in, every digit in it was turned up to 10° either way, zoomed up to 10% and shifted up to 10% across and up or down, and never flipped. The amounts were random for every digit, every time, so the network kept meeting the same digits slightly differently.
The page can't read your digit: it only shows what a digit looked like by the time the network saw it.
Arrow keys move the pen. Space puts it down or lifts it.
Examples, drawn for this page
Sixteen copies, each with its own random turn, zoom and shift. Point at one, or tap it, for its numbers.
Stop when it stops improving
After every epoch, training checked its accuracy on the 4,200 validation digits. Once 10 epochs had gone by without beating the best, it stopped and went back to the best epoch's weights. It was allowed up to 100 epochs; it stopped at 29 and kept 19.
Each epoch's score wasn't saved, so the strip has no curve: only where training peaked and where it stopped.
The optimizer, Adam, started at a learning rate of 0.001. A second rule watched the same score: each time 3 epochs passed without improvement, the rate halved, down to no less than 0.00001. When that happened wasn't recorded, so it isn't placed on the strip.
Epoch 29
10 epochs without beating 19. Training stops and goes back to epoch 19's weights.
Drag along the epochs, or use the arrow keys.
Kaggle's test digits
Then the CNN labeled the competition's 28,000 test digits, whose answers Kaggle keeps hidden. On Kaggle's hidden test digits it scored 0.99532, the only figure here from digits that played no part in training.
The honest limits
99.33% is on the split training watched to decide when to stop, so it isn't a clean test. Kaggle's 0.99532, on its hidden test digits, is the clean figure.
The baselines' scores are read from the notebook's chart, so they're approximate, and the notebook never recorded which digits they got wrong.
28 errors from one run, with no random seed set, are a few errors, scattered, not a reliable ranking of which digits get confused.
No model file was saved, so nothing on this page runs the real network or reads a digit you draw.
- Stack
- Python, TensorFlow and Keras, scikit-learn; pandas, NumPy and matplotlib for the rest.