Koi identifier Pt.2: LLM training
- David Turner
- Mar 22
- 11 min read
Updated: Aug 10
Recap of the project so far
In Part 1 of this project, I'd used Claude to vibe-code an app that identified koi varieties and individual fish sightings from uploaded photographs. However it became clear that there were limitations to using a general purpose LLM. Analysis of the same photograph twice would result in different results. So the answer was to train a model specifically for the task, and see if that performed any better.
Summary
Purpose: Training a model specifically on images of koi
Stack: Google Colab with Nvidia T4, Imagenet ResNet50, TensorFlow
Results: 20.8% Val Accuracy, model overfitting
Outcome: Not satisfactory fro use; need a much larger and varied data set to train on if I ever wanted to take this to Prod.
GitHub: koi-identifier
Requirements for training the learning model
The basic requirements for this phase of the project were to train a specialised deep-learning model that:
Recognises koi breed characteristics
Works locally (no API costs after training)
Learns from my specific photos
Improves with feedback
Gives consistent results every time
Design approach for visual recognition model training
The design concept here centred around building a Convolutional Neural Network (CNN) trained on a set of labeled koi photos from my portfolio.
In simplistic terms it would work as follows:
Koi photo
↓
Pre-trained model (ImageNet features)
↓
Fine-tuning layer (learns breed-specific patterns)
↓
Classification layer (outputs breed probability)
↓
Result: 98% confident → e.g. "Kohaku"
In the context of my project, there were two design options to consider.
Option A: Transfer Learning
Use pre-trained model (trained on millions of images)
Fine-tune on my koi data
Requires: 50-200 labeled photos per breed
Time: 1-2 weeks
Accuracy: 90-95%
Difficulty: Intermediate
Option B: Full Training
Train from scratch on koi images
Requires: 1000+ labeled photos per breed
Time: 4-8 weeks
Accuracy: 96-99%
Difficulty: Advanced
The choice was pretty obvious here - Option A. No way I was going to pull together 1000+ pictures across 12 varieties (plus another 1000 or so 'unclassified' examples).
The datasets
Training a model means you need training data. And for me this meant lots of pictures of koi. A minimum of 50 pictures per variety was required to reach a minimum-acceptable level of accuracy, with 200-300 pictures delivering better results.
There are a couple of ways to do this, and in reality they're all very tedious. However I submitted to the sunken-cost fallacy and decided that since I'd started, I might as well stick out the boredom and complete the task.
Method 1: Run through all my photos, sort them into varieties, and drop them into corresponding folders. I discounted this as (i) the whole purpose of building this project was because I didn't know what varieties they were and (ii) I may have had gaps where photo's didn't exist of certain varieties.
Method 2: Suggested by Claude, was to visit inaturalist.org and search each variety of koi, then download the license-free pictures. It even offered to write a script for me to automate that. One flaw in the plan though is that iNaturalist records sightings of wild animals, not captive ones like koi. So this method was discounted too.
Method 3: Google Images - search for each variety with the correct license settings configured in the search tools. Tedious, but viable and legal.
Method 4: Search forums, community pages and YouTube thumbnails for images. Request permission for use of the pictures for training. Frankly I don't have time to write to hundreds of people asking for permission to use their pics, and the YouTube option didn't sound particularly ethical, so this was out too.
That left Method 3 as the only viable and legally compliant option, and so I commenced with one of the most boring few hours of my life. Not something I want to repeat anytime soon, and not something I'd recommend where your sanity is concerned, or if you'd like to avoid a case of RSI.
BTW: If you ever wondered why review sites ask you to name the things in the pictures you've uploaded, it's because they also don't want to go through this the same tedium (albeit on a much larger scale) so have outsourced the labelling work to you. Enjoy.
Fast forward a couple of hours and I'd got my set of 600+ pics ready across 12 varieties. The experience made worse by listening to Liverpool get beaten by Brighton at the same time. Like I said, not an afternoon to remember.
A quite specific note here on koi as the subject matter; the variety of pictures available is really poor. Most tend to be identical in style - the fish in a collector's or breeder's blue artificial pond or bucket, photographed front-on or top-down. Same lighting, composition, angles, colours, and zero variety. These probably account for around 80-90% of all the pics out there. If I was doing such a project for real, I think the first step would be to take my camera out to Japan and visit every breeder, dealer and koi pond in the country. Probably a nice excuse for a trip, but not practical in the least.
Nevertheless I got them labelled by dropping each variety into it's own separate folder, and moved on.
The AI model training environment
There were a couple of options for how to execute the training:
Option 1: Google Colab
Free Google service, runs in browser, no installation needed.
Pros:
Free GPU access
No setup required
Very intuitive to use
Cons:
Session times out after 12 hours
Limited free GPU time
Option 2: Local Setup
Install locally on machine.
Pros:
Best for ongoing use.
Uses Python 3.8+
Cons:
NVIDIA GPU (recommended, as CPU will run much slower)
10GB free disk space
Since I don't have a spare GPU handy, and aren't prepared to give up 10GB that could be filled with more photos, it was clearly Option 1.
The AI model selected for training
After some back and forth with Claude, we settled on ResNet50 Transfer Learning, based upon the following rationale:
Pre-trained on ImageNet (1.2M images, 1000 classes)
Learns general visual features (edges, textures, shapes) applicable to any image classification task
Moderate size: faster training than ResNet101/152, more capacity than ResNet18
Well-documented, battle-tested in production systems
Strong performance on image classification benchmarks
Architecture wise this gave us:
Input (224×224×3)
↓
ResNet50 base (pre-trained, frozen)
↓
Global Average Pooling
↓
Dense(256, ReLU)
Dropout(0.5)
↓
Dense(128, ReLU)
Dropout(0.3)
↓
Dense(12, Softmax)
↓
Output (breed probabilities)
Transfer Learning Strategy:
Freeze early/middle layers (layer1, layer2) to preserve general features
Unfreeze final 20 layers for breed-specific learning
Separate optimiser for unfrozen layers (lower learning rate: 0.0001)
Worth mentioning TensorFlow at this juncture; chosen over PyTorch due its robustness with image formats, suitability for transfer learning (by virtue of Keras Sequential) and a generally better error tolerance. Keras itself selected over Functional API due to its simpler architecture and readability.
I've dropped a more comprehensive .MD file into the accompanying repo for those interested.
A note on metrics
Before jumping into the training and results. some explainers on the metrics we're looking for:
Accuracy (%): How many training images the model got right. Usually higher because the model has already seen them.
Validation accuracy (%): How many new, unseen images it classifies correctly. This is the real test.
Loss: How wrong the training predictions are. Lower is better. Should decrease as training progresses.
Speed (s/step): Seconds per batch processed. CPU is slow; GPU is ~35× faster.
In terms of accuracy versus validation accuracy, we can use the exam analogy to illustrate the relative importance:
Accuracy: You score 95% on practice tests you've already studied. The questions look familiar.
Validation accuracy: You score 70% on the real exam with questions you've never seen before. This tells you what you actually learned.
Validation accuracy is what matters. Accuracy can lie to you, whereas validation accuracy tells the truth about real-world performance.
Also worth mentioning epochs at this stage: being a relatively small dataset, a good guideline is to use 10 epochs. During training we'll adjust up or down based upon which of three different patterns we see in the results:
The right number of epochs is however many it takes for validation accuracy to stop improving. If we see over-fitting start to occur (when the model performs much better on training data than validation data - it's memorised the training set instead of learning patterns that work on new data.) then we need to stop early and address the data size, quality and model complexity.
In terms of end results, we're looking for final validation accuracy of:
Training: teaching the model to recognise images
Into Google Colab, which is incredibly intuitive for simple projects like this. Even faster and easier when using Claude to generate all the scripts too. First we set up the data path to the zip folder of training data on my desktop, and got that loaded:
A couple of minor syntax errors and bugs in the script, and we were making progress.
Now at this stage I have an argument with Claude due to an error we keep getting - Google Colab doesn't like .webp images. Right at the start of the Google Image downloading extravaganza, I explicitly asked Claude if .webp files were ok - it said yes, no problem, we can handle those along with .png and .jpg. I double checked this and it confirmed again 'no worries, in the worst case I'll just write a script to convert them, it's easy'.
So once we get the errors I remind Claude of this, and it does it's 'oh yeh I was wrong, I should have checked' routine. Not much help at this stage, so I told it to write the conversion script. Which didn't work. Over at least 5 attempts. It's best advice after the last failure was 'well we'll just have to use the smaller set of non-webp files then'.
So I'd wasted most of my afternoon collecting over 600 images and listening to another shambolic Liverpool performance, to be left with less than 200 usable files. Thanks Claude.
A number of other corrupted images were discounted from this set (due to attribute errors in the JPEGImageFile setting) further reducing the dataset. TensorFlow would skip through the corrupted images and use only the good ones.
The training loop itself consisted of 10 epochs: Epoch duration: 21–23 seconds per epoch (GPU)
Total training time: ~35 minutes for 10 epochs
Hardware: Google Colab T4 GPU
Learning rate schedule: StepLR (decay by 0.1 every 3 epochs, not applied in final run)
Results: did the models train successfully?
Undeterred and with the end in sight, I ran the training on those files regardless across 10 epochs. The final results came in as follows:
Dataset Summary
```
Breeds: 6 (asagi, bekko, kawarimono, kinginrin, matsuba, metallic)
Total Images: 243 (after automatic validation by TensorFlow)
Training Set: 195 images (80%)
Validation Set: 48 images (20%)
Training Batches: 7 (batch size 32)
Validation Batches: 2
Images per Breed: ~40 average
```Epoch-by-Epoch Performance
Epoch | Train Accuracy | Train Loss | Val Accuracy | Val Loss | Observation |
1. | 15.9% | 2.190 | 20.8% | 1.852 | Random chance initialization |
2. | 38.5% | 1.575 | 18.8% | 1.829 | Training improving, val noisy |
3. | 44.1% | 1.430 | 14.6% | 1.818 | Divergence beginning |
4. | 55.4% | 1.180 | 20.8% | 1.918 | Widening gap |
5. | 69.2% | 0.994 | 20.8% | 2.450 | Major divergence |
6. | 70.3% | 0.947 | 20.8% | 4.282 | Val loss exploding |
7. | 73.3% | 0.791 | 20.8% | 4.487 | Severe overfitting |
8. | 74.4% | 0.731 | 20.8% | 3.071 | Memorization complete |
9. | 80.5% | 0.602 | 20.8% | 2.745 | Model perfecting on training |
10. | 83.1% | 0.503 | 20.8% | 2.812 | Final: Extreme gap (62.3%) |
Key Metrics
Final Training Accuracy: 83.08%
Final Validation Accuracy: 20.83%
Accuracy Gap: 62.25 percentage points
Training Loss Trend: Smooth decrease (good)
Validation Loss Trend: Volatile, then increasing (bad)
Validation Loss Ratio: Starts at 1.85, ends at 2.81 (+52% increase)
What the Numbers Mean
Interpretation:
The model achieved 83% accuracy on images it had already seen during training
On images it had never seen (validation set), accuracy remained at 20.8%—essentially **random guessing for 5 classes** (20% = 1 in 5)
Validation loss increased by 52% while training loss decreased, classic overfitting signature
The 62-point gap is severe; healthy models show gap < 15 points
Root Cause: With only ~33 training images per breed, the model memorised specific patterns rather than learning generalisable breed characteristics.
Overfitting Analysis
What Happened Epoch-by-Epoch
Epochs 1–3: Learning Phase
Training accuracy climbs from 15.9% → 44.1%
Validation accuracy fluctuates (20.8% → 18.8% → 14.6%)
Model is learning some patterns, but validation is too small to show reliable progress
Validation batch size = only 8 images per breed (high variance)
Epochs 4–5: Divergence Begins
Training accuracy reaches 69.2%
Validation accuracy drops to 14.6%, then returns to 20.8%
Gap widens to ~50 points
Validation loss rises from 1.8 → 2.45 (35% increase)
Model starts memorising training set details
Epochs 6–10: Severe Memorisation
Training accuracy continues climbing (70.3% → 83.1%)
Validation accuracy stuck at 20.8%
Validation loss rises further (2.45 → 4.49 at peak)
Model has completely memorised training data
Zero transfer learning benefit
Why Early Stopping Would Have Helped:
Best validation accuracy: 20.83% (occurred at epoch 1–3)
Optimal stopping point: Epoch 3
What actually happened: Trained all 10 epochs
If we had used early stopping (patience=3), training would have halted at epoch 3, potentially avoiding the severe memorisation. Even at epoch 3 with 14.6% accuracy, we would have at least not gotten worse.
Diagnosis: Why 20.8% Validation Accuracy?
Factor 1: Insufficient Training Data
The Maths:
6 breeds × ~33 training images = 198 images total
Batch size: 32 images
Training batches: 7
Classes: 6
Random chance: 16.7% (1/6)
Achieved: 20.8%
The model barely outperforms random guessing. This strongly indicates it learned nothing generalisable.
Benchmark comparison:
ImageNet transfer learning: Requires ≥100 images per class
Fine-tuning ResNet50: Industry standard minimum
My dataset: 33 images per class (1/3 of minimum)
Factor 2: Image Quality & Diversity
While TensorFlow successfully loaded 243 images, quality issues remained:
Potential problems:
Variant angles (fish shot from side vs. top vs. front)
Lighting variation (sunlit pond, indoor aquarium, murky water)
Partial visibility (fish at edge of frame, partly obscured)
Breed confusion (WebP conversion may have lost color fidelity; Kohaku's red loses saturation)
ResNet50 is powerful, but with only 33 images per breed, it can't learn robust features across all these variations.
Factor 3: Validation Set Too Small
Challenge:
Validation set: 48 images (8 per breed)
With 8-image samples, random fluctuation is large
Val accuracy could fluctuate wildly between epochs due to chance
Example:
Epoch 2: Val accuracy 18.8% (might have gotten fewer breeds correct by chance)
Epoch 3: Val accuracy 14.6% (all 8 validation images were hard cases)
This noise masks any real learning signal
What would help:
Validation set of ≥50 per breed (300+ total)
More stable, reliable accuracy measurements
Clearer signal about whether model is actually learning
Factor 4: Class Imbalance
While breed distribution is relatively balanced (49–55 images per breed), with only 33 per breed, any slight imbalance matters more.
Conclusion: you need a lot of data to train an AI model
As expected, the training set let me down here. Too many similar pictures, the majority of which were rendered useless once it became apparent all the .webp files were to be discarded.
To make an improvement, I'd be looking in the region of 1000+ images of greater variety, running a quality audit on them prior to data loading (can be automated via Python script).
With the addition of early stopping too, Claude estimates potential results of:
Training accuracy: 85–95%
Validation accuracy: 75–85%
Gap: 10–20 points (healthy)
Model size: ~95MB (manageable)
Frankly there's more chance of Elvis riding past on Shergar than me going through the tedium of creating an expanded dataset, never mind training it again and then integrating to the frontend. If I was a tech company and had ways to get hold of huge datasets to create a monetisable product, then sure. But for a personal project it's a bridge too far - I'm happy enough with the Anthropic AI. And I never want to see a picture of a koi for a very long time.
The actual learning experience here was of great value though, and that's what really matters. To get hands-on with training and see how important the variety, quantity and quality of image data is to the end result, watching the epochs churn through on one of Google's Nvidia T4 GPUs brings a huge part of AI to life. Seeing how those results developed through the epochs and being able to spot patterns and identify issues was invaluable in getting further under the skin of AI and model training, and demystifying the topic into more practical, tangible outcomes.
And once again this was thanks to the brilliance of the tools (Claude) existing in the first place. Twelve to eighteen months ago I wouldn't have bothered attempting such a project, expecting to fully get bogged down in the tedium of bug fixes, writing code (which I have nowhere near enough patience for and rage-quit several years ago), generating all the accompanying docs, deciding on tech stack and toolchain, and summarising the whole thing in markdown files for my repos. This was all done on the free tier of Claude, with less than 5 USD of API credit (which also covered me for Claude Code training certifications).
It's not the purest way to develop projects, but frankly I don't care - I have no truck for purists or dogmas in any field. What's important is how easy it is to create, to learn and do it fast. If you've got a product or design idea and enough knowledge of regulations and security not to do anything stupid (and an aversion to writing proper code) then this is the way.
David Turner is the founder of Kói, an independent strategic consultancy advising senior leaders and investors on high-value decisions across technology and adjacent creative fields.
You can reach him at: enquiries@dkoi.design
© Kói Holdings Ltd 2026. All Rights Reserved.
