Koi-identifier Pt. 1: Anthropic API
- David Turner
- Mar 22
- 10 min read
Updated: Aug 25
Using AI to recognise images
One of my favourite places to test lenses and try out stuff with photography in general is the Kyoto Garden in Holland Park. This means I've got literally hundreds, if not thousands of pictures of Koi carp accumulated over the last 10+ yrs.
Sorting through those pics for my portfolio, I thought for the first time about how many different Koi I'd photographed, which varieties they were, and if it'd be possible to identify the individual specimens to get an idea of how long they'd been around. And that was the spark for this project...
Summary
Objective: create an AI-powered image analysis app for identifying varieties of koi and logging individual sightings
Stack: HTML5, CSS3, JavaScript, Express.js, Node,js, SQLite, Anthropic API
Results: Success - full working app validated. AI model not trained on koi though.
Outcome: Next step to improve AI element by training a specific model
GitHub: koi-identifier
AI Image recognition as the tool
Rather than waste days of my life running through a stack of around a thousand-ish pics manually, I wanted AI to do it for me. I've built all kinds of stuff with Claude, Gemini, Le Chat and Chat GPT, but never tried out image recognition - therefore it had to be done.
At this point I should say there's some method to the madness, and the premise is less facetious than it might at first appear. Koi don't fit easily into breed categories in the same way as say dogs do. Where dog breed standards are defined clearly by the Kennel Club, Koi varieties are more akin to layered traits which are far more fluid. For example, a Koi fish might be classified as follows:
Base variety: Defines the primary pattern/colour (e.g., Bekko - non-metallic Japanese koi with a solid white, red, or yellow base (shiroji) and black (sumi) markings that resemble a tortoiseshell pattern)
Additional Traits:
Doitsu: Little to no scales
Gin Rin: Sparkling scales
Tancho: Single red spot on the head.
Butterfly: Long, flowing fins.
Example: A "Doitsu Gin Rin Bekko" is a Bekko (base variety) with Doitsu scalation and Gin Rin sparkle.
Therefore it's a fairly involved process to identify each variety correctly, and would be a good test of an AI models capabilities
Design concept for an AI-powered image recognition tool
Dragging my Claude project agent out of bed, we set about vibe-PDR'ing a product spec. I wanted a web-page where I could upload photos, get the variety recognised, identify whether I'd seen this Koi before or not, and record the results. This meant a fundamental feature-set of:
Upload koi photos for AI analysis
Use Claude's vision capabilities to identify breed and distinctive features
Automatically detect duplicate fish from the existing collection
Store everything in a SQLite database on my machine
Track sighting history for each fish
For implementation, this meant a stack consisting of:
Frontend: HTML5, CSS3, Vanilla JavaScript
Backend: Express.js, Node.js
Database: SQLite (file-based, local)
AI: Gemini (later replaced by Anthropic Claude API - read on for details)
Development of the image recognition tool
As ever it was easy to get a first proto web page up and running - Claude churns out the code, I run it in VS Code and a browser window, and we run through a few simple iterations to get everything looking presentable.
Storing data only in the browser's LocalStorage however exposed a security vulnerability - the API key would be visible in the frontend code. I needed a backend server to handle API calls securely.
Therefore I built an Express.js server running on localhost:3001 with CORS middleware to handle requests from the frontend on localhost:8000.
Designing an API to interface to the models
The REST API follows standard conventions:
GET: for retrieval
POST: for creation
PUT: for updates
DELETE: for removal
JSON request/response bodies
Proper HTTP status codes (201 for creation, 500 for errors, etc.)
The `/api/analyze-koi` endpoint sends both photo and existing fish to Claude, asking it to:
1. Describe distinctive features
2. Check for matches in the collection
3. Suggest the correct breed from a predefined list
4. Return JSON with `isNewFish`, `confidence`, `suggestedBreed`, and reasoning
The biggest headache though came courtesy of utilising Gemini models for image recognition. I'd chosen Gemini over Claude for a couple of reasons; theoretically it's more capable of image-based tasks, and also being a Google One member I should get a generous free tier of usage before having to buy credits.
However (and not for the first time) Gemini's inconsistent API naming conventions and quotas proved to be tedious and frustrating in practice:
gemini-2.0-flash` with `v1` → "quota exceeded" (as it's a paid model)
gemini-1.5-flash` with `v1beta` → "not found for API version"
gemini-pro-vision` → finally worked, then quota issues appeared
gemini-1.5-flash-latest` with `v1beta` → "not found for API version"
At this point I got tired of the guesswork and decided to move over to Claude, where I had a few dollars of credit leftover from the certifications I'd previously done with them.
Documentation and consistency were better, and Claude code was far happier working with it's own product family.
With the switch to Claude's API, we get our first working version - upload a pic, the model analyses it, and we have a record form to populate and submit.
UI/UX improvements to the image recognition tool
Giving the first version a test uncovered the usual UX issues that come with vibe-coded projects. Analysis text that wasn't human-friendly, no mention of the variety of koi which had been recognised, no auto-population of the form based on the analysis, no navigation buttons between screens (collection and analysis)... etc. Took several iterations to rectify these, but we got there by implementing a few context-aware header buttons:
On Collection view → Show "+ Add Fish"
On Add Form → Show "← Collection"
On Detail view → Show "← Collection"
Logo always clickable to return home
From a UI standpoint, I wasn't impressed with the bland design, so gave Claude a screenshot of this website and told it to match the colour palette and typography.
One thing which really annoys me about pretty much all AI tools is their over-use of emojis, as if they've been trained exclusively on the Snapchat messages of hyperactive teenagers. So the fish emoji had to go, to be replaced by the Kói logo we'd already created for this site.
Logo issues - why can't Claude reproduce images correctly?
I gave Claude a screenshot of the logo (which it helped me to prototype remember) and it couldn't recreate it for the site. So I give it an SVG file, and again it fails, after confidently telling me what a great a job it had done (which is my biggest annoyance with Claude; it will over-confidently declare it's brilliance in the face of obvious failure, and even make up incorrect answers or facts to sound confident. Challenge it on that and it'll back down really quickly and admits what it's done. Be diligent in checking it's outputs and don't give it an inch is the best practice in this regard).
Eventually we get there after using the usual method of telling it to call up the file rather than try to recreate it (same lesson as from the Grand-Complication project).
Adding a backend to create persistence
With UI/UX sorted and bugs ironed out, we had a pretty good working version 2. However once the code was stopped or restarted, the collection would be lost - we need therefore to add persistence via a backend database.
Being stingy as ever I wasn't about to pay for that, so opted to run the SQLite locally on my machine, which is always fine for personal projects like this. I got a schema set up with two tables:
Fish: Stores fish records with name, breed, markings, notes, and photo URLs
Sightings: Tracks each sighting with a foreign key relationship to Fish (cascade delete on fish removal)
The schema uses:
INTEGER PRIMARY KEY: for auto-incrementing IDs
TEXT: for dates (stored as strings like "20/03/2026" for simplicity)
FOREIGN KEY: with CASCADE DELETE for referential integrity
Separate sightings table: rather than JSON array in Fish (normalised design, easier to query)
This simple schema supports the app perfectly while being normalisable for future expansion (e.g., environmental conditions, water parameters, size estimates).
The backend itself was then implemented with full CRUD operations:
GET all fish with their sightings
POST new fish
PUT to update fish
DELETE fish (which cascades to sightings)
POST to log new sightings
This design meant the foreign key relationship between Fish and Sightings would keep data integrity tight, and the cascade delete would prevent orphaned records.
Frontend-backend integration
Here's where things got tricky. The original frontend was saving everything to LocalStorage. I needed to switch it to sync with the backend database.
The frontend uses `fetch()` with promise chains for all backend calls:
```javascript
fetch('/api/fish', { method: 'POST', body: JSON.stringify(fish) })
.then(res => res.json())
.then(data => {
// Handle response
loadFish(); // Reload from database
})
.catch(err => console.error(err));
```This pattern ensures:
Proper error handling
Explicit async flow
Clear separation of concerns
Couple of errors to fix during this stage with Claude's help:
Error #1: Missing Required Headers
Error: `anthropic-version: header is required`
Fix: Added the required header to all Claude API calls:
```javascript
'anthropic-version': '2023-06-01'
```Error #2: Image Format Detection
Error: "The image was specified using image/jpeg but appears to be image/png"
Root Cause: The code hardcoded `image/jpeg` but users uploaded PNGs.
Fix: Implemented automatic image type detection from the base64 header:
```javascript
let mediaType = 'image/jpeg';
if (photoBase64.startsWith('iVBORw0KGgo')) {
mediaType = 'image/png';
} else if (photoBase64.startsWith('R0lGODlh')) {
mediaType = 'image/gif';
}
```Learning: Always handle different input formats gracefully. Users won't care about your assumptions; they'll just upload a PNG.
Error #3: Data Not Appearing After Save
Problem: Fish were saving to the database (confirmed in Terminal logs) but weren't appearing in the UI.
Root Cause: The frontend loaded fish once on page load from LocalStorage, but never refreshed after saving to the database.
Fix: Added `loadFish()` call after successfully saving:
```javascript
loadFish(); // Reload from database
showMainView();
```Learning: Frontend-backend sync requires explicit refresh calls. The browser doesn't magically know the server state changed.
Error #4: Duplicate Sightings
Problem: New fish were showing "2 sightings" instead of 1.
Root Cause: The sighting POST request was fire-and-forget (no `.then()` handler), so `loadFish()` was called before the sighting finished inserting. Race condition.
Fix: Made the sighting creation wait before reloading:
```javascript
fetch(/* sighting */)
.then(res => {
// Only NOW reload
loadFish();
showMainView();
})
```Learning: In JavaScript, asynchronous operations require explicit waiting. Fire-and-forget patterns cause race conditions. Always chain `.then()` for dependent operations.
Errors fixed, we arrive at v3, the MVP in terms of this product and based on a stack of:
Decision | Alternative | Rationale |
SQLite Database | Cloud database | Local-first approach, zero hosting costs, data ownership |
Anthropic Claude | Google Gemini | Better API consistency |
Local Development | Vercel deployment | Simpler iteration, no serverless complexity for SQLite |
Vanilla JS | React/Vue | Simpler setup, no build process, fewer dependencies for small project |
Express.js | Django/Flask | Familiar to JavaScript developers, lightweight, good for simple APIs |
Cost and Environmental Impact of the AI development
The MVP did everything it was supposed to in terms of features, and had a UI and UX I was happy with. One thing I wasn't sure about though was how efficient the product would be, as this was my first run-out with an image-based AI analysis project.
So with Claude's help I calculated the figures, and surprisingly it wasn't too harsh on the API billing or energy usage:
Costs: Per Photo Analysis
Claude API Pricing (as of March 2026):
Input tokens: $3 per 1M tokens
Output tokens: $15 per 1M tokens
Typical Koi Photo Analysis:
A single photo analysis involves:
1. Image encoding to base64: ~50-150KB file size
Base64 encoding adds ~33% overhead
A 100KB JPEG → ~133KB base64
2. Claude API vision token usage:
Image processing: ~500-1000 tokens (depends on photo quality and complexity)
System prompt + breed list: ~800 tokens
Existing fish collection context: ~500-2000 tokens (scales with collection size)
Total input tokens: ~1800-3800 tokens per analysis
Output response: ~200-300 tokens
Cost Breakdown Per Analysis:
Scenario 1: Small collection (1-5 fish)
Input: ~2000 tokens × ($3/1M) = $0.006
Output: ~250 tokens × ($15/1M) = $0.0038
Total per analysis: ~$0.01 (1 cent)
Scenario 2: Medium collection (50 fish)
Input: ~2500 tokens × ($3/1M) = $0.0075
Output: ~250 tokens × ($15/1M) = $0.0038
Total per analysis: ~$0.011 (1.1 cents)
Scenario 3: Large collection (500 fish)
Input: ~3500 tokens × ($3/1M) = $0.0105
Output: ~250 tokens × ($15/1M) = $0.0038
Total per analysis: ~$0.015 (1.5 cents)
Energy Consumption Per Token
Anthropic's Claude models run on cloud infrastructure (likely AWS or similar).
Estimated energy per token processed:
LLM inference: ~0.01-0.1 Wh per 1M tokens (varies by model)
For Claude 3.5 Sonnet: roughly ~0.05 Wh per 1M tokens
Per koi photo analysis:
Input tokens: ~2500 tokens
Output tokens: ~250 tokens
Total: ~2750 tokens
Energy: ~2750 tokens × (0.05 Wh / 1M tokens) = ~0.00014 Wh = 0.5 kJ
That's equivalent to:
A single LED lightbulb for 0.0001 hours
About 50 milliseconds of your laptop's power draw
Negligible in practical terms
Carbon Footprint Per Analysis
Using typical data centre carbon intensity:
Cloud data center: ~100-200 grams CO₂ per kWh
Let's use 150 grams CO₂/kWh (middle estimate)
Per koi photo:
Energy: 0.00014 kWh
Carbon: 0.00014 kWh × 150 g CO₂/kWh = ~0.021 grams CO₂
To put this in perspective:
One koi photo analysis creates as much CO₂ as:
Driving a car for ~1 metre (0.0006 miles)
A single sheet of paper printed and recycled
Typing 100 words on your laptop
Literally unmeasurable in human terms
Metric | Per Photo | Monthly (12 pics) | Annual (150 pics) |
Cost | ~$0.012 | ~$0.14 | ~$1.80 |
Energy | 0.5 kJ | 6 kJ | 75 kJ |
Carbon | 0.021g CO₂ | 0.25g CO₂ | 3.1g CO₂ |
Equivalent | 1m drive | 30m drive | 1.5km drive |
So as a personal project that I'll likely use a couple of times and forget, it's of negligible impact to the environment, it's cheap, and cost scales predictably. And I get to keep hold of all my pics locally.
If on the other hand I was running something like this on a wider scale, optimisations would be possible:
Batch processing
Caching (to skip the API call if the same pic is uploaded twice)
Pre-filter prior to analysis (discount same date/location pics, discount perfect image matches etc)
Validation
In terms of functional performance, characteristics are as follows:
Photo upload: 2-5 seconds (depends on AI API latency)
Database queries: Instant (SQLite is fast for small collections)
Frontend rendering: Instant for <1000 fish
Memory usage: Minimal; entire fish collection in memory
Storage: SQLite file grows ~500KB per 100 fish (with photos)
All ok at first glance, until I started to give it more of a workout. Then some patterns started to emerge:
The same fish could be analysed twice with different results.
The same photograph could be analysed twice with different results
At this point I could have overlooked this and used the app manually. But being a tedious sort, I just wouldn't let it lie (© Vic Reeves 1990)
So with a bit of digging, I found out why this was happening - the Anthropic LLM hadn't been specifically trained to do something as niche as identify koi, so it was prone to errors. And the only way to fix that would be to train a specific model.
Claude told me not to bother as it would be time consuming and a bit too much for a personal project, but for the second time in three paragraphs I just wouldn't let it lie - see Pt. II of this post...
David Turner is the founder of Kói, an independent strategic consultancy advising senior leaders and investors on high-value decisions across technology and adjacent creative fields.
You can reach him at: enquiries@dkoi.design
© Kói Holdings Ltd 2026. All Rights Reserved.
