This is an example of why I'm a little stumped about why people are excited about Jev-style decision models. For any application at scale it'd be way more accurate to have a dedicated classifier using the right architecture (neural net, bayesian, etc.) LLMs drive the cost of making those models down to near zero.

Just use the LLM to make a thing that's actually good at what you need it to do.

> I compared to a 10-year old CNN and to my surprise found that this was superior.

I'm surprised he's surprised.

> A small, old, specialised network using an outdated architecture recognises skin lesions far better than two frontier LLMs.

Yes of course. A 30m params Katago network with one playout is superhuman in playing go, while frontier LLMs have over 10,000 times more parameters and aren't any good at playing go. They're good at other things (such as helping you create a network that's good at playing go).

> I used Inception v3 pretrained on everyday photographs and then fine-tuned all layers on 68,011 dermoscopic and clinical images from the ISIC Archive on a single H100 GPU.

>These 200 test lesions, and every other photograph of the same lesion or patient, were removed from its data (including the validation set) before training, so it had never seen them, and they were not used to choose the model or set its threshold.

This is just benchmaxxing. A benchmaxxed model will always beat an unbenchmaxxed model, but it will be brittle.

You can always beat a specific dataset with a small model by finetuning on a train/test split. But it doesn't generalize well to 'wild' images, because they are out-of-domain.

And a calculator outperforms LLMs at multiplication and division...

As people have mentioned, it isn't that surprising that a specially trained model beats a general purpose model.

However, I think it kinda misses the potential of llms for diagnosis. What happens if we also include journal notes? What if we let the model generate questions to ask the patient and include those? What if the images aren't perfect examples and happen to contain out of distribution content like a tattoo?

I found SkinGPT-4 when I was looking into this article. From the nature paper "among the 150 cases, a significant percentage of SkinGPT-4’s diagnoses (80.63%) were evaluated as correct or relevant by board-certified dermatologists." Which does outperform the CNN in this article.

I remember being blown away by CNNs and GANs when they first gained popularity. I do think, with skin cancer diagnosis, a transformer architecture is going to be the better bet.

LLMs shouldn't be treated as if they were purpose-built for specific cases just because they generally output a plausible-looking answer, and this is not something that was discovered only recently

Here's a good heuristic for predicting the ability of a modern frontier LLM at any non-automatable task: Is it hugely overrepresented in their pre-training dataset (e.g. code), and were they specifically post-trained on it using reinforcement learning (e.g. coding)? If the answer to one or both is yes, you should expect them to be very good at it. Otherwise, you should expect them to be, at best, as good as an average human who's read a few books on the subject and is good at bullshitting.

I developed the current #1 ranking leaderboard model and seems like a good time to stop lurking and chime in... It's no surprise that a dramatically simpler model fit to the narrow ~11 label prediction task outperforms an LLM where the same task is a drop in the ocean with regards to its training corpus.

What is maybe more interesting is that, despite the availability of open weight medical image-language embedding models like Google's MedSigLIP AND the opportunity to adapt them, they don't seem to break into the Top-10. My impression is that the joint representation is valuable down-stream of the diagnostic classification task for interpretation, communication, and management of the risk, but the language corpus for dermatology images are full of 'human engineered' descriptive features (think Bitter Lesson), and we should be looking past clinical semantics for more informative visual features if we want to push the limits of performance and generalizability. Ultimately, this seems necessary: people are immediately and persistently aware of the presence and extent of their skin conditions, the skin is the most readily accessible to imaging by patients themselves, smartphones have remarkable macroscopic photo capability, and there is huge demand for dermatology care.

OK, so everyone in the replies is saying "yeah duh", but a few years ago people used to defend the negative externalities of LLM's by pointing to medical applications and saying "but think of how useful they would be for doing a cancer diagnosis"

This is true for a large subset of problems. LLMs are bad at basically all games, for instance. LLMs are even bad at chess, which has a huge amount of publicly available data. LLMs are really bad at games like backgammon, which has only 10s of millions of tokens of public data.

My dog outperforms the world's top researchers in an unassisted 100 meter sprint.

I kind of expected this. Fine-tuned models, among other tools, should be used by LLMs to interpret nuances in images. Just as I wouldn't trust LLMs to interpret a complex mathematical formula without it having access to Lean and other methods to check its reasoning.

This doesn't seem that surprising to me? One is tasked with a simple very narrow task. The other has a corpus including billions of other images and pieces of information. It'd seem to me that instinctively one would outperform the other, regardless of age.

I think an important comparison is the LLM with the supervised training data in context. I wonder how the LLM would perform if you provided all, or a representative subset of, the data used to fine tune the CNN in the prompt and then asked it to classify a new case.

This strategy is how LLM-like models have achieved leading performance on tabular prediction tasks (TabPFN).

Hardly surprising that models specialised at language aren't as good at a highly specialist task as a model trained specifically at that task, isn't it?

I know some people want to believe that LLMs are good at everything, but they're not. They're good at language.

Yeah obviously. Xgboost outperforms llms in algorithmic trading, even applying the token prediction method to markets xgboost is effectively better.

Llms have their applications. Other models still do.

This isn't really surprising. LLMs are intended to be generally intelligent. They are spiky on the things frontier labs tend to benchmark against (software/math) but are not intended or expected to be great at everything. The general intelligence unlocks new workflows, it does not perfect existing ones.

That being said, I'm sure a frontier LLM could make some improvements to your CNN.

The same holds true for most classification tasks - if you have sufficient samples and high quality labels, training an xgboost or a deep learning model with a classification head is the way to go. You can use a frontier LLM on high-effort to help you with classification reasoning and explainability.

But how well will the CNN be able to write me a fanfiction of Dobby being in Lord of the Rings?

This entire LLM hoo-ha is leading to a "revival" and introspection on other techniques that are being revisited. Which is wonderful ...

Seems obvious given how each works. Choose the right tool for the job, don't blame the tool.

This is not a compelling analysis.

First, the CNN is not 10 years old. The base is but it was fine tuned for this project on the same dataset the test images were derived from:

> The test: The lesions come from MILK10k, a public dataset released through the International Skin Imaging Collaboration [ISIC] in 2025

> The CNN is deliberately not state of the art. I used Inception v3 pretrained on everyday photographs and then fine-tuned all layers on 68,011 dermoscopic and clinical images from the ISIC Archive on a single H100 GPU

The test images themselves were filtered out at least, but that's not really enough. The CNN has a home field advantage because the test images are going to be in distribution with the training images.

Second, Looking at the code, for frontier models probably you should be giving them image tools and using them in an agent loop and not trying to one shot results (e.g. https://platform.claude.com/docs/en/build-with-claude/prompt...). It is also not meaningful to ask an LLM for a numeric confidence score.

Third, all 200 of the the test images were biopsied by the clinician originally. So when you ask the model "You are assisting a dermatologist" and "Decide whether this lesion can be safely discharged, meaning it needs no treatment, no biopsy and no follow-up" why would you penalize the model for making the same call the physician did even if it was benign in retrospect?

Are you evaluating the model on if it can replace the need for a biopsy itself (in which case the ground truth is whether it in retrospect was harmful or not) or in if it can match physician judgement in determining when a biopsy is needed (in which case the ground truth is if they physician ordered a biopsy)? You're asking the model to do the latter and grading it on the former.

Overall yeah probably specialized models make a ton of sense in a bunch of areas still but idk this does not seem like a good demonstration of that

I mean not too surprising, it's a specialised model that trained on relevant quantitive/qualitative inputs.

LLMs are very general and flexible, by all means they can do well on non-text based specialised problems, but it also does well because their harnesses know how to defer to the right tools a lot of the time, and produces good answers from inspecting the output. Like if you ask Codex/Claude Code to transcribe a videos, it's going to create a script that uses whisper to extract the transcript not manually process the binary. If you ask it for demographic information it'll pull census data and write a script which produces the summary stats and report it back to you (I imagine much of the recent math stuff relied on this).

Sometimes it'll pull a number from its training data but the strength of an LLM is its ability to autonomously seek out the right answer even when it doesn't know the answer initially.

You could definitely use train an LLM to utilise these efficient specialised models, by it may be more cost effective to ask the model to build a special purpose application with a new UI over the top of the efficient specialised model.

It's cool that there's an in-browser skin classifier...but it says "It is not a state-of-the-art model". The obvious question is, is there a free in browser state-of-the-art model?

This isn't surprising but it's good to have reporting to point to. The only way a general purpose LLM like these will match the performance of a bespoke classifier is if it has access to such a model.

The model architecture might be ten old, but it seems it was trained recently, so the title is a bit misleading.

It's going to be funny/sad to watch people who have deeply misapplied LLM tech discover "oh gee, this is pretty expensive. I guess I can just use ML?"

CNNs are going the way of radiologists...

I'd like to see it compared to chinese models like DeepSeek, Kimi, GLM, etc just for the sake of research

LLMs are not 10 years old and far far more than just a CN. By the time hey hit the 10 year mark I’d be stunned if they weren’t better, given the pace of change in math and AI. Once Lon’s get specifically trained on these and other targeted tasks, they will develope

Give to Caesar what is Caesar’s, and to God what is God’s.

The key is to use llms to create CNN models

it is a language model after all

Read this as: AI beaten by a 10yo.

An example of "Dumb it down"

In other news, 30-year-old chess engine outperforms frontier llms in chess

this should not be surprising. alphafold is supervised learning.

did you ask the LLM to make a CNN instead haha, might’ve won

Duh?

I asked ChatGPT what it thinks about this situation and it told me LLMs can diagnose skin cancer, so there.

Couldn't you just train a general purpose LLM on the output of this CNN?

Anyone wanna take a bet that the general purpose model will beat these specific models in a year or two?

It seems one is more specialized for the purpose and compare it against a general purpose LLM instead of a specalized/focused LLM?

This is like saying I compared this against photos for sure stored in a home swimming pool (CNN), compared to photos presumably stored in an ocean (generalized LLM).

I use LLMs where I can, but I'm no automatic fan of LLMs, a healthier comparison here seems to be what if a general purpose LLM was provided those CNN images, and perhaps what does a fine tuned local model with the CNN images perform like?

It still might not be better, but I would suspect it's closer, and a more realistic apples to apples comparison.

General assistant models compared to task specific models seem to be a well known spot where things don't always fit, but might be improving.

I am not sure I would expect a transformer model to excel on this kind of task.

Which CNN? A fine-tuned version of Inception v3, fine-tuned on 68,011 dermatology images

Which LLM? GPT-6 Astra and Claude Fable 5.1, not specified exactly which version

Obviously this is a bit like comparing an 100 iq radiologist, 20 years experience, with a just-tested-at-iq-130 6 year old and observing the radiologist beat the crap out of the kid. Was anybody really expecting a different outcome? Let's see after fine tuning.

Btw: the actual lesson from that network when it came out, which is really important is that Inception v6, the Google trained version (ie. can recognize cats, halloween masks and airplanes) beat, with a wide margin, the version that was trained from scratch on clinical images (and still didn't do so well, 62%. Not even really good enough for a first triage).

fyi CNN: convolutional neural network

I'm sorry but as an MLE this is not a remotely useful article, and does not belong on the front page.

It's comparing a fine-tuned CNN to general purpose models on a tiny dataset.