I mean we've been using technology to solve long-standing problems in mathematics that no human had been able to do for years. Is Deep Blue sentient because it can beat any human at chess?
Deep Blue's strength was leveraging massive compute to execute a task-specific, human-written algorithm. The problems which LLMs are tackling don't elude mathematicians because they require too much number-crunching, but because they demand creative problem solving. The latter seems more profound, even if it doesn't imply sentience.
Not to get too reductive - but LLMs are essentially performing massive number-crunching at a textual representation level. If you've ever played around with early NLP models, they can give you some astounding results for aspects like sentiment analysis that would seem like creative problem solving to someone in 1950. You could say the same for using a neural network to perform things like object recognition.
That's not removing the amazing ability of LLMs and the scale of this accomplishment, don't get me wrong - it's incredibly impressive that specialised models are now able to walk down pathways for creative problem solving - but I don't think that suggests a sentience or profoundness - rather it's just number-crunching at a astronomically larger scale.
Yeah agreed - Just as a future context I tend to close out of articles quickly if they have an AI generated image. Highly reccomend something like Unsplash or Comsos.so - They both have some brilliant visuals, especially public.work - cosmos.so/public-work
Even old archives are great to search for visuals and shows the passion that you've put in to the writing.
This is really helpful, thank you! This post was one of my first with relevant, 'real' images, and I really like how it came out relative to the generated stuff. Besides, I feel like image generation has lost its way since DALL-E got deprecated. I've bookmarked these resources!
I'm sorry - but you're not going to ban AI no more than you can ban the transistor. You could limit & limit the potential of who uses it - but historically that seems to benefit the few rather than the many.
Banning doesn't mean the thing you ban disappears, you're stretching the definition quite a lot. The limit you're talking about is what a ban is. I would personally support a complete ban on AI technology impersonating humans. At no point would I expect that would completely get rid of the phenomenon. But it should be seen as extremely cringe and face meaningful push back, given how antisocial and antihuman that is.
There's a massive difference between the hatred of a CEO who is actively wanting to replace workers with what is essentially applied mathematics. AI seems more like easy reasoning for mass-layoffs & cost saving measures - and I rarely see articles that actually attempt to delve into this, instead seeking to just cancel out an entire technology.
This article doesn't hate AI - it hates capitalism - which is a completely different argument, the underlying system was broken already, AI has just excasperated some of the concerns. Things like awful SEO + low effort art were already happening beforehand, they're just become far easier.
And maybe a big problem is that AI = ChatGPT for the vast majority of people, including the person who wrote this artcle.
This article specifically cites things like the Commonwealth Prize - a prize that if you look at historically, wasn't exactly an example of brilliant prose. Surely that's far more of a inditement on the quality of judging for a prize if it can be won by poor writing.
A lot of the issues cited within this article just seem hollow, as they're issues that were pervasive before ChatGPT. AI isn't a panacea, but hating a technology because bad people use it feels reductionist.
I think a far bigger problem is that the majority of the population doesn't have good knowledge of AI or Software in general, including CEOs. I'd love to see journalists that have a good understanding of the actual technology.
You don't need to have a "good knowledge" of a misused technology to hate it when it's used by malevolent people.
In the same way I don't need to be a virologist to know that is better to avoid the flu, I don't need to be a ML/AI expert to see its direct detrimental effects on people, communities, and the internet as a whole.
To use your analogy, I would say the "blanket ban" attitude would be more like wishing all viruses would just go away, or have never existed in the first place, which:
1) is an impossible and unproductive attitude, and
2) fails to recognize the important contribution to evolution, genetic diversity, and our immune systems that viruses introduced, not to mention the possible beneficial applications that could exist by understanding it.
Rejecting something without nuance makes you more vulnerable down the road because it prevents you from building an effective immunity. Engaging with it is the only productive way to mitigate its downsides and promote its benefits.
Exactly, my vaccine against GenAI chatbots is to not use them, which is the equivalent of masking when covid vaccines were not available.
I am just waiting for the GenAI vaccines (ideally regulations, practically people rioting in municipal councils against gigantic data centers opening at their door).
But given your example - I don't care much for the opinion of someone who believes flu is spread by sinful thoughts. It's good to have a base understanding of something that you'd like to speak about.
Are local LLM models also within this hate sphere? What about fully open source vision models? That's what makes an article like this feel hollow - it's just someone talking about vibes.
Or to quote the article:
" But while I took mental notes on what I was observing, I also felt a lack of representation for true, profound, and guttural loathing of AI. The people like me who have only the vaguest idea of what defines AI, but extremely specific examples of why it sucks. "
That's why I think this article is a criticism of neoliberal capitalism rather than anything else. If it wasn't AI, it would be robotics, if it wasn't robotics, it would be Quantum. But i'd like to see better substance in articles on this site rather than just a dislike of robots.
Not sure about this one - my slightly terrible portfolio website that i'm in the process of completely revamping (bemben.co.uk) managed to mog simon wilsons webblog (https://simonwillison.net/) - which is a far better website in all aspects!
Maybe introduce some additional stats like load times, content analysis etc - and tweak the prompt slightly - just because a website looks slightly newer, doesn't mean it's better at all!
Fun that we both used simonw as the "competitor", and both of our more minimalistic websites won over him! :)
So the obvious next step is to pit them against each other, seems your "premium, editorial feel" won out!
> bemben.co.uk wins due to its sophisticated use of typography and a cohesive dark-mode aesthetic that creates a premium, editorial feel. While emsh.cat is functional and clean, it lacks the intentional visual hierarchy and character found in the layout of bemben.co.uk.
I'm sure eventually i'll eat my words - but Quantum still seems like a massive marketing gimmick. The technology itself is incredibly interesting, but it feels as if CERN began advertising itself as a marketing stunt - there's just something about the way I see quantum marketed + advertised right now that doesn't seem to align with reality.
I suppose in spirit of the article - it's as if the manhattan project in 1944 was telling the world that theoretically it's 6-12 months away from igniting the entire upper atmosphere.
There's an incredibly serious lack of education with how LLMs & carb-counting works. This entire article would be better suited to astrology.com than hackernews.
When I opened it up, I assumed the author would have at least attempted a calculation service, maybe even placed something like the size of the meal into an actual model, using the integration of pre-existing tools that are (slightly more) accurate. Hell - most food literally is required to have calorie information, and you can query open source data for others!
But the author just took pictures of food & expected a realistic response?
Is this genuinely what amounts to a study in AI?
This is akin to the instagram reels that talk to chatGPT and ask it to time how long they're run is. Except those are treated as funny jokes rather than being turned into studies.
I'd like to see this study done using any kind of actual grounding knowledge, seeing what mistakes AI makes when attempting to query ground truth from picture analysis - there would at least be an interesting result methodology in that.
> But the author just took pictures of food & expected a realistic response?
There are very popular apps on the App Store right now that are going viral among non-techie people that do exactly this, and they have no concept of how AI works. My wife was talking about one and I had to give her a reality check that the AI had no idea what ingredients were used to make the food. And she's a licensed nutritionalist.
Studies like this create something to point at for people who are confused and serve as a springboard for a conversation in the media.
That's true - I suppose i'm just dissapointed that this study hasn't seemed to include those within any analysis. Being able to point out that the top 100 calorie counting apps on the app store return similiar results to simple frontier models would be of interest.
I think i'm just dissapointed that this study doesn't go deep enough, and stays at a surface level statistical analysis of frontier models.
To be fair my expectations is that those apps have done the prompt engineering, and schema, and tools (to query nutrition database), etc... and although they're not 100% consistent, the margin of errors should be narrow to the point that barely matter, and they should do a bit better than a random ChatGPT chat session.
the problem isn't one that can be solved with prompts. If I gave a panel of food and nutrition experts (human or machine) a bunch of pictures of food, they still wouldn't be able to tell if, e.g. a slice of cake was made with whole milk or skim.
The "pic of packaged food --> LLM --> nutrition DB call" pipeline is workable, but many users of these apps are using them for fresh prepared foods, which is just an unworkable problem without either an understanding of the preparation process or a bomb calorimeter.
The real benchmark should be comparing the amounts with a human guess. And aa far as I know with diabetes if you are within 30% of guessing carbs then you should be fine.
Yeah, for sure there are. And people will just ask ChatGPT as well.
The funny thing is that for people who are just trying to lose weight without managing any health issues precisely, this type of extreme variance doesn't really matter, because the mere act of consciously quantifying food consumption is, based on my experience counting calories, the single biggest factor in success with weight loss.
I actually think "just asking ChatGPT" is fine, because A) the data in these apps is suspect at best and B) the data behind calories is also pretty suspect (but we all play along because we can adjust other variables to make it all "work" well enough).
Once or twice a year I spend a few weeks meticulously measuring ingredients/cooked foods and recording calories and on complex recipes apps are next to useless at getting accurate data. You're trying to input five or ten relevant ingredients, and then weighing your cooked outcome to try and divide the ingredients by proportion. Frankly it's a mess and most people aren't doing it for home cooked meals, and are getting very lossy outcomes (weighing cooked chicken and marking it as raw chicken, etc)
With reasoning and tool calling (combined with me meticulously weighing before and after), it's producing fine data for my purposes.
I was complaining about AI generated clothes being misleading marketing, deceiving customers as to whether the garment even exists.
And then I learned that the pre-AI norms weren't any less fictional: they made an exemplar garment and did photoshoots, sure, but then they send the pictures and patterns to the lowest bidder factories with permission to make whatever edits are necessary to make it cheap and manufactureable. The whole thing was already a simulacrum.
I honestly think that, given the sorry state of the pre-GenAI internet, with all the SEO optimization nonsense, clickbait, and supplement peddling everywhere, LLMs are for now actually better than Google for “doing your own research” on many things.
At least at the entry level. Once you want to go in depth, the outcome in my experience is the same as with LLM use on any topic depends heavily on the domain knowledge of the prompter and their ability to steer it.
The author uses the prompts and method from an open-source app that connects to insulin pump, a medical device. I think AI food identification is an experimental feature in the app.
> The prompt was adapted from the one used in the iAPS open-source automated insulin delivery system — it’s a real production prompt, not a toy example.
Exactly. This is not paid software. We assume full responsibility for outcomes when using it. There's a reason it's not on any app store. I'm glad features like this are being experimented with. Not how I would use AI to estimate carbs...
True, but I'm working on a product that's "adjacent to" this sort of thing, and we also have a "food recognition" feature that's marked as experimental. Our users are using it, and now I plan to push fairly hard on at least measuring the accuracy and hopefully exposing those results to our users regardless of how well it performs.
One of the biggest gaps is that people don't understand that food labels are allowed by the FDA to be off by up to 20% in terms of the number of actual calories!
In the real world you need to calibrate your behavior with the results. Are you gaining weight? You'll need to eat less if you want to lose any. You can do all the math with nutrition labels and macros you want but that's all theoretical.
A large part of the effectiveness in counting calories is that you pay more attention and make more conscious decisions and are less likely to "cheat" if you have to enter it in your food log.
It's indeed like astrology. Simply thinking about personality traits and thinking through your life and your desires and goals and current situation is already beneficial to take charge and navigate your life.
I'd take the opposite point of view: just thinking about life, desires, and goals is how people wind up paying $10 to Whole Foods for a slice of "healthy pizza" that is nutritionally identical to any other pizza but comes out of a cute stone oven and is displayed on a wooden platform next to green leafy plants. Vibes astrology is notoriously easy to exploit, both by your own sugar/salt/fat-seeking instincts and by unscrupulous commercial forces, let alone the two working together. The unique thing about calorie counting is that it cannot be exploited like vibes astrology. Not even with a 20% error margin, which is (probably not coincidentally) the caloric deficit targeted by standard dieting advice.
That's an interesting bit, where reducing friction too much can eliminate the side effect that is actually driving the desired results.
Do you want to count calories, or do you want to lose weight? Sounds like it's possible to hyper-optimize calorie counting to the point that it becomes counter-productive...
Some good news from it. If you weigh the food instead of depending on the package size then the labels become much more accurate!!
"Serving size, by weight, exceeded label statements by 1.2% [median] (25th percentile −1.4, 75th percentile 4.3, p=0.10). When differences in serving size were accounted for, metabolizable calories were 6.8 kcal (0.5, 23.5, p=0.0003) or 4.3% (0.2, 13.7, p=0.001) higher than the label statement."
If you look at the table "Deviation of metabolizable calories from label calories" [https://pmc.ncbi.nlm.nih.gov/articles/PMC3605747/figure/F1/] you'll see that most labels even for service side are pretty good and there are some that are really bad.
If you'll look at one of the worst offenders Tostitos, the label has "Tostitos Tortilla Chips - serving size 24 chips", but chips vary a lot in size, so you could have a huge variance in weight. If instead you weighed them, which I do with my chips, I bet the calories are much closer to the label.
Body composition comes down to routine. I've found found I love to eat, but I pretty much eat the same meals week over week, that makes it extremely easy for me to lose or gain weight depending on my goals.
Moreover, the calorie numbers for raw ingredients are much more accurate than for snack foods, where the amount of each ingredient may vary from nominal, even when the total mass is nominal.
So when you cook yourself and you weigh the ingredients used for cooking, you can know the real calorie content with far more accuracy than when buying ready-to-eat food.
The errors either cancel out (if error in both directions) or they work in one direction ("bad" foods like junk systematically underestimate calories, "good" foods like protein powder systematically overestimate calories).
Either way, if you count calories and compare to your weight gain/loss over a few weeks and adjust your calorie target as warranted, assuming the types of food you are eating do not change drastically (e.g. you calibrated on regular diet and now have started an elimination diet), the error bars can be basically ignored.
Realistically the labels are going to be much closer for staples like long-grain Jasmine rice or olive oil, if they're measured by weight.
It's just not that easy to change the nutritional content of a kilogram of a known cultivar of dry rice when it's passed all the standard checks for moisture content, protein content, etc.
What has more calories: 1 lb of peanuts, OR 1 lb of peanuts ground into peanut butter?
I cant find the study, but the peanut butter has more calories since its pre-ground and more bioavailable. Peanuts get chomped up but larger pieces still remain and are not captured by the body.
I feel like you didn't understand the goal of this study
> The DTN-UK stated earlier this year that generic LLMs must never be used as autonomous advisory calculators for insulin delivery. This data is the quantitative evidence base for that statement.
This study is to prove that you should not rely on LLMs
The thing is it doesn't really prove LLMs can't do this, it proves no existing frontier LLMs can do this.
The part where they talk about sampling multiple runs is interesting - it suggests to me that in the next few years as the reasoning process is improved the models may be able to do that autonomously.
My mind really is going to using a dedicated object detection models fine-tuned with nutrition information, but I don't think there's a fundamental reason LLMs can't eventually manage this use case, except perhaps the size of the needed weights being prohibitively large.
This isn't something so fanciful as fusion power, this is reasonably something that might be within the capabilities of object detection transformers. Whether a different prompt/finetuning with a good dataset could make this work is very relevant here.
that is good to know. presented this way i find LLM behavior to be a feature, not a bug. then again i think everything is value add over pen and paper / notepad / spreadsheet and maybe a friend or doctor (or specialized equipment if you need more than calorie in, calorie out). just go exercise and don't be a lard lad
> If calories in, calories out was useful advice rather than a banal statement of physics
it's also wrong, or at least imprecise. fat and bone and muscle all weigh different amounts, at the same volume.
the only way i've been able to explain the science to laypersons is thus:
if you and a friend both weigh 200lbs, but you once weighed 250lbs and your friend has never weighed more than 200lbs; all else equal: you must ingest less calories than your friend to maintain 200lb body weight.
your body will try to "outlast the famine" if you had to lose a lot of weight (or lost a lot of weight for any reason).
That absolutely does not comport with "calories in, calories out". It's also why the people who were never fat have no problem "just eating a donut."
no, i won't cite, this has been published many times in the last decade, 13 years.
Eh, it totally does comport with calories in, calories out. We just don't hook people up to metabolic carts in day-to-day life, but that's really the only way to measure the calories out part.
The physics of CICO are undeniable. Humans don't photosynthesize. The biology of CICO is useless.
- former fat guy. I'm deeply, personally aware of how useless CICO is as dietary advice.
I disagree, it clearly explains that AI carb counting apps are a problem and shouldn’t be used.
They’re writing in a neutral way that reaches their audience without lecturing or being condescending. They lead the reader to the conclusion rather than shoving it at them. I think that’s why it’s triggering so many angry comments on HN, but it’s effective for the audience they’re writing for (non technical people who may need convincing but don’t like being preached at)
If you smack yourself in the head with a hammer and it injures you that's evidence that smacking people in the head with hammers is bad and shouldn't be done, right?
Here we’re at the origin of the tool and get to watch how many people hit themselves in the head before we learn this collective wisdom.
There’s a gap between what the tool will allow you to tell it to do, and what it’s good at. The feedback mechanism to tell the difference is deficient compared to a hammer.
> But the author just took pictures of food & expected a realistic response? Is this genuinely what amounts to a study in AI?
The article explains this: There are apps targeting people with diabetes that claim to count your carbs with AI.
> If you’re using AI carb counting in a diabetes app
Before you dismiss a study, try to understand where it’s coming from.
The authors of the study weren’t stupid. They knew the LLMs would provide poor results. They ran the study to quantify it and create a resource to spread the information in response to the rise of AI carb counting apps.
> The authors of the study weren’t stupid. They knew the LLMs would provide poor results. They ran the study to quantify it and create a resource to spread the information in response to the rise of AI carb counting apps.
Yeah. I think it is under-appreciated that much of science is intended for debugging purposes. Sure, you and I know that X is positive, but what's it actual value? Can we find the causes that make it that way? Et cetera.
I don't believe the authors of this study are stupid.
If there are apps targeting people with diabetes that claims to count your carbs with AI, why haven't those been analysed? That would be a far more effective claim.
I based the study off of the clickbait article that they wrote about the study - i'll read through the study to see whether they analyse that, but it would be far more effective to see if the 'carb-counting' AI app is returning similiar results to the frontier model - that's an interesting result that actually can forward discussion.
> If there are apps targeting people with diabetes that claims to count your carbs with AI, why haven't those been analysed? That would be a far more effective claim.
Because the apps aren’t going to let you submit 29,000 automated requests for statistical analysis.
And if you did, the authors of those apps would just release an update saying they changed models and try to dismiss the study.
The vitriol against this article on HN is sad. Commenters who agree with the article and its conclusions are grasping for reasons to be angry about it anyway
You can commit statistical analysis on frontier models and still use commercial applications as an identifier & comparison.
Criticism is not vitriol - it's possible to make a wider point about being taken aback by the lack of education within AI to the point that there's a critical mass of people using them for calorie counting; but there are many studies on effects of LLMs on psychology etc that are far more effective.
But for me - this is like creating a study that performing algebra & calculus is innacurate on LLMs. That should be common knowledge
Well, for me the comments that insist we don't need to study X because everybody knows LLMs can't do that is a very good justification to study exactly X.
Not to mention that this is now a standard thought-terminating cliché, where someone points out a use case where LLMs don't work at all well and irrate responses protest that LLMs aren't meant to be used in that way. Says who? If you ask an LLM a question and it answers it- then that's an LLM use case. If you can ask the same question many times and evaluate the results then that's an evaluation that is perfectly fine to make.
Yes - my original claim is not to not study it, it's to study it deeper than just surface level, which is my belief at what I've read from the site linked
The linked "click bait" article explains this very clearly as well. It clearly explains the methodology: they took the prompt sent to an LLM by a popular open source carb counting iOS app and sent it, together with five different pictures of food that a typical person might take, to all of the frontier models, and checked the responses. They also explain the purpose: to check the possible accuracy of this approach taken by a real app that real people use.
The fact that you somehow perceived this as an attack on LLMs as a technology is a failure entirely on your part. There is nothing in the article that suggests that people shouldn't use LLMs for other purposes - just a statistical verification of the fact that they shouldn't be used for this one particular thing.
I didn't take anything as an attack on LLMs. I took it as a severe misunderstanding of how technology works. I specifically outline that I would like to see the margin of error even when integrating actual apps that claim to achieve results, rather than using tools that don't.
None of my claim perceives anything as an attack on LLMs, which shows a mischaracterisation on your part of my entire point.
Typical case of the "curse of knowledge". We deal with AI on a daily basis on the technical level, so it's very easy to forget that the "common" folk really still believe that AI can replace dieticians, gym coaches, etc
And what’s the variance & accuracy of their responses? Isn’t comparing the models’ variance to baseline human variance what matters here? It seems like they didn’t do that, and I agree with parent’s call for that kind of baseline.
Having counted calories for years, I don’t think I could reliably estimate the calories or carbs in the example picture of a cheese sandwich. I can make assumptions about the bread and the cheese, but I might easily be off by 2-3x. Calorie counting apps that use text descriptions also have huge variance for the same thing. The problem might be the belief that a picture or description is enough, regardless of who or what is guessing…?
Edit: Ah, I see from sibling thread you meant commercial services are LLMs, I thought you meant there were human-backed services to compare to. Anyway, I totally agree there’s a problem if people rely on AI for safety, but I’m not sure LLMs are the core issue here, it seems like using vague information and guessing is the core issue.
> Isn’t comparing the models’ variance to baseline human variance what matters here?
You seem to be missing the context that this isn't just about diet apps - this is about apps claiming to be able to track carbs sufficiently accurately to be used in a medical context to dose insulin (a substance which can be lethal if incorrectly dosed)
No I understand apps are making dubious claims and implications; obviously claiming LLMs can accurately estimate carbs from a photo is just wrong. But that doesn’t necessarily change my question. Should people use photos to estimate carbs? Can people looking at photos do any better?
The presence of variance in the LLM output doesn’t actually prove anything, in fact I would expect and hope for variance when confidence is less than 1.0. I’m more curious about accuracy of the mean of guesses for different models, for example.
But should any diabetic expect photos to be reliable, regardless of whether it’s an app or an LLM or a human? I know some diabetics, and the people I know do not rely on photos for their safety. They don’t even rely on food labels either (which are far more accurate than photos), they measure their insulin.
It’s probably useful to raise awareness, and useful to scare app makers away from making bogus medical claims - products and scams that make bogus medical claims is of course a practice as old as history. But we can still hold the studies and PR around this up to high standards, right? Even assuming this article & the paper behind it are right, there are reasonable questions here about how to demonstrate the problem and what the baselines are.
It’s worth keeping in mind that trying to prove the bogus apps wrong with a flawed methodology or questionable reasoning or just an overly heavy handed style can cause backlash and do damage to the cause. We’re already seeing that effect play out with respect to vaccinations.
But I don't see them using those commercial services in this study - instead, they're using frontier model companies? Is Gemini advertising that you get a realistic calorie count from a picture? Maybe so - in which case i'd take it back!
The commercial services likely also have frontier model dependencies...
The opening to the actual paper is quite explicit that (i) other studies have already tested commercial apps with with unimpressive results and (ii) a popular open source app for carb counting directly relies on API calls from these frontier models, and this research batch tested the images used the exact same models and prompts as the popular open source app.
A carb counting app might use API calls to these frontier models and then do some kind of analysis. It could see if different models agree or not, or multiple calls, and with how much variance.
So it would be more accurate to test the apps rather than the APIs, unless the goal is to warn people that just open chatgpt and ask there.
The open source app could in theory do that, but the paper's authors would be able to determine whether it did or not by reading its code, which they evidently did to replicate the API calls it made with their own script.
(And of course it would also be far more tedious to submit each picture 500 times manually using an app and manually log the response than using a script which is designed to collect the data automatically as fast as API rate limits permit)
Great point - and i'd love a study to address that. If the study pointed out that X services sit perfectly within the analysis found, I think that would be a fantastic study that would be enlightening & useful to show.
The app the study is based on is open-source, so you yourself can verify that it does indeed just call a frontier model with the same prompts used in the study
That's not really the same thing as what I'm saying - which is to investigate the applications specifically advertising AI calorie counting capabilities
They investigated an open source application specifically advertising carb counting capabilities, replicated its prompts and API calls in a way optimised to collect data from 26000 queries (which is a lot to do using a GUI!). They also note other people have already done [necessarily] smaller scale studies of the commercial AI carb counting apps and been similarly unimpressed by the responses.
This is all in the first few paragraphs of a preprint paper describing the research in considerably more detail which is linked at the bottom of TFA
Meta: enjoying nearly half this HN thread being arguments that surely people care about what's in their food don't ask ChatGPT for comment instead of looking it up properly, and most of the rest of it being people who apparently care what's in a research paper asking HN for comment instead of looking it up :)
It amazes me how much people try to build AI systems relying on nothing more than the models knowledge. I suspect a great deal of "failed" AI experiments we keep reading are people just not having any idea how to use AI at what its good at.
As someone who used to do this. OpenAI models refuse to look up calories unless you explicitly tell them to and even then it is a hit and miss even if you tell them exactly what the product is. Easiest way to get good calculation is to just take a photo of the nutrition label or feed that info in by hand.
Funny thing is 4o did look up calories but I guess it was too good for this world
- AI will solve all of our problems
- No not like that!
Are the trillion dollars sloshing around the AI economy well-invested if the refrain is always “you’re holding it wrong”?
So we’re trying to define, through trial and error, what problems “AI” will actually solve, and this paper is one of the many cobblestones on that road.
"AI can solve this one problem, but it needs X, Y, Z, because it's not a omnipotent god entity"
"I tried it without any of those things and it didn't work - this is worthless tech!"
I don't know if more accurate calorie counting using AI exists - but it's like being upset that the screwdriver isn't gluing wood. AI is far more than frontier LLMs.
There are plenty of positions on the spectrum from “omnipotent God entity” and “Casio SL-300SV”. What does the current valuation of LLMaaS companies represent though?
LLMs are certainly not worthless, that’s a strawman in the same way my statement “AI will solve all of our problems” is a strawman. The question of their worth is being explored.
“AI is a black box that can solve problems”. Which problems? How consistently? At what cost? How quickly?
- A queryable large vector database containing calorie counts for specific meals.
- A vision model specifically trained on food images with labelled data containing approximate calorie counts.
- OCR model allowing reading of barcodes + calorie information.
A model trained to ask for additional context & information (e.g for pasta - please provide a photo of the original sauce tin/ect), (please approximate the weight of X meat)
I don't know how accurate integrating all of those aspects would be - and you could argue the end user would probably be incredibly annoyed and it wouldn't be a good app - but I'd argue you'd at least need that if you're developing an app for diabetes management.
> But the author just took pictures of food & expected a realistic response? Is this genuinely what amounts to a study in AI?
Reminds me of that one youtube video (I forget who it is so I have no idea how to pull it up) where he turns on the camera on his phone for ChatGPT and asks it what everything it sees weighs, then puts it on a scale, and ChatGPT was never right, ever, which makes sense, I couldnt tell you what most things weigh on sight alone either, but ChatGPT often got it dramatically off. I got the feeling he thought it was terrible AI for this, but I don't think a model looking at an image of something and trying to guess its weight / calories / etc... is a reason to call an AI model bad...
People do not understand this stuff at all. So many times I've seen people post the output from one of these services with full confidence that it has to be correct because it came from a computer.
I did this too! For months (almost a year) I used descriptions, pictures, and measurements of food to get rough calorie counts. My diet is pretty simple and repetitive.
I would occasionally check the estimates, maybe once every few days for meals I wasn't already pretty sure of, and it was generally accurate. Where it was extremely inaccurate was on portions, and anyone who has dealt with computer vision could tell you, you can't get scale from a picture. So I'd have to weigh some meals or ingredients, which would generally make things more accurate again.
So, I think it's possible, but you need multimodal data and grounded with regular checks.
About once a week I ask ChatGPT to give me a reasonable diet for recomp with weight loss. It consistently insisted I have at least 7 meals consisting of at least 30g of protein per meal, but the protein source can't be whey or casein. When I ask "why" it cites a bunch of studies ... but most of those "studies" are N=1 of a college or Olympic level athlete. If, instead, I grab a large scale lateral analysis, it says "3 meals" with about 1/2 of the protein.
It'll defend both sides (mutually contradictory) to the death. NOTHING will budge it from its initial stance.
To be fair this is a reflection of the general state of nutritional science and the actual answer seems to be "it depends on your genes".
Some people do well on 6 small meals, others do well on no breakfast and two large ones. Studies can't tell you anything useful about that, you have to experiment and find out what works best for you
The answer does not really depend on genes. There are personal preferences, there are sex differences (women prefer more carbs), and the biggest component is where you are and in which direction do you want to go to.
But in terms of physiology the answer is quite clear:
1. The protein is the most important macro to get, no matter if bulking or cutting. It is the building block.
2. Whatever the amounts (0.8g-1.8g/kg of bodyweight, depends a bit on a situation and the willingness to lose some potential marginal gains), try to divide your daily protein somewhat evenly between meals.
3. Pareto principle, you get the most benefits by having 3 meals. 4 if you really care about small differences and want to optimize. 5 meals give negligible additional benefits, for professional athletes who want to be anal.
4. So basically eat at least 3 meals and up to whatever works for you practically speaking.
Optimum meal timing in particular I believe is heavily influenced by genes - I have friends who never eat breakfast, survive on black coffee until 1PM, then eat a lot in the evening and feel good doing it. If I do that I feel terrible.
So yes eat 2g/kg protein but the best way to time that in terms of meals, best specific foods to eat etc is definitely influenced by your genes
But physiologically speaking there is no difference in the optimal recommendation in terms of protein synthesis: protein should be evenly divided between evenly spaced meals.
If people decide that this non-optimal way of life fits them better, then go for it. But let's stop with the "I feel better this way, therefore biologically my body works differently than any other human body." BS.
The author is doing what a non-sophisticated user would be doing, or would want to be able to do, and estimating calories based on a photo has been an often-cited potential or promised AI use case in recent years years. It makes a lot of sense to test current general-purpose AI’s performance on it as a reality check.
It also exemplifies how current AI offerings are still quite limited in their capabilities, because one would expect that they’d do the intelligent thing on their own that you had expected, instead of the user having to come up with a working methodology.
There are apps in the app store right now that pretend to do this kind of thing, so having somebody actually show that it doesn't work is valuable, even if we already knew the outcome ahead of time.
I suppose i'd much rather a study analyse the apps in the app store that are attempting and claiming to do that kind of thing - rather than the base model they might be using.
> The prompt I used asks each model to return a confidence score (0 to 1) for every food item it identifies. All four models dutifully returned confidence scores for 100% of items. Surely we can use those to filter out bad estimates?
This is a problem with the companies selling the AI models, not the customers. It is their responsibility to inform consumers about the limits of their services, and to train the models to say "I don't know, there is not enough information".
> There's an incredibly serious lack of education with how LLMs & carb-counting works. This entire article would be better suited to astrology.com than hackernews.
This is because the people who promote these technologies, and the companies that sell these technologies, engage in a massive amount of puffery (aka hyperbolizing aka just straight telling lies).
These technologies are painted as the magical solution to whatever problem you have (all it costs you is a few tens of thousands of tokens, aka your water supply). There is literally nothing they CAN'T do if you will just let us build these gigantic small town destroying, noise polluting, water and electricity hungry 'AI data-centers'. So that we can use those datacenters to sell you more tokens to put into their slot machines.
Honestly it's scary how misunderstood this is by the general public, the media and EVEN scientists.
There is a shocking amount of Computer Vision tasks where the scientists claim you can get X info from a picture of Y and it's like, even with ML/AI you can't extract data where there isn't any. The fact I can add an arbritrary amount of high-calorie fat to a meal without changing the appearance by defintion shows it's pointless. A 1000 calorie and 100 calorie milkshake can look identical, and you'd have no way of working that out via an image even if it was a super-intelligent system.
Similarly I see it in things like extracting material of an object from an image of it in serious research papers, which for the same reason cannot be done, since how an object looks has very little to do with what its made of, else painting and other art would clearly be impossible. The information is just not there within the data.
It’s like CSI “enhance!” AI image upscaling. People will do it, see it fabricated details, and then draw the wrong lesson from it, that “AI fabricates things!” when that is exactly what they asked the model to do and there is no magic math that would extract ground truth that was never in the image to begin with.
> But the author just took pictures of food & expected a realistic response?
Outside our tech-enabled bubble, there are folks who have been sold the idea that ChatGPT et al is a miracle worker capable of replacing dieticians, gym coaches, psychologists, etc.
So it's VERY plausible to believe that there are folks out there snapping pics of their meals and asking GPT to spit out nutritional values.
That's a good point - and I think there's a wider core lack of knowledge outside of the bubble.
I suppose I just expected this study to be a little less 'water is wet' which made me dissapointed, but that may be coming at it from a more technical perspective.
I guess that's partially my original point summarised; I'd like to see this criticism levied at the apps advertising it - rather than using something that doesn't
> But the author just took pictures of food & expected a realistic response?
You say this (and I agree), but I know of quite a few companies in this area, including a couple accelerated by YCombinator, and that's pretty much 100% of what they do in their backend.
> There's an incredibly serious lack of education with how LLMs & carb-counting works
Oh! Do the vendors offer trainings to make sure the users understand how LLMs work? If not, surely, the LLM itself is trained to know its limitations and politely decline in situations like that?...
The #1 use case for this tech is "here's a problem I don't feel like solving, let's have a computer do magic". It's how it's advertised on TV, it's how it promoted in the software I already use. Food preparation? Travel planning? Shopping? Tutoring your children? You can do anything now!
I just talked to a realtor who will make a killing on a real estate transaction. Instead of offering human insights, they sent me "AI reviews" of several properties. The AI has never been to any of these properties and has no idea how they actually look like. But I guess it's how we operate now as a society.
If you go to eBay, every other listing description for used items is AI-generated. This is an official platform feature for sellers. The AI doesn't know the condition of the item or what's included or missing. Doesn't matter, it's magic. It's AGI, it will figure it out.
Most of the uses of AI I encounter as a consumer are like that, and the companies selling this tech are 100% complicit.
yes - the vast majority of labs offer a whole host of training material for users to understand how LLMs work. There's entire course websites created by each of the major vendors specifically to understand how LLMs work. Here's a couple of examples:
>Getting the most out of generative AI depends on what you put in. To quote our Outreach team, "It's a tool, not magic!" As the technology evolves, more and more chatbots are designed for specific purposes.
All of the above are completely free, you could start 3 coursers today that specifically teach you how AI tools work in practice. yes, this is different from the marketing that some of these tools use, but these resources are there, free and available.
Maybe we need to create a form of driving license for responsible AI use, but saying the resources don't exist is not accurate
It doesn't really matter if the model cannot make a good educated guess about calories in the food if it cannot give a consistent response given the same input.
as a t-1 diabetic, this is exactly what we do for nearly every meal we eat, especially in restaurants, look at it and try to estimate the number of carbs.
a realistic response? What's a realistic response to "how many calories are in an avocado?"
If you are counting calories, you don't want the answer to "how many calories are in the average avocado?", you want to know how many calories are in this avocado. Remember that bodyweight is roughly linear with BMR, so a 10% error in calorie counting is an extra 10% of bodyweight.
How about instead of blaming the user for not understanding how AI works, the AI makers stop letting their chatbots answer questions so confidently that they clearly can't answer...
If I ask the AI about some health issue, it says something along the lines of warning I'm not a doctor etc. So if I show it a picture and ask it to tell me the carbs, how about a warning telling me it can try, but that it probably wont be very accurate.
> This entire article would be better suited to astrology.com than hackernews.
I laughed, but you nailed it. Sadly so many people lack even basic understanding of LLMs and the ViT tower that makes it vLLM, that I expect a whole industry, similar to fortune telling, to emerge out of it.
> But the author just took pictures of food & expected a realistic response?
If someone sent me a picture of a meal and asked me what the macros were or how many carbs this is, I would say "I can't tell from a photo. Nobody can". The problem is that current LLM chatbots don't seem to have a concept of telling you "I don't know", "you can't do that" or even "you're wrong".
You can say that somebody shouldn't trust an LLM for this but it's going to be a problem that LLMs give nonsencial answers. What I find particularly amusing is that there are still technical people (generally, not anyone specifically) who seem unable to acknowledge that LLMs hallucinate and lie.
There was a post on here recently that I couldn't find with some quick searching but the premise basically was that chatbots were trained like neurotypical people: A lot of affirmation and basically lying. Separately someone else characterized this NT style of communication as "tone poems" [1]. I keep thinking about that because to me that's so accurate.
Dunning-Kruger is a common refrain on HN, for good reason. Another way to put this is how often people are confidently wrong. I really wonder if this is an inevitable consequence of NT communication because most neurodivergent ("ND") people I know are incredibly intentional in what they say and mean.
>> But the author just took pictures of food & expected a realistic response? Is this genuinely what amounts to a study in AI?
The aim of the study was to understand the variation in results returned by models and how that could cause risks for patients using those models. The main result was measuring within-model variation.
We aimed to characterise the within-image reproducibility of carbohydrate estimates from four large language model (LLM) vision APIs and to quantify the clinical risk for insulin dosing, stratifying accuracy by reference value quality.
Methods
Thirteen food photographs were each submitted 495–561 times to four LLM vision
APIs (GPT-5.4, Claude Sonnet 4.6, Gemini 2.5 Pro, Gemini 3.1 Pro Preview) using
an identical structured prompt adapted from the iAPS automated insulin delivery
system (26,904 total queries, temperature 0.01). The primary outcome was within-
image variation (coefficient of variation [CV], range, distributional normality).
Secondary outcomes included accuracy against reference values for nine images,
stratified by quality tier (packet label, weighed/measured, portioned, or visual
estimate). Clinical risk was translated at an insulin-to-carbohydrate ratio of 1:10.
>> I'd like to see this study done using any kind of actual grounding knowledge, seeing what mistakes AI makes when attempting to query ground truth from picture analysis - there would at least be an interesting result methodology in that
The ground truth was established by the author. There's an appendix in the pre-print (Appendix I) that describes the methodology. Methods are described in page 4 of the pre-print:
Reference values for accuracy analysis
For nine of the thirteen images, the author estimated the carbohydrate content using
methods described in Appendix 1. Reference quality was categorised into four tiers:
Tier 1 (packet label): Carbohydrate values derived from manufacturer nutrition
labelling. Two images (cheese sandwich, soup with bread) used bread with
labelled carbohydrate content of 20 g per slice.
Tier 2 (weighed/measured): Portions directly weighed and cross-referenced
with established composition data. Three images (Bakewell tart, bakery cookie,
breakfast burrito).
Tier 3 (portioned): Portions estimated by the author (not weighed) and
combined with USDA composition data. Three images (roast dinner, chilli con
carne with rice, stuffed pork loin).
Tier 4 (visual estimate): Portions and composition estimated from visual
inspection. One image (churros).
For the four restaurant dishes (pizza capricciosa, eggs benedict, crema catalana,
paella), no reference value was established. These images were used for the primary
reproducibility analysis only.
Carbohydrate values follow the EU convention with dietary fibre excluded.
I've slowly began to write about abandoned side-projects. It's actually incredible how much you end up re-picking back up.
A gaussian splat converter that I made and abanonded became incredibly useful a few months later when I needed to do a visualisation for a really specific environment
I really dislike the default AI slop css - if you're going to do this - please have a design language and taste ideas beforehand. It can help so much in refining the look.
Genuine piece of feedback, as soon as I see those gradients + quirks. My perception immediately becomes - you put no effort into finding your own style, therefore you will not have put effort into creating this website.
> Genuine piece of feedback, as soon as I see those gradients + quirks. My perception immediately becomes - you put no effort into finding your own style, therefore you will not have put effort into creating this website.
yes. if instagram started performing intensive JPEG compression that made photos choppy and unpleasant, I would consider that an intentional degredation of the software.
reply