A personalized nutrition and fitness coach that uses multimodal AI and behavioral context to generate plans built around you as a whole person, not just your last meal.
Most nutrition apps start with what you ate. They log calories, compute macros from a BMI formula, and return a plan that looks the same for everyone at your height and weight. The plan ignores your sleep, your stress, your schedule, and what your body actually looks like.
The insight behind NutriCoach: personalization has to start with the whole person. Combining a photo (body composition context) with goals and lifestyle signals can produce recommendations that actually fit, and plans that fit get followed.
In short: my mandate was to build a health product that used AI in a meaningfully differentiated way, not just as a wrapper around calorie math.
I audited the top 6 nutrition and fitness apps (MyFitnessPal, Noom, Lose It, Calibrate, MacroFactor, Whoop) and ran qualitative interviews across three user archetypes: beginners trying to lose weight, intermediate users building muscle, and active people optimizing performance. Two artifacts shaped the strategy.
| Feature | Top Apps | NutriCoach |
|---|---|---|
| Body composition input | ❌ None | ✅ Photo + Multimodal AI |
| Integrated meal + workout | ❌ Siloed | ✅ Unified |
| Lifestyle context | ⚠️ Basic | ✅ Sleep, stress, schedule |
| Manual food logging | ✅ Required | ❌ Not needed |
| Personalization depth | ⚠️ BMI-based | ✅ Multimodal AI |
| Setup time | ⚠️ 15+ min | ✅ Under 3 min |
Three consistent failure patterns emerged from the research:
Shipping an AI product requires more than choosing a model and writing a prompt. Before building anything, I worked through the full set of decisions an AI PM needs to own: model selection, data inputs, cost modeling, infrastructure, and eval design.
The product hypothesis was clear: if we reduce input friction (no food logging) while increasing output relevance (photo + lifestyle context), plan completion rates improve. The question was how to build it responsibly.
I chose Claude Sonnet for its multimodal capability (vision and text in a single API call) combined with strong instruction-following for structured output. Compared to GPT-4o, Sonnet offered comparable quality at lower cost for this use case, which matters when modeling unit economics at scale.
The input pipeline conditionally constructs a multimodal message (base64-encoded photo + structured lifestyle context) or falls back to text-only if no photo is provided. This mirrors good production AI design: graceful degradation when optional inputs are absent, consistent output format regardless of input mode.
The prompt explicitly instructs the model to assess body composition, muscle development, and posture from the photo and state that assessment at the start of its response before generating the plan. This makes the personalization auditable rather than a black box, a user or reviewer can check whether the stated visual assessment is actually reflected in the plan's calorie target, protein target, and workout intensity. This was verified by testing with multiple different photos under identical text inputs and confirming the stated assessment and resulting plan differed accordingly.
A single NutriCoach generation uses approximately 2,500 input tokens plus ~1,600 image tokens, at Claude Sonnet pricing of ~$3/M tokens. Estimated cost per generation: ~$0.02. At 10K MAU with 2 plan generations per month, that's roughly $400/month, well within MVP budget before introducing a paywall.
An API key can never be exposed client-side. I built a Cloudflare Worker as a serverless proxy that handles CORS, authenticates with Anthropic, and forwards requests. The free tier handles up to 100K requests per day with zero backend infrastructure, appropriate for an MVP and easy to swap for a Node backend at scale.
I ran 11 test generations against a 5-criterion rubric: structural completeness, caloric accuracy, photo-responsiveness, dietary compliance, and actionability. The test set deliberately varied photo, goal, activity level, dietary restrictions, and input completeness, including edge cases like no photo provided and minimal field completion.
Results: structural completeness, caloric accuracy, actionability, and photo-responsiveness passed consistently across all runs, including the no-photo fallback case, which produced a transparently caveated estimate rather than a silent guess or a failure.
Dietary compliance surfaced a real bug: two of three restriction-based tests failed. A nut-allergy test included almond butter mislabeled as a "seeds-based alternative," a direct violation. A gluten-free test included unspecified "granola" and "whole grain toast" without confirming the compliant version was used. In both cases, the Grocery List section correctly included the right substitution, but the Meal Plan section did not consistently apply it, a cross-section consistency failure, not a misunderstanding of the restriction itself. This was fixed with an explicit instruction requiring the model to label compliance status on ambiguous ingredients and cross-check meal-plan items against grocery-list substitutions before finalizing output.
One additional finding worth noting honestly: when no goal was specified, the model inferred an intended goal directly from the photo's body composition rather than asking for clarification, a deliberate but debatable UX choice worth flagging as a tradeoff, not simply a strength.
The 3-step intake flow was designed to keep setup under 3 minutes, directly addressing the logging fatigue problem. Output is tabbed across 4 views so users navigate without scrolling a wall of text. The MVP targets fitness-motivated users 25-40, with a growth loop built around plan sharing and a retention hook of weekly plan refresh based on progress check-in.
With those findings in mind, I built a strategy around flipping the input model entirely. The core product decisions:
Upload a photo, share your goals, and get your personalized plan in seconds.
Based on your photo, goals, and lifestyle