How I built a personal expense and budget tracker with Flutter and Gemini

I built Peggy, a personal finance and budget tracker app for people who prefer manual tracking with privacy in mind. You can download it on Google Play and the App Store.

This article explains the exact architecture decisions behind the app, why each decision matters, and what I would recommend if you are building a similar product today.

The core product challenge was speed without trust issues. Logging an expense had to feel instant, even on unstable networks, while money data and AI behavior still had to remain predictable.

I used Flutter for a single mobile codebase, Supabase for backend services, Hive for local cache, and Gemini through a secure server-side proxy. That combination kept the app fast to iterate on and safe to ship.

The first key decision was cache-first data flow. Each screen subscribes to local data first, then triggers a server sync in the background. Users see data immediately, and refreshes happen without blocking interaction.

The second key decision was strict money modeling. Every amount is stored as integer minor units with an ISO currency code. This avoids floating-point drift and keeps calculations stable across budgets, summaries, and multi-currency displays.

The third key decision was AI through a backend boundary, never directly from the client. The app sends transcript context to a server function, and that function calls Gemini with a strict JSON response schema. This removes API keys from the app binary and makes parser behavior testable.

I constrained the AI output to six fields only: description, amount, transaction type, category, recurrence, and emoji. Constraining output shape is what turns an LLM feature from a demo into a production component.

Domain rules were also encoded in the prompt. For example, the meaning of income and expense changes by planning mode. In debt tracking, a payment reduces debt while a new charge increases it. Those semantics need to be explicit or the model will drift.

Voice capture is where the app earns trust or loses it.

Voice capture follows a simple reliability loop: listen, transcribe, parse, validate, then confirm. If parsing fails, the app returns a clear recovery state instead of silently creating broken records.

The hardest part was handling incomplete speech without punishing fast speakers. Early versions cut recording too aggressively, so the parser received half-finished intent like "coffee fi..." or missing amounts. The fix was not only prompt tuning. I had to improve the state transitions around recording stop, add better tolerance for natural pauses, and keep the user in a review state until the transcript looked complete enough to parse.

Another challenge was deciding when to trust AI output. If the parser returns a description but no amount, auto-creating a transaction is dangerous in a finance app. I explicitly gate creation on minimum valid fields, then show recovery UI when confidence is low. That one guardrail eliminated a large share of bad entries and did more for trust than any animation or visual polish.

Observability was built in from the start. I tracked where users start voice capture, where parsing fails, and where they abandon flows. Product decisions became much easier once I could see real drop-off points instead of guessing.

If you are building your own finance tracker, start with three non-negotiables: strict money types, secure AI proxying, and cache-first UX. You can evolve everything else after these are stable.

The highest leverage mindset is this: design your data contracts before your UI polish. A clean contract between client, backend, and AI parser prevents most painful rewrites later.