It was 3 AM. Again. My toddler was screaming, I was running on fumes, and somewhere between the fourth lullaby and the fifth trip to his room, I thought: “There has to be a better way.”

Spoiler: there was. After trying various traditional methods without much success, I decided to turn to AI for help.

TL;DR: I combined Frigate (AI-powered video surveillance), a custom CNN model, and Babybuddy to automatically track my toddler’s sleep patterns. The model detects not just if he’s in bed, but whether he’s lying down or sitting up—all in ~15ms with 99%+ accuracy.

The problem

Our toddler was having trouble sleeping through the night. He would wake up multiple times, cry, and have difficulty falling back asleep. This was taking a toll on our entire family, as we were all exhausted from the constant interruptions.

20-30% of toddlers experience sleep disturbances.1 In Western countries like the United States, Canada, and Australia, approximately 25% of parents report their babies or toddlers have sleep troubles2.

A survey of 1,300+ parents found that parents lose approximately 1,000+ hours of sleep in the first year alone3. That’s over a month of sleepless nights!

The Canonical solution

We read books, consulted pediatricians, and tried various sleep training methods. While some of these approaches provided temporary relief, the underlying issues persisted.

As a last resort we decided to try one of what felt to be a big scam… Sleep consultants. We hired not 1 but 2 sleep consultants who provided us with a structured plan to help our toddler develop better sleep habits. The consultant advertised their methods as “tailored to your child’s needs” and “based on the latest research in child development.” while in reality we spent about 1000$ to receive very well known generic advice that we could have found online (or already read in books).

So we were back to square one, feeling frustrated and desperate for a solution.

If you can’t measure it, you can’t improve it

Before our child was born, I’d already installed Babybuddy, an open source baby tracking web application, to track feedings, diaper changes, sleep, and other activities.

Babybuddy

The picture above shows the sleep patterns of our toddler over some weeks. We could see the frequent awakenings and the total sleep duration.

Tools like this allowed us to collect data on our toddler’s sleep patterns, which we could then analyze to identify potential issues.

Unfortunately, when you are a sleep deprived parent, you don’t have the time nor the energy to analyze data nor to update it regularly.

Enter AI

During a particularly rough night, I stumbled upon an article discussing the use of AI in healthcare and parenting. This sparked an idea: what if we could leverage AI to track automatically our toddler’s sleep patterns and then eventually provide personalized recommendations?

After all, AI has been successfully used in various domains to analyze complex data and provide insights.

Anyone familiar with the self-hosting community knows that Frigate is the go-to open source solution for local AI-powered video surveillance. It uses TensorFlow Lite models to perform real-time object detection on video streams from IP cameras.

I already had an ip-camera installed in the toddler’s room for monitoring purposes already hooked into Frigate. So I decided to explore whether I could use Frigate’s AI capabilities to monitor my toddler’s sleep patterns.

What does Frigate offer?

Well, frigate is not meant to be used for sleep tracking, so I had to get creative. Frigate has APIs that allow you to access detected objects and events. I set up Frigate to monitor the video feed from the toddler’s room and configured properly to record up to 24h of footage.

This gave me access to raw training data (the camera frames) I could use for later. Frigate also provides insights about movement in the room, which I could use to infer sleep patterns such as deep & light sleep.

What I (and mommy) want

For our son to sleep through the night!!

I would like to have:

  • Sleep patterns properly populated into babybuddy automatically
  • More insights about sleep quality (eg: number of awakenings, total sleep time, sleep efficiency, etc.)
  • Recommendations to improve sleep based on the data collected

The plan

So the idea was:

  1. Use current homelab systems such as Frigate, BabyBuddy, Home Assistant
  2. Build multiple AI models to analyze the video feed from the toddler’s room
  3. Feed in data automatically into babybuddy
  4. Provide actionable daily insights based on the data collected
  5. Keep everything local-only to preserve privacy

Implementation

I had Claude write for me a script to download past 24h images at 5s interval to have some training data for the CNN model it was going to help me build. To refine data I always used claude to perform tedious tasks like download the footage, extract frames, label them, etc.

Claude code

YoLOv8 + Transfer Learning

We used YOLOv8 as the base architecture for the CNN model. It’s a state-of-the-art object detection model that is both fast and accurate, making it suitable for real-time applications with low inference time like this.

We used a technique called transfer learning4, where we took a pre-trained YOLOv8 model and fine-tuned it on our specific dataset of toddler sleep images. This approach significantly reduces the amount of data and training time required to achieve good performance.

Two main classifiers were needed:

Presence classifier:

  1. empty - no toddler in bed
  2. baby_alone - toddler alone in bed
  3. baby_with_adult - toddler with adult in bed (likely mommy trying to soothe him back to sleep)

Posture classifier (only when baby_alone):

  1. lying_down
  2. sitting_up

I then proceed to label some of the frames keeping classes as balanced as possible.5

One challenge was the camera angle: it’s mounted almost top-down above the bed. While this made presence detection trivial, distinguishing between sitting_up and lying_down turned out to be surprisingly tricky from above, a sitting toddler doesn’t look that different from a lying one (to the AI “eye”). This required collecting significantly more training samples for the posture classifier than I initially expected.

After only a couple of training iterations, the models were already achieving >99% accuracy on the validation set.

Connecting the dots

With the classifier working reliably, I had Claude build a simple web interface so I could monitor the predictions from my phone during critical moments—like bedtime. This let me validate the model’s accuracy in real-time.

Web interface

The next step was to connect everything to BabyBuddy’s APIs, automatically logging sleep data without any manual input.

Provided the necessary API endpoints and authentication details, Claude generated the code to push sleep data into BabyBuddy based on the model’s predictions.

Fighting feature creep

Once the basic system was working, my brain started doing what engineer brains do:

“I should use the audio profile from the camera to detect crying…”

“Wait, why don’t I use the motion data from Frigate to detect restlessness?”

“Oh wow, I could even add a sensor in Home Assistant to track room temperature and correlate it with sleep quality!”

I had to stop myself. The simple classifier was already solving the problem. Every additional feature meant more complexity, more things to break, and more time spent tinkering instead of sleeping.

Sometimes good enough really is good enough.

Results

No more manual logging

The system now runs 24/7, quietly logging every sleep event to BabyBuddy with minute-level precision. I don’t even check the dashboard anymore each morning. When we need to, we just ask Claude to summarize the night using the BabyBuddy APIs, and we get a clear picture of how things went.

Before this, my wife and I would often disagree on how many times our toddler woke up—was it three times or four? Did that 2am crying spell last five minutes or twenty? These fuzzy memories led to unreliable data, which made it impossible to spot patterns or make informed decisions about naps and bedtime.

Hidden insights

The system also answered questions we couldn’t even ask before:

  • Was he awake but silent, hoping to go unnoticed?
  • How long did he sit there before the crying started?

These are things parents simply can’t know without continuous monitoring. Turns out, sometimes he’d be awake for 10-15 minutes before making a sound—information that changed how we thought about his sleep patterns entirely.

The uncomfortable truth

The data revealed something important: we were part of the problem.

The visualizations showed us patterns we hadn’t noticed… how our decisions sometimes made things worse, not better. Our toddler still doesn’t sleep through the night every time (sigh), but the improvement over the past few weeks has been undeniable. We’re making data-driven adjustments instead of guessing, and it’s working.

Of course AI alone did not solve our toddler’s sleep issues—we did. But having accurate data made it way easier to identify what worked and what didn’t.

Conclusion

This project started as a sleep-deprived experiment and turned into something genuinely useful. By combining off-the-shelf tools like Frigate and BabyBuddy with a custom-trained model and Claude as my coding assistant, I built a fully automated sleep tracking system in just a few days prompts.

Would I recommend this approach? If you’re comfortable with self-hosting and have a camera already set up, absolutely. The time investment paid for itself within the first week of not having to manually log anything.

Turns out, the best parenting tool is one that works while you’re too tired to think.

Footnotes

  1. Montgomery P, Dunne D. (2007). “Sleep disorders in children”. BMJ Clinical Evidence. 2007:2304. PMC2943792 ↩

  2. Mindell JA, Sadeh A, Wiegand B, How TH, Goh DYT. (2010). “Cross-cultural differences in infant and toddler sleep”. Sleep Medicine. 11(3):274-80. DOI:10.1016/j.sleep.2009.04.012 ↩

  3. Snuz Sleep Survey (2022). Survey of 1,300+ UK parents found 70% lose an average of 3 hours’ sleep per night in their baby’s first year (~1,095 hours annually). Source ↩

  4. Ribani R, Marengoni M. (2019). “A Survey of Transfer Learning for Convolutional Neural Networks”. 32nd SIBGRAPI Conference on Graphics, Patterns and Images Tutorials. pp. 47-57. DOI:10.1109/SIBGRAPI-T.2019.00010 ↩

  5. Buda M, Maki A, Mazurowski MA. (2018). “A systematic study of the class imbalance problem in convolutional neural networks”. Neural Networks. 106:249-259. DOI:10.1016/j.neunet.2018.07.011 ↩