Maybe create a gesture-capture system to help people work faster? But why stop there and not drive a virtual avatar with it? Then why not build a system that can control a small robot? Easy, right? Not so fast…
The situation is that Fable suggests some solutions, but integrating them without a deep understanding of what the frameworks actually do reminds me of the YOLO approach I tried on web pages half a year ago. I ended up with a bunch of spaghetti and unnecessary code because Sonnet thought it might be useful.
So, what actually worked? Face and gesture recognition. Great. I can’t say anything bad about that. It’s fast, doesn’t require much AI, CUDA, or MLX, and doesn’t need any fine-tuning. I think it would work even on a very small setup, not necessarily a Mac with unified memory.
But that was the last thing that worked out of the box. The rest, such as passing data to Kalidokit, works only partially:
- Face emotion detection and tracking work very well; lip-syncing does not work at all.
- In multiple iterations, the limbs often changed sides. I move my right hand, the avatar moves its left. I ask for a fix, it swaps them, but in the next iteration it introduces the bug again.
- Hands and fingers are mostly a disaster. It fixes some gestures and introduces new bugs in others. It cannot even consistently swap hand sides correctly from inner to outer.
This was my test to see where we are. Two sessions, maybe twelve iterations in total. So far, I’m not amused.
The most important question is: how do we improve this? It won’t be just about writing a better prompt or giving the AI more context. You need to understand the frameworks involved.
For example, I know how Java and Spring work. When something doesn’t work, I don’t just tell the AI that I can’t see something on the page. My intuition helps me suggest where it should look or which solution is worth trying. The problem space is already narrowed down, and the AI can usually fix it much faster than I could. Eventually, though, I would have found the solution myself.
With a project like this, I can’t even ask the AI the right questions. It’s similar to a project manager or product owner trying to describe a new feature without understanding the underlying technology—they often use the wrong terminology, so the conversation goes in the wrong direction. That’s exactly why my small PoC didn’t work (yet).
The first step is learning to ask the right questions.