Summary

During this chapter, we extended our audio visual capabilities we developed in the last chapter into real-time character lip syncing. We first put together a novel lip sync example using just our basic audio visualizer. From there, we looked at how we could make our lip syncing more realistic and natural by understanding speech phonemes. After that, we ran through an exercise on understanding how to classify speech phonemes by using our audio visualizer. This lead to our need to animate the characters jaw and facial muscles using a combination of bone and blend shape animation. We also took time exploring by attempting to model each of the base speech phonemes. With all that knowledge gained, we then put everything together and tested the real-time lip syncing. Finally, we looked at a component that would allow us to use the microphone to record directly to Unity. Something...