How to make auto-generated videos feel human-like

While everything we do is a delicate balancing act that will enable us to pass the Video Turing Test, our goal for now isn't to make the absolute most human-like automated video. Instead, we strive to create a great experience that doesn't distract the viewer.

Wabi-sabi, or the art of asymmetry, is the name of the game. We've worked hard to include flaws in what is otherwise a very precise set of movements and actions. The right amount of human-like chaos makes the end product feel, well... right.

When it's perfectly imperfect, the viewer forgets the video was made by automation and not a human. When we achieve that, we pass the Video Turing Test.

There are two main categories that make up our version of the Turing Test: voice and human behaviors.

Video Voice

We use text to speech generators to make the video voice overs. Videate utilizes voices from several different providers, including Microsoft , Amazon Polly, and IBM Watson. Providers have gotten incredibly good at making human-like text-to-speech voices. It's already difficult to tell the difference between a synthetic versus human voice.

One component to keep in mind when picking the right voice is timbre. Timbre is the quality of sound. The other component is the shape of the sentences - inflecting in the right way, and emphasizing the right words, pausing enough (like where a person would take a breath) and handling punctuation properly.

Providers use Al to figure this out. Synthetic speech handles short sentences just fine.

But the longer a sentence runs on, the harder it gets to make it sound just right. That's usually where the listener starts to notice the voice isn't human.

This is why we recommend customers adjust their video scripts to align more with how people talk versus how we write documentation. Shorter, more direct sentences translate better in the synthetic voiceover. 

Harnessing the power of text-to-speech generation

We listen and talk to devices all day long: Alexa, Siri, Google and even radio advertising. We're used to hearing these generated voices in our daily lives. Text to speech engines were fairly good when we originally launched Videate, and in just a few years they've gotten really good.

Providers like Amazon Polly, and Microsoft Cognitive Services already generate high-quality, custom voices for customers. The "Space Race" for voice is moving fast.

We expect synthetic speech to get better and better, making video voiceovers indistinguishable from a human's. And since Videate is speech engine agnostic, we can layer in audio from any source. Customers can pick and choose which synthetic voice sounds best for their brand.

Human behaviors

Another major aspect of the engine is the ability to mix-in behaviors and speech at the same time. This is what people naturally do when recording videos. You will notice mouse movements, clicks, scrolls, and typing while the engine is speaking. Without these interwoven actions, the videos feel slow and contrived. With them... it's like magic.🪄

In the early days of developing our engine, we would do something, then say something (rinse and repeat). However it didn't feel right. We need the engine to speak, move the mouse, and apply special effects all at the same time. The timing has to be perfect. We invested a lot of time into launching multiple events with parallel paths of execution while it's speaking.

We focus on simulating human behaviors in two areas: Mouse and Keyboard.

Mouse movement that feels organic

With mouse movements, a mouse cursor consistently going too perfectly and too quickly from-peint A to point B feels robotic. The key to making mouse movements feel human-like is keeping it "slightly flawed."

You don't want that perfect swipe, you want it to maybe swoop a bit, or overshoot the button and come back-to almost see the human hand at work moving their device on a desk.

When the mouse navigates around the screen, we use a number of algorithms to simulate the movements of a trackball, a mouse, a touchpad, or a library of precise curves. It takes a lot of precise math to make mouse movement look natural! We observed tens of thousands of mouse and touchpad movements to get this right. That, and several hundred recordings of co-founder and CEO Dave Gullo himself moving his mouse around on a screen :)

Videate offers a synthetic mouse movement in addition to the more human-like one, but we've found that our customers primarily use the human-like, "slightly flawed" version.
We also included mouse-based callouts. Like when you move a mouse back and forth or circle something on screen to emphasize an element or call attention to something.

Adding keyboard actions back in

As to the keyboard aspect, we've been careful to include little things that make the experience seem like a human is typing. For example, inserting micro times between keystrokes, a brief silence where you can imagine somebody "moving to hit the spacebar,  or the sound of pressing "enter."

We've worked hard to synchronize all of this with what's going on onscreenThis is the kind of stuff you don't really think about until It's missing, and when it's missing, the experience feels robotic and jarring

Additional "human-like" details

We have many different sound libraries available that play mouse and keyboard sounds (we see you, TikTok). When you watch a human-produced instructional video, you usually hear these things. When the viewer expects these sounds, even subconsciously, the lack of them becomes glaring.

Plus, reinforcing screen actions with sounds keeps the viewer's attention focused in the right place. Our customers can build a custom persona that provides voice characteristics, mouse and keyboard sounds, behavioral rates, custom colors, and effects that fit their brand or preference. We continue to add more special effects all the time that help drive the viewer's focus.

Closing the gap

With Videate's new innovations and a constant balancing act, we are closing the gap towards passing the Video Turing Test. Many clients already believe we pass and that's why they've joined this revolution. Our present workflow enables customers to produce and update an entire video library on every software release, in minutes.

More importantly, these efforts help our customers win. Big time. The ability to produce videos at scale using automation is critical in a world that demands more and more video. But it's also important to keep viewers engaged, not jarred out of their experience by something feeling too robotic.