Sunday, 29 July 2012

Manipulating tweet character count in Arduino to control lip-sync duration

Simplified serial data connection between Processing and Arduino

Detail of the Processing sketch, showing sub-clause of tweetCheck () function which is checking incoming tweets from @twitr_janus. When it detects one, it sends a trigger signal to the Arduino to start jaw movement, then immediately initiates text-to-speech on the text. This will make the jaw move as the text speech audio plays.
Download the complete Processing sketch running on the PC detecting tweets
Download the complete Arduino sketch running on the Arduino board
Twitr_janus Processing and Arduino sketches

Simple example - triggering an Arduino response to a tweet by sending a flag over the serial port.

This clause checks if it is a new tweet:

if (tweetText.equals(tweethCheck) == false)  

"tweetText" is the latest value of last tweet from twitr_janus, "tweetCheck" is the last new tweet.
It is doing two things here. Firstly it is sending a message of value NULL to the serial port (called imaginatively "port") which will send it over to the USB connection to the arduino board. This could be done wirelessly in principle.


The NULL character is converted to 0 when it is transmitted as data over the serial connection. The Arduino will start a jaw movement control signal if it detects a signal of value 0. In the code shown, it is simply turning a PIN on and off 8 times, with a 100ms delay before switching between HIGH and LOW, to give a 200ms period..

Adding logic - using tweet length to control the lip-sync signal duration

Using a simple value is fine for an on-off trigger, but the lip-sync should only occur for the duration of the speech generated by the tweet. This will vary depending on the number of characters in the tweet. This can be obtained using the length() method.

tweetLength= tweetText.length();

By sending the tweet length (number of characters) instead of a simple trigger, lip-sync logic can be created in the Arduino sketch running on the board, that will cause the lip sync duration to match the speech audio duration.

The Arduino function is called jawChomper ().

"incomingByte" is the tweet length in number of characters
"chompFactor" is a scaling factor that reduces the character length into jaw movement number. This is because when you speak your jaw does not move for every character. It moves based on words, which are groups of characters.
"chompDelay" is the standard duration in milliseconds, of the alternate HIGH and LOW values of the control signal.
"chompRand" is a controlled randomisation factor. This is added, to cause each HIGH or LOW to vary in duration. This is because the code cannot determine word length, so the jaw cannot be synced to the words. Just turning the signal on and off is not good enough. The brain can easily detect a rigidly uniform rhythm and it will jar. Varying the duration of up and down motion is a simple way to trick the brain into thinking the jaw is in sync with the speech, when it is actually just stopping and start in an asymmetrical rhythm.

void jawChomper ()
  // this function sets jaw bighting rate for
    // read the incoming byte:
   digitalWrite(13, HIGH);
   for (int i = 0; i < (incomingByte/chompFactor); i++) {
    // turn the pin on:
    digitalWrite(peakPin, HIGH);
    // turn the pin off:
    digitalWrite(peakPin, LOW);
   digitalWrite(13, LOW);
    // say what you got:
    Serial.print("I received: ");
    Serial.print(incomingByte, DEC);

The voltage signal sent from the Arduino (signal pin here is peakPin - pin 6 as it happens) is connected to the base of a transistor with a resistor to supply a signal current. When the pin is HIGH, the transistor will amplify it to a current high enough to trigger the relay and turn on the power to the motor (not shown here).
When the signal goes LOW, the transistor current will stop and turn of the relay.
Arduino and lip-sync relay circuit

Sunday, 22 July 2012

Twitr_janus talks through temporary animatronic jaw

Twitr_janus speaks and moves Finally, I have got the full loop through from sending a tweet to automatically make. Remote puppet talk and move its mouth in time as it speaks.

Twitter to speech converter kit list

  • Twitter app (here on iPad) to Internet
  • Internet to PC running Processing sketch periodically scans for twets and strips message
  • Processing converts to spoken output using tts.h text-to-speech library
  • Speech played through audio amplifier
  • Amplifier secondary speaker output connected to Arduino analog input
  • Voltage detection in Arduino triggers output pulses on peaks
  • Peaks switch power on and off, from model railway transformer connected to car door actuator motor
  • Actuator moves jaw in time to speech peaks

Twitr_janus audio out Twitr_janus amplifier audio connections
Twitr_janus model railway power unit and Arduino Twitr_janus relay circuit Twitr_janus Processing sketch Car door lock actuator driving cardboard mouth

Watch the complete loop from tweet to talking cardboard...

Thursday, 19 July 2012

Mad scientists...


You can't quite earn your mad scientist badge without having one of these babies...
I can't even remember where I obtained this, but it's lovely...
Analogue dials!

Sunday, 15 July 2012

Jaw muscle as car-part consumable

Lip-sync testing Car door locks - they are really rather understated

I am very pleased to have discovered car door lock mechanisms. They are a really powerful means of moving something in a straight line and dead easy to control. I am surprised he aren't as used as servos. A strong 12v locker is about the same price as a pretty small hobby servo. This will drive Twitr_janus' jaws.  This video shows his in action....
This video shows a simple switching circuit with a transistor triggering a relay. The input will force a transistor to produce a voltage that is sufficient to cause a relay coil to close a circuit for the door lock motor This combined with an Arduino will convert speech audio peaks into jaw movement = SWEET!

Friday, 13 July 2012

Paper prototyping jaw opening mechanism

Just discovered car-door lock mechanism which is an awesomely useful device that will push quite a big load from a 12V pulse. This may be the answer to lyp-synched puppet mouth operation via Twitter-to-Text-to-Speech-to-Jaw-Motion!

Nothing like a good Rotring pen for scrawling out circuit ideas
Car door lock motor circuit

Here's what the motor looks like (it's a bit of a crap photo, off Maplins

Monday, 9 July 2012

Solenoids for lip-syncing Twitr_janus

Been trying to work out how to get lip sync working. Thought this would need Arduino, but actually suspect analogue electronics will sort this easier.  BUT - what to use?

A keen fan of Gerry Anderson since a boy, I have looked into Supermarionation, which essentially is the brand he used for his semi-automated puppets. They had a solenoid linked to the speech input to sync-the lip movement.

This "intro to solenoids" video on YouTube is really rather good. Great Soviet Vibe. Nice, but too much effort to have to design a solenoid, and there are some serious overheating fire risks that need to be understood. This was deemed too dodgy.

BUT where to get a solenoid round the house? I thought about a doorbell solenoid first

BUT, then realised what I wanted was sound to movement. Of course this is what a loudspeaker does. It's just a coil that moves depending on the input. Just what I needed.

I had an old speaker in the garden for BBQ music, which was no longer in use... Old speaker
The casing was mouldering and full of creatures...
Old speaker
But conveniently had solid low pass and high pass filters built in...
Old speaker

All I have to do now is work out how to drive the speaker from the laptop output, without blowing either one of them. This means understanding the spec - yikes...

Old speaker

Thursday, 5 July 2012

V&A heads

Head in the V&AHead in the V&AHead in the V&AHead in the V&AHead in the V&AHead in the V&A
Head in the V&AHead in the V&AHead in the V&AHead in the V&AHead in the V&AHead in the V&A
Head in the V&AHead in the V&AHead in the V&AHead in the V&AHead in the V&AHead in the V&A
Head in the V&AHead in the V&AHead in the V&AHead in the V&AHead in the V&AHead in the V&A

V&A heads, a set on Flickr.

Some headshots taken for puppet head design inspiration

Twitr_janus speaks...


A successful test of sending a tweet from device to another remote device over the web to make Twitr_janus talk

This allows speech puppetry to Twitr_janus anywhere in the world!
  • on controller device (an iPad here, but could be any other web device)
  • text sent as tweet to Twitr_janus account on Twitter to web to Twitter
  • receiver device from Twitter API to web
  • to Processing sketch where API is parsed to extract tweet text
  • text is converted to speech using guru.freetts library and Processing native text-to-specch (TTS)

ha ha ha ha...

Watch the test in action as it was filmed!

Monday, 2 July 2012

Lip sync for puppet mouth

I have been trawling about the Internet looking for other people who have done audio to mechanical movement transduction. Of course the rather marvellous Gerry Anderson Supermarionation puppets for Thunderbirds etc., did this years ago.

This site has great shots of inside the heads of Gerry Anderson puppets:
Thanks to whoever put these pics up.

This is their image, but loads more on their site.