Schematic flowchart of how Psychic Hive-Mind Fortune Teller is put together.

I have been wondering if describing a daft physical object with lots of words might be making it harder to understand!

So, for all those who like visualisations, here a schematic diagram.

It represents how the Psychic Hive-Mind Fortune Teller works (more or less)!

What is @twitr_janus - A summary overview, November 2012 for MCN attendees

This post is an updated summary of what @twitr_janus is, and what it demonstrates

This is mainly aimed at attendees at the Museums Computer Network conference, Seattle, 2012, especially the Slack Day hacking workshop, where this cutey will be being demonstrated live. (if I can put it back together again)

Firstly, here is @twitr_janus in cabin baggage ready to be checked in as hand luggage. I am not sure it will be allowed on board the aircraft...

What is @twitr_janus?

@twitr_janus is a live physical avatar - a puppet that can be controlled remotely over the Internet using the data sources of common web services like Google Drive and Twitter and direct web-enabled services such as Skype. It is a (very) crude alpha prototype that demonstrates several simple ways to remotely interact physically using web data services or piggybacking the connectivity of audio/visual web services.

Its real point though, is to demonstrate how real-time sensory interaction can be achieved by hacking together feeds, APIs or just features of common freebie web services. It can see you, hear you, talk to you and move its eyes to watch you. That's not creepy at all.

How does this work?

To control @twitr_janus, data is sent to it over the Internet using free web tools that have interactive elements. These include sending a tweet in a specific Twitter account and creating custom forms that piggyback the form processing script of a Google spreadsheet, but many other web services could be used.

At the other end of the web, the structured data is retrieved so it can be manipulated. This is done using the open source Processing language running on a connected computer. Processing can parse (strip out selective data) from open RSS feeds and/or APIs.

The data is then converted into physical action. An open source text-to-speech library for Processing is used to generate speech from text data. It can speak either tweets from its Twitter account, or from the contents of a field in a Google spreadsheet.
@twitr_janus uses the open source hardware platform Arduino to convert other data to physical motion. Eye control is achieved by an Arduino sketch loaded on the board, which reacts to data parsed from feeds from a Google spreadsheet. This signal data controls the position of two servos that direct the focus of its stare.

Its jaw is lip-synched to speech output by converting the analogue audio signal levels into a control signal for a car-door lock actuator motor. An audio amplifer is used to create asignal large enough to detect, and the Arduino sketch triggers a relay power circuit for the actuator based on signal peaks. It also has LEDs to indicate the last source of its control data.

Finally it can see and hear via a webcam. It does this using web video software Skype. This can be remotely activated using the correct settings in Skype

WTF?

The important point is simply that almost any modern web service will use, or often be built upon, a data publishing model. Because these web services have built-in data publishing features, such as APIs (application programming interface) or feeds such as RSS, they convert the human input tools into predictable data output that can be manipulated. None of this requires you to set up any infrastructure.


This principle can be used to hack new things together. It's all very enjoyably Frankenstein. @twitr_janus is just a specimen beast, created to illustrate this with a physical example. It simply demonstrates some possibilities beyond obvious web front end stuff, of hacking data from current open source or freely available web services, to create physical interaction.

It's crude, often breaks and has cost the lives of two Arduino boards, at least five servos and one vintage glue gun (RIP), but it does show how web data doesn't have to be just on-screen.

Other things that could be quickly prototyped using this method can easily be imagined such as:

• Tweetometer for marketing campaigns. Use Twitter data about your campaign tags to make a physical thermometer go up and down to indicate success. (e.g. Trend ranking, number of retweets, etc.)
• Animatronic avatar interaction to query collections or events databases. If data can be extracted, it can be spoken
• Physical puppet automatically announces new events as they appear in the api of your events database
• etc. etc.

Free code and stuff...

(with apologies to my professional developer colleagues for the state of this hackery!)
The basic (and very rough) code is available for forking/hacking/recycling as you see fit. Note, this will be only the last working version:

Processing control sketch (version 15)
(polls web feeds from Google and Twitter to extract data)
Github code download
PDF version

Arduino sketch (used to manipulate data and/or analogue signals into physical actions)
Github code download
PDF version

There are detailed posts about the specific physical build of @twitr_janus in the archive if this blog. Physical build notes crop up in back posts here on makingweirdstuff (interspersed with other nonsense) 
E.g. Posts about building eyeballs

Hacking out an audio amplifier for a text-to-speech puppet voicebox

@Twitr_janus needs a voicebox. This will do two things:

• Amplify the Processing text-to-speech output from  a feeble PC audio speaker to a dedicated speaker in the puppet 
• Provide a voltage level that is high ebough to use as an analog input to be detected by the Arduino board, to power the speech to jaw motion sync

In the top left photo, is a standard PC audio amplifier rigged up to an iphone as a test source. In the right photo is a standard phono plug, which will provide the output..














Below left is where is he output plug comes out of the amplifier. The output cable was cut to expose the wires (below right)















This cable is coaxial. The outer sheath copper mesh is the earth level, with a sheathed inner copper core
















Below left is what the stripped cable looks like. Below right with the oscilloscope lead attached















And on the left here is the signal trace. Each square is 1V.


This is what will be used to drive the Arduino. The output which is here driving the oscilloscope will be attached to the Arduino analog input. The Arduino will convert the analog audio level into digital data. In fact a number between 0 and 1024.

The Arduino script will send pulses to a relay if the signal peak is higher than a preset threshold level.

In practise the jaw will trigger to move, if there is a peak speech output (e.g. a plosive B, etc)



This is done in Arduino with a function that polls the analog input, something like this. (This example uses variables that would normally be declared elsewhere in the sketch.)

void jawTrigger () { //  jawTrigger is a function to be called in the Arduino loop()

jawPeakVol = analogRead(analogPin); // turns voltage from amplifier signal wire into a number between 0 and 1024

  if (jawPeakVol >= jawThresholdVol) {

  digitalWrite (jawRelayPin, HIGH); 

// trigger the relay to make the jaw motor move if the voltages peaks above a preset threshold (jawThresholdVol)

     }   // end if

  else if (jawPeakVol < jawThresholdVol) {

      digitalWrite (jawRelayPin, LOW); 
// trigger the relay to make the jaw motor return to default rest position if the voltage drops below the threshold

  }  // end else if

}   // end jawTrigger

This is not able to simulate how a real mouth looks when speaking, which will change shape depending on the phonemes in the speech. Here the mouth movement is synced to an audible peak, which the audience will hear as they see movement, and the brain will believe it is speaking the words.

The speaker is stereo. There are two outputs

The PC amplifer has two speaker boxes. One master speaker with an amplifier in it and one slave speaker that is driven by it.

The lead above  is the feed from the master box to the slave box.

Inside the master box is the speaker that will be the voicebox itself. This was removed from the box, and the box was then cut up to use as a mount

Old long-handled, sheet-metal-cutting shears make light work of even such a thick plastic casing (below left)
Below right is the amplifier extracted from the box

MakeyMakey arrives...

Apologies, this is a non-making post - just getting excited about having finally got a new MakeyMakey after lengthy Kickstarter waiting...

The good news is...

The MakeyMakey arrived.

The bad news is...

Shame about the £12.50 customs tax :(

The good news is...

It's got a very nice usb cable

The bad news is...

The pug was asleep and missed all the action

The good news is...

even the packagin was rather tasty
OK that's enough now...

A self-contained relay circuit unit

Schematic and circuit unit

Having got the relay circuit to work on the testbed breadboard, it was time to build it into a unit...

Here's the schematic and the finished board together.



The relay circuit is very simple. This relay uses a 5V voltage to trigger the switchover. The transistor has no current flowing as default, which means the output power voltage is sent to one connector (middle right block).
Applying a signal voltage HIGH to the input, causes a current to flow in the transistor and therefore through the relay switching coil. This changes the power voltage to the alternate relay output pin.

Earthing

There are two separate circuits in a relay. There is a high current power switched circuit, which here is 12V. There is also a coil circuit that switches this power voltage from one output to another. Here the switch circuit is driven at 5V.

The earth of the two relay circuits should be connected or natural differences in voltage levels can cause unexpected problems.

Completed circuit unit

The left hand image shows the circuit in it's final form on the board with connectors. The red indicates power voltages with black as earth.

The left hand connectors are the coil circuit power in, the coil earth and an earth connector for the power circuit linked to the coil earth.
The right hand connectors are for the power supply voltage (top, all red), the power voltage with signal switched on (also all red), and power voltage with signal off (black and red), respectively.

The right hand image shows the underside with connectors and mounting screws.

The top, black connector block is for the signal in.

This unit is now ready to be deployed as required.

Manipulating tweet character count in Arduino to control lip-sync duration

Simplified serial data connection between Processing and Arduino

Detail of the Processing sketch, showing sub-clause of tweetCheck () function which is checking incoming tweets from @twitr_janus. When it detects one, it sends a trigger signal to the Arduino to start jaw movement, then immediately initiates text-to-speech on the text. This will make the jaw move as the text speech audio plays.
Download the complete Processing sketch running on the PC detecting tweets
Download the complete Arduino sketch running on the Arduino board

Simple example - triggering an Arduino response to a tweet by sending a flag over the serial port.

This clause checks if it is a new tweet:

if (tweetText.equals(tweethCheck) == false)  

"tweetText" is the latest value of last tweet from twitr_janus, "tweetCheck" is the last new tweet.
It is doing two things here. Firstly it is sending a message of value NULL to the serial port (called imaginatively "port") which will send it over to the USB connection to the arduino board. This could be done wirelessly in principle.

port.write("NULL") 

The NULL character is converted to 0 when it is transmitted as data over the serial connection. The Arduino will start a jaw movement control signal if it detects a signal of value 0. In the code shown, it is simply turning a PIN on and off 8 times, with a 100ms delay before switching between HIGH and LOW, to give a 200ms period..

Adding logic - using tweet length to control the lip-sync signal duration

Using a simple value is fine for an on-off trigger, but the lip-sync should only occur for the duration of the speech generated by the tweet. This will vary depending on the number of characters in the tweet. This can be obtained using the length() method.

tweetLength= tweetText.length();

By sending the tweet length (number of characters) instead of a simple trigger, lip-sync logic can be created in the Arduino sketch running on the board, that will cause the lip sync duration to match the speech audio duration.

The Arduino function is called jawChomper ().

"incomingByte" is the tweet length in number of characters
"chompFactor" is a scaling factor that reduces the character length into jaw movement number. This is because when you speak your jaw does not move for every character. It moves based on words, which are groups of characters.
"chompDelay" is the standard duration in milliseconds, of the alternate HIGH and LOW values of the control signal.
"chompRand" is a controlled randomisation factor. This is added, to cause each HIGH or LOW to vary in duration. This is because the code cannot determine word length, so the jaw cannot be synced to the words. Just turning the signal on and off is not good enough. The brain can easily detect a rigidly uniform rhythm and it will jar. Varying the duration of up and down motion is a simple way to trick the brain into thinking the jaw is in sync with the speech, when it is actually just stopping and start in an asymmetrical rhythm.

void jawChomper ()
{
  // this function sets jaw bighting rate for
    // read the incoming byte:
 
   digitalWrite(13, HIGH);
   for (int i = 0; i < (incomingByte/chompFactor); i++) {
    // turn the pin on:
    digitalWrite(peakPin, HIGH);
    delay(chompDelay+random(chompRand));                
    // turn the pin off:
    digitalWrite(peakPin, LOW);
     delay(chompDelay+random(chompRand));
   }
 
   digitalWrite(13, LOW);
   
    // say what you got:
    Serial.print("I received: ");
    Serial.print(incomingByte);
    Serial.print(incomingByte, DEC);
    }

The voltage signal sent from the Arduino (signal pin here is peakPin - pin 6 as it happens) is connected to the base of a transistor with a resistor to supply a signal current. When the pin is HIGH, the transistor will amplify it to a current high enough to trigger the relay and turn on the power to the motor (not shown here).
When the signal goes LOW, the transistor current will stop and turn of the relay.

Twitr Janus has sight and hearing

Twitr Janus is a remote control puppet head.
Previously, the basic means by which to make it to speak was established. This will work by enabling it to listen (via a Processing sketch) to tweets from the @Twitr_janus account on Twitter. The tweets will eventually get converted into spoken sentences using text to speech.

and so onto hearing and vision...

To get sight requires an eye. In this case an HD video camera. a cheap one will do, but this was handy for testing. A lower res camera may actually be more desirable to keep a fast connection and make it reactive. This one has a decent microphone too, which is harder to find. It is likely that this will be used, but with the video streaming settings set low, but the audio high.


Various possible options were considered to enable streaming of video and audio from the remote head to a control PC elsewhere.
These included:

• Google hangouts but this is aimed at socialising, and can't be easily activated remotely. 
• running a mini server with a/v/ streaming, but I couldn't be bothered to work out the security/ firewall/port settings. 
• a plug and play IP or ethernet webcam, but this was not proven and would need to be bought before testing.

In the end, after some trawling discussions, it occurred to me that Skype might work. This proved to be rather useful because:

• it can be set up to make a video and audio streaming connection really easily
• it can run in the background, and be activated remotely by calling it
• security is handled by  the in-built privacy settings and by creating controlled accounts for Twitr Janus and a controller, which can only access calls by each other
• once account access is restricted, the connection can be activated remotely by the controlling account by setting the slave account to automatically connect (set in Skype) 

Here is a prototype of the communication technology components, joined up to demo the following:

• it is possible to start the video/audio link to enable Twitr Janus to see and hear (by calling on Skype - the slave account on Twitr Janus control PC would need to be in tray mode)
• it is possible to respond remotely to someone talking to Twitr Janus by tweeting the response to the @Twitr_janus Twitter account which will eventually be converted to speech

and so...

The iPad (master):

• the controlling device here. It also could be a phone or laptop as the Twitter and Skype services are web based (or app based if needed)

The laptop (slave control PC):

• has the web cam attached pointing at the Arduino board.
• has Skype running streaming video and audio from the webcam over the Internet to the remote master control (iPad here)
• has a Processing sketch running which is listening to Twitter and sending data to the Arduino if a new tweet from @Twitr_janus has been sent

The arduino:

• Has the Firmata for arduino library loaded, which is making the serial connection possible to the Processing script, and hence to Twitter
• Has an LED indicating when a tweet has been received

very satisfying!!


Close up of the eye watching the response LED indicating that a Tweet has been received!

Twitr Janus is created

It lives!

I must stop watching films like Promethius...

Twitr Janus is a 2-way communication puppet with remote animatronic control built on common technologies (well, it will be eventually!)
When finished it will be a physical avatar that can be controlled remotely to interact with people over the Internet. 


Basic objectives for this project are:

• to make a talking puppet head that can be manipulated remotely
• to build a completed working puppet
• as far as possible to build all this in open source software such as Arduino, Processing with communication using open data such as  APIs from common services such as Twitter and Google Docs 

This could take a while!
So far:

• it is just an Arduino puppet brain but no physical form yet
• it has a Twitter feed through which it can think and communicate @twitr_janus
• It has a puppetmaster brain running  Processing on a master computer that can control it across a serial port connection 

---

A quick video showing the initial brain functions....

 

Processing script that is returning tweets using the Twitter API and passing them to the arduino which has a Firmata sketch loaded on it as firmware.

Initial thoughts on making Twitter drive a remote talking head

This is a "design" and I use that term loosely, that is an attempt to make a real head speak out tweets, from anywhere in the world - a sort of remote Twitter-to-speech-convertor!






























To make this work there are a number of challenges, which include:

• Connecting the head to the web, ideally remotely using wireless
• Once connected getting the last tweet from a specific Twitter account (the control account)
• Converting the tweet into speech
• Making the speech signal animate a mouth to simulate talking

This is visualised above!

The step for getting the last tweet from a twitter account

Step one of this I appear to have solved in Processing (which is Java-based), which should mean it works in arduino.  

Here's a snap shot of the function and the output that it can produce.

It's calling the Twitter API, then parsing it and splitting it to create an array, then calling one element of that array, which is the last Tweet from twitter.com/rosemarybeetle

For details on the Twitter API, see https://dev.twitter.com/docs/api/1/get/statuses/user_timeline

Making an Arduino power supply cable look wooden

With nice wooden plugs an ugly black plastic lead is not really what anyone wants!

The black lead on the left was revamped into the lead on the right by sawdust coating it using latex as a flexible bonding agent.

Latex and sawdust forms a composite material which is much physically robust than either on their own. The latex provides flexibility and the dust gives bulk and resilience.



Liquid latex from an art supply shop was used, without any thickener adding. It uses ammonia as a suspension fluid which does smell rather strongly. Use in a ventilated space!








Latex was poured into a shallow container










Sawdust was sifted to get a fine powder.










The sawdust was dusted over the wet latex.










This was left to dry, then the process was repeated.













Eventually after three coats it ended up looking rather tasty and root-like!













A close-up of the joint with the mains plug.










Here is the head complete with swanky new cable.

Adding wooden plugs to the power unit

The casing for the power supply to the wooden head has cracked. This was pulled apart completely on the crack, and glued back at the same time as gluing the casing onto the power unit transformer.









Here it is being held temporarily in the vice, while a cramp is attached.










A conical casing for the %v end of the power lead had been turned, which had also cracked. This turned out to be helpful, as it was the easiest way to fit the plug










A pair of locking grips was used as a cramp to hold it while the glue set.

















Once glued and cramped, they were let to dry









Here is the finished mains end of the plug (face side)











Here is the business end of the finished mains end of the plug.

The transformer is held in place firmly using hot glue










And finally the finished end of the 5v plug