Reality matters

This is a guest post by Frederic Schroyens from The Reference digital agency

Elegy for forgotten sounds is an art installation at the Concert Hall of Bruges commemorating World War I. It allows visitors to experience the cacophony of WWI’s battlefield, using no more than timpani sticks and Estimote Beacons. This post describes our path to developing it, issues we experienced along the way, and how we solved them.

The image below gives an impression of the setup–the installation is placed in the center of the room. There’s a lot of technology hidden inside: an iPhone, Intel Edison units, speakers, audio control panel, and even a smoke machine. The board contains five timpani sticks. Each stick has two Estimote Beacons inside and is “charged” with a different set of sounds. All sounds are designed to evoke an impression of WWI battlefield.

The experience is as follows: people choose a stick and start moving around the room. The audio behavior triggered by the system will create a WWI soundscape. It’s even possible to trigger smoke with the right combination of sticks. The area surrounding the installation is divided into three invisible zones, each of them approximately 2 meters wide. The technology inside the central installation keeps track of the sticks and also checks which zone the sticks are in. Ultimately, the combination of sticks and zones determine the audio playback and volume.

The installation uses a total of ten beacons, placed in five timpani sticks. Each pair of beacons are combined at a 90-degree angle to provide the best possible signal quality. Both beacons are used for distance tracking, but one of them also employs Motion UUID, which is responsible for motion detection.

Figure 1. A conceptual drawing of the installation zones.

Challenges

At the same time, creating a beacon deployment like this one poses some difficulties: iOS limitations, signal interference, and Bluetooth behavior make it hard to receive consistent RSSI readings. In most situations it is enough to detect the presence of the user within the beacon’s range. In our case we also needed to know the approximate position of the stick almost in real-time. An unstable or slow signal can disrupt the application flow which will be audible for the users.

At first it seems simple enough to use the Estimote SDK and listen to the distance updates. Unfortunately, this approach has some drawbacks. First, iOS limits the beacon update speed to one update per second. However, this is too slow when you need real-time distance tracking.

One alternative would have been to implement the tracking part of the application on Android, which does not have this limitation. We tried this approach by tracking the beacons on Android and passing the data to iPhone over Bluetooth. Unfortunately, while this improved the tracking speed, the signal was still jumping around a lot due to interference in the room.

Interference causes the RSSI of the beacon signal to jump all over the place. Zone switches would often occur even when standing completely still, because the RSSI value can randomly jump 5dB in an instant.

We couldn’t get rid of the interference in the building, but we did come up with some steps to reduce its impact on our app.

Tracking with Intel Edison

Simple presence detection works well enough with an iPhone, but it’s not enough when you need real-time distance information like we did. The update speed is too slow and there’s nothing to do about interference. Theoretically, a larger Bluetooth antenna would help, but one can’t just add one to the iPhone!

If you’re not familiar with the Intel Edison, it’s a computer about the size of a credit card running Yocto Linux. It comes with Bluetooth 4.0 support and you can attach an external antenna with a coax cable.

To cover as many approach angles as possible, we configured three Intel Edisons and placed them in a triangular shape around the installation. Each Edison has an external antenna attached to receive a tremendous amount of Bluetooth signal from the beacons. We tested seven different antennas, but eventually settled on the ANT-24G-WHJ-SMA which gave us the most consistent results.

The job of the Edisons is quite simple. They set up a Bluetooth connection with the iPhone and then start capturing the Estimote RSSI broadcast values; then they process those values and pass them to the iPhone. Thanks to the Edisons, the maximum range of the beacons is strongly extended and the signal is far more stable.

An entire explanation of the code running on the Edisons would make this blog post too long, but their configuration looks as follows:

• A system.d script launches the program at boot and restarts it in case of a crash
• The program itself is written in C and uses the Linux BlueZ stack to set up Bluetooth connections. The program will capture the RSSI of all beacons with a major in the 600 range. It will then perform a weighted rolling average on the values received, using the following steps:
• Take the average RSSI value of the last five values as the new rolling average
• Take the weighted average of the last rolling average and the new one, whereby it attaches a bit more weight to the old value
• Compare the received value to that of the ’sibling’ beacon and check which one has the best (lowest) value. If the value is different from the last value sent to the iPhone, the new value is sent. This helps to smooth the readings and avoid any sudden jumps in the RSSI signal

Remember that each stick contains two beacons and we’re looking for the best value of the two. So after we calculate and smooth out the RSSI value, we compare the values between sibling beacons and send the better one.

Edisons send data to the iPhone in the following format:

INTEL_EDISON_ID|STICK_MAJOR|VALUE

So for example, the iPhone will receive: 3|610|48

Note that we send the stick’s major ID and not the beacon’s major ID. This is because the beacon’s major is only relevant to the Edisons, but as far as the iPhone is concerned, there are only 5 beacons (sticks) involved. Therefore, the Edison will map the majors of each pair of beacons to the major of the corresponding stick as the iPhone expects them. So beacon majors 600 and 610 become stick major 600, and beacon majors 620 and 630 become stick major 610, etc.

Applying weighted rolling average X2

The first round of weighted rolling average described above occurs on the Edisons. The iPhone will run a second round once it receives data from one of the Edisons.

The iPhone contains a dictionary in which it keeps the last received value from each Edison for each stick. Upon receiving a new value, the iPhone will check the ID of the Edison that sent it and then store it at the appropriate place in the dictionary.

It will then take each value in the dictionary and calculate the best one. In other words, this is the lowest RSSI value received from one of the Edisons. This value is then stored in the list of last five ’best’ values for the stick.

The second round of weighted rolling average is then executed on those values. The outcome of this round is the final value that will be used in the application flow. The entire process will repeat itself every time a new RSSI value is received. Because the beacons are configured with an advertising interval of 200 ms this flow will occur approximately five times per second.

Figure 2. The green line displays the effect of signal smoothing applied to the distance returned by the Estimote SDK.

Setting up ’dynamic’ zone borders

The above steps work well to keep a fairly smooth and continuous feed of RSSI readings. However, it does not yet stop our zones from switching back and forth if a user happens to stand near the edge.

For example, if the near/close zone border is defined by RSSI value of 50 dBm, with near < 50 dBm and close >= 50 dBm and the current RSSI value is 49 dBm, then a 1 dBm fluctuation will trigger a switch. Since RSSI is never completely stable, this can occur often.

To mitigate the problem, we implemented ‘dynamic’ zone borders. The zone border is essentially “moved” a few dBms in the opposite direction after the user crosses it. As an example, imagine the near/close border is defined at 50 dBm and the user crosses from near to close. The moment the user enters the close zone, the required RSSI threshold for going back into the near zone is reduced by 2 dBm (thus 48 dBm). If RSSI now jumps by 1 or 2 dBms, it’s not a problem. It will still be greater than 48 dBm and the user will remain in the close zone. Moving back from close to near works the same way. The new border of 48 dBm will be moved back to 50 dBm. Note that the border only changes for the particular stick that crossed it. Thus at any given moment sticks can have different values for the zone borders.

The exact values with which to update each border is gained from a lot of of trial and error. We don’t use an offset of 2 dBm everywhere and in every direction, but the concept generally works pretty well for our purposes.

Using motion detection to detect unused sticks

One problem still remains. When sticks are placed on the board they should not be part of the application flow. But if an unstable signal suddenly causes an RSSI spike, the stick may jump into one of the zones and trigger audio playback. The further the beacon, the less stable its signal, so this was a regular occurrence.

The solution for us was to combine the distance tracking with motion detection. The motion detection is implemented using the Estimote SDK on the iPhone itself and does not use the Intel Edisons. If a stick doesn’t move (so when it’s placed on the board) we consider it to be out of range. The iPhone will simply ignore all updates from sticks that are motionless. This will also cause the sound to go quiet if the user keeps the stick very still, which turned out to be an interesting side effect in our case.

With all of the above measures, the installation works pretty well though not completely flawless. All the smoothing in the world won’t help if a beacon’s signal suddenly jumps 10 dBms and remains there for the next 10 seconds. But the behavior is generally consistent. Actually, it’s reliable enough for us to implement dynamic volume (the audio volume changes with the user’s distance in the close range).

We suggest starting with the Estimote SDK alongside Estimote Beacons–it allowed us to quickly set up basic tracking so we could focus on the application and flow logic first. Only afterwards did we move towards improving the signal by switching to the Intel Edison setup. And depending on the location where you will deploy your beacons, simply tracking with iPhone may already work well enough.

Frederic Schroyens, Head of Mobile & Tablets at The Reference