RTK GPS for Outdoor Augmented Reality

What is an RTK GPS system?

Basically, GPS is inaccurate. I'm sure I've mentioned this before but if I haven't, it it. Very, very inaccurate. It may not seem that way because of the way we use GPS to help us locate ourselves and get directions, but often this is because averaging a position over a greater period of time leads to greater accuracy, and a constant heading is more accurate than standing still. Portable devices may have an accuracy of 5m-10m radius under the best circumstances which means while you are standing still the device things you are wandering around in various directions (a phenomenon known as "drift"). But RTK GPS systems use a signal called "correction data" from a base station at a known, surveyed point to locate roving units at an accuracy of 1 cm over several miles. This is the sort of accuracy required to do outdoor augmented reality. Instead of talking general theory I'm going to get down to the nitty gritty technical nonsense one might have to endure to set up a system like this (if you are interested, I recommend http://www.tinmith.net/ or http://www.tinmith.net/wayne/thesis/piekarski-ch0-start.pdf for a little light reading on the most advanced self contained units being used in university research). My hope is not even that someone might try and replicate the system, but more that the general technical tidbits may be otherwise useful.

During the summer of 2012, on top of my usual job running the augmented reality lab at York University, I've been working to help set up the Augmented Reality Research Lab in the Centre for Digital Humanities at Brock University in St. Catharines. In addition to the Intersense 900 hardware we have at York, John Bonnett, CRC at York also has a couple of RTK GPS devices that he's been anxiously waiting to get into the field. Fortunately, most of the tedious stuff was taken care of for me. Thanks to the UART Augmented Reality Plugins for Unity from the AEL in Georgia Tech I was able to unify the tracking technologies in the lab into one software, just as we'd done at York. The only thing that hadn't been done yet in our lab was to work with the GPS devices. One other handy addition to VRPN library from Alex Hill at Georgia Tech was the first step in getting one of these RTK GPS devices reporting positional information to talk to Unity.

 We are using the following configuration:

• Head Tracking: Intersense InertiaCube 3 connected by serial adapter to Keyspan Adapter to USB and powered over USB. 
• Camera: Point Grey Firefly2 over Firewire (to PCI card with external power source). 
• HMD: emagin z800 (modified with Dual Rachet Headband). 
• GPS: Magellan Zmax Thales RTK base/rovers with a Magellan U-link radio and external GNSS-750 GPS Receiving Antenna. 

VRPN
The first challenge was simply getting these GPS systems talking to the computer. After a little digging through the technical manuals, I realized that the line was dead because by default there is no NMEA data being reported. This is where the $PASHS,NME commands come in. The Fischer Plug on Port A of the unit connects, via serial cable (an extra component at $200), to a keyspan adapter making it USB compatible. Firing Up Wincomm (9600, 8, N, 1) I could connect to the device. The manual is out there online  and it appeared that to get this talking like a normal GPS device I needed to input $PASHS,NME,RMC,A,ON,0.2 $PASHS,NME,GGA,A,ON,0.2 $PASHS,NME,GLLA,A,ON,0.2 With positional information streaming from the rover, I at least had a sense this was possible to do. The most daunting hurdle was that for some reason VRPN wasn't talking with my GPS device. It was talking fine to my bluetooth GPS device (sort of) but it took a clue from Russell Taylor (University of North Carolina) that the RTS bit wasn't being set. With that added to the VRPN_Tracker_GPS code, the device finally started filling the cue (cbInQue). A couple more tweaks to the GPS code and VRPN was ready to use, with the GPS, in Unity.

Unity
I found the examples in UART for using GPS systems in Unity quite good, but I was a bit concerned (needlessly?) about the conversion happening to transform latitude and longitude to meters (units) for use in Unity's coordinate system. I couldn't pass up an opportunity to use Vincenty's formula wrapped up with a bow in C#. More details on that to follow.

DragonFly 2 Cameras
The trickiest part was wiring a solution for these cameras. Not being a big electronics person (remember my former post on soldering? I've gotten a *bit* better since then) I was caught in this seemingly endless debate online about whether or not an Amperage rating matters. However, instead of having to worry about heat issues on top of that (dropping the voltage, etc...) I thought these little 12VDC ryobi packs would be perfect: http://cheesycam.com/cordless-tool-batteries-solid-dc-power-packs/ And indeed they have been. After finally tracking down the right size plug for the Firewire card (hint: try an external USB hub power supply) it was just a matter of adding a switch and a 12Ohm resistor to the mix to ensure only 1Amp was running throughout the circuit. 

How does everything fit together? Where's the source code? More soon...

SnapDragonAR 1.0.8 Released

Lot of work went into this update. Lots of great new features including the ability to greenscreen images on to the markers to create transparent borders. Technologically it's not as advanced as creating a true alpha channel but it's a lot less processor expensive when running large amounts of video like we are doing. The result is pretty cool. The only downside is that the images can't overlap. They create a blank space. Not the end of the world, after all this software is intended to work with one movie per marker. It's only if you get totally carried away with scale and offset that it's at all noticable.
Otherwise it's becoming a seriously cool little app. It compresses your videos so you can instantly test different codecs for your project, allows you a bunch of camera configurations and works with any camera that works in Quicktime. You can even add content from http:// and rtsp:// streams. I've only tested it with static videos but I would imagine you could stream a live webcam feed directly to a marker and then chromakey it.

Installing opencv 2.0.0 through macports... Piece of cake or Pain and misery?

ARRGH! Okay now that out of my system... I hope someone finds this post useful.

http://opencv.willowgarage.com/wiki/Mac_OS_X_OpenCV_Port

"If you encounter errors, try installing its dependents with +universal whenever possible."

Great... but what if that doesn't solve your problem.

Mac OS 10.5.8, Macports 1.9.1, OpenCV 2.0.0

Perhaps this is a naive approach, I've since learned about checking "port variants opencv" so perhaps "port variants lame" would have turned up a helpful clue... but this is what I did to solve my issues. Take it or leave it.

My first error:

:info:build xmm_quantize_sub.c:37:23: error: xmmintrin.h: No such file or directory

If this error looks familiar it's because LAME didn't build (LAME 3.98.4).

"If I run make immediately after configure, it fails. What I have to do
to make successfully is to first comment out the following line in
config.h:

#define HAVE_XMMINTRIN_H 1"

Taken From http://www.opensource-archive.org/archive/index.php/t-88245.html

Tried even setting the define to 0 but that didn't work, so commenting it out may be your only option. Let's try building opencv again...

AND... Ka-blamo... another error. This ones worse than the first. It seems to be with ORC (formerly liboil) 0.4.5

It has no idea what the function get_cpuid is, any why should it? It's supposed to know better thanks to ifdef __APPLE__, right?

Look at the bottom of orccpu-x86.c (in the macports work directory).

orc_mmx_get_cpu_flags(void)

{

//orc_cpu_detect_kernel_support ();

#ifdef USE_I386_CPUID

return orc_mmx_detect_cpuid ();

Oops, if I'm reading this right then regardless of whether it's been defined Apple it's still going looking for the get_cpuid function because we're on an intel Mac. Naughty, naughty. Let's comment those out and cross our fingers.

orc_mmx_get_cpu_flags(void) { //orc_cpu_detect_kernel_support (); #ifdef USE_I386_CPUID return orc_mmx_detect_cpuid ();

There's another one in the matching function orc_sse_get_cpu_flags(void) too. Make sure you get both and "orc" will compile.

This post likely explains why the error may not appear for some people (looks like it depends on which assembly compiler you are using, but I'm guessing that this is a bonafide bug).:

http://sourceware.org/ml/crossgcc/2008-01/msg00017.html

Okay, so far so good, ffmpeg is the last to go... Totally expecting it to crash and burn.

Oh, okay, that worked. It's building and installing opencv now. Could I possibly be that lucky? Guess I'll have to build a project in Xcode and find out.

Banff CAVE redux

I was fortunate enough to help restore the Banff CAVE to a functioning state. We used the Cosm Library for Max/MSP to create our own series of cave.* objects for Max. Those can be downloaded in their rough form from www.futurecinema.ca/arlab. Several modifications have to be made to use them in your own cave like environment, mainly the windows have to be aligned to work with the projectors you are using. You have to enable/disable the dual windows manually that are being used for stereo. And you will have to repurpose the cave.send and cave.jit.send patches so they are hard coded with the IP of your destination machines (and new ports may have to be added depending on the number of machines you are using). We had two renderers and one server.

Storytelling X.0

This was a really great session to be involved with at FITC.

Storytelling x.0

There was a truly exceptional lineup of speakers. Everyone really brought something interesting to the discussions.

Adrian Belina

jam3media.com

More Info

Todd Denis

swarment.com

More Info

Anita Doron

avalanchefilms.com

More Info

Caitlin Fisher

futurecinema.ca

More Info

Victoria Ha

stitchmedia.ca

More Info

Tali Krakowsky

apologuestudio.com

More Info

Alex Mayhew

alexmayhew.com

More Info

Alex McDowell

5dconference.com

More Info

James Milward

thesecretlocation.com

More Info

Siobhan O'Flynn

1001tales.posterous.com

More Info

Andrew Roth

futurecinema.ca

More Info

John Underkoffler

oblong.com

More Info

Marian Ursu

shapeshift.tv

More Info

Tony Walsh

PhantomCompass.com

More Info

Gremlins are back - need proof?

Some strange things have been happening to the technonolgy around me.
Of course, I blame gremlins...

"So close and yet..."

Hmmm... The good news or the bad news? Well here's both.

As you can see, the ladybug2 can now talk directly to Max... But it's green and unhappy. There are two major problems, one which I couldn't have foreseen, the other I'm an idiot for not considering.

First: We were lucky enough not to have the Bayer filters included in Libdc1394v2, to try and decode the RGGB that the ladybug is outputting but as you see it doesn't work as expected. Our theory is that the code is expecting the image to be upright, but as you can see, our image directly out of memory is 90º from where it should be. So why don't we just rotate the image before processing? Well, as it is we are copying the matrix once, from memory, to the outlet. This image is huge, a whopping 4608x1024. That's 6x768x1024. So of course that brings me to the second problem. Under no circumstances have I ever been amazed with the framerate of an incoming image in Max at 1024x768. Let alone 6 of them simultaneously. So is this even a good idea? Perhaps there's just too much visual information to process to be useful at all. Not one to give up too easily, I may have another solution. If we output the raw image into a jit.slab and crunch it all on the GPU (which is what they do at Point Grey if I'm not mistaken anyways), then maybe we can get the performance up to something halfway decent. Definitely open to suggestions.

Finally something to contribute to the Max Community after a long time. Thanks to Rob and Andrei from York and Rand at Intersense. Download at futurecinema.ca/arlab

Follow the instructions in the top corner of the .maxhelp patch to get the dylib installed and get you up and running. Sorry but I think the object only works on OS X 10.5 (Leopard) and up.

Looking Back at Almost perfect...

Hey I'm on TV.... ;)

Ladybug Woes on a Mac

http://sourceforge.net/projects/libdc1394/files/

Been working with the Libdc1394 version 2 library thanks to a tip (thanks Louis!) and am trying to get the ladybug working with Max/MSP. The OSX port of Libdc seems fine until you try to capture a series of images and I get a "bus Error". Tried posting to the forums, no luck, but I solved it, it seems there's an error in the "BASENAME" string in the Sprintf function. If you change:
sprintf(filename,"%s-%05d-%d-%d.jpg",BASENAME,i,cam,k);
to
sprintf(filename,"%04d-%d-%d.jpg",i,cam,k);

AND
sprintf(filename,"%s-%05d.raw",BASENAME,i);
to
sprintf(filename,"%04d.raw",i);

Tah-Dah, it works (though it drops all your frames in your current folder... sorry about that). Will try and figure out how to get that running soon, and then, on to getting it running with Max/MSP.

Wrapping up the Year

The year ends with one of my favorite things (something I'm terrible
at). I haven't had to solder much in my life. I must have seen it
done a hundred times. I even went out an bought a soldering iron for
the AR Lab to swap LEDs for infrared emitters in dollar store
booklights. The picture is of Stefan at Brock showing me how it's
done. He really had the knack but I think I held my own pretty well.
The component is a Parallax RFID reader, I'll post the code I got from
the arduino playground and the Max/MSP patch I've been using from the
forums.

IASC Media Tools for Interactivity

I've been keeping busy since I've been back in Ontario.  I've been teaching part time at Brock University in St. Catherines.  The course is Media Tools for Interactivity as part of the Interactive Arts and Sciences program.  It's a course on graphical programming languages and peripheral devices framed by a survey of New Media Art  History.  It's teaching students to explore digital art and interactivity as practitioners.  It's loosely based on programs that I took in the Digital Media stream of Fine Arts Cultural Studies at York University though the focus on graphical programming languages as their own entity is an important part of the course.  And of course there will be good fun with Arduino boards.

It's got me thinking about what sort of courses I'd be interested in pursuing.  I'm definitely still interested in taking a course on 3D Modeling and perhaps one on C programming (I'm kind of getting anxious to program for platforms like the iPhone).  But what to do about a Masters or PhD?  While taking more interactive art programs such as ITP in New York sound great, I can't help but think I still want to pursue Augmented Reality as a field of research (both practical and theoretical).  I'm not sure yet what courses are better suited to that kind of research.

All Good Things...

The "Almost Perfect: Call and Response" Locative Media Co-Production Residency grinds to a halt as all things must. I will post more about the Gremlin-infested Positioning System shortly but would like to take this moment to acknowledge the wonderful people I worked with and met at the Banff Centre. To all of you, my advisors, the staff at the BNMI, and the rest of the Almost Perfectionists, I'd like to say thank you for making it an incredible experience. I learned so much and can't wait to expand my practice further into the field of Locative Media.  I will cherish the memories and endeavor to keep in contact with all of you.

Here Be Gremlins

Well, it serves me right invoking the gremlins name as part of my project. Technical glitches abound. Though it's finally taking shape tonight, there was one perfect run, and another in which the sound from all the videos suddenly cut out for no reason. And the keyboard for the tablet kept popping up which has never happened before. Oh, and I got chased by deer again. I think they see a kindred spirit in my glassy eyed stare into the bright light of a computer screen. Either that or they want to bite me.

Gremlin-infested Positioning System

With open studios around the corner I've been wondering how this project is going to received. It's a little out there conceptually, but I've heard people say that the intro video (sort of a last minute consideration) really puts the things you experience into context. Essentially I want users to be able to experince hybrid tracking technology as if the gremlins are the only thing preventing it from working properly. Today I decided that since the GPS zones all seem to work it was about time to have participants try it. It was a semi-guided tour. Even with technical glitches (irony aside) it seemed to be a compelling enough experience. So with more participants trying the system tomorrow I can't wait to see more reactions. The picture is an early attempt at turning Sulphur Mountain into a Volcano (one of the many uncomfortable situations the Gremlins inflict on the user).

Graffiti Research Lab hits Banff

GRL in Banff from GRL Canada on Vimeo.

Good clean fun.  Projected on a 30 foot elevator shaft-to-be on the construction site in Banff (shown below).  I've been following the GRL here and there but was just too happy about getting to try it out yesterday.  Music is NSFW, Explicit Content, blah blah blah...  You've been warned.  Mute if you must.

Scene of the crime:

Footnotes - Kay Burns Audio Walk

Today was a terrific day to try out the large scale audio walk
"Footnotes" produced by Kay Burns. The tour consisted of GPS
activated sound clips of which many were lamentations about the
changing landscape and ecology of Banff. I found several of the
experiences particularly appealing, one of which involved the sound of
crickets followed by an antiquated ad for Bug Bombs and an anecdote
about children running behind DDT fogging trucks. The sound of frogs
then began to play hinting at the bugs role in the ecosystem. I found
it moving since I so much enjoyed the sound of the crickets.

Tour of the VR CAVE at the Banff Centre

Today we had a group tour of the VR CAVE in the Banff Centre.  The project we were shown was merely for demonstration purposes.

mScapes Workshop

This workshop was one I'd been anticipating, it was our first chance to design programs in mScapes, the project that emerged from the Mobile Bristol platform.  This program is Windows only and works on Windows Mobile devices.

Overall the program is quite powerful and scalable.  The trickiest part of the whole procedure is getting your own custom map into mScapes from something like Open Street Map, or an artfully hand drawn map, so you aren't stealing maps off Google.  So that's the part I will outline here.

When you've decided on a map to use in Windows, we'll use Open Street Maps for our example, we'll need a map saved as a .jpg or .gif.  Before we get there we will need to write some stuff down.  When you "Export" the map you've zoomed into you will see some numbers below "Area to Export".  You can either manually draw an oddly shaped map or go with the square map.  The numbers that indicate the top, bottom, left, and right of the map.  Be sure to record those clearly and in the right order.  Otherwise things will be confusing later.

Open Street Map saves as a .jpg but unfortunately you'll notice it may muck up the colours if you import directly into mScapes.  Convert it to .gif with a graphic converter program then we'll import it into mScapes.  

Once you've installed mScapes (you'll notice it'll also install the .NET framework once you run the installer), create yourself a project file so you can keep all your files in the same place, it'll get a little messy otherwise.  Open the application and create a new mScape.  Save the new mScape and you'll get a dialogue asking for a LOT of descriptive information.  Don't make the mistake I did, fill in the first field (title) because it won't give it a default name like "untitled" and you'll end up with an unlabeled file (potentially annoying).

Go to the Tools menu> then Import Map from Image.  You can select your freshly converted .gif and it will create a maplib file.  It prompts you to enter your coordniates you can do so now according to the coordinates you recorded from Open Street Maps.  Notice how it will ask for the Latitude and Longitude of each corner.  The coordinates you've recorded from Open Street Maps will be the Top, Right, Left, and Bottom lines of the box you've chosen.  Thankfully mScapes only asks for 3 of the four corners to be entered.  So the fields will be:

Top/Left:

(x) Longitude = left;

(y) Latitude = top;

Top/Right:

(x) Longitude = right;

(y) Latitude = top;

Bottom/Left:

(x) Longitude = left;

(y) Latitude = Bottom;

Bottom/Right:

(x) Longitude = right;

(y) Latitude = bottom;

Whew! Hope I got that right.  You'll know you've done it right if a Map located in North America has negative (x) Longitude values, indicating it's located in the Western Hemisphere.

After all of that, your map won't instantly appear in the window, you have to manually Add Map, and choose the newly created maplib.  I can't recall how to import a map and I'm blogging this without the program infront of me, but just to be aware if you choose at any point not to enter the coordinates in the first step, you reimport the map and will be given the option to change the coordinates of an existing map.  I'll clarify this post at a later time with more detailed instructions.  If I have the opportunity I'll create something within mScapes before I leave Banff so I'll know the program a bit better.