Raspberry Pi Robot with PiShield, Part 3: Fast video streaming

I fiddled for a good part of the day trying to find a low-latency local network video streaming solution to implement a "FPV"-like control for the robot using a RPi camera module. It turns out a bunch of existing solutions (like VLC streaming etc) have really high latencies. This is OK for things like a security camera, but not so much for realtime control.

Finally, this rather hacky solution using netcat appeared to work best:

1. On the client (viewer/controller, running on a Mac) side, run:

 nc -l 5000 | mplayer -fps 30 -cache 1024 - -x 640 -y 360

 2. On the RPi side, run:

 raspivid -n -t 999999 -fps 12 -rot 180 -o - | nc IP_OF_VIEWER 5000

 netcat is used to listen and send; raspivid grabs the video and pipes it to netcat, and on the other side netcat pipes the received data and spits it into mplayer.

 This was the only way I could get less than 1s of latency on the video feed, although it still wasn't great. Having a good USB wifi dongle also helped a bit. Finally, changing the dimensions of the video using raspivid actually made it much slower, probably due to software resizing of each frame on the fly compared to just sending out the raw feed from the camera.

 Next steps: get some more interesting sensor info from the PiShield, and get a separate power source since there seem to be random system freezes when the motor and Pi are both running off the same power bank.

UPDATE Oct 26:

with the advice of a very helpful redditor who suggested gstreamer, I gave it a shot this evening.

On Raspbian, gstreamer1.0 is currently already in the official repos, so no need to add custom sources as these earlier instructions.

For OSX, I just downloaded and installed the latest compiled version from the official source. It also handily tells you that by default the path for the commands are at

/Library/Frameworks/GStreamer.framework/Commands/

From here, we can either have a tcpserver on the Pi that a client can connect to, in which case:

RPi:

raspivid -t 0 -h 720 -w 1080 -fps 25 -hf -b 2000000 -o - | gst-launch-1.0 -v fdsrc ! h264parse !  rtph264pay config-interval=1 pt=96 ! gdppay ! tcpserversink host=minibian.local port=5000

Mac (note explict path to gst-launch-1.0 command, since I haven't added it to path):

/Library/Frameworks/GStreamer.framework/Commands/gst-launch-1.0 -v tcpclientsrc host=IP_OF_RPi port=5000  ! gdpdepay !  rtph264depay ! avdec_h264 ! videoconvert ! osxvideosink sync=false

Now UDP would be faster, and in this case it would be similar to the netcat example where the RPi would define the UDP "sink" using the IP address and PORT of the viewer (on the Mac), which would yield the following:

RPi (note destination should be explicitly defined by the source this time):

raspivid -t 0 -h 720 -w 1080 -fps 25 -hf -b 2000000 -o - | gst-launch-1.0 -v fdsrc ! h264parse !  rtph264pay config-interval=1 pt=96 ! gdppay ! udpsink host=IP_OF_VIEWER port=5000

Mac:

/Library/Frameworks/GStreamer.framework/Commands/gst-launch-1.0 -v udpsrc port=5000  ! gdpdepay !  rtph264depay ! avdec_h264 ! videoconvert ! osxvideosink sync=false

Eye-balling the two versions, I feel like the UDP version is slightly snappier, but considering that my network is not very congested, I have a feeling the difference wouldn't be as much as if I had a lot of traffic running. Regardless, this is a HUGE improvement over the existing solution! Should try to do some timing tests to see if I could get a better measure of the actual latency - perhaps using a photo of a stopwatch like this guy did (which was one of many sources I originally consulted for the netcat solution above as well)!

finally, here's a test on the actual video latency of the setup above: looks like its somewhere between 150 and 200 ms.

FINAL UPDATE (Jan 2017)

Yet another option, is to use RPiCamWebInterface. In the end I didn't get around to do the latency measurement, but it feels *almost* as fast as the previous solution. The other bonus is the client user interface is far nicer!

Here's a sample video showing the stream in action:

Raspberry Pi Robot with PiShield, Part 2

The previous post showed the basic functionality of the Raspberry Pi Robot. The Arduino-based motor driver takes in commands to drive the robot in different directions, can be controlled remotely in a serial terminal via SSH using the screen application.

This is what the system looks like:

The Raspberry Pi talks to the Arduino via USB serial. The movement commands are sent from the Pi to the Arduino, currently, 'w', 's', 'a', 'd' to toggle movement, and ' ' (space) to stop. there are also ASCII numbers 1-6 that correspond to PWM values to control the speed of the motors. While it would be possible to control the motor driver with the GPIO pins of the RPi directly, I had that other circuit all hooked up from a previous project, and all it required was a single USB cable to the Pi, so I went with that instead of rewiring everything.

This time, we want to start doing something a bit more fun. Here's a preview of the end result:

The premise of the example is quite simple: get too close to the robot, and it runs away from you!

We use Python this time to implement the sensing behaviour. The source code that implements the above is available here. I call this one the PyPiBot as it runs on Python... ;)

Basically, what it does is the following:

Initialization:

1. Open SPI device for getting sensor data from the PiShield 

2. Open serial port (for sending commands to the motor driver)

Main loop:

1. Get data from sensor port

2. If sensor value is greater than a certain threshold trigger a move forward routine:

   - send a 'w' to the robot, which puts it in forward mode

   - wait a second (sleep)

   - send a ' ' to make it stop

3. Check for ctrl+c and quit the program

What you notice about this particular script is that it is blocking, which means that while the robot is in the move forward mode and the script sleeps, you can't respond to anything else. We'll tackle that in another example in the future!

Raspberry Pi Robot with PiShield, Part 1

Using the base from Tippy, I've started building a Raspberry Pi powered, sensor enabled robot. Currently, you can SSH into it and control it via the serial port. The original Arduino-based motor driver from Tippy is still there, but in theory we could just send out PWM and control signals from the Pi directly.

Main components:

- Raspberry Pi Zero with inline 2-port USB adapter:

- - Port 1: wifi dongle

- - Port 2: Arduino USB serial

- PiShield Sensor Interface (current Kickstart project here)

- 5000 mAh OCHO mobile power bank (1A and 2.1A outputs).

- Arduino and Sparkfun motor driver. Its been a few years since I ordered that part. This one has two channels with PWM input for each channel for speed control. There are now ones that fit directly on the Raspberry Pi as well which could be more convenient to use...

- Tamiya Track and wheel set. Really handy little platform for making a basic tracked robot. One of the cheapest options out there too.

My idea is to add sensors to the PiShield, and then have it react to local environment conditions in addition to receiving wireless commands. The software I'm running on the Pi is the latest minibian, which usually boots up and gets an IP from my router in about 10~15 seconds.

Could this have been done on an Arduino or even an ESP8266 based board? Sure - but its kinda neat to be able to run linux on-board in a relatively compact package. Stay tuned for more updates!

p.s. apologies to /r/cablemanagement. This part is clearly not my strong suit... ;)

Tippy for ICEC 2011

What is Tippy?

Tippy:

- is a Telepresence Robot

- has a simple microcontroller-based drive system

- runs on ANY mobile device with a front facing camera and 2-way video application (such as Skype)

- uses optic coupling to provide a novel interface between the mobile device and drive control hardware

- does not require any other mobile app for operation

- costs less than $100 in parts (excluding iPod touch) if you build it yourself

Tippy allows the user to achieve telepresence by leveraging powerful, existing mobile devices quickly and easily. Tippy can be easily packaged into other formats using different sized devices (smartphones, handhelds and tablets) without custom mobile applications and the control scheme can be adapted to control any kind of external hardware.

For more information, take a look at our conference paper HERE.

To see Tippy in action, check out the following video:

If you're interested in finding out more, please find our contact info here!

Tippy the Telepresence Robot

So Vincent, Benny and I have been working on a cute little project over the past few weeks. It's a compact telepresence robot, similar to the one that the famous Johnny Lee wrote an instructional for on his procrastineering blog. The key factor of our design is the optical coupling between the mobile device and the robot control hardware: in our implementation, we use Skype for two-way video but embed the control signals in the video stream as well - thereby reducing the amount of development (in terms of hardware and software interfacing). We've submitted an ICEC demo paper along with the following video, which should do a slightly better job at explaining how it works:


We hope to get together an instructional, and a DIY kit soon so people in the community can play around with the idea (if the video isn't enough to get you going :)

A few points on my personal motivation/comments for this project:

1.) Robots are cool; telepresence robots are also cool.

2.) Mobile devices are so powerful these days that you can do so much with them. However, for the DIY hobbyist/hacker, it's not that easy to interface these devices with custom hardware that you build. For example, on an iOS device (iPhone, iTouch, iPad etc), you are quite limited when it comes to interfacing hardware. Even if you do have a developer license, it's still pretty hard to get a physical connection to the serial port working (at the risk of voiding the warranty and blowing up the device, etc). Using platforms such as Android does away with the license hurdle, but the hardware interface is still not easy. This project is essentially a demonstration at a quick and dirty (but working!) solution to this problem. There are many ways that it can be improved (e.g. modulating the visual signal, auto-calibration of the sensors, etc etc), but it should provide a good starting point.

3.) I'm surrounded by a bunch of amazing people with different skills and really appreciate the opportunity to work with them. This project was definitely worth the evenings and weekends we spent at home or in the lab tinkering away. Good job guys!