2011-10-13

Ach Track

Attendees

• Nick Armstrong-Crews
• Brian Gerkey
• Geoffrey Biggs
• Joe Romano
• Nico Hochgeschwender
• Bob Dean
• Azamat Shakhimardanov
• Mike Stilman
• Neil Dantam

ROS Threading and Communications Model

• One thread for sockets and XML-RPC
  • Multiplexes file descriptors with poll/select
  • Puts messages into queue
• One thread to process messages
  • ros::spin()
  • processes the messages queued up by the socket thread
• Other threads
  • Can register specific callbacks with specific threads
  • Can start up other threads as desired

ACHROS Approach

• Inital Code

• To minimize latency, it is desirable to minimize context switching.
  • Thus, the thread that pulls the message out of the channel should be the thread to handle (invoke the callback on) that message.
  • Callback is stored in the per-topic subscription object
  • ACHROS could pull that callback out of the subscription object and
    • invoke it directly.
• The File Descriptor / Condition Variable Disconnect
  • ROS topics accessed via file descriptors which are waited on in a
    • poll/select
  • Access to ach channel is mediated by a mutex/condition variable
    • which subscribers will wait on
  • POSIX offers no way for one thread to wait on multiple condition
    • variables
  • Two options to access Ach channel
    • (1) Poll the channel
    • (2) Wait (on condition variable) for the channel
    • Both used by GT-Humanoids, waiting when process has only couple
      • channels, polling when there are many channels.
• SVN
  • Electric already branched out
  • Work off of trunk
  • Even/Odd Stable/Unstable version numbering
• ROS REP 106 describes a polling API for ROS subscriptions. This
  • might be a good fit for Ach's model which supports both waiting and polling for messages.

TODO

• Nico: try to build Ach
• Neil: continue Ach integration and send patches to Troy Straszheim
  • at WG

WIFI Track

Specific Issues from Lincoln Lab Group (Nick)

• Project: multi-robot mapping, teleop with video and laser data to
  • operator
• Problems when out of WIFI range, messages queue up then burst when
  • back in WIFI range, even with UDP.
• Bufferbloat: the WIFI layer detects packet loss (no ACK of WIFI
  • packet) and retransmits. This causes large and unacceptable latency.
• Used Linksys Router and 1w (WIFI max power) Ubiquiti router.
• Sending large UDP messages works fine when extra network capacity

QOS

• Lincoln developed network monitor to track health of connections, ie
  • ping times
  • Communicates data as ROS messages
  • nodes can then dynamically adjust ie jpeg compression based on
    • available bandwidth
• Possible to release the Network Manager
  • 45-day review process at LL
• Want some way to request a certain amount of network bandwidth
  • LL usings trickle/IPtables to enforce contracts
• Perhaps ROS needs a "Network Master" to manage QoS
• Question: what are the IP and Kernel layer QoS options?

Kinect/Pointclouds over WIFI

• Some Smart compression for point clouds
  • Uses deltas, so good for static scenes but bad for mobile robots
• Naive GZIP stream compression: not good enough
• WG sends point clouds over WIFI
  • low frequency
  • takes over network

TODO

• Nick: Start release process for LL Network Monitor
• Nick: Check Network/Kernel QoS options

2011-09-22

Attendees

• Joe Romano, PhD at UPenn, interest in haptics and feedback control
• Brian Gerkey, Director of Open Source at Willow Garage, did lots of early ROS development
• Azamat Shakhimardanov, PhD at Bonn, interest in control and middleware
• Mike Stilman, Professor at Georgia Tech, interest in planning and control
• Neil Dantam, PhD at Georgia Tech, interest in formal methods for control
• Patrick Mihelich

IPC Use Cases

1. Real-time control: minimal latency
2. Vision-Pipelines: minimize copying and latency
3. Wireless communication over unreliable links

Multiple Transports

• Different applications probably require different transport semantics and implementation tradeoffs.
• Appropriate to support multiple transports, and ROS provides some infrastructure to do this.
• Implement multiple transports. Community can try and decide which are appropriate.

ROS Transport Status

• Supports UDP and TCP

Multi-Transport Support

• Trivial to add to XMLRPC protocol
• More work to add roscpp implimentation.
• Keeping the same external API across multiple transports also difficult.

Current WIFI Issues

• TCP: stream backs up when many packets dropped
• UDP: multi-packet messages (ie big images) get lost when any packet in the message is dropped
• If initial connect to publisher fails (due to dropped packets), subscriber assumes publisher is dead.

Fast, single-host transport

• Real-Time control needs minimal latency
• Vision needs minimal copies and low latency
• Vision probably will still requires each process to copy the message to its own address space.

ZeroMQ

• Socket-like interface
• Can be run over unix-domain sockets
• Also does networking

RCSLib / NML

• Supports single-slot buffer or queued messages.
• Multiple subscribers will each "remove" a message from the queue, preventing others from seeing it.
• Handles serialization also via XDR.
• Does networking, reliable/unreliable message transport.

Ach

• Unique semantics for message-passing.
• Always access newest message. Older messages on best-effort basis.
• Distribution to n subscribes require n+1 memcpy's. (Point-to-point sockets generally take 2*n memcpy's due to in-kernel buffer)

Initial Plan

Two Tracks

1. Investigate wireless transport needs.
2. Integrate Ach as a roscpp transport. Maybe stretch internal API. Try not to change external API.

Individual Tasks

• Everybody: Consider transport-over-WIFI needs and investigate options.
• Brian: Poll ROS users on their needs for wireless transports.
• Neil: Get Ach building as a ROS package. Check roscpp code to for how to integrate Ach.
• Joe: Test a ros-integrated Ach.
• Azamat: Review API differences between ZeroMQ, RCSLib, and Ach

Future Work

• Use Ach integration experience to integrate other transports (ie, ZeroMQ)
• Ach-ros bindings for other languages (ie Python, Common Lisp)

Next Meeting

• No need for full meeting at IROS, but introductions are good.
• Continue discussion on mailing list.
• Skype again in a couple weeks to make sure everyone is caught up.