Moer and more, I'm finding it necessary to do some sort of network communication. Sometimes it's to talk to hardware, sometimes there is something that needs to be captured and sent off via FTP or email. To date, I've used a mish-mash of boost::asio, third parties libraries, or platform specific libraries. The reality is that boost::asio can literally do anything related to networking. It's already a dependency in Cinder, so it makes sense to leverage it as much as possible.
A lot of these network communication tasks seem trivial because they are quite mundane. But they are necessary. I know a lot of people out there have worked on this stuff. I would like to invite you to take part in creating a central boost::asio wrapper.
Here's the outline I have around building this:
Role: Client (base class)
Simple FTP client sample application
Communicates with server sample by default
Sample which communicates with server app
Sample app which communicates with RESTful service over HTTP (to demo various methods)
OSC sample application (ie, maybe leads to boost::asio based OSC block?)
Include Arduino project
Role: Server (base class)
Subclass: FtpServer (lowest priority)
Simple FTP server sample application, receives and saves files
Subclass: HttpServer (lowest priority)
Sample app handles HTTP requests
Communicates with client app by default
Subclass: SmtpServer (low priority)
Sample application which can receive email
TCP server sample application
Communicates with client sample
Subclass: WebSocketServer (low priority)
Web socket server sample application
UDP server sample application
Communicates with client sample by default
Client class should wrap up for its server counterpart
All communication is asynchronous, using boost::signals2 for callbacks (already implemented in Client)
Persistent connections (FTP, web sockets, etc) should not block (
Anything marked "low priority" in the outline is a nice-to-have
Anything marked "lowest priority" is essentially unnecessary, but would be a nice to have just to be thorough
Do not rewrite anything that exists (ie, endpoint, socket, etc)
Look first in std:: namespace, then cinder::, then boost::
For now, all classes have a callback (build into base) which returns a ci::Buffer
Converting ci::Buffer to "useful" data will not happen in the library, but in sample applications.
Leverage CInder's ability to work with ci::Buffer to get images, JSON, XML, etc
May need to add callbacks in subclasses for specific events (ie, file download complete or aborted in FtpClient)
Do you see anything missing? Do you see anything in this plan that should be omitted, merged, or improved? Do you have something you've made that could fill one or more of these gaps? Would you like to make something for this?
I'm dreaming of a day when we don't have to grumble every time we hear "control it with your iPad from home", "attach the photo to an email", "post to social networks", "receive an email", "upload a file to server", "play audio over the network", etc. boost::asio has everything we need to make it happen and it's just sitting there neglected inside all our "cinder/boost/boost/" folders. We just need to put it to work.
Also, one thing that would be useful, a bit out of scope, but something to think about for future integration when posting to social networks is using OAuth.
From what I've seen most OAuth implementations require a user to sign in through their facebook, twitter, linkedIn, foursquare, etc. etc, account info..Twitter is still supporting their old REST api at the moment that does not require OAuth, but it's becoming a standard to use OAuth for other social networks.
Anyways, it seems that there would need to be some sort of integration into a web browser via Awesomium or some other method.
@zsolt_ero Well, I don't plan on doing it on my own. I'm hoping that as people need to build things like a FTP client, mail server, etc, they'll build it around boost::asio and contribute it towards the library. I seem to constantly have a need for things like TcpClient, UdpClient, WebSocketClient/Server, and SmtpClient/Server, so I'll likely be filling those in.
@justin OAuth should definitely be in there, but that would probably be a sample app using HttpClient. The primary need for a HttpClient class in this environment is mimicking a request from a web browser.
I'm all for this. In dev we now have the App maintain an ASIO io_service, which should simplify integration in the cases where you don't want a separate thread/io_service combination. Additionally I have a reasonable implementation of ASIO-based HTTP Server which I can clean up and contribute once we get 0.8.5 out the door.
@justinmaurer I believe Oauth 2 uses SSL. I'm not sure if it's the best strategy for integrating Oauth but I have experience integrating OpenSSL into Cinder apps. Working with OpenSSL is NOT fun at all, but that's all the more reason it would be valuable to have in tandem with boost::asio logic. Perhaps that should be a separate thread though(?).
Otherwise, I would be interested in contributing stuff. For example, I have an HTTP streaming client for streaming video playback. This is surprisingly hard to get from Cocoa too. Even though it exists "out of the box" for iOS. I think the same thing may also be valuable for streaming audio...
I'm going to start a dev branch where folks can contribute.
@sy1vain Will take a closer look for SmtpClient. I noticed that your class is set up to use an internal shared_ptr, but is also used as an explicit shared_ptr. So it's basically a shared_ptr to a shared_ptr. I would recommend moving everything in the internal object to the class level. That said, it looks like you've put a lot of work into it and it should be a big help.
@cloblob Yes, streaming media is a must -- and part of the impetus for this library. Would love some help there.
I removed the internal shared pointer in the mailer class, just to be neat(er). It still isn't the neatest code, just really getting into cinder and c++ and it kind of was a rushed job. The thing to peek at is how an e-mail message is built with all the different MIME-parts so that it can both support attachments as inline attachments.
Caleb Johnston has been a huge help in bringing this together. The library has been simplified dramatically. We're currently working on implementing server classes and building out test/sample applications (check the dev branch).
Something I realized as I dug into the library and looked at a few others, is that I was making the same fatal flaw which overcomplicates similar APIs.
boost::asio is strictly about transferring data from one place to another. It is completely agnostic of what that data is. boost doesn't care if I'm following the right protocol to send or receive an email, nor does it mind if I'm working with a proprietary binary format to talk to networked micro-controllers. It doesn't care if I'm moving audio or video data between devices or if I want to fire off a billion 0's for no reason.
Therefore, a boost::asio wrapper should only cover transport, not protocol. A properly designed protocol library can be used with this block, or WinINet, or Posix, or anything that can send/receive data (ie, boost::asio directly), but should not be integrated directly into it. Ideally, a protocol API will manage things like request/response lines, header fields, security, etc. This API might act as a framework for simplifying tasks like email, FTP, etc.
For the moment, we're relying on sample applications to demonstrate how to use the block for email, FTP, HTTP video streaming, etc. The idea is these samples may grow into their own blocks, or a larger protocol library which you can use with any transport layer library.
What's the motivation to wrap asio? In my opinion, what we really need is the protocol api's, TcpClient, HttpClient... asio's job is the transport and users of it write the protocol (they also provide samples like you are describing).
It would be helpful to make both easier to use. And more importantly, the two should be decoupled from each other. There shouldn't be a HttpClient. There should be a TcpClient and a HttpRequest/HttpResponse.
Internet communication is a just a portion of what boost::asio can do. It's how you communicate between devices. It's a way to talk to hundreds of networked micro-controllers sending tiny pieces of proprietary data, just as it is a way to stream video, load a web site, send an email, etc. Personally, I usually use boost::asio with hardware that doesn't adhere to any sort of protocol.
Despite the fact that boost has tutorials on some basic Internet tasks, it's less-than-trivial to implement a high performance, generic, dead-simple, reliable interface to the technology. One that is asynchronous/multi-threaded, yet maintains order/sequence. And one that integrates nicely into Cinder. The asio wrapper we're finishing up now addresses these issues.
As we build the samples, we'll be looking for patterns to start working on an extensible protocol API. There will probably be Request and Response base classes which have a status line, header fields, and a body. This generic set up plus sample apps might be enough. Or maybe it will make sense to go a step deeper and create rqeuest/response classes for specific protocols.
I hope I don't sound like I am discouraging this initiative when I write this, because I think its great one and we could all use better networking support, but gotta write it anyway..... I don't think this is a wheel that needs re-inventing. I think a common, low-level interface to boost::asio may help to organize a networking library (or 'communication library' if you prefer), but in the end when I start a project with a 2 week deadline and there is an http server I need to talk to, I need a high-level HttpClient (like this one, or this one, or this one, or... well you get the picture :) with an intuitive interface. Actualy by this time in the Internet Revolution, the interfaces pretty much all look the same / accomplish the same things, although implementation may vary because of language constructs.
About your response to have HttpRequest + HttpResponse but not HttpClient, you'll see in most frameworks they have all three - a request is no good without a response and vice-versa, and the client is the guy who organizes how those two get along (e.x. OAuth or timeouts).
Maybe I'm jumping ahead too far and again, I don't mean to discourage the work you and Caleb have already accomplished, I just mean to shed light on what currently missing from the C++ / Cinder toolbox in hopes that we'll overcome that.
@reakinator I completely agree with your sentiments about a missing communication which adheres to protocols in Cinder. But all that means in this context is that Cinder-Asio will be a base layer to something that will need to be a larger and lengthier effort. In cases like yours (2 week project using off-the-shelf communication), it's probably wisest to use a package that supports the full set of features you need. But we can't ignore that there are many of us (not just Steve and I) who still need the low-level structures to be organized. Boost::asio is quite bare bones for creating a custom client-server and/or peer-to-peer communication system. This project simply organizes the boost parts in a Cinder-friendly way.
Additionally, most commonly used protocols can be very intricate. To my surprise, even HTTP has tons of infrequently used fields that various servers interpret differently. Check out this rant to get an idea of what I mean:
I don't have the UDP side finished just yet, but it's coming shortly. It looks like there are a lot of classes, but generally, you'll just use one or two.
TCP and UDP each have a Client and Server. The role of these is to create Sessions from connections. The Session is handed to the user in a callback and then that's what you use for reading and writing. There is one example right now, HttpClient, which simply loads the HTML of libcinder.org. But it's worth a quick study. Separating the "Session" from the rest adds a step, but greatly increases the power and flexibility of this block by allowing a practically unlimited number of sessions to run concurrently.
There is also a WaitTimer class, which is basically just a handy asynchronous timer that leverages boost::asio.
The nice thing is all of it is powered by the boost::asio::io_service that is already running in your Cinder app. :)
Just trying to get this working and having trouble with the TcpServer on OSX (10.8.5). I already switched to the dev branch since I noticed you found a bug in the TcpServer sample, but it's not working out for me.
What version of boost and Cinder is this code written against? Wondering if that might be an issue as I am working from 0.8.5.
Server::cancel() is at the root of all the crashes I'm seeing (called in the dtor and at the end of every message received). I'll see if I can fix it up and make a pull request. Alternatively, if you have a fix and it's just not on github, that would be thrilling.
The error is actually happening when accept() is called in the main application. On quit, I believe the issue is that mServer no longer exists when we try to use it. Why accept() fails after each message is received, I'm not sure. Digging.
EDIT: The weird thing about quitting is that the breakpoint inside that conditional is still reached.
Cool. I fixed the big error with a one-line addition. Unfortunately, I also changed all the line endings by opening the file on OSX. The crash-on-exit can be fixed by disconnecting the event handlers in the main app's destructor (setting them all to nullptr).
Hey, this is awesome that you put together this code. I'm intrigued by the possibility of handling all network communication via a unified clean interface so I thought I'd give it a try to use for a udp client.
Looking at the sample udpclient app code I have a couple of questions. Why am I required to provide an ip for a udp client? AFAIK, it is typically the udp server which maintains the ip address to the clients. Also, why is there only an example of sending data from client to server and not reading data from the server into the client? That seems to be the most common use case for a client. Anyway, I tried getting it to work with my udp server application for the last hour or so but no luck. Let me know if there are any tips and I will try again some other time. Thanks!
> Why am I required to provide an ip for a udp client? AFAIK, it is typically the udp server which maintainsthe ip address to the clients.
The IP:PORT pair specifies where the client is to send datagrams. Even when broadcasting, an IP:PORT is needed.
> Also, why is there only an example of sending data from client to server and not reading data from theserver into the client? That seems to be the most common use case for a client.
I *think* think is because TCP was the priority and UDP isn't complete yet. If you look at the UdpServer, the read eventhandler only takes a buffer, it does not have any information on the sender, making a response ratherdifficult. I filed#6 Unable to divine sender from UdpServer read event handlerwhich with a workaround, but I have to imagine there is a better way than provided.
One of the reasons why the native socket is exposed is to do exactly what you're asking about. You need to create a custom event handler structure and map it to the session. Your event handler should have a member pointing to the session's socket (or the session itself) and anything else you want (ie, a ci::Buffer). When the event handler is finished reading, the main app can grab any data it needs and erase the session/event handler pair from the map.
EDIT: You can also just use a std::list (not vector) to store callbacks, or some other mapping ID. I forget why, but I needed the originating session to be the key in my map on this one project.
It's OK to leave a session open, but make sure you send future requests through the existing session rather than open new ones. Otherwise, close your session after the round trip is complete and use a new one for the next request.
Which port number are you using? If it's over 5000, try something lower.
thanks for the great block. I've used it for TCP before but now need to use it for UDP for the first time and am having some issues (I need to send messages to some lighting hardware using the DDP protocol and plan to create a block when working)
I got a test app sending messages to the example server app no worries but I notice that I am not able to send UDP messages if a server is not running. This also prevents me from being able to use broadcast IPs.
UDP shouldn't require establishing a connection first but I can't see a way to use Cinder-Asio without doing this.
However, I am getting connection refused error when trying to send UDP messages to the specific IP of my hardware device, where as I think I should be able to send messages to the IP whether or not it's there, no? (although I can ping it)
Are there any other settings I should be looking at? Excuse my ignorance but I am not very networking/socket savvy - I browsed what's available in boost::asio::socket_base:: but didn't see anything obious to try.
Hey, all. I've made some pretty big updates to the ASIO block.
I've added two new samples called "MultiUdpClient" and
"MultiUdpServer". It's essentially an echo set up that
works with multiple clients. The server app demonstrates how to
use the new event handler interfaces track multiple connections and talk
back to them. The client sample sends data off to the host and
then listens on the same port for a response from the server.
The event handler interfaces are totally optional, but they are
really helpful to give you a nice starting point for making event
handlers that you can tie to sessions.
All of this points to some revisions under the hood for both UDP
and TCP. Namely, the right socket options are being set at the
right time. This also address all of the issues reported on the repo.
Also, looking for commentary on whether or not MultiTcp* samples are
needed. I added local and remote endpoint getters to UdpSession,
but not TcpSession. With TCP, the connection just stays open which
allows you to talk back without ever needing that info. If you
want to respond, just write to the still open session. With UDP,
you can't get the endpoints from the socket with the block because by
the time you ask for it, the connection is closed. The endpoint getters
were added to UdpSession because they can really only be divined during
If you do need the endpoint of your TcpSession, you can get it
from the socket. I'm just wondering if this really needs to be
demoed or not.
Sorry to resurrect this old thread but I am stuck getting multiple TCP clients to connect to one server - seems I could use that sample! onAccept in the server only gets called for the first client. Details over on the new forum here. Cheers.
Leave a comment on naychrist's reply
Change topic type
Link this topic
Provide the permalink of a topic that is related to this topic