Introducing secushare's Multicast Publish/Subscribe API

Introducing secushare's Multicast Publish/Subscribe API

The Subscribe & Publish paradigm, recently nicknamed "pubsub," has been essential to Internet technology whenever something was intended for multiple recipients. It is an ancient usage pattern in any computer network, but currently all scalable solutions depend on a cloud infrastructure and are usually owned by a Faceboogle company.

secushare intends to provide a scalable publish and subscribe solution for the free and GNU Internet. This is probably what most of all sets it apart from any other privacy-enhancing project.

Pubsub used to do a better job before the web came

Pubsub exists on the Internet at least since mailing lists where invented in the early 80's. It's also in the signaling protocols around 1992's IP Multicast and in the /join command of 1987's IRC. Or just think of subscribing newsgroups in UseNET's NNTP.

The latter was arguably one of the best implementations of the pubsub concept to date, because it didn't just model the subscription process - it also provided for a standard to efficiently distribute the content to all the intended recipients. Nodes of the UseNET news system were organized in a multicast tree network with the intention of minimizing required bandwidth, which happened to be particularly precious prior to the arrival of web commerce. Similarly IRC organizes its servers into a multicast distribution tree.

Many lessons learned before the web were forgotten in the times after. Blogs and podcasts have been creating content, but you had to poll their RSS feeds to know if there's something new. No distribution strategy there. Discussion groups have devolved from newsgroups back to mailing lists, no distribution strategy either.

Scalability is such a boring topic, it is usually left for last - and then it is too late, when millions of people want to use your software but you can't fix it to make it scale. I'm glad that you're still reading.

Scalability is proprietary.. or absent.. or you have to roll it yourself

Frequently, in modern designs, scalability isn't even part of the equation. Google's PubSubHubbub provides a simple HTTP-based signaling method, completely leaving out the distribution aspect – knowing that, should it become popular, only a cloud architecture such as Google's would be capable of actually providing a functioning and scalable implementation of the protocol. Thus, in the current pubsub universe, the Google PubSubHubbub server is the most popular one. Surprise.

Similar case with XMPP where the scalability issues have intentionally not been taken care of during the process of protocol design, ignoring all the warning voices. XEP-0060 now just adds a publish/subscribe signaling procedure on top. In order to achieve scalable XMPP applications it is therefore necessary to centralize the users on cloud-based servers, as the open federation network would not be able to perform fast enough.

All attempts to introduce Multicast into XMPP were rejected by the XSF, ironically because of privacy issues. The term was even subjected to mean something else: Extension 33 just adds carbon-copy (cc:) to XMPP like it exists in e-mail. In both cases no distribution strategy is applied - it's just cosmetic.

Other protocols that implement the pubsub paradigm, but leave the implementation of a distribution strategy to the respective software vendor, or even the final user, seem to be AMQP, MQTT and ICE. 0MQ has a binding for IP Multicast's PGM of which we already know that it either isn't available or it doesn't scale - so it is only interesting for deployment in the cloud.

PSYC has been providing a pubsub interface to chat rooms, friendships and other subscription channels, garnished with a decentralized multicast strategy, since the late 90's. It has been hosting large audience chat events for media enterprises without resorting to cloud technology.

Note: PubsubHubbub is also mentioned on the XMPP page.

Introducing secushare's Multicast and PubSub API

In the secushare project we have chosen to adapt and optimize PSYC's approach to the GNUnet architecture, for better independence from the Internet as it has come to be. The new API reflects a revamp of the protocol and an upcoming new implementation of the pubsub paradigm, which still is a great way to model most social interactions on the Internet.

The design is described in Gabor's masters thesis. See the protocol page for details.

This time we are handing out a publish/subscribe API that actually works out of the user's own computer, not depending on servers, not exposing interests to other people than to the intended ones, and protecting the flow of information in transit. Designed to function also in cases of extreme popularity such as a pop star's Facebook or Twitter account.

Now that this fundamental piece of a GNU Internet has been created, we are adapting the ActivityStreams protocol, enabling developers to create user interfaces and further applications on top of an infrastructure that provides similar social functionality as the social services we are familiar with, but in a distributed and encrypted fashion.

This could be the foundation of a new free and libre Internet. Wish us luck. Or, even better.. participate, contribute. There's no business model for liberty, so we depend on folks like you.