Introducing secushare's Multicast Publish/Subscribe API

Introducing secushare's Multicast Publish/Subscribe API

The Subscribe & Publish paradigm, recently nicknamed "pubsub," has been essential to Internet technology whenever something was intended for multiple recipients. It is an ancient usage pattern in any computer network, but since the Internet failed to provide for a generic distribution layer for all, all scalable solutions now depend on a content delivery network or cloud infrastructure and are usually owned by a Faceboogle kind of company.

secushare intends to provide a scalable publish and subscribe solution for the free and GNU Internet. This is probably what most of all sets it apart from any other privacy-enhancing project.

Video summary from ChaosWest: Scalable and Privacy-Respectful Distributed Systems - Our Chance to Avoid Cloud Computing?

Pubsub used to do a better job before the web came

Pubsub exists on the Internet at least since mailing lists where invented in the early 80's. It's also in the signaling protocols around 1992's IP Multicast and in the /join command of 1987's IRC. Or just think of subscribing newsgroups in UseNET's NNTP.

The latter was arguably one of the best implementations of the pubsub concept to date, because it didn't just model the subscription process - it also provided for a standard to efficiently distribute the content to all the intended recipients. Nodes of the UseNET news system were organized in a multicast tree network with the intention of minimizing required bandwidth, which happened to be particularly precious prior to the arrival of web commerce. Similarly, IRC organizes its servers into a multicast distribution tree.

Many lessons learned before the web were forgotten in the times after. Blogs and podcasts have been creating content, but you had to poll their RSS feeds to know if there's something new. No distribution strategy there. Discussion groups have devolved from newsgroups back to mailing lists, no distribution strategy either.

Scalability is such a boring topic, it is usually left for last - and then it is too late, when millions of people want to use your software but you can't fix it to make it scale. I'm glad that you're still reading.

Scalability is proprietary.. or absent.. or you have to roll it yourself

Frequently, in modern designs, scalability isn't even part of the equation. Google's PubSubHubbub provides a simple HTTP-based signaling method, completely leaving out the distribution aspect – knowing that, should it become popular, only a cloud architecture such as Google's would be capable of actually providing a functioning and scalable implementation of the protocol. Thus, in the current pubsub universe, the Google PubSubHubbub server is the most popular one. Surprise.

Similar case with XMPP where the scalability issues have intentionally not been taken care of during the process of protocol design, ignoring all the warning voices. XEP-0060 just adds a publish/subscribe signaling procedure on top. In order to achieve scalable XMPP applications it is therefore necessary to centralize the users on cloud-based servers, as the open federation network would not be able to perform fast enough.

All attempts to introduce Multicast into XMPP were rejected by the XSF, ironically because of perceived privacy issues. The term was even subjected to mean something else: Extension 33 just adds carbon-copy (cc:) to XMPP like it exists in e-mail. In both cases no distribution strategy is applied - it's just cosmetic.

Other protocols that implement the pubsub paradigm, but leave the implementation of a distribution strategy to the respective software vendor, or even the final user, seem to be AMQP, MQTT and ICE. 0MQ has a binding for IP Multicast's PGM of which we already know that it either isn't available or it doesn't scale - so it is only interesting for deployment within the cloud.

PSYC has been providing a pubsub interface to chat rooms, friendships and other subscription channels, garnished with a decentralized multicast strategy, since the late 90's. It has been hosting large audience chat events for media enterprises without resorting to cloud technology.

Introducing secushare's Multicast and PubSub API

In the secushare project we have chosen to adapt and optimize PSYC's approach to the GNUnet architecture, for better independence from the Internet as it has come to be. The new API reflects a revamp of the protocol and an upcoming new implementation of the pubsub paradigm, which still is a great way to model most social interactions on the Internet. The design is described in Gabor's masters thesis. See the protocol page for details.

This time we are handing out a publish/subscribe API that actually works out of the user's own computer, not depending on servers, not exposing interests to other people than to the intended ones, and protecting the flow of information in transit. Designed to function also in cases of extreme popularity such as a pop star's Facebook or Twitter account.

Now that this fundamental piece of a GNU Internet has been created, we are adapting the ActivityStreams protocol (without the cruft of later versions though), enabling developers to create user interfaces and further applications on top of an infrastructure that provides similar social functionality as the social services we are familiar with, but in a distributed and encrypted fashion.

This could be the foundation of a new free and libre Internet. Wish us luck. Or, even better.. participate, contribute. There's no business model for liberty and no funds for products in the interest of the common good, so we depend on folks like you. But if you only invest into the blockchain hype, we have some good news for you:

Private scalable blockchains for everyone

Depending on the definition of "blockchain", if, as Tony Arcieri argues a chain of proof-of-work transactions is what makes up a blockchain, then we are proud to say we are a blockchain-free technology that doesn't burn up a household's amount of electricity per pubsub transaction and as much as the entire nation of Iceland in total. Or is it Chile already?

If, instead, blockchain is just a special one-dimensional case of a Merkle tree, then we are planning to provide optional blockchains for each pubsub as we are considering to synchronize chatrooms and such using Merkle trees.

So, if you made any plans on how you could be using blockchains, with secushare you can have as many private or public Merkle trees as you like, keenly anonymized by plenty of other traffic on the network rather than being under five eyes scrutiny, not consuming large amounts of energy and scalable by design.

If you remove Bitcoin's proof-of-work from the definition of blockchain, then there are dozens of mostly scientific projects out there, which are already providing distributed blockchains since their invention in 1979.

It's interesting how we are living in times when hobbyist projects can stir up much larger hype than scientific ones. But, what do you expect from times when science deniers can become president?