The Pigeon protocol is an off-grid, serverless, peer-to-peer protocol for building software that works on poor internet connections, or entirely offline.
HIBERNATION. Pigeon was an exploration of ideas that I embarked on in 2020. Now (2021) I am changing my focus. If more people are interested in the project I might start work on it again. Please let me know by raising an issue. For now, I am shifting my focus to other areas unless interest in the project changes.
- Introduction
- Why build software that follows the Pigeon protocol?
- How does the Pigeon protocol benefit users?
- Use-case scenarios
- Software implementation ideas
- Implementations
- How does the Pigeon protocol work?
- What do Pigeon messages look like as data?
- Constraints and Design Philosophy
- What the Pigeon protocol is not trying to address
- How the Pigeon protocol differs from Sneakernet
- How the Pigeon protocol differs From Secure Scuttlebutt
- Influences
- Getting involved with the Pigeon protocol development
- For more information
This document will describe why you may want to build software that follows the Pigeon protocol, how the protocol works, and how you can write software that follows this protocol.
This document provides a reference to the Pigeon protocol specifications, and is written for developers.
Software that follows the Pigeon protocol can operate online or offline, regardless of connectivity quality. This allows you to focus on software development, instead of worrying about network types and quality, or third-party services.
The Pigeon protocol outlines several ways users can benefit from using software that follows this protocol.
This section consists of the following subsections:
- Data is backed up by default
- Data can be shared anywhere
- Data is tamper-resistant
- Data is always available
Software that follows the Pigeon protocol allows users to store data on their nodes, and copy their data to other nodes. By copying data to other nodes, this not only backs up data, but also provides users with data they can access offline.
Every user in the mesh has a local database. You can share your database entries anywhere, such as CD-R, email, public forums, and USB dead drops.
Entries in the database are cryptographically signed. Because the entries are signed, you don't have to worry about malicious third parties tampering with your data.
Any data the user is currently accessing is already available offline, and in case of a sudden internet outage, or poor connectivity issues, the access to the data is not affected, because the data doesn't rely on an internet connection.
You can "follow" peers, by copying their database to your machine. Updates are performed by retrieving newer bundles from your peers. This allows you to share your peers' bundles on behalf of them. This is useful when the peer is unavailable and another peer wants their bundle.
The act of sharing peers' data on behalf of other peers is called "gossiping".
The Pigeon protocol may be useful in scenarios such as:
- Systems with low connectivity or uptime such as remote sensor logging, maritime systems, solar systems with intermittent power, IoT systems with poor network connectivity.
- Store-and-forward message gateways, such as a data mule.
- Censorship resistant applications, such as peer-to-peer messaging and blogging.
- Delay tolerant networking
- Applications that require a high level of data-integrity or auditing.
- Delay-tolerant peer-to-peer social networks, games, file sharing etc...
- Time series data storage
- A messenger application
- An IoT data logger
- A newsgroup or NNTP analog
- A turn-based board game
- A microblogging social network
- An e-commerce application for P2P cryptocurrency users
- A Secure Scuttlebutt import / export / gateway tool
- A social mapping / point-of-interest sharing site
- A Bluetooth file sharing application
- A graphical database browser for developers that want to use the protocol for log storage or as a time series database
- A synchronization system over actual pigeons, possibly soliciting help from the world-famous boxer and pigeon-racing enthusiast, Mike Tyson
There is currently one working implementation available here. A long term goal is to support all major platforms and languages.
Each node in a social network of peers has a local "log". The log is an append-only feed of messages written in an ASCII-based serialization format.
Messages are signed with a secret key to validate a message's integrity and to prevent tampering by malicious peers.
Nodes in the social network "follow" other logs from peers. Nodes replicate the logs of their peers and "gossip" information about peers across the social network.
Gossip information is packaged into "bundles" which contain backups of peer logs in a format that can be transmitted through sneakernet, direct serial connection, or any high-throughput medium, regardless of latency.
Log synchronization through Sneakernet is the main use case for Pigeon messages to be transmitted. SD Cards sent via postal mail offer an excellent medium for transmission of Pigeon messages, although any data transfer medium is possible.
Pigeon messages use a custom ASCII-based encoding scheme, as seen in Example 1 and Example 2 below.
Note: Although the message encoding scheme is human-readable the messages should be parsed by third-party applications rather than written by hand.
Example 1:
author USER.R68Q26P1GEFC0SNVVQ9S29SWCVVRGCYRV7D96GAN3XVQE3F9AZJ0
depth 0
kind my_blog
prev NONE
current_status:"Another great day. Nice weather, too!"
signature 2VMAG4SCX5RHVBKCB1RNZCB0AJN4WN6FEMS7W9FM1CVYSZXMX7CPQFCDPYEKCTGG91Y1YSGY4G5K8XAGQ67HEPDFRMRYQHWQBATAC2R
Example 2:
author USER.R68Q26P1GEFC0SNVVQ9S29SWCVVRGCYRV7D96GAN3XVQE3F9AZJ0
depth 1
kind like
prev TEXT.6CBA4J3756A5SNM1W1GHNCTT9EG95ZP3ZMAT5Z1EJP7TXMNNVZC0
liked_post_multihash:TEXT.6CBA4J3756A5SNM1W1GHNCTT9EG95ZP3ZMAT5Z1EJP7TXMNNVZC0
signature Y34Q47V0BY370RM5KWGRJRN9HFNGJN0C3DEYVB2V2476CW9RN5HD4XD7KMQ6T4T42N36R5P3XX6E3FYEWVZR25AVCF6KQPZHJP6EM10
For examples of how messages are handled, see the Ruby-implementation tutorial here
-
Offline-first means offline-only. Never incorporate TCP or UDP features. TCP- or UDP-related concerns must be handled by higher-level protocols or by application developers. This is to ensure the protocol is always an option for off-grid usage.
-
Natural is better than simple. Convention over configuration. Do not make plugins for common use cases unless it would hurt portability.
-
Prefer a monolithic internal structure. Avoid external dependencies except for limited use cases (Eg: crypto libs). Do not break things into smaller pieces until there are at least three real-world reasons to do so. Decoupling a library into a package for only 2 use cases is not acceptable.
-
Maintain ecosystem diversity by having a protocol that can be easily and entirely ported to new languages and platforms.
-
No singletons. No signing authorities, no servers of any kind, even locally, no differentiation between peers (eg: no "super peers").
-
Configuration is always a design compromise. We will allow a limit of 10 configuration options for all eternity. These are simple key/value pairs. No nesting, no namespacing, no dots, no dashes, no nested config names, no arrays, none of that crap. Seriously, I'm watching you.
-
Assume CPU and RAM are not plentiful.
-
Assume platform has no networking support. No servers. No hooks for startup, shutdown, or reboot.
-
Assume block storage is plentiful when making resource allocation trade-offs.
-
Files are better than sessions, but be filesystem agnostic. Persistence mechanisms are implementation-specific.
-
Provide tamper resistance. Privacy features will be added later.
-
Enable "Free listening"
-
Have a formal specification (reference implementations are not OK).
-
Minimize conceptual overhead (If it's not needed at least 80% of the time, don't add it).
-
Use a serialization format that is deterministic and easy to parse on constrained devices.
-
Backwards compatibility. Numerous compromises have been made to support legacy systems, such as devices that use FAT16 file systems or lack network support.
- Networking support: The protocol will make no mention of TCP, UDP, SSH, HTTP, Etc..
- Configurability: Configuration is the root of all evil. Instead of trying to create something that serves every use case, focus on over 80% use cases and allow edge cases to be handled by application developers.
- Anonymity: Though pseudonymity is possible via the use of multiple identities, the concept of anonymity is not well suited to this protocol. Every message is signed by a known identity, though identities are easily created and can be backed by pseudonyms for privacy.
- OPEN TO DISCUSSION: Encryption. Like many internet protocols, encryption concerns are offloaded to the transport layer. Consider using an encrypted file system or encrypted compression application to store data. If you have ideas on how we could support encryption, please contact us. Please keep in mind that anonymity is not necessarily a project goal.
Sneakernet is a protocol used by ancient civilizations to exchange files between computers with limited internet connectivity.
Although the Pigeon protocol messages can be exchanged over sneakernet, the Pigeon protocol is not sneakernet. Sneakernet messages by themselves are not tamper-resistant, nor does it provide redundant backup using peers.
In contrast, a Pigeon protocol message is redundantly replicated beyond its intended recipient to neighboring peers, "a friend of a friend", through gossip. Unlike sneakernet, the Pigeon protocol also uses cryptography to guarantee that a message's content has not been altered by a third party.
Secure Scuttlebut is a decentralized, peer-to-peer protocol that also works offline.
The Pigeon protocol does not rely on networking in the core library. Although Secure Scuttlebutt is able to support Sneakernet-only operation, it is difficult in practice due to reliance on UDP, TCP, and in the case of pubs, DNS.
The Pigeon protocol uses a custom key/value serialization format instead of JSON. This has two benefits:
- Serialization and signing is more simple, because indentation and whitespace are less likely to cause verification problems.
- Unlike JSON, pigeon messages do not allow nesting, which promotes simplified message schemas.
The Pigeon protocol uses Crockford-flavored Base32 rather than URL safe Base64. This makes it easier to support old or low-powered systems and is easier to support FAT16 or embedded systems.
Unlike Secure Scuttlebutt, the Pigeon protocol was designed for portability and simplicity. It allows support for platforms that aren't limited to NodeJS.
Complicated features, such as network support, are ignored in favor of an easy-to-implement standard.
Pigeon borrows many of the ideas set forth by the Secure Scuttlebutt protocol. It is our opinion that SSB is one of the most innovative protocols created in recent years. Without the research and efforts of the Secure Scuttlebutt Consortium , this project would not be possible, so a big thanks goes out to all the people who make SSB possible.
We've also been inspired by the compactness and minimalism of SQLite, which should serve as a role model for all of us.
In many ways, this protocol can be considered an amalgam of the best ideas from both SQLite and Secure Scuttlebutt.
PIGEON PROTOCOL DEVELOPERS MAILING LIST: Send an email with "subscribe" in the subject line to pigeon_developers@vaporsoft.xyz.
If you want to be involved with the Pigeon protocol development, there are areas we need help in currently:
- Documentation, editors, and feedback.
- We need applications to be built using the protocol! We are happy to assist you along the way.
- We need a BNF grammar for Pigeon messages.
Your feedback really helps. Feel free to email us your opinions and
ideas at contact at vaporsoft.xyz. We will reply to all messages.
Check out the following links:
- Message Format Specification
- Bundle Format Specification
- Frequently Asked Questions
- Roadmap
- Ideas and Features
TODO:
- Explain what forking is and why it is not supported.
- Explain how verification happens (Soapdog)
- Question: "How do apps find what they need in the Pigeon feeds and/or accumulate state forward? Is this the responsibility of the Pigeon libraries, or do apps do it themselves?"