Ice Year

Everyone has a corresponding star shining in this universe.

Arthur C. Clarke "2001: A Space Odyssey"

Introduction#

  Due to some controversial actions during Musk's transition of Twitter to 𝕏, a large number of users chose to migrate from Twitter to platforms like Mastodon and Threads. The term Fediverse gradually entered the public's view, and even Threads claimed it would support the ActivityPub protocol in the future to achieve interconnectivity and join the Fediverse (which even sparked a series of discussions within the Fediverse community).

  On February 6, 2024, Bluesky announced the end of its public beta that had been running since 2021, allowing users to register without an invitation code. ¹ As a newcomer to the Fediverse created by a team of Twitter co-founders, Bluesky did not adopt the currently popular ActivityPub protocol in the Fediverse, but instead developed its own AT Protocol (Authenticated Transfer Protocol).

  Having followed the Fediverse for over two and a half years, I browsed through the relevant content of the AT Protocol out of curiosity. In the spirit of sharing, I wrote this article, which is neither a paper nor a report nor a blog post, to document my thoughts and understanding of the AT Protocol. Due to my limited level, this is for reference only, and I welcome corrections and discussions.

Initial Impressions and Comparisons#

  This analogy may not be very appropriate, but if we compare the development of blockchain, I believe that protocols represented by OStatus are similar to Adam Back's hashcash, ActivityPub is akin to Bitcoin, while the AT Protocol feels somewhat like Ethereum and EOS.

  If you have previously used instances based on the ActivityPub protocol like Mastodon, you might be aware of the interconnectivity flaws between instances: it is impossible to migrate to a new domain when the original domain has stopped working, and interactions with other instances can only display interactions from instances that your instance has federated with. If you have ever operated such an instance, you might also feel the financial burden of the Fediverse (which has successfully made IDC providers like Hetzner the hosting choice for over half of the Fediverse instances, humorously dubbed the capital of the Fediverse), as well as the exhaustion of having to manage both technology and administration. Of course, this does not hinder ActivityPub from being one of the very excellent protocols in the Fediverse, but these bugs always make one ponder whether there are ways to improve.

  In 2023, Nostr, representing blockchain-based Web3 social media, gradually "went viral" due to promotion led by Edward Snowden, and a series of "mysterious codes" became popular on Twitter. Nostr introduced blockchain into social platforms, adding a layer of value (or incentive layer) and achieving interconnectivity through P2P protocols like DHT. However, due to Nostr's design philosophy, it lacks an inherent mechanism for information filtering. The current lack of cross-chain mechanisms, along with the high costs and risks of operating chains or nodes, has led to users primarily concentrating on the Nostr main chain. Moreover, Nostr's "mysterious codes" also reveal the flaws of this social method: a lack of readability. While computers can easily interpret hashes, the vast majority of humans cannot memorize large amounts of randomly generated hash addresses. Although other blockchain-based Web3 social projects like Farcaster and Status I have experienced have attempted to map using human-friendly methods like ENS, the high costs of ENS have hindered user participation.

  Furthermore, both ActivityPub and implementations similar to Nostr and Farcaster adopt absolute timelines and follow pushes. While this avoids the "information cocoon" caused by a single platform's algorithm, it leads to another problem. Since users' attention is limited, ActivityPub prioritizes (or even only pushes) information from its own instance and federated instances, requiring users to filter time-sorted information when discovering information, thus shifting the filtering cost onto users. Meanwhile, Nostr also employs a strongly correlated timeline mechanism, but due to the centralization of the main chain, information overflow exceeds that of any large instance using the ActivityPub protocol, evolving from an "information cocoon" into an "information explosion." The design of the ActivityPub protocol means that users can only take a one-size-fits-all approach to blocking information or wait for instance administrators to delete or block information, without considering the potential to reduce push demands. The situation is even worse for Nostr, where the abundance of information makes it nearly impossible for users to block when exploring the timeline, and both client-based filtering and manual review become extraordinarily difficult.

Architecture of the AT Protocol#

  The clever design of the AT Protocol lies in the separation of data storage, pushing and discovery, filtering, and the interface that users can access, achieving openness, friendliness, disaster recovery, ease of migration, and scalability.

  As Bluesky elaborated in its paper: the design of the AT Protocol references today's internet, ² where websites should not all bear the responsibility of mutual discovery and federating all websites on the internet, nor should they all bear the work of filtering and screening the entire internet. It can be said that the World Wide Web is currently the most successful decentralized autonomous organization, and referencing it is undoubtedly very wise, significantly reducing the amount of thought and work required. The internet has hardware, software, and standards, and the design of the AT Protocol can also be said to have hardware, software, and standards. However, just as the internet has blurred the boundaries between software and hardware through virtualization and container technology, it is foreseeable that the software and hardware of the AT Protocol will not be limited to the original standard implementations in the future, but may see integration and expansion.

Hardware#

  Assuming that the hardware in the AT Protocol refers to parts of traditional hardware in the internet, then in the official standard implementation of the AT Protocol, Personal Data Server (PDS), Relay, and App View ³ can be considered the hardware of the AT Protocol.

Personal Data Server (PDS)#

  The Personal Data Server is, as the name suggests, a place to store user data. In previous ActivityPub implementations, this relied on various instances, but in the design of the AT Protocol, the Personal Data Server is equivalent to a server on the World Wide Web, only needing to implement the storage of relevant data, greatly reducing performance requirements and costs. It can even be set up on a home NAS or Raspberry Pi, which will make the decentralized network built on the AT Protocol more robust due to the addition of more nodes. Additionally, just as virtualization technology allows a single independent server to host multiple virtual servers, Personal Data Servers can also aggregate operations to provide services. For example, the official operation of Bluesky is currently the largest aggregation of Personal Data Servers, similar to IDC in cloud computing.

Relay#

  The Relay is similar to gateways and routers in the internet, serving as intermediaries connecting Personal Data Servers. The first component of the Relay is the Relay Server, responsible for fetching push data from known Personal Data Servers, verifying the data, discarding data that fails verification, and initially filtering high-frequency spam. The Relay aggregates this data through the Relay Server to create a Firehose ³, which is received by the App View ³. The benefit of this approach is that it significantly reduces the number of requests to Personal Data Servers and App Views compared to separately subscribing to multiple Personal Data Servers, and it can be verified using timestamp-based Merkle trees. Similar to how there are few operators responsible for operating lines and gateways in the internet, operating a Relay is the highest cost part among all nodes in the AT Protocol, and its expected quantity is relatively small.

App View#

  The App View is the component most perceptible to users, analogous to terminals accessing the internet or more vividly, APPs. The App View receives the Firehose from the Relay and processes the corresponding records according to the lexicon. In Bluesky, this corresponds to counting the likes on each post, organizing reply threads for posts, maintaining each user's collection of followers, and constructing a timeline containing posts from followed users, while also parsing media file addresses in the information and delivering the parsed files (with necessary processing, such as compression) to users. Just as nearly all connected applications fundamentally communicate based on TCP or UDP, the App View also communicates based on the AT Protocol. However, TCP and UDP only provide communication protocols, while the input, parsing, and output of the transmitted content rely on APPs. Based on the AT Protocol, many applications similar to the World Wide Web ecosystem can be developed. Currently, the largest application is the Bluesky microblogging platform, but it is foreseeable that in the future, many App Views similar to those in the existing Fediverse will be developed.

Software#

  Similarly, defining the software in the AT Protocol as parts of traditional software in the internet, in the official standard implementation of the AT Protocol, Feed Generators and Labelers ³ can be considered the hardware of the AT Protocol.

Feed Generator#

  Website directories and searches rely more on search engines and their algorithms. Correspondingly, the Feed Generator in the AT Protocol serves the role of indexing and selectively pushing information. Just as users can freely choose and even write their own search algorithms to build search engines, users can also freely choose and even build their own Feed Generators. Bluesky also provides a convenient startup toolkit on Github. ⁴ Moreover, similar to SearXNG, which allows users to select and aggregate information from search engines, users can also choose and aggregate Feed Generators in the App View to obtain and explore information, simply by clicking to add, just like downloading an APP from an app store. In other words, a large number of Feed Generators forming an algorithm market is itself the vision of the AT Protocol.

Labelers#

  For filtering and screening website content, more reliance is placed on ad blockers and fraud protection, or firewalls. The Labelers in the AT Protocol serve a similar function. Just as users can subscribe to and overlay different ad blocking rules, users can also use different labeling nodes to tag data, achieving more refined filtering and control, catering to the different needs of various users while effectively separating operations and management tasks.

Standards#

  Continuing with the previous definitions, the standards in the AT Protocol can be defined as parts of traditional standards in the internet. In the official standard implementation of the AT Protocol, DIDs (Decentralized Identifiers), Handles, Repositories, and Lexicons ³ can be considered the standards of the AT Protocol.

  Behind the standards is the data model of the AT Protocol. The AT Protocol applies various resource identification specifications, currently including DIDs (Decentralized IDs), CIDs (Content IDs), NSIDs (Namespaced Identifiers), and rkeys (Record Keys). ⁵

DID (Decentralized Identifiers)#

  Due to the brilliant design of the identity identification system in the AT Protocol, it deserves to be highlighted separately (not really)

  Well, it's just because the length is quite long, but the design is indeed very clever.

  DID superficially resembles an account system in the internet, but I believe it is more akin to the implementation of domain names in the internet, and it largely references the internet's certificate issuance mechanism. To be precise, DID is not an original invention of the AT Protocol; the true inventor is W3C, and there are currently hundreds of DID methods (which shows humanity's penchant for tinkering). ⁶ After discussions, the developers of the AT Protocol decided to only adopt the did:web method and the did:plc method to better avoid overtime meet the requirements.

  The cleverness of DID lies in that it is not a simple character combination, but can be resolved into a JSON array called a DID Document ³ through a mechanism similar to DNS records in the internet (essentially still a character combination). The DID Document in the AT Protocol follows the Merkle tree specification to ensure immutability and verifiability, so the Merkle tree of the DID Document will gradually grow over time and with operations.

  However, because DID often adopts a hash-based compression mechanism, it frequently appears as a combination of random characters, which is not user-friendly. To address this, the developers of the AT Protocol adopted the Handle method to solve this issue, establishing a mapping between domain names in the internet and DID through a human-friendly domain name mechanism, using TXT records or the /.well-known/ path via HTTPS (familiar to those who know about free SSL like Let's Encrypt). Therefore, in Bluesky, a username like @xxx.com (which indirectly resolves the blue verification issue, as long as example.com is the official website of a person or organization, it is equivalent to obtaining verification, shifting the verification responsibility to ICANN).

  The implementation of the did:web method is similar to the Handle mechanism, where the user's DID is the domain name, and the mechanism for resolving the DID to return a JSON array is through querying the specific TXT record corresponding to the domain name, with the return value being the DID Document (in this case, we can remove "similar"; it is indeed DNS resolution/doge), or by accessing the /.well-known/ path via HTTPS to obtain the DID Document. This approach combines Handle and DID, making it user-friendly. However, the downside is that the domain name serving as DID and Handle is immutable due to the immutability of DID, which can easily encounter events like domain changes in reality. Additionally, with each growth of the Merkle tree, manual updates to the TXT records or /.well-known/ path are required, which seems inefficient in large-scale applications and may lead to delays in updates due to varying network environments and potential hijacking.

  The did:plc method was created to address the above pain points. It is called did:plc because the developers originally intended it as a "placeholder," ³ but clearly, it is no longer the case, and there is no need to worry about the future of did:plc, as the AT Protocol provides backward compatibility in its corresponding implementation of DID to avoid potential issues with the addition of new DID methods rendering old methods ineffective. The did:plc method implements encryption and compression of the DID Document using the secp256k1 and NIST P-256 algorithms from did:key, possessing native encryption features.

  The request process for did:plc is as follows: send a request to the PLC Server ³, which checks whether it has already been registered. If registered, it returns a status code and the DID Document if necessary. If it is an update operation for the DID, the PLC Server will make a judgment through a verification mechanism. The PLC Server also determines which version to retain in case of a fork in the DID. The PLC Server offers various verification mechanisms to choose from, including using the rotation key pairs provided by ddi:plc for verification or employing traditional email-password mechanisms for verification using a hosted signing key pair by the PLC Server (for example, the official PLC Server of Bluesky), achieving a balance between convenience and security.

  The application of DID makes data migration in the AT Protocol exceptionally easy. Users only need to update their DID Merkle tree, adding records of changes to the new Personal Data Server, and back up and transfer media files from the original Personal Data Server to the new one. Even if the original Personal Data Server has completely disappeared, the original social network can still be retained. Additionally, through the combination of did:plc and the extended Oauth 2.1, it is even possible to log in to applications compatible with the corresponding Oauth standards using the AT Protocol. ⁷

Lexicon#

  If you have played in the Fediverse based on the ActivityPub protocol, you should have some impression of the differences between various projects: for example, Mastodon only allows likes on a post, while Misskey can have multiple reactions. Both Mastodon and Misskey use ActivityPub as the protocol for interconnectivity, but if Mastodon is to support Misskey's reaction features, the work required would involve modifying the entire Mastodon project. This design is not very elegant, while the design of the Lexicon in the AT Protocol elegantly solves this problem.

  The architecture of Lexicon references JSON and OpenAPI. Personally, I feel that NSID serves a role similar to Android package names, while rkey is somewhat akin to individual Activities. The namespace mechanism of NSID makes Lexicon more extensible, allowing projects developed based on the AT Protocol to focus solely on designing the Lexicon and corresponding elements in the App View, making interconnectivity and interoperability in the Fediverse more convenient than ever.

Repository#

  If you are familiar with IPFS, seeing CID should feel familiar. You are correct; the Repository in the AT Protocol follows the IPLD specification and implements version control through the Merkle tree mechanism of Git. The advantages of content addressing, avoiding duplicate data, and the signature verification and version control of Git and Merkle trees will not be reiterated here.

Conclusion#

  It can be said that the development of the AT Protocol stands on the shoulders of countless giants, from the mapping of ubiquitous internet elements to claiming it is not blockchain, ⁸ while its body honestly applies a large number of mature applications of Merkle trees, to the unprecedented customizability of pushes that can be seen as a shadow of RSS, and the application of Git in the data model. The AT Protocol has learned from the pitfalls of Twitter, ActivityPub, Nostr, and others, continuously evolving and surpassing, achieving the most promising form of Web3 social protocols currently.

  Currently, the application of the AT Protocol still exhibits a relatively centralized phenomenon, with Bluesky remaining the only choice in many cases. However, if it continues to develop according to Bluesky's declarations, the AT Protocol will become increasingly open, and it is foreseeable that its excellent features will attract many developers, promising a bright future for the ecosystem. There is a strange feeling that 𝕏 resembles the Netscape company of yesteryear, while Bluesky, emerging from it, hopes to become an open organization like Mozilla, promoting technology towards a more open, transparent, fair, and interconnected direction.

Original article link: https://blog.iceyear.eu.org/atprotocol-study