ABI Management

Solana ABI management process

This document proposes the Solana ABI management process. The ABI management process is an engineering practice and a supporting technical framework to avoid introducing unintended incompatible ABI changes.

Problem

The Solana ABI (binary interface to the cluster) is currently only defined implicitly by the implementation and requires a very careful eye to notice breaking changes. This makes it extremely difficult to upgrade the software on an existing cluster without rebooting the ledger.

Requirements and objectives

  • Unintended ABI changes can be detected as CI failures mechanically.

  • Newer implementation must be able to process the oldest data (since genesis)

    once we go mainnet.

  • The objective of this proposal is to protect the ABI while sustaining rather

    rapid development by opting for a mechanical process rather than a very long

    human-driven auditing process.

  • Once signed cryptographically, data blob must be identical, so no

    in-place data format update is possible regardless of inbound and outbound of

    the online system. Also, considering the sheer volume of transactions we're

    aiming to handle, retrospective in-place update is undesirable at best.

Solution

Instead of natural human's eye due-diligence, which should be assumed to fail regularly, we need a systematic assurance of not breaking the cluster when changing the source code.

For that purpose, we introduce a mechanism of marking every ABI-related things in source code (structs, enums) with the new #[frozen_abi] attribute. This takes hard-coded digest value derived from types of its fields via ser::Serialize. And the attribute automatically generates a unit test to try to detect any unsanctioned changes to the marked ABI-related things.

However, the detection cannot be complete; no matter how hard we statically analyze the source code, it's still possible to break ABI. For example, this includes not-derived hand-written ser::Serialize, underlying library's implementation changes (for example bincode), CPU architecture differences. The detection of these possible ABI incompatibilities is out-of-scope for this ABI management.

Definitions

ABI item/type: various types to be used for serialization, which collectively comprises the whole ABI for any system components. For example, those types include structs and enums.

ABI item digest: Some fixed hash derived from type information of ABI item's fields.

Example

+#[frozen_abi(digest="1c6a53e9")]
#[derive(Serialize, Default, Deserialize, Debug, PartialEq, Eq, Clone)]
pub struct Vote {
/// A stack of votes starting with the oldest vote
pub slots: Vec<Slot>,
/// signature of the bank's state at the last slot
pub hash: Hash,
}

Developer's workflow

To know the digest for new ABI items, developers can add frozen_abi with a random digest value and run the unit tests and replace it with the correct digest from the assertion test error message.

In general, once we add frozen_abi and its change is published in the stable release channel, its digest should never change. If such a change is needed, we should opt for defining a new struct like FooV1. And special release flow like hard forks should be approached.

Implementation remarks

We use some degree of macro machinery to automatically generate unit tests and calculate a digest from ABI items. This is doable by clever use of serde::Serialize ([1]) and any::typename ([2]). For a precedent for similar implementation, ink from the Parity Technologies [3] could be informational.

References