Protobuf

StreamingFast Substreams protobuf schemas

Protobuf overview

Substreams uses Google Protocol Buffers extensively. Protocol Buffers, also referred to as protobufs, are used as the API for data models specific to the different blockchains. Manifests contain references to the protobufs for your Substreams module.

Tip: Protobufs define the input and output for modules.

Learn more about the details of Google Protocol Buffers in the official documentation provided by Google.

Google Protocol Buffer Documentation

Learn more about Google Protocol Buffers in the official documentation provided by Google.

Google Protocol Buffer Tutorial

Explore examples and additional learning material for Google Protocol Buffers provided by Google.

Protobuf definition for Substreams

Define a protobuf model as proto:eth.erc721.v1.Transfers representing a list of ERC721 transfers.

Note: The Transfers protobuf in the Substreams Template example is located in the proto directory.

eth/erc721/v1/erc721.proto

syntax = "proto3";

package eth.erc721.v1;

message Transfers {
  repeated Transfer transfers = 1;
}

message Transfer {
  bytes from = 1;
  bytes to = 2;
  uint64 token_id = 3;
  bytes trx_hash = 4;
  uint64 ordinal = 5;
}

View the erc721.proto file in the official Substreams Template example repository.

Identifying data types

The ERC721 smart contract used in the Substreams Template example contains a Transfer event. You can use the event data through a custom protobuf.

The protobuf file serves as the interface between the module handlers and the data being provided by Substreams.

Tip: Protobufs are platform-independent and are defined and used for various blockchains.

• The ERC721 smart contracts used in the Substreams Template example are generic contracts used across many different Ethereum applications.

• The size and scope of the Substreams module dictates the number of and complexity of protobufs.

The Substreams Template example extracts Transfer events from the Bored Ape Yacht Club smart contract which is located on the Ethereum blockchain.

Several specific data types exist in the Ethereum smart contract ecosystem, some extending the ERC20 and ERC721 base modules. Complex protobufs are created and refined based on the various data types used across the different blockchains.

Tip: The use of fully qualified protobuf file paths reduces the risk of naming conflicts when other community members build their Substreams packages.

Generating protobufs

The substreams CLI is used to generate the associated Rust code for the protobuf.

Notice the protogen command and Substreams manifest passed into the substreams CLI.

substreams protogen ./substreams.yaml --exclude-paths="sf/ethereum,sf/substreams,google"

The pairing code is generated and saved into the src/pb/eth.erc721.v1.rsRust file.

The mod.rs file located in the src/pb directory of the Substreams Template example is responsible for exporting the freshly generated Rust code.

src/pb/mod.rs

#[path = "eth.erc721.v1.rs"]
#[allow(dead_code)]
pub mod erc721;

View the mod.rs file in the repository.

Protobuf and Rust optional fields

Protocol buffers define fields' type by using standard primitive data types, such as integers, booleans, and floats or a complex data type such as message, enum, oneof or map. View the full list of types in the Google Protocol Buffers documentation.

Any primitive data types in a message generate the corresponding Rust type,String for string, u64 for uint64, and assign the default value of the corresponding Rust type if the field is not present in a message, an empty string for String, 0 for integer types, false for bool.

Rust generates the corresponding message type wrapped by an Option enum type for fields referencing other complex messages. The None variant is used if the field is not present in the message.

The Option enum is used to represent the presence through Some(x) or absence through None of a value in Rust. Option allows developers to distinguish between a field containing a value versus a field without an assigned a value.

Note: The standard approach to represent nullable data in Rust is to wrap optional values in Option<T>.

The Rust match keyword is used to compare the value of an Option to a Some or None variant. Handle a type wrapped Option in Rust by using:

match person.Location {
    Some(location) => { /* Value is present, do something */ }
    None => { /* Value is absent, do something */ }
}

If you are only interested in finding the presence of a value, use the if let statement to handle the Some(x) arm of the match code.

if let Some(location) = person.location {
    // Value is present, do something
}

If a value is present, use the .unwrap() call on the Option to obtain the wrapped data. You'll need to account for these types of scenarios if you control the creation of the messages yourself or if the field is documented as always being present.

Note: You need to be absolutely sure the field is always defined, otherwise Substreams panics and never completes, getting stuck on a block indefinitely.

PROST! is a tool for generating Rust code from protobuf definitions. Learn more about prost in the project's official GitHub repository.

Learn more about Option in the official Rust documentation.