Substreams Foundational Stores

Foundational Stores Overview

A high-performance, multi-backend key-value storage system designed for Substreams ingestion and serving within the StreamingFast ecosystem. The foundational store provides a unified interface to persist and query time-series blockchain data with fork-awareness and efficient batch processing.

What are Foundational Stores?

Foundational stores are persistent storage systems that serve as the foundation for complex blockchain data applications. They provide a unified interface for ingesting, storing, and serving time-series blockchain data with fork-awareness capabilities.

  • Block-level versioning: Every piece of data is tagged with the block number where it originated

  • Fork-aware operations: Automatic handling of blockchain reorganizations with rollback capabilities

  • Real-time ingestion: Continuous processing of streaming Substreams output

  • High-performance serving: Optimized gRPC API for fast data retrieval

Architecture

Foundational stores consist of three main components working together:

Sink Component

The Sink handles streaming data ingestion from Substreams:

  • Continuous Processing: Consumes Substreams output in real-time

  • Batch Management: Groups data into efficient batches for storage

  • Cursor Management: Maintains resumption state for reliable streaming

  • Fork Detection: Processes BlockUndoSignal messages for reorganizations

Store Component

The Store provides a unified interface across multiple storage backends:

  • ForkAware Layer: In-memory cache with block-level versioning

  • Backend Abstraction: Support for Badger (embedded) and PostgreSQL

  • Flush Operations: Persists finalized data below Last Irreversible Block (LIB)

  • Eviction Handling: Removes reorganized data during fork events

Server Component

The Server exposes a high-performance gRPC API:

  • Get/GetAll Operations: Retrieve single or multiple keys efficiently

  • Block Validation: Ensures requested blocks have been processed

  • Response Codes: Clear status indicators (FOUND, NOT_FOUND, NOT_FOUND_BLOCK_NOT_REACHED)

Data Flow Architecture

Foundational Store Data Flow

User Consumption Patterns

Foundational Store Intrinsics

Foundational stores are consumed through the substreams-rs FoundationalStore type in your Substreams modules:

#[substreams::handlers::map]
fn my_module(
    block: Block,
    foundational_store: FoundationalStore,
) -> Result<Output, Error> {
    // Query foundational store for block-scoped data
    let response = foundational_store.get_all(&keys);
    // ...
}

Response Codes

Foundational stores return detailed status information for each query:

  • FOUND: Key exists and value was retrieved successfully

  • NOT_FOUND: Key does not exist at the requested block

  • NOT_FOUND_FINALIZE: Key was deleted after finality (LIB) - historical reference

  • NOT_FOUND_BLOCK_NOT_REACHED: Requested block has not been processed yet

See the complete ResponseCode definition for additional details.

See Also

Last updated

Was this helpful?