LogoLogo
Package RegistryThe Graph
  • Introduction
  • Getting Started
  • Tutorials
    • Develop Your First Substreams
      • on EVM
      • on Solana
        • Transactions & Instructions
        • Account Changes
      • on Cosmos
        • Injective
        • MANTRA
      • on Starknet
      • on Stellar
    • Publishing a Substreams Package
  • How-To Guides
    • Developing Substreams
      • on EVM
        • Exploring Ethereum
          • Mapping Blocks
          • Filter Transactions
          • Retrieve Events of a Smart Contract
      • on Solana
        • Explore Solana
          • Filter Instructions
          • Filter Transactions
        • SPL Token Tracker
        • NFT Trades
        • DEX Trades
      • on Cosmos
        • Injective
          • Simple Substreams Example
          • Foundational Modules
          • Dojo DEX USDT Volume Subgraph Example
    • Using a Substreams Sink
      • Substreams:SQL
      • Substreams:Subgraph
        • Triggers
        • Graph Out
      • Substreams:Stream
        • JavaScript
        • Go
      • Substreams:PubSub
      • Community Sinks
        • MongoDB
        • Files
        • Key-Value Store
        • Prometheus
    • EVM Extensions
      • Making eth_calls
    • Getting Started Using Rust and Protobuf
      • Rust
        • Option struct
        • Result struct
      • Protobuf Schemas
    • From Yellowstone to Substreams
  • Reference Material
    • Chains and endpoints
      • Ethereum Data Model
    • Never Miss Data
    • Development Container Reference
    • Substreams CLI
      • Install the CLI
      • Authentication
      • Substreams CLI reference
    • Substreams Components
      • Packages
      • Modules
        • Module types
        • Inputs
        • Output
        • Module handlers
        • Module handler creation
        • Indexes
        • Keys in stores
        • Dynamic data sources
        • Aggregation Windows
        • Parameterized Modules
      • Manifests Reference
    • Substreams Architecture
    • Graph-Node
      • Local Development
      • Publish to The Graph Network
    • Indexer Reference
      • Test Substreams Locally
    • Logging, Debugging & Testing
    • Change log
    • FAQ
  • Decentralized Indexing
    • What is The Graph?
Powered by GitBook
On this page

Was this helpful?

Edit on GitHub
  1. Reference Material
  2. Substreams Components
  3. Modules

Aggregation Windows

Building and freeing up aggregation windows

Store module key-value storage can hold at most 1 GiB. It is usually enough if used correctly, but it is still a good idea (and sometimes even necessary) to free up unused keys. It is especially true for cases where you work with aggregation windows.

Consider this store module that aggregates hourly trade counter for each token:

#[substreams::handlers::store]
pub fn store_total_tx_counts(clock: Clock, events: Events, output: StoreAddBigInt) {
    let timestamp_seconds = clock.timestamp.unwrap().seconds;
    let hour_id = timestamp_seconds / 3600;
    let prev_hour_id = hour_id - 1;

    output.delete_prefix(0, &format!("TokenHourData:{prev_hour_id}:"));

    for event in events.pool_events {
        output.add_many(
            event.log_ordinal,
            &vec![
                format!("TokenHourData:{}:{}", hour_id, event.token0),
                format!("TokenHourData:{}:{}", hour_id, event.token1),
            ],
            &BigInt::from(1 as i32),
        );
    }
}

Let's break it down.

First, we use Clock input source to get the current and previous hour id for the block.

let hour_id = timestamp_seconds / 3600;
let prev_hour_id = hour_id - 1;

Then we build hourly keys for our counters and use add_many method to increment them. These counters will be consumed downstream by other modules.

output.add_many(
    event.log_ordinal,
    &vec![
        format!("TokenHourData:{}:{}", hour_id, event.token0),
        format!("TokenHourData:{}:{}", hour_id, event.token1),
    ],
    &BigInt::from(1 as i32),
);

Here's the trick. Since we don't need these counters outside of the hourly window, we can safely delete these key-value pairs for the previous hourly window and free up the memory.

This is done using delete_prefix method:

output.delete_prefix(0, &format!("TokenHourData:{prev_hour_id}:"));

Links

PreviousDynamic data sourcesNextParameterized Modules

Last updated 1 year ago

Was this helpful?

Uniswap-v3 Subgraph and Substreams