The Graph Node: The Indexing Powerhouse

At the heart of The Graph’s architecture lies the Graph Node. This key component is responsible for indexing subgraphs and making the resultant data accessible via a GraphQL API. It’s the core of the indexer stack, and its effective operation is crucial for running a successful indexer. Graph Node operates with versatility, capable of running on both bare metal and cloud environments, reflecting the adaptability required in the dynamic landscape of blockchain technology.

Storing Data in PostgreSQL Databases

Integral to the operation of the Graph Node is the PostgreSQL database, which acts as the main store. This database houses not just the subgraph data but also metadata about subgraphs and essential network data like block and eth_call caches. The organization and management of this database are vital for the smooth operation of the Graph Node, ensuring data integrity and accessibility.

Network Clients and Firehoses

For indexing blockchain networks, Graph Node connects to network clients through an EVM-compatible JSON-RPC API. This setup can vary from connecting to a single client to a more complex arrangement involving load balancing across multiple clients. In addition, The Graph has developed the Network Firehoses - a gRPC service that provides an ordered, fork-aware stream of blocks. While not currently a requirement for indexers, the Firehose represents a significant advancement in supporting performant indexing at scale.

Metadata on IPFS Nodes and Monitoring with Prometheus

Graph Node’s interaction with the IPFS network is crucial for storing subgraph deployment metadata. An IPFS node hosted at the network level simplifies this process for indexers. Additionally, the optional integration with a Prometheus metrics server for monitoring and reporting adds another layer of sophistication, enabling indexers to track and optimize the performance of the Graph Node.

Scalability and Advanced Configurations

The flexible setup of the Graph Node, from installation options to scaling capabilities, highlights The Graph’s commitment to accommodating various operational needs. The system can be scaled horizontally with multiple Graph Nodes and databases, catering to the increasing demands of the network. Advanced users can leverage the Graph Node’s configuration options, managed through a TOML file or environment variables, to optimize data processing and workload distribution.

Data Indexing and Processing: Firehose and Substreams Redefined

Firehose: Reinventing Blockchain Data Extraction

Firehose, conceptualized and developed by StreamingFast, marks a revolution in extracting data from blockchain nodes. This innovative tool breaks down every transaction within a blockchain block into its smallest elements, saving them as simple, flat files. These flat files are not just a storage format; they embody a paradigm shift in data indexing. They facilitate parallel processing, which drastically speeds up indexing operations. This technology delivers rich, fork-aware data from instrumented blockchain nodes directly to consumers. In practical terms, Firehose has shown its prowess by offering capture and processing speeds that were once thought unattainable, thus setting new standards for data extraction in The Graph’s ecosystem.

Substreams: Pioneering Efficient Data Processing

Substreams, an extension of the capabilities provided by Firehose, are designed for high-performance data processing in a parallel, streaming-first fashion. These Rust-written modules allow developers to compose, sort, store, and transform blockchain data for various uses. The ingenuity of Substreams lies in their ability to utilize Firehose’s flat files for indexing data at exceptionally fast speeds. This approach ensures that Substreams are not only highly efficient in processing data but also in distributing it as soon as it becomes available, rather than relying on continuous requests.

The Combination of Firehose and Substreams

The integration of Firehose and Substreams within The Graph’s ecosystem provides a powerful combination for data processing. Firehose ensures the swift delivery of blockchain data in an optimized format, while Substreams further refine and process this data. This synergistic relationship results in unparalleled efficiency in handling large volumes of blockchain data, significantly elevating the capabilities of The Graph.

Subgraphs: The Industry Standard in Data Indexing

Subgraphs have become the industry standard in indexing blockchain data since their introduction by The Graph in 2018. They are essentially open APIs that extract, process, and store data from blockchains, thereby making it easily queryable via a GraphQL interface. With over 85,000 subgraphs supported across more than 40 chains, subgraphs have become indispensable for web3 developers. They allow for the rapid deployment of a Postgres database filled with indexed data, ready to be queried using a GraphQL layer. Subgraphs enable developers to display a wide array of blockchain data in their DApps, from DeFi transactions to NFT provenance, in an organized and efficient manner.

In the ever-evolving landscape of blockchain technology, subgraphs have emerged as a pivotal concept, transforming the way we interact with and utilize blockchain data. These open APIs act as intermediaries, seamlessly bridging the gap between the decentralized world of blockchains and the familiar realm of structured data. By extracting, processing, and organizing blockchain data into a queryable format, subgraphs empower developers to build innovative and data-driven applications.

Subgraphs offer a plethora of benefits, making them an attractive choice for developers and users alike. Their decentralized nature ensures resistance to censorship and downtime, fostering a secure and reliable data ecosystem. Additionally, subgraphs are inherently scalable, capable of handling massive amounts of data without compromising performance. Cost-effectiveness is another key advantage, with subgraphs often being more affordable than traditional data APIs.

The Subgraph Architecture

Subgraphs are composed of three essential components that work in harmony to deliver their transformative capabilities:

1. Manifest: The manifest serves as the blueprint of the subgraph, outlining its data sources, schema, and AssemblyScript code. It defines the boundaries of the data that the subgraph will index, ensuring that only relevant information is captured.

2. Schema: The schema defines the structure of the data, akin to the blueprint of a building. It outlines the entities, fields, and relationships between entities, providing a clear and organized way to represent the data.

3. AssemblyScript Code: This executable code acts as the workhorse of the subgraph, translating the raw data from blockchains into a format that the GraphQL engine can comprehend. It also handles the indexing and storage of data, ensuring its accessibility and reliability.

Subgraph Development Process

Creating a subgraph entails a series of steps, each carefully crafted to ensure the subgraph’s functionality and effectiveness:

1. Conceptualizing and Designing: The journey begins with a clear idea of the data to be indexed and the applications it will power. This involves defining the entities, fields, and relationships between them, ensuring that the subgraph’s structure aligns with its intended purpose.

2. Manifest and Schema Development: The manifest and schema are meticulously crafted, providing the foundation for the subgraph’s data architecture. The manifest specifies the datasources, while the schema outlines the data structure, ensuring data integrity and consistency.

3. AssemblyScript Code Implementation: The AssemblyScript code is written, translating the raw blockchain data into a format that GraphQL can understand. It handles indexing, storage, and data retrieval, enabling efficient access to the indexed data.

Deploying and Publishing Subgraphs

Once the subgraph is developed, it undergoes a deployment process that introduces it to the world:

1. Subgraph Studio Integration: The Subgraph Studio serves as a centralized platform for managing subgraphs. It facilitates the deployment process, allowing developers to publish their subgraphs to the decentralized network.

2. Indexing and Curating: Indexers, responsible for fetching and storing blockchain data, are essential for making subgraphs accessible to developers. Curating, typically done through GRT tokens, incentivizes indexers to prioritize subgraphs with high demand.

3. Querying and Utilizing: Developers can now query the deployed subgraph using GraphQL queries, retrieving specific data tailored to their application’s needs. This seamless integration empowers developers to harness blockchain data for innovation.

The Future: The Graph’s New Era

As The Graph embarks on its New Era (We will explore it in Lesson 5), the continued evolution of these core technologies – subgraphs, Firehose, and Substreams – is eagerly anticipated. These components are set to expand and evolve, playing vital roles in introducing new data services and ensuring faster, more modular flows of data. The Verifiable Firehose, for example, is poised to be a groundbreaking solution for accessing historical Ethereum data, addressing the challenges of evolving blockchain standards.

Differentiating Subgraphs from Substreams

It’s crucial to distinguish between subgraphs and Substreams as they serve different purposes. Subgraphs are ideal for standard data retrieval and management, offering ease of setup and use with a GraphQL query layer. Conversely, Substreams are tailored for more complex analytics and big data needs, offering parallelized data processing and greater flexibility in data handling and storage. Substreams allow developers to transform data from basic file formats into more usable forms, catering to sophisticated data processing requirements.

Organizing Off-chain Data: The Graph’s Approach and Impact on Web3 Data Pipelines

The Graph, traditionally known for its prowess in organizing on-chain data, is now expanding its horizons by venturing into the realm of off-chain data. This approach aligns with The Graph’s long-term mission to provide easy access to the world’s public knowledge and information.

Web3 and The Trade-offs in Data Management

In the Web3 architecture, while users can directly interact with the blockchain through middleware services, there’s a trade-off, especially when it comes to cost. On-chain transaction costs, often referred to as gas fees, can be prohibitively high for complex computations or extensive data storage. This limitation has historically curtailed the complexity of applications or led developers to create proprietary off-chain APIs, departing from open-source models.

The Graph’s Unique Solution for Off-chain Data

The Graph presents a unique solution to this challenge by enabling the organization and serving of off-chain data through its decentralized network. This method involves a workflow where traditionally off-chain data is posted to IPFS (InterPlanetary File System), and the IPFS hashes are then recorded on-chain. Subsequently, this data can be indexed by subgraphs and made available for querying. This approach offers a scalable and economical way to publish and serve complex, dynamic data without the overhead of building and maintaining proprietary APIs.

The Workflow: A Three-Step Process

The workflow consists of three key components:

1. Cron Jobs for Data Computation and Posting: An off-chain cron job performs complex computations and posts the results to a permaweb source, like IPFS, indexable by The Graph. This job also generates an on-chain transaction to post the IPFS file hash and any relevant metadata.

2. Subgraph Publication for Indexing: The next step involves publishing a subgraph that indexes these IPFS files based on the file hashes posted on the chain. Once the subgraph is published, it can be picked up and served by Indexers in The Graph’s network, allowing third-party developers and users to query the data.

3. Robust and Reliable Data Access: By leveraging The Graph’s distributed network of Indexers, data access remains robust and reliable without additional effort from the data publisher. This decentralized structure significantly enhances the data’s availability and integrity.

Practical Implementation and Cost-Efficiency

A practical example in The Graph ecosystem is the oracle developed by Edge & Node for publishing network cost and quality of service metrics. This oracle posts aggregated data to IPFS every five minutes and logs the IPFS file hash on the Gnosis chain. This data is then indexed in a subgraph, which can be consumed by protocol stakeholders. The cost associated with this workflow is surprisingly low, making it an attractive option for data publishers.

New Possibilities with Dynamic Data

This method of using The Graph for dynamic data opens up exciting new possibilities for permaweb sites, including lean back-ends for blogs, algorithmic content curation, and real-time monitoring systems. It represents a significant shift in how data is published, indexed, and accessed, promoting a more open and collaborative web3 ecosystem.

Organizing Off-chain Data: The Graph’s Innovative Approach

The Graph’s expansion into managing off-chain data opens new avenues in the Web3 ecosystem, creating a bridge between the decentralized and traditional data realms. This initiative reflects The Graph’s mission to make a wider spectrum of information accessible in a decentralized manner, addressing the inherent limitations of on-chain data storage and computation.

Balancing On-chain and Off-chain Data Needs

The Graph recognizes the cost-related trade-offs of on-chain data storage and computation in Web3 architectures. While direct interaction with the blockchain is straightforward, complex computations and large-scale data storage can become prohibitively expensive. To circumvent these limitations, The Graph introduces a method that combines off-chain data storage with on-chain data referencing, thereby retaining the decentralized ethos while enhancing functionality.

The Graph’s Methodology for Off-chain Data

This approach involves a three-step workflow:

1. Off-chain Computation and IPFS Posting: Complex computations are performed off-chain, and the resulting data is posted to IPFS, a decentralized storage solution. This step ensures that the data, while off-chain, is stored in a verifiable and decentralized manner.

2. On-chain Linking via Transactions: Alongside storing data on IPFS, a corresponding on-chain transaction is made to log the IPFS hash and other relevant metadata. This method anchors the off-chain data to the blockchain, providing a layer of trust and traceability.

3. Subgraph Indexing for Accessibility: The final step involves indexing the IPFS-stored data using subgraphs. This process makes the off-chain data easily queryable and accessible through The Graph’s decentralized network.
   Practical Implementation: Edge & Node’s Oracle

A practical application of this methodology within The Graph’s ecosystem is the oracle developed by Edge & Node. This oracle publishes network cost and quality of service metrics every five minutes in the following manner:

• Aggregated data is posted to IPFS.

• The corresponding IPFS file hash is then recorded on the Gnosis chain via a DataEdge contract.

• These IPFS files are indexed in a subgraph, making the data available for stakeholders in a decentralized manner.
  This implementation demonstrates the low-cost, scalable, and efficient approach to publishing and serving complex data without the need for proprietary APIs. It exemplifies how The Graph’s method can be leveraged to create dynamic data sources for a variety of applications.

Cost Efficiency and Scalability

The costs associated with this workflow are surprisingly low, making it an attractive solution for data publishers. For instance, the oracle implementation by Edge & Node incurs minimal expenses for on-chain transactions and IPFS node pinning, with the serving costs borne by the data consumer. This model effectively reduces the operational overhead for data publishers while ensuring robust and reliable data access.

Enabling New Web3 Use Cases

This method unlocks new possibilities for permaweb applications, such as dynamic back-ends for blogs, algorithmic content curation, and real-time monitoring systems. It allows for the partitioning of data publishers from app/front-end operators, encouraging specialization and division of labor in an open-source community. This approach holds promise for decentralized social applications and protocols, offering a new path forward for decentralized data publication and consumption.

Query Language and APIs in The Graph: Embracing GraphQL

The integration of GraphQL as the query language of choice. This decision significantly shapes the way data is accessed and interacted with through The Graph’s APIs, providing a streamlined and efficient method for querying blockchain data.

Understanding GraphQL’s Role in The Graph

GraphQL stands at the forefront of modern API design, offering a flexible and efficient approach to data retrieval. In the blockchain context, where data structures are complex and ever-evolving, GraphQL’s ability to fetch exactly what is needed becomes invaluable.

1. Tailored Data Queries: At the core of GraphQL’s appeal is its ability to allow clients to precisely define the structure of the data they require. This capability is a significant departure from traditional fixed-structure responses, enabling a more focused and efficient data interaction.

2. Enhancing Real-time Interactions: Beyond just querying, GraphQL in The Graph supports real-time data subscriptions. This feature is vital for blockchain applications where timely updates and responsiveness are key to user experience.

3. Decentralized and Trustless Data Access: The Graph’s use of GraphQL extends its philosophy of decentralization into the realm of data access. By interfacing with a network of decentralized nodes, GraphQL queries ensure that data remains open, transparent, and censorship-resistant.
   The Convergence of APIs and GraphQL

In The Graph’s ecosystem, the melding of APIs with GraphQL creates a harmonious and powerful system for data retrieval:

1. Schema Definition and Data Mapping: Developers define a GraphQL schema within their subgraph, outlining the structure of the queryable data. The schema is then intricately mapped to blockchain events, translating on-chain activities into structured data.

2. Executing Queries through Indexers: When a GraphQL query is submitted to a subgraph API, it’s processed by The Graph’s decentralized network of Indexers. This process exemplifies how queries are executed in a distributed manner, upholding the principles of blockchain technology.

3. Handling Complex Data Relationships: With complex data relationships being commonplace in blockchain, GraphQL’s ability to handle intricate queries, including various forms of data filtering and sorting, is especially beneficial.

The Benefits Unfold for Developers and End-users

The integration of GraphQL in The Graph brings forth numerous benefits:

• Flexibility for Developers: Developers can harness the full power of GraphQL to craft queries that align precisely with their application’s requirements.

• Efficient Data Fetching: Reduced data fetching times enhance application performance, directly translating to a better end-user experience.

• Seamless Data Integration: The standardized nature of GraphQL queries simplifies the process of integrating diverse blockchain data sources into applications.