TheGraph: Fixing the Web3 data querying

Why we need TheGraph and how to use it

Previously we looked at the big picture of Solidity and the create-eth-app which already mentioned TheGraph before. This time we will take a closer look at TheGraph which essentially became part of the standard stack for developing Dapps in the last year.

But let's first see how we would do things the traditional way...

Without TheGraph...

So let's go with a simple example for illustration purposes. We all like games, so imagine a simple game with users placing bets:

pragma solidity 0.7.1;

contract Game {
    uint256 totalGamesPlayerWon = 0;
    uint256 totalGamesPlayerLost = 0;
    event BetPlaced(address player, uint256 value, bool hasWon);

    function placeBet() external payable {
        bool hasWon = evaluateBetForPlayer(msg.sender);

        if (hasWon) {
            (bool success, ) = msg.sender.call{ value: msg.value * 2 }('');
            require(success, "Transfer failed");
            totalGamesPlayerWon++;
        } else {
            totalGamesPlayerLost++;
        }
        
        emit BetPlaced(msg.sender, msg.value, hasWon);
    }
}
Now let's say in our Dapp, we want to display total the total games lost/won and also update it whenever someone plays again. The approach would be:


  1. Fetch totalGamesPlayerWon.
  2. Fetch totalGamesPlayerLost.
  3. Subscribe to BetPlaced events.


We can listen to the event in Web3 as shown on the right, but it requires handling quite a few cases.

GameContract.events.BetPlaced({
    fromBlock: 0
}, function(error, event) { console.log(event); })
.on('data', function(event) {
    // event fired
})
.on('changed', function(event) {
    // event was removed again
})
.on('error', function(error, receipt) {
    // tx rejected
});

Now this is still somewhat fine for our simple example. But let's say we want to now display the amounts of bets lost/won only for the current player. Well we're out of luck, you better deploy a new contract that stores those values and fetch them. And now imagine a much more complicated smart contract and Dapp, things can get messy quickly.

You can see how this is not optimal:

  • Doesn't work for already deployed contracts.
  • Extra gas costs for storing those values.
  • Requires another call to fetch the data for an Ethereum node.
not good enough

Now let's look at a better solution.

Let me introduce you to GraphQL

First let's talk about GraphQL, originally designed and implemented by Facebook. You might be familiar with the traditional Rest API model. Now imagine instead you could write a query for exactly the data that you wanted:

GraphQL
GraphQL query

The two images pretty much capture the essence of GraphQL. With the query on the right we can define exactly what data we want, so there we get everything in one request and nothing more than exactly what we need. A GraphQL server handles the fetching of all data required, so it is incredibly easy for the frontend consumer side to use. This is a nice explanation of how exactly the server handles a query if you're interested.

Now with that knowledge, let's finally jump into blockchain space and TheGraph.

What is TheGraph?

A blockchain is a decentralized database, but in contrast to what's usually the case, we don't have a query language for this database. Solutions for retrieving data are painful or completely impossible. TheGraph is a decentralized protocol for indexing and querying blockchain data. And you might have guessed it, it's using GraphQL as query language.

Examples are always the best to understand something, so let's use TheGraph for our GameContract example.

How to create a Subgraph

The definition for how to index data is called subgraph. It requires three components:

  1. Manifest (subgraph.yaml)
  2. Schema (schema.graphql)
  3. Mapping (mapping.ts)

Manifest (subgraph.yaml)

The manifest is our configuration file and defines:

  • which smart contracts to index (address, network, ABI...)
  • which events to listen to
  • other things to listen to like function calls or blocks
  • the mapping functions being called (see mapping.ts below)

You can define multiple contracts and handlers here. A typical setup would have a subgraph folder inside the Truffle/Buidler project with its own repository. Then you can easily reference the ABI.

For convenience reasons you also might want to use a template tool like mustache. Then you create a subgraph.template.yaml and insert the addresses based on the latest deployments. For a more advanced example setup, see for example the Aave subgraph repo.

And the full documentation can be seen here: https://thegraph.com/docs/define-a-subgraph#the-subgraph-manifest.

specVersion: 0.0.1
description: Placing Bets on Ethereum
repository: - Github link -
schema:
  file: ./schema.graphql
dataSources:
  - kind: ethereum/contract
    name: GameContract
    network: mainnet
    source:
      address: '0x2E6454...cf77eC'
      abi: GameContract
      startBlock: 6175244
    mapping:
      kind: ethereum/events
      apiVersion: 0.0.1
      language: wasm/assemblyscript
      entities:
        - GameContract
      abis:
        - name: GameContract
          file: ../build/contracts/GameContract.json
      eventHandlers:
        - event: PlacedBet(address,uint256,bool)
          handler: handleNewBet
      file: ./src/mapping.ts

Schema (schema.graphql)

The schema is the GraphQL data definition. It will allow you to define which entities exist and their types. Supported types from TheGraph are

  • Bytes
  • ID
  • String
  • Boolean
  • Int
  • BigInt
  • BigDecimal

You can also use entities as type to define relationships. In our example we define a 1-to-many relationship from player to bets. The ! means the value can't be empty. The full documentation can be seen here: https://thegraph.com/docs/define-a-subgraph#the-graphql-schema.

type Bet @entity {
  id: ID!
  player: Player!
  playerHasWon: Boolean!
  time: Int!
}

type Player @entity {
  id: ID!
  totalPlayedCount: Int
  hasWonCount: Int
  hasLostCount: Int
  bets: [Bet]!
}

Mapping (mapping.ts)

The mapping file in TheGraph defines our functions that transform incoming events into entities. It is written in AssemblyScript, a subset of Typescript. This means it can be compiled into WASM (WebAssembly) for more efficient and portable execution of the mapping.

You will need to define each function named in the subgraph.yaml file, so in our case we need only one: handleNewBet. We first try to load the Player entity from the sender address as id. If it doesn't exist, we create a new entity and fill it with starting values.

Then we create a new Bet entity. The id for this will be event.transaction.hash.toHex() + "-" + event.logIndex.toString() ensuring always a unique value. Using only the hash isn't enough as someone might be calling the placeBet function several times in one transaction via a smart contract.

Lastly we can update the Player entity will all the data. Arrays cannot be pushed to directly, but need to be updated as shown here. We use the id to reference the bet. And .save() is required at the end to store an entity.

The full documentation can be seen here: https://thegraph.com/docs/define-a-subgraph#writing-mappings. You can also add logging output to the mapping file, see here.

import { Bet, Player } from '../generated/schema';
import { PlacedBet }
    from '../generated/GameContract/GameContract';

export function handleNewBet(event: PlacedBet): void {
  let player = Player.load(
    event.transaction.from.toHex()
  );

  if (player == null) {
    // create if doesn't exist yet
    player = new Player(event.transaction.from.toHex());
    player.bets = new Array<string>(0);
    player.totalPlayedCount = 0;
    player.hasWonCount = 0;
    player.hasLostCount = 0;
  }

  let bet = new Bet(
    event.transaction.hash.toHex()
        + '-'
        + event.logIndex.toString()
  );
  bet.player = player.id;
  bet.playerHasWon = event.params.hasWon;
  bet.time = event.block.timestamp;
  bet.save();

  player.totalPlayedCount++;
  if (event.params.hasWon) {
    player.hasWonCount++;
  } else {
    player.hasLostCount++;
  }

  // update array like this
  let bets = player.bets;
  bets.push(bet.id);
  player.bets = bets;

  player.save();
}

Using it in the Frontend

Using something like Apollo Boost, you can easily integrate TheGraph in your React Dapp (or Apollo-Vue). Especially when using React hooks and Apollo, fetching data is as simple as writing a single GraphQl query in your component. A typical setup might look like this:


// See all subgraphs: https://thegraph.com/explorer/
const client = new ApolloClient({
  uri: "{{ subgraphUrl }}",
});

ReactDOM.render(
  <ApolloProvider client={client}>
    <App />
  </ApolloProvider>,
  document.getElementById("root"),
);
const { loading, error, data } = useQuery(myGraphQlQuery);

React.useEffect(() => {
    if (!loading && !error && data) {
        console.log({ data });
    }
}, [loading, error, data]);

And now we can write for example a query like this. This will fetch us

  • how many times current user has won
  • how many times current user has lost
  • a list of timestamps with all his previous bets

    All in one single request to the GraphQL server.
const myGraphQlQuery = gql`
    players(where: { id: $currentUser }) {
      totalPlayedCount
      hasWonCount
      hasLostCount
      bets {
        time
      }
    }
`;
Magic

But we're missing one last piece of the puzzle and that's the server. You can either run it yourself or use the hosted service.

TheGraph server

Graph Explorer: The hosted service

The easiest way is to use the hosted service. Follow the instructions here to deploy a subgraph. For many projects you can actually find exisiting subgraphs in the explorer at https://thegraph.com/explorer/.

TheGraph Explorer

Running your own node

Alternatively you can run your own node: https://github.com/graphprotocol/graph-node#quick-start. One reason to do this might be using a network that's not supported by the hosted service. Currently supported are mainnet, Kovan, Rinkeby, Ropsten, Goerli, PoA-Core, xDAI and Sokol.

The decentralized future

GraphQL supports streams as well for newly incoming events. This is not yet fully supported by TheGraph, but it will be released soon.

One missing aspect though is still decentralization. TheGraph has future plans for eventually becoming a fully decentralized protocol. Those are two great articles explaining the plan in more detail:


Two key aspects are:
  1. Users will be paying the indexers for queries.
  2. Indexers will be staking Graph Tokens (GRT).

TheGraph architecture
g

Markus Waas

Solidity Developer

More great blog posts from Markus Waas

  • xDai

    How to use xDai in your Dapp

    Deploying and onboarding users to xDai to avoid the high gas costs

    Gas costs are exploding again, ETH2.0 is still too far away and people are now looking at layer 2 solutions. Here's a good overview of existing layer 2 projects: https://github.com/Awesome-Layer-2/awesome-layer-2 . Today we will take a closer look at xDai as a solution for your Dapp. What are...

  • 15 Stacks

    Stack Too Deep

    Three words of horror

    You just have to add one tiny change in your contracts. You think this will take you only a few seconds. And you are right, adding the code took you less than a minute. All happy about your coding speed you enter the compile command. With such a small change, you are confident your code is...

  • Chainlink Thumbnail

    Integrating the new Chainlink contracts

    How to use the new price feeder oracles

    By now you've probably heard of Chainlink. Maybe you are even participating the current hackathon ? In any case adding their new contracts to retrieve price feed data is surprisingly simple. But how does it work? Oracles and decentralization If you're confused about oracles, you're not alone. The...

  • truffle buidler typescript

    Adding Typescript to Truffle and Buidler

    How to use TypeChain to utilize the powers of Typescript in your project

    Unlike compiled languages, you pretty much have no safeguards when running JavaScript code. You'll only notice errors during runtime and you won't get autocompletion during coding. With Typescript you can get proper typechecking as long as the used library exports its types. Most Ethereum...

  • Balance Rope

    Integrating Balancer in your contracts

    What is Balancer and how to use it

    What is Balancer? Balancer is very similar to Uniswap . If you're not familiar with Uniswap or Balancer yet, they are fully decentralized protocols for automated liquidity provision on Ethereum. An easier-to-understand description would be that they are decentralized exchanges (DEX) relying on...

  • mousetrap

    Navigating the pitfalls of securely interacting with ERC20 tokens

    Figuring out how to securely interact might be harder than you think

    You would think calling a few functions on an ERC-20 token is the simplest thing to do, right? Unfortunately I have some bad news, it's not. There are several things to consider and some errors are still pretty common. Let's start with the easy ones. Let's take a very common token: ... Now to...

  • Aave

    Why you should automatically generate interests from user funds

    How to integrate Aave and similar systems in your contracts

    If you're writing contracts that use, hold or manage user funds, you might want to consider using those funds for generating free extra income. What's the catch? That's right, it's basically free money and leaving funds unused in a contract is wasting a lot of potential. The way these...

  • Matic Logo

    How to use Matic in your Dapp

    Deploying and onboarding users to Matic to avoid the high gas costs

    Gas costs are exploding again, ETH2.0 is still too far away and people are now looking at layer 2 solutions. Here's a good overview of existing layer 2 projects: https://github.com/Awesome-Layer-2/awesome-layer-2 . Today we will take a closer look at Matic as a solution for your Dapp. Why Matic...

  • Migrating from Truffle to Buidler

    And why you should probably keep both.

    Why Buidler? Proper debugging is a pain with Truffle. Events are way too difficult to use as logging and they don't even work for reverted transactions (when you would need them most). Buidler gives you a console.log for your contracts which is a game changer. And you'll also get stack traces...

  • Factory

    Contract factories and clones

    How to deploy contracts within contracts as easily and gas-efficient as possible

    The factory design pattern is a pretty common pattern used in programming. The idea is simple, instead of creating objects directly, you have an object (the factory) that creates objects for you. In the case of Solidity, an object is a smart contract and so a factory will deploy new contracts for...

  • IPFS logo

    How to use IPFS in your Dapp?

    Using the interplanetary file system in your frontend and contracts

    You may have heard about IPFS before, the Interplanetary File System. The concept has existed for quite some time now, but with IPFS you'll get a more reliable data storage, thanks to their internal use of blockchain technology. Filecoin is a new system that is incentivizing storage for IPFS...

  • tiny-kitten

    Downsizing contracts to fight the contract size limit

    What can you do to prevent your contracts from getting too large?

    Why is there a limit? On November 22, 2016 the Spurious Dragon hard-fork introduced EIP-170 which added a smart contract size limit of 24.576 kb. For you as a Solidity developer this means when you add more and more functionality to your contract, at some point you will reach the limit and when...

  • EXTCODEHASH

    Using EXTCODEHASH to secure your systems

    How to safely integrate anyone's smart contract

    What is the EXTCODEHASH? The EVM opcode EXTCODEHASH was added on February 28, 2019 . Not only does it help to reduce external function calls for compiled Solidity contracts, it also adds additional functionality. It gives you the hash of the code from an address. Since only contract addresses...

  • Uniswap

    Using the new Uniswap v2 in your contracts

    What's new in Uniswap v2 and how to integrate Uniswap v2

    What is UniSwap? If you're not familiar with Uniswap yet, it's a fully decentralized protocol for automated liquidity provision on Ethereum. An easier-to-understand description would be that it's a decentralized exchange (DEX) relying on external liquidity providers that can add tokens to smart...

  • Continuous Integration

    Solidity and Truffle Continuous Integration Setup

    How to setup Travis or Circle CI for Truffle testing along with useful plugins.

    Continuous integration (CI) with Truffle is great for developing once you have a basic set of tests implemented. It allows you to run very long tests, ensure all tests pass before merging a pull request and to keep track of various statistics using additional tools. We will use the Truffle...

  • Devcon 6

    Upcoming Devcon 2021 and other events

    The Ethereum Foundation just announced the next Devcon in 2021 in Colombia

    Biggest virtual hackathon almost finished First of all, the current HackMoney event has come to an end and it has been a massive success. One can only imagine what kind of cool projects people have built in a 30 days hackathon. All final projects can be seen at:...

  • ERC-2020

    The Year of the 20: Creating an ERC20 in 2020

    How to use the latest and best tools to create an ERC-20 token contract

    You know what an ERC-20 is, you probably have created your own versions of it several times (if not, have a look at: ERC-20 ). But how would you start in 2020 using the latest tools? Let's create a new ERC-2020 token contract with some basic functionality which focuses on simplicity and latest...

  • hiring

    How to get a Solidity developer job?

    There are many ways to get a Solidity job and it might be easier than you think!

    You have mastered the basics of Solidity, created your first few useful projects and now want to get your hands on some real-world projects. Getting a Solidity developer job might be easier than you think. There are generally plenty of options to choose from and often times not a lot of...

  • People making fun

    Design Pattern Solidity: Mock contracts for testing

    Why you should make fun of your contracts

    Mock objects are a common design pattern in object-oriented programming. Coming from the old French word 'mocquer' with the meaning of 'making fun of', it evolved to 'imitating something real' which is actually what we are doing in programming. Please only make fun of your smart contracts if you...

  • React and Ethereum

    Kickstart your Dapp frontend development with create-eth-app

    An overview on how to use the app and its features

    Last time we looked at the big picture of Solidity and already mentioned the create-eth-app . Now you will find out how to use it, what features are integrated and additional ideas on how to expand on it. Started by Paul Razvan Berg, the founder of sablier , this app will kickstart your frontend...

  • Solidity Overview

    The big picture of Solidity and Blockchain development in 2020

    Overview of the most important technologies, services and tools that you need to know

    Now, I do not know about you, but I remember when I first started with Solidity development being very confused by all the tools and services and how they work in connection with one another. If you are like me, this overview will help you understand the big picture of Solidity development. As I...

  • Design Pattern Solidity: Free up unused storage

    Why you should clean up after yourself

    You may or may not be used to a garbage collectors in your previous programming language. There is no such thing in Solidity and even if there was a similar concept, you would still be better off managing state data yourself. Only you as a programmer can know exactly which data will not be used...

  • How to setup Solidity Developer Environment on Windows

    What you need to know about developing on Windows

    Using Windows for development, especially for Solidity development, can be a pain sometimes, but it does not have to be. Once you have configured your environment properly, it can actually be extremely efficient and Windows is a very, very stable OS, so your overall experience can be amazing. The...

  • Avoiding out of gas for Truffle tests

    How you do not have to worry about gas in tests anymore

    You have probably seen this error message a lot of times: Error: VM Exception while processing transaction: out of gas Disclaimer : Unfortunately, this does not always actually mean what it is saying when using Truffle , especially for older versions. It can occur for various reasons and might be...

  • Design Pattern Solidity: Stages

    How you can design stages in your contract

    Closely related to the concept of finite-state machines, this pattern will help you restrict functions in your contract. You will find a lot of situations where it might be useful. Any time a contract should allow function calls only in certain stages. Let's look at an example: contract Pool {...

  • Web3 1.2.5: Revert reason strings

    How to use the new feature

    A new Web3 version was just released and it comes with a new feature that should make your life easier. With the latest version 1.2.5 , you can now see the the revert reason if you use the new handleRevert option. You can activate it easily by using web3.eth.handleRevert = true . Now when you use...

  • Gaining back control of the internet

    How Ocelot is decentralizing cloud computing

    I recently came across an ambitious company that will completely redefine the way we are using the internet. Or rather, the way we are using its underlying infrastructure which ultimately is the internet. While looking at their offering, I also learned how to get anonymous cloud machines, you...

  • Devcon 5 - Review

    Impressions from the conference

    I had a lot to catch up on after Devcon. Also things didn't go quite as planned, so please excuse my delayed review! This year's Devcon was certainly stormy with a big typhoon warning already on day 1. Luckily (for us, not the people in Tokyo), it went right past Osaka. Nevertheless, a lot of...

  • Devcon 5 - Information, Events, Links, Telegram

    What you need to know

    Devcon 5 is coming up soon and there are already lots of events available, information about Osaka and more. Here is a short overview: Events Events Calendar Events Google Docs Events Kickback Most events are in all three, but if you really want to see all, you will have to look at all three...

  • Design Pattern Solidity: Off-chain beats on-chain

    Why you should do as much as possible off-chain

    As you might have realized, Ethereum transactions are anything but cheap. In particular, if you are computing complex things or storing a lot of data. That means sometimes we cannot put all logic inside Solidity. Instead, we can utilize off-chain computations to help us. A very simple example...

  • Design Pattern Solidity: Initialize Contract after Deployment

    How to use the Initializable pattern

    There are a few reasons why you might want to initialize a contract after deployment and not directly by passing constructor arguments. But first let's look at an example: contract MyCrowdsale { uint256 rate; function initialize(uint256 _rate) public { rate = _rate; } } What's the advantage over...

  • Consensys Blockchain Jobs Report

    What the current blockchain job market looks like

    Consensys published their blockchain jobs report which you can checkout in their Blockchain Developer Job Kit . The most interesting aspects are Blockchain developer jobs have been growing at a rate of 33x of the previous year according to LinkedIns jobs report Typical salary is about...

  • Provable — Randomness Oracle

    How the Oraclize random number generator works

    One particularly interesting approach by Provable is the usage of a hardware security device, namely the Ledger Nano S. It uses a trusted execution environment to generate random numbers and provides a Provable Connector Contract as interface. How to use the Provable Randomness Oracle? Use the...

  • Solidity Design Patterns: Multiply before Dividing

    Why the correct order matters!

    There has been a lot of progress since the beginning of Ethereum about best practices in Solidity. Unfortunately, I have the feeling that most of the knowledge is within the circle of experienced people and there aren’t that many online resources about it. That is why I would like to start this...

  • Devcon 5 Applications closing in one week

    Devcon 5 Applications closing

    Watch out for the Devcon 5 applications. You only have one week left to apply either as Buidler Student Scholarship Press Devcon is by far the biggest and most impressive Ethereum conference in the world. And it's full of developers! I am especially excited about the cool location this year in...

  • Randomness and the Blockchain

    How to achieve secure randomness for Solidity smart contracts?

    When we talk about randomness and blockchain, these are really two problems: How to generate randomness in smart contracts? How to produce randomness for proof-of-stake (POS) systems? Or more generally, how to produce trusted randomness in public distributed systems? There is some overlap of...