Issues of Returning Arrays of Dynamic Size in Solidity Smart Contracts

Limitations of functions that return arrays of big size and ways to handle them

Published in

Better Programming

5 min readSep 5, 2022

flex tape meme. man slapping black patch on container leak. caption: developer misusing smart contracts. leaking container caption: EVM which is not designed for getting arrays of big size. black patch caption: pagination — image by author

Since smart contracts were introduced, the complexity of systems based on them has grown a lot. It began with simple voting contracts, ERC20 tokens and went to sophisticated architectures like Uniswap, Aragon, etc.

Now closer to our story: recently Custom App (development studio), where I’m working, was developing a Minter Guru dApp, and I was working on system architecture and smart contracts. By the philosophy of Minter Guru, we have to move as much logic as possible to smart contracts.

While implementing read(from CRUD pattern) functions, we’ve faced the issue of returning dynamic-sized arrays, which can consist of hundreds or even thousands of elements. So, we’ve researched the limitations of implementing such functions.

In this article, we will discuss these limitations, ways to handle them, and the feasibility of “Implementing Web2 backend with smart contracts” approach to building applications. However, if you are an experienced web3 developer, our results may be obvious to you. :)

For better understanding, you should be familiar with Ethereum blockchain basics (or any other EVM-compatible blockchain like Polygon or Binance Smart Chain), Solidity, and Hardhat. All code from the examples and instructions on how to run it is available at our GitHub repository.

Gas Limit

Our first thought was that, theoretically, we could reach the gas limit.

In EVM, every function call requires gas. Since calls don’t spend real gas, we can set any gas limit, but due to EVM implementation maximum gas limit is equal to the maximum for uint64 type, which equals to 18446744073709551615.

Let’s check how fast we will achieve this limit on simple contract example. We will use a simple contract with uint256[] storage variable, push n values to array function and getter method. Also, we need two scripts to deploy the contract, and to test the required gas limit. We will use Hardhat for our local network and for interacting with contracts.

Gas spent is based on array size, as EVM will copy array data to memory. So, we will research the dependency between gas spent and array size in bytes. We are using uint256[] array for simplicity. For other data types calculation of encoded array size in bytes differs. According to Solidity spec, Enc(uint256[])=Enc(array.size)(Enc(array[0],…,array[array.size-1])). Hence, the size in bytes of our array equals to size(uint256)+size(uint256)*array.size=32+32*array.size.

While running this script, we saw that for loop iterations are starting to get ridiculously slow. The 10th iteration took 10+ seconds, and the 50th took 4+ minutes. By the way, let’s check some of the results (the whole table can be found in our GitHub repository):

Estimated gas values for some array sizes

On the chart, we can see that estimated gas depends on array size linearly, so let’s build extrapolation to estimate the theoretical size of the array. We’ve done it using Google Spreadsheets and linear regression. You can check it here.

As one can see, the theoretical uint256 array size limit approximately equals to 1.8e+17 in kilobytes (163709 in terabytes) which is practically unachievable. However, as we have seen, execution time is a great issue.

Execution Time

First, let’s mention that we are doing tests on Hardhat network, which was implemented with Node.js, so its performance will be worse than Golang implementation, which is used in Ethereum nodes. However, the performance of smart contract functions is more affected by the design of EVM than by EVM implementation language.

Parameters of the machine on which the test was performed:

CPU — 11th Gen Intel® Core™ i7–1165G7 @ 2.80GHz × 8
RAM — 16 GB
OS — Ubuntu 22.04 LTS
Disk — 512 GB SSD

Now let’s modify our test script to measure execution time. We only need to change the for loop in our test script to log execution time. Here’s the code:

Execution time of getting array of some sizes

Now we can see that the high execution time problem appears fast. Twelve seconds for only 320 kilobytes of data is too big an execution time, so time becomes our main issue, which we need to overcome.

Pagination

Unfortunately, there is no magic way to get an array of big sizes in a reasonable time. But if your application doesn’t need a full array at one moment, you can use a pagination pattern just like in Web2. However, if it does, this is a sign of architecture problems, which can’t be solved in smart contract systems or classic Web2 applications.

Pagination is often used by developers to limit the size of returning data in one API call. This approach decreases latency and reduces the number of UI elements to render for the browser and mobile applications.

Implementation of pagination consists of the following steps:

Defining sorting on our array
Adding page number and page size to API method (contract function in our case). Your new method should return a slice of the sorted array: array[page*size:min((page+1)*size, array.length)]
Adding the total quantity of elements in an array to the response lets you find the page quantity on the client.

Now let’s modify our test smart contract, as shown below:

And test script. We will also save execution time for each get page call to show that it stays the same for any page. Here’s the code:

Let’s check our results. Again, full results can be found in our repository. Also, to reduce the table size, we will show only the average and maximum execution time for a get page call:

Execution time of getting arrays by pages

So now we can get parts of our array in a reasonable time.

One last detail: if you want to iterate through the array in which a state can be modified too often, you can use a fixed block number while calling the function. This guarantees the state will remain the same at every one-page call. With Hardhat contract bindings, it can be done as follows:

Conclusion

So, there are two main points which we found out:

The main problem with functions returning arrays of big size is call execution time.
With pagination, we can make call time reasonable enough to get part of the data. That pattern can be used if the for logic of your application is OK to work with parts of the array at one moment.

We have done all our tests with the simplest getter logic, but in your application, getter logic can be much more complex, so while designing your architecture, you should be careful about moving everything to smart contracts.

You may face issues you can’t solve after deploying to mainnet (if your contracts are not upgradable). If you feel your system is too complex to put it all in smart contracts, combining smart contracts with classic Web2 solutions will be wise to make a better application.