For years now, TPS, or transactions per second, has been the go-to metric for blockchain developers trying to hype up their snazzy new networks.
Compared to Bitcoin’s seven transactions per second maximum (usually closer to four), for years, Ripple claimed that XRP was able to handle 1,500 TPS, on par with Visa’s transactions per second, although chief technology officer David Schwartz admitted last year that it has never actually hit that figure.
Solana transactions per second are claimed to be 65,000 TPS in benchmark testing, although they are closer to 3,000 TPS in the real world — and it’s “real TPS” metric is much lower again. Even newer blockchain networks now tout up to 297,000 transactions per second — “theoretically,” of course.
But how much stock should one place in the metric itself?
Various blockchain industry leaders argue that the advent of transaction bundling today makes TPS a pretty flawed measure of blockchain performance and that TPS can be — and often is — gamed.
But the reason it’s still the most common metric may be that there’s no single better alternative.
Rise and limitations of blockchain transactions per second metric
In the earlier days of crypto, TPS was the only metric that mattered, given that blockchains like Bitcoin and Litecoin mostly just sent transactions from one address to another — like when Laszlo Hanyecz spent 10,000 BTC to buy two pizzas in 2010.
At the time, TPS helped users understand how different block size choices or cryptographic algorithms from potential forks or chains could impact the speed by which their transactions could be processed. “Scaling” was mainly a debate over how many TPS a blockchain could handle, which was important if crypto was going to become the world’s digital currency.
When Ethereum, with its 13 or so TPS and programmable smart contracts, entered the scene in July 2015, many operations became far more computationally intensive than simple transactions.
Account abstraction and blockchain transactions bundling
Years later, account abstraction would allow users to group multiple actions (called UserOps) through transaction bundling, further widening TPS’ growing blindspot.
Offchain Labs co-founder Steven Goldfeder explains that focusing on the raw TPS number is a bit like “counting the number of bills in your wallet but ignoring that some are singles, some are twenties, and some are hundreds.”
Some transactions could have much more computational value than others, though under TPS, all would be counted as one and the same.
Offchain Labs is the developer of the Ethereum layer-2 scaling solution Arbitrum One, aimed at providing a scalable environment for decentralized applications (DApps) and smart contracts. It has native account abstraction and averages around 9.95 TPS — though it claims to be capable of 40,000 TPS.
flexing TPS is like saying rice is the best food because you eat 10,000 per meal
— pseudo (on farcaster)
(@pseudotheos) January 23, 2024
Solana is also grappling with ever-increasing transaction complexity, says Austin Federa, head of strategy.
“Solana actually is probably at least five times faster now than it was when I joined. But you don’t see it because […] the complexity of the transactions has massively increased.”
A simple transaction such as a consensus vote or sending SOL to another user has a low compute cost, while an arbitrage transaction or minting an NFT could be “100 times more compute-intensive.” Both would be counted as the same under TPS, explained Federa.
“So, Solana today is powering many more complex transactions than it was back in 2021 — even though the number of transactions per second has not astronomically gone up.”
But according to their marketing, Solana’s TPS is already astronomical, which has been a source of controversy.
Solana transactions per second claims: Inflated or misunderstood?
Solana’s marketed TPS has borne the brunt of industry criticism, with detractors taking issue with Solana’s claimed 65,000 TPS in “benchmark testing,” while its white paper touts a theoretical 710,000 TPS at a 1-gigabit-per-second network connection.
The solana.com website currently posts a live feed of transactions per second at nearly 3,000. Critics argue that 80%–90% of it is made up of non-user transactions.
Federa argues the criticisms are unwarranted. “Votes are real transactions on Solana that pay their way,” he says.
“Some people are like, ‘I don’t want to count votes; votes aren’t counted in other networks.’ Okay, I will accept your premise. Let’s look at the real TPS number, then.”
Staking tool provider Solana Compass currently lists a “true TPS” — which excludes consensus vote transactions — of 704 for the network.
“The true TPS number that Solana Compass reports is entirely designed to dispel it, and that number is still 10x what the nearest blockchain is able to do,” said Federa.
“Then if they look at that and are like, ‘Oh, well, it’s all spam from X or Y or Z,’ like at that point, we’re not dealing with serious people anymore.”
Blockchain transactions: How TPS are pumped up
American author and humorist Mark Twain once famously said, “There are three kinds of lies: Lies, damned lies and statistics.”
Neil Davies, a systems performance scientist at Input Output — the firm behind Cardano — told Magazine that this is certainly the case with the use of TPS in the industry today.
“People like to latch on to ‘simple’ ideas,” explains Davies.
“Benchmarks have value when the quantity they report is a good proxy for a performance characteristic that they have a real need for.”
Unfortunately, “bragging rights” appear to have been the more prominent development driver, he argued.
Davies criticizes chains that count “extensive inter-node messaging” as part of their TPS metric.
“Such ‘transactions’ are not a proxy for any end-user activity — it would seem that they are endeavoring to make a ‘virtue’ of their overheads,” he said.
Offchain Labs’ Goldfeder agrees:
“They’ll say, ‘Oh, TPS, my chain can do 1,000 TPS,’ then there’s an asterisk to say […] they did the most basic transactions like a no-op. Could be literally a transaction that does nothing or a transfer of the underlying asset.”
Solana’s Federa believes that “every metric and blockchain and everything is gamed to some extent. So, be very suspicious of peak numbers.”
“Solana is such a mature network that there is no one sitting there just sending transactions to make numbers look good. That’s not the case on other networks necessarily.”
But if TPS is often gamed and not that illustrative of capacity anyway, are there any better alternatives?
Alternatives to blockchain TPS: User Operations (UserOps) per second
Anthony Rose, senior vice president of technology at Matter Labs — the developers of zkSync — believes user operations (UserOps) per second could be a “more meaningful metric” but admits it’s a measurement that the community is not “super well calibrated with.”
Ethereum’s account abstraction standard, known as ERC-4337 introduced “pseudo-transaction” objects called UserOperations. They work like instructions to tell a smart account what action to take on behalf of a user.
TPS is a very faulty metric to measure blockchain throughput. And it's going to get worse in the future. One of the potential alternatives is to measure UOPS – User Ops per second. But what are they, how they relate to AA and is it really a feasible alternative ?
pic.twitter.com/wTx16AJra7
— bartek.eth (@bkiepuszewski) October 31, 2023
For example, making a simple swap from one token to another on a decentralized exchange normally requires two separate transactions. With account abstraction, these UserOps are batched together. Unfortunately, TPS still sees it all as just one and the same transaction.
“As we see more adoption for account abstraction and this improved UX, TPS gets increasingly further from being a useful metric,” says Rose.
However, there’s a major flaw to UserOps per second: The metric isn’t really applicable outside the Ethereum ecosystem. Federa isn’t a fan, arguing it rewards overly-complicated products.
“It’s like counting how many visits to a website based on how many HTTP requests there are,” he says.
Typically, a well-optimized website will make fewer HTTP requests, allowing the site to load faster.
“Like, YouTube is pulling in 60 concurrent HTTP requests, whereas Wikipedia, it’s like two or three.”
“And so, this is where the user operations per second metric really fall apart. It rewards complicated products.”
Gas per second (GPS) as blockchain transctions per second replacement
“We will all evolve to some kind of equivalent of gas per second,” says Avihu Levy, chief product officer of StarkWare.
Users pay gas to process smart contracts on the Ethereum network, so this metric is a way to account for the computational work of a system.
Gas per second, or GPS, would thus account for the size and type of transaction, as well as block size and block time to measure throughput. Levy argues it’s the closest metric to what a network can do in terms of computation or resources consumed per second.
“So, if I know my network can do medium computation for a second, then I can try to derive a comparison between other networks that can do different numbers of computational steps.”
The downside, Levy admits, is that networks measure computation differently.
“So, if you talk about EVM using gas per second, it’s a very good metric because they all measure the complexity of the computation in exactly the same way.”
Starknet, however, measures computation via Cairo steps. Solana uses “Compute Units,” while Aptos, another layer-1 blockchain, uses gas units.
Levy explains that for all of these non-Ethereum ecosystem blockchains, the challenge would be to create a “canonical benchmark” to properly convert the various computation measures.
The difficulty with making standardized comparisons between different, complicated blockchains and projects is probably the reason TPS has remained top dog, even with all its flaws.
The future of blockchain transactions performance metrics like TPS
While he’s no fan, Rose says TPS seems likely to be around for a while longer, as most of the market isn’t advanced enough to grasp more insightful metrics.
“It’s a flawed metric, for sure […] I don’t even think there are any strong counterarguments,” says Rose, though he admits that TPS is still one that “many people are anchored on.”
“People understand it in a way [unlike] significantly more meaningful metrics […]. People don’t have deep intuition about them yet.”
“There’s a trade-off between explainability, understandability and then, obviously, the quality of the metric as well,” he adds.
Rose says Matter Labs actually still uses the metric to see how design changes can impact performance in internal testing environments.
However, with the adoption of account abstraction on Ethereum, Rose says a move toward UOPS “absolutely makes sense.”
Meanwhile, Davies says the inertia of TPS will remain until players start using such blockchains beyond simple “stores of value.”
“If you are speculating (in blockchains), TPS is only valuable to the extent it helps ‘shill’ your ‘investment,’” he says.
Federa, however, thinks it’s likely that TPS won’t even matter for most users once the technology is fast enough — just like the case for phones and computers today.
“The success of the personal computer industry is you don’t have to really care that much about what the computer in front of you is.”
“We are not there with blockchains yet, but I really hope we get to a place where all blockchains are functionally fast enough that the only people who actually need to worry about this are developers,” said Federa.
week.
The post Is measuring blockchain transactions per second (TPS) stupid in 2024? Big Questions appeared first on Cointelegraph Magazine.