geth is underfilling mainnet blocks

[dataalways]

Starting with Edolus (v1.13.12), a couple weeks before Deneb, geth blocks began to use significantly less gas than other builders/engines when the network isn't over capacity. Currently the median block built by geth is using less than 3 million gas and carries less than 50 transactions.

Figure 1: average gas usage of blocks separated by builder.

Transactions on the network tend to be heavily skewed toward low priority fees, with over 40% of transactions using less than 1.5% of their tips in gas (over 50% in locally built blocks).

Figure 2: the distribution of priority fee share of gas price for transactions on the Ethereum Network.

As can be seen in Figure 3, in blocks built by geth the majority of the left spike is missing, accounting for approximately 50% of transactions. Because these transactions contain little-to-no tips, the value of geth blocks is relatively unaffected and remains competitive with Nethermind blocks.

Figure 3: the distribution of priority fee share of gas price for transactions in blocks built by geth.

[karalabe]

Sigh, answering it for the 3rd time on a 3rd completely different channel.

First answer:

Ok, so we had a bug in Geth since the merge
Where instead of enforcing the user supplied (or default) minimum gas tip for miners, we elegantly ignored it altogether, adding even txs that paid as low as 1 wei to blocks (if there was free space)
I noticed the bug when doing some refactors and fixed it in 1.13.12 https://github.com/ethereum/go-ethereum/releases/tag/v1.13.12
Specifically #28933
This all of a sudden start enforcing the default 1 gwei gas price. I think mst other clients accept whatever from the network as tips
So, it does work as intended, maybe the question is whether the 1gwei is a good enough number or not to be used as a default.
But anyone can set it now on their own node and it should accept lower tip txs even if the default is "too high"
As for the default of 1gwei, it's an interesting question, maybe we could bring it up on ACD as to what should be a safe correct one.

Second answer:

our default miner tip is 1 gwei
everyone else is 0
it was updated in 1.13.12
until then we had a bug and accepted everything
lowering it to 0.1 would probably already fill blocks mostly
but that's up to miners really
we could change the default, but unsure what's the best
maybe bring it up at the interop or at an acd

[karalabe]

40% of transactions using less than 1.5% of their tips in gas

This makes no sense. Gas is one unit and gas price is an independent unit.

[karalabe]

Because these transactions contain little-to-no tips, the value of geth blocks is relatively unaffected and remains competitive with Nethermind blocks.

Yes, working as intended. A transaction either pays the miner's minimum tip requirement, or it's not getting included.

[karalabe]

If you want your miner, to accept close-to-no-tip transactions, run it with --miner.gasprice=1.

[dataalways]

Peter, you and I have never private messaged so I reject your implication that you're being harassed. And I've never claimed geth is bad, simply that it is making an opinionated choice and that the community has not been briefed on its impact to the network. I don't know who your quoted private messages are with, but they are not with me.

My contact with the geth team has been: pointing out the reduced gas usage to Marius who responded with:

Nope wasn't aware, super interesting. So our blocks are still valuable, just not as full? It can be a symptom of our recent miner refactor.

And then after I asked him to confirm that it wasn't a security issue and that I could talk about it publicly he responded with:

Nope, you can talk about it publicly :) Thanks for sharing!

The only other 'contact' I've had with the geth team was a private document that I shared with a couple fellow researchers to provide them access to cleaned data on the topic, since I don't believe you provide any public monitoring of your blocks.

Someone (not me) must have shared the link with Martin, and he commented:

Yes. We don't accept zero-tip transactions in blocks any more. This is working as intended, I don't think there's any mystery here.

I responded:

okay, i opened an issue because i think intentionally filling blocks with only 2 million gas is not working as intended.

And that gets us to here, where you instantly closed the only public forum on the topic stating that geth is "working as intended".

---

On to the topic:

What geth has done is create a tragedy of the commons and offloaded compute onto subsequent proposers and builders. Over half of blocks following geth blocks now run up against the gas limit which implies multi-block delays for valid transactions. It also destroys the UX gains from EIP-1559 and causes severe oscillations in gas prices. And now that geth is doing it, MEV builders will follow suit because the reduction in block size will really help them with latency.

As you correctly point out elsewhere, this situation is now far more complicated because of multidimensional fee markets. One of the tenets that makes these markets work well is the assumption that all valid transactions get included in blocks. By choosing to reject the consensus transaction inclusion rules, geth is complicating this future.

The change is further complicating discussions on how to improve the censorship resistance of the network. Right now, geth's behavior can be handled by the network because it only builds about 6% of blocks, but there's been a serious push to get more validators to build blocks locally--with @timbeiko going as far as hoping for 50% of blocks to be built locally. If that PR had been accepted and geth were building ~40% of blocks on the network, it would be a disaster with geth's minimum tip policy.

Since geth seems prone to continue enforcing minimum without researching the effects and communicating the drawbacks to stakeholders, the rest of the community needs to identify where previous assumptions have been broken.

I look forward to ACD. Thank you for opening the ticket, I hadn't gone around to it yet.

[karalabe]

Peter, you and I have never private messaged so I reject your implication that you're being harassed.

Uri from bloXroute shared your charts with me on the 12th and told me you're investigating this. I made a very long reply to him on the 13th explaining what's happening. If he didn't forward it, then you need better friends.

Less than 6 hours later, Marius pings me with the same charts rehashing the issues from zero. I explained to him too what's happening. If @MariusVanDerWijden didn't forward my reply either, then you really need better friends.

15 hours later, a new post of yours was shared by Marius on the Geth channel, again stirring up crap, where @fjl, @holiman and myself explained to him why this is happening. Again, if Marius is a black hole not forwarding information, I'm sorry.

Then a couple hours later, this issue appears, restarting the exact same problem we've answered 3 times already, so I - with next to no patience left - answer here too and close the issue.

Then a day later you open a PR against Prysm to penalize Geth builders because you didn't like my response.

---

Dude, I responded to the issue you raised 6 different times already. Find better friends if nobody who contacts me directly is actually bothering to send any information back to you.

[dataalways]

I never asked any of them to ask you anything. I only pointed out some weird behavior and asked a few people if they knew what was going on. If your DMs on Twitter were open I would have pinged you originally instead of Marius--who is not my friend and I had never previously talked to. Apology accepted I guess? From my side, I'm sorry that this spiralled into a big thing, it was not my intention.

And it's not about not liking your response, I have no issue with you or geth making opinionated decisions and excluding transactions. The prysm PR would not penalize geth builders at all, it would simply remove the economic penalty for mev bids compared to geth block values to reflect that geth might not be providing better CR than mev builders. That seems totally reasonable to me, do you believe that geth is providing better CR than the average mev builder currently?

And I did not maliciously remove your read rights to my charts. The document was a placeholder and the queries are all publicly available here. The specific chart you linked is available here with the source code for anyone that wants to do their own analysis.