Skip to content

Commit

Permalink
Trace/db traces (#4535)
Browse files Browse the repository at this point in the history
* Implement TraceStore plugin

* Add compression to db

* Allow unsafe to avoid allocation

* Fix EndBlockTrace to be after Applying rewards

* Add ByteArrayPoolList

* Make ResettableList implement IReadOnlyCollection

* Make DbOnTheRocks default to nameless configuration if named not found

* Make BlockTracerBase based on ResettableList

* Fixes and improvements

* Add plugin to pipelines

* fix test

* Add temporary traces

* Revert "Add temporary traces"

This reverts commit 9190ed0.

* Add exception data to json rpc errors

* Increase MaxDepth to 128

* Refactor product versioning and revise Docker files (#4781)

* Upgrade version to 14.4 (#4796)

Co-authored-by: Kamil Chodoła <kamil@nethermind.io>

* Chiado TTD (#4801)

* Updating Fast Sync config files (#4803)

Co-authored-by: LukaszRozmej <LukaszRozmej@users.noreply.github.com>

* Feature: added extra data length checks (#4805)

* Added 32 bytes check on extra data

Added checks and tests for the ExtraData not to be longer than 32 bytes

* Fix workflow to pass parameters to scripts (#4799)

* Autorecover when disconnected chain is found (#4816)

* Autorecover when disconnected chain is found

* Fix tests

* Updating Fast Sync config files (#4819)

Co-authored-by: MarekM25 <MarekM25@users.noreply.github.com>

* Fix environment variable definition (#4825)

* Revert "Fix sepolia invalid block (#4694)" (#4838)

This reverts commit 8f28e4a.

* Bump build version to 14.5 (#4839)

Co-authored-by: Kamil Chodoła <kamil@nethermind.io>

* fix transition creation

* Don't log error, when initialization is stopped (like when syncing previously downloaded blocks from db)

* Add VerifySerialized option

* Add serialization tests

* Move VerifySerialized to TraceSerializer, add MaxDepth config

* fix whitespace

* fix null handling in TxBroadcaster

* Refactor ITraceSerializer

* fix test

* undo fix

* whitespace fix

* test fix

* fixes

* Set chiado and xdai `Merge.SecondsPerSlot=5` (#4880)

* Set chiado `Merge.SecondsPerSlot=5`

* Update xdai configs with `Merge.SecondsPerSlot=5`

This commit is not relevant before the merge.

* Updating Fast Sync config files (#4897)

Co-authored-by: LukaszRozmej <LukaszRozmej@users.noreply.github.com>

* Set C# version to 10.0 (#4881)

* Add gnosischain TTD value (#4901)

* Add gnosischain TTD value

Scheduled for `Dec 5 2022` assuming 5s block times: `8626000000000000000000058750000000000000000000`

* Add EnginePort xdai configs

* Add xdai TTD test to ChainSpecLoaderTests

* Fix runner tests

* Override MergePlugin Name and Description fields in AuRaMergePlugin

* fix eth_call mix_hash and beneficiary (#4873)

* fix eth_call mix_hash and beneficiary

* always set MixHash
# Conflicts:
#	src/Nethermind/Nethermind.Facade/BlockchainBridge.cs

* Changes the docker password to a token (#4916)

* fix: changes the docker password to token

* fix: update docker pass for main release workflow

* parallelized trace_filter execution per block

* Remove unneded filters

* Reduce traces memory allocation

* fix trace filtering

* fix whitespace

* Updating Fast Sync config files (#4920)

Co-authored-by: LukaszRozmej <LukaszRozmej@users.noreply.github.com>

* Bump build version to 14.6 (#4923)

Co-authored-by: Kamil Chodoła <kamil@nethermind.io>

* Changes the docker password variable to token (#4924)

* fix: changes the docker password to token

* fix: update docker pass for main release workflow

* fix: change DOCKER_PASSWORD to token

* null action result when it is empty

* Temporarily switch to `ubuntu-20.04` image (#4927)

* Ignore ParityResult.IsEmpty in JSON

* Fix remove empty results

* Merge 1.14.7

* fix licensing header

* fix whitespace

* Final touches

* Make ITraceSerializer generic

* refactor

* Adds IsTracingFees docs

* FIx issue when Result was added to json, because it was created by default on deserialization, even if not serialized

* fix shutdown if TraceStorePlugin not Enabled

* Review fixes

* Fix tests

* fix shitespace

Co-authored-by: Ruben Buniatyan <rubo@users.noreply.github.com>
Co-authored-by: Kamil Chodoła <43241881+kamilchodola@users.noreply.github.com>
Co-authored-by: Kamil Chodoła <kamil@nethermind.io>
Co-authored-by: Jorge Mederos <46798594+jmederosalvarado@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: LukaszRozmej <LukaszRozmej@users.noreply.github.com>
Co-authored-by: Oleg Jakushkin <oleg.jakushkin@gmail.com>
Co-authored-by: Amirul Ashraf <asdacap@gmail.com>
Co-authored-by: MarekM25 <MarekM25@users.noreply.github.com>
Co-authored-by: Marek Moraczyński <marekm2504@gmail.com>
Co-authored-by: Falco <1364936+FalcoXYZ@users.noreply.github.com>
  • Loading branch information
12 people authored Dec 6, 2022
1 parent 1da7431 commit 2e8630b
Show file tree
Hide file tree
Showing 66 changed files with 1,741 additions and 215 deletions.
3 changes: 3 additions & 0 deletions .github/workflows/run-nethermind-tests-with-code-coverage.yml
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,9 @@ jobs:
- name: Nethermind.Sockets.Test
run: |
dotnet test -c Release $EXCLUDE_TEST_PROJECTS src/Nethermind/Nethermind.Sockets.Test
- name: Nethermind.JsonRpc.TraceStore.Tests
run: |
dotnet test -c Release $EXCLUDE_TEST_PROJECTS src/Nethermind/Nethermind.JsonRpc.TraceStore.Tests
- name: Upload Codecov Report
if: matrix.os == 'ubuntu-latest'
uses: actions/upload-artifact@v2
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/run-nethermind-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,9 @@ jobs:
- name: Nethermind.AccountAbstraction.Test
run: |
dotnet test src/Nethermind/Nethermind.AccountAbstraction.Test -c ${{ env.BUILD_CONFIG }}
- name: Nethermind.JsonRpc.TraceStore.Tests
run: |
dotnet test src/Nethermind/Nethermind.JsonRpc.TraceStore.Tests -c ${{ env.BUILD_CONFIG }}
neth-tests2:
name: Running Nethermind Tests 2
Expand Down
10 changes: 5 additions & 5 deletions scripts/deployment/archive-packages.sh
Original file line number Diff line number Diff line change
Expand Up @@ -35,11 +35,11 @@ dotnet build -c release Nethermind.Runner.csproj

cd $RELEASE_DIRECTORY

cp $RELEASE_DIRECTORY/$RELEASE_PATH/Nethermind.{Api,HealthChecks,EthStats,Merge.Plugin,Mev}.dll $LIN_RELEASE/plugins
cp $RELEASE_DIRECTORY/$RELEASE_PATH/Nethermind.{Api,HealthChecks,EthStats,Merge.Plugin,Mev}.dll $OSX_RELEASE/plugins
cp $RELEASE_DIRECTORY/$RELEASE_PATH/Nethermind.{Api,HealthChecks,EthStats,Merge.Plugin,Mev}.dll $WIN_RELEASE/plugins
cp $RELEASE_DIRECTORY/$RELEASE_PATH/Nethermind.{Api,HealthChecks,EthStats,Merge.Plugin,Mev}.dll $LIN_ARM64_RELEASE/plugins
cp $RELEASE_DIRECTORY/$RELEASE_PATH/Nethermind.{Api,HealthChecks,EthStats,Merge.Plugin,Mev}.dll $OSX_ARM64_RELEASE/plugins
cp $RELEASE_DIRECTORY/$RELEASE_PATH/Nethermind.{Api,HealthChecks,EthStats,Merge.Plugin,Mev,JsonRpc.TraceStore}.dll $LIN_RELEASE/plugins
cp $RELEASE_DIRECTORY/$RELEASE_PATH/Nethermind.{Api,HealthChecks,EthStats,Merge.Plugin,Mev,JsonRpc.TraceStore}.dll $OSX_RELEASE/plugins
cp $RELEASE_DIRECTORY/$RELEASE_PATH/Nethermind.{Api,HealthChecks,EthStats,Merge.Plugin,Mev,JsonRpc.TraceStore}.dll $WIN_RELEASE/plugins
cp $RELEASE_DIRECTORY/$RELEASE_PATH/Nethermind.{Api,HealthChecks,EthStats,Merge.Plugin,Mev,JsonRpc.TraceStore}.dll $LIN_ARM64_RELEASE/plugins
cp $RELEASE_DIRECTORY/$RELEASE_PATH/Nethermind.{Api,HealthChecks,EthStats,Merge.Plugin,Mev,JsonRpc.TraceStore}.dll $OSX_ARM64_RELEASE/plugins

cd $LIN_RELEASE && zip -r $LIN-$VERSION-$COMMIT_HASH-$DATE.zip . && cd ..
cd $OSX_RELEASE && zip -r $OSX-$VERSION-$COMMIT_HASH-$DATE.zip . && cd ..
Expand Down
2 changes: 1 addition & 1 deletion src/Nethermind/Directory.Build.props
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
<Copyright>Demerzel Solutions Limited</Copyright>
<Product>Nethermind</Product>
<SourceRevisionId Condition="'$(Commit)' != ''">$(Commit.Substring(0, 8))</SourceRevisionId>
<VersionPrefix>1.14.6</VersionPrefix>
<VersionPrefix>1.14.7</VersionPrefix>
<VersionSuffix></VersionSuffix>
</PropertyGroup>

Expand Down
18 changes: 2 additions & 16 deletions src/Nethermind/Nethermind.Consensus/BlocksConfig.cs
Original file line number Diff line number Diff line change
@@ -1,19 +1,5 @@
// Copyright (c) 2021 Demerzel Solutions Limited
// This file is part of the Nethermind library.
//
// The Nethermind library is free software: you can redistribute it and/or modify
// it under the terms of the GNU Lesser General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// The Nethermind library is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Lesser General Public License for more details.
//
// You should have received a copy of the GNU Lesser General Public License
// along with the Nethermind. If not, see <http://www.gnu.org/licenses/>.
//
// SPDX-FileCopyrightText: 2022 Demerzel Solutions Limited
// SPDX-License-Identifier: LGPL-3.0-only

using System;
using System.Reflection;
Expand Down
17 changes: 2 additions & 15 deletions src/Nethermind/Nethermind.Consensus/IBlocksConfig.cs
Original file line number Diff line number Diff line change
@@ -1,18 +1,5 @@
// Copyright (c) 2021 Demerzel Solutions Limited
// This file is part of the Nethermind library.
//
// The Nethermind library is free software: you can redistribute it and/or modify
// it under the terms of the GNU Lesser General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// The Nethermind library is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Lesser General Public License for more details.
//
// You should have received a copy of the GNU Lesser General Public License
// along with the Nethermind. If not, see <http://www.gnu.org/licenses/>.
// SPDX-FileCopyrightText: 2022 Demerzel Solutions Limited
// SPDX-License-Identifier: LGPL-3.0-only

using System.Collections.Generic;
using System.Reflection;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -223,10 +223,10 @@ protected virtual TxReceipt[] ProcessBlock(
_receiptsTracer.SetOtherTracer(blockTracer);
_receiptsTracer.StartNewBlockTrace(block);
TxReceipt[] receipts = _blockTransactionsExecutor.ProcessTransactions(block, options, _receiptsTracer, spec);
_receiptsTracer.EndBlockTrace();

block.Header.ReceiptsRoot = receipts.GetReceiptsRoot(spec, block.ReceiptsRoot);
ApplyMinerRewards(block, blockTracer, spec);
_receiptsTracer.EndBlockTrace();

_stateProvider.Commit(spec);
_stateProvider.RecalculateStateRoot();
Expand Down
2 changes: 2 additions & 0 deletions src/Nethermind/Nethermind.Consensus/Processing/BlockRef.cs
Original file line number Diff line number Diff line change
Expand Up @@ -46,5 +46,7 @@ public bool Resolve(IBlockTree blockTree)

return true;
}

public override string ToString() => Block?.ToString() ?? BlockHash.ToString();
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -175,5 +175,18 @@ public void Set_should_throw(int item)
Action action = () => list[item] = 1;
action.Should().Throw<ArgumentOutOfRangeException>();
}

[TestCase(1, 16)]
[TestCase(14, 16)]
[TestCase(15, 32)]
[TestCase(20, 32)]
[TestCase(100, 128)]
public void AddRange_should_expand(int items, int expectedCapacity)
{
ArrayPoolList<int> list = new(16) { 0, 1 };
list.AddRange(Enumerable.Range(2, items));
list.Should().BeEquivalentTo(Enumerable.Range(0, items + 2));
list.Capacity.Should().Be(expectedCapacity);
}
}
}
2 changes: 2 additions & 0 deletions src/Nethermind/Nethermind.Core/BlockInfo.cs
Original file line number Diff line number Diff line change
Expand Up @@ -76,5 +76,7 @@ public bool IsBeaconInfo
/// This property is not serialized
/// </summary>
public long BlockNumber { get; set; }

public override string ToString() => BlockHash.ToString();
}
}
22 changes: 17 additions & 5 deletions src/Nethermind/Nethermind.Core/Collections/ArrayPoolList.cs
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,13 @@
using System.Collections.Generic;
using System.Runtime.CompilerServices;
using System.Threading;
using System.Threading.Tasks;
using Nethermind.Core.Crypto;

namespace Nethermind.Core.Collections
{
public class ArrayPoolList<T> : IList<T>, IReadOnlyList<T>, IDisposable
{
private readonly ArrayPool<T> _arrayPool;
private T[] _array;
protected T[] _array;
private int _count = 0;
private int _capacity;
private bool _disposed;
Expand Down Expand Up @@ -60,6 +58,13 @@ public void Add(T item)
_array[_count++] = item;
}

public void AddRange(Span<T> items)
{
GuardResize(items.Length);
items.CopyTo(_array.AsSpan(_count, items.Length));
_count += items.Length;
}

public void Clear()
{
_count = 0;
Expand Down Expand Up @@ -100,12 +105,17 @@ public void Insert(int index, T item)
_count++;
}

private void GuardResize()
private void GuardResize(int itemsToAdd = 1)
{
GuardDispose();
if (_count == _capacity)
int newCount = _count + itemsToAdd;
if (newCount > _capacity)
{
int newCapacity = _capacity * 2;
while (newCount > newCapacity)
{
newCapacity *= 2;
}
T[] newArray = _arrayPool.Rent(newCapacity);
_array.CopyTo(newArray, 0);
T[] oldArray = Interlocked.Exchange(ref _array, newArray);
Expand Down Expand Up @@ -198,5 +208,7 @@ public void Dispose()
_disposed = true;
}
}

public Span<T> AsSpan() => _array.AsSpan(0, _count);
}
}
25 changes: 25 additions & 0 deletions src/Nethermind/Nethermind.Core/Collections/ByteArrayPoolList.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
// SPDX-FileCopyrightText: 2022 Demerzel Solutions Limited
// SPDX-License-Identifier: LGPL-3.0-only

using System.Buffers;
using System.Collections.Generic;
using System.IO;

namespace Nethermind.Core.Collections;

public class ByteArrayPoolList : ArrayPoolList<byte>
{
public ByteArrayPoolList(int capacity) : base(capacity)
{
}

public ByteArrayPoolList(int capacity, IEnumerable<byte> enumerable) : base(capacity, enumerable)
{
}

public ByteArrayPoolList(ArrayPool<byte> arrayPool, int capacity) : base(arrayPool, capacity)
{
}

public MemoryStream AsMemoryStream() => new(_array, 0, Count);
}
16 changes: 2 additions & 14 deletions src/Nethermind/Nethermind.Core/Extensions/Bytes.cs
Original file line number Diff line number Diff line change
Expand Up @@ -504,20 +504,8 @@ public static void StreamHex(this byte[] bytes, StreamWriter streamWriter)
}
}

public static string ToHexString(this byte[] bytes, bool withZeroX)
{
return ToHexString(bytes, withZeroX, false, false);
}

public static string ToHexString(this byte[] bytes, bool withZeroX, bool noLeadingZeros)
{
return ToHexString(bytes, withZeroX, noLeadingZeros, false);
}

public static string ToHexString(this byte[] bytes, bool withZeroX, bool noLeadingZeros, bool withEip55Checksum)
{
return ByteArrayToHexViaLookup32(bytes, withZeroX, noLeadingZeros, withEip55Checksum);
}
public static string ToHexString(this byte[] bytes, bool withZeroX, bool noLeadingZeros = false, bool withEip55Checksum = false) =>
ByteArrayToHexViaLookup32(bytes, withZeroX, noLeadingZeros, withEip55Checksum);

private struct StateSmall
{
Expand Down
4 changes: 2 additions & 2 deletions src/Nethermind/Nethermind.Core/Resettables/ResettableList.cs
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@

namespace Nethermind.Core.Resettables;

public class ResettableList<T> : IList<T>
public class ResettableList<T> : IList<T>, IReadOnlyCollection<T>
{
private List<T> _wrapped;
private readonly List<T> _wrapped;
private readonly int _startCapacity;
private readonly int _resetRatio;
private int _currentCapacity;
Expand Down
8 changes: 5 additions & 3 deletions src/Nethermind/Nethermind.Db.Rocks/DbOnTheRocks.cs
Original file line number Diff line number Diff line change
Expand Up @@ -196,9 +196,11 @@ protected internal void UpdateWriteMetrics()
propertyName);
try
{
return (T?)dbConfig.GetType()
.GetProperty(prefixed, BindingFlags.IgnoreCase | BindingFlags.Public | BindingFlags.Instance)?
.GetValue(dbConfig);
Type type = dbConfig.GetType();
PropertyInfo? propertyInfo = type.GetProperty(prefixed, BindingFlags.IgnoreCase | BindingFlags.Public | BindingFlags.Instance);
// if no custom db property default to generic one
propertyInfo ??= type.GetProperty(propertyName, BindingFlags.IgnoreCase | BindingFlags.Public | BindingFlags.Instance);
return (T?)propertyInfo?.GetValue(dbConfig);
}
catch (Exception e)
{
Expand Down
5 changes: 1 addition & 4 deletions src/Nethermind/Nethermind.Db/RocksDbInitializer.cs
Original file line number Diff line number Diff line change
Expand Up @@ -68,9 +68,6 @@ protected async Task InitAllAsync()
await Task.WhenAll(allInitializers);
}

protected static string GetTitleDbName(string dbName)
{
return char.ToUpper(dbName[0]) + dbName.Substring(1);
}
protected static string GetTitleDbName(string dbName) => char.ToUpper(dbName[0]) + dbName.Substring(1);
}
}
19 changes: 10 additions & 9 deletions src/Nethermind/Nethermind.Evm/Tracing/BlockTracerBase.cs
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
using System.Collections.Generic;
using Nethermind.Core;
using Nethermind.Core.Crypto;
using Nethermind.Core.Resettables;
using Nethermind.Int256;

namespace Nethermind.Evm.Tracing
Expand All @@ -16,13 +17,13 @@ public abstract class BlockTracerBase<TTrace, TTracer> : IBlockTracer where TTra

protected BlockTracerBase()
{
TxTraces = new List<TTrace>();
TxTraces = new ResettableList<TTrace>();
}

protected BlockTracerBase(Keccak? txHash)
{
_txHash = txHash;
TxTraces = new List<TTrace>();
TxTraces = new ResettableList<TTrace>();
}

private TTracer? CurrentTxTracer { get; set; }
Expand All @@ -36,7 +37,10 @@ public virtual void ReportReward(Address author, string rewardType, UInt256 rewa
{
}

public abstract void StartNewBlockTrace(Block block);
public virtual void StartNewBlockTrace(Block block)
{
TxTraces.Reset();
}

ITxTracer IBlockTracer.StartNewTxTrace(Transaction? tx)
{
Expand All @@ -58,18 +62,15 @@ void IBlockTracer.EndTxTrace()
}
}

public abstract void EndBlockTrace();
public virtual void EndBlockTrace() { }

protected virtual bool ShouldTraceTx(Transaction? tx)
{
return IsTracingEntireBlock || tx?.Hash == _txHash;
}

protected List<TTrace> TxTraces { get; }
protected ResettableList<TTrace> TxTraces { get; }

public IReadOnlyCollection<TTrace> BuildResult()
{
return TxTraces;
}
public IReadOnlyCollection<TTrace> BuildResult() => TxTraces;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,5 @@ public GethLikeBlockTracer(Keccak txHash, GethTraceOptions options)
protected override GethLikeTxTracer OnStart(Transaction? tx) => new(_options);

protected override GethLikeTxTrace OnEnd(GethLikeTxTracer txTracer) => txTracer.BuildResult();

public override void StartNewBlockTrace(Block block)
{
}

public override void EndBlockTrace()
{
}
}
}
29 changes: 29 additions & 0 deletions src/Nethermind/Nethermind.Evm/Tracing/IBlockTracer.cs
Original file line number Diff line number Diff line change
Expand Up @@ -15,16 +15,45 @@ namespace Nethermind.Evm.Tracing
/// </remarks>
public interface IBlockTracer
{
/// <summary>
/// Is reward state change traced
/// </summary>
/// <remarks>
/// Controls
/// - <see cref="ReportReward"/>
/// </remarks>
bool IsTracingRewards { get; }

/// <summary>
/// Reports rewards for bock.
/// </summary>
/// <param name="author">Author/coinbase for reward.</param>
/// <param name="rewardType">Type of reward.</param>
/// <param name="rewardValue">Value of reward.</param>
/// <remarks>Depends on <see cref="IsTracingRewards"/></remarks>
void ReportReward(Address author, string rewardType, UInt256 rewardValue);

/// <summary>
/// Starts a trace for new block.
/// </summary>
/// <param name="block">Block to be traced.</param>
void StartNewBlockTrace(Block block);

/// <summary>
/// Starts new transaction trace in a block.
/// </summary>
/// <param name="tx">Transaction this trace is started for. Null if it's reward trace.</param>
/// <returns>Returns tracer for transaction.</returns>
ITxTracer StartNewTxTrace(Transaction? tx);

/// <summary>
/// Ends last transaction trace <see cref="StartNewTxTrace"/>.
/// </summary>
void EndTxTrace();

/// <summary>
/// Ends block trace <see cref="StartNewBlockTrace"/>.
/// </summary>
void EndBlockTrace();
}
}
Loading

0 comments on commit 2e8630b

Please sign in to comment.