High Latency and Endorse/Commit Deadline Exceeded errors

[MartinStuwe]

I'm currently benchmarking our HF 2.5 network with Caliper 0.5.

We have 4 peer organizations, each has 1 peer, each peer is an anchor peer and they use LevelDB as state database.
Further we have 3 orderer organizations, each with 1 orderer.
They are joined to one channel, where the orderers use Raft.
The endorsement policy is set to a majority:

    Policies: &ApplicationDefaultPolicies
        LifecycleEndorsement:
            Type: ImplicitMeta
            Rule: "MAJORITY Endorsement"
        Endorsement:
            Type: ImplicitMeta
            Rule: "MAJORITY Endorsement"
        Readers:
            Type: ImplicitMeta
            Rule: "ANY Readers"
        Writers:
            Type: ImplicitMeta
            Rule: "ANY Writers"
        Admins:
            Type: ImplicitMeta
            Rule: "MAJORITY Admins"

    SampleAppChannelEtcdRaft:
        <<: *ChannelDefaults
        Orderer:
            <<: *OrdererDefaults
            OrdererType: etcdraft
            Organizations:
                - <<: *orderer1
                  Policies:
                      <<: *orderer1Policies
                      Admins:
                          Type: Signature
                          Rule: "OR('orderer1.kibiosense.com.member')"
                - <<: *orderer2
                  Policies:
                      <<: *orderer2Policies
                      Admins:
                          Type: Signature
                          Rule: "OR('orderer2.kibiosense.com.member')"
                - <<: *orderer3
                  Policies:
                      <<: *orderer3Policies
                      Admins:
                          Type: Signature
                          Rule: "OR('orderer3.kibiosense.com.member')"
        Application:
            <<: *ApplicationDefaults
            Organizations:
                - <<: *fishsupplier1
                  Policies:
                      <<: *fishsupplier1Policies
                - <<: *deliverer1
                  Policies:
                      <<: *deliverer1Policies
                - <<: *storage1
                  Policies:
                      <<: *storage1Policies
                - <<: *supermarket1
                  Policies:
                      <<: *supermarket1Policies
                

Channel/Orderer settings

        "values": {
          "BatchSize": {
            "mod_policy": "Admins",
            "value": {
              "absolute_max_bytes": 51380224,
              "max_message_count": 10,
              "preferred_max_bytes": 2097152
            },
            "version": "1"
          },
          "BatchTimeout": {
            "mod_policy": "Admins",
            "value": {
              "timeout": "1.0s"
            },

They are deployed on 7 Debian VMs with 2 vCPUs, 4 peer VMs and 3 orderer VMs.

Our chaincodes are written in Go.

Now during the benchmark, where we target peers through the gateway functionality, we often encounter high latency, resulting in the following errors:

I tried to reduce the maximum message amount from 500 to 10,
reduce the batch timeout from 2.5s to 1.0s and to add more peers s.t. higher block count peers would take over as endorsing peers.

But we still encounter such issues.

It's independent of the chaincode method, every method sometimes works with about 200 TPS and on other occasions results in the unfinished transactions seen in the second image and in most cases end up in deadline exceeded errors seen in the first image.

The first and second image are from the same workload.

Did someone encounter such problems and knows how to resolve this?