failed to invoke chaincode, error: tls: failed to find any PEM data in certificate input failed to load client certificate

[GeorgePals]

Hello!

Please, read the edit at the end of the post.

I'm trying to bring up some external chaincodes. However, I'm having some difficulties.
While the external CCs work without TLS, when I'm trying to enable it, I'm getting the title's error.

failed to invoke chaincode, error: tls: failed to find any PEM data in certificate input failed to load client certificate

The connection.json file (I removed some parts from the certs to reduce their size):

{"address": "cc-some_chaincode-some_org:4441", "dial_timout": "10s", "tls_required": true, "client_auth_required": true, "client_cert": "-----BEGIN CERTIFICATE-----MIICxzCCAm6gAwIBAgIUDhUYNeQRdqLJ5nP20lbGpK7kM9QwCgYIKoZIzj0EAwIwczELMAkGA1UEBhMCR1IxDzANBgNVBAg...(reduced)...../Oq7cHiKZhIuw1e52Y97OZVRKv7M/A==-----END CERTIFICATE-----", "client_key": "-----BEGIN PRIVATE KEY-----MIGHAgE..(reduced)...yWkNfdRR8/h-----END PRIVATE KEY-----", "root_cert": "-----BEGIN CERTIFICATE-----MIICLDCCAdOgAwIBAgIUFGvZTPpBXuFEmoynKWm49kQa+RwwCgYIKoZIzj0EAwIwczELMAkGA1UEBhMCR1IxDzANBgNVBAgTBkdyZW...(reduced)....DaGKADp98gPAJEedpsJJGPTLwRO+AwBkErbw6xp2tSY=-----END CERTIFICATE-----"}

Here you can see the error log from the peer:

2021-05-31 18:47:01.417 UTC [chaincode.externalbuilder] PackageMetadata -> DEBU 3114 Walking package release dir '/var/hyperledger/production/externalbuilder/builds/some_chaincode_34.0-f2e1a8de0f2d59fdfbaeb1e61c81b940878512cf15d67ebb8d8891f81d4cdca2/release'
2021-05-31 18:47:01.417 UTC [chaincode.externalbuilder] func1 -> DEBU 3115 Adding file '/var/hyperledger/production/externalbuilder/builds/some_chaincode_34.0-f2e1a8de0f2d59fdfbaeb1e61c81b940878512cf15d67ebb8d8891f81d4cdca2/release' to tar with header name 'META-INF/'
2021-05-31 18:47:01.417 UTC [chaincode.externalbuilder] func1 -> DEBU 3116 Adding file '/var/hyperledger/production/externalbuilder/builds/some_chaincode_34.0-f2e1a8de0f2d59fdfbaeb1e61c81b940878512cf15d67ebb8d8891f81d4cdca2/release/chaincode' to tar with header name 'META-INF/chaincode/'
2021-05-31 18:47:01.417 UTC [chaincode.externalbuilder] func1 -> DEBU 3117 Adding file '/var/hyperledger/production/externalbuilder/builds/some_chaincode_34.0-f2e1a8de0f2d59fdfbaeb1e61c81b940878512cf15d67ebb8d8891f81d4cdca2/release/chaincode/server' to tar with header name 'META-INF/chaincode/server/'
2021-05-31 18:47:01.417 UTC [extcc] Stream -> DEBU 3118 Starting external chaincode connection: some_chaincode_34.0:f2e1a8de0f2d59fdfbaeb1e61c81b940878512cf15d67ebb8d8891f81d4cdca2
2021-05-31 18:47:01.417 UTC [chaincode.externalbuilder] func1 -> DEBU 3119 Adding file '/var/hyperledger/production/externalbuilder/builds/some_chaincode_34.0-f2e1a8de0f2d59fdfbaeb1e61c81b940878512cf15d67ebb8d8891f81d4cdca2/release/chaincode/server/connection.json' to tar with header name 'META-INF/chaincode/server/connection.json'
2021-05-31 18:47:01.417 UTC [chaincode] Launch -> DEBU 311a stopping due to error while launching: tls: failed to find any PEM data in certificate input
failed to load client certificate
error creating grpc client to some_chaincode_34.0:f2e1a8de0f2d59fdfbaeb1e61c81b940878512cf15d67ebb8d8891f81d4cdca2
error cannot create connection for some_chaincode_34.0:f2e1a8de0f2d59fdfbaeb1e61c81b940878512cf15d67ebb8d8891f81d4cdca2
connection to some_chaincode_34.0:f2e1a8de0f2d59fdfbaeb1e61c81b940878512cf15d67ebb8d8891f81d4cdca2 failed
2021-05-31 18:47:01.417 UTC [chaincode] Launch -> DEBU 311b launch complete
2021-05-31 18:47:01.417 UTC [chaincode] Deregister -> DEBU 311c deregister handler: some_chaincode_34.0:f2e1a8de0f2d59fdfbaeb1e61c81b940878512cf15d67ebb8d8891f81d4cdca2
2021-05-31 18:47:01.417 UTC [lifecycle] Work -> WARN 311d could not launch chaincode 'some_chaincode_34.0:f2e1a8de0f2d59fdfbaeb1e61c81b940878512cf15d67ebb8d8891f81d4cdca2': connection to some_chaincode_34.0:f2e1a8de0f2d59fdfbaeb1e61c81b940878512cf15d67ebb8d8891f81d4cdca2 failed: error cannot create connection for some_chaincode_34.0:f2e1a8de0f2d59fdfbaeb1e61c81b940878512cf15d67ebb8d8891f81d4cdca2: error creating grpc client to some_chaincode_34.0:f2e1a8de0f2d59fdfbaeb1e61c81b940878512cf15d67ebb8d8891f81d4cdca2: failed to load client certificate: tls: failed to find any PEM data in certificate input
2021-05-31 18:47:01.417 UTC [chaincode.externalbuilder] func1 -> DEBU 311e Adding file '/var/hyperledger/production/externalbuilder/builds/some_chaincode_34.0-f2e1a8de0f2d59fdfbaeb1e61c81b940878512cf15d67ebb8d8891f81d4cdca2/release/chaincode/server/tls' to tar with header name 'META-INF/chaincode/server/tls/'
2021-05-31 18:47:01.417 UTC [chaincode.externalbuilder] func1 -> DEBU 311f Adding file '/var/hyperledger/production/externalbuilder/builds/some_chaincode_34.0-f2e1a8de0f2d59fdfbaeb1e61c81b940878512cf15d67ebb8d8891f81d4cdca2/release/chaincode/server/tls/client.crt' to tar with header name 'META-INF/chaincode/server/tls/client.crt'
2021-05-31 18:47:01.417 UTC [chaincode.externalbuilder] func1 -> DEBU 3120 Adding file '/var/hyperledger/production/externalbuilder/builds/some_chaincode_34.0-f2e1a8de0f2d59fdfbaeb1e61c81b940878512cf15d67ebb8d8891f81d4cdca2/release/chaincode/server/tls/client.key' to tar with header name 'META-INF/chaincode/server/tls/client.key'
2021-05-31 18:47:01.417 UTC [chaincode.externalbuilder] func1 -> DEBU 3121 Adding file '/var/hyperledger/production/externalbuilder/builds/some_chaincode_34.0-f2e1a8de0f2d59fdfbaeb1e61c81b940878512cf15d67ebb8d8891f81d4cdca2/release/chaincode/server/tls/root.crt' to tar with header name 'META-INF/chaincode/server/tls/root.crt'

and

2021-05-31 18:47:07.663 UTC [chaincode] Launch -> DEBU 3206 stopping due to error while launching: tls: failed to find any PEM data in certificate input
failed to load client certificate
error creating grpc client to some_chaincode_34.0:f2e1a8de0f2d59fdfbaeb1e61c81b940878512cf15d67ebb8d8891f81d4cdca2
error cannot create connection for some_chaincode_34.0:f2e1a8de0f2d59fdfbaeb1e61c81b940878512cf15d67ebb8d8891f81d4cdca2
connection to some_chaincode_34.0:f2e1a8de0f2d59fdfbaeb1e61c81b940878512cf15d67ebb8d8891f81d4cdca2 failed
2021-05-31 18:47:07.663 UTC [chaincode] Launch -> DEBU 3207 launch complete
2021-05-31 18:47:07.663 UTC [chaincode] Deregister -> DEBU 3208 deregister handler: some_chaincode_34.0:f2e1a8de0f2d59fdfbaeb1e61c81b940878512cf15d67ebb8d8891f81d4cdca2
2021-05-31 18:47:07.664 UTC [endorser] callChaincode -> INFO 3209 finished chaincode: some_chaincode duration: 0ms channel=basechannel txID=72d6f434
2021-05-31 18:47:07.664 UTC [endorser] SimulateProposal -> ERRO 320a failed to invoke chaincode some_chaincode, error: tls: failed to find any PEM data in certificate input

The Builders
Build:

#!/bin/sh

set -euo pipefail

CHAINCODE_SOURCE_DIR=$1
CHAINCODE_METADATA_DIR="$2"
BUILD_OUTPUT_DIR=$3

#external chaincodes expect connection.json file in the chaincode package
if [ ! -f "$CHAINCODE_SOURCE_DIR/connection.json" ]; then
    >&2 echo "$CHAINCODE_SOURCE_DIR/connection.json not found"
    exit 1
fi

>&2 echo "$(cat $CHAINCODE_SOURCE_DIR/connection.json)"
>&2 echo "$(cat $BUILD_OUTPUT_DIR/connection.json)"

#simply copy the endpoint information to specified output location
mkdir -p $BUILD_OUTPUT_DIR/chaincode/
echo "$(cat $CHAINCODE_SOURCE_DIR/connection.json)" > $BUILD_OUTPUT_DIR/chaincode/connection.json

if [ -d "$CHAINCODE_SOURCE_DIR/metadata" ]; then
    echo "$(cat $CHAINCODE_SOURCE_DIR/metadata)" > $BUILD_OUTPUT_DIR/chaincode/metadata
fi

exit 0

Detect:

#!/bin/sh

CHAINCODE_METADATA_DIR="$2"

TYPE=$(tr -d '\n' < "$CHAINCODE_METADATA_DIR/metadata.json" | awk -F':' '{ for (i = 1; i < NF; i++){ if ($i~/type/) { print $(i+1); break }}}'| cut -d\" -f2)
if [ "$TYPE" = "external" ]; then
    exit 0
fi

exit 1

Release:

#!/bin/sh

set -euo pipefail

BUILD_OUTPUT_DIR="$1"
RELEASE_OUTPUT_DIR="$2"

if [ -d "$BUILD_OUTPUT_DIR/META-INF" ] ; then
   cp -a "$BUILD_OUTPUT_DIR/META-INF/"* "$RELEASE_OUTPUT_DIR/"
fi

#external chaincodes expect artifacts to be placed under "$RELEASE_OUTPUT_DIR"/chaincode/server
if [ -f $BUILD_OUTPUT_DIR/chaincode/connection.json ]; then
   mkdir -p "$RELEASE_OUTPUT_DIR"/chaincode/server/tls
   echo "$(cat $BUILD_OUTPUT_DIR/chaincode/connection.json)" > "$RELEASE_OUTPUT_DIR"/chaincode/server/connection.json
   #if tls_required is true, copy TLS files (using above example, the fully qualified path for these fils would be "$RELEASE_OUTPUT_DIR"/chaincode/server/tls)

   echo "$(tr -d '\n' < "$BUILD_OUTPUT_DIR/chaincode/connection.json" | awk -F':' '{ for (i = 1; i < NF; i++){ if ($i~/client_cert/) { print $(i+1); break }}}'| cut -d\" -f2)" > "$RELEASE_OUTPUT_DIR"/chaincode/server/tls/client.crt
   echo "$(tr -d '\n' < "$BUILD_OUTPUT_DIR/chaincode/connection.json" | awk -F':' '{ for (i = 1; i < NF; i++){ if ($i~/client_key/) { print $(i+1); break }}}'| cut -d\" -f2)" > "$RELEASE_OUTPUT_DIR"/chaincode/server/tls/client.key
   echo "$(tr -d '\n' < "$BUILD_OUTPUT_DIR/chaincode/connection.json" | awk -F':' '{ for (i = 1; i < NF; i++){ if ($i~/root_cert/) { print $(i+1); break }}}'| cut -d\" -f2)" > "$RELEASE_OUTPUT_DIR"/chaincode/server/tls/root.crt

   exit 0
fi

exit 1

Run:

#!/bin/sh

BUILD_OUTPUT_DIR="$1"
RUN_METADATA_DIR="$2"

# setup the environment expected by the go chaincode shim
export CORE_CHAINCODE_ID_NAME=$(tr -d '\n' < "$RUN_METADATA_DIR/chaincode.json" | awk -F':' '{ for (i = 1; i < NF; i++){ if ($i~/chaincode_id/) { print $(i+1); break }}}'| cut -d\" -f2)
export CORE_PEER_TLS_ENABLED="true"
export CORE_TLS_CLIENT_CERT_FILE="$RUN_METADATA_DIR/client.crt"
export CORE_TLS_CLIENT_KEY_FILE="$RUN_METADATA_DIR/client.key"
export CORE_PEER_TLS_ROOTCERT_FILE="$RUN_METADATA_DIR/root.crt"
export CORE_PEER_LOCALMSPID=$(tr -d '\n' < "$RUN_METADATA_DIR/chaincode.json" | awk -F':' '{ for (i = 1; i < NF; i++){ if ($i~/mspid/) { print $(i+1); break }}}'| cut -d\" -f2)

# populate the key and certificate material used by the go chaincode shim
echo "$(tr -d '\n' < "$RUN_METADATA_DIR/chaincode.json" | awk -F':' '{ for (i = 1; i < NF; i++){ if ($i~/client_cert/) { print $(i+1); break }}}'| cut -d\" -f2)" | CORE_TLS_CLIENT_CERT_FILE
echo "$(tr -d '\n' < "$RUN_METADATA_DIR/chaincode.json" | awk -F':' '{ for (i = 1; i < NF; i++){ if ($i~/client_key/) { print $(i+1); break }}}'| cut -d\" -f2)" | CORE_TLS_CLIENT_KEY_FILE"
echo "$(tr -d '\n' < "$RUN_METADATA_DIR/chaincode.json" | awk -F':' '{ for (i = 1; i < NF; i++){ if ($i~/root_cert/) { print $(i+1); break }}}'| cut -d\" -f2)" | CORE_PEER_TLS_ROOTCERT_FILE"
if [ -z echo "$(tr -d '\n' < "$RUN_METADATA_DIR/chaincode.json" | awk -F':' '{ for (i = 1; i < NF; i++){ if ($i~/client_cert/) { print $(i+1); break }}}'| cut -d\" -f2)" ]; then
    export CORE_PEER_TLS_ENABLED="false"
fi

# exec the chaincode to replace the script with the chaincode process
exec "$BUILD_OUTPUT_DIR/chaincode" -peer.address=$(tr -d '\n' < "$ARTIFACTS/chaincode.json" | awk -F':' '{ for (i = 1; i < NF; i++){ if ($i~/peer_address/) { print $(i+1); break }}}'| cut -d\" -f2)

Thanks! Any suggestions will be much appreciated!

Edit: After some time, I "achieved" to have a different error message:

2021-05-31 21:29:55.707 UTC [endorser] ProcessProposal -> WARN 40db Failed to invoke chaincode channel=basechannel chaincode=some_chaincode error="error in simulation: failed to execute transaction 77a7014fb510c5a2de23bf04f3d93e90935160c466544b777a4edaea1552691a: could not launch chaincode some_chaincode_11.0:90c917ef6cd6e82e8cb04b216b8fce810bd5332f9f990450638c839163f11a74: connection to some_chaincode_11.0:90c917ef6cd6e82e8cb04b216b8fce810bd5332f9f990450638c839163f11a74 failed: error cannot create connection for some_chaincode_11.0:90c917ef6cd6e82e8cb04b216b8fce810bd5332f9f990450638c839163f11a74: error creating grpc connection to cc-some_chaincode-some_org:4441: failed to create new connection: context deadline exceeded"

2021-05-31 21:07:22.595 UTC [grpc] InfoDepth -> DEBU 220c [core]pickfirstBalancer: UpdateSubConnState: 0xc003763830, {CONNECTING <nil>}
2021-05-31 21:07:22.595 UTC [grpc] InfoDepth -> DEBU 220d [core]Channel Connectivity change to CONNECTING
2021-05-31 21:07:22.596 UTC [comm.tls] ClientHandshake -> ERRO 220e Client TLS handshake failed after 650.553µs with error: tls: first record does not look like a TLS handshake remoteaddress=10.152.183.111:4441
2021-05-31 21:07:22.597 UTC [grpc] WarningDepth -> DEBU 220f [core]grpc: addrConn.createTransport failed to connect to {cc-some_chaincode-some_org:4441 cc-some_chaincode-some_org:4441 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake". Reconnecting...
2021-05-31 21:07:22.597 UTC [grpc] InfoDepth -> DEBU 2210 [core]Subchannel Connectivity change to TRANSIENT_FAILURE

[sownak]

@GeorgePals I was having same error and as there is hardly any documentation on how to create connection.json or related items for TLS enabled chaincode, I also spent many days and google searches to find a solution. I was finally able to run the chaincode with tls. Here are my observations:

• I have used the default detect/release/build scripts from fabric samples
• I did not use a run script
• The TLS files are read directly from connection.json
• Hence, the certs in connection.json must contain \n characters

In your updated error message cc-some_chaincode-some_org:4441: failed to create new connection means that the chaincode server is not running at this URL or the peer is not able to reach this URL