Hyperledger Blockchain Explorer-Fail to connect before the deadline on Endorser, fail to connect to remote gRPC server

I am trying to set up a Hyperledger Fabric Network with Hyperledger Explorer. I spin up a VM on the digital ocean cloud with ubuntu OS. From there, I spin up 3 orderers node, and 2 peers node. Which result in total of 5 nodes. (I am using RAFT setup).

However, I encounter the error as below when trying to start the hyperledger fabric explorer docker-container images.

Error:

> [email protected] app-start /opt/explorer
> ./start.sh

[2021-02-18T07:32:21.828] [INFO] PgService - SSL to Postgresql disabled
[2021-02-18T07:32:21.829] [INFO] PgService - connecting to Postgresql postgres://hppoc:******@explorerdb.mynetwork.com:5432/fabricexplorer
[2021-02-18T07:32:21.898] [INFO] Platform -  network_config.id  test-network  network_config.profile  ./connection-profile/test-network.json
[2021-02-18T07:32:22.013] [INFO] FabricConfig - config.client.tlsEnable  true
[2021-02-18T07:32:22.013] [INFO] FabricConfig - FabricConfig, this.config.channels  airlinechannel
[2021-02-18T07:32:22.016] [INFO] FabricGateway - enrollUserIdentity: userName : exploreradmin
2021-02-18T07:32:25.221Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Endorser- name: peer1.acme.com, url:grpcs://peer1.acme.com:7051, connected:false, connectAttempted:true
2021-02-18T07:32:25.222Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server peer1.acme.com url:grpcs://peer1.acme.com:7051 timeout:3000
2021-02-18T07:32:25.223Z - info: [NetworkConfig]: buildPeer - Unable to connect to the endorser peer1.acme.com due to Error: Failed to connect before the deadline on Endorser- name: peer1.acme.com, url:grpcs://peer1.acme.com:7051, connected:false, connectAttempted:true
2021-02-18T07:32:28.250Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Discoverer- name: peer1.acme.com, url:grpcs://peer1.acme.com:7051, connected:false, connectAttempted:true
2021-02-18T07:32:28.250Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server peer1.acme.com url:grpcs://peer1.acme.com:7051 timeout:3000
2021-02-18T07:32:28.250Z - error: [ServiceEndpoint]: ServiceEndpoint grpcs://peer1.acme.com:7051 reset connection failed :: Error: Failed to connect before the deadline on Discoverer- name: peer1.acme.com, url:grpcs://peer1.acme.com:7051, connected:false, connectAttempted:true
2021-02-18T07:32:28.251Z - error: [DiscoveryService]: send[airlinechannel] - no discovery results
[2021-02-18T07:32:28.251] [ERROR] FabricClient - Error: DiscoveryService has failed to return results
    at DiscoveryService.send (/opt/explorer/node_modules/fabric-network/node_modules/fabric-common/lib/DiscoveryService.js:370:10)
    at process._tickCallback (internal/process/next_tick.js:68:7)
[2021-02-18T07:32:28.252] [INFO] FabricClient - ********* call to initializeDetachClient **********
[2021-02-18T07:32:28.253] [INFO] FabricClient - initializeDetachClient, network config)  { name: 'test-network',
  version: '1.0.0',
  client:
   { tlsEnable: true,
     adminCredential: { id: 'exploreradmin', password: 'exploreradminpw' },
     enableAuthentication: true,
     organization: 'AcmeMSP',
     connection: { timeout: [Object] } },
  channels: { airlinechannel: { peers: [Object] } },
  organizations:
   { AcmeMSP:
      { mspid: 'AcmeMSP',
        adminPrivateKey: [Object],
        peers: [Array],
        signedCert: [Object] } },
  peers:
   { 'peer1.acme.com': { tlsCACerts: [Object], url: 'grpcs://peer1.acme.com:7051' } } }
[2021-02-18T07:32:28.253] [INFO] FabricClient - ************************************* initializeDetachClient *************************************************
[2021-02-18T07:32:28.254] [INFO] FabricClient - Error : Failed to connect client peer, please check the configuration and peer status
[2021-02-18T07:32:28.254] [INFO] FabricClient - Info :  Explorer will continue working with only DB data
[2021-02-18T07:32:28.254] [INFO] FabricClient - ************************************** initializeDetachClient ************************************************
[2021-02-18T07:32:28.259] [INFO] Platform - initializeListener, network_id, network_client  test-network { name: 'test-network',
  version: '1.0.0',
  client:
   { tlsEnable: true,
     adminCredential: { id: 'exploreradmin', password: 'exploreradminpw' },
     enableAuthentication: true,
     organization: 'AcmeMSP',
     connection: { timeout: [Object] } },
  channels: { airlinechannel: { peers: [Object] } },
  organizations:
   { AcmeMSP:
      { mspid: 'AcmeMSP',
        adminPrivateKey: [Object],
        peers: [Array],
        signedCert: [Object] } },
  peers:
   { 'peer1.acme.com': { tlsCACerts: [Object], url: 'grpcs://peer1.acme.com:7051' } } }
[2021-02-18T07:32:28.260] [INFO] main - Please open web browser to access :http://localhost:8080/
[2021-02-18T07:32:28.261] [INFO] main - pid is 20
[2021-02-18T07:32:28.263] [ERROR] main - <<<<<<<<<<<<<<<<<<<<<<<<<< Explorer Error >>>>>>>>>>>>>>>>>>>>>
[2021-02-18T07:32:28.263] [ERROR] main - Error :  [ 'Default client peer is down and no channel details available database' ]
[2021-02-18T07:32:30.264] [INFO] main - Received kill signal, shutting down gracefully
[2021-02-18T07:32:30.266] [INFO] Platform - <<<<<<<<<<<<<<<<<<<<<<<<<< Closing explorer  >>>>>>>>>>>>>>>>>>>>>
[2021-02-18T07:32:30.266] [INFO] main - Closed out connections

Version Detail

  1. Hyperledger Fabric: 2.3.1
  2. Hyperledger Explorer: v1.1.1 (latest tag)

Part 1: Docker Container Setup

a) Docker PS

CONTAINER ID   IMAGE                                                                                                                                                                  COMMAND                  CREATED          STATUS                   PORTS                                            NAMES
9e8768914140   hyperledger/explorer:latest                                                                                                                                            "docker-entrypoint.s…"   3 minutes ago    Up 3 minutes             0.0.0.0:8080->8080/tcp                           explorer.mynetwork.com
903c8c4a4326   hyperledger/explorer-db:latest                                                                                                                                         "docker-entrypoint.s…"   3 minutes ago    Up 3 minutes (healthy)   5432/tcp                                         explorerdb.mynetwork.com
9fed267ae9b1   dev-peer1.budget.com-gocc1.1.0-1.0-2593f1a95def85c64fdfed35e3d3b1051b92ed91549cfe789292ec5475d7db86-e014c6d933da036b6c79b53d29a800d8a6886e374ceb435a30642b885735f8f5   "chaincode -peer.add…"   21 minutes ago   Up 21 minutes                                                             dev-peer1.budget.com-gocc1.1.0-1.0-2593f1a95def85c64fdfed35e3d3b1051b92ed91549cfe789292ec5475d7db86
12407497fa4c   dev-peer1.acme.com-gocc1.1.0-1.0-2593f1a95def85c64fdfed35e3d3b1051b92ed91549cfe789292ec5475d7db86-a3b6caa9293bb826d231b1a31cc47437e58764abe5169a20ed0ee43f25c16b24     "chaincode -peer.add…"   21 minutes ago   Up 21 minutes                                                             dev-peer1.acme.com-gocc1.1.0-1.0-2593f1a95def85c64fdfed35e3d3b1051b92ed91549cfe789292ec5475d7db86
0448e71f68e1   hyperledger/fabric-peer:latest                                                                                                                                         "peer node start"        22 minutes ago   Up 22 minutes            0.0.0.0:8051->7051/tcp, 0.0.0.0:8052->7052/tcp   peer1.budget.com
3266ec37b360   hyperledger/fabric-peer:latest                                                                                                                                         "peer node start"        22 minutes ago   Up 22 minutes            0.0.0.0:7051-7052->7051-7052/tcp                 peer1.acme.com
47ebe9ad79d1   hyperledger/fabric-orderer:latest                                                                                                                                      "orderer"                22 minutes ago   Up 22 minutes            0.0.0.0:8050->7050/tcp                           orderer2.acme.com
09a5f771f47f   hyperledger/fabric-tools:latest                                                                                                                                        "/bin/bash"              22 minutes ago   Up 22 minutes                                                             tools
e132bb01ce22   hyperledger/fabric-orderer:latest                                                                                                                                      "orderer"                22 minutes ago   Up 22 minutes            0.0.0.0:9050->7050/tcp                           orderer3.acme.com
3c61b0316385   hyperledger/fabric-orderer:latest                                                                                                                                      "orderer"                22 minutes ago   Up 22 minutes

b) I use 3 docker-compose files for my configuration setting.

$ docker-compose -f ./config/docker-compose-base.yaml -f ./tls/docker-compose-tls.yaml -f ./raft/docker-compose-raft.yaml up -d

  1. docker-compose-base.yaml

https://gist.github.com/Skyquek/03d1ffad5643d67d8da5b268a4814a7d

  1. docker-compose-tls.yaml

https://gist.github.com/Skyquek/b3b314cb2152ab541e822f72c60a2cbd

  1. docker-compose-raft.yaml

https://gist.github.com/Skyquek/7f8ec2d4d1876283f4a9444675971be8

c) Core.yaml

  1. acme core.yaml

https://gist.github.com/Skyquek/8cdcbc4ee3d53a2277b1c34bb2fca704

Part 2: Blockchain Explorer Setting

1. connection-profile.json

{
        "name": "test-network",
        "version": "1.0.0",
        "client": {
                "tlsEnable": true,
                "adminCredential": {
                        "id": "exploreradmin",
                        "password": "exploreradminpw"
                },
                "enableAuthentication": true,
                "organization": "AcmeMSP",
                "connection": {
                        "timeout": {
                                "peer": {
                                        "endorser": "300"
                                },
                                "orderer": "300"
                        }
                }
        },
        "channels": {
                "airlinechannel": {
                        "peers": {
                                "peer1.acme.com": {}
                        }
                }
        },
        "organizations": {
                "AcmeMSP": {
                        "mspid": "AcmeMSP",
                        "adminPrivateKey": {
                                "path": "/tmp/crypto/peerOrganizations/acme.com/users/[email protected]/msp/keystore/priv_sk"
                        },
                        "peers": ["peer1.acme.com"],
                        "signedCert": {
                                "path": "/tmp/crypto/peerOrganizations/acme.com/users/[email protected]/msp/signcerts/[email protected]"
                        }
                }
        },
        "peers": {
                "peer1.acme.com": {
                        "tlsCACerts": {
                                "path": "/tmp/crypto/peerOrganizations/acme.com/tlsca/tlsca.acme.com-cert.pem"
                        },
                        "url": "grpcs://peer1.acme.com:7051"
                }
        }
}

2. docker-compose.yaml


# SPDX-License-Identifier: Apache-2.0
version: '2.1'

volumes:
  pgdata:
  walletstore:

networks:
  mynetwork.com:
    external:
      name: acloudfan_airline

services:

  explorerdb.mynetwork.com:
    image: hyperledger/explorer-db:latest
    container_name: explorerdb.mynetwork.com
    hostname: explorerdb.mynetwork.com
    environment:
      - DATABASE_DATABASE=fabricexplorer
      - DATABASE_USERNAME=hppoc
      - DATABASE_PASSWORD=password
    healthcheck:
      test: "pg_isready -h localhost -p 5432 -q -U postgres"
      interval: 30s
      timeout: 10s
      retries: 5
    volumes:
      - pgdata:/var/lib/postgresql/data
    networks:
      - mynetwork.com

  explorer.mynetwork.com:
    image: hyperledger/explorer:latest
    container_name: explorer.mynetwork.com
    hostname: explorer.mynetwork.com
    environment:
      - DATABASE_HOST=explorerdb.mynetwork.com
      - DATABASE_DATABASE=fabricexplorer
      - DATABASE_USERNAME=hppoc
      - DATABASE_PASSWD=password
      - LOG_LEVEL_APP=debug
      - LOG_LEVEL_DB=debug
      - LOG_LEVEL_CONSOLE=info
      - LOG_CONSOLE_STDOUT=true
      - DISCOVERY_AS_LOCALHOST=true
    volumes:
      - ./config.json:/opt/explorer/app/platform/fabric/config.json
      - ./connection-profile:/opt/explorer/app/platform/fabric/connection-profile
      - ./organizations:/tmp/crypto
      - walletstore:/opt/explorer/wallet
    ports:
      - 8080:8080
    depends_on:
      explorerdb.mynetwork.com:
        condition: service_healthy
    networks:
      - mynetwork.com

Solution that I tried

1. Change the tlscs certs path

As stated in
Hyperledger Fabric 2.0.1: Error: Failed to connect before the deadline on Discoverer- name:

The problem most likely is due to the error from the path. But the problem still persists.

2. Change the env variable DISCOVERY_AS_LOCALHOST=true to false

Some of them mention that this will fixed the problem. But I can’t seem to fix it this way.

3. Tried with hyperledger fabric 2.0 test-network

I tried to run with fabric sample test-network and its run perfectly fine.

4. docker exec -it sh to the explorer.mynetwork.com to ping the peer

The ping can run perfectly fine.

/opt/explorer # ping peer1.acme.com:7051
PING peer1.acme.com:7051 (172.23.0.6): 56 data bytes
64 bytes from 172.23.0.6: seq=0 ttl=64 time=0.138 ms
64 bytes from 172.23.0.6: seq=1 ttl=64 time=0.087 ms
64 bytes from 172.23.0.6: seq=2 ttl=64 time=0.090 ms
64 bytes from 172.23.0.6: seq=3 ttl=64 time=0.089 ms
64 bytes from 172.23.0.6: seq=4 ttl=64 time=0.101 ms
64 bytes from 172.23.0.6: seq=5 ttl=64 time=0.088 ms
^C
--- peer1.acme.com:7051 ping statistics ---
6 packets transmitted, 6 packets received, 0% packet loss
round-trip min/avg/max = 0.087/0.098/0.138 ms

Its keep on showing that the peer is down or the connection to the peer is fail

I am struggling with this error for few days now. Hope someone can help me to identify the problem. Thank you very much.

Source: Docker Questions

LEAVE A COMMENT