Strato_bloc_1 & strato_strato_1 is unhealthy


#1

strato_bloc_1 & strato_strato_1 is unhealthy

It was in operation for half a year but stopped suddenly.
Something like this has happened before, but I solved it by rebooting.
It did not solve this time.
Does anybody have a solution for this?

Environment:

  • ubuntu 18.04.1 LTS (Bionic Beaver)
  • Docker version 18.06.0-ce, build 0ffa825
  • STRATO 4.0.0

$ docker ps -a

CONTAINER ID        IMAGE                                                                COMMAND                  CREATED             STATUS                     PORTS                                                          NAMES
31e9ece61074        registry-aws.blockapps.net:5000/blockapps-repo/nginx:4.0.0           "/docker-run.sh"         8 months ago        Up 6 minutes               0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                       strato_nginx_1
9e972c1ab7a6        registry-aws.blockapps.net:5000/blockapps-repo/smd:4.0.0             "sh /usr/src/app/doc…"   8 months ago        Up 6 minutes               3002/tcp                                                       strato_smd_1
5d09eb427442        registry-aws.blockapps.net:5000/blockapps-repo/dappstore:4.0.0       "sh /usr/src/app/doc…"   8 months ago        Up 6 minutes               3000/tcp                                                       strato_dappstore_1
f5d61579488f        registry-aws.blockapps.net:5000/blockapps-repo/apex:4.0.0            "sh /usr/src/app/doc…"   8 months ago        Up 5 minutes               3001/tcp                                                       strato_apex_1
78fc8b31818d        registry-aws.blockapps.net:5000/blockapps-repo/postgrest:4.0.0       "sh /doit.sh"            8 months ago        Up 6 minutes               3001/tcp                                                       strato_postgrest_1
7a56805190fb        registry-aws.blockapps.net:5000/blockapps-repo/bloc:4.0.0            "/usr/bin/bloc/doit.…"   8 months ago        Up 6 minutes (unhealthy)   8000/tcp, 8002/tcp                                             strato_bloc_1
0cdfc2d78e7c        registry-aws.blockapps.net:5000/blockapps-repo/strato:4.0.0          "/var/lib/doit.sh"       8 months ago        Up 6 minutes (unhealthy)   3000/tcp, 0.0.0.0:30303->30303/tcp, 0.0.0.0:30303->30303/udp   strato_strato_1
3a41d088be73        registry-aws.blockapps.net:5000/blockapps-repo/vault-wrapper:4.0.0   "/doit.sh"               8 months ago        Up 5 minutes (healthy)     8000/tcp                                                       strato_vault-wrapper_1
d2c07b2fa219        registry-aws.blockapps.net:5000/blockapps-repo/docs:4.0.0            "sh /usr/share/nginx…"   8 months ago        Up 5 minutes               8080/tcp                                                       strato_docs_1
52571ffcda9f        wurstmeister/kafka:1.1.0                                             "start-kafka.sh"         8 months ago        Up 5 minutes                                                                              strato_kafka_1
52d298f85900        postgres:9.6                                                         "docker-entrypoint.s…"   8 months ago        Up 5 minutes               5432/tcp                                                       strato_postgres_1
0faaa6391749        wurstmeister/zookeeper:3.4.6                                         "/bin/sh -c '/usr/sb…"   8 months ago        Up 5 minutes               22/tcp, 2181/tcp, 2888/tcp, 3888/tcp                           strato_zookeeper_1
8dc537e3ab5f        redis:3.2                                                            "docker-entrypoint.s…"   8 months ago        Up 5 minutes               6379/tcp                                                       strato_redis_1

$ docker logs strato_bloc_1

Environment variables:
slipstream:
--pghost=$postgres_host=postgres
--pgport=$postgres_port=5432
--pguser=$postgres_user=postgres
--password=$postgres_password=api
--database=$postgres_slipstream_db=cirrus
--stratourl=$stratoRoot=http://strato:3000/eth/v1.2
--vaultwrapperurl=$vaultWrapperRoot=http://vault-wrapper:8000/strato/v2.3
--kafkahost=$kafkaHostkafka
--kafkaport=9092

strato-server:
no vars/flags set

bloc:
stratoHost=strato:3000
vaultWrapperHost=vault-wrapper:8000
--stratourl=$stratoRoot=http://strato:3000/eth/v1.2
--vaultwrapperurl=$vaultWrapperRoot=http://vault-wrapper:8000/strato/v2.3
--pghost=$postgres_host=postgres
--pgport=$postgres_port=5432
--pguser=$postgres_user=postgres
--password=$postgres_password=api
--loglevel=$loglevel=4

Generating locales (this might take a while)...
  en_US.UTF-8... done
Generation complete.
Waiting for STRATO to be available...
STRATO is available
Waiting for Vault Wrapper to be available...
Vault Wrapper is available
Waiting for postgres to be available...
Slipstream is waiting for bloc to come up...
Bloc is up - running slipstream now...
Monitoring the background processes. Making checks every 5 sec. If you don't see any error messages below - all processes are healthy...
Process /usr/bin/slipstream --pghost=postgres --pgport=5432 --pguser=postgres --password=api --database=cirrus --stratourl=http://strato:3000/eth/v1.2 --vaultwrapperurl=http://vault-wrapper:8000/strato/v2.3 --kafkahost=kafka --kafkaport=9092 with pid 122 crashed - killing all monitored processes but keeping the container running...
killing process /usr/bin/blockapps-strato-server (pid: 112)
done
killing process /usr/bin/blockapps-bloc --pghost=postgres --pgport=5432 --pguser=postgres --password=api --stratourl=http://strato:3000/eth/v1.2 --vaultwrapperurl=http://vault-wrapper:8000/strato/v2.3 --loglevel=4 +RTS -N1 (pid: 113)
done
CONTAINER IS DOWN: Process with pid 122 crashed so all background processes were killed. Check /logs/ in the container

$ docker logs strato_strato_1

Environment variables:
pgUser = postgres
pgPass = api
pgHost = postgres
kafkaHost = kafka
zkHost = zookeeper
redisBDBHost = redis
redisBDBPort = 6379
redisBDBNumber = 0
genesis = gettingStarted
miningAlgorithm = Instant
maxTxsPerBlock = 500
networkID = 6
genesisBlock =
bootnode =
mineBlocks = true
verifyBlocks = false
instantMining = true
lazyBlocks = true
serveBlocks = true
receiveBlocks = true
addBootnodes = false
numMinPeers = 0
useSyncMode = false
minQuorumSize = 1
maxConn = 20
difficultyBomb = false
sqlDiff = true
diffPublish = true
backupLocation = /var/lib/strato/backup_strato_block
statsEnable = false
statsHost = telegraf
statsPort = 8125
statsFlush = 1000
statsPrefix =
statsSuffix =
evmDebugMode = false
evmTraceMode = false
                                                  List of databases
                     Name                     |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges
----------------------------------------------+----------+----------+------------+------------+-----------------------
 apex_prod                                    | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
 bloc22                                       | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
 blockchain                                   | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
 cirrus                                       | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
 eth_15191863d9c07e9ada83c1c1f580b3357ce73f4f | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
 oauth                                        | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
 postgres                                     | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
 template0                                    | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
                                              |          |          |            |            | postgres=CTc/postgres
 template1                                    | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
                                              |          |          |            |            | postgres=CTc/postgres
(9 rows)

global-db with args:
ESC[33m>>>> Creating global databaseESC[0m
Ignoring.
Starting Strato processes. All output is logged to /var/lib/strato/logs.
Starting strato-adit
Starting strato-p2p-server
Starting ethereum-discover
Starting strato-p2p-client
Starting strato-sequencer
Starting strato-api-indexer
Starting strato-p2p-indexer
Starting strato-txr-indexer
Starting ethereum-vm
Starting strato-api
Configuring log maintenance
process pid:: 31 (command: cleanupLogs)
Monitoring the background processes. Making checks every 5 sec. If you don't see any error messages below - all processes are healthy...
Process strato-api-indexer +RTS -N1 with pid 26 crashed - killing all monitored processes but keeping the container running...
killing process strato-p2p-indexer +RTS -N1 (pid: 27)
done
killing process strato-p2p-client --cNetworkID=6 --maxConn=20 --sqlPeers=true --debugFail=true (pid: 24)
done
killing process strato-sequencer     --minLogLevel=LevelInfo (pid: 25)
done
killing process strato-p2p-server --runUDPServer=false --networkID=6 (pid: 22)
done
killing process ethereum-discover (pid: 23)
done
killing process strato-adit --useSyncMode=false --minQuorumSize=1 --threads=1 --aMiner=Instant (pid: 21)
done
killing process strato-txr-indexer +RTS -N1 (pid: 28)
done
killing process ethereum-vm --useSyncMode=false --miner=Instant --maxTxsPerBlock=500 --diffPublish=true --sqlDiff=true --createTransactionResults=true --miningVerification=false --difficultyBomb=false --trace=false --debug=false --minLogLevel=LevelInfo  +RTS -N1 (pid: 29)
done
killing process cleanupLogs (pid: 31)
done
killing process strato-api +RTS -N1 (pid: 30)
done
STRATO IS DOWN: Process with pid 26 crashed so all background processes were killed. Check /var/lib/strato/logs/ in the container

$ docker exec -it strato_strato_1 /bin/bash
$ cat/var/lib/strato/logs/strato-api-indexer

[2019-04-16 01:29:38.750871 UTC]  INFO | ThreadId 4     | apiIndexer                          | About to fetch blocks
[2019-04-16 01:33:06.66256 UTC ]  INFO | ThreadId 4     | apiIndexer                          | Fetched 0 events starting from Offset 64963
[2019-04-16 01:33:07.733987 UTC]  INFO | ThreadId 4     | apiIndexer                          | 0 of them are blocks
[2019-04-16 01:33:08.130797 UTC]  INFO | ThreadId 4     | setKafkaCheckpoint                  | Setting checkpoint to Offset 64963 / IndexerBestBlockInfo (BlockDataRefKey {unBlockDataRefKey = SqlBackendKey {unSqlBackendKey = 16329}})
[2019-04-16 01:33:08.393973 UTC]  INFO | ThreadId 4     | apiIndexer                          | About to fetch blocks
[2019-04-16 01:36:05.808527 UTC]  INFO | ThreadId 4     | apiIndexer                          | Fetched 0 events starting from Offset 64963
[2019-04-16 01:36:06.513709 UTC]  INFO | ThreadId 4     | apiIndexer                          | 0 of them are blocks
[2019-04-16 01:36:06.513944 UTC]  INFO | ThreadId 4     | setKafkaCheckpoint                  | Setting checkpoint to Offset 64963 / IndexerBestBlockInfo (BlockDataRefKey {unBlockDataRefKey = SqlBackendKey {unSqlBackendKey = 16329}})
[2019-04-16 01:36:06.570921 UTC]  INFO | ThreadId 4     | apiIndexer                          | About to fetch blocks
+ proc_pid=80
+ MONITORED_PIDS[${proc_pid}]='strato-api-indexer +RTS -N1'
+ echo 'process pid:: 80 (command: strato-api-indexer' +RTS '-N1)'
process pid:: 80 (command: strato-api-indexer +RTS -N1)
+ disown %
+ strato-api-indexer +RTS -N1
[2019-05-07 06:57:08.512895 UTC]  INFO | ThreadId 4     | runIContextM                        | Creating PG connection pool of size 20
[2019-05-07 06:57:09.329349 UTC]  INFO | ThreadId 4     | apiIndexer                          | About to fetch blocks
strato-api-indexer: ApiIndexerBestBlock was never initialized in strato-setup!
CallStack (from HasCallStack):
  error, called at src/Blockchain/Strato/Indexer/ApiIndexer.hs:124:37 in strato-index-0.1.0.0-9yTavtPJCMp8O5DcmX89Vc:Blockchain.Strato.Indexer.ApiIndexer
+ proc_pid=24
+ MONITORED_PIDS[${proc_pid}]='strato-api-indexer +RTS -N1'
+ echo 'process pid:: 24 (command: strato-api-indexer' +RTS '-N1)'
process pid:: 24 (command: strato-api-indexer +RTS -N1)
+ disown %
+ strato-api-indexer +RTS -N1
[2019-05-07 07:02:55.576165 UTC]  INFO | ThreadId 4     | runIContextM                        | Creating PG connection pool of size 20
[2019-05-07 07:02:55.581313 UTC]  INFO | ThreadId 4     | apiIndexer                          | About to fetch blocks
strato-api-indexer: ApiIndexerBestBlock was never initialized in strato-setup!
CallStack (from HasCallStack):
  error, called at src/Blockchain/Strato/Indexer/ApiIndexer.hs:124:37 in strato-index-0.1.0.0-9yTavtPJCMp8O5DcmX89Vc:Blockchain.Strato.Indexer.ApiIndexer
+ proc_pid=68
+ strato-api-indexer +RTS -N1
+ MONITORED_PIDS[${proc_pid}]='strato-api-indexer +RTS -N1'
+ echo 'process pid:: 68 (command: strato-api-indexer' +RTS '-N1)'
process pid:: 68 (command: strato-api-indexer +RTS -N1)
+ disown %
[2019-05-07 07:08:22.895406 UTC]  INFO | ThreadId 4     | runIContextM                        | Creating PG connection pool of size 20
[2019-05-07 07:08:25.964136 UTC]  INFO | ThreadId 4     | apiIndexer                          | About to fetch blocks
strato-api-indexer: ApiIndexerBestBlock was never initialized in strato-setup!
CallStack (from HasCallStack):
  error, called at src/Blockchain/Strato/Indexer/ApiIndexer.hs:124:37 in strato-index-0.1.0.0-9yTavtPJCMp8O5DcmX89Vc:Blockchain.Strato.Indexer.ApiIndexer
+ proc_pid=48
+ MONITORED_PIDS[${proc_pid}]='strato-api-indexer +RTS -N1'
+ echo 'process pid:: 48 (command: strato-api-indexer' +RTS '-N1)'
process pid:: 48 (command: strato-api-indexer +RTS -N1)
+ disown %
+ strato-api-indexer +RTS -N1
[2019-05-07 07:15:26.339768 UTC]  INFO | ThreadId 4     | runIContextM                        | Creating PG connection pool of size 20
[2019-05-07 07:15:26.357501 UTC]  INFO | ThreadId 4     | apiIndexer                          | About to fetch blocks
strato-api-indexer: ApiIndexerBestBlock was never initialized in strato-setup!
CallStack (from HasCallStack):
  error, called at src/Blockchain/Strato/Indexer/ApiIndexer.hs:124:37 in strato-index-0.1.0.0-9yTavtPJCMp8O5DcmX89Vc:Blockchain.Strato.Indexer.ApiIndexer
+ proc_pid=25
+ MONITORED_PIDS[${proc_pid}]='strato-api-indexer +RTS -N1'
+ echo 'process pid:: 25 (command: strato-api-indexer' +RTS '-N1)'
process pid:: 25 (command: strato-api-indexer +RTS -N1)
+ disown %
+ strato-api-indexer +RTS -N1
[2019-05-07 07:41:36.45305 UTC ]  INFO | ThreadId 4     | runIContextM                        | Creating PG connection pool of size 20
[2019-05-07 07:41:36.460885 UTC]  INFO | ThreadId 4     | apiIndexer                          | About to fetch blocks
strato-api-indexer: ApiIndexerBestBlock was never initialized in strato-setup!
CallStack (from HasCallStack):
  error, called at src/Blockchain/Strato/Indexer/ApiIndexer.hs:124:37 in strato-index-0.1.0.0-9yTavtPJCMp8O5DcmX89Vc:Blockchain.Strato.Indexer.ApiIndexer
+ proc_pid=58
+ MONITORED_PIDS[${proc_pid}]='strato-api-indexer +RTS -N1'
+ strato-api-indexer +RTS -N1
+ echo 'process pid:: 58 (command: strato-api-indexer' +RTS '-N1)'
process pid:: 58 (command: strato-api-indexer +RTS -N1)
+ disown %
[2019-05-07 07:46:38.164704 UTC]  INFO | ThreadId 4     | runIContextM                        | Creating PG connection pool of size 20
[2019-05-07 07:46:39.031233 UTC]  INFO | ThreadId 4     | apiIndexer                          | About to fetch blocks
strato-api-indexer: ApiIndexerBestBlock was never initialized in strato-setup!
CallStack (from HasCallStack):
  error, called at src/Blockchain/Strato/Indexer/ApiIndexer.hs:124:37 in strato-index-0.1.0.0-9yTavtPJCMp8O5DcmX89Vc:Blockchain.Strato.Indexer.ApiIndexer
+ proc_pid=26
+ MONITORED_PIDS[${proc_pid}]='strato-api-indexer +RTS -N1'
+ echo 'process pid:: 26 (command: strato-api-indexer' +RTS '-N1)'
process pid:: 26 (command: strato-api-indexer +RTS -N1)
+ disown %
+ strato-api-indexer +RTS -N1
[2019-05-14 06:27:28.433798 UTC]  INFO | ThreadId 4     | runIContextM                        | Creating PG connection pool of size 20
[2019-05-14 06:27:28.462827 UTC]  INFO | ThreadId 4     | apiIndexer                          | About to fetch blocks
strato-api-indexer: ApiIndexerBestBlock was never initialized in strato-setup!
CallStack (from HasCallStack):
  error, called at src/Blockchain/Strato/Indexer/ApiIndexer.hs:124:37 in strato-index-0.1.0.0-9yTavtPJCMp8O5DcmX89Vc:Blockchain.Strato.Indexer.ApiIndexer

Thanks for the help.


#2

Thank you for contacting the BlockApps support team. We will respond to your inquiry shortly.

Thanks,
Zach