Setup Issue- Nginx 502


#1

I’m having a similar issue to this thread, which doesn’t seems to have been resolved consistently-

I’ve followed the getting started guide on a fresh remote install of Ubuntu 17.04, and any API calls are returning a 502 from Nginx. Example-

[email protected]:~$ curl "http://localhost/bloc/v2.1/users"
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.10.3 (Ubuntu)</center>
</body>
</html>

I can access the Strato management dashboard, but nothing is functional since it can’t access any of the endpoints. I’ve had this issue on local installs as well in the past, but have only been able to resolve it by completely starting fresh (which was inconsistent at best).


#2

Hi @APmoss - can you try

./strato-run.sh --wipe
./strato-run.sh
npm run deploy
npm run start

as in the other post and report the results?
Also, FWIW, the current bloc version is 2.2. On my machine:

Kierens-MacBook-Pro:strato-getting-started kjameslubin$ ./strato-run.sh --stable
Kierens-MacBook-Pro:strato-getting-started kjameslubin$ curl "http://localhost/bloc/v2.1/users"
[]

Kierens-MacBook-Pro:strato-getting-started kjameslubin$ curl "http://localhost/bloc/v2.2/users"
[]

Kierens-MacBook-Pro:strato-getting-started kjameslubin$ curl "http://localhost/bloc/v2.1/users"
[]


 ... *make user on dashboard*


Kierens-MacBook-Pro:strato-getting-started kjameslubin$ curl "http://localhost/bloc/v2.2/users"
["kjl"]

#3

I’ve just tried wiping again but still have the same result-

[email protected]:~/code/strato-getting-started$ curl "http://localhost/bloc/v2.2/users"
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.10.3 (Ubuntu)</center>
</body>
</html>

However, i have noticed something that might be contributing to the issue. I’ve compared the output of docker ps on a working machine and on the machine with the problem.

Working-

[email protected]:~/code/strato-getting-started$ sudo docker ps
CONTAINER ID        IMAGE                                                                       COMMAND                  CREATED              STATUS              PORTS                                                          NAMES
851f2d4adceb        registry-aws.blockapps.net:5000/blockapps-repo/silo-nginx:latest            "/run.sh"                About a minute ago   Up About a minute   0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                       strato_nginx_1
8ee1f7493cf0        registry-aws.blockapps.net:5000/blockapps-repo/silo-smd-ui:latest           "sh /usr/src/app/d..."   About a minute ago   Up About a minute   3002/tcp                                                       strato_smd-ui_1
6965f8b18019        registry-aws.blockapps.net:5000/blockapps-repo/silo-cirrus:latest           "/doit.sh"               About a minute ago   Up About a minute                                                                  strato_cirrus_1
90be55c48ad5        registry-aws.blockapps.net:5000/blockapps-repo/silo-bloch:latest            "/usr/bin/bloc/doi..."   About a minute ago   Up About a minute   8000/tcp                                                       strato_bloch_1
8d5e7df9ae51        registry-aws.blockapps.net:5000/blockapps-repo/silo-strato:latest           "/var/lib/doit.sh"       About a minute ago   Up About a minute   3000/tcp, 0.0.0.0:30303->30303/tcp, 0.0.0.0:30303->30303/udp   strato_strato_1
8b4997b393bc        registry-aws.blockapps.net:5000/blockapps-repo/silo-kafka:latest            "/doit.sh"               About a minute ago   Up About a minute   9092/tcp                                                       strato_kafka_1
13ee42fdc1d4        registry-aws.blockapps.net:5000/blockapps-repo/silo-postgrest:latest        "/doit.sh"               About a minute ago   Up About a minute   3001/tcp                                                       strato_postgrest_1
13d3b4c25c13        redis:3.2                                                                   "docker-entrypoint..."   About a minute ago   Up About a minute   6379/tcp                                                       strato_redis_1
cc56c06ce9be        registry-aws.blockapps.net:5000/blockapps-repo/silo-postgres:latest         "/run.sh"                About a minute ago   Up About a minute                                                                  strato_postgres_1
d4938af5ab46        zookeeper:3.4.9                                                             "/docker-entrypoin..."   About a minute ago   Up About a minute   2181/tcp, 2888/tcp, 3888/tcp                                   strato_zookeeper_1
d74d8b59f410        postgres:9.6                                                                "docker-entrypoint..."   About a minute ago   Up About a minute   5432/tcp                                                       strato_postgres-cirrus_1
22a3d8346e1b        registry-aws.blockapps.net:5000/blockapps-repo/silo-blockapps-docs:latest   "sh /usr/share/ngi..."   About a minute ago   Up About a minute   8080/tcp                                                       strato_blockapps-docs_1

Not working-

[email protected]:~/code/strato-getting-started$ sudo docker ps
CONTAINER ID        IMAGE                                                                        COMMAND                  CREATED             STATUS                          PORTS                                                          NAMES
e73a7c7860b4        registry-aws.blockapps.net:5000/blockapps-repo/silo-nginx:eb563ae            "/run.sh"                13 minutes ago      Up 13 minutes                   0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                       strato_nginx_1
067d8ef48de3        registry-aws.blockapps.net:5000/blockapps-repo/silo-smd-ui:25cac97           "sh /usr/src/app/d..."   13 minutes ago      Up 13 minutes                   3002/tcp                                                       strato_smd-ui_1
1e39c5cb6ffa        registry-aws.blockapps.net:5000/blockapps-repo/silo-cirrus:d9d20d3           "/doit.sh"               13 minutes ago      Up 13 minutes                                                                                  strato_cirrus_1
b32d1f2bedf0        registry-aws.blockapps.net:5000/blockapps-repo/silo-bloch:520aebd            "/usr/bin/bloc/doi..."   13 minutes ago      Restarting (1) 59 seconds ago                                                                  strato_bloch_1
46a59e048690        registry-aws.blockapps.net:5000/blockapps-repo/silo-strato:6adbfdd           "/var/lib/doit.sh"       13 minutes ago      Up 13 minutes                   3000/tcp, 0.0.0.0:30303->30303/tcp, 0.0.0.0:30303->30303/udp   strato_strato_1
56a42b6ead2b        registry-aws.blockapps.net:5000/blockapps-repo/silo-kafka:648c004            "/doit.sh"               13 minutes ago      Restarting (1) 20 seconds ago                                                                  strato_kafka_1
c999d53716da        redis:3.2                                                                    "docker-entrypoint..."   13 minutes ago      Up 13 minutes                   6379/tcp                                                       strato_redis_1
767b4ac8932b        registry-aws.blockapps.net:5000/blockapps-repo/silo-postgres:750b87e         "/run.sh"                13 minutes ago      Up 13 minutes                                                                                  strato_postgres_1
8ad9271539ac        postgres:9.6                                                                 "docker-entrypoint..."   13 minutes ago      Up 13 minutes                   5432/tcp                                                       strato_postgres-cirrus_1
8e999c9f9849        registry-aws.blockapps.net:5000/blockapps-repo/silo-blockapps-docs:ebc2107   "sh /usr/share/ngi..."   13 minutes ago      Up 13 minutes                   8080/tcp                                                       strato_blockapps-docs_1
bb77e3ab323d        zookeeper:3.4.9                                                              "/docker-entrypoin..."   13 minutes ago      Up 13 minutes                   2181/tcp, 2888/tcp, 3888/tcp                                   strato_zookeeper_1
b7b3eafaab47        registry-aws.blockapps.net:5000/blockapps-repo/silo-postgrest:ae5c1c5        "/doit.sh"               13 minutes ago      Up 13 minutes                   3001/tcp                                                       strato_postgrest_1

It seems like strato_bloch_1 and strato_kafka_1 are not starting correctly? The two machines I’m using are near identical, so I’m not sure what could be causing the issue.


#4

Can you check the output of
docker logs strato_kafka_1?


#5

It seems like it might be a memory issue-

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 1073741824 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /usr/lib/kafka/hs_err_pid14.log

...

/proc/meminfo:
MemTotal:        1013060 kB
MemFree:           24548 kB
MemAvailable:     613264 kB
Buffers:          186288 kB
Cached:           388484 kB
SwapCached:           64 kB
Active:           621232 kB
Inactive:         191532 kB
Active(anon):     209836 kB
Inactive(anon):    52640 kB
Active(file):     411396 kB
Inactive(file):   138892 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:        262140 kB
SwapFree:         262072 kB
Dirty:             24804 kB
Writeback:             0 kB
AnonPages:        238028 kB
Mapped:           134380 kB
Shmem:             24412 kB
Slab:             111856 kB
SReclaimable:      58360 kB
SUnreclaim:        53496 kB
KernelStack:        6000 kB
PageTables:         7020 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      768668 kB
Committed_AS:    2069152 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
DirectMap4k:       14200 kB
DirectMap2M:     1034240 kB
DirectMap1G:           0 kB

That might explain why it’s inconsistent, but it’s still strange since both machines should have 1024mb allocated to them. I’ll do some testing with more/less memory to see if it resolves the issue.

Is there a known minimum memory requirement to be able to run all services?


#6

Is there a known minimum memory requirement to be able to run all services?

We will have to investigate. A few people have run into memory related issues recently - we have some tickets in our backlog to figure it out.


#7

Ok, I don’t think memory capacity is the problem here. I’ve reduced the working machine to 512mb of base memory for testing purposes (which is less than the non-working machine), but it still seems to run all services correctly.

[email protected]:~/code/strato-getting-started$ free
              total        used        free      shared  buff/cache   available
Mem:         497660      325332        5668        4228      166660      155664
Swap:        969960      675824      294136

# Run strato services...

[email protected]:~/code/strato-getting-started$ sudo docker ps
CONTAINER ID        IMAGE                                                                       COMMAND                  CREATED             STATUS              PORTS                                                          NAMES
e023963bf9b3        registry-aws.blockapps.net:5000/blockapps-repo/silo-nginx:latest            "/run.sh"                19 minutes ago      Up 5 minutes        0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                       strato_nginx_1
98a2d87f2342        registry-aws.blockapps.net:5000/blockapps-repo/silo-smd-ui:latest           "sh /usr/src/app/d..."   19 minutes ago      Up 5 minutes        3002/tcp                                                       strato_smd-ui_1
20eb6c4648e9        registry-aws.blockapps.net:5000/blockapps-repo/silo-cirrus:latest           "/doit.sh"               19 minutes ago      Up About a minute                                                                  strato_cirrus_1
1bef52af9931        registry-aws.blockapps.net:5000/blockapps-repo/silo-bloch:latest            "/usr/bin/bloc/doi..."   19 minutes ago      Up 4 minutes        8000/tcp                                                       strato_bloch_1
e962aa93c61c        registry-aws.blockapps.net:5000/blockapps-repo/silo-strato:latest           "/var/lib/doit.sh"       19 minutes ago      Up 5 minutes        3000/tcp, 0.0.0.0:30303->30303/tcp, 0.0.0.0:30303->30303/udp   strato_strato_1
4a6ee52018ff        registry-aws.blockapps.net:5000/blockapps-repo/silo-kafka:latest            "/doit.sh"               19 minutes ago      Up 5 minutes        9092/tcp                                                       strato_kafka_1
79678936e122        registry-aws.blockapps.net:5000/blockapps-repo/silo-postgrest:latest        "/doit.sh"               19 minutes ago      Up 5 minutes        3001/tcp                                                       strato_postgrest_1
1ca0f8704199        registry-aws.blockapps.net:5000/blockapps-repo/silo-postgres:latest         "/run.sh"                19 minutes ago      Up 5 minutes                                                                       strato_postgres_1
b423ddd5b0d3        zookeeper:3.4.9                                                             "/docker-entrypoin..."   19 minutes ago      Up 5 minutes        2181/tcp, 2888/tcp, 3888/tcp                                   strato_zookeeper_1
7101120ef492        redis:3.2                                                                   "docker-entrypoint..."   19 minutes ago      Up 5 minutes        6379/tcp                                                       strato_redis_1
0e72f02d6128        postgres:9.6                                                                "docker-entrypoint..."   19 minutes ago      Up About a minute   5432/tcp                                                       strato_postgres-cirrus_1
36e885b2e8e6        registry-aws.blockapps.net:5000/blockapps-repo/silo-blockapps-docs:latest   "sh /usr/share/ngi..."   19 minutes ago      Up 5 minutes        8080/tcp                                                       strato_blockapps-docs_1

#8

are you running SMD in the browser on the same machine where you have STRATO deployed?


#9

No, it’s a headless remote server. I test it both locally and remotely however, either by sshing in then accessing localhost, or by just using its remote ip. Both methods result in the same error.


#10

makes sense for me. Please provide the NODE_HOST when you run STRATO:
NODE_HOST=<VM EXTERNAL IP> ./strato-run.sh
where <VM EXTERNAL IP> is VM’s IP reachable from your machine.

Refer to the comments for the Step 3 of the setup guide: https://developers.blockapps.net/install/local/#setup


#11

I have tried that before-

[email protected]:~/code/strato-getting-started$ echo $NODE_HOST
<server ip here>

sudo ./strato-run.sh

[email protected]:~/code/strato-getting-started$ sudo docker ps
CONTAINER ID        IMAGE                                                                       COMMAND                  CREATED             STATUS                         PORTS                                                          NAMES
ec3358433c06        registry-aws.blockapps.net:5000/blockapps-repo/silo-nginx:latest            "/run.sh"                8 seconds ago       Up 7 seconds                   0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                       strato_nginx_1
989478cc0b5e        registry-aws.blockapps.net:5000/blockapps-repo/silo-smd-ui:latest           "sh /usr/src/app/d..."   9 seconds ago       Up 8 seconds                   3002/tcp                                                       strato_smd-ui_1
190dd1c552ec        registry-aws.blockapps.net:5000/blockapps-repo/silo-cirrus:latest           "/doit.sh"               10 seconds ago      Up 8 seconds                                                                                  strato_cirrus_1
72cb9ed1c0e9        registry-aws.blockapps.net:5000/blockapps-repo/silo-bloch:latest            "/usr/bin/bloc/doi..."   11 seconds ago      Up Less than a second          8000/tcp                                                       strato_bloch_1
4fc8f6c0574a        registry-aws.blockapps.net:5000/blockapps-repo/silo-strato:latest           "/var/lib/doit.sh"       11 seconds ago      Up 10 seconds                  3000/tcp, 0.0.0.0:30303->30303/tcp, 0.0.0.0:30303->30303/udp   strato_strato_1
a28ebdee7b7e        registry-aws.blockapps.net:5000/blockapps-repo/silo-kafka:latest            "/doit.sh"               14 seconds ago      Restarting (1) 3 seconds ago                                                                  strato_kafka_1
9262afe92e97        registry-aws.blockapps.net:5000/blockapps-repo/silo-postgrest:latest        "/doit.sh"               15 seconds ago      Up 12 seconds                  3001/tcp                                                       strato_postgrest_1
b485f4e62a37        zookeeper:3.4.9                                                             "/docker-entrypoin..."   15 seconds ago      Up 13 seconds                  2181/tcp, 2888/tcp, 3888/tcp                                   strato_zookeeper_1
e424d62dfc8a        redis:3.2                                                                   "docker-entrypoint..."   15 seconds ago      Up 11 seconds                  6379/tcp                                                       strato_redis_1
615f3809c512        postgres:9.6                                                                "docker-entrypoint..."   15 seconds ago      Up 10 seconds                  5432/tcp                                                       strato_postgres-cirrus_1
dd50776ad562        registry-aws.blockapps.net:5000/blockapps-repo/silo-postgres:latest         "/run.sh"                15 seconds ago      Up 11 seconds                                                                                 strato_postgres_1
230479bb5e4c        registry-aws.blockapps.net:5000/blockapps-repo/silo-blockapps-docs:latest   "sh /usr/share/ngi..."   15 seconds ago      Up 11 seconds                  8080/tcp                                                       strato_blockapps-docs_1

Notice not all processes have started.


#12

Ok, I think I’ve resolved the issue. I increased the swap space on my server and everything seems to be working well so far. In the end it probably just wasn’t getting enough memory allocated to start the services.

My specs to get it working was 1024mb main memory and 512mb of swap.