Faceswap connectivity to Server

[asadch]

Hi i'm trying to connect faceswap app with my server. But it results in connection closed error. If anyone can support it will be really appreciated. If anyone working on the same architecture please comment OR if there is any online forum / community on faceswap??
Please find below the start_demo output.
asad@asad-HP-Pavilion-15-Notebook-PC:~/FaceSwap-server/server$ ./start_demo.sh
sourcing torch:
/home/asad/torch/install/bin/torch-activate
launching FaceSwap server at dir /home/asad/FaceSwap-server/server
INFO Start RESTful API Server (port :8021)
INFO Start UPnP Server
INFO Start monitoring offload engines
INFO * Mobile server() configuration
INFO - Open TCP Server at ('0.0.0.0', 22222)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * Mobile server() configuration
INFO - Open TCP Server at ('0.0.0.0', 9098)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * Mobile server() configuration
INFO - Open TCP Server at ('0.0.0.0', 9099)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * Mobile server() configuration
INFO - Open TCP Server at ('0.0.0.0', 9100)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * Mobile server() configuration
INFO - Open TCP Server at ('0.0.0.0', 9111)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * UComm relay server() configuration
INFO - Open TCP Server at ('0.0.0.0', 9090)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * Application server() configuration
INFO - Open TCP Server at ('0.0.0.0', 10101)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * Application server() configuration
INFO - Open TCP Server at ('0.0.0.0', 10102)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * Application server() configuration
INFO - Open TCP Server at ('0.0.0.0', 10103)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO Gabriel Server :
INFO {u'acc_tcp_streaming_ip': u'192.168.100.11',
u'acc_tcp_streaming_port': 10102,
u'audio_tcp_streaming_ip': u'192.168.100.11',
u'audio_tcp_streaming_port': 10103,
u'ucomm_relay_ip': u'192.168.100.11',
u'ucomm_relay_port': 9090,
u'ucomm_server_ip': None,
u'ucomm_server_port': None,
u'video_tcp_streaming_ip': u'192.168.100.11',
u'video_tcp_streaming_port': 10101}
INFO connecting to 192.168.100.11:9090
INFO Result forwarding thread created
INFO * UCOMM server configuration
INFO User communication module is connected
INFO - Open TCP Server at ('0.0.0.0', 10120)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO Mobile client is connected for (Mobile Video Server)
INFO Connection closed (Mobile Video Server)
INFO [TERMINATE] Finish Mobile Video Server
INFO Mobile client is connected for (Mobile Video Server)
INFO Connection closed (Mobile Video Server)
INFO [TERMINATE] Finish Mobile Video Server
INFO Mobile client is connected for (Mobile Video Server)
INFO Connection closed (Mobile Video Server)
INFO [TERMINATE] Finish Mobile Video Server
INFO Mobile client is connected for (Mobile Video Server)
INFO Connection closed (Mobile Video Server)
INFO [TERMINATE] Finish Mobile Video Server
INFO Mobile client is connected for (Mobile Video Server)
INFO Connection closed (Mobile Video Server)
INFO [TERMINATE] Finish Mobile Video Server

below are the app snaphots

[teiszler]

@asadch,

I apologize for not responding sooner. It looks from the log like the client is connecting and then immediately disconnecting. So it would seem that the IP address is correct. Is it possible that there are some firewall rules blocking some of the ports that are used here? The Gabriel architecture communicates to different ports for different services, so all of those listed in the log need to be open (even though not all of them are used between the client and the server).

There is some documentation for FaceSwap available here: https://cmusatyalab.github.io/faceswap/, thought I don't see your specific issue. If you are still having difficulty, I can speak with the author of FaceSwap to see if they have any thoughts.

Please let me know.

Regards,
Tom

[MSoares]

Dear Tom,

I tryed also to install Faceswap via https://cmusatyalab.github.io/faceswap/, method 1.
I have change firewalll to accept all ports used.

I am able to start training, it can send name and devolve also "dummy" and after that an error message appears on Mobile "Error" and also on Server, as you can observe below.

faceswap-admin@faceswap-server-debug:~/FaceSwap-server$ ./start_demo.sh
sourcing torch:
/home/faceswap-admin/torch/install/bin/torch-activate
launching FaceSwap server at dir /home/faceswap-admin/FaceSwap-server
INFO Start RESTful API Server (port :8021)
INFO Start UPnP Server
INFO Start monitoring offload engines
INFO * Mobile server() configuration
INFO - Open TCP Server at ('0.0.0.0', 9098)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * Mobile server() configuration
INFO - Open TCP Server at ('0.0.0.0', 9099)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * Mobile server() configuration
INFO - Open TCP Server at ('0.0.0.0', 9101)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * Application server() configuration
INFO - Open TCP Server at ('0.0.0.0', 10101)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * Application server() configuration
INFO - Open TCP Server at ('0.0.0.0', 10102)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * UComm server configuration
INFO - Open TCP Server at ('0.0.0.0', 9090)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
[Error] REST API Server is closed unexpectedly
INFO Google Glass is connected for (Mobile Video Server)
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread
INFO Google Glass is connected for (Mobile Video Server)
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread
{u'objects': {u'acc_tcp_streaming_address': u'10.11.12.61', u'video_tcp_streaming_port': 10101, u'result_return_server_list': [u'10.11.12.61:10120'], u'ucomm_data_address': u'10.11.12.61', u'ucomm_data_port': 9090, u'acc_tcp_streaming_port': 10102, u'video_tcp_streaming_address': u'10.11.12.61'}, u'return': u'success'}
INFO Gabriel Server :
INFO {u'acc_tcp_streaming_address': u'10.11.12.61',
u'acc_tcp_streaming_port': 10102,
u'result_return_server_list': [u'10.11.12.61:10120'],
u'ucomm_data_address': u'10.11.12.61',
u'ucomm_data_port': 9090,
u'video_tcp_streaming_address': u'10.11.12.61',
u'video_tcp_streaming_port': 10101}
INFO connecting to 10.11.12.61:9090
INFO User communication module is connected
INFO * UCOMM server configuration
INFO - Open TCP Server at ('0.0.0.0', 10120)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO Start forwarding data
INFO Google Glass is connected for (Mobile Video Server)
2018-01-29 23:07:18+0000 [-] Log opened.
2018-01-29 23:07:18+0000 [-] Traceback (most recent call last):
2018-01-29 23:07:18+0000 [-] File "/home/faceswap-admin/FaceSwap-server/openface-server/cloudlet-demo-openface-server.py", line 595, in
2018-01-29 23:07:18+0000 [-] reactor.listenTCP(args.port, factory)
2018-01-29 23:07:18+0000 [-] File "/usr/local/lib/python2.7/dist-packages/twisted/internet/posixbase.py", line 492, in listenTCP
2018-01-29 23:07:18+0000 [-] p.startListening()
2018-01-29 23:07:18+0000 [-] File "/usr/local/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 984, in startListening
2018-01-29 23:07:18+0000 [-] raise CannotListenError(self.interface, self.port, le)
2018-01-29 23:07:18+0000 [-] twisted.internet.error.CannotListenError: Couldn't listen on any:9000: [Errno 98] Address already in use.
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread
checking openface server status:
1
openface server has not finished starting. wait for another 20 seconds...
checking openface server status:
2
openface server has not finished starting. wait for another 20 seconds...
INFO Google Glass is connected for (Mobile Video Server)
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread
INFO Google Glass is connected for (Mobile Video Server)
checking openface server status:
3
openface server has not finished starting. wait for another 20 seconds...
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread
INFO Offloading engine is connected
INFO new Offlaoding Engine is connected
INFO Google Glass is connected for (Mobile Video Server)
checking openface server status:
4
openface server has not finished starting. wait for another 20 seconds...
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread
INFO Google Glass is connected for (Mobile Video Server)
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread
checking openface server status:
5
openface server has not finished starting. wait for another 20 seconds...
2018-01-29 23:09:07,828 INFO MainProcess resp: {"type": "GET_TRAINING_RESP", "val": false}
2018-01-29 23:09:07,829 INFO MainProcess openface is training?False
Discovery Control VM
INFO Gabriel Server :
INFO {u'acc_tcp_streaming_address': u'10.11.12.61',
u'acc_tcp_streaming_port': 10102,
u'result_return_server_list': [u'10.11.12.61:10120'],
u'ucomm_data_address': u'10.11.12.61',
u'ucomm_data_port': 9090,
u'video_tcp_streaming_address': u'10.11.12.61',
u'video_tcp_streaming_port': 10101}
TOKEN SIZE OF OFFLOADING ENGINE: 1
INFO Success to connect to (u'10.11.12.61', 10101)
[INFO/DetectionProcess] child process calling self.run()
2018-01-29 23:09:07,847 INFO DetectionProcess created
INFO Offloading engine is connected
INFO Start getting data from the server
INFO Start publishing data
INFO new Offlaoding Engine is connected
[INFO/receive_thread] child process calling self.run()
INFO Google Glass is connected for (Mobile Video Server)
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread
INFO Google Glass is connected for (Mobile Video Server)
INFO New connection is starting at 1517267366.198832
adding person
training_cnt :0
INFO returning result: {"training": "trt", "engine_id": "dummy"
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread
INFO Google Glass is connected for (Mobile Video Server)
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread
INFO Google Glass is connected for (Mobile Video Server)
INFO New connection is starting at 1517267397.277692
adding person
training_cnt :0
INFO returning result: {"training": "joke", "engine_id": "dummy
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread
INFO Google Glass is connected for (Mobile Video Server)
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread
INFO Google Glass is connected for (Mobile Video Server)
INFO New connection is starting at 1517267433.212038
adding person
training_cnt :0
INFO returning result: {"training": "tab", "engine_id": "dummy"
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread
^C[INFO/DetectionProcess] process shutting down
[INFO/receive_thread] process shutting down
Process receive_thread:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
[INFO/DetectionProcess] calling join() for process receive_thread
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self.target(*self.args, **self._kwargs)
File "/home/faceswap-admin/FaceSwap-server/openfaceClient.py", line 165, in async_on_receive
inputready,outputready,exceptready = select.select(input,[],[])
KeyboardInterrupt
[INFO/receive_thread] process exiting with exitcode 1
Process DetectionProcess:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self.target(*self.args, **self._kwargs)
File "/home/faceswap-admin/FaceSwap-server/face_swap.py", line 417, in detection_update_thread
frame = img_queue.get(timeout=1)
File "/usr/lib/python2.7/multiprocessing/queues.py", line 131, in get
if timeout < 0 or not self._poll(timeout):
KeyboardInterrupt
[INFO/DetectionProcess] process exiting with exitcode 1
user exits
2018-01-29 23:23:12,097 INFO MainProcess detection process shutdown!
2018-01-29 23:23:12,099 INFO MainProcess sync faces thread shutdown!
INFO App thread terminated
INFO Streaming thread terminated
INFO terminate thread
INFO Publish thread terminated
[INFO/MainProcess] process shutting down
DEBUG Socket is closed
INFO Offloading engine is disconnected
INFO <main.UCommServerHandler object at 0x7fe36784d890> terminate thread
faceswap-admin@faceswap-server-debug:~/FaceSwap-server$

Please, it would be great to finish testing that part.
Thanks for your estimated help.

faceswap-admin@faceswap-server-debug:~/FaceSwap-server$ sudo ufw status
sudo: unable to resolve host faceswap-server-debug
[sudo] password for faceswap-admin:
Status: active

To Action From

---

8021 ALLOW Anywhere
22222 ALLOW Anywhere
9098 ALLOW Anywhere
9099 ALLOW Anywhere
9100 ALLOW Anywhere
9111 ALLOW Anywhere
9090 ALLOW Anywhere
10101 ALLOW Anywhere
10102 ALLOW Anywhere
10103 ALLOW Anywhere
8021 (v6) ALLOW Anywhere (v6)
22222 (v6) ALLOW Anywhere (v6)
9098 (v6) ALLOW Anywhere (v6)
9099 (v6) ALLOW Anywhere (v6)
9100 (v6) ALLOW Anywhere (v6)
9111 (v6) ALLOW Anywhere (v6)
9090 (v6) ALLOW Anywhere (v6)
10101 (v6) ALLOW Anywhere (v6)
10102 (v6) ALLOW Anywhere (v6)
10103 (v6) ALLOW Anywhere (v6)

In Android side, I have tryed with 2 Android Mobile Smartphones: Samsung S8 (Android 7) and Benq Aquaris M5 (Android 6), and it seems that Bq Aquaris M5 have better results because it is possible to see on server that "dummy" is devolved, maybe Android version is important.

I tryed many times to make training with different names and result is ever the same error window.
Then it came back to general page and try to connect to server again.

Regarding Gabriel process, I have checked https://github.com/cmusatyalab/gabriel and verified each point and also 5.)

HTTP display

If you want to have a quick test of whether your image stream transmission is working fine, you can run another cognitive engine by

$ cd /server/bin/example-proxies/gabriel-proxy-http-display
$ ./proxy.py

This cognitive engine sets up an HTTP server to publish the received images. If all properly set, you should now be able to see the camera images from mobile devices by connecting to http://gabriel_ip:7070/index.html using your browser.

I have tryed it and it is working, I am able to see what the mobile device is displaying.

Thanks for your estimated help.

[teiszler]

@MSoares,

I will see if the FaceSwap expert can take a look. In the mean time, I see that in the log, there is a complaint that the address is already in use.

2018-01-29 23:07:18+0000 [-] twisted.internet.error.CannotListenError: Couldn't listen on any:9000: [Errno 98] Address already in use.

Is it possible that you have launched ./start_demo.sh multiple times? If you do netstat -pl does it show that something is already listening on any of those ports that faceswap is trying to listen on (8021, 9098, 9099 etc)? Maybe you just need to kill the processes that are still executing from previous attempts.

Regards,
Tom

[MSoares]

Dear Tom,

I have not tell you before but when we launch instance 2 processes of faceswap start running. So in order to launch ./start_demo.sh, I had to kill them first.

I am going to try a netstat -pl so we can analyze what is going.

Bests regards,
Micael

[junjuew]

Hi Micael @MSoares,

The server image starts the Faceswap backend automatically using upstart. There is no need to log-in, manually kill those processes, and restart them. Just wait until the server finishes its boot process to use the client.

I agree with Tom. From the log, port 9000 is occupied. (probably because an old faceswap process?)

[MSoares]

Hi @teiszler and @junjuew,

I have tried to wait for 5 minutes in order that server could boot all processes and result is the same.
If I login and wait for process to boot, I can see that there is many gabriel-control instantiated.
Is that normal?

When I put command netstat -pl, it presents information as below:

faceswap-admin@faceswap-server-debug:~$ netstat -pl
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 :zope-ftp *: LISTEN 760/python
tcp 0 0 :10101 *: LISTEN 735/python
tcp 0 0 :ssh *: LISTEN -
tcp 0 0 :10102 *: LISTEN 735/python
tcp 0 0 :9090 *: LISTEN 735/python
tcp 0 0 :9000 *: LISTEN 1040/python2
tcp 0 0 :10120 *: LISTEN 913/python
tcp 0 0 :xinetd *: LISTEN 735/python
tcp 0 0 :9099 *: LISTEN 735/python
tcp 0 0 :bacula-dir *: LISTEN 735/python
tcp 0 0 :http-alt *: LISTEN 735/python
tcp6 0 0 [::]:ssh [::]:* LISTEN -
tcp6 0 0 faceswap-server.n:42874 [::]:* LISTEN 768/java
udp 0 0 :bootpc *: -
udp 0 0 :18615 *: -
udp6 0 0 faceswap-server.n:40354 [::]:* 768/java
udp6 0 0 [::]:30324 [::]:* -
udp6 0 0 [::]:1900 [::]:* 768/java
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node PID/Program name Path
unix 2 [ ACC ] STREAM LISTENING 7003 - @/com/ubuntu/upstart
unix 2 [ ACC ] STREAM LISTENING 9383 - /var/run/acpid.socket
unix 2 [ ACC ] STREAM LISTENING 8368 - /var/run/dbus/system_bus_socket
unix 2 [ ACC ] SEQPACKET LISTENING 7123 - /run/udev/control

if I kill process and start manually ./start_demo.sh, I have got error message "Error" after making picture training and next lines in server:

faceswap-admin@faceswap-server-debug:~$ top

top - 22:55:03 up 18 min, 1 user, load average: 0.13, 0.37, 0.43
Tasks: 84 total, 1 running, 82 sleeping, 0 stopped, 1 zombie
%Cpu(s): 6.2 us, 2.7 sy, 0.0 ni, 91.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 4048352 total, 523620 used, 3524732 free, 18832 buffers
KiB Swap: 0 total, 0 used, 0 free. 316232 cached Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
913 faceswa+ 20 0 262100 13080 3480 S 14.6 0.3 2:50.80 gabriel-ucomm
735 faceswa+ 20 0 857308 14476 3592 S 3.3 0.4 0:55.01 gabriel-control
1 root 20 0 33504 2816 1444 S 0.0 0.1 0:00.79 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
4 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
6 root 20 0 0 0 0 S 0.0 0.0 0:00.03 kworker/u4:0
7 root 20 0 0 0 0 S 0.0 0.0 0:00.08 rcu_sched
8 root 20 0 0 0 0 S 0.0 0.0 0:00.05 rcuos/0
9 root 20 0 0 0 0 S 0.0 0.0 0:00.03 rcuos/1
10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/0
12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/1
13 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
14 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
15 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
16 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/1
17 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1
19 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0H
20 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 khelper
21 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
22 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns
23 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback
24 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kintegrityd
25 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
26 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/u5:0
27 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kblockd
28 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ata_sff
29 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khubd
30 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 md
31 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 devfreq_wq
faceswap-admin@faceswap-server-debug:~$ kill 913
faceswap-admin@faceswap-server-debug:~$ kill 735
faceswap-admin@faceswap-server-debug:~$ cd FaceSwap-server/
faceswap-admin@faceswap-server-debug:~/FaceSwap-server$ ls
LICENSE NetworkProtocol.py camShift.pyc face_swap.py install-deps.sh notes.txt openfaceClient.py script
MyUtils.py NetworkProtocol.pyc demo_config.py face_swap.pyc kill_demo.sh openface-server openfaceClient.pyc start_demo.sh
MyUtils.pyc camShift.py demo_config.pyc faceswap-proxy.py mirror-proxy.py openface-server.log requirements.txt
faceswap-admin@faceswap-server-debug:~/FaceSwap-server$ ./start_demo.sh
sourcing torch:
/home/faceswap-admin/torch/install/bin/torch-activate
launching FaceSwap server at dir /home/faceswap-admin/FaceSwap-server
INFO Start RESTful API Server (port :8021)
INFO Start UPnP Server
INFO Start monitoring offload engines
INFO * Mobile server() configuration
INFO - Open TCP Server at ('0.0.0.0', 9098)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * Mobile server() configuration
INFO - Open TCP Server at ('0.0.0.0', 9099)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * Mobile server() configuration
INFO - Open TCP Server at ('0.0.0.0', 9101)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * Application server() configuration
INFO - Open TCP Server at ('0.0.0.0', 10101)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * Application server() configuration
INFO - Open TCP Server at ('0.0.0.0', 10102)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * UComm server configuration
INFO - Open TCP Server at ('0.0.0.0', 9090)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
[Error] REST API Server is closed unexpectedly
{u'objects': {u'acc_tcp_streaming_address': u'10.11.12.64', u'video_tcp_streaming_port': 10101, u'result_return_server_list': [u'10.11.12.64:10120'], u'ucomm_data_address': u'10.11.12.64', u'ucomm_data_port': 9090, u'acc_tcp_streaming_port': 10102, u'video_tcp_streaming_address': u'10.11.12.64'}, u'return': u'success'}
INFO Gabriel Server :
INFO {u'acc_tcp_streaming_address': u'10.11.12.64',
u'acc_tcp_streaming_port': 10102,
u'result_return_server_list': [u'10.11.12.64:10120'],
u'ucomm_data_address': u'10.11.12.64',
u'ucomm_data_port': 9090,
u'video_tcp_streaming_address': u'10.11.12.64',
u'video_tcp_streaming_port': 10101}
INFO connecting to 10.11.12.64:9090
INFO User communication module is connected
INFO * UCOMM server configuration
INFO - Open TCP Server at ('0.0.0.0', 10120)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO Start forwarding data
INFO Google Glass is connected for (Mobile Video Server)
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread
INFO Google Glass is connected for (Mobile Video Server)
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread
2018-01-30 22:55:48+0000 [-] Log opened.
2018-01-30 22:55:48+0000 [-] WebSocketServerFactory starting on 9000
2018-01-30 22:55:48+0000 [-] Starting factory
2018-01-30 22:55:57+0000 [-] Client connecting: tcp4:127.0.0.1:45555
2018-01-30 22:55:57+0000 [-] server state: []
2018-01-30 22:55:57+0000 [-] WebSocket connection open.
2018-01-30 22:55:57,379 INFO MainProcess resp: {"type": "GET_TRAINING_RESP", "val": false}
2018-01-30 22:55:57,379 INFO MainProcess openface is training?False
Discovery Control VM
[INFO/DetectionProcess] child process calling self.run()
2018-01-30 22:55:57,386 INFO DetectionProcess created
INFO Gabriel Server :
INFO {u'acc_tcp_streaming_address': u'10.11.12.64',
u'acc_tcp_streaming_port': 10102,
u'result_return_server_list': [u'10.11.12.64:10120'],
u'ucomm_data_address': u'10.11.12.64',
u'ucomm_data_port': 9090,
u'video_tcp_streaming_address': u'10.11.12.64',
u'video_tcp_streaming_port': 10101}
TOKEN SIZE OF OFFLOADING ENGINE: 1
INFO Success to connect to (u'10.11.12.64', 10101)
INFO Offloading engine is connected
INFO Start getting data from the server
INFO Start publishing data
INFO new Offlaoding Engine is connected
2018-01-30 22:55:57+0000 [-] Client connecting: tcp4:127.0.0.1:45559
2018-01-30 22:55:57+0000 [-] server state: []
2018-01-30 22:55:57+0000 [-] WebSocket connection open.
[INFO/receive_thread] child process calling self.run()
INFO Google Glass is connected for (Mobile Video Server)
INFO New connection is starting at 1517352972.487156
adding person
training_cnt :0
2018-01-30 22:56:12+0000 [-] ['mic']
INFO returning result: {"training": "mic", "engine_id": "dummy"
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread
INFO Google Glass is connected for (Mobile Video Server)
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread

Could you please give me some help, I am a little bit lost here.

Bests regards,
Micael

[junjuew]

@MSoares 5 gabriel-control processes are not normal. It means the faceswap server failed to start, and upstart kept trying for 5 times. Can you double-check if your VM supports AVX? If not please double-check if you're using the correct image. (see https://cmusatyalab.github.io/faceswap/faq/)

To furthur debug, I suggest
1. use kill_demo.sh in FaceSwap-server/ to kill all faceswap + gabriel processes
2. command "ps -aux | grep python" shouldn't return any processes after 1. If you still see processes, use kill_demo.sh multiple times or use "kill -9 pid" to forcefully kill all remaining processes
2. turn on faceswap debug flag: FaceSwap-server/demo_config.py line 3, change to "DEBUG=True"
3. reset gabriel log by "rm /var/tmp/cloudlet/log-gabriel"

It would be helpful to include the content of /var/tmp/cloudlet/log-gabriel besides stdout

[MSoares]

Dear @junjuew,

when I use kill_demo.sh, it appears to kill gabriel_ucomm, gabriel_control and python, but after a while it start running again, i tried also to delete them using kill -9 PID.

Anyway if I turn on debug and try to make a training, error message "Error" on Mobile will appear and in the logs present on /var/tmp/cloudlet/log-gabriel, we will have this below:

01-31 19:31 main INFO Start RESTful API Server (port :8021)
01-31 19:31 main INFO Start UPnP Server
01-31 19:31 main INFO Start monitoring offload engines
01-31 19:32 face_swap INFO resp: {"type": "GET_TRAINING_RESP", "val": false}
01-31 19:32 face_swap INFO openface is training?False
01-31 19:32 face_swap INFO created
01-31 19:32 gabriel.proxy.common INFO Gabriel Server :
01-31 19:32 gabriel.proxy.common INFO {u'acc_tcp_streaming_address': u'10.11.12.65',
u'acc_tcp_streaming_port': 10102,
u'result_return_server_list': [u'10.11.12.65:10120'],
u'ucomm_data_address': u'10.11.12.65',
u'ucomm_data_port': 9090,
u'video_tcp_streaming_address': u'10.11.12.65',
u'video_tcp_streaming_port': 10101}
01-31 19:32 gabriel.proxy.common INFO Success to connect to (u'10.11.12.65', 10101)
01-31 19:32 gabriel.proxy.common INFO Start getting data from the server
01-31 19:32 gabriel.proxy.common INFO Start publishing data
01-31 19:32 gabriel.proxy.common INFO New connection is starting at 1517427159.785977
01-31 19:32 face_swap DEBUG detector run: 101.779937744
01-31 19:32 face_swap DEBUG # detected : 0
01-31 19:32 face_swap DEBUG No faces detected in training frame. abandon frame
01-31 19:32 gabriel.proxy.common INFO returning result: {"training": "mic", "engine_id": "dummy"
01-31 19:33 gabriel.proxy.common INFO New connection is starting at 1517427186.994237
01-31 19:33 face_swap DEBUG detector run: 70.9369182587
01-31 19:33 face_swap DEBUG # detected : 0
01-31 19:33 face_swap DEBUG No faces detected in training frame. abandon frame
01-31 19:33 gabriel.proxy.common INFO returning result: {"training": "nin", "engine_id": "dummy"
01-31 19:33 gabriel.proxy.common INFO New connection is starting at 1517427221.284511
01-31 19:33 face_swap DEBUG detector run: 70.7261562347
01-31 19:33 face_swap DEBUG # detected : 0
01-31 19:33 face_swap DEBUG No faces detected in training frame. abandon frame
01-31 19:33 gabriel.proxy.common INFO returning result: {"training": "pgg", "engine_id": "dummy"
01-31 19:34 face_swap INFO detection process shutdown!
01-31 19:34 face_swap INFO sync faces thread shutdown!
01-31 19:34 face_swap DEBUG transformer terminate!
01-31 19:34 gabriel.proxy.common INFO App thread terminated
01-31 19:34 gabriel.proxy.common INFO Streaming thread terminated
01-31 19:34 gabriel.proxy.common INFO Publish thread terminated

[junjuew]

@MSoares
Gabriel is the underlying communication library for faceswap. start_demo.sh starts gabriel and the demo. kill_demo.sh kills the demo app and gabriel. If you just started the VM and upstart kept trying to launch faceswap, you can use "sudo service faceswap stop" to prevent upstart from trying. And then use kill_demo.sh to make sure system is clean and use start_demo.sh to start a new demo.

From the above log, it seems that the connection was successfully established and the backend processed a few frames before crashed.

Can you provide the stdout from start_demo.sh? What's the compute power of your VM?

[MSoares]

@junjuew,

if I launch VM and use commands "sudo service faceswap stop", then "kill_demo.sh", it will erase all gabriel and python components once for all. After I can launch demo with "start_demo.sh" but error is the same:

faceswap-admin@faceswap-server-debug:~$ sudo service faceswap stop
sudo: unable to resolve host faceswap-server-debug
faceswap stop/waiting
faceswap-admin@faceswap-server-debug:~$ cd FaceSwap-server/
faceswap-admin@faceswap-server-debug:~/FaceSwap-server$ ls
LICENSE camShift.pyc install-deps.sh openfaceClient.py
MyUtils.py demo_config.py kill_demo.sh openfaceClient.pyc
MyUtils.pyc demo_config.pyc mirror-proxy.py requirements.txt
NetworkProtocol.py face_swap.py notes.txt script
NetworkProtocol.pyc face_swap.pyc openface-server start_demo.sh
camShift.py faceswap-proxy.py openface-server.log
faceswap-admin@faceswap-server-debug:~/FaceSwap-server$ ./kill_demo.sh
killing gabriel...
start_demo
openface-server/start_server.sh
faceswap-proxy.py
gabriel-upnp
gabriel_upnp_server
gabriel_REST_server
gabriel-ucomm
gabriel-control
cloudlet-demo-openface-server
openface
All killed
faceswap-admin@faceswap-server-debug:~/FaceSwap-server$ top

top - 22:12:35 up 3 min, 1 user, load average: 0.15, 0.25, 0.12
Tasks: 80 total, 1 running, 79 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.2 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 4048352 total, 404816 used, 3643536 free, 18384 buffers
KiB Swap: 0 total, 0 used, 0 free. 312468 cached Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1795 faceswa+ 20 0 23536 1424 1024 R 0.3 0.0 0:00.06 top
1 root 20 0 33764 3000 1444 S 0.0 0.1 0:00.79 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
4 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:+
6 root 20 0 0 0 0 S 0.0 0.0 0:00.03 kworker/u4+
7 root 20 0 0 0 0 S 0.0 0.0 0:00.08 rcu_sched
8 root 20 0 0 0 0 S 0.0 0.0 0:00.07 rcuos/0
9 root 20 0 0 0 0 S 0.0 0.0 0:00.03 rcuos/1
10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/0
12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/1
13 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
14 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
15 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
16 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/1
faceswap-admin@faceswap-server-debug:~/FaceSwap-server$ rm /var/tmp/cloudlet/log-gabriel
faceswap-admin@faceswap-server-debug:~/FaceSwap-server$ ./start_demo.sh
sourcing torch:
/home/faceswap-admin/torch/install/bin/torch-activate
launching FaceSwap server at dir /home/faceswap-admin/FaceSwap-server
INFO Start RESTful API Server (port :8021)
INFO Start UPnP Server
INFO Start monitoring offload engines
INFO * Mobile server() configuration
INFO - Open TCP Server at ('0.0.0.0', 9098)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * Mobile server() configuration
INFO - Open TCP Server at ('0.0.0.0', 9099)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * Mobile server() configuration
INFO - Open TCP Server at ('0.0.0.0', 9101)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * Application server() configuration
INFO - Open TCP Server at ('0.0.0.0', 10101)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * Application server() configuration
INFO - Open TCP Server at ('0.0.0.0', 10102)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * UComm server configuration
INFO - Open TCP Server at ('0.0.0.0', 9090)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
{u'objects': {u'acc_tcp_streaming_address': u'10.11.12.67', u'video_tcp_streaming_port': 10101, u'result_return_server_list': [u'10.11.12.67:10120'], u'ucomm_data_address': u'10.11.12.67', u'ucomm_data_port': 9090, u'acc_tcp_streaming_port': 10102, u'video_tcp_streaming_address': u'10.11.12.67'}, u'return': u'success'}
INFO Gabriel Server :
INFO {u'acc_tcp_streaming_address': u'10.11.12.67',
u'acc_tcp_streaming_port': 10102,
u'result_return_server_list': [],
u'ucomm_data_address': u'10.11.12.67',
u'ucomm_data_port': 9090,
u'video_tcp_streaming_address': u'10.11.12.67',
u'video_tcp_streaming_port': 10101}
INFO connecting to 10.11.12.67:9090
INFO User communication module is connected
INFO * UCOMM server configuration
INFO - Open TCP Server at ('0.0.0.0', 10120)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO Start forwarding data
2018-01-31 22:16:09+0000 [-] Log opened.
2018-01-31 22:16:09+0000 [-] WebSocketServerFactory starting on 9000
2018-01-31 22:16:09+0000 [-] Starting factory
2018-01-31 22:16:18+0000 [-] Client connecting: tcp4:127.0.0.1:45555
2018-01-31 22:16:18+0000 [-] server state: []
2018-01-31 22:16:18+0000 [-] WebSocket connection open.
2018-01-31 22:16:18,748 INFO MainProcess resp: {"type": "GET_TRAINING_RESP", "val": false}
2018-01-31 22:16:18,748 INFO MainProcess openface is training?False
Discovery Control VM
[INFO/DetectionProcess] child process calling self.run()
2018-01-31 22:16:18,753 INFO DetectionProcess created
INFO Gabriel Server :
INFO {u'acc_tcp_streaming_address': u'10.11.12.67',
u'acc_tcp_streaming_port': 10102,
u'result_return_server_list': [u'10.11.12.67:10120'],
u'ucomm_data_address': u'10.11.12.67',
u'ucomm_data_port': 9090,
u'video_tcp_streaming_address': u'10.11.12.67',
u'video_tcp_streaming_port': 10101}
TOKEN SIZE OF OFFLOADING ENGINE: 1
INFO Success to connect to (u'10.11.12.67', 10101)
INFO Start getting data from the server
INFO Offloading engine is connected
INFO Start publishing data
INFO new Offlaoding Engine is connected
2018-01-31 22:16:19+0000 [-] Client connecting: tcp4:127.0.0.1:45559
2018-01-31 22:16:19+0000 [-] server state: []
2018-01-31 22:16:19+0000 [-] WebSocket connection open.
[INFO/receive_thread] child process calling self.run()
INFO Google Glass is connected for (Mobile Video Server)
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread
INFO Google Glass is connected for (Mobile Video Server)
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread
INFO Google Glass is connected for (Mobile Video Server)
INFO New connection is starting at 1517437016.344165
adding person
training_cnt :0
2018-01-31 22:16:56+0000 [-] ['mic']
INFO returning result: {"training": "mic", "engine_id": "dummy"
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread
INFO Google Glass is connected for (Mobile Video Server)
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread
INFO Google Glass is connected for (Mobile Video Server)
INFO New connection is starting at 1517437041.547975
adding person
2018-01-31 22:17:21+0000 [-] ['mic', 'mic2']
training_cnt :0
INFO returning result: {"training": "mic2", "engine_id": "dummy
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread
INFO Google Glass is connected for (Mobile Video Server)
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread

Regarding the logs since start_demo:

01-31 22:15 main INFO Start RESTful API Server (port :8021)
01-31 22:15 main INFO Start UPnP Server
01-31 22:15 main INFO Start monitoring offload engines
01-31 22:15 gabriel.control.mobile_server INFO * Mobile server() configuration
01-31 22:15 gabriel.control.mobile_server INFO - Open TCP Server at ('0.0.0.0', 9098)
01-31 22:15 gabriel.control.mobile_server INFO - Disable nagle (No TCP delay) : 1
01-31 22:15 gabriel.control.mobile_server INFO --------------------------------------------------
01-31 22:15 gabriel.control.mobile_server INFO * Mobile server() configuration
01-31 22:15 gabriel.control.mobile_server INFO - Open TCP Server at ('0.0.0.0', 9099)
01-31 22:15 gabriel.control.mobile_server INFO - Disable nagle (No TCP delay) : 1
01-31 22:15 gabriel.control.mobile_server INFO --------------------------------------------------
01-31 22:15 gabriel.control.mobile_server INFO * Mobile server() configuration
01-31 22:15 gabriel.control.mobile_server INFO - Open TCP Server at ('0.0.0.0', 9101)
01-31 22:15 gabriel.control.mobile_server INFO - Disable nagle (No TCP delay) : 1
01-31 22:15 gabriel.control.mobile_server INFO --------------------------------------------------
01-31 22:15 gabriel.control.app_server INFO * Application server() configuration
01-31 22:15 gabriel.control.app_server INFO - Open TCP Server at ('0.0.0.0', 10101)
01-31 22:15 gabriel.control.app_server INFO - Disable nagle (No TCP delay) : 1
01-31 22:15 gabriel.control.app_server INFO --------------------------------------------------
01-31 22:15 gabriel.control.app_server INFO * Application server() configuration
01-31 22:15 gabriel.control.app_server INFO - Open TCP Server at ('0.0.0.0', 10102)
01-31 22:15 gabriel.control.app_server INFO - Disable nagle (No TCP delay) : 1
01-31 22:15 gabriel.control.app_server INFO --------------------------------------------------
01-31 22:15 gabriel.control.ucomm_relay INFO * UComm server configuration
01-31 22:15 gabriel.control.ucomm_relay INFO - Open TCP Server at ('0.0.0.0', 9090)
01-31 22:15 gabriel.control.ucomm_relay INFO - Disable nagle (No TCP delay) : 1
01-31 22:15 gabriel.control.ucomm_relay INFO --------------------------------------------------
01-31 22:16 gabriel.control.ucomm_relay INFO User communication module is connected
01-31 22:16 face_swap INFO resp: {"type": "GET_TRAINING_RESP", "val": false}
01-31 22:16 face_swap INFO openface is training?False
01-31 22:16 face_swap INFO created
01-31 22:16 gabriel.proxy.common INFO Gabriel Server :
01-31 22:16 gabriel.proxy.common INFO {u'acc_tcp_streaming_address': u'10.11.12.67',
u'acc_tcp_streaming_port': 10102,
u'result_return_server_list': [u'10.11.12.67:10120'],
u'ucomm_data_address': u'10.11.12.67',
u'ucomm_data_port': 9090,
u'video_tcp_streaming_address': u'10.11.12.67',
u'video_tcp_streaming_port': 10101}
01-31 22:16 gabriel.proxy.common INFO Success to connect to (u'10.11.12.67', 10101)
01-31 22:16 gabriel.proxy.common INFO Start getting data from the server
01-31 22:16 gabriel.control.app_server INFO Offloading engine is connected
01-31 22:16 gabriel.proxy.common INFO Start publishing data
01-31 22:16 gabriel.control.mobile_server INFO Google Glass is connected for (Mobile Video Server)
01-31 22:16 gabriel.control.mobile_server INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server
01-31 22:16 gabriel.control.mobile_server DEBUG Recv 0 data at Mobile Video Server

01-31 22:16 gabriel.control.mobile_server INFO Mobile Video Server terminate thread
01-31 22:16 gabriel.control.mobile_server INFO Google Glass is connected for (Mobile Video Server)
01-31 22:16 gabriel.control.mobile_server INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

01-31 22:16 gabriel.control.mobile_server DEBUG Recv 0 data at Mobile Video Server

01-31 22:16 gabriel.control.mobile_server INFO Mobile Video Server terminate thread
01-31 22:16 gabriel.control.mobile_server INFO Google Glass is connected for (Mobile Video Server)
1-31 22:16 gabriel.proxy.common INFO New connection is starting at 1517437016.344165
01-31 22:16 gabriel.proxy.common INFO returning result: {"training": "mic", "engine_id": "dummy"
01-31 22:17 gabriel.control.mobile_server INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

01-31 22:17 gabriel.control.mobile_server DEBUG Recv 0 data at Mobile Video Server

01-31 22:17 gabriel.control.mobile_server INFO Mobile Video Server terminate thread
01-31 22:17 gabriel.control.mobile_server INFO Google Glass is connected for (Mobile Video Server)
01-31 22:17 gabriel.control.mobile_server INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

01-31 22:17 gabriel.control.mobile_server DEBUG Recv 0 data at Mobile Video Server

01-31 22:17 gabriel.control.mobile_server INFO Mobile Video Server terminate thread
01-31 22:17 gabriel.control.mobile_server INFO Google Glass is connected for (Mobile Video Server)
01-31 22:17 gabriel.proxy.common INFO New connection is starting at 1517437041.547975
01-31 22:17 gabriel.proxy.common INFO returning result: {"training": "mic2", "engine_id": "dummy
01-31 22:17 gabriel.control.mobile_server INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

01-31 22:17 gabriel.control.mobile_server DEBUG Recv 0 data at Mobile Video Server

01-31 22:17 gabriel.control.mobile_server INFO Mobile Video Server terminate thread
01-31 22:17 gabriel.control.mobile_server INFO Google Glass is connected for (Mobile Video Server)
01-31 22:17 gabriel.control.mobile_server INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

01-31 22:17 gabriel.control.mobile_server DEBUG Recv 0 data at Mobile Video Server
01-31 22:17 gabriel.control.mobile_server INFO Mobile Video Server terminate thread
01-31 22:21 face_swap INFO detection process shutdown!
01-31 22:21 face_swap INFO sync faces thread shutdown!
01-31 22:21 gabriel.proxy.common INFO Streaming thread terminated
01-31 22:21 gabriel.proxy.common INFO App thread terminated
01-31 22:21 gabriel.proxy.common INFO Publish thread terminated
01-31 22:21 gabriel.control.app_server DEBUG Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/app_server.py", line 84, in handle
LOG.e("Offload engine should not return directly")
AttributeError: 'Logger' object has no attribute 'e'

01-31 22:21 gabriel.control.app_server DEBUG 'Logger' object has no attribute 'e'
01-31 22:21 gabriel.control.app_server INFO terminate thre$

At last, my VM specification data is:
Instance Overview
Information

Name
Faceswap-Server
ID
5d88f966-2f30-40a4-8a89-d209dcc8737e
Status
Active
Availability Zone
nova
Created
Jan. 31, 2018, 10:08 p.m.
Time Since Created
18 minutes
Host
mickael-SATELLITE-L755

Specs

Flavor
m1.medium
Flavor ID
3
RAM
4GB
VCPUs
2 VCPU
Disk
40GB

[junjuew]

@MSoares
I don't see anything wrong in the stdout nor log. I couldn't reproduce this error on my side either.

Can you turn on the faceswap debug flag (FaceSwap-server/demo_config.py line 3, change to "DEBUG=True") and post the stdout and log with debug messages? BTW, there is no need to re-create a VM every time.

[MSoares]

@junjuew,

sorry my bad I forget to put debug function ON.

I have make 2 test with user "mic" and "mic2" and the stdout is:

faceswap-admin@faceswap-server-debug:~/FaceSwap-server$ ./start_demo.sh
sourcing torch:
/home/faceswap-admin/torch/install/bin/torch-activate
launching FaceSwap server at dir /home/faceswap-admin/FaceSwap-server
INFO Start RESTful API Server (port :8021)
INFO Start UPnP Server
INFO Start monitoring offload engines
INFO * Mobile server() configuration
INFO - Open TCP Server at ('0.0.0.0', 9098)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * Mobile server() configuration
INFO - Open TCP Server at ('0.0.0.0', 9099)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * Mobile server() configuration
INFO - Open TCP Server at ('0.0.0.0', 9101)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * Application server() configuration
INFO - Open TCP Server at ('0.0.0.0', 10101)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * Application server() configuration
INFO - Open TCP Server at ('0.0.0.0', 10102)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO * UComm server configuration
INFO - Open TCP Server at ('0.0.0.0', 9090)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
{u'objects': {u'acc_tcp_streaming_address': u'10.11.12.67', u'video_tcp_streaming_port': 10101, u'result_return_server_list': [u'10.11.12.67:10120'], u'ucomm_data_address': u'10.11.12.67', u'ucomm_data_port': 9090, u'acc_tcp_streaming_port': 10102, u'video_tcp_streaming_address': u'10.11.12.67'}, u'return': u'success'}
INFO Gabriel Server :
INFO {u'acc_tcp_streaming_address': u'10.11.12.67',
u'acc_tcp_streaming_port': 10102,
u'result_return_server_list': [],
u'ucomm_data_address': u'10.11.12.67',
u'ucomm_data_port': 9090,
u'video_tcp_streaming_address': u'10.11.12.67',
u'video_tcp_streaming_port': 10101}
INFO connecting to 10.11.12.67:9090
INFO User communication module is connected
INFO * UCOMM server configuration
INFO - Open TCP Server at ('0.0.0.0', 10120)
INFO - Disable nagle (No TCP delay) : 1
INFO --------------------------------------------------
INFO Start forwarding data
2018-01-31 22:58:17+0000 [-] Log opened.
2018-01-31 22:58:17+0000 [-] WebSocketServerFactory starting on 9000
2018-01-31 22:58:17+0000 [-] Starting factory
2018-01-31 22:58:25+0000 [-] Client connecting: tcp4:127.0.0.1:45579
2018-01-31 22:58:25+0000 [-] server state: []
2018-01-31 22:58:25+0000 [-] WebSocket connection open.
2018-01-31 22:58:25,701 INFO MainProcess resp: {"type": "GET_TRAINING_RESP", "val": false}
2018-01-31 22:58:25,702 INFO MainProcess openface is training?False
[DEBUG/MainProcess] created semlock with handle 140329532518400
[DEBUG/MainProcess] created semlock with handle 140329532514304
[DEBUG/MainProcess] created semlock with handle 140329532510208
[DEBUG/MainProcess] created semlock with handle 140329532506112
[DEBUG/MainProcess] created semlock with handle 140329532502016
[DEBUG/MainProcess] created semlock with handle 140329532497920
[DEBUG/MainProcess] created semlock with handle 140329531686912
[DEBUG/MainProcess] created semlock with handle 140329531682816
[DEBUG/MainProcess] created semlock with handle 140329531678720
[DEBUG/MainProcess] created semlock with handle 140329531674624
Discovery Control VM
[DEBUG/DetectionProcess] Queue.afterfork()
[DEBUG/DetectionProcess] Queue.afterfork()
[DEBUG/DetectionProcess] Queue.afterfork()
[INFO/DetectionProcess] child process calling self.run()
[DEBUG/DetectionProcess] created semlock with handle 140329531670528
[DEBUG/DetectionProcess] created semlock with handle 140329531666432
[DEBUG/DetectionProcess] created semlock with handle 140329531662336
[DEBUG/DetectionProcess] created semlock with handle 140329531658240
[DEBUG/DetectionProcess] created semlock with handle 140329531654144
2018-01-31 22:58:25,708 INFO DetectionProcess created
INFO Gabriel Server :
INFO {u'acc_tcp_streaming_address': u'10.11.12.67',
u'acc_tcp_streaming_port': 10102,
u'result_return_server_list': [u'10.11.12.67:10120'],
u'ucomm_data_address': u'10.11.12.67',
u'ucomm_data_port': 9090,
u'video_tcp_streaming_address': u'10.11.12.67',
u'video_tcp_streaming_port': 10101}
TOKEN SIZE OF OFFLOADING ENGINE: 1
INFO Success to connect to (u'10.11.12.67', 10101)
[DEBUG/MainProcess] created semlock with handle 140329531670528
[DEBUG/MainProcess] created semlock with handle 140329531666432
[DEBUG/MainProcess] created semlock with handle 140329531662336
[DEBUG/MainProcess] Queue.afterfork()
INFO Offloading engine is connected
[DEBUG/MainProcess] created semlock with handle 140329531658240
INFO Start getting data from the server
[DEBUG/MainProcess] created semlock with handle 140329531654144
[DEBUG/MainProcess] created semlock with handle 140329531650048
[DEBUG/MainProcess] Queue.afterfork()
[DEBUG/MainProcess] created semlock with handle 140329531645952
[DEBUG/MainProcess] created semlock with handle 140329531641856
[DEBUG/MainProcess] created semlock with handle 140329531637760
[DEBUG/MainProcess] Queue.afterfork()
INFO Start publishing data
INFO new Offlaoding Engine is connected
2018-01-31 22:58:26+0000 [-] Client connecting: tcp4:127.0.0.1:45583
2018-01-31 22:58:26+0000 [-] server state: []
2018-01-31 22:58:26+0000 [-] WebSocket connection open.
[DEBUG/DetectionProcess] created semlock with handle 140329531650048
[DEBUG/DetectionProcess] created semlock with handle 140329531645952
[DEBUG/DetectionProcess] created semlock with handle 140329531641856
[DEBUG/DetectionProcess] created semlock with handle 140329531637760
[DEBUG/DetectionProcess] created semlock with handle 140329531633664
[DEBUG/receive_thread] Queue.afterfork()
[DEBUG/receive_thread] Queue.afterfork()
[DEBUG/receive_thread] Queue.afterfork()
[INFO/receive_thread] child process calling self.run()
INFO Google Glass is connected for (Mobile Video Server)
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread
INFO Google Glass is connected for (Mobile Video Server)
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread
INFO Google Glass is connected for (Mobile Video Server)
INFO New connection is starting at 1517439530.972028
packet interval: 25725
adding person
training_cnt :0
2018-01-31 22:58:50+0000 [-] ['mic']
2018-01-31 22:58:51,130 DEBUG MainProcess detector run: 146.707057953
2018-01-31 22:58:51,131 DEBUG MainProcess # detected : 0
2018-01-31 22:58:51,132 DEBUG MainProcess No faces detected in training frame. abandon frame
total processing time: 159.695863724
[DEBUG/MainProcess] Queue.startthread()
[DEBUG/MainProcess] doing self._thread.start()
[DEBUG/MainProcess] starting thread to feed data to pipe
[DEBUG/MainProcess] ... done self._thread.start()
INFO returning result: {"training": "mic", "engine_id": "dummy"
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread
INFO Google Glass is connected for (Mobile Video Server)
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread
INFO Google Glass is connected for (Mobile Video Server)
INFO New connection is starting at 1517439553.962068
packet interval: 22829
adding person
training_cnt :0
2018-01-31 22:59:13+0000 [-] ['mic', 'mic2']
2018-01-31 22:59:14,037 DEBUG MainProcess detector run: 71.0678100586
2018-01-31 22:59:14,037 DEBUG MainProcess # detected : 0
2018-01-31 22:59:14,037 DEBUG MainProcess No faces detected in training frame. abandon frame
total processing time: 75.3419399261
INFO returning result: {"training": "mic2", "engine_id": "dummy
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread
INFO Google Glass is connected for (Mobile Video Server)
INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

INFO Mobile Video Server terminate thread
^C[INFO/receive_thread] process shutting down
[DEBUG/receive_thread] running all "atexit" finalizers with priority >= 0
[DEBUG/receive_thread] running the remaining "atexit" finalizers
[INFO/DetectionProcess] process shutting down
user exits
[DEBUG/DetectionProcess] running all "atexit" finalizers with priority >= 0
[INFO/DetectionProcess] calling join() for process receive_thread
Process receive_thread:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self.target(*self.args, **self._kwargs)
File "/home/faceswap-admin/FaceSwap-server/openfaceClient.py", line 165, in async_on_receive
inputready,outputready,exceptready = select.select(input,[],[])
KeyboardInterrupt
[INFO/receive_thread] process exiting with exitcode 1
[DEBUG/DetectionProcess] running the remaining "atexit" finalizers
Process DetectionProcess:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self.target(*self.args, **self._kwargs)
File "/home/faceswap-admin/FaceSwap-server/face_swap.py", line 417, in detection_update_thread
frame = img_queue.get(timeout=1)
File "/usr/lib/python2.7/multiprocessing/queues.py", line 131, in get
if timeout < 0 or not self._poll(timeout):
KeyboardInterrupt
[INFO/DetectionProcess] process exiting with exitcode 1
2018-01-31 23:00:04,770 INFO MainProcess detection process shutdown!
2018-01-31 23:00:04,770 INFO MainProcess sync faces thread shutdown!
2018-01-31 23:00:04+0000 [-] WebSocket connection closed: connection was closed uncleanly (peer dropped the TCP connection without previous WebSocket closing handshake)
2018-01-31 23:00:04+0000 [-] WebSocket connection closed: None
2018-01-31 23:00:04,772 DEBUG MainProcess transformer terminate!
[DEBUG/MainProcess] Queue.startthread()
[DEBUG/MainProcess] doing self._thread.start()
[DEBUG/MainProcess] starting thread to feed data to pipe
[DEBUG/MainProcess] ... done self._thread.start()
[DEBUG/MainProcess] Queue.startthread()
INFO App thread terminated
[DEBUG/MainProcess] doing self._thread.start()
[DEBUG/MainProcess] ... done self._thread.start()
[DEBUG/MainProcess] telling queue thread to quit
INFO Streaming thread terminated
INFO terminate thread
[DEBUG/MainProcess] feeder thread got sentinel -- exiting
[DEBUG/MainProcess] joining queue thread
[DEBUG/MainProcess] ... queue thread already dead
[DEBUG/MainProcess] starting thread to feed data to pipe
INFO Publish thread terminated
DEBUG Socket is closed
INFO Offloading engine is disconnected
INFO <main.UCommServerHandler object at 0x7f118ecd55d0> terminate thread
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[DEBUG/MainProcess] telling queue thread to quit
[DEBUG/MainProcess] telling queue thread to quit
[DEBUG/MainProcess] feeder thread got sentinel -- exiting
[DEBUG/MainProcess] running the remaining "atexit" finalizers
[DEBUG/MainProcess] joining queue thread
[DEBUG/MainProcess] ... queue thread joined
[DEBUG/MainProcess] joining queue thread
[DEBUG/MainProcess] feeder thread got sentinel -- exiting
[DEBUG/MainProcess] ... queue thread joined
faceswap-admin@faceswap-server-debug:~/FaceSwap-server$

Regarding the logs on /var/tmp/cloudlet/log-gabriel:

01-31 22:58 main INFO Start RESTful API Server (port :8021)
01-31 22:58 main INFO Start UPnP Server
01-31 22:58 main INFO Start monitoring offload engines
01-31 22:58 gabriel.control.mobile_server INFO * Mobile server() configuration
01-31 22:58 gabriel.control.mobile_server INFO - Open TCP Server at ('0.0.0.0', 9098)
01-31 22:58 gabriel.control.mobile_server INFO - Disable nagle (No TCP delay) : 1
01-31 22:58 gabriel.control.mobile_server INFO --------------------------------------------------
01-31 22:58 gabriel.control.mobile_server INFO * Mobile server() configuration
01-31 22:58 gabriel.control.mobile_server INFO - Open TCP Server at ('0.0.0.0', 9099)
01-31 22:58 gabriel.control.mobile_server INFO - Disable nagle (No TCP delay) : 1
01-31 22:58 gabriel.control.mobile_server INFO --------------------------------------------------
01-31 22:58 gabriel.control.mobile_server INFO * Mobile server() configuration
01-31 22:58 gabriel.control.mobile_server INFO - Open TCP Server at ('0.0.0.0', 9101)
01-31 22:58 gabriel.control.mobile_server INFO - Disable nagle (No TCP delay) : 1
01-31 22:58 gabriel.control.mobile_server INFO --------------------------------------------------
01-31 22:58 gabriel.control.app_server INFO * Application server() configuration
01-31 22:58 gabriel.control.app_server INFO - Open TCP Server at ('0.0.0.0', 10101)
01-31 22:58 gabriel.control.app_server INFO - Disable nagle (No TCP delay) : 1
01-31 22:58 gabriel.control.app_server INFO --------------------------------------------------
01-31 22:58 gabriel.control.app_server INFO * Application server() configuration
01-31 22:58 gabriel.control.app_server INFO - Open TCP Server at ('0.0.0.0', 10102)
01-31 22:58 gabriel.control.app_server INFO - Disable nagle (No TCP delay) : 1
01-31 22:58 gabriel.control.app_server INFO --------------------------------------------------
01-31 22:58 gabriel.control.ucomm_relay INFO * UComm server configuration
01-31 22:58 gabriel.control.ucomm_relay INFO - Open TCP Server at ('0.0.0.0', 9090)
01-31 22:58 gabriel.control.ucomm_relay INFO - Disable nagle (No TCP delay) : 1
01-31 22:58 gabriel.control.ucomm_relay INFO --------------------------------------------------
01-31 22:58 gabriel.control.ucomm_relay INFO User communication module is connected
01-31 22:58 face_swap INFO resp: {"type": "GET_TRAINING_RESP", "val": false}
01-31 22:58 face_swap INFO openface is training?False
01-31 22:58 face_swap INFO created
01-31 22:58 gabriel.proxy.common INFO Gabriel Server :
01-31 22:58 gabriel.proxy.common INFO {u'acc_tcp_streaming_address': u'10.11.12.67',
u'acc_tcp_streaming_port': 10102,
u'result_return_server_list': [u'10.11.12.67:10120'],
u'ucomm_data_address': u'10.11.12.67',
u'ucomm_data_port': 9090,
u'video_tcp_streaming_address': u'10.11.12.67',
u'video_tcp_streaming_port': 10101}
01-31 22:58 gabriel.proxy.common INFO Success to connect to (u'10.11.12.67', 10101)
01-31 22:58 gabriel.control.app_server INFO Offloading engine is connected
01-31 22:58 gabriel.proxy.common INFO Start getting data from the server
01-31 22:58 gabriel.proxy.common INFO Start publishing data
01-31 22:58 gabriel.control.mobile_server INFO Google Glass is connected for (Mobile Video Server)
01-31 22:58 gabriel.control.mobile_server INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

01-31 22:58 gabriel.control.mobile_server DEBUG Recv 0 data at Mobile Video Server

01-31 22:58 gabriel.control.mobile_server INFO Mobile Video Server terminate thread
01-31 22:58 gabriel.control.mobile_server INFO Google Glass is connected for (Mobile Video Server)
01-31 22:58 gabriel.control.mobile_server INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server
01-31 22:58 gabriel.control.mobile_server DEBUG Recv 0 data at Mobile Video Server

01-31 22:58 gabriel.control.mobile_server INFO Mobile Video Server terminate thread
01-31 22:58 gabriel.control.mobile_server INFO Google Glass is connected for (Mobile Video Server)
01-31 22:58 gabriel.proxy.common INFO New connection is starting at 1517439530.972028
01-31 22:58 face_swap DEBUG detector run: 146.707057953
01-31 22:58 face_swap DEBUG # detected : 0
01-31 22:58 face_swap DEBUG No faces detected in training frame. abandon frame
01-31 22:58 gabriel.proxy.common INFO returning result: {"training": "mic", "engine_id": "dummy"
01-31 22:58 gabriel.control.mobile_server INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

01-31 22:58 gabriel.control.mobile_server DEBUG Recv 0 data at Mobile Video Server

01-31 22:58 gabriel.control.mobile_server INFO Mobile Video Server terminate thread
01-31 22:58 gabriel.control.mobile_server INFO Google Glass is connected for (Mobile Video Server)
01-31 22:59 gabriel.control.mobile_server INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

01-31 22:59 gabriel.control.mobile_server DEBUG Recv 0 data at Mobile Video Server
01-31 22:59 gabriel.control.mobile_server INFO Mobile Video Server terminate thread
01-31 22:59 gabriel.control.mobile_server INFO Google Glass is connected for (Mobile Video Server)
01-31 22:59 gabriel.proxy.common INFO New connection is starting at 1517439553.962068
01-31 22:59 face_swap DEBUG detector run: 71.0678100586
01-31 22:59 face_swap DEBUG # detected : 0
01-31 22:59 face_swap DEBUG No faces detected in training frame. abandon frame
01-31 22:59 gabriel.proxy.common INFO returning result: {"training": "mic2", "engine_id": "dummy
01-31 22:59 gabriel.control.mobile_server INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

01-31 22:59 gabriel.control.mobile_server DEBUG Recv 0 data at Mobile Video Server

01-31 22:59 gabriel.control.mobile_server INFO Mobile Video Server terminate thread
01-31 22:59 gabriel.control.mobile_server INFO Google Glass is connected for (Mobile Video Server)
01-31 22:59 gabriel.control.mobile_server INFO Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 98, in handle
self.handleinput_data()
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 134, in handleinput_data
header_size = struct.unpack("!I", self.recvall(4))[0]
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/mobile_server.py", line 71, in recvall
raise MobileCommError("Recv 0 data at %s" % str(self))
MobileCommError: Recv 0 data at Mobile Video Server

01-31 22:59 gabriel.control.mobile_server DEBUG Recv 0 data at Mobile Video Server
01-31 22:59 gabriel.control.mobile_server INFO Mobile Video Server terminate thread
01-31 23:00 face_swap INFO detection process shutdown!
01-31 23:00 face_swap INFO sync faces thread shutdown!
01-31 23:00 face_swap DEBUG transformer terminate!
01-31 23:00 gabriel.proxy.common INFO App thread terminated
01-31 23:00 gabriel.proxy.common INFO Streaming thread terminated
01-31 23:00 gabriel.control.app_server DEBUG Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gabriel/control/app_server.py", line 84, in handle
LOG.e("Offload engine should not return directly")
AttributeError: 'Logger' object has no attribute 'e'

01-31 23:00 gabriel.control.app_server DEBUG 'Logger' object has no attribute 'e'
01-31 23:00 gabriel.control.app_server INFO terminate thre$
01-31 23:00 gabriel.proxy.common INFO Publish thread terminated

Thanks for your estimated help.

[junjuew]

@MSoares

I saw nothing wrong in the server log. It seemed that the client just suddenly disconnected. I launched a AWS instance using the same face swap image. Can you quickly check using your client if the same problem still occur? 52.27.88.197 is the IP. I tested with my google pixel and everything worked fine.

[MSoares]

@junjuew,

I have tested on 52.27.88.197 and it worked fine so mobile application apk is working fine in my side.

[junjuew]

@MSoares So both the client and the server seem to be correct. Perhaps you want to try increase your VM size. I would suggest 8 cores and 16GB RAM.

[MSoares]

@junjuew,
unfortunately I do not have the possibility to make this kind of VM, because of my HW capabilities, I have to choose 2 cores and 4gb ram.

There coud be something missing?

[junjuew]

@MSoares I don't think 2 cores and 4 GB RAM are big enough to run the demo. Both the face detection and the DNN-based recognition in the demo are compute-intensive. The failure is probably due to the computation being so slow that the client timed out or the server run out of memory.

[MSoares]

@junjuew,

I was thinking and i have to double check, it is possible to make 4cores.

I will come back to you as soon possible with the result.

Thanks for your understanding.

[MSoares]

@junjuew,

I tried to launch VM with 4 cores ans 4gb ram but same issue ever.

Is there any possibility to put it working with kind of config?