i have 2 tornado servers running in same python process. 1 tcpserver, while other httpserver. tcpserver receives data source, , httpserver broadcasts data out clients connected served webpage. have 2 servers communicating data between each other tornado.queues.queue(), wherein tcpserver puts data queue, , httpserver gets data , sends off webpage clients.
the question is, how make take advantage of multiple cores? data coming in once source, makes sense dedicate 1 core tcpserver, since there can arbitrarily many clients i'd spin number of processes each own instance of httpserver serve clients.
from tornado documentation, see if of servers unrelated, i'd have this:
def main(): app = make_app() server = tornado.httpserver.httpserver(app) server.bind(8888) server.start(0) # forks 1 process per cpu ioloop.current().start() however, since have 2 servers running in single process, can't quite mirror that. when try that, error saying that:
traceback (most recent call last): file "./tcp_server.py", line 28, in <module> http_server.start(0) file "/usr/local/lib/python3.5/dist-packages/tornado/tcpserver.py", line 205, in start process.fork_processes(num_processes) file "/usr/local/lib/python3.5/dist-packages/tornado/process.py", line 130, in fork_processes raise runtimeerror("cannot run in multiple processes: ioloop instance " runtimeerror: cannot run in multiple processes: ioloop instance has been initialized. cannot call ioloop.instance() before calling start_processes() how can reorganize startup procedure have tcpserver in 1 process, , multiple instances of httpserver on other processes?
import logging import tornado import tornado.queues buggy_data_server import buggydataserver buggy_http_server import buggyhttpserver if __name__ == "__main__": logging.basicconfig(level=logging.debug) data_queue = tornado.queues.queue() # currently, both servers started in single process. needs # split multiprocess configuration later. data_sock = tornado.netutil.bind_sockets(4242) data_server = buggydataserver(data_queue) # tcpserver data_server.add_sockets(data_sock) http_sock = tornado.netutil.bind_sockets(8080) http_server = buggyhttpserver(data_queue) # httpserver http_server.add_sockets(http_sock) http_server.start(0) tornado.ioloop.ioloop.instance().start()
this falls under "advanced multi-process" category of tcpserver docs. need call fork_processes explicitly instead of letting 1 of servers it. reorder setup calls bind sockets before fork, , add them servers after (as rule of thumb, want little possible before fork):
data_sock = tornado.netutil.bind_sockets(4242) http_sock = tornado.netutil.bind_sockets(8080) tornado.process.fork_processes(0) data_server = buggydataserver(data_queue) # tcpserver data_server.add_sockets(data_sock) http_server = buggyhttpserver(data_queue) # httpserver http_server.add_sockets(http_sock) this starts http server , tcp server in each process. if need run 1 tcp server (and not run http server in process), can @ return value of fork_processes:
http_sock = tornado.netutil.bind_sockets(8080) proc_num = tornado.process.fork_processes(0) if proc_num == 0: # close http sockets in tcp server's process s in http_sock: s.close() buggydataserver(data_queue).listen(4242) else: buggyhttpserver(data_queue).add_sockets(http_sock) 
Comments
Post a Comment