Constructing Realtime Simulations Engine: Breaking Shared Memory Limits in High-Level Scripting Languages
DOI:
https://doi.org/10.24191/jcrinn.v11i1.617Keywords:
Distributed systems, Engineering Simulation, inter-process communication, multiprocessing, IPC, Multiprocessing, shared memory, Realtime simulationsAbstract
In critical applications, particularly in areas of Realtime Simulations, usages of Unix-based Shared Memory (Shared Memory) concepts help greatly in managing the system’s resources, to be called into play at any moment by the other subsystems. The fact that it is memory-based makes inter-process transactions very fast, as befitting the requirements of critical systems. Coupled with some semi-low-level compiled languages like C/C++ that incorporates the Unix Shared Memory easily, Shared Memories formed the very basis of most Realtime Simulations Engines. In modern times however, both the computer Operating System (OS) as well as programming languages have evolved: the former to cater more for distributed processes and global internet communications, while the latter into more rapid proto-typing as well as very high-level hierarchy in the man-machine interface, which resulted in many, many new languages being used, the most popular of which would be Python. In the OS side, issues about integrity, safety and privacy drive OS manufacturers to do away with many would-be leeways in which security issues (or lack thereof) could creep in, and the usage of Shared Memories would be the primeval talking point. In short, while operating systems like Windows and macOS still support shared memory mechanisms (e.g., POSIX SHM, memory-mapped files), these are limited to processes on a single host. Creating true shared-memory spaces across multiple networked systems is no longer natively supported. In this paper, we investigate the constraints of Python’s Shared Memory model in distributed contexts and, realizing of its futility in fully utilizing it, propose four practical alternatives to overcome the hurdles, eventually opting one of them after being convinced that it is the nearest in behaviour to the Shared Memory model. This work serves as a guide for developers and engineers facing similar constraints in using Python for distributed high-frequency data sharing.Downloads
References
Bershad, B. N., Anderson, T. E., Lazowska, E. D., & Levy, H. M. (1991). User-level interprocess communication for shared memory multiprocessors. ACM Transactions on Computer Systems (TOCS), 9(2), 175–198. https://doi.org/10.1145/103720.114701
Carlson, J. L. (2013). Redis in action. Manning Publications.
Couvée, P., Lemarinier, P., Cedeyn, A., Agosta, G., Cattaneo, D., Fornaciari, W., Zaccaria, V., Bartolini, A., Torquati, M., Aldinucci, M., Colonnelli, I., Martinelli, A. R., Guimarães, F., Mohr, B., Goglin, B., Clauss, C., Krempel, S., Moschny, T., Pickartz, S., … Vysocký, O. (2023). European pilot for exascale - Software specification (101033975).
Eli, B. (2012, January 24). Distributed computing in Python with multiprocessing. Eli Bendersky's Website. https://eli.thegreenplace.net/2012/01/24/distributed-computing-in-python-with-multiprocessing
Garfinke, T. (2010). Paradigms for virtualization based host security [Doctoral dissertation, Stanford University]. Stanford University Repository.
Heroux, M. A., McInnes, L. C., Thakur, R., Vetter, J. S., Li, S., Ahrens, J., Munson, T., & Mohror, K. (2020). ECP software technology capability assessment report – public (ECP-RPT-ST). Exascale Computing Project. https://doi.org/10.2172/1760096
Islam, A. A. R., Narayanan, A., York, C., & Dai, D. (2020). A performance study of Optane persistent memory: From indexing data structures’ perspective. In 36th International Conference on Massive Storage Systems and Technology (MSST 2020).
Kerrisk, M. (2010). The Linux programming interface: A Linux and UNIX system programming handbook. No Starch Press.
Kumar, A. S. (2013). Virtualizing Intelligent River: A comparative study of alternative virtualization technologies [Master's thesis, Clemson University]. Clemson University TigerPrints.
Leite, L., & Tretyak, A. (2015). In-memory databases and Redis (Technical Report).
Liu, Q., & Yuan, H. (2019). A high performance memory key-value database based on Redis. Journal of Computers, 14(3), 170–183. https://doi.org/10.17706/jcp.14.3.170-183
McKusick, M. K., Neville-Neil, G. V., & Watson, R. N. M. (2014). Design and implementation of the FreeBSD operating system (2nd ed.). Addison-Wesley Professional.
Moritz, P., Nishihara, R., Wang, S., Tumanov, A., Liaw, R., Liang, E., Elibol, M., Yang, Z., Paul, W., Jordan, M. I., & Stoica, I. (2018). Ray: A distributed framework for emerging AI applications. In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2018) (pp. 561–577).
Python Software Foundation. (2023). Multiprocessing — Process-based parallelism. Python 3.12 Documentation. https://docs.python.org/3/library/multiprocessing.html
Rockenbach, D. A., Anderle, N., Griebler, D., & Souza, S. (2017). Estudo comparativo de banco de dados chave-valor com armazenamento em memória. In 13th Escola Regional de Banco de Dados (ERBD) (pp. 67–76). http://www.upf.br/_uploads/Conteudo/erbd2017/anais_ERBD2017_final_pos.pdf
Steven, R., Ferner, B., & Rudoff, A. M. (1998). UNIX network programming: The sockets networking API (3rd ed., vol. 1). Addison-Wesley.
Steven, W. R., & Rago, S. A. (2013). Advanced programming in the UNIX® environment (3rd ed.). Addison-Wesley Professional.
Turkstra, J. A. (2013). Metachory: An unprivileged OS kernel for general purpose distributed computing (Vol. 9) [Doctoral dissertation, Purdue University]. Purdue e-Pubs.
