Towards automatic infrastructure provisioning for highly dynamic streaming applications

Resource type
Conference
Author(s)
P. Zehnder and D. Riemer
Year
2017
Pages
4869-4871
Month
Dec
Book title
2017 IEEE International Conference on Big Data (Big Data)
Abstract
In recent years, there has been an increasing need for flexible stream processing platforms that focus on enabling domain experts to model data processing pipelines using graphical user interfaces. Such pipelines can then be automatically deployed and executed in a distributed big data infrastructure. However, a still remaining problem of such stream systems is the manual management of resources, which could be automated by adding an infrastructure manager. The manager uses available information about running pipelines, especially their semantics. This information combined with the real-time information of the underlying services executing pipelines can be used to automate the configuration of modern cluster management tools. With those tools, it is possible to add new resources into the infrastructure of existing systems and to allocate the running application accordingly. This paper presents a novel approach how information about the application and its usage can be used to adapt the infrastructure, which further reduces the manual configuration effort of such systems.
DOI
10.1109/BigData.2017.8258571
Download .bib
Download .bib
Published by
Philipp Zehnder