The stream processing framework HarmonicIO is a prototype that addresses the needs for processing streams based on relatively large individual objects. In this regard, it is a specialized streaming framework well-suited for scientific workflows.
Salman Toor and Oliver Stein presented this work, and our latest publication Smart Resource Management for Data Streaming using an Online Bin-packing Strategy in IEEE BigData 2020. The main contribution in this paper is a new solution based on online bin-packing that improves resource-efficient autoscaling. Our results once again highlight that, in the case of scientific workflows, HarmonicIO performs better than Spark and Kafka based streaming solutions.
The paper will soon be available in thee IEEE Explorer. In the meantime, the preprint is available on ArXiv: https://arxiv.org/abs/2001.10865 .
