As for stream applications, the input data are a real-time data stream that typically experiences significant variations in workload properties.

Lack of input data statistics: Data statistics are almost never available for MapReduce and Spark applications, since data often reside in semi- or un-structured files and are opaque until accessed.

In addition, executing MapReduce or Spark workloads with iterative stages and tasks in parallel or serial makes it challenging to observe and model workload performance.

System scale and complexity: As data analytics platforms have grown in scale and complexity, system administrators may need to configure and tune hundreds to thousands of nodes, some equipped with different CPUs, memory, storage media, and network stacks.

To make matters worse, some parameters might affect the performance of different jobs in different ways, while certain groups of parameters may have dependent effects (i.e., a good setting for one parameter may depend on the setting of a different parameter).

Large and complex parameter space: Hadoop, Spark, and Storm have over 200 configurable parameters each.

The use of automated parameter tuning techniques is a promising, yet challenging approach for optimizing system performance. Improper settings of configuration parameters are shown to have detrimental effects on the overall system performance and stability. However, system performance is directly linked to a vast array of configuration parameters, which control various aspects of system execution, ranging from low-level memory settings and thread counts to higher-level decisions such as resource management and load balancing. Achieving good and robust system performance at such a scale is the foundation for successfully performing timely and cost-effective analytics (either offline or in real-time).

Large-scale data analytics platforms, including both batch data processing systems (e.g., Hadoop MapReduce, Spark ), as well as stream data processing systems (e.g., Apache Storm, Heron, Flink, Samza ), have emerged to assist with the Big Data challenge, i.e., to efficiently collect, process, and analyze massive volumes of heterogeneous data. The continuous growth of the World Wide Web, E-commerce, Internet of Things (IoT), and other applications is generating massive amounts of ever-increasing raw data every day.

Autotune 8.1 present