Apache YuniKorn

From Wikitia
Jump to navigation Jump to search
Apache Yunikorn
Developer(s)Apache Software Foundation
RepositoryYunikorn Repository
Written inGo (programming language)
Operating systemCross-platform
LicenseApache License 2.0
Websiteyunikorn.apache.org

Apache YuniKorn is an Open-source software, lightweight resource scheduler that supports large-scale, multi-tenant, and cloud-native environments. Apache YuniKorn also adapting different ResourceManager YuniKorn's architecture design implementations like Apache Hadoop YARN [1] [2] Its primary purpose was to replace Apache YARN. As for scheduling Apache Spark on Kubernetes and now it became the widely accepted method. [3]

On a large scale, in multi-tenant settings on the one hand, and in dynamically created cloud-native environments on the other, YuniKorn facilitates efficient fine-grained resource sharing for different Spark workloads. By providing a solid foundation for a range of applications, including large-scale data transformation, analytics, and machine learning, YuniKorn enables Apache Spark to become an enterprise-grade fundamental platform for users. [4]

History

In order to build a contemporary method for scheduling massive, distributed operations in a future where clouds predominate, Apache YuniKorn was initially created at Cloudera in 2019.[5] The small group of individuals were searching for a truly solid solution to the scheduling and Kubernetes scheduling issue that they had seen. At that time, they also believed that Yarn and Kubernetes should be merged, therefore they decided on a single scheduling method. Apache incubation beginning on January 21, 2020, and the stable version 0.8.0 was published on May 4, 2020, nearly six months later.[6]

Naming

The meaning of Yunikorn [Y Apache Hadoop + K Kubernetes + Uni Unified] [7] and the purpose of a scheduler support any container orchestration system both on-cloud and on-prem use cases.[8]

Development

Apache YuniKorn is developed under the Apache License 2.0 [9] by the Apache YuniKorn Community within the Apache Software Foundation. The project is driven by over 25 Committer[10]

Overview

Apache YuniKorn is a resource scheduler that is designed for scheduling and managing larger dynamic applications on the container orchestration structure and became an List of Apache Software Foundation projects in March 2022.[11] [12]

The use of conventional scheduling techniques with Big data tools has a number of shortcomings. Due to the absence of fine-grained control over Autoscaling in the majority of resource schedulers, resource usage is often out of sync, autoscaling periods for both upscaling and downscaling are prolonged, and as a result, cloud costs are higher and throughput/performance are decreased.

Resource consumption is enhanced and autoscaling performance is boosted thanks to YuniKorn's Gang scheduling and bin-packing.[13] YuniKorn simultaneously ran periodic Apache Spark tasks and saw almost a 2-fold increase in throughput, with a 2-fold decrease in average job runtime and a 3-fold decrease in scale-up and scale-down latencies for 200 nodes. [14]

There are four components in YuniKorn:[15]

  • yunikorn-core
  • yunikorn-web
  • yunikorn-scheduler-interface
  • yunikorn-k8shim

References

  1. "Welcome to Apache YuniKorn | Apache YuniKorn". yunikorn.apache.org.
  2. "Apache YuniKorn - A Universal Scheduler". The Apache Software Foundation. 24 October 2022.
  3. Desk, AIT News (2022-05-17). "The Apache Software Foundation Announces Apache YuniKorn as a Top-Level Project". AiThority. Retrieved 2022-11-22.
  4. "Apache Spark on Kubernetes: How Apache YuniKorn helps - openthings". my.oschina.net. Retrieved 2022-11-22.
  5. "Apache YuniKorn becomes a Top-Level Project". DataXP. 2022-05-16. Retrieved 2022-11-22.
  6. "The Apache Software Foundation Announcement". sg.news.yahoo.com. Retrieved 2022-11-22.
  7. Yang, WeiWei (17 July 2019). "YuniKorn: a universal resources scheduler". Cloudera Blog.
  8. Schmidt, Julia (2019-07-18). "YuniKorn to unify cross-platform scheduling • DEVCLASS". DEVCLASS. Retrieved 2022-11-22.
  9. Apache YuniKorn License, The Apache Software Foundation, 2022-11-22, retrieved 2022-11-22
  10. "Contributors to apache/yunikorn-core". GitHub. Retrieved 2022-11-22.
  11. Foundation, The Apache Software (2022-05-16). "The Apache Software Foundation Announces Apache® YuniKorn™ as a Top-Level Project". GlobeNewswire News Room. Retrieved 2022-11-22.
  12. "The Apache Software Foundation Announces Apache® YuniKorn™ as a Top-Level Project". news.apache.org. 16 May 2022.
  13. Mayer, Marton (2021-09-02). "Optimizing Cloudera Data Engineering Autoscaling Performance". Cloudera Blog. Retrieved 2022-11-22.
  14. datafoam (2021-09-03). "Optimizing YuniKorn Autoscaling Performance". Datafoam. Retrieved 2022-11-22.
  15. "Apache YuniKorn Repro". The Apache Software Foundation. 23 August 2022.

External links

Add External links

This article "Apache YuniKorn" is from Wikipedia. The list of its authors can be seen in its historical. Articles taken from Draft Namespace on Wikipedia could be accessed on Wikipedia's Draft Namespace.