diff --git a/docs/client/advanced/features/engine_pool.rst b/docs/client/advanced/features/engine_pool.rst deleted file mode 100644 index ed97f3c98e4..00000000000 --- a/docs/client/advanced/features/engine_pool.rst +++ /dev/null @@ -1,18 +0,0 @@ -.. Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - -.. http://www.apache.org/licenses/LICENSE-2.0 - -.. Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. - -Enabling Kyuubi Engine Pool -=========================== - diff --git a/docs/deployment/engine_share_level.md b/docs/deployment/engine_share_level.md index 8e6c210efc5..f96e1723a09 100644 --- a/docs/deployment/engine_share_level.md +++ b/docs/deployment/engine_share_level.md @@ -42,12 +42,13 @@ With this feature, Kyuubi give you a more flexible way to handle different big d The current supported share levels are, -| Share Level | Syntax | Scenario | Isolation Degree | Shareability | -|----------------|------------------------------|------------------------------|----------------------------------------------|-----------------------| -| **CONNECTION** | One engine per session | Large-scale ETL
Ad hoc | High | Low | -| **USER** | One engine per user | Ad hoc
Small-scale ETL | Medium | Medium | -| **GROUP** | One engine per primary group | Ad hoc
Small-scale ETL | Low | High | -| **SERVER** | One engine per cluster | Admin | Highest If Secured
Lowest If Unsecured | Admin ONLY If Secured | +| Share Level | Syntax | Scenario | Isolation Degree | Shareability | +|------------------|------------------------------|------------------------------|----------------------------------------------|-----------------------| +| **CONNECTION** | One engine per session | Large-scale ETL
Ad hoc | High | Low | +| **USER** | One engine per user | Ad hoc
Small-scale ETL | Medium | Medium | +| **GROUP** | One engine per primary group | Ad hoc
Small-scale ETL | Low | High | +| **SERVER_LOCAL** | One engine per Kyuubi server | Resource load balancing | Very Low | Very High | +| **SERVER** | One engine per cluster | Admin | Highest If Secured
Lowest If Unsecured | Admin ONLY If Secured | - Better isolation degree of engines gives us better stability of an engine and the query executions running on it. - Better shareability of engines means we are more likely to reuse an engine which is already in full speed. @@ -167,6 +168,21 @@ The session user and the primary group name(as sparkUser/execute user) will be b By default, the sparkUser will be used to check the YARN/HDFS ACLs. If you want fine-grained access control for session user, you need to get it from `SparkContext.getLocalProperty("kyuubi.session.user")` and send it to security service, like Apache Ranger. +### SERVER_LOCAL + +
+ + +
+ +*Figure.4 SERVER_LOCAL Share Level* + +
+ +An engine with SERVER_LOCAL share level is private to a single Kyuubi Server instance and is shared by all user sessions +on that instance.In this mode, an engine instance is created when it is first needed by the Kyuubi Server and is +associated with the host address of that server. + ### SERVER
@@ -174,7 +190,7 @@ If you want fine-grained access control for session user, you need to get it fro
-*Figure.4 SERVER Share Level* +*Figure.5 SERVER Share Level*
@@ -188,6 +204,15 @@ For example, in USER share level, you can use `kyuubi.engine.share.level.subdoma The `kyuubi.engine.share.level.subdomain` shall be configured in the JDBC connection URL to tell the Kyuubi server which engine you want to use. +### Engine Pool + +Engine pool works by creating multiple engine subdomains (e.g., +`engine-pool-0`, `engine-pool-1`) and distributing sessions across these +subdomains. + +The Engine Pool's configurations will not take effect when `kyuubi.engine.share.level.subdomain` is set +or `kyuubi.engine.share.level` is set as `CONNECTION`. + ### Hybrid All supported share levels can be used together in a single Kyuubi server or cluster. @@ -196,7 +221,7 @@ All supported share levels can be used together in a single Kyuubi server or clu - kyuubi.engine.share.level(kyuubi.session.engine.share.level) - Default: USER - - Candidates: USER, CONNECTION, GROUP, SERVER + - Candidates: USER, CONNECTION, GROUP, SERVER_LOCAL, SERVER - Meaning: The base level for how an engine is created, cached and shared to sessions. - Usage: It can be set both in the server configuration file and also connection URL. The latter has higher priority. - kyuubi.session.engine.idle.timeout @@ -209,6 +234,26 @@ All supported share levels can be used together in a single Kyuubi server or clu - Candidates: a valid zookeeper a child node - Meaning: Add a subdomain under the base level to make further isolation for engines - Usage: It can be set both in the server configuration file and also connection URL. The latter has higher priority. +- kyuubi.engine.pool.size + - Default: -1 + - Candidates: a positive integer + - Meaning: The number of engine pools + - Usage: It can be set both in the server configuration file and also connection URL. The latter has higher priority. +- kyuubi.engine.pool.name + - Default: engine-pool + - Candidates: a valid name + - Meaning: The name of the engine pool + - Usage: It can be set both in the server configuration file and also connection URL. The latter has higher priority. +- kyuubi.engine.pool.selectPolicy + - Default: RANDOM + - Candidates: RANDOM, POLLING + - Meaning: The selection policy of the engine pool + - Usage: It can be set both in the server configuration file and also connection URL. The latter has higher priority. +- kyuubi.engine.pool.size.threshold + - Default: 9 + - Candidates: a positive integer + - Meaning: The threshold of the engine pool size + - Usage: It can be set only in the server configuration file. ## Conclusion