Skip to content

Commit 5200518

Browse files
popcornylupopcornylu
authored
Merge pull request #57 from InfuseAI/feature/sc-21471/datasets
add admin datasets group
2 parents 363b35c + 3a6f14e commit 5200518

23 files changed

+1617
-38
lines changed

‎docs/CLI/admin/datasets.md

Lines changed: 446 additions & 0 deletions
Large diffs are not rendered by default.

‎docs/CLI/apps.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -214,4 +214,4 @@ stop: False
214214
status: Ready
215215
message: Deployment is ready
216216
pods: [{'logEndpoint': 'http://primehub-python-sdk.primehub.io/api/logs/pods/app-code-server-26fcc-765bf579c5-srcft'}]
217-
```
217+
```

‎docs/CLI/deployments.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -364,4 +364,4 @@ endpoint: http://primehub-python-sdk.primehub.io/deployment/quickstar
364364
availableReplicas: 1
365365
message: Deployment is being deployed and not available now
366366
pods: [{'name': 'deploy-quickstart-iris-cfmnh-68889b97cc-8qgvf'}, {'name': 'deploy-quickstart-iris-cfmnh-68889b97cc-wxzs8'}]
367-
```
367+
```

‎docs/notebook/admin/datasets.ipynb

Lines changed: 228 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,228 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"id": "3365d96c",
6+
"metadata": {},
7+
"source": [
8+
"# [admin] Datasets command\n",
9+
"\n",
10+
"\n",
11+
"The `datasets` command in `admin` scope could help you manage datasets.\n"
12+
]
13+
},
14+
{
15+
"cell_type": "markdown",
16+
"id": "0e684e90",
17+
"metadata": {},
18+
"source": [
19+
"## Setup PrimeHub Python SDK\n"
20+
]
21+
},
22+
{
23+
"cell_type": "code",
24+
"execution_count": null,
25+
"id": "2ed0bf22",
26+
"metadata": {},
27+
"outputs": [],
28+
"source": [
29+
"from primehub import PrimeHub, PrimeHubConfig\n",
30+
"ph = PrimeHub(PrimeHubConfig())\n",
31+
"\n",
32+
"if ph.is_ready():\n",
33+
" print(\"PrimeHub Python SDK setup successfully\")\n",
34+
"else:\n",
35+
" print(\"PrimeHub Python SDK couldn't get the group information, follow the 00-getting-started.ipynb to complete it\")"
36+
]
37+
},
38+
{
39+
"cell_type": "markdown",
40+
"id": "3720ad82",
41+
"metadata": {},
42+
"source": [
43+
"## Help documentation"
44+
]
45+
},
46+
{
47+
"cell_type": "code",
48+
"execution_count": null,
49+
"id": "2055801e",
50+
"metadata": {},
51+
"outputs": [],
52+
"source": [
53+
"help(ph.admin.datasets)"
54+
]
55+
},
56+
{
57+
"cell_type": "markdown",
58+
"id": "ac0e2516",
59+
"metadata": {},
60+
"source": [
61+
"## Dataset management\n",
62+
"\n",
63+
"---\n",
64+
"\n",
65+
"\n",
66+
"```\n",
67+
"$ primehub admin datasets\n",
68+
"\n",
69+
"Usage:\n",
70+
" primehub admin datasets <command>\n",
71+
"\n",
72+
"Manage datasets\n",
73+
"\n",
74+
"Available Commands:\n",
75+
" create Create a dataset\n",
76+
" delete Delete a dataset by id\n",
77+
" get Get a dataset by name\n",
78+
" list Delete a dataset by id\n",
79+
" update Update the dataset\n",
80+
" upload_secret Regenerate the secret of the upload server\n",
81+
"```\n",
82+
"\n",
83+
"---\n",
84+
"\n",
85+
"\n",
86+
"## Dataset configuration\n",
87+
"\n",
88+
"You need a configuration `create` and `update` to operate. Here is an example to create a `pv-dataset`:\n",
89+
"\n",
90+
"```json\n",
91+
"{\n",
92+
" \"name\": \"pv-dataset\",\n",
93+
" \"displayName\": \"the dataset created by SDK\",\n",
94+
" \"description\": \"It is a PV dataset\",\n",
95+
" \"type\": \"pv\",\n",
96+
" \"global\": true,\n",
97+
" \"pvProvisioning\": \"auto\",\n",
98+
" \"volumeSize\": 1\n",
99+
"}\n",
100+
"```\n",
101+
"\n",
102+
"In our system, there are 5 types for datasets: `['pv', 'nfs', 'hostPath', 'git', 'env']`. Please check the fields reference to give a proper configuration to create your own dataset.\n",
103+
"\n",
104+
"\n",
105+
"\n",
106+
"## Fields for creating or updating\n",
107+
"\n",
108+
"| field | required | type | description |\n",
109+
"| --- | --- | --- | --- |\n",
110+
"| name | required | string | it should be a valid resource name for kubernetes |\n",
111+
"| displayName | optional | string | display name for this dataset |\n",
112+
"| description | optional | string | |\n",
113+
"| global | optional | boolean | when a dataset is global, it could be seen for each group |\n",
114+
"| type | required | string | one of ['pv', 'nfs', 'hostPath', 'git', 'env'] |\n",
115+
"| url | conditional | string | **MUST** use with `git` type |\n",
116+
"| pvProvisioning | conditional | string | onf of ['auto', 'manual'], **MUST** use with `pv` type. This field only uses in `CREATE` action |\n",
117+
"| nfsServer | conditional | string | **MUST** use with `nfs` type |\n",
118+
"| nfsPath | conditional | string | **MUST** use with `nfs` type |\n",
119+
"| hostPath | conditional | string | **MUST** use with `hostPath` type |\n",
120+
"| variables | optional | dict | **MAY** use with `env` type. It is key value pairs. All values have to a string value. For example: `{\"key1\":\"value1\",\"key2\":\"value2\"}`. |\n",
121+
"| groups | optional | list of connected groups (dict) | please see the `connect` examples |\n",
122+
"| secret | optional | dict | **MAY** use with `git` type. it binds a `secret` to the `git` dataset |\n",
123+
"| volumeSize | conditional | integer | **MUST** use with `pv` type. The unit is `GB`.|\n",
124+
"| enableUploadServer | optional | boolean | it only works with one of ['pv', 'nfs', 'hostPath'] writable types |\n",
125+
"\n",
126+
"> There is a simple rule to use fields for `UPDATE`. All required fields should not be in the payload.\n",
127+
"\n",
128+
"For example, there is a configuration for creating env dataset:\n",
129+
"\n",
130+
"```bash\n",
131+
"primehub admin datasets create <<EOF\n",
132+
"{\n",
133+
" \"name\": \"env-dataset\",\n",
134+
" \"description\": \"\",\n",
135+
" \"type\": \"env\",\n",
136+
" \"variables\": {\n",
137+
" \"ENV\": \"prod\",\n",
138+
" \"LUCKY_NUMBER\": \"7\"\n",
139+
" }\n",
140+
"}\n",
141+
"EOF\n",
142+
"```\n",
143+
"\n",
144+
"After removing required `name` and `type` fields, it could be used with updating:\n",
145+
"\n",
146+
"```bash\n",
147+
"primehub admin datasets update env-dataset <<EOF\n",
148+
"{\n",
149+
" \"description\": \"make changes to the description\",\n",
150+
" \"variables\": {\n",
151+
" \"ENV\": \"prod\",\n",
152+
" \"LUCKY_NUMBER\": \"8\"\n",
153+
" }\n",
154+
"}\n",
155+
"EOF\n",
156+
"```\n",
157+
"\n"
158+
]
159+
},
160+
{
161+
"cell_type": "markdown",
162+
"id": "dcb77e3d",
163+
"metadata": {},
164+
"source": [
165+
"## Examples"
166+
]
167+
},
168+
{
169+
"cell_type": "markdown",
170+
"id": "b4eb5746",
171+
"metadata": {},
172+
"source": [
173+
"You could find [more examples on our github](https://github.com/InfuseAI/primehub-python-sdk/blob/main/docs/CLI/admin/datasets.md)."
174+
]
175+
},
176+
{
177+
"cell_type": "code",
178+
"execution_count": null,
179+
"id": "639e6281",
180+
"metadata": {},
181+
"outputs": [],
182+
"source": [
183+
"# List datasets\n",
184+
"list(ph.admin.datasets.list())"
185+
]
186+
},
187+
{
188+
"cell_type": "code",
189+
"execution_count": null,
190+
"id": "908759a6",
191+
"metadata": {},
192+
"outputs": [],
193+
"source": [
194+
"# Get a dataset\n",
195+
"ph.admin.datasets.get('primehub')"
196+
]
197+
},
198+
{
199+
"cell_type": "code",
200+
"execution_count": null,
201+
"id": "ef3c1a91",
202+
"metadata": {},
203+
"outputs": [],
204+
"source": []
205+
}
206+
],
207+
"metadata": {
208+
"kernelspec": {
209+
"display_name": "PrimeHub SDK",
210+
"language": "python",
211+
"name": "myenv"
212+
},
213+
"language_info": {
214+
"codemirror_mode": {
215+
"name": "ipython",
216+
"version": 3
217+
},
218+
"file_extension": ".py",
219+
"mimetype": "text/x-python",
220+
"name": "python",
221+
"nbconvert_exporter": "python",
222+
"pygments_lexer": "ipython3",
223+
"version": "3.7.6"
224+
}
225+
},
226+
"nbformat": 4,
227+
"nbformat_minor": 5
228+
}

‎docs/notebook/groups.ipynb

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -98,4 +98,4 @@
9898
},
9999
"nbformat": 4,
100100
"nbformat_minor": 5
101-
}
101+
}

‎primehub/__init__.py

Lines changed: 32 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -192,7 +192,9 @@ class PrimeHub(object):
192192
def __init__(self, config: PrimeHubConfig):
193193
self.primehub_config = config
194194
self.json_output = True
195+
self.usage_role = 'user'
195196
self.commands: Dict[str, Module] = dict()
197+
self.admin_commands: Dict[str, Module] = dict()
196198
self._stderr = sys.stderr
197199
self._stdout = sys.stdout
198200

@@ -212,6 +214,9 @@ def __init__(self, config: PrimeHubConfig):
212214
self.register_command('apptemplates', 'AppTemplate')
213215
self.register_command('apps', 'Apps')
214216

217+
# register admin commands
218+
self.register_admin_command('admin_datasets', 'AdminDatasets', 'datasets')
219+
215220
# initial
216221
self._ensure_config_details(config)
217222

@@ -236,19 +241,42 @@ def request_logs(self, endpint: str, follow: bool, tail: int):
236241
def request_file(self, endpint: str, dest: str):
237242
return Client(self.primehub_config).request_file(endpint, dest)
238243

239-
def register_command(self, module_name: str, command_class: Union[str, Callable], command_name=None):
240-
if not command_name:
241-
command_name = module_name
242-
244+
def _find_command_class(self, command_class, module_name):
243245
# create command instance
244246
if isinstance(command_class, str):
245247
clazz = importlib.import_module('primehub.' + module_name).__getattribute__(command_class)
246248
else:
247249
clazz = command_class
250+
return clazz
251+
252+
def register_command(self, module_name: str, command_class: Union[str, Callable], command_name=None):
253+
if not command_name:
254+
command_name = module_name
255+
256+
clazz = self._find_command_class(command_class, module_name)
248257

249258
# register to the commands table
250259
self.commands[command_name] = clazz(self)
251260

261+
def register_admin_command(self, module_name: str, command_class: Union[str, Callable], command_name=None):
262+
if not command_name:
263+
command_name = module_name
264+
265+
clazz = self._find_command_class(command_class, module_name)
266+
267+
# register to the commands table
268+
self.admin_commands[command_name] = clazz(self)
269+
270+
def switch_admin_role(self):
271+
self.usage_role = 'admin'
272+
self.commands = self.admin_commands
273+
274+
@property
275+
def admin(self):
276+
admin_primehub = PrimeHub(self.primehub_config)
277+
admin_primehub.commands = self.admin_commands
278+
return admin_primehub
279+
252280
def __getattr__(self, item):
253281
if item in self.commands:
254282
return self.commands[item]

0 commit comments

Comments
 (0)