dsscli tool¶
dsscli is a command-line tool that can perform a variety of runtime administration tasks on DSS. It can be used directly by a DSS administrator, or incorporated into automation scripts.
Most dsscli operations are performed through the DSS public API and can thus also be performed using any REST client, or using the DSS public API Python client.
Running dsscli¶
dsscli is made of a large number of commands. Each command performs a single administration task. Each command takes arguments and options
From the DSS data directory, run ./bin/dsscli <command> <arguments>
- Running
./bin/dsscli -hwill list the available commands. - Running
./bin/dsscli <command> -hwill show the detaildd help of the selected command.
For example, to list jobs history in project MYPROJECT, use ./bin/dsscli jobs-list MYPROJECT
dsscli vs dssadmin¶
Another command-line tool is available in the DSS data directory for performing management tasks on DSS: ./bin/dssadmin
- dssadmin is mostly for “installation-kind” of commands (setting up R, Spark or Hadoop integration for example)
- dsscli is mostly for “day-to-day” routine operations (creating users, running jobs, …)
Security-related commands¶
dsscli provides commands to:
- Create, delete, list and edit users
- Create, delete, list and edit groups
- Create, delete, list and edit API keys
user-create¶
dsscli user-create [-h] [--email EMAIL]
[--source-type SOURCE_TYPE]
[--display-name DISPLAY_NAME]
[--user-profile USER_PROFILE] [--group GROUP]
login password
SOURCE_TYPEmust be eitherLOCALto create a regular user (in the DSS users database),LDAPto create a user that will authenticate through LDAP orLOCAL_NO_AUTHfor authentication through SSO. See Configuring LDAP authentication and Single Sign-On for more information. Note that even for LDAP and LOCAL_NO_AUTH, dsscli expects a “password” argument. Enter any random string, it will be ignored. The default isLOCALUSER_PROFILEis one of the possible user profiles defined by your license. For most DSS licenses, it is one ofREADER,DATA_ANALYSTorDATA_SCIENTIST. The default is specified by your configuration- The
--group GROUPargument can be specified multiple times to place the user in multiple groups
users-list¶
dsscli users-list
user-delete¶
dsscli user-delete [-h] login
positional arguments:
login Login to delete
user-edit¶
Modifies the settings of an existing user.
dsscli user-edit [-h] [--password PASSWORD]
[--display-name DISPLAY_NAME] [--email EMAIL]
[--user-profile USER_PROFILE] [--group GROUP]
login
Each of password, display name, email, user-profile and groups can be modified independently.
For example, running dsscli user-edit --email mynewemail@company.com user will only modify the email address, and leave all other fields unmodified.
All groups are modified at once: thus to modify groups, you need to pass a new list of groups, which will be the new complete list.
It is not possible to modify password for a LDAP or LOCAL_NO_AUTH user
groups-list¶
Lists all groups in DSS
dsscli groups-list [-h] [--with-permissions] [--output OUTPUT]
[--no-header]
If --with-permissions is specified, additional columns are added to the output with global permissions, as detailed in Main permissions
group-create¶
Creates a group by name
dsscli group-create [-h] [--description DESCRIPTION]
[--source-type SOURCETYPE] [--admin ADMIN]
[--may-manage-code-envs MAYMANAGECODEENVS]
[--may-create-code-envs MAYCREATECODEENVS]
[--may-write-unsafe-code MAYWRITEUNSAFECODE]
[--may-write-safe-code MAYWRITESAFECODE]
[--may-create-projects MAYCREATEPROJECTS]
[--may-manage-udm MAYMANAGEUDM]
[--may-edit-lib-folders MAYEDITLIBFOLDERS]
[--may-develop-plugins MAYDEVELOPPLUGINS]
[--may-create-authenticated-connections MAYCREATEAUTHENTICATEDCONNECTIONS]
name
All of the --may-xxx flags take “true” or “false” as argument, and refer to one of the global permissions as detailed in Main permissions
SOURCETYPE can be either LDAP or LOCAL. Note that LDAP groups need to declare mappings to LDAP groups to be functional, but this feature is not currently in dsscli. You need to use the DSS API clients.
Adding users in groups is done by editing these users.
group-edit¶
Edits the settings of a group by name
dsscli group-edit [-h] [--description DESCRIPTION]
[--source-type SOURCETYPE] [--admin ADMIN]
[--may-manage-code-envs MAYMANAGECODEENVS]
[--may-create-code-envs MAYCREATECODEENVS]
[--may-write-unsafe-code MAYWRITEUNSAFECODE]
[--may-write-safe-code MAYWRITESAFECODE]
[--may-create-projects MAYCREATEPROJECTS]
[--may-manage-udm MAYMANAGEUDM]
[--may-edit-lib-folders MAYEDITLIBFOLDERS]
[--may-develop-plugins MAYDEVELOPPLUGINS]
[--may-create-authenticated-connections MAYCREATEAUTHENTICATEDCONNECTIONS]
name
All of the --may-xxx flags take “true” or “false” as argument, and refer to one of the global permissions as detailed in Main permissions
SOURCETYPE can be either LDAP or LOCAL. Note that LDAP groups need to declare mappings to LDAP groups to be functional, but this feature is not currently in dsscli. You need to use the DSS API clients.
Adding users in groups is done by editing these users.
api-keys-list¶
Lists global API keys
api-key-create¶
Creates a global API key
dsccli api-key-create [-h] [--output OUTPUT] [--no-header]
[--description DESCRIPTION] [--label LABEL]
[--admin ADMIN]
[--may-manage-code-envs MAYMANAGECODEENVS]
[--may-create-code-envs MAYCREATECODEENVS]
[--may-write-unsafe-code MAYWRITEUNSAFECODE]
[--may-write-safe-code MAYWRITESAFECODE]
[--may-create-projects MAYCREATEPROJECTS]
[--may-manage-udm MAYMANAGEUDM]
[--may-edit-lib-folders MAYEDITLIBFOLDERS]
[--may-develop-plugins MAYDEVELOPPLUGINS]
[--may-create-authenticated-connections MAYCREATEAUTHENTICATEDCONNECTIONS]
All of the --may-xxx flags take “true” or “false” as argument, and refer to one of the global permissions as detailed in Main permissions
api-key-edit¶
Edits a global API key
dsccli api-key-edit [-h] [--output OUTPUT] [--no-header]
[--description DESCRIPTION] [--label LABEL]
[--admin ADMIN]
[--may-manage-code-envs MAYMANAGECODEENVS]
[--may-create-code-envs MAYCREATECODEENVS]
[--may-write-unsafe-code MAYWRITEUNSAFECODE]
[--may-write-safe-code MAYWRITESAFECODE]
[--may-create-projects MAYCREATEPROJECTS]
[--may-manage-udm MAYMANAGEUDM]
[--may-edit-lib-folders MAYEDITLIBFOLDERS]
[--may-develop-plugins MAYDEVELOPPLUGINS]
[--may-create-authenticated-connections MAYCREATEAUTHENTICATEDCONNECTIONS]
key
All of the --may-xxx flags take “true” or “false” as argument, and refer to one of the global permissions as detailed in Main permissions
api-key-delete¶
Deletes a global API key
Jobs-related commands¶
These commands are used to trigger, list and abort jobs and scenarios
jobs-list¶
Lists jobs for a given project, both running ones and past ones
dsscli jobs-list [-h] [--output OUTPUT] [--no-header] project_key
Returns a list like:
| Job id | State | From scenario |
|---|---|---|
| Build_dataset1_2017-10-25T13-05-23.615 | RUNNING | |
| Build_dataset1_2017-10-25T13-05-23.615 | FAILED | |
| Build_dataset1_2017-10-25T12-45-32.864 | FAILED | |
| Build_Other_2017-10-25T12-43-31.463 | DONE |
build¶
Runs a DSS job to build one or several datasets, saved models or managed folders
build [-h] [--output OUTPUT] [--no-header] [--wait]
[--mode MODE] [--dataset DATASET [DATASET ...]]
[--folder FOLDER] [--model MODEL]
project_key
Specifying outputs to build¶
To build “dataset1” and “dataset2” in project “PROJECT1”, run: dsscli build PROJECT1 --dataset dataset1 --dataset dataset2
For partitioned datasets, use the following syntax:
dsscli build PROJECT1 --dataset dataset1 partition1For multiple partitions, use the regular partition specification syntax:
dsscli build PROJECT1 --dataset dataset1 FR,ENdsscli build PROJECT1 --dataset dataset1 2017-01-02/2017-01-14dsscli build PROJECT1 --dataset dataset1 2017-01-02/2017-01-14|FR,2017-01-02/2017-01-30|EN
Build modes¶
Use –mode to switch between the different build modes of the DSS flow. The argument must be one of:
- RECURSIVE_BUILD,
- NON_RECURSIVE_FORCED_BUILD
- RECURSIVE_FORCED_BUILD
- RECURSIVE_MISSING_ONLY_BUILD
Other¶
The –wait argument makes dsscli wait for the end of the job (either success or failure) before returning. If the job fails or is aborted, dsscli returns with a non-zero exit code.
If not waiting, dsscli prints the new job id
job-abort¶
Aborts a running job
dsscli job-abort [-h] project_key job_id
The job_id is the first column returned by the dsscli jobs-list command
job-status¶
Gets the status of a job
dsscli job-status [-h] [--output OUTPUT] [--no-header]
project_key job_id
Scenarios-related commands¶
scenarios-list¶
Lists scenarios of a project
dsscli scenarios-list [-h] [--output OUTPUT] [--no-header]
project_key
scenario-runs-list¶
Lists previous and current runs of a scenario
dsscli scenario-runs-list [-h] [--output OUTPUT] [--no-header]
[--limit LIMIT] [--only-finished-runs]
project_key scenario_id
--only-finished-runslimits output to runs that are finished (either succeeded, failed or was aborted)--limitlimits the number of returned runs. Default is 10
scenario-run¶
Runs a scenario
dsscli scenario-run [-h] [--output OUTPUT] [--no-header] [--wait]
[--no-fail] [--params RUN_PARAMS]
project_key scenario_id
If the scenario was already running, the run is cancelled, and a flag is returned in the output
If --wait is passed, command waits for the scenario to be complete and fails if the scenario fails, except if --no-fail is passed. It also fails if the run is cancelled because the scenario was already running
--params is an optional file containing run parameters as a JSON dict. Use ‘-‘ for stdin
scenario-abort¶
dsscli scenario-abort [-h] project_key scenario_id
Aborts the current run of a scenario, if any. Does not fail if the scenario was not running
Projects-related commands¶
projects-list¶
Lists projects
project-export¶
project-import¶
project-delete¶
bundle-export (Design node only)¶
bundles-list-exported (Design node only)¶
bundle-download-archive (Design node only)¶
project-create-from-bundle (Automation node only)¶
bundles-list-imported (Automation node only)¶
bundle-import (Automation node only)¶
bundle-activate (Automation node only)¶
Datasets related commands¶
datasets-list¶
dataset-schema-dump¶
dataset-list-partitions¶
dataset-clear¶
dataset-delete¶
Managed folders related commands¶
managed-folders-list¶
managed-folder-list-contents¶
managed-folder-get-file¶
Connections-related commands¶
connections-list¶
Lists all connections with their type and flags
Controlling dsscli output¶
All dsscli commands that display results take two additional arguments:
--no-headerremoves column headers from display to make them easier to parse (each line of the output directly corresponds to one data item)--output jsonto format output as JSON for machine consumption (the default is--output fancy)