Skip to content

Monitoring an Experiment

At any point in time you can see the status of your running experiments by issuing the riseml status command:

$ riseml status
ID     PROJECT    STATE    AGE           TYPE
138    ai-toaster RUNNING  2 second(s)   Experiment

You can continue streaming the logs with the riseml logs command:

$ riseml logs 138
138       | [2017-08-30T09:36:11Z] --> FINISHED
138.train | [2017-08-30T09:36:11Z] --> FINISHED
138.build | [2017-08-30T09:36:11Z] --> FINISHED
138.build | [2017-08-30T09:21:53Z] Building your image
138.build | [2017-08-30T09:21:53Z] Downloading code
...
138.train | [2017-08-30T09:26:20Z] Successfully downloaded inception-2015-12-05.tgz
...
This will output the last known states followed by a chronological stream of the produced output.

In addition, you can obtain real-time statistics about CPU utilization with the riseml monitor command:

$ riseml monitor 138
ID        PROJECT     STATE      CPU       MEM        GPU    GPU MEM
138.train ai-toaster  ● RUNNING  1.1/4 |2  2.0/60 |4  -      -

The syntax for CPU and MEM statistics shows current/available |requested. In the example, 1.1 CPUs are used and the host has a total of 4 CPUs available. As per the job definition, you have requested and are guaranteed at least 2 cpus.