Monitoring an Experiment
At any point in time you can see the status of your running experiments by issuing the
riseml status command:
$ riseml status ID PROJECT STATE AGE TYPE 138 ai-toaster RUNNING 2 second(s) Experiment
You can continue streaming the logs with the
riseml logs command:
$ riseml logs 138 138 | [2017-08-30T09:36:11Z] --> FINISHED 138.train | [2017-08-30T09:36:11Z] --> FINISHED 138.build | [2017-08-30T09:36:11Z] --> FINISHED 138.build | [2017-08-30T09:21:53Z] Building your image 138.build | [2017-08-30T09:21:53Z] Downloading code ... 138.train | [2017-08-30T09:26:20Z] Successfully downloaded inception-2015-12-05.tgz ...
This will output the last known states followed by a chronological stream of the produced output.
In addition, you can obtain real-time statistics about CPU utilization with the
riseml monitor command:
$ riseml monitor 138 ID PROJECT STATE CPU MEM GPU GPU MEM 138.train ai-toaster ● RUNNING 1.1/4 |2 2.0/60 |4 - -
The syntax for CPU and MEM statistics shows
In the example, 1.1 CPUs are used and the host has a total of 4 CPUs available.
As per the job definition, you have requested and are guaranteed at least 2 cpus.