- Running UNIX Commands in the Cloud [You are here]
- Interacting with Cloud Object Storage
- Visually Browsing Cloud Object Storage
- Parallelizing your UNIX Pipeline
- Injecting Custom Binaries
- Examples of parallel analytics against CommonCrawl data
- Automating Periodic Tasks
- The Super Dashboard
Super allows you to run a UNIX command line in parallel, using
auto-scaling Cloud compute resources, against a set of Cloud data.
Super takes care of provisioning the right amount of compute, memory,
and disk capacity, scheduling your jobs, granting the needed data
access authority to your work, and streaming out logs --- all in one
command: super run.
Super uses containers running in IBM Cloud Code Engine as the compute layer, and gives your jobs access to data via IBM Cloud Object Storage.
This document demonstrates how to use super run via examples ranging
from data preparation tasks to more complex analytics.
Note: Make sure to verify you are good to go, by first running
super up.
The command lscpu provides information about the CPU your on which
your command executes. Using super run, we can easily run this in
the Cloud. Behind the scenes, CPU resources will automatically be spun
up and torn down, without your having to worry about anything beyond
the commands you wish to execute:
❯ super run -- lscpu
Architecture: ...You can also link together commands into pipelines, in the normal
UNIX way. For
example, to extract the CPU model, you can pipe through
grep to filter the output:
❯ super run -- 'lscpu | grep "Model name"'
Model name: Intel Core Processor (Broadwell, IBRS)Note: Surround your command with
'single quotes'to ensure that the|is executed in the Cloud, not your laptop.
Your Cloud pipelines are given a default allocation of CPU shares and
phystical memory: 1 (whole) CPU and 1024Mi of memory. If the default
settings prove insufficient for your needs, then the --cpu and
--memory options can help:
super run --cpu 4 --memory=2048Mi -- lscpu