UNIX pipelines perform incredibly well, and let you mix and match
off-the-shelf tools in flexible ways. With Super, you can leverage
all of this power. Use high-performance UNIX pipes |, but against
Cloud data and compute.
Following on from our previous cp example,
this super run will spawn 3 jobs and produce the word count output
on your console. Here we are using off-the-shelf tools (grep and
wc), but in the Cloud, to compute a partial sum of matches.
super run -- 'cat /s3/ibm/tmp/*.wet.gz | gunzip -c - | grep "WARC-Type: conversion" | wc -l'
[Job 1] 40711
[Job 2] 40880
[Job 3] 40681
