Skip to content

GoTutorial.md

Latest commit

 

History

History
210 lines (181 loc) · 7.41 KB

GoTutorial.md

File metadata and controls

210 lines (181 loc) · 7.41 KB
Raw

Tutorial

In this tutorial we will develop a Colonies Executor using the Golang SDK. The executor calculates the last number in a given Fibonacci serie.

1. Set up a Colonies development server

The easiest way to get a local server is to run the all-in-one container (see Deployment.md):

./scripts/generate-dev-env.sh
docker compose --profile allinone up -d

Alternatively, start a server directly (requires a running Postgres, e.g. make startdb):

colonies server start --initdb --insecure

2. Environment variables

Load the generated .env file into your shell and set the executor type used by the examples:

set -a; . ./.env; set +a
export COLONIES_EXECUTOR_TYPE="cli"

3. Fibonacci job generator code (examples/fibonacci/generator/generator.go)

func main() {
	colonyName := os.Getenv("COLONIES_COLONY_NAME")
	executorPrvKey := os.Getenv("COLONIES_PRVKEY")
	coloniesHost := os.Getenv("COLONIES_SERVER_HOST")
	coloniesPortStr := os.Getenv("COLONIES_SERVER_HTTP_PORT")
	if coloniesPortStr == "" {
		coloniesPortStr = os.Getenv("COLONIES_SERVER_PORT") // deprecated alias
	}
	coloniesPort, err := strconv.Atoi(coloniesPortStr)
	if err != nil {
		fmt.Println(err)
		os.Exit(-1)
	}

	funcSpec := core.CreateEmptyFunctionSpec()
	funcSpec.Conditions.ColonyName = colonyName
	funcSpec.Conditions.ExecutorType = os.Getenv("COLONIES_EXECUTOR_TYPE")
	funcSpec.Env["fibonacciNum"] = os.Args[1]

	client := client.CreateColoniesClient(coloniesHost, coloniesPort, true, false)
	addedProcess, err := client.Submit(funcSpec, executorPrvKey)
	if err != nil {
		fmt.Println(err)
		return
	}

	fmt.Println("Submitted a new process to the Colonies server with Id <" + addedProcess.ID + ">")
}

4. Fibonacci solver executor code (examples/fibonacci/solver/solver.go)

func main() {
	colonyName := os.Getenv("COLONIES_COLONY_NAME")
	executorPrvKey := os.Getenv("COLONIES_PRVKEY")
	coloniesHost := os.Getenv("COLONIES_SERVER_HOST")
	coloniesPortStr := os.Getenv("COLONIES_SERVER_HTTP_PORT")
	if coloniesPortStr == "" {
		coloniesPortStr = os.Getenv("COLONIES_SERVER_PORT") // deprecated alias
	}
	coloniesPort, err := strconv.Atoi(coloniesPortStr)
	if err != nil {
		fmt.Println(err)
		os.Exit(-1)
	}

	// Ask the Colonies server to assign a process to this executor
	client := client.CreateColoniesClient(coloniesHost, coloniesPort, true, false)
	assignedProcess, err := client.Assign(colonyName, 100, "", "", executorPrvKey) // Max wait 100 seconds for assignment request
	if err != nil {
		fmt.Println(err)
		return
	}

	// Parse env attribute and calculate the given Fibonacci number
	for _, attribute := range assignedProcess.Attributes {
		if attribute.Key == "fibonacciNum" {
			fmt.Println("We were assigned process " + assignedProcess.ID)
			fmt.Println("Calculating Fibonacci serie for " + attribute.Value)
			nr, _ := strconv.Atoi(attribute.Value)
			fibonacci := fibonacciBig(uint(nr))
			fmt.Println("Result: The last number in the Fibonacci serie " + attribute.Value + " is " + fibonacci.String())

			attribute := core.CreateAttribute(assignedProcess.ID, colonyName, "", core.OUT, "result", fibonacci.String())
			client.AddAttribute(attribute, executorPrvKey)

			// Close the process as successful
			client.Close(assignedProcess.ID, executorPrvKey)
			return
		}
	}

	// Close the process as failed
	client.Fail(assignedProcess.ID, []string{"invalid arg"}, executorPrvKey)
}

// fibonacciBig returns the n-th Fibonacci number.
func fibonacciBig(n uint) *big.Int {
	a := big.NewInt(0)
	b := big.NewInt(1)
	for i := uint(0); i < n; i++ {
		a.Add(a, b)
		a, b = b, a
	}
	return a
}

5. Calculating Fibonacci numbers

Generate a job

go run ./examples/fibonacci/generator 12

Output:

Submitted a new process to the Colonies server with Id <705abd98cb2f801aa4c0a357c367ea8a5cc89a51d24aaadbca89abbb4be00b7e>

Look up the job in the queue

colonies process psw

The table lists FuncName, Args, KwArgs, submission time, executor name, executor type, initiator, and label for each waiting process. Pass --ids (or -i) to also show the process IDs.

Run one job from the queue

go run ./examples/fibonacci/solver

Output:

We were assigned process 705abd98cb2f801aa4c0a357c367ea8a5cc89a51d24aaadbca89abbb4be00b7e
Calculating Fibonacci serie for 12
Result: The last number in the Fibonacci serie 12 is 144

Inspect the finished process

colonies process get --processid 705abd98cb2f801aa4c0a357c367ea8a5cc89a51d24aaadbca89abbb4be00b7e
Process:
+--------------------+------------------------------------------------------------------+
| ID                 | 705abd98cb2f801aa4c0a357c367ea8a5cc89a51d24aaadbca89abbb4be00b7e |
| IsAssigned         | True                                                             |
| AssignedExecutorID | 3fc05cf3df4b494e95d6a3d297a34f19938f7daa7422ab0d4f794454133341ac |
| State              | Successful                                                       |
| Priority           | 0                                                                |
| SubmissionTime     | 2022-05-27 14:10:03                                              |
| StartTime          | 2022-05-27 14:12:41                                              |
| EndTime            | 2022-05-27 14:12:41                                              |
| WaitDeadline       | 0001-01-01 01:12:12                                              |
| ExecDeadline       | 0001-01-01 01:12:12                                              |
| WaitingTime        | 2m37.735526s                                                     |
| ProcessingTime     | 9.11ms                                                           |
| Retries            | 0                                                                |
| ErrorMsg           |                                                                  |
+--------------------+------------------------------------------------------------------+

Function Specification:
+-------------+------+
| Func        | None |
| Args        | None |
| KwArgs      | None |
| MaxWaitTime | -1   |
| MaxExecTime | -1   |
| MaxRetries  | 3    |
| Label       |      |
+-------------+------+

Conditions:
+------------------+------+
| Colony           | dev  |
| ExecutorNames    | None |
| ExecutorType     | cli  |
| Location         | None |
| Dependencies     |      |
| Nodes            | 0    |
| CPU              |      |
| Memory           |      |
| Processes        | 0    |
| ProcessesPerNode | 0    |
| Storage          |      |
| Walltime         | 0    |
| GPUName          |      |
| GPUs             | 0    |
| GPUPerNode       | 0    |
| GPUMemory        |      |
+------------------+------+

Attributes:
+------------------------------------------------------------------+--------------+-------+------+
|                                ID                                |     KEY      | VALUE | TYPE |
+------------------------------------------------------------------+--------------+-------+------+
| c288d631ae86efc84c54b4c40e2420845d9ac04aecfa614d30f2a509441994b2 | fibonacciNum | 12    | Env  |
| 798040fccd6100fd68f680cdd962c87caf5098e826797c1e85b154dbecf87a27 | result       | 144   | Out  |
+------------------------------------------------------------------+--------------+-------+------+

See examples/fibonacci/generator_sub/generator_sub.go and examples/fibonacci/solver_sub/solver_sub.go for an event-driven version of the generator and solver.