Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions .vale/styles/config/vocabularies/Utopia/accept.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@


# Brands, products, people
# Protocols, formats, acronyms
# Tech vocabulary
[Cc]harset(?:s)?
[Cc]onfig(?:s)?
[Cc]ron
Expand All @@ -13,9 +14,6 @@
[Ss]tdout
[Ss]ubtree
[Vv]alidator(?:s)?
# Brands, products, people
# Protocols, formats, acronyms
# Tech vocabulary
accessor(?:s)?
ACLs?
aiohttp
Expand Down Expand Up @@ -68,6 +66,7 @@ Fedora
FPM
Fux
GET
getters?
GPTBot
Grafana
Guzzle
Expand Down Expand Up @@ -148,6 +147,7 @@ Sailfish
sanitization
Scrypt
Sentry
setters?
Silk
Sinch
SMS
Expand Down
2 changes: 1 addition & 1 deletion packages/client/composer.json
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
"psr/http-client": "^1.0",
"psr/http-factory": "^1.0",
"psr/http-message": "^1.1 || ^2.0",
"utopia-php/pools": "^1.0",
"utopia-php/pools": "^2.0",
"utopia-php/psr7": "^0.2",
"utopia-php/span": "^1.1 || ^3.0 || ^4.0"
},
Expand Down
6 changes: 3 additions & 3 deletions packages/client/tests/Client/PoolTest.php
Original file line number Diff line number Diff line change
Expand Up @@ -58,13 +58,13 @@ public function testItReclaimsTheConnectionSoItCanBeReused(): void
*
* @template T of \Psr\Http\Client\ClientInterface&StreamingClientInterface
*
* @param callable(): T $init
* @param \Closure(): T $init
*
* @return Connections<T>
*/
private function connections(callable $init, int $size = 4): Connections
private function connections(\Closure $init, int $size = 4): Connections
{
return new Connections(new Stack(), 'test', $size, $init);
return new Connections(new Stack(), 'test', $size, $init, timeout: 0.0);
}

private function request(): RequestInterface
Expand Down
2 changes: 1 addition & 1 deletion packages/messaging/composer.json
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
"psr/http-client": "^1.0",
"psr/http-message": "^1.1 || ^2.0",
"utopia-php/client": "^0.2",
"utopia-php/pools": "^1.0",
"utopia-php/pools": "^2.0",
"utopia-php/telemetry": "^0.4"
},
"require-dev": {
Expand Down
7 changes: 5 additions & 2 deletions packages/messaging/src/Utopia/Messaging/Adapter.php
Original file line number Diff line number Diff line change
Expand Up @@ -271,10 +271,13 @@ protected function requestMulti(

$run = function () use ($requests, $timeout, $connectTimeout, &$results): void {
$pool = new ConnectionPool(
pool: new SwoolePoolAdapter(),
adapter: new SwoolePoolAdapter(),
name: self::CONNECTION_POOL_NAME,
size: min(\count($requests), self::MAX_CONCURRENT_REQUESTS),
size: max(1, min(\count($requests), self::MAX_CONCURRENT_REQUESTS)),
init: $this->clientFactory ?? $this->defaultClient($timeout, $connectTimeout)->withConnectionReuse(...),
// A slot per request, so acquisition never queues; the request
// timeouts belong to the client, not to getting hold of one.
timeout: 0.0,
);

$group = new WaitGroup();
Expand Down
150 changes: 94 additions & 56 deletions packages/pools/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,79 +27,117 @@ composer require utopia-php/pools

```php
use PDO;
use Utopia\Pools\Pool;
use Utopia\Pools\Adapter\Swoole;
use Utopia\Pools\Group;
use Utopia\Pools\Pool;

$pool = new Pool('mysql-pool', 1 /* number of connections */, function() {
$host = '127.0.0.1';
$db = 'test';
$user = 'root';
$pass = '';
$charset = 'utf8mb4';

try {
$pdo = new PDO("mysql:host=$host;dbname=$db;charset=$charset", $user, $pass);
} catch (\PDOException $e) {
throw new \PDOException($e->getMessage(), (int)$e->getCode());
}

return $pdo;
});

$pool->setReconnectAttempts(3); // number of attempts to reconnect
$pool->setReconnectSleep(5); // seconds to sleep between reconnect attempts

$pool->setRetryAttempts(3); // number of attempts to get connection
$pool->setRetrySleep(5); // seconds to sleep between failed pop-connection attempts

$connection = $pool->pop(); // Get a connection from the pool
$connection->getID(); // Get the connection ID
$connection->getResource(); // Get the connection resource

$pool->push($connection); // Return the connection to the pool
$pool = new Pool(
adapter: new Swoole,
name: 'mysql-pool',
size: 8,
init: fn () => new PDO('mysql:host=127.0.0.1;dbname=test;charset=utf8mb4', 'root', ''),
timeout: 2.0,
);

// Preferred: the pool hands the resource to the callback and takes it back
// afterwards, discarding it if the callback threw.
$rows = $pool->use(fn (PDO $pdo) => $pdo->query('SELECT 1')->fetchAll());

// Manual lifetime, when `use()` does not fit.
$connection = $pool->pop();
$connection->id; // "mysql-pool-6885a1f2c4e19"
$connection->resource; // the PDO instance
$connection->reclaim(); // return it for reuse
$connection->destroy(); // or discard it and free the capacity

$pool->count(); // connections a caller could still obtain
$pool->isEmpty(); // none available without one being returned first
$pool->isFull(); // nothing checked out

$group = new Group;
$group->add($pool);
$group->get('mysql-pool');
$group->use(['mysql-pool'], fn (PDO $pdo) => $pdo->query('SELECT 1'));
```

$pool->reclaim(); // Recalim the pool, return all active connections automatically
## Acquiring connections

$pool->count(); // Get the number of available connections
`timeout` bounds how long `pop()` waits for a connection to become available.
The pool creates one when it has spare capacity, otherwise it waits for one to
come back, and it throws once the budget is spent. There is one number for
waiting, and it is the number a caller experiences.

$pool->isEmpty(); // Check if the pool is empty
It does not cover time spent inside `init`. The pool cannot interrupt a blocking
connect, so give `init` its own connect timeout if you need the total bounded —
for PDO that is `PDO::ATTR_TIMEOUT`.

$pool->isFull(); // Check if the pool is full
If creating a connection fails, that exception propagates untouched, so callers
keep the original type to act on. The pool does not retry and does not fall back
to waiting, because waiting for another caller's connection after your own
create failed is a retry in disguise. Own retries in `init`:

$group = new Group(); // Create a group of pools
$group->add($pool); // Add a pool to the group
$group->get('mysql-pool'); // Get a pool from the group
$group->setReconnectAttempts(3); // Set the number of reconnect attempts for all pools
$group->setReconnectSleep(5); // Set the sleep time between reconnect attempts for all pools
```php
$pool = new Pool(
adapter: new Swoole,
name: 'mysql-pool',
size: 8,
init: fn () => $breaker->call(fn () => new PDO(/* ... */)),
timeout: 2.0,
);
```

## Reconnect and retry

Both reconnect and retry logic is used in `pop()` method to handle problematic scenarios. Both allow you to configure 2 properties:

- `attempts` - How many times library will retry when problem occurs
- `sleep` - How long will library wait for before next retry attempt

**Reconnect** settings are used when your connection initialization callback throws an exception. This can occur for example, when a handshake with SQL server fails.

**Retry** settings are used when pool of connection is empty and there is no more connections to pop. This can occur for example, when your server supports more concurrent actions than your pool.
Compose [`utopia-php/circuit-breaker`](https://github.com/utopia-php/circuit-breaker)
there rather than counting attempts, because a breaker carries state between
calls and an attempt counter cannot.

## Upgrading from 1.x

Configuration is constructor-only, and the retry knobs are gone.

- `timeout` is a required constructor argument, and it is the pool's only
timeout. It replaces `setRetryAttempts()`, `setRetrySleep()` and
`setSynchronizationTimeout()`. Pass the total wait you want, in seconds.
- `setReconnectAttempts()` and `setReconnectSleep()` are removed with no
replacement. Wrap `init` if you want to retry a failed connect.
- `setTelemetry()` is removed. Pass `telemetry:` to the constructor.
- `Group::setReconnectAttempts()`, `Group::setReconnectSleep()` and
`Group::setTelemetry()` are removed. Pools arrive configured.
- `Pool::getName()` and `Pool::getSize()` are now the `$name` and `$size`
properties. The `getReconnectAttempts()`, `getReconnectSleep()`,
`getRetryAttempts()`, `getRetrySleep()` and `getSynchronizationTimeout()`
getters are removed.
- `Connection` is `final readonly`. `getID()`, `getResource()` and `getPool()`
become the `$id` and `$resource` properties; `getPool()` has no replacement.
`setID()`, `setResource()` and `setPool()` are removed.
- `Connection::reclaim()` and `Connection::destroy()` return `void` instead of
the pool.
- `Pool::release()` no longer takes a `$start` timestamp. The pool measures use
time itself.
- `Adapter::pop()` takes a `float` timeout. Custom adapters need the wider type.
- `init` is typed `Closure` rather than `callable`. Pass a closure, or wrap a
string or array callable with `Closure::fromCallable()`.

Two behaviour changes worth checking against your own code:

- A failed connect used to be absorbed by an internal retry and can now reach
the caller on the first attempt, as whatever type `init` threw rather than a
pool exception.
- `destroy()` no longer creates the replacement connection itself. Capacity is
freed immediately and the next `pop()` creates one, so `destroy()` cannot
block on a connect or fail for reasons unrelated to the connection you asked
it to discard.

## System requirements

Utopia Framework requires PHP 8.0 or later. We recommend using the latest PHP version whenever possible.
Utopia pools requires PHP 8.4 or later. We recommend using the latest PHP version whenever possible.

## Tests

To run all unit tests, use the following Docker command:

```bash
docker compose exec tests vendor/bin/phpunit --configuration phpunit.xml tests
```

To run static code analysis, use the following Psalm command:
Run the test suite and static analysis from the monorepo root:

```bash
docker compose exec tests vendor/bin/psalm --show-info=true
bin/monorepo test pools
bin/monorepo check pools
```

## Copyright and license
Expand Down
Loading
Loading