Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 74 additions & 1 deletion UPGRADE-1.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ a small, pluggable interface hierarchy.
| Class / Interface | Description |
|-------------------|-------------|
| `Yokai\Batch\Logger\JobExecutionLoggerInterface` | PSR-3 logger + `getReference()`, `getLogs()`, `getLogsContent()` |
| `Yokai\Batch\Factory\JobExecutionLoggerFactoryInterface` | Creates and restores loggers (`create()` / `restore(string $ref)`) |
| `Yokai\Batch\Factory\JobExecutionLoggerFactoryInterface` | Creates and restores loggers (`create(string $id)` / `restore(string $ref)`) |
| `Yokai\Batch\Logger\InMemoryJobExecutionLogger` | Default implementation — accumulates logs in-memory |
| `Yokai\Batch\Logger\NullJobExecutionLogger` | Discards all logs — useful in tests |
| `Yokai\Batch\Factory\JobExecutionLoggerFactory\InMemoryJobExecutionLoggerFactory` | Factory for the in-memory logger (registered by default by the Symfony bundle) |
Expand Down Expand Up @@ -169,6 +169,79 @@ forward to the current job execution logger during job execution. Replace any re

---

### New bridge: `yokai/batch-monolog`

A new optional bridge package adds file-based log storage for job executions via
[Monolog](https://seldaek.github.io/monolog)'s `StreamHandler`.

```bash
composer require yokai/batch-monolog
```

Each job execution gets its own log file named after its id:

```php
use Yokai\Batch\Bridge\Monolog\StreamJobExecutionLoggerFactory;

new StreamJobExecutionLoggerFactory(
directory: '/var/log/batch',
);
```

For large workloads, files can be spread across nested subdirectories (git-object style):

```php
new StreamJobExecutionLoggerFactory(
directory: '/var/log/batch',
subDirectories: 2,
charsPerDirectory: 2,
// a job "60996f72" is stored at /var/log/batch/60/99/60996f72.log
);
```

Monolog processors and a custom formatter can also be injected to control how records are written.

In a Symfony application, configure the bridge through the bundle (see below).

---

### Symfony bundle configuration — job execution log storage

A new `logging` key controls how job execution logs are stored.
The default behaviour is unchanged (`memory` — in-memory, lost at process end):

```yaml
# config/packages/yokai_batch.yaml
yokai_batch:
logging:
type: memory # memory (default) | null | stream | service
```

To persist logs to files (requires `yokai/batch-monolog`):

```yaml
yokai_batch:
logging:
type: stream
stream:
directory: '%kernel.logs_dir%/batch'
sub_directories: 0 # subdirectory depth (0 = flat)
chars_per_directory: 0 # characters per subdirectory level
processors: [] # Monolog processor service ids
formatter: ~ # Monolog formatter service id (optional)
```

To use a custom service implementing `JobExecutionLoggerFactoryInterface`:

```yaml
yokai_batch:
logging:
type: service
service: App\Batch\MyJobExecutionLoggerFactory
```

---

### Symfony bundle configuration — storage DSN

The `storage` key of the `yokai_batch` bundle now accepts a **DSN string** as a shorthand:
Expand Down
10 changes: 7 additions & 3 deletions composer.json
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
"doctrine/orm": "^3.0",
"doctrine/persistence": "^4.0",
"league/flysystem": "^3.0",
"monolog/monolog": "^3.0",
"openspout/openspout": "^4.0",
"psr/container": "^1.0",
"psr/event-dispatcher": "^1.0",
Expand Down Expand Up @@ -63,7 +64,8 @@
"yokai/batch-symfony-messenger": "self.version",
"yokai/batch-symfony-serializer": "self.version",
"yokai/batch-symfony-uid": "self.version",
"yokai/batch-symfony-validator": "self.version"
"yokai/batch-symfony-validator": "self.version",
"yokai/batch-monolog": "self.version"
},
"autoload": {
"psr-4": {
Expand All @@ -78,7 +80,8 @@
"Yokai\\Batch\\Bridge\\Symfony\\Messenger\\": "src/batch-symfony-messenger/src/",
"Yokai\\Batch\\Bridge\\Symfony\\Serializer\\": "src/batch-symfony-serializer/src/",
"Yokai\\Batch\\Bridge\\Symfony\\Uid\\": "src/batch-symfony-uid/src/",
"Yokai\\Batch\\Bridge\\Symfony\\Validator\\": "src/batch-symfony-validator/src/"
"Yokai\\Batch\\Bridge\\Symfony\\Validator\\": "src/batch-symfony-validator/src/",
"Yokai\\Batch\\Bridge\\Monolog\\": "src/batch-monolog/src/"
}
},
"autoload-dev": {
Expand All @@ -98,7 +101,8 @@
"Yokai\\Batch\\Tests\\Bridge\\Symfony\\Messenger\\": "src/batch-symfony-messenger/tests/",
"Yokai\\Batch\\Tests\\Bridge\\Symfony\\Serializer\\": "src/batch-symfony-serializer/tests/",
"Yokai\\Batch\\Tests\\Bridge\\Symfony\\Uid\\": "src/batch-symfony-uid/tests/",
"Yokai\\Batch\\Tests\\Bridge\\Symfony\\Validator\\": "src/batch-symfony-validator/tests/"
"Yokai\\Batch\\Tests\\Bridge\\Symfony\\Validator\\": "src/batch-symfony-validator/tests/",
"Yokai\\Batch\\Tests\\Bridge\\Monolog\\": "src/batch-monolog/tests/"
}
}
}
1 change: 1 addition & 0 deletions docs/docs/bridges.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ Here is the complete list of what to expect:
Doctrine ORM </bridges/doctrine-orm>
Doctrine Persistence </bridges/doctrine-persistence>
Flysystem </bridges/league-flysystem>
Monolog </bridges/monolog>
OpenSpout </bridges/openspout>
Symfony Console </bridges/symfony-console>
Symfony Messenger </bridges/symfony-messenger>
Expand Down
45 changes: 45 additions & 0 deletions docs/docs/bridges/monolog.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
Bridge with ``monolog/monolog``
============================================================

Refer to the `official documentation <https://seldaek.github.io/monolog/>`__ on Monolog's website.

This bridge provides file-based log storage for job executions using Monolog's ``StreamHandler``.


Store job execution logs in files
------------------------------------------------------------

| The
`StreamJobExecutionLoggerFactory <https://github.com/yokai-php/batch-monolog/blob/1.x/src/StreamJobExecutionLoggerFactory.php>`__
creates one log file per job execution, named after the job execution id.
| Logs written during execution are readable afterwards via the same reference.

.. literalinclude:: monolog/stream-logger-factory.php
:language: php

.. seealso::
| :doc:`What is a job execution logger? </core-concepts/job-execution-logger>`
| :doc:`Bridge with Symfony Framework </bridges/symfony-framework>`


Organize log files in subdirectories
------------------------------------------------------------

| By default, all log files land in the same directory.
| For large workloads this can become a filesystem performance issue.
| You can configure the factory to spread files across nested subdirectories,
similarly to how Git stores objects:

.. literalinclude:: monolog/stream-logger-factory-with-subdirectories.php
:language: php


Customize the Monolog stack
------------------------------------------------------------

| You can provide Monolog processors and a custom formatter to control how records are written.
| Processors are applied to every log record before it is written.
| The formatter controls the final string representation written to the file.

.. literalinclude:: monolog/stream-logger-factory-with-processors.php
:language: php
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
<?php

declare(strict_types=1);

use Monolog\Formatter\JsonFormatter;
use Monolog\Processor\PsrLogMessageProcessor;
use Yokai\Batch\Bridge\Monolog\StreamJobExecutionLoggerFactory;

new StreamJobExecutionLoggerFactory(
directory: '/var/log/batch',
processors: [new PsrLogMessageProcessor()],
formatter: new JsonFormatter(),
);
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
<?php

declare(strict_types=1);

use Yokai\Batch\Bridge\Monolog\StreamJobExecutionLoggerFactory;

// With subDirectories: 2, charsPerDirectory: 2, a job execution id
// "60996f72-4f54-4184-9268-35ffdecf0de6" would be stored at:
// /var/log/batch/60/99/60996f72-4f54-4184-9268-35ffdecf0de6.log

new StreamJobExecutionLoggerFactory(
directory: '/var/log/batch',
subDirectories: 2,
charsPerDirectory: 2,
);
9 changes: 9 additions & 0 deletions docs/docs/bridges/monolog/stream-logger-factory.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
<?php

declare(strict_types=1);

use Yokai\Batch\Bridge\Monolog\StreamJobExecutionLoggerFactory;

new StreamJobExecutionLoggerFactory(
directory: '/var/log/batch',
);
63 changes: 63 additions & 0 deletions docs/docs/bridges/symfony-framework.rst
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,69 @@ You can configure what your id will be like:
# id: symfony.uuid.time
# id: symfony.ulid

Configure job execution log storage
------------------------------------------------------------

| By default, job execution logs are kept in memory and are lost when the process ends.
| You can configure a different strategy using the ``logging`` key:

.. code-block:: yaml

# config/packages/yokai_batch.yaml
yokai_batch:
logging:
type: memory # memory (default) | null | stream | service

* ``memory``: stores logs in memory (default) — suitable when you have few logs per job
* ``null``: discards all logs
* ``stream``: writes one log file per job execution via Monolog (requires ``yokai/batch-monolog``)
* ``service``: delegates to a custom service implementing ``JobExecutionLoggerFactoryInterface``

Stream logging (file-based)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The ``stream`` type writes one log file per job execution via Monolog's ``StreamHandler``.
It requires the ``yokai/batch-monolog`` package:

.. code-block:: shell

composer require yokai/batch-monolog

.. code-block:: yaml

# config/packages/yokai_batch.yaml
yokai_batch:
logging:
type: stream
stream:
directory: '%kernel.logs_dir%/batch' # where log files are stored
sub_directories: 0 # number of subdirectory levels (0 = flat)
chars_per_directory: 0 # characters from the id used per level
processors: [] # list of Monolog processor service ids
formatter: ~ # Monolog formatter service id (optional)

.. note::
With ``sub_directories: 2`` and ``chars_per_directory: 2``, a job execution with id
``60996f72`` would be stored at ``{directory}/60/99/60996f72.log``.
This is useful when a large number of job executions are expected.

.. seealso::
| :doc:`Bridge with Monolog </bridges/monolog>`

Custom logger factory service
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If none of the built-in types fit your needs, you can point to your own service:

.. code-block:: yaml

# config/packages/yokai_batch.yaml
yokai_batch:
logging:
type: service
service: App\Batch\MyJobExecutionLoggerFactory


User interface to visualize ``JobExecution``
------------------------------------------------------------

Expand Down
56 changes: 56 additions & 0 deletions docs/docs/core-concepts/job-execution-logger.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
Job execution logger
============================================================

What is a job execution logger?
------------------------------------------------------------

Every ``JobExecution`` carries a
`JobExecutionLoggerInterface <https://github.com/yokai-php/batch/tree/1.x/src/Logger/JobExecutionLoggerInterface.php>`__
instance that records log messages produced during the execution.

It extends PSR-3's ``LoggerInterface``, so anything you can do with a standard logger
(``info``, ``warning``, ``error``, etc.) works here.
On top of that, it exposes two extra methods to read the logs back:

* ``getLogs(): iterable<string>`` — streams log lines lazily, suitable for large files or HTTP streaming
* ``getLogsContent(): string`` — returns the full log content as a string (loads everything in memory)

It also exposes ``getReference(): string``, a string serialized alongside the ``JobExecution``
that allows the log storage to be restored after the execution is loaded back from storage.

How are loggers created and restored?
------------------------------------------------------------

The
`JobExecutionLoggerFactoryInterface <https://github.com/yokai-php/batch/tree/1.x/src/Factory/JobExecutionLoggerFactoryInterface.php>`__
is responsible for the full lifecycle:

* ``create(string $jobExecutionId)`` — called when a new ``JobExecution`` is created.
Returns a fresh logger ready to receive messages.
* ``restore(string $logsReference)`` — called when a ``JobExecution`` is loaded from storage.
Reconstructs the logger from the reference that was serialized with the execution.

You should never have to call these methods yourself; the framework handles it internally.

What implementations exist?
------------------------------------------------------------

**Built-in implementations:**

* `InMemoryJobExecutionLoggerFactory <https://github.com/yokai-php/batch/tree/1.x/src/Factory/JobExecutionLoggerFactory/InMemoryJobExecutionLoggerFactory.php>`__
(default): keeps logs in memory. Logs are lost when the process ends.
* `NullJobExecutionLoggerFactory <https://github.com/yokai-php/batch/tree/1.x/src/Factory/JobExecutionLoggerFactory/NullJobExecutionLoggerFactory.php>`__:
discards all log messages.

**From bridges:**

* From ``monolog/monolog`` bridge:

* `StreamJobExecutionLoggerFactory <https://github.com/yokai-php/batch-monolog/blob/1.x/src/StreamJobExecutionLoggerFactory.php>`__:
writes one log file per job execution using Monolog's ``StreamHandler``.
Logs survive the process and can be read back from the file.

.. seealso::
| :doc:`What is a job execution? </core-concepts/job-execution>`
| :doc:`Bridge with Monolog </bridges/monolog>`
| :doc:`Bridge with Symfony Framework </bridges/symfony-framework>`
3 changes: 2 additions & 1 deletion docs/docs/core-concepts/job-execution.rst
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,10 @@ What kind of information does it hold?
* ``JobExecution::$failures``: A list of failures (usually exceptions)
* ``JobExecution::$warnings``: A list of warnings (usually skipped items)
* ``JobExecution::$summary``: A summary (can contain any data you wish to store)
* ``JobExecution::$logs``: Some logs
* ``JobExecution::$logger``: The associated logger
* ``JobExecution::$childExecutions``: Some child execution

.. seealso::
| :doc:`How is a job execution created? </core-concepts/job-launcher>`
| :doc:`How can I retrieve a job execution afterwards? </core-concepts/job-execution-storage>`
| :doc:`How are job execution logs stored? </core-concepts/job-execution-logger>`
2 changes: 2 additions & 0 deletions docs/docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ Explore some of the things that could be built with **Yokai Batch**:
core-concepts/item-job
core-concepts/job-execution
core-concepts/job-execution-storage
core-concepts/job-execution-logger
core-concepts/job-with-children
core-concepts/job-parameter-accessor
core-concepts/aware-interfaces
Expand Down Expand Up @@ -98,6 +99,7 @@ Explore some of the things that could be built with **Yokai Batch**:
Doctrine ORM </bridges/doctrine-orm>
Doctrine Persistence </bridges/doctrine-persistence>
Flysystem </bridges/league-flysystem>
Monolog </bridges/monolog>
OpenSpout </bridges/openspout>
Symfony Console </bridges/symfony-console>
Symfony Framework </bridges/symfony-framework>
Expand Down
1 change: 1 addition & 0 deletions phpstan.neon
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,4 @@ parameters:
- src/batch-symfony-messenger/src/
- src/batch-symfony-serializer/src/
- src/batch-symfony-validator/src/
- src/batch-monolog/src/
8 changes: 8 additions & 0 deletions src/batch-monolog/.gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
.github export-ignore
.gitattributes export-ignore
.gitignore export-ignore
docs/ export-ignore
tests/ export-ignore
LICENSE export-ignore
phpunit.xml export-ignore
*.md export-ignore
Loading
Loading