feat(server): implement log rotation based on size and retention

[Svecco]

Redirected from issue #46 , the progress:

• Implemented _install_log_rotation_handler function, which was previously marked as todo;
• Implemented cleanup_log_files to delete logs based on the configured time threshold & size;
• Merged multiple directory traversals into a single traversal;
• Reduced multiple metadata() calls by collecting data in single traversal;
• Split log cleanup logic into smaller, more manageable functions;
• Improved error handling with detailed error messages and proper error propagation;
• Fixed an issue where the directory did not exist when loading logs;
• Added disk space checking before initializing file logging;

Before merge:

• Verified to comply with Apache specifications;
• Add a relevant unit test for logging;
• Confirm that the code has been optimized well?
• Squash commits to a proper number after polishing;
• Solve conflicts and confirm it is ready for being merged;

Plan:
Around the next several weekends.

[hubcio]

I think changes in core/message_bus are not related to log retention, looks like merge problem.

[Svecco]

I think changes in core/message_bus are not related to log retention, looks like merge problem.

Yes, that was the case. I have just rebased it to the correct state.

[Svecco]

Result of running RUST_BACKTRACE=1 cargo test -p server:

Terminal

running 62 tests
test configs::sharding::tests::test_parse_all ... ok
test configs::sharding::tests::test_parse_range ... ok
test configs::sharding::tests::test_parse_numa_auto ... ok
test log::logger::tests::test_graceful_shutdown_flag ... ok
test log::logger::tests::test_calculate_max_files_with_values ... ok
test configs::sharding::tests::test_parse_count ... ok
test log::logger::tests::test_logging_config_default_values ... ok
test configs::sharding::tests::test_parse_numa_explicit ... ok
test log::logger::tests::test_cleanup_log_files_functions ... ok
test log::logger::tests::test_calculate_max_files ... ok
test log::logger::tests::test_log_directory_creation ... ok
test binary::command::tests::should_be_serialized_as_bytes_and_deserialized_from_bytes ... ok
test log::logger::tests::test_config_values_used_in_handlers ... ok
test log::logger::tests::test_disk_space_check ... ok
test log::logger::tests::test_logging_creation ... ok
test log::logger::tests::test_user_defined_rotation_check_interval_parsing ... ok
test log::logger::tests::test_user_defined_max_total_size_parsing ... ok
test shard::task_registry::shutdown::tests::test_child_token_child_trigger ... ok
test shard::task_registry::registry::tests::test_oneshot_critical_failure ... ok
test shard::task_registry::registry::tests::test_shutdown_prevents_spawning ... ok
test shard::task_registry::registry::tests::test_oneshot_completion_detection ... ok
test shard::task_registry::shutdown::tests::test_shutdown_trigger ... ok
test shard::task_registry::shutdown::tests::test_child_token_parent_trigger ... ok
test streaming::personal_access_tokens::personal_access_token::tests::personal_access_token_should_be_created_with_random_secure_value_and_hashed_successfully ... ok
test streaming::personal_access_tokens::personal_access_token::tests::personal_access_token_should_be_expired_given_passed_expiry ... ok
test streaming::polling_consumer::tests::given_consumer_group_with_numeric_id_polling_consumer_group_should_be_created ... ok
test streaming::polling_consumer::tests::given_consumer_with_named_id_polling_consumer_should_be_created ... ok
test streaming::polling_consumer::tests::given_consumer_with_numeric_id_polling_consumer_should_be_created ... ok
test streaming::polling_consumer::tests::given_distinct_named_ids_unique_polling_consumer_ids_should_be_created ... ok
test shard::task_registry::shutdown::tests::test_child_token_no_polling_overhead ... ok
test streaming::utils::address::tests::test_extract_ip_ipv4 ... ok
test streaming::utils::address::tests::test_extract_ip_ipv6 ... ok
test streaming::utils::address::tests::test_extract_port ... ok
test streaming::utils::hash::tests::given_same_input_calculate_should_produce_same_output ... ok
test tcp::tcp_socket::tests::given_override_defaults_socket_should_be_configured ... ok
test versioning::tests::should_parse_basic_semantic_version ... ok
test versioning::tests::should_parse_semantic_version_with_build_metadata ... ok
test versioning::tests::should_parse_semantic_version_with_prerelease ... ok
test versioning::tests::should_parse_semantic_version_with_prerelease_and_build_metadata ... ok
test versioning::tests::should_parse_semver_on_compile_time ... ok
test streaming::utils::address::tests::test_extract_ip_no_port ... ok
test streaming::utils::address::tests::test_extract_port_no_port ... ok
test versioning::tests::should_compare_versions_correctly ... ok
test versioning::tests::should_parse_semantic_version_with_alpha ... ok
test versioning::tests::should_load_the_expected_version_from_package_definition ... ok
test versioning::tests::test_const_parse_u32_range ... ok
test versioning::tests::test_const_str_slice ... ok
test versioning::tests::test_find_byte_in_range ... ok
test versioning::tests::test_find_byte_pos_or_len ... ok
test versioning::tests::test_semantic_version_current ... ok
test streaming::deduplication::message_deduplicator::tests::message_deduplicator_should_insert_only_unique_identifiers ... ok
test shard::task_registry::shutdown::tests::test_sleep_or_shutdown_completes ... ok
test shard::task_registry::shutdown::tests::test_sleep_or_shutdown_interrupted ... ok
test shard::task_registry::registry::tests::test_timeout_error ... ok
test shard::task_registry::registry::tests::test_composite_timeout_insufficient ... ok
test versioning::tests::test_const_parse_u32_range_empty - should panic ... ok
test versioning::tests::test_const_parse_u32_range_invalid - should panic ... ok
test shard::task_registry::registry::tests::test_composite_timeout ... ok
test shard::task_registry::registry::tests::test_periodic_last_tick_timeout ... ok
test streaming::users::user::tests::should_be_created_given_specific_status ... ok
test streaming::users::user::tests::given_root_user_data_and_credentials_should_be_valid ... ok
test streaming::deduplication::message_deduplicator::tests::message_deduplicator_should_evict_identifiers_after_given_time_to_live ... ok

test result: ok. 62 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.60s

     Running unittests src/main.rs (target/debug/deps/iggy_server-195528b665373263)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

   Doc-tests server

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

[Svecco]

@hubcio @spetz I've just squashed commits into one.