diff --git a/CHANGELOG.md b/CHANGELOG.md index 66f1fa7b4459..9f184b80a0dc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,14 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html). +## [7.0.13] + +[7.0.13]: https://github.com/microsoft/CCF/releases/tag/ccf-7.0.13 + +### Changed + +- TLS is now terminated by OpenSSL directly on the socket, rather than being relayed over the ringbuffer and decrypted through a memory BIO. The session interfaces in `include/ccf/node/session.h` and `include/ccf/research/custom_protocol_subsystem_interface.h` have changed shape accordingly: a session now receives and emits plaintext, taking ownership of the inbound buffer, and writes its output through a `ccf::SessionWriter` rather than a `tls::Context` (#8117). + ## [7.0.12] [7.0.12]: https://github.com/microsoft/CCF/releases/tag/ccf-7.0.12 diff --git a/CMakeLists.txt b/CMakeLists.txt index e136af2d2920..f362ae88662b 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -706,10 +706,10 @@ if(BUILD_TESTS) ) add_unit_test( - rpc_connections_test - ${CMAKE_CURRENT_SOURCE_DIR}/src/host/test/rpc_connections.cpp + openssl_server_test + ${CMAKE_CURRENT_SOURCE_DIR}/src/host/test/openssl_server_test.cpp ) - target_link_libraries(rpc_connections_test PRIVATE uv) + target_link_libraries(openssl_server_test PRIVATE ccf_tasks http_parser uv) add_unit_test( raft_test @@ -870,7 +870,10 @@ if(BUILD_TESTS) ${CMAKE_CURRENT_SOURCE_DIR}/src/node/test/node_info_json.cpp ) - add_unit_test(tls_test ${CMAKE_CURRENT_SOURCE_DIR}/src/tls/test/main.cpp) + add_unit_test( + tls_test + ${CMAKE_CURRENT_SOURCE_DIR}/src/clients/tls/test/main.cpp + ) target_link_libraries(tls_test PRIVATE ${CMAKE_THREAD_LIBS_INIT}) add_unit_test( diff --git a/doc/architecture/tls_internals.rst b/doc/architecture/tls_internals.rst index e7acbd4802af..2353a2237512 100644 --- a/doc/architecture/tls_internals.rst +++ b/doc/architecture/tls_internals.rst @@ -4,106 +4,156 @@ TLS Internals Overview ~~~~~~~~ -In CCF, the :term:`TLS` layer is implemented using OpenSSL 3.3. However, the original implementation was using OE's MbedTLS library, which the current implementation replaced. During the transition period, the OpenSSL implementation had to emulate the previous MbedTLS one and the remaining code isn't particularly suited to OpenSSL. +In CCF, the :term:`TLS` layer is implemented using OpenSSL (3.3 or later). -This document describes how that works, to facilitate further changes. +TLS is terminated in the **connection layer**: OpenSSL owns the socket file descriptor directly. The existing host ``libuv`` loop reports file descriptor readiness, and a per-connection worker drives the non-blocking handshake, reads, writes and graceful close. Everything above the connection layer, including HTTP parsing and endpoint dispatch, only ever sees plaintext. -Enclave Connections -~~~~~~~~~~~~~~~~~~~ +This document describes the connection layer and its interface to the session layer above it. -CCF handles RPC requests by managing a collection of HTTPS sessions on each node. The HTTP session receives raw TCP bytes from the :term:`ring buffer` and passes it to the TLS implementation that tries to decrypt on read (after a successful handshake). +Layers +~~~~~~ -If there isn't enough information to decrypt on read, the TLS layer responds with a 'WANTS_READ' message, meaning more packets are needed to complete the message, so the endpoint tried to get more data from the ring buffer. +A single RPC interface is served by these pieces: -Once all incoming data is decrypted, the HTTP session parses the plain text data and passes the parsed request to the application, which process it and produces a plain text response. The session then serialises this response, encrypts it through the TLS layer, and writes the encrypted data to the :term:`ring buffer` to be sent to the original caller. +- :ccf_repo:`OpenSSLServer ` owns the listening and accepted sockets and registers ``uv_poll_t`` handles for them on the existing host loop. Each accepted connection holds an ``SSL`` object bound to its file descriptor with ``SSL_set_fd``. It emits decrypted bytes through an ``OnData`` callback and reports teardown through ``OnClose``. +- :ccf_repo:`OpenSSLSessionManager ` bridges the transport to the session layer. It lazily creates one ``ccf::Session`` per connection using a caller-supplied factory, and implements :ccf_repo:`ccf::SessionWriter ` so that a session's outbound plaintext is handed back to the transport. +- :ccf_repo:`ccf::PlaintextSession ` is the base for the protocol sessions (:ccf_repo:`HTTPServerSession `, :ccf_repo:`HTTP2ServerSession `). It receives plaintext, and emits plaintext through its ``SessionWriter``. +- :ccf_repo:`RPCConnectionManager ` owns one of these stacks per configured RPC interface, and holds the cross-interface policy: certificates, session caps, and metrics. -TLS Implementation -~~~~~~~~~~~~~~~~~~ +Because TLS lives below the session, there is no separate "encrypted session" type. The difference between a TLS interface and an ``UNSECURED`` one is a flag on the connection layer, not a different session class. -The TLS implementation in 'src/tls' has three main components: +Note that the inbound and outbound paths are not symmetric. Inbound plaintext is pushed straight into the session, but a session cannot touch the socket: it hands bytes to a ``SessionWriter``, which queues them for the owning connection. This is what allows sessions to run on any worker thread while each socket is still only ever touched by one thread at a time. -- A 'Certificate Authority' (CA), which has a root certificate that can sign other certificates in the network. This CA certificate is generated by the network outside of the TLS implementation and is imported when clients and servers are created. -- A 'Certificate' (Cert), which is the server/client's own certificate, signed by the CA and unique to this server or client. -- A 'Context', which is the common implementation of both 'Client' and 'Server' and provides input (read and decrypt) and output (encrypt and write) over the ring buffer and its triggers. Clients and servers are basically just a Context with a certificate. +Listening +~~~~~~~~~ -Both CA and Cert have internal logic to validate their certificates and private keys, and the Context has logic to complete a TLS handshake, read and write using encryption and to query the peer's certificate. +The listening socket is resolved with ``getaddrinfo`` (rather than ``inet_pton``) so that interfaces can be configured with hostnames such as ``localhost`` and with IPv6 addresses, not only IPv4 literals. ``SO_REUSEADDR`` is set, so that a port left in ``TIME_WAIT`` by a previous process can be rebound on restart. -The next layer up is the 'TLSSession' (of which 'HTTPSession' owns an instance) that holds the ring buffer, the pending read and write buffers, and the call backs for sending and receiving data. +``SO_REUSEPORT`` is deliberately not set. There is one listener per interface, so it would buy nothing today, and it would weaken two useful properties: a second node misconfigured onto the same port would bind successfully instead of failing with ``EADDRINUSE``, with connections then split between the two at random; and any process running with the same effective UID could bind the same port and siphon off a share of inbound connections. If per-worker listeners are added later, it should be enabled explicitly for that case rather than unconditionally. -Receiving Messages -~~~~~~~~~~~~~~~~~~ +After binding, the actually bound port is read back with ``getsockname``. This supports configuring port ``0`` to request an ephemeral port, which the test infrastructure relies on, and is why resolved addresses are only known once the interface is listening. -When a REST node message is received through the ring buffer, the session's ``handle_incoming_data_thread()`` method passes it to ``TLSSession::read()``, which in turn calls ``tls::Context::read()``. +Certificates +~~~~~~~~~~~~ -However, when creating the TLS 'Context', the 'TLSSession' had to register some callbacks of its own, too. This is one of the parts that was specific to MbedTLS and that was (unnaturally) replicated to OpenSSL. +An interface starts listening before its certificate is necessarily known. A joining node, for example, only obtains the service certificate once its join request has been accepted. ``OpenSSLServer`` therefore accepts an empty certificate at construction, and refuses TLS connections until one is supplied. -Those callbacks are implemented in 'TLSSession' because they need access to the ``pending_read`` buffers to read from and the 'ring buffer' to write to. But the TLS context ('SSL' and 'SSL_CTX' objects) need to encrypt/decrypt data, and they can only do that from 'BIO's in OpenSSL, so the callback has a mix of ring buffer and BIO handling that tricks both sides to take the appropriate steps at the right time in this complex dance. +``set_server_cert()`` may be called from any thread. It does not build the ``SSL_CTX`` inline: it queues the request and wakes the loop, so the context is only ever created or replaced on the loop thread. This avoids locking the context on the hot path. Replacing the context affects only subsequent connections; existing connections keep the context they were created with, which OpenSSL keeps alive by reference counting. -In the 'read' case, the message has arrived into the ``pending_read`` buffer via a (previously triggered) ring buffer callback, and that data is written to TLS's 'read' BIO (via ``BIO_write_ex``) `before` TLS itself reads it. That data is still encrypted, so when the TLS reads it with ``SSL_read_ex``, it tries to decrypt and on success, returns a plain text buffer. On error, it emits the error or a 'WANTS_READ' status, so the endpoint can try to extract more information from the ``pending_read`` buffer again. +The server context sets a minimum version of TLS 1.2 and, when the interface is configured for HTTP/2, advertises ``h2`` via ALPN. -Upon success, the read BIO is flushed and new data can be read again. The plain text result is returned to the HTTP session, which sends it to the application to process. +Cryptographic policy +~~~~~~~~~~~~~~~~~~~~ -Sending Messages -~~~~~~~~~~~~~~~~ +The context restricts what the handshake may negotiate: -When the application finishes, it returns a plain text response. The session passes that message to ``TLSSession::send_data()``, which fills the ``pending_write`` buffer and calls ``write_some()`` via ``flush()``. ``write_some()`` itself calls ``tls::Context::write()``. +- TLS 1.2 is limited to four AES-GCM cipher suites, all with ECDHE key exchange and either ECDSA or RSA authentication. +- TLS 1.3 is limited to ``TLS_AES_256_GCM_SHA384`` and ``TLS_AES_128_GCM_SHA256``. +- The key exchange group list prefers the hybrid post-quantum groups ``SecP384r1MLKEM1024``, ``SecP256r1MLKEM768`` and ``X25519MLKEM768``, falling back to ``P-521``, ``P-384`` and ``P-256``. The hybrid groups are prefixed with ``?`` so that the list still loads on OpenSSL versions which do not implement them. +- Renegotiation is disabled, and session resumption on renegotiation with it, to avoid the associated denial-of-service vectors. +- Where a choice remains, the server's ordering wins over the client's. -This is the same process as for reads: a callback in 'TLSSession' was registered for the 'write' BIO, with access to the ``pending_write`` and the ring buffer. But in this case, the callback is only executed `after` the TLS layer has encrypted the data and written to the 'write' BIO. +Note that in TLS 1.3 the *client* effectively chooses the group: OpenSSL picks the first client-offered group that the server also supports, so the server list acts as a filter rather than a preference. -Now, we take that (encrypted) data, flush the BIO ourselves (to avoid clogging the pipes with the next message) and send it through the ring buffer with the ``RINGBUFFER_TRY_WRITE_MESSAGE(::tcp::tcp_outbound, ...)`` macro. +This policy is defined in one place, ``build_server_ctx()``. It is asserted from the wire by the ``openssl_server_test`` unit tests and, for a running service, by the :ccf_repo:`tls_groups ` end-to-end test. Both only check the hybrid groups when built with ``-DTEST_HYBRID_TLS_GROUPS=ON``, since those groups require OpenSSL 3.5 or later. -This is what actually sends the message back to the client, so when this callback returns, the remaining stack just returns the status of that write. Different errors are treated at different levels (TLS errors in Context, ring buffer errors in TLSSession and HTTP errors in HTTPSession). +The context also sets ``SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER`` and ``SSL_MODE_ENABLE_PARTIAL_WRITE``, both of which are required by the write path described below. -Why OpenSSL? -~~~~~~~~~~~~ +Client authentication +~~~~~~~~~~~~~~~~~~~~~ + +The server calls ``SSL_CTX_set_verify`` with ``SSL_VERIFY_PEER`` and an accept-all callback. This is deliberate: it makes the server *request* the client certificate during the handshake so that it is available for application-level caller authentication, without the TLS layer itself rejecting clients. Whether a certificate is required, and whether it is acceptable, is decided by the endpoint's authentication policy. + +Reading +~~~~~~~ + +When ``libuv`` reports a connection readable, the loop hands the connection to its worker (see `Threading`_), which calls ``SSL_read`` repeatedly until it reports ``SSL_ERROR_WANT_READ``. Every chunk of decrypted bytes is passed to the ``OnData`` callback as it is produced, so a single readable event may yield several callbacks. + +A single pass is capped at a fixed number of bytes so that one busy connection cannot monopolise its worker. Reaching that cap does not end the read: OpenSSL may still be holding buffered records, in which case the file descriptor is *not* readable and waiting for a further ``uv_poll`` event would stall the connection. The worker therefore reports ``SSL_has_pending()`` back to the loop, which immediately schedules another pass instead of re-arming ``UV_READABLE``. ``SSL_has_pending()`` rather than ``SSL_pending()``, because it also covers bytes which have been read off the socket but not yet processed into plaintext. + +``OpenSSLSessionManager`` receives those bytes, finds or creates the session for that connection, and calls ``handle_incoming_data``. The session dispatches the actual parsing to its own ``OrderedTasks``, so neither the loop thread nor the connection worker blocks on application work. + +``SSL_ERROR_WANT_WRITE`` on a read is not an error: a TLS 1.3 key update needs the socket to become writable, so the connection is left open with ``UV_WRITABLE`` armed. Any other result, whether a clean ``SSL_ERROR_ZERO_RETURN``, an unclean EOF, or a fatal error, closes the connection. + +Inbound flow control +~~~~~~~~~~~~~~~~~~~~ + +How much the node reads is bounded by how far behind the application is. ``OpenSSLSessionManager`` charges every chunk it hands to a session against a node-wide budget, and releases it once the session reports that chunk processed. Because request execution happens inside the parse task, "processed" means the request has actually run, so the budget measures unexecuted work rather than merely unparsed bytes. + +While the budget is exhausted, no interface arms ``UV_READABLE``. The sending client's own :term:`TCP` window then closes, and it cannot queue more work than the node can retire. Nothing is dropped or refused: reads simply pause and resume. + +The budget is node-wide rather than per-connection, so a single connection can exhaust it and pause the others. That is deliberate: a per-connection limit would have to be smaller than one maximum-sized request before it could bound total node memory to the same figure. -The main reasons why we moved to OpenSSL are: +Releasing the budget wakes every registered transport, not only the one whose session made room, so an interface with no traffic of its own does not stay paused. A session which never reports - a custom protocol, or one whose queued work is cancelled as it is destroyed - cannot strand the budget either: whatever is still outstanding for a connection is released when that connection is torn down. -- We already use OpenSSL for our crypto library ('src/crypto'). -- We wanted TLS 1.3 support that the MbedTLS version in use did not provide. -- We wanted to support QUIC through OpenSSL. +A connection paused this way is performing no I/O, so it ages towards the idle timeout in the usual way. That is intentional: a connection which cannot make progress in either direction for the whole idle period is one the node is too far behind to serve. -MbedTLS has since been removed from the runtime implementation. +Writing and backpressure +~~~~~~~~~~~~~~~~~~~~~~~~ -MbedTLS vs OpenSSL -~~~~~~~~~~~~~~~~~~ +``send()`` is thread-safe. It appends the plaintext to a queue guarded by a mutex and signals a ``uv_async_t``, waking the loop thread, which moves the bytes onto the target connection and schedules a worker pass to attempt the write. -As stated above, the current OpenSSL implementation is `emulating` the previous MbedTLS one, so some oddities are observed. +Writes are where genuine backpressure appears. ``SSL_write`` on a non-blocking socket may report ``SSL_ERROR_WANT_WRITE`` after consuming only part of the buffer. The remainder stays buffered against the connection, ``UV_WRITABLE`` is armed, and the write resumes when the socket next becomes writable. Because the socket is owned by OpenSSL and is non-blocking, this reflects the real state of the :term:`TCP` send buffer rather than an internal approximation. -First, MbedTLS returns errors as negative values and amount of data handled as positive values. OpenSSL concurs on positive values but returns 0 (or -1 in previous versions) for all errors, using ``SSL_get_error`` to then classify which error and what to do. The error values are also positive. +Closing +~~~~~~~ -To simulate this, we implement the error handling at each invocation and, on error, we negate the value of the error so that we can retain the old behavior of checking for negative values. +Closing is deferred rather than immediate. A close requested while output is still buffered sets a flag and flushes; the file descriptor is only closed once the buffered bytes have been written. -Second, MbedTLS keeps all its context (configuration, connection info, read and write buffers) in a single large structure, while OpenSSL has separate structures for each and uses 'BIO' objects for buffers. Reads and writes in MbedTLS is done exclusively via callbacks. +This matters because the common pattern is to write a response and immediately close. Closing eagerly truncates any response large enough to have been backpressured, which the client observes as a connection reset partway through the body rather than as a well-formed response. -OpenSSL callbacks, however, are very different from MbedTLS ones. They are called twice for each action, one before the actual action and another after. +Server shutdown is the exception: there the close is forced rather than deferred. Each connection takes exactly one further worker pass, writing whatever the socket will accept, and then goes. Waiting for the flush would let a peer which has stopped reading hold the whole node's shutdown open indefinitely. -To simulate this we had to implement a read callback `before` the BIO read (so we could fill it up with the contents of the ring buffer) and the write callback `after` the BIO write (so we could pick up its contents and send it into the ring buffer). +Idle connections are closed separately. If an idle timeout is configured, a repeating ``uv_timer_t`` periodically sweeps connections whose last I/O is older than the timeout. This retains the once-per-second scheduling used by the previous RPC transport while comparing actual ``steady_clock`` timestamps rather than counting timer ticks. -There is a complex dance of return values in OpenSSL's callbacks. If any returns errors the action is canceled immediately. On reads, because the BIO was empty, the initial return value is an error, so we must make sure that, if there is anything in the ``pending_read`` buffer, we have to change the status to the amount of bytes read, so it can continue. +Threading +~~~~~~~~~ -Third, the handshake in MbedTLS had various types of errors, which we had to emulate by making the appropriate ``SSL_*`` calls, check the peer certificate, etc. to get the same types of responses for the same situations. +``OpenSSLServer`` does not create a thread or a private reactor. It splits its work between the existing host ``libuv`` thread and the general task system, along a single dividing line: -Finally, in MbedTLS, the configuration and session objects were setup at the same time, while in OpenSSL they're separate. We ended up duplicating every single configuration, but this is unnecessary, because once the config object is correct, any session object created from it has the same properties. +- The **loop thread** owns everything that touches the server's own state: accepting connections, the connection and identifier maps, ``uv_poll_t`` registration, building and replacing the ``SSL_CTX``, sweeping idle connections, and finally closing file descriptors. It performs no ``SSL`` operations. +- Each connection owns an :ccf_repo:`OrderedTasks ` queue, and every ``SSL`` operation for that connection - ``SSL_new``, ``SSL_accept``, ``SSL_read``, ``SSL_write``, ``SSL_shutdown``, ``SSL_free`` - runs there, as does the ``OnData`` callback. This keeps handshakes and bulk encryption off the loop thread. -But the TLS Context doesn't handle more than one session per configuration, so we could set either of them once and ignore the other. The simplest thing would be to setup just the session, but if we end up having more than one session later, we'd have to refactor that. +Work reaches the loop in one of two ways: file descriptor readiness reported through ``uv_poll_t``, or a cross-thread request (a queued write, a close, or a certificate update) posted to a mutex-guarded queue and signalled through ``uv_async_t``. Either way the loop only ever *schedules* a pass over the connection; it never performs the I/O itself. -Simplifying the OpenSSL Implementation -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +A connection is serviced by at most one worker pass at a time. Before dispatching, the loop calls ``uv_poll_stop`` for that connection, so no second pass can be scheduled while one is running. Readiness events and cross-thread commands that arrive meanwhile accumulate in loop-thread-only fields and are handed to the next pass. When a pass finishes it posts its result back to the loop, which re-arms polling, schedules another pass if more work accumulated, or tears the connection down. -With MbedTLS gone from the code base, the OpenSSL implementation can be simplified. +.. warning:: -The considerations are: + OpenSSL's error queue is **thread-local**, and successive passes over the same connection may run on different worker threads. ``SSL_get_error`` consults that queue, so an error left behind by unrelated work on the same thread can be misattributed to this connection. -- We don't need to handle errors inside the calls to read/write, but can leave for each caller to handle IFF there is an error by calling ``SSL_get_error``. This also means we don't need to negate error values, as they're in different domains. -- We can simplify the SSL configuration on startup, handshake handling and peer certificate handling. + Every ``SSL_accept``, ``SSL_read`` and ``SSL_write`` is therefore preceded by ``ERR_clear_error()``. Omitting this causes healthy keep-alive connections to be closed spuriously, because a stale error (for example a previous client disconnecting without ``close_notify``) is read as a fatal error on an unrelated connection. -However, getting rid of the callbacks and using BIOs directly is going to be hard. +Unsecured interfaces +~~~~~~~~~~~~~~~~~~~~ -First, the current callback is in 'TLSSession' because it has access to both pending buffers and the ring buffer. The TLS Context does not have access to it nor it would be wise to pass references to it, as that'd make the Context exclusive to the TLSSession. +An interface configured as ``UNSECURED`` runs the same loop with no ``SSL`` object. Connections are immediately ready, and reads and writes use ``::recv`` and ``::send`` directly. The buffering, backpressure, graceful close and idle handling are identical, so the session layer above is unchanged. + +Outbound connections +~~~~~~~~~~~~~~~~~~~~ + +``OpenSSLServer`` is inbound only. Outbound requests - fetching quote endorsements, refreshing JWT signing keys, and the recovery decision protocol - are made with libcurl, which does its own TLS. + +The only remaining OpenSSL client helpers, :ccf_repo:`CA ` and :ccf_repo:`Cert `, are used solely by the C++ test and perf clients in :ccf_repo:`src/clients `. No node code uses them. + +Why OpenSSL? +~~~~~~~~~~~~ + +CCF originally used MbedTLS, and the OpenSSL implementation that replaced it emulated MbedTLS conventions, notably negated error codes and a memory-BIO indirection driven by callbacks. + +Both are now gone. ``OpenSSLServer`` calls ``SSL_read``/``SSL_write`` on a socket-backed ``SSL`` and interprets ``SSL_get_error`` directly, and ``src/tls`` no longer exists. + +The reasons for OpenSSL remain: + +- OpenSSL is already used for the :doc:`cryptography ` implementation in ``src/crypto``. +- TLS 1.3 support. +- A path to QUIC. + +Future: QUIC +~~~~~~~~~~~~ -Second, both endpoint and TLS have a need to read and write asynchronously. Data arrives from the ring buffer at any time and the TLS implementation can request reads and writes (for example, during handshake) that the endpoint didn't request itself. +QUIC is not yet implemented. Server-side QUIC requires OpenSSL 3.5 or later, which adds ``SSL_new_listener``, ``SSL_accept_connection`` and ``OSSL_QUIC_server_method``; these are absent from the 3.3.x baseline CCF currently supports. -So if ``SSL_handshake``, ``SSL_read_ex`` and ``SSL_write_ex`` don't have direct access to read and write from the ring buffers without direct requests from the endpoints, it won't be able to conclude the asynchronous handshake and start the connection. +:ccf_repo:`DatagramServer ` exists as the substrate for that work. It is deliberately shaped as the UDP socket a QUIC server operates on: socket creation, binding, ``uv_poll_t`` readiness and per-datagram dispatch are all reusable as-is. The points that change for QUIC are marked ``QUIC EXTENSION POINT`` inline, and consist of wrapping the socket with ``BIO_new_dgram``/``SSL_set_fd`` on a listener ``SSL``, adding the OpenSSL event timeout, and replacing the datagram callback with ``SSL_handle_events``. -One possible way out of it is to create a `BIO pair `_ for each read/write action between the 'TLSSession' and the TLS 'Context', driven by two asynchronous tasks in 'TLSSession' that just poll the BIOs and buffers and pass data across. This removes a callback, but introduces polling, which is not an actual improvement. +Until then, an interface whose ``app_protocol`` is ``QUIC`` echoes each datagram straight back, statelessly. Other UDP interfaces are served by the custom protocol subsystem (:ccf_repo:`custom_protocol_subsystem_interface.h `) in the usual way, with one session per source address, swept on the same idle timeout as TCP connections. diff --git a/doc/contribute/onboarding.rst b/doc/contribute/onboarding.rst index 3cfc66930e34..49ee4cc9223c 100644 --- a/doc/contribute/onboarding.rst +++ b/doc/contribute/onboarding.rst @@ -63,12 +63,13 @@ Note that this diagram deliberately does not represent host-to-enclave communica .. mermaid:: flowchart TB - Client[HTTPS/1.1 Client auth] -- TLS 1.2 or 1.3 --> TLSSession - TLSSession[TLS Session src] -- PlainText --> HTTPSession + Client[HTTPS/1.1 Client auth] -- TLS 1.2 or 1.3 --> TLSConnection + TLSConnection[TLS Connection src] -- PlainText --> HTTPSession + HTTPSession -- PlainText --> SessionWriter[Session Writer src] + SessionWriter --> TLSConnection HTTPSession[HTTP Session src] -- Request --> Endpoint[Application Endpoint doc] Endpoint -- Response --> HTTPSession - HTTPSession --> TLSSession - TLSSession --> Client + TLSConnection --> Client Endpoint -- WriteSet --> Store[Store doc] Store -- LedgerEntry --> Ledger[Ledger doc] Ledger -- LedgerEntry --> Disk diff --git a/include/ccf/node/session.h b/include/ccf/node/session.h index da58f763a5ec..7f72a7b0d625 100644 --- a/include/ccf/node/session.h +++ b/include/ccf/node/session.h @@ -3,7 +3,7 @@ #pragma once #include -#include +#include namespace ccf { @@ -12,8 +12,55 @@ namespace ccf public: virtual ~Session() = default; - virtual void handle_incoming_data(std::span data) = 0; + // Inbound bytes for this session. Ownership is transferred, so that the + // buffer the transport has already built is moved through the session + // rather than copied again. + virtual void handle_incoming_data(std::vector&& data) = 0; virtual void send_data(std::vector&& data) = 0; virtual void close_session() = 0; }; + + // Abstract output sink injected into Sessions: a Session hands its outbound + // bytes (and connection-teardown requests) to a SessionWriter, which is + // implemented by the RPC transport. + // + // IMPORTANT: Sessions may invoke these methods from worker threads, so + // implementations MUST be thread-safe and must marshal any socket operations + // onto their I/O thread if required. + class SessionWriter + { + public: + virtual ~SessionWriter() = default; + + // `id` throughout is the connection identifier the session was created + // with (ccf::tls::ConnID). + + // Queue bytes to be written to the socket associated with `id`. Ownership + // is transferred, so that a response which may be arbitrarily large is + // moved through to the transport rather than copied again. + // + // Fire-and-forget: there is currently no backpressure signal. + // + // FUTURE: to surface genuine TCP-layer backpressure, an implementation + // should report when a connection's pending-write queue exceeds a watermark + // (tracking per-connection queued bytes) and return a writable/would-block + // status here. + virtual void write_outbound(int64_t id, std::vector&& data) = 0; + + // Tear down the connection: stop the underlying socket and drop the + // session. + virtual void close_socket(int64_t id) = 0; + + // Report that `bytes` of previously delivered inbound data have now been + // processed. The transport uses this to decide when it may read more: it + // stops reading once the node is holding more unprocessed inbound data + // than it is willing to, and resumes as sessions catch up. Without it a + // client could make the node queue work faster than it retires it, for as + // long as it liked. + // + // A session which does not report is not penalised beyond its own + // connection - the transport releases whatever is still outstanding when + // the connection closes - so this defaults to a no-op. + virtual void inbound_consumed(int64_t /*id*/, size_t /*bytes*/) {} + }; } \ No newline at end of file diff --git a/include/ccf/node/startup_config.h b/include/ccf/node/startup_config.h index 3d6cef413798..1dd33d0dfe32 100644 --- a/include/ccf/node/startup_config.h +++ b/include/ccf/node/startup_config.h @@ -29,6 +29,12 @@ namespace ccf ccf::ds::SizeString historical_cache_soft_limit = {"512MB"}; + // How long an idle RPC (client TLS) connection is kept before it is closed. + // std::nullopt disables idle closure (connections are never closed for + // being idle). + std::optional idle_connection_timeout = + ccf::ds::TimeString("60s"); + ccf::consensus::Configuration consensus = {}; ccf::NodeInfoNetwork network; diff --git a/include/ccf/research/custom_protocol_subsystem_interface.h b/include/ccf/research/custom_protocol_subsystem_interface.h index 53092b6340b2..d9f89f069128 100644 --- a/include/ccf/research/custom_protocol_subsystem_interface.h +++ b/include/ccf/research/custom_protocol_subsystem_interface.h @@ -16,7 +16,6 @@ namespace ccf { namespace tls { - class Context; using ConnID = int64_t; } @@ -24,7 +23,7 @@ namespace ccf { public: using CreateSessionFn = std::function( - ccf::tls::ConnID, const std::unique_ptr&&)>; + ccf::tls::ConnID, ccf::SessionWriter&)>; ~CustomProtocolSubsystemInterface() override = default; @@ -41,7 +40,7 @@ namespace ccf virtual std::shared_ptr create_session( const std::string& protocol_name, ccf::tls::ConnID conn_id, - const std::unique_ptr&& ctx) = 0; + ccf::SessionWriter& writer) = 0; struct Essentials { diff --git a/python/pyproject.toml b/python/pyproject.toml index e8ee11b20a3d..18462482f034 100644 --- a/python/pyproject.toml +++ b/python/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "ccf" -version = "7.0.12" +version = "7.0.13" authors = [ { name="CCF Team", email="CCF-Sec@microsoft.com" }, ] diff --git a/src/clients/rpc_tls_client.h b/src/clients/rpc_tls_client.h index a4a8251ef945..80037ca0e642 100644 --- a/src/clients/rpc_tls_client.h +++ b/src/clients/rpc_tls_client.h @@ -8,6 +8,7 @@ #include "tls_client.h" #define FMT_HEADER_ONLY +#include #include #include #include @@ -90,7 +91,7 @@ namespace client return call_raw(prep.encoded); } - std::optional last_response; + std::deque pending_responses; public: using TlsClient::TlsClient; @@ -206,20 +207,20 @@ namespace client Response read_response() { - last_response = std::nullopt; - - while (!last_response.has_value()) + while (pending_responses.empty()) { const auto next = read_all(); parser.execute(next.data(), next.size()); } - return std::move(last_response.value()); + auto response = std::move(pending_responses.front()); + pending_responses.pop_front(); + return response; } std::optional read_response_non_blocking() { - if (bytes_available()) + if (!pending_responses.empty() || bytes_available()) { return read_response(); } @@ -232,8 +233,8 @@ namespace client ccf::http::HeaderMap&& headers, std::vector&& body) override { - last_response = { - next_recv_id++, status, std::move(headers), std::move(body)}; + pending_responses.push_back( + {next_recv_id++, status, std::move(headers), std::move(body)}); } void set_prefix(const std::string& prefix_) diff --git a/src/clients/tls/README.md b/src/clients/tls/README.md new file mode 100644 index 000000000000..763759b37a26 --- /dev/null +++ b/src/clients/tls/README.md @@ -0,0 +1,14 @@ +# TLS certificate helpers (client tooling only) + +These helpers configure an OpenSSL client with a trusted root and an owned +identity: + +- `CA` parses one or more root certificates and populates an `SSL_CTX`'s + trusted certificate store. +- `Cert` holds a client's own certificate and private key, and applies them + (and the peer verification mode) to an `SSL_CTX` or `SSL`. + +Their only consumer is `TlsClient` in the parent directory, used by the C++ +test and perf clients. **No node code uses them.** The node's inbound TLS is +handled by `src/host/tls`, which builds its own `SSL_CTX`, and its outbound +requests go through libcurl. diff --git a/src/tls/ca.h b/src/clients/tls/ca.h similarity index 100% rename from src/tls/ca.h rename to src/clients/tls/ca.h diff --git a/src/tls/cert.h b/src/clients/tls/cert.h similarity index 99% rename from src/tls/cert.h rename to src/clients/tls/cert.h index 5e7776e3e058..c8cd67fe9319 100644 --- a/src/tls/cert.h +++ b/src/clients/tls/cert.h @@ -3,9 +3,9 @@ #pragma once #include "ccf/crypto/openssl/openssl_wrappers.h" +#include "clients/tls/ca.h" #include "crypto/openssl/ec_key_pair.h" #include "ds/internal_logger.h" -#include "tls/ca.h" #include #include diff --git a/src/clients/tls/test/main.cpp b/src/clients/tls/test/main.cpp new file mode 100644 index 000000000000..b71c1db7363b --- /dev/null +++ b/src/clients/tls/test/main.cpp @@ -0,0 +1,129 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the Apache 2.0 License. + +// Unit tests for the certificate helpers used by the C++ test clients, which +// configure an OpenSSL context and connection with a trusted root and an owned +// identity. The TLS transport itself is tested in +// src/host/test/openssl_server_test.cpp. + +#include "ccf/crypto/ec_key_pair.h" +#include "ccf/crypto/verifier.h" +#include "ccf/ds/x509_time_fmt.h" +#include "clients/tls/ca.h" +#include "clients/tls/cert.h" +#include "crypto/certs.h" + +#define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN +#include +#include +#include +#include +#include + +namespace +{ + constexpr size_t certificate_validity_period_days = 365; + + std::string valid_from_yesterday() + { + using namespace std::literals; + return ccf::ds::to_x509_time_string(std::chrono::system_clock::now() - 24h); + } + + ccf::crypto::Pem generate_self_signed_cert( + const ccf::crypto::ECKeyPairPtr& kp, const std::string& name) + { + return ccf::crypto::create_self_signed_cert( + kp, name, {}, valid_from_yesterday(), certificate_validity_period_days); + } + + struct NetworkCA + { + ccf::crypto::ECKeyPairPtr kp; + ccf::crypto::Pem cert; + }; + + /// Get self-signed CA certificate. + NetworkCA get_ca() + { + auto kp = ccf::crypto::make_ec_key_pair(); + return {kp, generate_self_signed_cert(kp, "CN=issuer")}; + } + + /// Creates a ::tls::Cert endorsed by the given CA. + std::unique_ptr<::tls::Cert> get_dummy_cert( + NetworkCA& net_ca, const std::string& name, bool auth_required = true) + { + auto ca = std::make_unique<::tls::CA>(net_ca.cert.str()); + + // Create a signing request and sign with the CA + auto kp = ccf::crypto::make_ec_key_pair(); + auto crt = ccf::crypto::create_endorsed_cert( + kp, + "CN=" + name, + {}, + valid_from_yesterday(), + certificate_validity_period_days, + net_ca.kp->private_key_pem(), + net_ca.cert); + + // Verify node certificate with the CA's certificate + auto v = ccf::crypto::make_verifier(crt); + REQUIRE(v->verify_certificate({&net_ca.cert})); + + return std::make_unique<::tls::Cert>( + std::move(ca), crt, kp->private_key_pem(), std::nullopt, auth_required); + } +} + +TEST_CASE("CA configures trusted certificate store") +{ + auto ca = get_ca(); + ::tls::CA trusted_ca(ca.cert.str(), true); + ccf::crypto::OpenSSL::Unique_SSL_CTX ctx(TLS_method()); + + trusted_ca.configure_trusted_cert_store(ctx); + + auto* store = SSL_CTX_get_cert_store(ctx); + REQUIRE(store != nullptr); + auto* params = X509_STORE_get0_param(store); + REQUIRE(params != nullptr); + REQUIRE( + (X509_VERIFY_PARAM_get_flags(params) & X509_V_FLAG_PARTIAL_CHAIN) != 0); +} + +TEST_CASE("Cert configures TLS verification and own certificate") +{ + auto ca = get_ca(); + + SUBCASE("auth_required requires a peer certificate") + { + auto cert = get_dummy_cert(ca, "server"); + ccf::crypto::OpenSSL::Unique_SSL_CTX ctx(TLS_method()); + + cert->configure_context(ctx); + ccf::crypto::OpenSSL::Unique_SSL ssl(ctx); + cert->configure_connection(ssl); + + constexpr auto expected_verify_mode = + SSL_VERIFY_PEER | SSL_VERIFY_FAIL_IF_NO_PEER_CERT; + REQUIRE(SSL_CTX_get_verify_mode(ctx) == expected_verify_mode); + REQUIRE(SSL_get_verify_mode(ssl) == expected_verify_mode); + REQUIRE(SSL_CTX_get0_certificate(ctx) != nullptr); + REQUIRE(SSL_get_certificate(ssl) != nullptr); + } + + SUBCASE("without auth_required a peer certificate is requested, not required") + { + auto cert = get_dummy_cert(ca, "server", false); + ccf::crypto::OpenSSL::Unique_SSL_CTX ctx(TLS_method()); + + cert->configure_context(ctx); + ccf::crypto::OpenSSL::Unique_SSL ssl(ctx); + cert->configure_connection(ssl); + + // The connection inherits the context's verification mode + REQUIRE((SSL_get_verify_mode(ssl) & SSL_VERIFY_PEER) != 0); + REQUIRE((SSL_get_verify_mode(ssl) & SSL_VERIFY_FAIL_IF_NO_PEER_CERT) == 0); + } +} diff --git a/src/clients/tls_client.h b/src/clients/tls_client.h index bdeee21ca057..2b574bee4fae 100644 --- a/src/clients/tls_client.h +++ b/src/clients/tls_client.h @@ -3,9 +3,9 @@ #pragma once #include "ccf/crypto/openssl/openssl_wrappers.h" +#include "clients/tls/ca.h" +#include "clients/tls/cert.h" #include "ds/internal_logger.h" -#include "tls/ca.h" -#include "tls/cert.h" #include #include diff --git a/src/common/configuration.h b/src/common/configuration.h index d11b775b26e6..05d47975f0ae 100644 --- a/src/common/configuration.h +++ b/src/common/configuration.h @@ -143,7 +143,8 @@ namespace ccf files_cleanup, node_to_node_message_limit, historical_cache_soft_limit, - identity_history_fetch); + identity_history_fetch, + idle_connection_timeout); DECLARE_JSON_TYPE_WITH_OPTIONAL_FIELDS(RecoveryDecisionProtocolConfig); DECLARE_JSON_REQUIRED_FIELDS( diff --git a/src/ds/work_beacon.h b/src/ds/work_beacon.h index 277f505f19d7..c603d1bf2d7a 100644 --- a/src/ds/work_beacon.h +++ b/src/ds/work_beacon.h @@ -55,6 +55,24 @@ namespace ccf::ds condition_variable.notify_all(); } + + void notify_work_available_coalesced() + { + bool notify = false; + { + std::lock_guard lock(mutex); + if (work_available == 0) + { + work_available = 1; + notify = true; + } + } + + if (notify) + { + condition_variable.notify_all(); + } + } }; using WorkBeaconPtr = std::shared_ptr; diff --git a/src/enclave/abstract_rpc_sessions.h b/src/enclave/abstract_rpc_sessions.h new file mode 100644 index 000000000000..f1cc85c7c062 --- /dev/null +++ b/src/enclave/abstract_rpc_sessions.h @@ -0,0 +1,42 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the Apache 2.0 License. +#pragma once + +#include "ccf/crypto/pem.h" +#include "ccf/service/node_info_network.h" +#include "forwarder_types.h" +#include "node/session_metrics.h" + +#include +#include + +namespace ccf +{ + class CustomProtocolSubsystem; + class CommitCallbackSubsystem; + + // The slice of RPC session management that the node (NodeState, frontends, + // Enclave, jwt refresh) depends on, independent of how connections are + // actually serviced. RPCConnectionManager implements this, so node-side code + // can hold a reference without depending on the concrete networking backend. + class AbstractRPCSessions : public AbstractRPCResponder + { + public: + ~AbstractRPCSessions() override = default; + + virtual ccf::SessionMetrics get_session_metrics() = 0; + + virtual void set_node_cert( + const ccf::crypto::Pem& cert, const ccf::crypto::Pem& pk) = 0; + virtual void set_network_cert( + const ccf::crypto::Pem& cert, const ccf::crypto::Pem& pk) = 0; + + virtual void update_listening_interface_options( + const ccf::NodeInfoNetwork& node_info) = 0; + + virtual void set_custom_protocol_subsystem( + std::shared_ptr cpss) = 0; + virtual void set_commit_callbacks_subsystem( + std::shared_ptr ccss) = 0; + }; +} diff --git a/src/enclave/enclave.h b/src/enclave/enclave.h index 5e864b7f9e0e..9485b491a50b 100644 --- a/src/enclave/enclave.h +++ b/src/enclave/enclave.h @@ -11,6 +11,7 @@ #include "ds/oversized.h" #include "ds/work_beacon.h" #include "host/ledger.h" +#include "host/rpc_connection_manager.h" #include "indexing/enclave_lfs_access.h" #include "indexing/historical_transaction_fetcher.h" #include "interface.h" @@ -34,7 +35,6 @@ #include "node/rpc/user_frontend.h" #include "node/signature_cache_subsystem.h" #include "rpc_map.h" -#include "rpc_sessions.h" #include "tasks/worker.h" namespace ccf @@ -48,7 +48,7 @@ namespace ccf ccf::ds::WorkBeaconPtr work_beacon; ccf::NetworkState network; std::shared_ptr rpc_map; - std::shared_ptr rpcsessions; + std::shared_ptr rpcsessions; std::unique_ptr node; ringbuffer::WriterPtr to_host = nullptr; std::chrono::high_resolution_clock::time_point last_tick_time; @@ -92,7 +92,7 @@ namespace ccf writer_factory(std::move(writer_factory_)), work_beacon(std::move(work_beacon_)), rpc_map(std::make_shared()), - rpcsessions(std::make_shared(*writer_factory, rpc_map)) + rpcsessions(std::make_shared(rpc_map)) { to_host = writer_factory->create_writer_to_outside(); @@ -184,23 +184,110 @@ namespace ccf signature_cache, sig_tx_interval, sig_ms_interval); + + ccf::tasks::get_main_job_board().set_work_beacon(work_beacon); } ~Enclave() { + ccf::tasks::get_main_job_board().set_work_beacon(nullptr); LOG_TRACE_FMT("Shutting down enclave"); } CreateNodeStatus create_new_node( StartType start_type_, - const ccf::StartupConfig& ccf_config_, + ccf::StartupConfig ccf_config_, std::vector& node_cert, - std::vector& service_cert) + std::vector& service_cert, + std::vector& rpc_addresses) { start_type = start_type_; rpcsessions->update_listening_interface_options(ccf_config_.network); + // Idle RPC connections are closed after this period (nullopt = never). + std::optional rpc_idle_timeout; + if (ccf_config_.idle_connection_timeout.has_value()) + { + rpc_idle_timeout = std::chrono::milliseconds( + ccf_config_.idle_connection_timeout->count_ms()); + } + rpcsessions->set_idle_connection_timeout(rpc_idle_timeout); + + // Bind and start listening on each configured RPC interface. TLS is now + // terminated in the connection: an interface whose certificate is not yet + // available refuses connections until set_*_cert provides one (a joining + // node receives the service cert later). Ephemeral ports (bind ":0") are + // assigned here, so the resolved addresses are reported back to the host + // (which writes the rpc addresses file). + { + // An object rather than a default-constructed (null) json, so that a + // node with no RPC interfaces still writes a well-formed, empty map. + nlohmann::json resolved_rpc_addresses = nlohmann::json::object(); + + // Bind interfaces with an explicit (non-zero) port before those + // requesting an ephemeral port (port 0 or unspecified). Multiple + // interfaces on a node can share a single host address (for example + // the sole ::1 IPv6 loopback), and if an ephemeral interface is bound + // first the OS may assign it the exact port that another interface is + // configured to bind, making that later bind fail with "address + // already in use". + const auto is_ephemeral = [](const auto& interface) { + const auto port = + ccf::split_net_address(interface.bind_address).second; + return port.empty() || port == "0"; + }; + std::vector ordered_interface_names; + ordered_interface_names.reserve( + ccf_config_.network.rpc_interfaces.size()); + for (const auto& [name, interface] : ccf_config_.network.rpc_interfaces) + { + if (!is_ephemeral(interface)) + { + ordered_interface_names.push_back(name); + } + } + for (const auto& [name, interface] : ccf_config_.network.rpc_interfaces) + { + if (is_ephemeral(interface)) + { + ordered_interface_names.push_back(name); + } + } + + for (const auto& name : ordered_interface_names) + { + auto& interface = ccf_config_.network.rpc_interfaces.at(name); + const auto [host, port] = + ccf::split_net_address(interface.bind_address); + // UDP interfaces use the datagram (echo) path; TCP interfaces the + // OpenSSL stream path. + const uint16_t bound = (interface.protocol == "udp") ? + rpcsessions->listen_udp(name, host, port) : + rpcsessions->listen(name, host, port); + interface.bind_address = + ccf::make_net_address(host, std::to_string(bound)); + + if (interface.published_address.empty()) + { + interface.published_address = interface.bind_address; + } + else + { + const auto [phost, pport] = + ccf::split_net_address(interface.published_address); + if (pport == "0") + { + interface.published_address = + ccf::make_net_address(phost, std::to_string(bound)); + } + } + resolved_rpc_addresses[name] = interface.bind_address; + } + const auto dumped = resolved_rpc_addresses.dump(); + rpc_addresses.assign(dumped.begin(), dumped.end()); + } + node->set_n2n_message_limit(ccf_config_.node_to_node_message_limit); historical_state_cache->set_soft_cache_limit( @@ -392,18 +479,20 @@ namespace ccf } }); - rpcsessions->register_message_handlers(bp.get_dispatcher()); - // Maximum number of inbound ringbuffer messages which will be // processed in a single iteration static constexpr size_t max_messages = 256; + bool should_wait_for_work = true; while (!bp.get_finished()) { - // Wait until the host indicates that some ringbuffer messages are - // available, but wake at least every 100ms to check thread messages - work_beacon->wait_for_work_with_timeout( - std::chrono::milliseconds(100)); + if (should_wait_for_work) + { + // Wait until the host indicates that some ringbuffer messages or + // tasks are available, but wake at least every 100ms. + work_beacon->wait_for_work_with_timeout( + std::chrono::milliseconds(100)); + } // First, read some messages from the ringbuffer auto read = bp.read_n(max_messages, circuit->read_from_outside()); @@ -422,6 +511,10 @@ namespace ccf } task = job_board.get_task(); } + // Hitting the task budget may leave queued work behind. Continue + // immediately rather than consuming the only coalesced wake and then + // sleeping with a non-empty JobBoard. + should_wait_for_work = tasks_done < max_messages; // If no messages were read from the ringbuffer and tasks were // executed, idle @@ -431,6 +524,11 @@ namespace ccf } } + LOG_INFO_FMT("Stopping RPC transports"); + // The host is still running the libuv loop at this point - it only + // exits once we send AdminMessage::stopped below. + rpcsessions->stop(asynchost::OpenSSLServer::LoopState::Running); + LOG_INFO_FMT("Enclave stopped successfully. Stopping host..."); RINGBUFFER_WRITE_MESSAGE(AdminMessage::stopped, to_host); diff --git a/src/enclave/entry_points.h b/src/enclave/entry_points.h index 8dc35be490d7..9fcca727ab02 100644 --- a/src/enclave/entry_points.h +++ b/src/enclave/entry_points.h @@ -15,6 +15,7 @@ namespace ccf const ccf::StartupConfig& ccf_config, std::vector& node_cert, std::vector& service_cert, + std::vector& rpc_addresses, StartType start_type, ccf::LoggerLevel log_level, size_t num_worker_thread, diff --git a/src/enclave/main.cpp b/src/enclave/main.cpp index fe1b11530989..cc556daff90e 100644 --- a/src/enclave/main.cpp +++ b/src/enclave/main.cpp @@ -30,6 +30,7 @@ namespace ccf const ccf::StartupConfig& ccf_config, std::vector& node_cert, std::vector& service_cert, + std::vector& rpc_addresses, StartType start_type, ccf::LoggerLevel log_level, size_t num_worker_threads, @@ -132,7 +133,7 @@ namespace ccf try { status = enclave->create_new_node( - start_type, ccf_config, node_cert, service_cert); + start_type, ccf_config, node_cert, service_cert, rpc_addresses); } catch (...) { diff --git a/src/enclave/no_more_sessions.h b/src/enclave/no_more_sessions.h new file mode 100644 index 000000000000..a37a431b428a --- /dev/null +++ b/src/enclave/no_more_sessions.h @@ -0,0 +1,34 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the Apache 2.0 License. +#pragma once + +#include "ccf/odata_error.h" + +namespace ccf +{ + // Session wrapper used when an interface is over its soft session limit. + // + // It sends a single 503 response explaining that the service is busy, then + // closes the connection. It is templated on the concrete server session type + // (HTTPServerSession / HTTP2ServerSession) so it reuses that session's + // response machinery. + template + class NoMoreSessionsImpl : public Base + { + public: + template + NoMoreSessionsImpl(Ts&&... ts) : Base(std::forward(ts)...) + {} + + void handle_incoming_data_thread(std::vector&& /*data*/) override + { + // The transport already terminated TLS, so we can respond immediately. + Base::send_odata_error_response(ccf::ErrorDetails{ + HTTP_STATUS_SERVICE_UNAVAILABLE, + ccf::errors::SessionCapExhausted, + "Service is currently busy and unable to serve new connections"}); + + Base::close_session(); + } + }; +} diff --git a/src/enclave/rpc_sessions.h b/src/enclave/rpc_sessions.h deleted file mode 100644 index ec919cda8041..000000000000 --- a/src/enclave/rpc_sessions.h +++ /dev/null @@ -1,632 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the Apache 2.0 License. -#pragma once - -#include "ccf/pal/locking.h" -#include "ccf/service/node_info_network.h" -#include "ds/internal_logger.h" -#include "ds/serialized.h" -#include "enclave/session.h" -#include "forwarder_types.h" -#include "http/http2_session.h" -#include "http/http_responder.h" -#include "http/http_session.h" -#include "node/rpc/custom_protocol_subsystem.h" -#include "node/session_metrics.h" -#include "rpc_handler.h" -#include "tls/cert.h" -#include "tls/context.h" -#include "tls/plaintext_server.h" -#include "tls/server.h" -#include "udp/msg_types.h" - -// NB: This should be HTTP3 including QUIC, but this is -// ok for now, as we only have an echo service for now -#include "quic/quic_session.h" - -#include -#include -#include -#include - -namespace ccf -{ - using QUICSessionImpl = quic::QUICEchoSession; - - static constexpr size_t max_open_sessions_soft_default = 1000; - static constexpr size_t max_open_sessions_hard_default = 1010; - static const ccf::Endorsement endorsement_default = {ccf::Authority::SERVICE}; - - class RPCSessions : public std::enable_shared_from_this, - public AbstractRPCResponder, - public ::http::ErrorReporter - { - private: - struct ListenInterface - { - size_t open_sessions = 0; - size_t peak_sessions = 0; - size_t max_open_sessions_soft = 0; - size_t max_open_sessions_hard = 0; - ccf::Endorsement endorsement{}; - http::ParserConfiguration http_configuration; - ccf::SessionMetrics::Errors errors{}; - ccf::ApplicationProtocol app_protocol; - }; - std::map listening_interfaces; - - ringbuffer::AbstractWriterFactory& writer_factory; - ringbuffer::WriterPtr to_host = nullptr; - std::shared_ptr rpc_map; - std::unordered_map> certs; - std::shared_ptr custom_protocol_subsystem = - nullptr; - std::shared_ptr commit_callbacks_subsystem = - nullptr; - - ccf::pal::Mutex lock; - std::unordered_map< - ccf::tls::ConnID, - std::pair>> - sessions; - size_t sessions_peak = 0; - - template - class NoMoreSessionsImpl : public Base - { - public: - template - NoMoreSessionsImpl(Ts&&... ts) : Base(std::forward(ts)...) - {} - - void handle_incoming_data_thread(std::vector&& data) override - { - Base::tls_io->recv_buffered(data.data(), data.size()); - - if (Base::tls_io->get_status() == ccf::SessionStatus::ready) - { - // Send response describing soft session limit - Base::send_odata_error_response(ccf::ErrorDetails{ - HTTP_STATUS_SERVICE_UNAVAILABLE, - ccf::errors::SessionCapExhausted, - "Service is currently busy and unable to serve new connections"}); - - // Close connection - Base::tls_io->close(); - } - } - }; - - ListenInterface& get_interface_from_interface_id( - const ccf::ListenInterfaceID& id) - { - auto it = listening_interfaces.find(id); - if (it != listening_interfaces.end()) - { - return it->second; - } - - throw std::logic_error( - fmt::format("No RPC interface for interface ID {}", id)); - } - - std::shared_ptr make_server_session( - const std::string& app_protocol, - ccf::tls::ConnID id, - const ListenInterfaceID& listen_interface_id, - std::unique_ptr&& ctx, - const http::ParserConfiguration& parser_configuration) - { - if (app_protocol == "HTTP2") - { - return std::make_shared<::http::HTTP2ServerSession>( - rpc_map, - id, - listen_interface_id, - writer_factory, - std::move(ctx), - parser_configuration, - shared_from_this()); - } - if (app_protocol == "HTTP1") - { - return std::make_shared<::http::HTTPServerSession>( - rpc_map, - id, - listen_interface_id, - writer_factory, - std::move(ctx), - parser_configuration, - shared_from_this(), - commit_callbacks_subsystem); - } - if (custom_protocol_subsystem) - { - return custom_protocol_subsystem->create_session( - app_protocol, id, std::move(ctx)); - } - - throw std::runtime_error(fmt::format( - "unknown protocol '{}' and custom protocol subsystem missing", - app_protocol)); - } - - public: - RPCSessions( - ringbuffer::AbstractWriterFactory& writer_factory, - std::shared_ptr rpc_map_) : - writer_factory(writer_factory), - rpc_map(std::move(rpc_map_)) - { - to_host = writer_factory.create_writer_to_outside(); - } - - void set_custom_protocol_subsystem( - std::shared_ptr cpss) - { - custom_protocol_subsystem = cpss; - } - - void set_commit_callbacks_subsystem( - std::shared_ptr fcss) - { - commit_callbacks_subsystem = fcss; - } - - void report_parsing_error(const ccf::ListenInterfaceID& id) override - { - std::lock_guard guard(lock); - get_interface_from_interface_id(id).errors.parsing++; - } - - void report_request_payload_too_large_error( - const ccf::ListenInterfaceID& id) override - { - std::lock_guard guard(lock); - get_interface_from_interface_id(id).errors.request_payload_too_large++; - } - - void report_request_header_too_large_error( - const ccf::ListenInterfaceID& id) override - { - std::lock_guard guard(lock); - get_interface_from_interface_id(id).errors.request_header_too_large++; - } - - void update_listening_interface_options( - const ccf::NodeInfoNetwork& node_info) - { - std::lock_guard guard(lock); - - for (const auto& [name, interface] : node_info.rpc_interfaces) - { - auto& li = listening_interfaces[name]; - - li.max_open_sessions_soft = interface.max_open_sessions_soft.value_or( - max_open_sessions_soft_default); - - li.max_open_sessions_hard = interface.max_open_sessions_hard.value_or( - max_open_sessions_hard_default); - - li.endorsement = interface.endorsement.value_or(endorsement_default); - - li.http_configuration = - interface.http_configuration.value_or(http::ParserConfiguration{}); - - li.app_protocol = interface.app_protocol.value_or("HTTP1"); - - LOG_INFO_FMT( - "Setting max open sessions on interface \"{}\" ({}) to [{}, " - "{}] and endorsement authority to {}", - name, - interface.bind_address, - li.max_open_sessions_soft, - li.max_open_sessions_hard, - li.endorsement.authority); - } - } - - ccf::SessionMetrics get_session_metrics() - { - ccf::SessionMetrics sm; - std::lock_guard guard(lock); - - sm.active = sessions.size(); - sm.peak = sessions_peak; - - for (const auto& [name, interface] : listening_interfaces) - { - sm.interfaces[name] = { - interface.open_sessions, - interface.peak_sessions, - interface.max_open_sessions_soft, - interface.max_open_sessions_hard, - interface.errors}; - } - - return sm; - } - - ccf::ApplicationProtocol get_app_protocol_main_interface() const - { - // Note: this is a temporary function to conveniently find out which - // protocol to use when creating client endpoints (e.g. for join - // protocol). This can be removed once the HTTP and HTTP/2 endpoints have - // been merged. - if (listening_interfaces.empty()) - { - throw std::logic_error("No listening interface for this node"); - } - - return listening_interfaces.begin()->second.app_protocol; - } - - void set_node_cert( - const ccf::crypto::Pem& cert_, const ccf::crypto::Pem& pk) - { - set_cert(ccf::Authority::NODE, cert_, pk); - } - - void set_network_cert( - const ccf::crypto::Pem& cert_, const ccf::crypto::Pem& pk) - { - set_cert(ccf::Authority::SERVICE, cert_, pk); - } - - void set_cert( - ccf::Authority authority, - const ccf::crypto::Pem& cert_, - const ccf::crypto::Pem& pk) - { - // Caller authentication is done by each frontend by looking up - // the caller's certificate in the relevant store table. The caller - // certificate does not have to be signed by a known CA (nullptr) and - // verification is not required here. - auto cert = std::make_shared<::tls::Cert>( - nullptr, cert_, pk, std::nullopt, /*auth_required ==*/false); - - std::lock_guard guard(lock); - - for (auto& [listen_interface_id, interface] : listening_interfaces) - { - if (interface.endorsement.authority == authority) - { - certs.insert_or_assign(listen_interface_id, cert); - } - } - } - - void accept( - ccf::tls::ConnID id, - const ListenInterfaceID& listen_interface_id, - bool udp = false) - { - std::lock_guard guard(lock); - - if (sessions.find(id) != sessions.end()) - { - throw std::logic_error( - fmt::format("Duplicate conn ID received inside enclave: {}", id)); - } - - auto it = listening_interfaces.find(listen_interface_id); - if (it == listening_interfaces.end()) - { - throw std::logic_error(fmt::format( - "Can't accept new RPC session {} - comes from unknown listening " - "interface {}", - id, - listen_interface_id)); - } - - auto& per_listen_interface = it->second; - - if ( - per_listen_interface.endorsement.authority != Authority::UNSECURED && - certs.find(listen_interface_id) == certs.end()) - { - LOG_DEBUG_FMT( - "Refusing TLS session {} inside the enclave - interface {} " - "has no TLS certificate yet", - id, - listen_interface_id); - - RINGBUFFER_WRITE_MESSAGE( - ::tcp::tcp_stop, to_host, id, std::string("Session refused")); - } - else if ( - per_listen_interface.open_sessions >= - per_listen_interface.max_open_sessions_hard) - { - LOG_INFO_FMT( - "Refusing TLS session {} inside the enclave - already have {} " - "sessions from interface {} and limit is {}", - id, - per_listen_interface.open_sessions, - listen_interface_id, - per_listen_interface.max_open_sessions_hard); - - RINGBUFFER_WRITE_MESSAGE( - ::tcp::tcp_stop, to_host, id, std::string("Session refused")); - } - else if ( - per_listen_interface.open_sessions >= - per_listen_interface.max_open_sessions_soft) - { - LOG_INFO_FMT( - "Soft refusing session {} (returning 503) inside the enclave - " - "already have {} sessions from interface {} and limit is {}", - id, - per_listen_interface.open_sessions, - listen_interface_id, - per_listen_interface.max_open_sessions_soft); - - auto ctx = std::make_unique<::tls::Server>(certs[listen_interface_id]); - std::shared_ptr capped_session; - if (per_listen_interface.app_protocol == "HTTP2") - { - capped_session = - std::make_shared>( - rpc_map, - id, - listen_interface_id, - writer_factory, - std::move(ctx), - per_listen_interface.http_configuration, - shared_from_this()); - } - else - { - capped_session = - std::make_shared>( - rpc_map, - id, - listen_interface_id, - writer_factory, - std::move(ctx), - per_listen_interface.http_configuration, - shared_from_this(), - commit_callbacks_subsystem); - } - sessions.insert(std::make_pair( - id, std::make_pair(listen_interface_id, std::move(capped_session)))); - per_listen_interface.open_sessions++; - per_listen_interface.peak_sessions = std::max( - per_listen_interface.peak_sessions, - per_listen_interface.open_sessions); - } - else - { - LOG_DEBUG_FMT( - "Accepting a session {} inside the enclave from interface \"{}\"", - id, - listen_interface_id); - - if (udp) - { - LOG_DEBUG_FMT("New UDP endpoint at {}", id); - if (per_listen_interface.app_protocol == "QUIC") - { - auto session = std::make_shared( - rpc_map, id, listen_interface_id, writer_factory); - sessions.insert(std::make_pair( - id, std::make_pair(listen_interface_id, std::move(session)))); - } - else if (custom_protocol_subsystem) - { - // We know it's a custom protocol, but the session creation function - // hasn't been registered yet, so we keep a nullptr until the first - // udp::udp_inbound message. - sessions.insert( - std::make_pair(id, std::make_pair(listen_interface_id, nullptr))); - } - else - { - throw std::runtime_error( - "unknown UDP protocol and custom protocol subsystem missing"); - } - per_listen_interface.open_sessions++; - per_listen_interface.peak_sessions = std::max( - per_listen_interface.peak_sessions, - per_listen_interface.open_sessions); - } - else - { - std::unique_ptr ctx; - if ( - per_listen_interface.endorsement.authority == Authority::UNSECURED) - { - ctx = std::make_unique(); - } - else - { - ctx = std::make_unique<::tls::Server>( - certs[listen_interface_id], - per_listen_interface.app_protocol == "HTTP2"); - } - - auto session = make_server_session( - per_listen_interface.app_protocol, - id, - listen_interface_id, - std::move(ctx), - per_listen_interface.http_configuration); - - sessions.insert(std::make_pair( - id, std::make_pair(listen_interface_id, std::move(session)))); - per_listen_interface.open_sessions++; - per_listen_interface.peak_sessions = std::max( - per_listen_interface.peak_sessions, - per_listen_interface.open_sessions); - } - } - - sessions_peak = std::max(sessions_peak, sessions.size()); - } - - std::shared_ptr find_session(ccf::tls::ConnID id) - { - std::lock_guard guard(lock); - - auto search = sessions.find(id); - if (search == sessions.end()) - { - return nullptr; - } - - return search->second.second; - } - - bool reply_async( - ccf::tls::ConnID id, - bool terminate_after_send, - std::vector&& data) override - { - auto session = find_session(id); - if (session == nullptr) - { - LOG_DEBUG_FMT("Refusing to reply to unknown session {}", id); - return false; - } - - LOG_DEBUG_FMT("Replying to session {}", id); - - session->send_data(std::move(data)); - - if (terminate_after_send) - { - session->close_session(); - } - - return true; - } - - void remove_session(ccf::tls::ConnID id) - { - std::lock_guard guard(lock); - LOG_DEBUG_FMT("Closing a session inside the enclave: {}", id); - const auto search = sessions.find(id); - if (search != sessions.end()) - { - auto it = listening_interfaces.find(search->second.first); - if (it != listening_interfaces.end()) - { - it->second.open_sessions--; - } - sessions.erase(search); - } - else - { - // Enclave doesn't know this ID, but host is still talking about it. - // Continue with the normal closure flow - RINGBUFFER_WRITE_MESSAGE(::tcp::tcp_closed, to_host, id); - } - } - - void register_message_handlers( - messaging::Dispatcher& disp) - { - DISPATCHER_SET_MESSAGE_HANDLER( - disp, ::tcp::tcp_start, [this](const uint8_t* data, size_t size) { - auto [new_tls_id, listen_interface_name] = - ringbuffer::read_message<::tcp::tcp_start>(data, size); - accept(new_tls_id, listen_interface_name); - }); - - DISPATCHER_SET_MESSAGE_HANDLER( - disp, ::tcp::tcp_inbound, [this](const uint8_t* data, size_t size) { - auto id = serialized::peek(data, size); - - auto session = find_session(id); - if (session == nullptr) - { - LOG_DEBUG_FMT( - "Ignoring tls_inbound for unknown or refused session: {}", id); - return; - } - - session->handle_incoming_data({data, size}); - }); - - DISPATCHER_SET_MESSAGE_HANDLER( - disp, ::tcp::tcp_close, [this](const uint8_t* data, size_t size) { - auto [id] = ringbuffer::read_message<::tcp::tcp_close>(data, size); - remove_session(id); - }); - - DISPATCHER_SET_MESSAGE_HANDLER( - disp, udp::udp_start, [this](const uint8_t* data, size_t size) { - auto [new_id, listen_interface_name] = - ringbuffer::read_message(data, size); - accept(new_id, listen_interface_name, true); - }); - - DISPATCHER_SET_MESSAGE_HANDLER( - disp, udp::udp_inbound, [this](const uint8_t* data, size_t size) { - auto id = serialized::peek(data, size); - - std::shared_ptr session; - { - std::lock_guard guard(lock); - - auto search = sessions.find(id); - if (search == sessions.end()) - { - LOG_DEBUG_FMT( - "Ignoring udp::udp_inbound for unknown or refused session: {}", - id); - return; - } - - if (!search->second.second && custom_protocol_subsystem) - { - LOG_DEBUG_FMT("Creating custom UDP session {}", id); - - try - { - const auto& conn_id = search->first; - const auto& interface_id = search->second.first; - - auto iit = listening_interfaces.find(interface_id); - if (iit == listening_interfaces.end()) - { - LOG_DEBUG_FMT( - "Failure to create custom protocol session because of " - "unknown interface '{}', ignoring udp::udp_inbound for " - "session: " - "{}", - interface_id, - id); - } - - const auto& interface = iit->second; - - search->second.second = - custom_protocol_subsystem->create_session( - interface.app_protocol, conn_id, nullptr); - - if (!search->second.second) - { - LOG_DEBUG_FMT( - "Failure to create custom protocol session, ignoring " - "udp::udp_inbound for session: {}", - id); - return; - } - } - catch (const std::exception& ex) - { - LOG_DEBUG_FMT( - "Failure to create custom protocol session: {}", ex.what()); - return; - } - } - - session = search->second.second; - } - - session->handle_incoming_data({data, size}); - }); - } - }; -} diff --git a/src/enclave/session.h b/src/enclave/session.h index bf8a031eccb1..70fc4f9ef22e 100644 --- a/src/enclave/session.h +++ b/src/enclave/session.h @@ -3,13 +3,13 @@ #pragma once #include "ccf/node/session.h" -#include "enclave/tls_session.h" #include "tasks/ordered_tasks.h" #include "tasks/task.h" #include "tasks/task_system.h" #include "tcp/msg_types.h" #include +#include namespace ccf { @@ -25,12 +25,13 @@ namespace ccf std::vector data; std::shared_ptr self; + // The caller has already built a buffer for us, so take it rather than + // copying a request or response which may be arbitrarily large. SessionDataTask( - std::span d, std::shared_ptr s) : + std::vector&& d, std::shared_ptr s) : + data(std::move(d)), self(std::move(s)) - { - data.assign(d.begin(), d.end()); - } + {} }; struct HandleIncomingDataTask : public SessionDataTask @@ -39,6 +40,19 @@ namespace ccf void do_action() override { + // The transport is holding this data against a node-wide budget until + // we say we are done with it, so report on every exit path - including + // the early return below, and an exception out of the parser. + struct ReportConsumed + { + ThreadedSession& session; + size_t bytes; + ~ReportConsumed() + { + session.on_inbound_consumed(bytes); + } + } report{*self, data.size()}; + if (self->is_closing.load()) { return; @@ -86,16 +100,20 @@ namespace ccf // Implement Session::handle_incoming_data by dispatching a thread message // that eventually invokes the virtual handle_incoming_data_thread() - void handle_incoming_data(std::span data) override + void handle_incoming_data(std::vector&& data) override { - auto [_, body] = ringbuffer::read_message<::tcp::tcp_inbound>(data); - - task_scheduler->add_action( - std::make_shared(body, shared_from_this())); + task_scheduler->add_action(std::make_shared( + std::move(data), shared_from_this())); } virtual void handle_incoming_data_thread(std::vector&& data) = 0; + // Called once the data from a single handle_incoming_data() has been + // processed, so that the transport can resume reading. Sessions which have + // no transport to report to leave this as a no-op; the transport then + // reclaims their share when the connection closes. + virtual void on_inbound_consumed(size_t /*bytes*/) {} + // Implement Session::sent_data by dispatching a thread message // that eventually invokes the virtual send_data_thread() void send_data(std::vector&& data) override @@ -117,110 +135,82 @@ namespace ccf virtual void close_session_thread() = 0; }; - class EncryptedSession : public ThreadedSession + // A protocol session (HTTP/HTTP2/...) running over a transport that owns the + // TLS connection (the host-side OpenSSL connection). It receives and emits + // plaintext: inbound bytes are already decrypted, and outbound bytes are + // handed to a SessionWriter which encrypts and writes them. The peer + // certificate and SNI (captured by the transport at handshake) are provided + // for caller authentication. + class PlaintextSession : public ThreadedSession { public: virtual bool parse(std::span data) = 0; protected: - std::shared_ptr tls_io; ::tcp::ConnID session_id; - - EncryptedSession( + // Not owned. The writer is the transport's per-interface bridge, which is + // owned by the RPCConnectionManager and outlives every session on that + // interface: the manager is only destroyed if node creation fails, before + // any session exists, and is otherwise never destroyed. That matters + // because a queued task holds a shared_ptr to its session, so a session + // can outlive its removal from the transport's connection map. If the + // manager ever becomes destructible on the normal path, this must become a + // weak_ptr. + ccf::SessionWriter& session_writer; + std::vector peer_cert_; + // Set once parse() has reported that it will process no more data (a parse + // error, an oversized request, a protocol-level shutdown). The protocol + // session has already emitted its error response and closed the session by + // that point, but the transport may still deliver bytes which were already + // in flight. Feeding those to a parser which has errored produces + // duplicate responses on a socket which is closing, so drop them instead. + // + // Only touched from handle_incoming_data_thread(), which runs on this + // session's own OrderedTasks and is therefore serialised. + bool parsing_finished = false; + + PlaintextSession( ::tcp::ConnID session_id_, - ringbuffer::AbstractWriterFactory& writer_factory, - std::unique_ptr ctx) : + ccf::SessionWriter& writer, + std::vector peer_cert = {}) : ThreadedSession(session_id_), - tls_io(std::make_shared( - session_id_, writer_factory, std::move(ctx))), - session_id(session_id_) + session_id(session_id_), + session_writer(writer), + peer_cert_(std::move(peer_cert)) {} public: + const std::vector& peer_cert() const + { + return peer_cert_; + } + void send_data_thread(std::vector&& data) override { - tls_io->send_data(data.data(), data.size()); + session_writer.write_outbound(session_id, std::move(data)); } void handle_incoming_data_thread(std::vector&& data) override { - tls_io->recv_buffered(data.data(), data.size()); - - LOG_TRACE_FMT("recv called with {} bytes", data.size()); - - // Try to parse all incoming data, reusing the vector we were just passed - // for storage. Increase the size if the received vector was too small - // (for the case where this chunk is very small, but we had some previous - // data to continue reading). - constexpr auto min_read_block_size = 4096; - if (data.size() < min_read_block_size) + if (parsing_finished) { - data.resize(min_read_block_size); + return; } - auto n_read = tls_io->read(data.data(), data.size(), false); - - while (true) + if (!parse({data.data(), data.size()})) { - if (n_read == 0) - { - return; - } - - LOG_TRACE_FMT("Going to parse {} bytes", n_read); - - bool cont = parse({data.data(), n_read}); - if (!cont) - { - return; - } - - // Used all provided bytes - check if more are available - n_read = tls_io->read(data.data(), data.size(), false); + parsing_finished = true; } } - void close_session_thread() override - { - tls_io->close(); - } - }; - - class UnencryptedSession : public ccf::ThreadedSession - { - public: - virtual bool parse(std::span data) = 0; - - protected: - ::tcp::ConnID session_id; - ringbuffer::WriterPtr to_host; - - UnencryptedSession( - ::tcp::ConnID session_id_, - ringbuffer::AbstractWriterFactory& writer_factory_) : - ccf::ThreadedSession(session_id_), - session_id(session_id_), - to_host(writer_factory_.create_writer_to_outside()) - {} - - void send_data_thread(std::vector&& data) override + void on_inbound_consumed(size_t bytes) override { - RINGBUFFER_WRITE_MESSAGE( - ::tcp::tcp_outbound, - to_host, - session_id, - serializer::ByteRange{data.data(), data.size()}); + session_writer.inbound_consumed(session_id, bytes); } void close_session_thread() override { - RINGBUFFER_WRITE_MESSAGE( - ::tcp::tcp_stop, to_host, session_id, std::string("Session closed")); - } - - void handle_incoming_data_thread(std::vector&& data) override - { - parse(data); + session_writer.close_socket(session_id); } }; } diff --git a/src/enclave/tls_session.h b/src/enclave/tls_session.h deleted file mode 100644 index ee5ca7121c61..000000000000 --- a/src/enclave/tls_session.h +++ /dev/null @@ -1,671 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the Apache 2.0 License. -#pragma once - -#include "ds/internal_logger.h" -#include "ds/messaging.h" -#include "ds/ring_buffer.h" -#include "tcp/msg_types.h" -#include "tls/context.h" -#include "tls/tls.h" - -#include - -namespace ccf -{ - enum SessionStatus : uint8_t - { - handshake, - ready, - closing, - closed, - authfail, - error - }; - - class TLSSession : public std::enable_shared_from_this - { - public: - using HandshakeErrorCB = std::function; - - protected: - ringbuffer::WriterPtr to_host; - ::tcp::ConnID session_id; - - private: - std::vector pending_write; - std::vector pending_read; - // Decrypted data - std::vector read_buffer; - - std::unique_ptr ctx; - SessionStatus status = handshake; - - HandshakeErrorCB handshake_error_cb; - - bool can_send() - { - // Closing endpoint should still be able to respond to clients (e.g. to - // report errors) - return status == ready || status == closing; - } - - bool can_recv() - { - return status == ready || status == handshake; - } - - public: - TLSSession( - int64_t session_id_, - ringbuffer::AbstractWriterFactory& writer_factory_, - std::unique_ptr ctx_) : - to_host(writer_factory_.create_writer_to_outside()), - session_id(session_id_), - ctx(std::move(ctx_)) - { - ctx->set_bio(this, send_callback_openssl, recv_callback_openssl); - } - - virtual ~TLSSession() - { - RINGBUFFER_WRITE_MESSAGE(::tcp::tcp_closed, to_host, session_id); - } - - SessionStatus get_status() const - { - return status; - } - - void on_handshake_error(std::string&& error_msg) - { - if (handshake_error_cb) - { - handshake_error_cb(std::move(error_msg)); - } - else - { - LOG_TRACE_FMT("{}", error_msg); - } - } - - void set_handshake_error_cb(HandshakeErrorCB&& cb) - { - handshake_error_cb = std::move(cb); - } - - std::string hostname() - { - if (status != ready) - { - return {}; - } - - return ctx->host(); - } - - std::vector peer_cert() - { - return ctx->peer_cert(); - } - - // Returns count N of bytes read, which will be the first N bytes of data, - // up to a maximum of size. If exact is true, will only return either size - // or 0 (when size bytes are not currently available). data may be accessed - // beyond N during operation, up to size, but only the first N should be - // used by caller. - size_t read(uint8_t* data, size_t size, bool exact = false) - { - // This will return empty if the connection isn't - // ready, but it will not block on the handshake. - do_handshake(); - - if (status != ready) - { - LOG_TRACE_FMT("Not ready to read {} bytes", size); - return 0; - } - - LOG_TRACE_FMT("Requesting up to {} bytes", size); - - // Send pending writes. - flush(); - - size_t offset = 0; - - if (!read_buffer.empty()) - { - LOG_TRACE_FMT( - "Have existing read_buffer of size: {}", read_buffer.size()); - offset = std::min(size, read_buffer.size()); - ::memcpy(data, read_buffer.data(), offset); - - if (offset < read_buffer.size()) - { - read_buffer.erase(read_buffer.begin(), read_buffer.begin() + offset); - } - else - { - read_buffer.clear(); - } - - if (offset == size) - { - return size; - } - - // NB: If we continue past here, read_buffer is empty - } - - auto r = ctx->read(data + offset, size - offset); - LOG_TRACE_FMT("ctx->read returned: {}", r); - - switch (r) - { - case 0: - case TLS_ERR_CONN_CLOSE_NOTIFY: - { - LOG_TRACE_FMT( - "TLS {} close on read: {}", session_id, ::tls::error_string(r)); - - stop(closed); - - if (!exact) - { - // Hit an error, but may still have some useful data from the - // previous read_buffer - return offset; - } - - return 0; - } - - case TLS_ERR_WANT_READ: - case TLS_ERR_WANT_WRITE: - { - if (!exact) - { - return offset; - } - - // May have read something but not enough - copy it into read_buffer - // for next call - read_buffer.insert(read_buffer.end(), data, data + offset); - return 0; - } - - default: - { - } - } - - if (r < 0) - { - LOG_TRACE_FMT( - "TLS {} error on read: {}", session_id, ::tls::error_string(r)); - stop(error); - return 0; - } - - auto total = r + offset; - - // We read _some_ data but not enough, and didn't get - // TLS_ERR_WANT_READ. Probably hit an internal size limit - try - // again - if (exact && (total < size)) - { - LOG_TRACE_FMT( - "Asked for exactly {}, received {}, retrying", size, total); - read_buffer.insert(read_buffer.end(), data, data + total); - return read(data, size, exact); - } - - return total; - } - - void recv_buffered(const uint8_t* data, size_t size) - { - if (can_recv()) - { - pending_read.insert(pending_read.end(), data, data + size); - } - - do_handshake(); - } - - void close() - { - status = closing; - - switch (status) - { - case handshake: - { - LOG_TRACE_FMT("TLS {} closed during handshake", session_id); - stop(closed); - break; - } - - case ready: - case closing: - { - int r = ctx->close(); - - switch (r) - { - case TLS_ERR_WANT_READ: - case TLS_ERR_WANT_WRITE: - { - LOG_TRACE_FMT("TLS {} has pending data ({})", session_id, r); - // FALLTHROUGH - } - case 0: - { - LOG_TRACE_FMT("TLS {} closed ({})", session_id, r); - stop(closed); - break; - } - - default: - { - LOG_TRACE_FMT( - "TLS {} error on_close: {}", - session_id, - ::tls::error_string(r)); - stop(error); - break; - } - } - break; - } - - case closed: - case authfail: - case error: - { - break; - } - } - } - - void send_data(const uint8_t* data, size_t size) - { - // Writes as much of the data as possible. If the data cannot all - // be written now, we store the remainder. We - // will try to send pending writes again whenever write() is called. - do_handshake(); - - if (status == handshake) - { - pending_write.insert(pending_write.end(), data, data + size); - return; - } - - if (!can_send()) - { - return; - } - - pending_write.insert(pending_write.end(), data, data + size); - - flush(); - } - - private: - void send_buffered(const std::vector& data) - { - pending_write.insert(pending_write.end(), data.begin(), data.end()); - } - - void flush() - { - do_handshake(); - - if (!can_send()) - { - return; - } - - while (!pending_write.empty()) - { - auto r = write_some(pending_write); - - if (r > 0) - { - pending_write.erase(pending_write.begin(), pending_write.begin() + r); - } - else if (r == 0) - { - break; - } - else - { - LOG_TRACE_FMT("TLS session {} error on flush: {}", session_id, -r); - stop(error); - break; - } - } - } - - void do_handshake() - { - // This should be called when additional data is written to the - // input buffer, until the handshake is complete. - if (status != handshake) - { - return; - } - - auto rc = ctx->handshake(); - - switch (rc) - { - case 0: - { - status = ready; - break; - } - - case TLS_ERR_WANT_READ: - case TLS_ERR_WANT_WRITE: - break; - - case TLS_ERR_NEED_CERT: - { - on_handshake_error(fmt::format( - "TLS {} verify error on handshake: {}", - session_id, - ::tls::error_string(rc))); - stop(authfail); - break; - } - - case TLS_ERR_CONN_CLOSE_NOTIFY: - { - LOG_TRACE_FMT( - "TLS {} closed on handshake: {}", - session_id, - ::tls::error_string(rc)); - stop(closed); - break; - } - - case TLS_ERR_X509_VERIFY: - { - auto err = ctx->get_verify_error(); - on_handshake_error(fmt::format( - "TLS {} invalid cert on handshake: {} [{}]", - session_id, - err, - ::tls::error_string(rc))); - stop(authfail); - return; - } - - default: - { - on_handshake_error(fmt::format( - "TLS {} error on handshake: {}", - session_id, - ::tls::error_string(rc))); - stop(error); - break; - } - } - } - - int write_some(const std::vector& data) - { - auto r = ctx->write(data.data(), data.size()); - - switch (r) - { - case TLS_ERR_WANT_READ: - case TLS_ERR_WANT_WRITE: - return 0; - - default: - return r; - } - } - - void stop(SessionStatus status_) - { - switch (status) - { - case closed: - case authfail: - case error: - return; - - case handshake: - case ready: - case closing: - { - break; - } - } - - status = status_; - - switch (status) - { - case handshake: - case ready: - { - break; - } - case closing: - case closed: - { - RINGBUFFER_WRITE_MESSAGE( - ::tcp::tcp_stop, - to_host, - session_id, - std::string("Session closed")); - break; - } - - case authfail: - { - RINGBUFFER_WRITE_MESSAGE( - ::tcp::tcp_stop, - to_host, - session_id, - std::string("Authentication failed")); - } - case error: - { - RINGBUFFER_WRITE_MESSAGE( - ::tcp::tcp_stop, to_host, session_id, std::string("Error")); - break; - } - - default: - throw std::logic_error( - fmt::format("TLS {} unknown status: {}", session_id, status)); - } - } - - int handle_send(const uint8_t* buf, size_t len) - { - // Either write all of the data or none of it. - auto wrote = RINGBUFFER_TRY_WRITE_MESSAGE( - ::tcp::tcp_outbound, - to_host, - session_id, - serializer::ByteRange{buf, len}); - - if (!wrote) - { - return TLS_WRITING; - } - - return static_cast(len); - } - - int handle_recv(uint8_t* buf, size_t len) - { - if (!pending_read.empty()) - { - // Use the pending data vector. This is populated when the host - // writes a chunk larger than the size requested by the enclave. - size_t rd = std::min(len, pending_read.size()); - ::memcpy(buf, pending_read.data(), rd); - - if (rd >= pending_read.size()) - { - pending_read.clear(); - } - else - { - pending_read.erase(pending_read.begin(), pending_read.begin() + rd); - } - - return (int)rd; - } - - return TLS_READING; - } - - static int send_callback(void* ctx, const unsigned char* buf, size_t len) - { - return reinterpret_cast(ctx)->handle_send(buf, len); - } - - static int recv_callback(void* ctx, unsigned char* buf, size_t len) - { - return reinterpret_cast(ctx)->handle_recv(buf, len); - } - - // These callbacks below are complex, using the callbacks above and - // manipulating OpenSSL's BIO objects accordingly. This is just so we can - // emulate what MbedTLS used to do. - // Now that we have removed it from the code, we can move the callbacks - // above to handle BIOs directly and hopefully remove the complexity below. - // This work will be carried out in #3429. - static long send_callback_openssl( - BIO* b, - int oper, - const char* argp, - size_t len, - int argi, - long argl, - int ret, - size_t* processed) - { - // Unused arguments - (void)argi; - (void)argl; - (void)argp; - - if (ret != 0 && len > 0 && oper == (BIO_CB_WRITE | BIO_CB_RETURN)) - { - // Flush BIO so the "pipe doesn't clog", but we don't use the - // data here, because 'argp' already has it. - BIO_flush(b); - size_t pending = BIO_pending(b); - if (pending != 0) - { - BIO_reset(b); - } - - // Pipe object - void* ctx = BIO_get_callback_arg(b); - int put = - send_callback(ctx, reinterpret_cast(argp), len); - - // WANTS_WRITE - if (put == TLS_WRITING) - { - BIO_set_retry_write(b); - LOG_TRACE_FMT("TLS Session::send_cb() : WANTS_WRITE"); - *processed = 0; - return -1; - } - - LOG_TRACE_FMT("TLS Session::send_cb() : Put {} bytes", put); - - // Update the number of bytes to external users - *processed = put; - } - - // Unless we detected an error, the return value is always the same as the - // original operation. - return ret; - } - - static long recv_callback_openssl( - BIO* b, - int oper, - const char* argp, - size_t len, - int argi, - long argl, - int ret, - size_t* processed) - { - // Unused arguments - (void)argi; - (void)argl; - - if (ret == 1 && oper == (BIO_CB_CTRL | BIO_CB_RETURN)) - { - // This callback may be fired at the end of large batches of TLS frames - // on OpenSSL 3.x. Note that processed == nullptr in this case, hence - // the early exit. - return 0; - } - - if (ret != 0 && (oper == (BIO_CB_READ | BIO_CB_RETURN))) - { - // Pipe object - void* ctx = BIO_get_callback_arg(b); - // NOLINTNEXTLINE(cppcoreguidelines-pro-type-const-cast) - int got = recv_callback( - ctx, reinterpret_cast(const_cast(argp)), len); - - // WANTS_READ - if (got == TLS_READING) - { - BIO_set_retry_read(b); - LOG_TRACE_FMT("TLS Session::recv_cb() : WANTS_READ"); - *processed = 0; - return -1; - } - - LOG_TRACE_FMT("TLS Session::recv_cb() : Got {} bytes of {}", got, len); - - // If got less than requested, return WANT_READ - if ((size_t)got < len) - { - *processed = got; - return 1; - } - - // Write to the actual BIO so SSL can use it - BIO_write_ex(b, argp, got, processed); - - // The buffer should be enough, we can't return WANT_WRITE here - if ((size_t)got != *processed) - { - LOG_TRACE_FMT("TLS Session::recv_cb() : BIO error"); - *processed = got; - return -1; - } - - // If original return was -1 because it didn't find anything to read, - // return 1 to say we actually read something. This is common when the - // buffer is empty and needs an external read, so let's not log this. - if (got > 0 && ret < 0) - { - return 1; - } - } - - // Unless we detected an error, the return value is always the same as the - // original operation. - return ret; - } - }; -} diff --git a/src/host/configuration.h b/src/host/configuration.h index e19e5bcdfad8..5e702684948d 100644 --- a/src/host/configuration.h +++ b/src/host/configuration.h @@ -44,8 +44,6 @@ namespace host ccf::ds::TimeString slow_io_logging_threshold = {"10ms"}; std::optional node_client_interface = std::nullopt; ccf::ds::TimeString client_connection_timeout = {"2000ms"}; - std::optional idle_connection_timeout = - ccf::ds::TimeString("60s"); std::optional node_data_json_file = std::nullopt; std::optional service_data_json_file = std::nullopt; bool ignore_first_sigterm = false; @@ -187,7 +185,6 @@ namespace host slow_io_logging_threshold, node_client_interface, client_connection_timeout, - idle_connection_timeout, node_data_json_file, service_data_json_file, ignore_first_sigterm, diff --git a/src/host/datagram_server.h b/src/host/datagram_server.h new file mode 100644 index 000000000000..66d43c863dba --- /dev/null +++ b/src/host/datagram_server.h @@ -0,0 +1,394 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the Apache 2.0 License. +#pragma once + +// A minimal UDP datagram server: it owns a SOCK_DGRAM socket polled by the +// host libuv loop and delivers each received datagram to a handler. It backs +// UDP interfaces, leaving protocol behaviour to its handler. +// +// =========================================================================== +// QUIC EXTENSION POINT +// --------------------------------------------------------------------------- +// This is deliberately the substrate a future OpenSSL-native QUIC server would +// build on. The UDP socket created and bound here is exactly the datagram +// socket OpenSSL QUIC operates on. The pieces that change for QUIC are marked +// "QUIC EXTENSION POINT" inline; socket creation, binding and lifecycle remain. +// +// To become a QUIC server (needs OpenSSL >= 3.5, which adds SSL_new_listener / +// SSL_accept_connection / OSSL_QUIC_server_method): +// * wrap `sock` with BIO_new_dgram()/SSL_set_fd() on a QUIC listener SSL; +// * poll the descriptor returned by SSL_get_rpoll_descriptor() and use an +// SSL_get_event_timeout() timer; +// * on readability/timeout call SSL_handle_events(), then +// SSL_accept_connection()/SSL_accept_stream()/SSL_read_ex(), and reply with +// SSL_write_ex(). +// =========================================================================== + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace asynchost +{ + class DatagramServer + { + public: + using OnDatagram = std::function; + + // Whether anything is running the libuv loop when stop() is called. See + // stop() for why this has to be stated rather than inferred. + enum class LoopState : uint8_t + { + Running, + NotRunning, + }; + + private: + static constexpr size_t max_datagram = 65535; + // Datagrams handled per readable event. The handler runs inline on the + // loop thread, so an unbounded drain would let a UDP flood starve every + // other handle on the loop. The socket stays level-triggered, so any + // remainder is picked up on the next iteration. This replaces the read + // quota the previous ringbuffer-based UDP transport applied. + static constexpr size_t max_datagrams_per_event = 64; + + uv_loop_t* loop = nullptr; + int sock = -1; + // Heap-allocated and freed by their own close callbacks, so that this + // server can be destroyed without waiting for the loop to run them. See + // the equivalent note in host/tls/openssl_server.h. + uv_poll_t* socket_poll = nullptr; + uv_async_t* stop_handle = nullptr; + uint16_t bound_port = 0; + OnDatagram on_datagram; + + std::mutex lifecycle_mutex; + std::condition_variable teardown_cv; + bool started = false; + bool stopping = false; + bool torn_down = false; + + template + static void close_handle(THandle*& handle) + { + if (handle == nullptr) + { + return; + } + auto* as_handle = reinterpret_cast(handle); + handle = nullptr; + if (uv_is_closing(as_handle) != 0) + { + return; + } + uv_close(as_handle, [](uv_handle_t* h) { + // NOLINTNEXTLINE(cppcoreguidelines-owning-memory) + delete reinterpret_cast(h); + }); + } + + static bool set_nonblocking(int fd) + { + const int flags = fcntl(fd, F_GETFL, 0); + if (flags < 0) + { + return false; + } + return fcntl(fd, F_SETFL, flags | O_NONBLOCK) == 0; + } + + void drain() + { + size_t handled = 0; + while (handled < max_datagrams_per_event) + { + uint8_t buf[max_datagram]; + sockaddr_storage peer{}; + socklen_t peerlen = sizeof(peer); + const ssize_t n = ::recvfrom( + sock, + buf, + sizeof(buf), + 0, + reinterpret_cast(&peer), + &peerlen); + if (n < 0) + { + if (errno == EAGAIN || errno == EWOULDBLOCK) + { + break; + } + if (errno == EINTR) + { + // Interrupted before receiving anything, so this does not count + // against the quota. + continue; + } + break; + } + + ++handled; + if (on_datagram) + { + // === QUIC EXTENSION POINT === + // A QUIC server would feed these bytes to SSL_handle_events(). + on_datagram(buf, static_cast(n), peer, peerlen); + } + } + } + + static void on_socket_poll(uv_poll_t* handle, int status, int events) + { + auto* self = static_cast(handle->data); + if (status < 0) + { + self->tear_down_on_loop(); + return; + } + if ((events & UV_READABLE) != 0) + { + // === QUIC EXTENSION POINT === + // For QUIC this becomes SSL_handle_events() on the listener. + self->drain(); + } + } + + static void on_stop(uv_async_t* handle) + { + auto* self = static_cast(handle->data); + self->tear_down_on_loop(); + } + + // Close the socket and hand every handle to uv_close(). Must only run + // where it cannot race the loop - see stop(). Does not wait for the + // closes: the handles own themselves. + void tear_down_on_loop() + { + std::lock_guard guard(lifecycle_mutex); + if (torn_down) + { + return; + } + stopping = true; + + if (socket_poll != nullptr) + { + (void)uv_poll_stop(socket_poll); + } + // Closed under the lock so that a concurrent send_to() cannot be left + // holding a descriptor which has been closed (and possibly reused) + // underneath it. + if (sock >= 0) + { + ::close(sock); + sock = -1; + } + + close_handle(socket_poll); + close_handle(stop_handle); + + torn_down = true; + teardown_cv.notify_all(); + } + + public: + DatagramServer( + const std::string& host, + uint16_t port, + OnDatagram on_datagram_, + uv_loop_t* loop_ = uv_default_loop()) : + loop(loop_), + on_datagram(std::move(on_datagram_)) + { + addrinfo hints{}; + hints.ai_family = AF_UNSPEC; + hints.ai_socktype = SOCK_DGRAM; + hints.ai_flags = AI_PASSIVE; + addrinfo* res = nullptr; + const std::string port_str = std::to_string(port); + if (getaddrinfo(host.c_str(), port_str.c_str(), &hints, &res) != 0) + { + throw std::runtime_error("getaddrinfo (udp) failed for " + host); + } + + const int one = 1; + bool bound = false; + for (addrinfo* ai = res; ai != nullptr; ai = ai->ai_next) + { + sock = ::socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol); + if (sock < 0) + { + continue; + } + if (setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one)) != 0) + { + ::close(sock); + sock = -1; + continue; + } + if (::bind(sock, ai->ai_addr, ai->ai_addrlen) == 0) + { + bound = true; + break; + } + ::close(sock); + sock = -1; + } + freeaddrinfo(res); + if (!bound) + { + cleanup(); + throw std::runtime_error("bind (udp) failed for " + host); + } + if (!set_nonblocking(sock)) + { + cleanup(); + throw std::runtime_error("set_nonblocking (udp) failed"); + } + + sockaddr_storage bound_address{}; + socklen_t bound_address_len = sizeof(bound_address); + if ( + getsockname( + sock, + reinterpret_cast(&bound_address), + &bound_address_len) == 0) + { + bound_port = (bound_address.ss_family == AF_INET6) ? + ntohs(reinterpret_cast(&bound_address)->sin6_port) : + ntohs(reinterpret_cast(&bound_address)->sin_port); + } + } + + DatagramServer(const DatagramServer&) = delete; + DatagramServer& operator=(const DatagramServer&) = delete; + DatagramServer(DatagramServer&&) = delete; + DatagramServer& operator=(DatagramServer&&) = delete; + + ~DatagramServer() + { + stop(); + cleanup(); + } + + void start() + { + std::lock_guard guard(lifecycle_mutex); + if (started) + { + return; + } + + // Marked started before any handle is created, so that a failure part + // way through still leaves a server whose destructor closes the handles + // which were created (stop() is a no-op unless `started` is set). + started = true; + stopping = false; + torn_down = false; + + socket_poll = new uv_poll_t{}; // NOLINT(cppcoreguidelines-owning-memory) + socket_poll->data = this; + int rc = uv_poll_init_socket(loop, socket_poll, sock); + if (rc != 0) + { + throw std::runtime_error( + std::string("uv_poll_init_socket(udp) failed: ") + uv_strerror(rc)); + } + + stop_handle = new uv_async_t{}; // NOLINT(cppcoreguidelines-owning-memory) + stop_handle->data = this; + rc = uv_async_init(loop, stop_handle, on_stop); + if (rc != 0) + { + throw std::runtime_error( + std::string("uv_async_init(udp) failed: ") + uv_strerror(rc)); + } + + rc = uv_poll_start(socket_poll, UV_READABLE, on_socket_poll); + if (rc != 0) + { + throw std::runtime_error( + std::string("uv_poll_start(udp) failed: ") + uv_strerror(rc)); + } + } + + // Tear the server down. Idempotent, and safe to call from the destructor. + // See OpenSSLServer::stop() for why `loop_state` is stated rather than + // inferred. + void stop(LoopState loop_state = LoopState::NotRunning) + { + std::unique_lock lock(lifecycle_mutex); + if (!started || torn_down) + { + return; + } + + if (loop_state == LoopState::NotRunning) + { + lock.unlock(); + tear_down_on_loop(); + return; + } + + if (!stopping) + { + stopping = true; + if (stop_handle != nullptr) + { + (void)uv_async_send(stop_handle); + } + } + teardown_cv.wait(lock, [this]() { return torn_down; }); + } + + [[nodiscard]] uint16_t port() const + { + return bound_port; + } + + // Thread-safe: replies are sent from session workers, while the socket may + // be closed concurrently by shutdown on the loop thread. + [[nodiscard]] bool send_to( + const sockaddr_storage& peer, + socklen_t peerlen, + const uint8_t* data, + size_t len) + { + std::lock_guard guard(lifecycle_mutex); + if (sock < 0) + { + return false; + } + const auto rc = ::sendto( + sock, data, len, 0, reinterpret_cast(&peer), peerlen); + return rc >= 0; + } + + private: + void cleanup() + { + if (sock >= 0) + { + ::close(sock); + sock = -1; + } + } + }; +} \ No newline at end of file diff --git a/src/host/rpc_connection_manager.h b/src/host/rpc_connection_manager.h new file mode 100644 index 000000000000..fc6e37572245 --- /dev/null +++ b/src/host/rpc_connection_manager.h @@ -0,0 +1,1016 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the Apache 2.0 License. +#pragma once + +// Host-side, OpenSSL-native RPC connection manager. +// +// Owns one OpenSSL transport per listening interface (TLS terminated in the +// connection, see host/tls/openssl_server.h), creates the protocol session for +// each connection, and applies per-interface session caps and certificates. It +// implements ccf::AbstractRPCSessions so the node (NodeState/frontends) reaches +// it without depending on the transport backend. +// +// Inbound only: outbound requests (quote endorsements, JWT refresh, node join) +// are made with libcurl and do not come through here. +// +// Cert-deferred listening: interfaces bind at startup even before their +// certificate exists (a joining node receives the service cert later). A TLS +// interface with no cert yet refuses connections until set_cert() supplies one; +// UNSECURED interfaces listen in plaintext. + +#include "ccf/crypto/pem.h" +#include "ccf/service/node_info_network.h" +#include "ds/internal_logger.h" +#include "enclave/abstract_rpc_sessions.h" +#include "enclave/no_more_sessions.h" +#include "enclave/rpc_map.h" +#include "host/datagram_server.h" +#include "host/tls/openssl_session_manager.h" +#include "http/error_reporter.h" +#include "http/http2_session.h" +#include "http/http_session.h" +#include "node/rpc/custom_protocol_subsystem.h" +#include "node/session_metrics.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace ccf +{ + static constexpr size_t max_open_sessions_soft_default = 1000; + static constexpr size_t max_open_sessions_hard_default = 1010; + // Node-wide bound on inbound data which has been read off sockets but not + // yet processed. Sized well above any legitimate working set - clients are + // expected to pipeline deeply - so that it only engages under overload. + static constexpr size_t inbound_queue_limit = 16UL * 1024 * 1024; + static const ccf::Endorsement endorsement_default = {ccf::Authority::SERVICE}; + // How often idle UDP sessions are swept, mirroring the TCP idle sweep. + static constexpr auto udp_idle_sweep_interval = std::chrono::seconds(1); + + // Per-interface error counters, deliberately kept out of the manager. + // + // Sessions are handed an ErrorReporter and hold it for their whole lifetime. + // If that were the manager itself, the manager would own the interfaces, + // which own the transport, which owns the sessions, which own the manager - + // a cycle keeping the manager alive for as long as any session exists. + // Owning the counts separately breaks it, and they are all a session + // actually needs. + class InterfaceErrorCounts : public ::http::ErrorReporter + { + public: + struct Counts + { + size_t parsing = 0; + size_t payload_too_large = 0; + size_t header_too_large = 0; + }; + + Counts get(const ccf::ListenInterfaceID& id) + { + std::lock_guard guard(mutex); + auto it = counts.find(id); + return it == counts.end() ? Counts{} : it->second; + } + + void report_parsing_error(const ccf::ListenInterfaceID& id) override + { + std::lock_guard guard(mutex); + ++counts[id].parsing; + } + + void report_request_payload_too_large_error( + const ccf::ListenInterfaceID& id) override + { + std::lock_guard guard(mutex); + ++counts[id].payload_too_large; + } + + void report_request_header_too_large_error( + const ccf::ListenInterfaceID& id) override + { + std::lock_guard guard(mutex); + ++counts[id].header_too_large; + } + + private: + std::mutex mutex; + std::map counts; + }; + + class RPCConnectionManager : public ccf::AbstractRPCSessions + { + private: + struct ListenInterface + { + std::string name; + size_t max_open_sessions_soft = max_open_sessions_soft_default; + size_t max_open_sessions_hard = max_open_sessions_hard_default; + ccf::Endorsement endorsement = endorsement_default; + http::ParserConfiguration http_configuration; + ccf::ApplicationProtocol app_protocol = "HTTP1"; + + std::atomic open_sessions{0}; + std::atomic peak_sessions{0}; + + // The transport for this interface (created on listen()). Held by + // shared_ptr so that callers which snapshot it under interfaces_mutex + // and then use it without the lock (stop(), reply_async()) cannot race + // with it being replaced or destroyed. + std::shared_ptr bridge; + }; + + class DatagramSessionWriter : public ccf::SessionWriter + { + private: + std::function)> write; + std::function close; + + public: + DatagramSessionWriter( + std::function)> write_, + std::function close_) : + write(std::move(write_)), + close(std::move(close_)) + {} + + void write_outbound( + ::tcp::ConnID id, std::vector&& data) override + { + write(id, data); + } + + void close_socket(::tcp::ConnID id) override + { + close(id); + } + }; + + struct UdpSession + { + std::shared_ptr session; + ::tcp::ConnID conn_id = 0; + std::chrono::steady_clock::time_point last_active = + std::chrono::steady_clock::now(); + }; + + struct DatagramInterface + { + std::shared_ptr server; + std::unique_ptr writer; + std::map sessions_by_peer; + std::map<::tcp::ConnID, std::string> peer_by_id; + std::chrono::steady_clock::time_point last_idle_sweep = + std::chrono::steady_clock::now(); + }; + + std::shared_ptr rpc_map; + + // Subsystems are installed by NodeState during node creation, which now + // happens after the interfaces are already listening, so they can be + // assigned while sessions are being created on TLS worker threads. Guarded + // by their own mutex rather than interfaces_mutex, so that reading them on + // the session-creation path does not contend with the interface map. + std::mutex subsystems_mutex; + std::shared_ptr custom_protocol_subsystem; + std::shared_ptr commit_callbacks_subsystem; + + // Set at the start of stop(). Once set no further sessions are created: + // the transports are being torn down. + std::atomic stopping{false}; + + std::mutex interfaces_mutex; + std::map> interfaces; + + // The transport each admitted TCP connection belongs to, so that a reply + // can find its session directly. Guarded by its own mutex rather than + // interfaces_mutex: replies arrive on session workers, while the entries + // are added and removed on the libuv loop thread, and putting that on the + // interface lock would serialise every forwarded reply behind unrelated + // loop-thread work. + std::mutex connection_interfaces_mutex; + std::unordered_map< + ::tcp::ConnID, + std::shared_ptr> + connection_transports; + // UDP interface state, keyed by interface name. Only custom UDP protocols + // hold state here, one session per peer; "QUIC" interfaces are echoed + // statelessly (see listen_udp) and so have no entries at all. + std::map> udp_interfaces; + // cert/key PEM per endorsement authority (for cert-deferred listening). + std::map> certs; + + // Global connection-id source shared by all interface transports, so the + // session registry / reply routing have a single id space. + std::atomic<::tcp::ConnID> shared_conn_id{1}; + std::atomic active_sessions{0}; + std::atomic peak_sessions{0}; + + // How long an idle connection is kept before being closed (nullopt = + // never). Applied to each interface transport at listen() time. + std::optional idle_connection_timeout; + + // Shared by every interface transport, so the bound is on the node rather + // than on any one interface or connection. + std::shared_ptr inbound_admission = + std::make_shared(inbound_queue_limit); + + // Outlives this manager if a session does - see InterfaceErrorCounts. + std::shared_ptr error_counts = + std::make_shared(); + + std::shared_ptr<::http::ErrorReporter> error_reporter() + { + return error_counts; + } + + std::shared_ptr get_custom_protocol_subsystem() + { + std::lock_guard guard(subsystems_mutex); + return custom_protocol_subsystem; + } + + std::shared_ptr get_commit_callbacks_subsystem() + { + std::lock_guard guard(subsystems_mutex); + return commit_callbacks_subsystem; + } + + static std::string peer_key(const sockaddr_storage& peer, socklen_t peerlen) + { + return { + reinterpret_cast(&peer), + std::min(peerlen, sizeof(peer))}; + } + + // Ports reach here from operator configuration, so a malformed or + // out-of-range value must be reported rather than narrowed to whatever it + // happens to alias ("70000" would otherwise bind port 4464). An empty port + // requests an ephemeral one. + static uint16_t parse_port(const std::string& name, const std::string& port) + { + if (port.empty()) + { + return 0; + } + + uint16_t parsed = 0; + const auto* const end = port.data() + port.size(); + const auto [read_to, ec] = std::from_chars(port.data(), end, parsed); + if (ec != std::errc() || read_to != end) + { + throw std::logic_error(fmt::format( + "Invalid port '{}' for interface '{}' - expected a number in " + "[0, 65535]", + port, + name)); + } + return parsed; + } + + void increment_active_sessions() + { + const size_t now_active = ++active_sessions; + size_t prev_peak = peak_sessions.load(); + while (now_active > prev_peak && + !peak_sessions.compare_exchange_weak(prev_peak, now_active)) + { + } + } + + void decrement_active_sessions() + { + active_sessions.fetch_sub(1); + } + + void increment_interface_peak(ListenInterface* li, size_t now_open) + { + size_t prev_peak = li->peak_sessions.load(); + while (now_open > prev_peak && + !li->peak_sessions.compare_exchange_weak(prev_peak, now_open)) + { + } + } + + void decrement_interface_sessions(ListenInterface* li) + { + li->open_sessions.fetch_sub(1); + } + + // Admission control, run by the transport for every accepted connection + // before any TLS state exists. + // + // Reserving here rather than when the first request arrives is what makes + // max_open_sessions_hard a bound on *connections*: a client which + // completes the TCP and TLS handshakes and then sends nothing still holds + // a file descriptor and TLS state on the node, and previously went + // entirely uncounted. + // + // Returns nullopt to refuse, otherwise whether the connection was admitted + // above the soft limit (and so should be answered with a 503). Runs on the + // interface's libuv loop thread. + std::optional admit_connection( + ListenInterface* li, ::tcp::ConnID conn_id) + { + if (stopping.load()) + { + return std::nullopt; + } + + const size_t open = li->open_sessions.fetch_add(1); + if (open >= li->max_open_sessions_hard) + { + decrement_interface_sessions(li); + LOG_INFO_FMT( + "Refusing connection {} on interface {} - {} open, hard limit {}", + conn_id, + li->name, + open, + li->max_open_sessions_hard); + return std::nullopt; + } + + increment_interface_peak(li, open + 1); + increment_active_sessions(); + { + // li->bridge is assigned by listen() before the transport is started, + // and so before it can accept anything and reach here. + std::lock_guard guard(connection_interfaces_mutex); + connection_transports[conn_id] = li->bridge; + } + + const bool soft_limited = open >= li->max_open_sessions_soft; + if (soft_limited) + { + LOG_INFO_FMT( + "Soft-refusing connection {} (503) on interface {} - {} open, soft " + "limit {}", + conn_id, + li->name, + open, + li->max_open_sessions_soft); + } + return soft_limited; + } + + // Release the reservation taken by admit_connection. Invoked exactly once + // per admitted connection, when the transport has torn it down. + void release_connection(ListenInterface* li, ::tcp::ConnID conn_id) + { + { + std::lock_guard guard(connection_interfaces_mutex); + connection_transports.erase(conn_id); + } + decrement_interface_sessions(li); + decrement_active_sessions(); + } + + // Build the protocol session for a connection on `li`, which has already + // been admitted (see admit_connection). Returns nullptr to close the + // connection. Runs on the connection's TLS worker (invoked from + // OpenSSLServer's OnData), not on the libuv loop thread, so it must be + // safe to call concurrently for other connections. + std::shared_ptr make_session( + ListenInterface* li, + ::tcp::ConnID conn_id, + ccf::SessionWriter& writer, + std::vector peer_cert, + bool soft_limited) + { + if (stopping.load()) + { + // Shutting down - do not build new sessions, and in particular do not + // call error_reporter() (see `stopping`). + return nullptr; + } + + try + { + if (soft_limited) + { + return make_capped_session(li, conn_id, writer, std::move(peer_cert)); + } + return make_server_session(li, conn_id, writer, std::move(peer_cert)); + } + catch (const std::exception& e) + { + // Runs on a worker, so an escaping exception would be fatal. Closing + // the connection is the only useful response to being unable to build + // a session for it. + LOG_FAIL_FMT( + "Failed to create session {} on interface {}: {}", + conn_id, + li->name, + e.what()); + return nullptr; + } + } + + std::shared_ptr make_server_session( + ListenInterface* li, + ::tcp::ConnID conn_id, + ccf::SessionWriter& writer, + std::vector peer_cert) + { + if (li->app_protocol == "HTTP2") + { + return std::make_shared<::http::HTTP2ServerSession>( + rpc_map, + conn_id, + li->name, + writer, + std::move(peer_cert), + li->http_configuration, + error_reporter()); + } + if (li->app_protocol == "HTTP1") + { + return std::make_shared<::http::HTTPServerSession>( + rpc_map, + conn_id, + li->name, + writer, + std::move(peer_cert), + li->http_configuration, + error_reporter(), + get_commit_callbacks_subsystem()); + } + auto cpss = get_custom_protocol_subsystem(); + if (cpss != nullptr) + { + return cpss->create_session(li->app_protocol, conn_id, writer); + } + throw std::runtime_error(fmt::format( + "Unknown application protocol '{}' and custom protocol subsystem " + "missing", + li->app_protocol)); + } + + std::shared_ptr make_capped_session( + ListenInterface* li, + ::tcp::ConnID conn_id, + ccf::SessionWriter& writer, + std::vector peer_cert) + { + if (li->app_protocol == "HTTP2") + { + return std::make_shared>( + rpc_map, + conn_id, + li->name, + writer, + std::move(peer_cert), + li->http_configuration, + error_reporter()); + } + return std::make_shared>( + rpc_map, + conn_id, + li->name, + writer, + std::move(peer_cert), + li->http_configuration, + error_reporter(), + get_commit_callbacks_subsystem()); + } + + void send_udp_reply( + const std::string& name, ::tcp::ConnID id, std::span data) + { + std::lock_guard guard(interfaces_mutex); + auto it = udp_interfaces.find(name); + if (it == udp_interfaces.end()) + { + return; + } + + auto kit = it->second->peer_by_id.find(id); + if (kit == it->second->peer_by_id.end()) + { + return; + } + + const auto& key = kit->second; + const auto peerlen = static_cast(key.size()); + sockaddr_storage peer{}; + std::memcpy(&peer, key.data(), std::min(key.size(), sizeof(peer))); + + if (!it->second->server->send_to(peer, peerlen, data.data(), data.size())) + { + LOG_DEBUG_FMT("Failed to send UDP reply on interface {}", name); + } + } + + void close_udp_session( + const std::string& name, ListenInterface* li, ::tcp::ConnID id) + { + std::lock_guard guard(interfaces_mutex); + auto it = udp_interfaces.find(name); + if (it == udp_interfaces.end()) + { + return; + } + + auto kit = it->second->peer_by_id.find(id); + if (kit == it->second->peer_by_id.end()) + { + return; + } + + it->second->sessions_by_peer.erase(kit->second); + it->second->peer_by_id.erase(kit); + decrement_interface_sessions(li); + decrement_active_sessions(); + } + + // Drop UDP sessions which have seen no traffic for + // idle_connection_timeout. Unlike TCP there is no connection teardown to + // react to, and source addresses are trivially spoofable, so without this + // a UDP interface accumulates one session per distinct source address + // until it reaches max_open_sessions_hard and stops responding entirely. + // + // Sweeping here (on datagram arrival, rate-limited) rather than from a + // timer keeps the work where the growth happens and avoids a second timer + // per interface. Caller must hold interfaces_mutex. + void sweep_idle_udp_sessions(ListenInterface* li, DatagramInterface* udp) + { + if (!idle_connection_timeout.has_value()) + { + return; + } + + const auto now = std::chrono::steady_clock::now(); + if (now - udp->last_idle_sweep < udp_idle_sweep_interval) + { + return; + } + udp->last_idle_sweep = now; + + for (auto it = udp->sessions_by_peer.begin(); + it != udp->sessions_by_peer.end();) + { + if (now - it->second.last_active <= *idle_connection_timeout) + { + ++it; + continue; + } + + LOG_DEBUG_FMT( + "Closing idle UDP session {} on interface {}", + it->second.conn_id, + li->name); + udp->peer_by_id.erase(it->second.conn_id); + it = udp->sessions_by_peer.erase(it); + decrement_interface_sessions(li); + decrement_active_sessions(); + } + } + + std::shared_ptr get_or_create_udp_session( + ListenInterface* li, + DatagramInterface* udp, + ccf::SessionWriter& writer, + const sockaddr_storage& peer, + socklen_t peerlen) + { + std::lock_guard guard(interfaces_mutex); + sweep_idle_udp_sessions(li, udp); + + const auto key = peer_key(peer, peerlen); + auto sit = udp->sessions_by_peer.find(key); + if (sit != udp->sessions_by_peer.end()) + { + sit->second.last_active = std::chrono::steady_clock::now(); + return sit->second.session; + } + + if (stopping.load()) + { + return nullptr; + } + + auto cpss = get_custom_protocol_subsystem(); + if (cpss == nullptr) + { + LOG_DEBUG_FMT( + "Unknown UDP protocol '{}' and custom protocol subsystem missing", + li->app_protocol); + return nullptr; + } + + const size_t open = li->open_sessions.fetch_add(1); + if (open >= li->max_open_sessions_hard) + { + decrement_interface_sessions(li); + LOG_INFO_FMT( + "Refusing UDP session on interface {} - {} open, hard limit {}", + li->name, + open, + li->max_open_sessions_hard); + return nullptr; + } + const size_t now_open = open + 1; + increment_interface_peak(li, now_open); + increment_active_sessions(); + + const auto conn_id = + static_cast<::tcp::ConnID>(shared_conn_id.fetch_add(1)); + std::shared_ptr session; + try + { + session = cpss->create_session(li->app_protocol, conn_id, writer); + } + catch (const std::exception& e) + { + decrement_interface_sessions(li); + decrement_active_sessions(); + // Driven entirely by unauthenticated datagrams from a trivially + // spoofable source, so this must stay at a level which cannot be used + // to flood the log. + LOG_DEBUG_FMT( + "Failed to create UDP session on interface {}: {}", + li->name, + e.what()); + return nullptr; + } + + if (session == nullptr) + { + decrement_interface_sessions(li); + decrement_active_sessions(); + return nullptr; + } + + udp->peer_by_id.emplace(conn_id, key); + udp->sessions_by_peer.emplace( + key, UdpSession{session, conn_id, std::chrono::steady_clock::now()}); + return session; + } + + public: + explicit RPCConnectionManager(std::shared_ptr rpc_map_) : + rpc_map(std::move(rpc_map_)) + {} + + ~RPCConnectionManager() override + { + // By this point either stop() has already run (the normal path, from + // Enclave::run() while the loop was still going), in which case this is + // a no-op, or node startup failed before the event loop was ever + // entered - which is exactly LoopState::NotRunning. + stop(asynchost::OpenSSLServer::LoopState::NotRunning); + } + + // Tear down every transport. `loop_state` says whether another thread is + // running the libuv loop, which the transports cannot determine for + // themselves - see OpenSSLServer::stop(). + void stop( + asynchost::OpenSSLServer::LoopState loop_state = + asynchost::OpenSSLServer::LoopState::NotRunning) + { + // Refuse new sessions before tearing anything down. This is what makes + // it safe for the destructor to call stop(): make_session() will no + // longer reach error_reporter()/shared_from_this(). + stopping.store(true); + + // Snapshot the transports under the lock, but stop them without it. + // Stopping blocks until the libuv loop has torn down every connection, + // and the loop thread itself needs interfaces_mutex (for example in + // get_or_create_udp_session(), send_udp_reply() or a drained task + // calling report_parsing_error()), so holding it across the stop would + // deadlock. The snapshots are shared_ptr, so the transports stay alive + // for the duration even if the maps change. + std::vector> bridges; + std::vector> datagram_servers; + { + std::lock_guard guard(interfaces_mutex); + for (auto& [name, li] : interfaces) + { + if (li->bridge != nullptr) + { + bridges.push_back(li->bridge); + } + } + for (auto& [name, interface] : udp_interfaces) + { + datagram_servers.push_back(interface->server); + } + } + + for (const auto& bridge : bridges) + { + bridge->stop(loop_state); + } + for (const auto& server : datagram_servers) + { + server->stop( + loop_state == asynchost::OpenSSLServer::LoopState::Running ? + asynchost::DatagramServer::LoopState::Running : + asynchost::DatagramServer::LoopState::NotRunning); + } + } + + // Bind and start listening on `name` (which must have been configured via + // update_listening_interface_options). Returns the bound port, which for a + // configured port of 0 is the ephemeral port the OS assigned. Throws if the + // interface is unconfigured, or if the socket cannot be bound. + uint16_t listen( + const std::string& name, const std::string& host, const std::string& port) + { + std::lock_guard guard(interfaces_mutex); + auto it = interfaces.find(name); + if (it == interfaces.end()) + { + throw std::logic_error( + fmt::format("Cannot listen on unconfigured interface '{}'", name)); + } + auto* li = it->second.get(); + + const bool plaintext = + li->endorsement.authority == ccf::Authority::UNSECURED; + const std::string alpn = + plaintext ? "" : (li->app_protocol == "HTTP2" ? "h2" : "http/1.1"); + + std::string cert_pem; + std::string key_pem; + if (!plaintext) + { + auto c = certs.find(li->endorsement.authority); + if (c != certs.end()) + { + cert_pem = c->second.first; + key_pem = c->second.second; + } + } + + auto on_accept = [this, li](::tcp::ConnID cid) { + return admit_connection(li, cid); + }; + auto factory = [this, li]( + ::tcp::ConnID cid, + ccf::SessionWriter& w, + std::vector pc, + bool soft_limited) { + return make_session(li, cid, w, std::move(pc), soft_limited); + }; + auto on_closed = [this, li](::tcp::ConnID cid) { + release_connection(li, cid); + }; + + LOG_INFO_FMT( + "Registering RPC interface {}, on tcp {}:{}", name, host, port); + li->bridge = std::make_shared( + asynchost::OpenSSLServer::Config{ + .host = host, + .port = parse_port(name, port), + .cert_pem = cert_pem, + .key_pem = key_pem, + .alpn = alpn, + .plaintext = plaintext, + .idle_timeout = idle_connection_timeout, + .shared_next_id = &shared_conn_id, + .inbound_admission = inbound_admission}, + factory, + on_closed, + on_accept); + li->bridge->start(); + const uint16_t bound = li->bridge->port(); + LOG_INFO_FMT( + "Registered RPC interface {}, on tcp {}:{}", name, host, bound); + return bound; + } + + // Bind and start a UDP listener for `name` (interfaces with protocol + // "udp"). + // + // === QUIC EXTENSION POINT === + // A real QUIC interface would, instead of echoing, hand each datagram to an + // OpenSSL QUIC listener (OpenSSL >= 3.5). The DatagramServer below is the + // shared substrate for that (see host/datagram_server.h). + uint16_t listen_udp( + const std::string& name, const std::string& host, const std::string& port) + { + std::lock_guard guard(interfaces_mutex); + auto li_it = interfaces.find(name); + if (li_it == interfaces.end()) + { + throw std::logic_error(fmt::format( + "Cannot listen on unconfigured UDP interface '{}'", name)); + } + auto* li = li_it->second.get(); + + auto udp = std::make_unique(); + auto* udp_ptr = udp.get(); + udp->writer = std::make_unique( + [this, name](::tcp::ConnID id, std::span data) { + send_udp_reply(name, id, data); + }, + [this, name, li](::tcp::ConnID id) { + close_udp_session(name, li, id); + }); + auto* writer = udp->writer.get(); + + LOG_INFO_FMT( + "Registering RPC interface {}, on udp {}:{}", name, host, port); + udp->server = std::make_shared( + host, + parse_port(name, port), + [this, li, udp_ptr, writer]( + const uint8_t* data, + size_t len, + const sockaddr_storage& peer, + socklen_t peerlen) { + if (li->app_protocol == "QUIC") + { + // Placeholder behaviour until OpenSSL-native QUIC: echo the + // datagram straight back. + // + // Deliberately stateless. UDP has no connection to close, and + // source addresses are trivially spoofable, so retaining anything + // per peer here would let an attacker grow the session map and + // consume the interface's session capacity permanently with a + // handful of forged packets. The echo needs no state, so it keeps + // none. + if (!udp_ptr->server->send_to(peer, peerlen, data, len)) + { + LOG_DEBUG_FMT( + "Failed to echo UDP datagram on interface {}", li->name); + } + return; + } + + auto session = + get_or_create_udp_session(li, udp_ptr, *writer, peer, peerlen); + if (session == nullptr) + { + return; + } + session->handle_incoming_data(std::vector(data, data + len)); + }); + udp->server->start(); + const uint16_t bound = udp->server->port(); + udp_interfaces.emplace(name, std::move(udp)); + LOG_INFO_FMT( + "Registered RPC interface {}, on udp {}:{}", name, host, bound); + return bound; + } + + // ----- AbstractRPCSessions / AbstractRPCResponder ----------------------- + + bool reply_async( + int64_t id, + bool terminate_after_reply, + std::vector&& data) override + { + // The transport is snapshotted under the lock and used without it: + // stopping a transport blocks on the libuv loop, which itself needs + // interfaces_mutex, so no manager lock may be held across a send. + std::shared_ptr bridge; + { + std::lock_guard guard(connection_interfaces_mutex); + auto it = connection_transports.find(id); + if (it != connection_transports.end()) + { + bridge = it->second; + } + } + + auto session = bridge == nullptr ? nullptr : bridge->get_session(id); + if (session == nullptr) + { + LOG_DEBUG_FMT("Refusing to reply to unknown session {}", id); + return false; + } + + session->send_data(std::move(data)); + if (terminate_after_reply) + { + session->close_session(); + } + return true; + } + + ccf::SessionMetrics get_session_metrics() override + { + ccf::SessionMetrics sm; + std::lock_guard guard(interfaces_mutex); + for (auto& [name, li] : interfaces) + { + const auto counts = error_counts->get(name); + ccf::SessionMetrics::Errors errs{}; + errs.parsing = counts.parsing; + errs.request_payload_too_large = counts.payload_too_large; + errs.request_header_too_large = counts.header_too_large; + + sm.interfaces[name] = { + li->open_sessions.load(), + li->peak_sessions.load(), + li->max_open_sessions_soft, + li->max_open_sessions_hard, + errs}; + } + sm.active = active_sessions.load(); + sm.peak = peak_sessions.load(); + return sm; + } + + void set_node_cert( + const ccf::crypto::Pem& cert, const ccf::crypto::Pem& pk) override + { + set_cert(ccf::Authority::NODE, cert, pk); + } + + void set_network_cert( + const ccf::crypto::Pem& cert, const ccf::crypto::Pem& pk) override + { + set_cert(ccf::Authority::SERVICE, cert, pk); + } + + void set_cert( + ccf::Authority authority, + const ccf::crypto::Pem& cert, + const ccf::crypto::Pem& pk) + { + std::lock_guard guard(interfaces_mutex); + certs[authority] = {cert.str(), pk.str()}; + for (auto& [name, li] : interfaces) + { + if (li->endorsement.authority == authority && li->bridge != nullptr) + { + li->bridge->set_server_cert(cert.str(), pk.str()); + } + } + } + + // Set the idle-connection timeout applied to interfaces bound after this + // call (nullopt disables idle closure). Call before listen(). + void set_idle_connection_timeout( + std::optional timeout) + { + std::lock_guard guard(interfaces_mutex); + idle_connection_timeout = timeout; + } + + void update_listening_interface_options( + const ccf::NodeInfoNetwork& node_info) override + { + std::lock_guard guard(interfaces_mutex); + for (const auto& [name, interface] : node_info.rpc_interfaces) + { + auto it = interfaces.find(name); + if (it == interfaces.end()) + { + it = + interfaces.emplace(name, std::make_unique()).first; + it->second->name = name; + } + auto* li = it->second.get(); + + li->max_open_sessions_soft = interface.max_open_sessions_soft.value_or( + max_open_sessions_soft_default); + li->max_open_sessions_hard = interface.max_open_sessions_hard.value_or( + max_open_sessions_hard_default); + li->endorsement = interface.endorsement.value_or(endorsement_default); + li->http_configuration = + interface.http_configuration.value_or(http::ParserConfiguration{}); + li->app_protocol = interface.app_protocol.value_or("HTTP1"); + + LOG_INFO_FMT( + "Setting max open sessions on interface \"{}\" ({}) to [{}, {}] and " + "endorsement authority to {}", + name, + interface.bind_address, + li->max_open_sessions_soft, + li->max_open_sessions_hard, + li->endorsement.authority); + } + } + + void set_custom_protocol_subsystem( + std::shared_ptr cpss) override + { + std::lock_guard guard(subsystems_mutex); + custom_protocol_subsystem = std::move(cpss); + } + + void set_commit_callbacks_subsystem( + std::shared_ptr fcss) override + { + std::lock_guard guard(subsystems_mutex); + commit_callbacks_subsystem = std::move(fcss); + } + }; +} diff --git a/src/host/rpc_connections.h b/src/host/rpc_connections.h deleted file mode 100644 index 78b8df00f033..000000000000 --- a/src/host/rpc_connections.h +++ /dev/null @@ -1,524 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the Apache 2.0 License. -#pragma once - -#include "../tcp/msg_types.h" -#include "../udp/msg_types.h" -#include "ds/messaging.h" -#include "tcp.h" -#include "timer.h" -#include "udp.h" - -#include -#include -#include -#include - -namespace // NOLINT(cert-dcl59-cpp) -{ - template - constexpr bool isTCP() - { - return std::is_same(); - } - - template - constexpr bool isUDP() - { - return std::is_same(); - } - - template - constexpr const char* getConnTypeName() - { - if constexpr (isTCP()) - { - return "TCP"; - } - else if constexpr (isUDP()) - { - return "UDP"; - } - else - { - throw std::runtime_error("Invalid connection type"); - } - } -} - -namespace asynchost -{ - /** - * Generates next ID, passed as an argument to RPCConnectionsImpl so that we - * can have multiple and avoid reusing the same ConnID across each. - */ - class ConnIDGenerator - { - public: - /// This is the same as ccf::tls::ConnID and udp::ConnID - using ConnID = int64_t; - static_assert(std::is_same<::tcp::ConnID, udp::ConnID>()); - static_assert(std::is_same<::tcp::ConnID, ConnID>()); - - ConnIDGenerator() : next_id(1) {} - - template - ConnID get_next_id(T& sockets) - { - auto id = next_id++; - const auto initial = id; - - if (next_id < 0) - { - next_id = 1; - } - - while (sockets.find(id) != sockets.end()) - { - id++; - - if (id < 0) - { - id = 1; - } - - if (id == initial) - { - throw std::runtime_error( - "Exhausted all IDs for host RPC connections"); - } - } - - return id; - } - - private: - std::atomic next_id; - }; - - template - class RPCConnectionsImpl - { - using ConnID = ConnIDGenerator::ConnID; - - class RPCClientBehaviour : public SocketBehaviour - { - public: - RPCConnectionsImpl& parent; - ConnID id; - - RPCClientBehaviour(RPCConnectionsImpl& parent, ConnID id) : - SocketBehaviour("RPC Client", getConnTypeName()), - parent(parent), - id(id) - { - parent.mark_active(id); - } - - void on_resolve_failed() override - { - LOG_DEBUG_FMT("rpc resolve failed {}", id); - cleanup(); - } - - void on_connect_failed() override - { - LOG_DEBUG_FMT("rpc connect failed {}", id); - cleanup(); - } - - bool on_read(size_t len, uint8_t*& data, sockaddr /*unused*/) override - { - LOG_DEBUG_FMT("rpc read {}: {}", id, len); - - parent.mark_active(id); - - RINGBUFFER_WRITE_MESSAGE( - ::tcp::tcp_inbound, - parent.to_enclave, - id, - serializer::ByteRange{data, len}); - - return true; - } - - void on_disconnect() override - { - LOG_DEBUG_FMT("rpc disconnect {}", id); - cleanup(); - } - - void cleanup() - { - if constexpr (isTCP()) - { - RINGBUFFER_WRITE_MESSAGE(::tcp::tcp_close, parent.to_enclave, id); - } - } - }; - - class RPCServerBehaviour : public SocketBehaviour - { - public: - RPCConnectionsImpl& parent; - ConnID id; - - RPCServerBehaviour(RPCConnectionsImpl& parent, ConnID id) : - SocketBehaviour("RPC Client", getConnTypeName()), - parent(parent), - id(id) - {} - - void on_accept(ConnType& peer) override - { - // UDP connections don't register peers - if constexpr (isUDP()) - { - return; - } - - auto client_id = parent.get_next_id(); - peer->set_behaviour( - std::make_unique(parent, client_id)); - parent.sockets.emplace(client_id, peer); - - on_start(client_id); - } - - void on_start(int64_t peer_id) override - { - const auto interface_name = parent.get_interface_listen_name(id); - - LOG_DEBUG_FMT( - "rpc start {} on interface \"{}\" as {}", - peer_id, - interface_name, - this->conn_name); - - if constexpr (isTCP()) - { - RINGBUFFER_WRITE_MESSAGE( - ::tcp::tcp_start, parent.to_enclave, peer_id, interface_name); - return; - } - - if constexpr (isUDP()) - { - RINGBUFFER_WRITE_MESSAGE( - udp::udp_start, parent.to_enclave, peer_id, interface_name); - return; - } - } - - bool on_read(size_t len, uint8_t*& data, sockaddr addr) override - { - // UDP connections don't have clients, it's all done in the server - if constexpr (isUDP()) - { - auto [addr_family, addr_data] = udp::sockaddr_encode(addr); - - LOG_DEBUG_FMT("rpc udp read into ring buffer {}: {}", id, len); - RINGBUFFER_WRITE_MESSAGE( - udp::udp_inbound, - parent.to_enclave, - id, - addr_family, - addr_data, - serializer::ByteRange{data, len}); - } - - return true; - } - - void cleanup() - { - parent.sockets.erase(id); - } - }; - - std::unordered_map sockets; - // The timer close callback deletes this object asynchronously. - std::shared_ptr id_gen; - - // Measured in seconds - std::unordered_map idle_times; - - std::optional client_connection_timeout = - std::nullopt; - - std::optional idle_connection_timeout = std::nullopt; - - ringbuffer::WriterPtr to_enclave; - - public: - RPCConnectionsImpl( - ringbuffer::AbstractWriterFactory& writer_factory, - std::shared_ptr id_gen_, - std::optional client_connection_timeout_ = - std::nullopt, - std::optional idle_connection_timeout_ = - std::nullopt) : - id_gen(std::move(id_gen_)), - client_connection_timeout(client_connection_timeout_), - idle_connection_timeout(idle_connection_timeout_), - to_enclave(writer_factory.create_writer_to_inside()) - { - if (id_gen == nullptr) - { - throw std::invalid_argument( - "RPC connections require a connection ID generator"); - } - } - - bool listen( - ConnID id, std::string& host, std::string& port, const std::string& name) - { - if (id == 0) - { - id = get_next_id(); - } - - if (sockets.find(id) != sockets.end()) - { - LOG_FAIL_FMT("Cannot listen on id {}: already in use", id); - return false; - } - - ConnType s; - s->set_behaviour(std::make_unique(*this, id)); - - if (!s->listen(host, port, name)) - { - return false; - } - - host = s->get_host(); - port = s->get_port(); - - sockets.emplace(id, s); - - // UDP connections don't have peers, so we need to register the main - // socket TCP connections started via peer, on on_accept behaviour call - if constexpr (isUDP()) - { - s->start(id); - } - - return true; - } - - bool connect(ConnID id, const std::string& host, const std::string& port) - { - if (id == 0) - { - id = get_next_id(); - } - - if (sockets.find(id) != sockets.end()) - { - LOG_FAIL_FMT("Cannot connect on id {}: already in use", id); - return false; - } - - auto s = ConnType(true, client_connection_timeout); - s->set_behaviour(std::make_unique(*this, id)); - - if (!s->connect(host, port)) - { - return false; - } - - sockets.emplace(id, s); - return true; - } - - bool write(ConnID id, size_t len, const uint8_t* data, sockaddr addr = {}) - { - auto s = sockets.find(id); - - if (s == sockets.end()) - { - LOG_FAIL_FMT( - "Received an outbound message for id {} which is not a known " - "connection. Ignoring message of {} bytes", - id, - len); - return false; - } - - if (s->second.is_null()) - { - return false; - } - - mark_active(id); - - return s->second->write(len, data, addr); - } - - bool stop(ConnID id) - { - // Invalidating the TCP socket will result in the handle being closed. No - // more messages will be read from or written to the TCP socket. - sockets[id] = nullptr; - - RINGBUFFER_WRITE_MESSAGE(::tcp::tcp_close, to_enclave, id); - - return true; - } - - bool close(ConnID id) - { - if (sockets.erase(id) < 1) - { - LOG_DEBUG_FMT("Cannot close id {}: does not exist", id); - return false; - } - - // Make sure idle_times is cleaned up here, though in practice it should - // have been cleared when an earlier stop() was called - idle_times.erase(id); - - return true; - } - - void register_message_handlers( - messaging::Dispatcher& disp) - { - DISPATCHER_SET_MESSAGE_HANDLER( - disp, ::tcp::tcp_outbound, [this](const uint8_t* data, size_t size) { - auto [id, body] = - ringbuffer::read_message<::tcp::tcp_outbound>(data, size); - - auto connect_id = static_cast(id); - LOG_DEBUG_FMT("rpc write from enclave {}: {}", connect_id, body.size); - - write(connect_id, body.size, body.data); - }); - - DISPATCHER_SET_MESSAGE_HANDLER( - disp, ::tcp::tcp_connect, [this](const uint8_t* data, size_t size) { - auto [id, host, port] = - ringbuffer::read_message<::tcp::tcp_connect>(data, size); - - LOG_DEBUG_FMT("rpc connect request from enclave {}", id); - - if (check_enclave_side_id(id)) - { - connect(id, host, port); - } - else - { - LOG_FAIL_FMT( - "rpc session id is not in dedicated from-enclave range ({})", id); - } - }); - - DISPATCHER_SET_MESSAGE_HANDLER( - disp, ::tcp::tcp_stop, [this](const uint8_t* data, size_t size) { - auto [id, msg] = - ringbuffer::read_message<::tcp::tcp_stop>(data, size); - - LOG_DEBUG_FMT("rpc stop from enclave {}, {}", id, msg); - stop(id); - - // Immediately stop tracking idle timeout for this ID too - idle_times.erase(id); - }); - - DISPATCHER_SET_MESSAGE_HANDLER( - disp, ::tcp::tcp_closed, [this](const uint8_t* data, size_t size) { - auto [id] = ringbuffer::read_message<::tcp::tcp_closed>(data, size); - - LOG_DEBUG_FMT("rpc closed from enclave {}", id); - close(id); - }); - } - - void register_udp_message_handlers( - messaging::Dispatcher& disp) - { - DISPATCHER_SET_MESSAGE_HANDLER( - disp, udp::udp_outbound, [this](const uint8_t* data, size_t size) { - auto [id, addr_family, addr_data, body] = - ringbuffer::read_message(data, size); - - auto connect_id = static_cast(id); - LOG_DEBUG_FMT("rpc write from enclave {}: {}", connect_id, body.size); - - auto addr = udp::sockaddr_decode(addr_family, addr_data); - write(connect_id, body.size, body.data, addr); - }); - } - - void mark_active(ConnID id) - { - idle_times[id] = 0; - } - - void on_timer() - { - if (!idle_connection_timeout.has_value()) - { - return; - } - - const size_t max_idle_time = idle_connection_timeout->count(); - - auto it = idle_times.begin(); - while (it != idle_times.end()) - { - auto& [id, idle_time] = *it; - if (idle_time > max_idle_time) - { - LOG_INFO_FMT( - "Closing socket {} after {}s idle (max = {}s)", - id, - idle_time, - max_idle_time); - stop(id); - it = idle_times.erase(it); - } - else - { - idle_time += 1; - ++it; - } - } - } - - private: - ConnID get_next_id() - { - return id_gen->get_next_id(sockets); - } - - bool check_enclave_side_id(ConnID id) - { - return id < 0; - } - - std::string get_interface_listen_name(ConnID id) - { - const auto it = sockets.find(id); - if (it == sockets.end()) - { - LOG_FAIL_FMT( - "Requested interface number {}, has {}", id, sockets.size()); - throw std::logic_error(fmt::format("No socket with id {}", id)); - } - - auto listen_name = it->second->get_listen_name(); - if (!listen_name.has_value()) - { - throw std::logic_error( - fmt::format("Interface {} has no listen name", id)); - } - - return listen_name.value(); - } - }; - - template - using RPCConnections = proxy_ptr>>; -} diff --git a/src/host/run.cpp b/src/host/run.cpp index 68eb671002a8..31ef5d661fca 100644 --- a/src/host/run.cpp +++ b/src/host/run.cpp @@ -36,12 +36,10 @@ #include "lfs_file_handler.h" #include "node_connections.h" #include "pal/quote_generation.h" -#include "rpc_connections.h" #include "sig_term.h" #include "tcp.h" #include "ticker.h" #include "time_bound_logger.h" -#include "udp.h" #include #include @@ -79,9 +77,6 @@ size_t asynchost::TCPImpl::remaining_read_quota = asynchost::TCPImpl::max_read_quota; bool asynchost::TCPImpl::alloc_quota_logged = false; -size_t asynchost::UDPImpl::remaining_read_quota = - asynchost::UDPImpl::max_read_quota; - void print_version(int64_t ignored) { (void)ignored; @@ -190,89 +185,6 @@ namespace ccf {} }; - void setup_rpc_interfaces( - host::HostConfig& config, - asynchost::RPCConnections& rpc, - asynchost::RPCConnections& rpc_udp) - { - ResolvedAddresses resolved_rpc_addresses; - - // Bind interfaces with an explicit (non-zero) port before those requesting - // an ephemeral port (port 0). Multiple interfaces on a node can share a - // single host address (for example the sole ::1 IPv6 loopback), and if an - // ephemeral interface is bound first the OS may assign it the exact port - // that another interface is configured to bind, making that later bind fail - // with "address already in use". - std::vector ordered_interface_names; - ordered_interface_names.reserve(config.network.rpc_interfaces.size()); - for (const auto& [name, interface] : config.network.rpc_interfaces) - { - if (cli::validate_address(interface.bind_address).second != "0") - { - ordered_interface_names.push_back(name); - } - } - for (const auto& [name, interface] : config.network.rpc_interfaces) - { - if (cli::validate_address(interface.bind_address).second == "0") - { - ordered_interface_names.push_back(name); - } - } - - for (const auto& name : ordered_interface_names) - { - auto& interface = config.network.rpc_interfaces.at(name); - auto [rpc_host, rpc_port] = cli::validate_address(interface.bind_address); - LOG_INFO_FMT( - "Registering RPC interface {}, on {} {}:{}", - name, - interface.protocol, - rpc_host, - rpc_port); - - if (interface.protocol == "udp") - { - rpc_udp->behaviour.listen(0, rpc_host, rpc_port, name); - } - else - { - rpc->behaviour.listen(0, rpc_host, rpc_port, name); - } - - LOG_INFO_FMT( - "Registered RPC interface {}, on {} {}:{}", - name, - interface.protocol, - rpc_host, - rpc_port); - - resolved_rpc_addresses[name] = ccf::make_net_address(rpc_host, rpc_port); - interface.bind_address = ccf::make_net_address(rpc_host, rpc_port); - - // If public RPC address is not set, default to local RPC address - if (interface.published_address.empty()) - { - interface.published_address = interface.bind_address; - } - - auto [pub_host, pub_port] = - cli::validate_address(interface.published_address); - if (pub_port == "0") - { - pub_port = rpc_port; - interface.published_address = ccf::make_net_address(pub_host, pub_port); - } - } - - if (!config.output_files.rpc_addresses_file.empty()) - { - files::dump( - nlohmann::json(resolved_rpc_addresses).dump(), - config.output_files.rpc_addresses_file); - } - } - void configure_snp_attestation(ccf::StartupConfig& startup_config) { if (ccf::pal::platform != ccf::pal::Platform::SNP) @@ -478,6 +390,7 @@ namespace ccf ccf::StartupConfig& startup_config, std::vector& node_cert, std::vector& service_cert, + std::vector& rpc_addresses, ccf::LoggerLevel log_level, ringbuffer::NotifyingWriterFactory& notifying_factory, asynchost::Ledger& ledger) @@ -499,6 +412,7 @@ namespace ccf startup_config, node_cert, service_cert, + rpc_addresses, config.command.type, log_level, config.worker_threads, @@ -529,6 +443,13 @@ namespace ccf } LOG_INFO_FMT("Created new node"); + + // The enclave resolves and binds the RPC interfaces (including ephemeral + // ports), and reports the resolved addresses back here to be written out. + if (!config.output_files.rpc_addresses_file.empty()) + { + files::dump(rpc_addresses, config.output_files.rpc_addresses_file); + } return std::nullopt; } @@ -614,9 +535,6 @@ namespace ccf // reset the inbound-TCP processing quota each iteration const asynchost::ResetTCPReadQuota reset_tcp_quota; - // reset the inbound-UDP processing quota each iteration - const asynchost::ResetUDPReadQuota reset_udp_quota; - // handle outbound logging and admin messages from the enclave const asynchost::HandleRingbuffer handle_ringbuffer( 1ms, @@ -642,7 +560,6 @@ namespace ccf "snapshots.read_only_directory is deprecated and will be removed in a " "future release"); } - std::optional files_cleanup; if ( (config.files_cleanup.max_snapshots.has_value() || @@ -693,37 +610,44 @@ namespace ccf config.output_files.node_to_node_address_file); } - const auto id_gen = std::make_shared(); - - asynchost::RPCConnections rpc( - 1s, // Tick once-per-second to track idle connections, - writer_factory, - id_gen, - config.client_connection_timeout, - config.idle_connection_timeout); - rpc->behaviour.register_message_handlers(buffer_processor.get_dispatcher()); - - asynchost::RPCConnections rpc_udp( - 1s, - writer_factory, - id_gen, - config.client_connection_timeout, - config.idle_connection_timeout); - rpc_udp->behaviour.register_udp_message_handlers( - buffer_processor.get_dispatcher()); - // Initialise the curlm singleton curl_global_init(CURL_GLOBAL_DEFAULT); auto curl_libuv_context = curl::CurlmLibuvContextSingleton(uv_default_loop()); - // Setup RPC interfaces - setup_rpc_interfaces(config, rpc, rpc_udp); + // Validate and normalise the configured RPC addresses here, at the input + // boundary, so that everything downstream sees a well-formed "host:port" + // with a port in range. ccf::split_net_address, used to bind them, is + // deliberately lenient and does no validation of its own. + for (auto& [name, interface] : config.network.rpc_interfaces) + { + try + { + const auto [rpc_host, rpc_port] = + cli::validate_address(interface.bind_address); + interface.bind_address = ccf::make_net_address(rpc_host, rpc_port); + + if (!interface.published_address.empty()) + { + const auto [pub_host, pub_port] = + cli::validate_address(interface.published_address); + interface.published_address = + ccf::make_net_address(pub_host, pub_port); + } + } + catch (const std::exception& e) + { + LOG_FATAL_FMT( + "Invalid address for RPC interface {}: {}. Exiting.", name, e.what()); + return static_cast(CLI::ExitCodes::ValidationError); + } + } // Prepare startup configuration const size_t certificate_size = 4096; std::vector node_cert(certificate_size); std::vector service_cert(certificate_size); + std::vector rpc_addresses; ccf::StartupConfig startup_config(config); @@ -829,6 +753,7 @@ namespace ccf startup_config, node_cert, service_cert, + rpc_addresses, log_level, factories.notifying_factory, ledger); diff --git a/src/host/test/openssl_server_test.cpp b/src/host/test/openssl_server_test.cpp new file mode 100644 index 000000000000..dbf175e05bb9 --- /dev/null +++ b/src/host/test/openssl_server_test.cpp @@ -0,0 +1,2077 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the Apache 2.0 License. + +// Unit tests for the OpenSSL-native RPC transport and SessionWriter bridge. + +#include "ccf/crypto/ec_key_pair.h" +#include "ccf/ds/x509_time_fmt.h" +#include "clients/rpc_tls_client.h" +#include "crypto/certs.h" +#include "host/datagram_server.h" +#include "host/tls/openssl_server.h" +#include "host/tls/openssl_session_manager.h" +#include "tasks/task_system.h" + +#define DOCTEST_CONFIG_IMPLEMENT +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +using namespace asynchost; + +namespace +{ + // The host process ignores SIGPIPE (see src/host/run.cpp), so writes to a + // socket the peer has already closed return EPIPE rather than killing it. + // Tests must do the same to reproduce production behaviour. + [[maybe_unused]] const bool ignore_sigpipe = []() { + return signal(SIGPIPE, SIG_IGN) != SIG_ERR; + }(); + + std::pair make_server_cert() + { + using namespace std::literals; + auto kp = ccf::crypto::make_ec_key_pair(); + const auto valid_from = + ccf::ds::to_x509_time_string(std::chrono::system_clock::now() - 24h); + auto cert = ccf::crypto::create_self_signed_cert( + kp, "CN=localhost", {}, valid_from, /*validity_days*/ 365); + return {cert.str(), kp->private_key_pem().str()}; + } + + struct TestCA + { + ccf::crypto::ECKeyPairPtr kp; + ccf::crypto::Pem cert; + }; + + TestCA make_ca() + { + using namespace std::literals; + auto kp = ccf::crypto::make_ec_key_pair(); + const auto valid_from = + ccf::ds::to_x509_time_string(std::chrono::system_clock::now() - 24h); + return { + kp, + ccf::crypto::create_self_signed_cert( + kp, "CN=issuer", {}, valid_from, /*validity_days*/ 365)}; + } + + std::pair make_endorsed_server_cert( + const TestCA& ca) + { + using namespace std::literals; + auto kp = ccf::crypto::make_ec_key_pair(); + const auto valid_from = + ccf::ds::to_x509_time_string(std::chrono::system_clock::now() - 24h); + auto cert = ccf::crypto::create_endorsed_cert( + kp, + "CN=localhost", + {}, + valid_from, + /*validity_days*/ 365, + ca.kp->private_key_pem(), + ca.cert); + return {cert.str(), kp->private_key_pem().str()}; + } + + // Blocking TLS client: connects, sends `req` in full, reads exactly + // `expected_resp` bytes. Verification is disabled for the self-signed test + // certificate. + std::vector tls_client_exchange( + uint16_t port, + const std::vector& req, + size_t expected_resp, + const std::string& client_cert = {}, + const std::string& client_key = {}) + { + const int fd = ::socket(AF_INET, SOCK_STREAM, 0); + REQUIRE(fd >= 0); + + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_port = htons(port); + REQUIRE(inet_pton(AF_INET, "127.0.0.1", &addr.sin_addr) == 1); + REQUIRE( + ::connect(fd, reinterpret_cast(&addr), sizeof(addr)) == 0); + + SSL_CTX* cctx = SSL_CTX_new(TLS_client_method()); + REQUIRE(cctx != nullptr); + if (!client_cert.empty()) + { + BIO* cb = BIO_new_mem_buf( + client_cert.data(), static_cast(client_cert.size())); + X509* xc = PEM_read_bio_X509(cb, nullptr, nullptr, nullptr); + BIO_free(cb); + REQUIRE(xc != nullptr); + REQUIRE(SSL_CTX_use_certificate(cctx, xc) == 1); + X509_free(xc); + BIO* kb = + BIO_new_mem_buf(client_key.data(), static_cast(client_key.size())); + EVP_PKEY* pk = PEM_read_bio_PrivateKey(kb, nullptr, nullptr, nullptr); + BIO_free(kb); + REQUIRE(pk != nullptr); + REQUIRE(SSL_CTX_use_PrivateKey(cctx, pk) == 1); + EVP_PKEY_free(pk); + } + SSL* ssl = SSL_new(cctx); + REQUIRE(ssl != nullptr); + REQUIRE(SSL_set_fd(ssl, fd) == 1); + SSL_set_connect_state(ssl); + REQUIRE(SSL_connect(ssl) == 1); + + size_t off = 0; + while (off < req.size()) + { + const int n = + SSL_write(ssl, req.data() + off, static_cast(req.size() - off)); + REQUIRE(n > 0); + off += static_cast(n); + } + + std::vector resp; + resp.reserve(expected_resp); + while (resp.size() < expected_resp) + { + uint8_t buf[16384]; + const int n = SSL_read(ssl, buf, static_cast(sizeof(buf))); + if (n <= 0) + { + break; + } + resp.insert(resp.end(), buf, buf + static_cast(n)); + } + + SSL_shutdown(ssl); + SSL_free(ssl); + SSL_CTX_free(cctx); + ::close(fd); + return resp; + } + + std::vector random_bytes(size_t n) + { + std::vector v(n); + std::mt19937 rng(12345); + for (auto& b : v) + { + b = static_cast(rng()); + } + return v; + } + + std::string negotiated_group_name(SSL* ssl) + { + const auto group_id = SSL_get_negotiated_group(ssl); + if (group_id == NID_undef) + { + return {}; + } + + const auto* group_name = SSL_group_to_name(ssl, group_id); + if (group_name != nullptr) + { + return group_name; + } + + return std::to_string(group_id); + } + + struct HandshakeResult + { + bool succeeded = false; + std::string group; + std::string cipher; + }; + + // Handshakes against the server, optionally restricting what the client + // offers, and reports what was negotiated. Used to assert the server's + // configured cipher/group policy from the wire rather than by inspecting + // the SSL_CTX. + HandshakeResult handshake_and_inspect( + uint16_t port, + const std::string& client_groups = {}, + const std::string& client_ciphersuites = {}) + { + const int fd = ::socket(AF_INET, SOCK_STREAM, 0); + REQUIRE(fd >= 0); + + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_port = htons(port); + REQUIRE(inet_pton(AF_INET, "127.0.0.1", &addr.sin_addr) == 1); + REQUIRE( + ::connect(fd, reinterpret_cast(&addr), sizeof(addr)) == 0); + + SSL_CTX* cctx = SSL_CTX_new(TLS_client_method()); + REQUIRE(cctx != nullptr); + if (!client_groups.empty()) + { + REQUIRE(SSL_CTX_set1_groups_list(cctx, client_groups.c_str()) == 1); + } + if (!client_ciphersuites.empty()) + { + REQUIRE(SSL_CTX_set_ciphersuites(cctx, client_ciphersuites.c_str()) == 1); + } + + SSL* ssl = SSL_new(cctx); + REQUIRE(ssl != nullptr); + REQUIRE(SSL_set_fd(ssl, fd) == 1); + SSL_set_connect_state(ssl); + + HandshakeResult result; + result.succeeded = SSL_connect(ssl) == 1; + if (result.succeeded) + { + result.group = negotiated_group_name(ssl); + const auto* cipher = SSL_get_current_cipher(ssl); + if (cipher != nullptr) + { + result.cipher = SSL_CIPHER_get_name(cipher); + } + SSL_shutdown(ssl); + } + + SSL_free(ssl); + SSL_CTX_free(cctx); + ::close(fd); + return result; + } + + bool supports_hybrid_groups() + { + SSL_CTX* ctx = SSL_CTX_new(TLS_client_method()); + REQUIRE(ctx != nullptr); + const bool supported = + SSL_CTX_set1_groups_list( + ctx, "SecP384r1MLKEM1024:SecP256r1MLKEM768:X25519MLKEM768") == 1; + SSL_CTX_free(ctx); + return supported; + } + + // Handshakes with a client that verifies the server certificate against + // `ca`. Returns whether the handshake completed. + bool verifying_client_handshake(uint16_t port, const ccf::crypto::Pem& ca) + { + const int fd = ::socket(AF_INET, SOCK_STREAM, 0); + REQUIRE(fd >= 0); + + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_port = htons(port); + REQUIRE(inet_pton(AF_INET, "127.0.0.1", &addr.sin_addr) == 1); + REQUIRE( + ::connect(fd, reinterpret_cast(&addr), sizeof(addr)) == 0); + + SSL_CTX* cctx = SSL_CTX_new(TLS_client_method()); + REQUIRE(cctx != nullptr); + { + const auto ca_pem = ca.str(); + BIO* cb = BIO_new_mem_buf(ca_pem.data(), static_cast(ca_pem.size())); + X509* root = PEM_read_bio_X509(cb, nullptr, nullptr, nullptr); + BIO_free(cb); + REQUIRE(root != nullptr); + REQUIRE(X509_STORE_add_cert(SSL_CTX_get_cert_store(cctx), root) == 1); + X509_free(root); + } + SSL_CTX_set_verify(cctx, SSL_VERIFY_PEER, nullptr); + + SSL* ssl = SSL_new(cctx); + REQUIRE(ssl != nullptr); + REQUIRE(SSL_set_fd(ssl, fd) == 1); + SSL_set_connect_state(ssl); + + const bool ok = SSL_connect(ssl) == 1; + if (ok) + { + REQUIRE(SSL_get_verify_result(ssl) == X509_V_OK); + SSL_shutdown(ssl); + } + + SSL_free(ssl); + SSL_CTX_free(cctx); + ::close(fd); + return ok; + } + + // Echoes received plaintext back to the same connection via send(). + struct UVLoopRunner + { + std::thread thread; + + void start() + { + thread = std::thread([]() { uv_run(uv_default_loop(), UV_RUN_DEFAULT); }); + } + + ~UVLoopRunner() + { + if (thread.joinable()) + { + thread.join(); + } + } + }; + + struct EchoServer + { + std::shared_ptr server; + UVLoopRunner loop; + + EchoServer( + const std::string& cert, + const std::string& key, + const std::string& host = "127.0.0.1") + { + server = std::make_shared( + OpenSSLServer::Config{.host = host, .cert_pem = cert, .key_pem = key}, + [this]( + uint64_t id, + std::vector d, + const std::vector&, + bool) { server->send(id, d.data(), d.size()); }); + server->start(); + loop.start(); + } + + ~EchoServer() + { + server->stop(OpenSSLServer::LoopState::Running); + } + + uint16_t port() const + { + return server->port(); + } + }; + + // A minimal ccf::Session that echoes received bytes back through its writer, + // exercising the real Session / SessionWriter path over TLS. + struct EchoSession : public ccf::Session + { + ::tcp::ConnID id; + ccf::SessionWriter& writer; + + EchoSession(::tcp::ConnID id_, ccf::SessionWriter& w) : id(id_), writer(w) + {} + + void handle_incoming_data(std::vector&& data) override + { + writer.write_outbound(id, std::move(data)); + } + + void send_data(std::vector&& /*data*/) override {} + + void close_session() override + { + writer.close_socket(id); + } + }; + + // Writes a (large) response then immediately closes - reproduces the close- + // truncation bug: without graceful close, the buffered response is discarded. + struct LargeThenCloseSession : public ccf::Session + { + ::tcp::ConnID id; + ccf::SessionWriter& writer; + std::vector payload; + + LargeThenCloseSession( + ::tcp::ConnID id_, ccf::SessionWriter& w, std::vector p) : + id(id_), + writer(w), + payload(std::move(p)) + {} + + void handle_incoming_data(std::vector&& /*data*/) override + { + writer.write_outbound(id, std::vector(payload)); + writer.close_socket(id); + } + + void send_data(std::vector&& /*data*/) override {} + void close_session() override {} + }; +} + +// Admission control must run at accept time, not when the first request +// arrives. A client which completes the TCP and TLS handshakes and then sends +// nothing still holds a file descriptor and TLS state on the node, so it has +// to be counted - and the count has to be released exactly once when the +// connection goes away, whether or not it ever produced a session. +TEST_CASE("Connections are admitted at accept time and released once") +{ + auto [cert, key] = make_server_cert(); + + std::mutex m; + std::condition_variable cv; + std::vector<::tcp::ConnID> admitted; + std::vector<::tcp::ConnID> closed; + size_t data_callbacks = 0; + // Refuse everything after the first two connections, as a hard cap would. + constexpr size_t cap = 2; + + auto server = std::make_shared( + OpenSSLServer::Config{ + .host = "127.0.0.1", .cert_pem = cert, .key_pem = key}, + [&]( + ::tcp::ConnID, std::vector, const std::vector&, bool) { + std::lock_guard guard(m); + ++data_callbacks; + cv.notify_all(); + }, + [&](::tcp::ConnID id) { + std::lock_guard guard(m); + closed.push_back(id); + cv.notify_all(); + }, + [&](::tcp::ConnID id) -> std::optional { + std::lock_guard guard(m); + if (admitted.size() >= cap) + { + return std::nullopt; + } + admitted.push_back(id); + cv.notify_all(); + return false; + }); + UVLoopRunner loop; + server->start(); + const auto port = server->port(); + loop.start(); + + const auto connect_tls = [port](int& fd, SSL_CTX*& ctx, SSL*& ssl) { + fd = ::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0); + REQUIRE(fd >= 0); + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_port = htons(port); + REQUIRE(inet_pton(AF_INET, "127.0.0.1", &addr.sin_addr) == 1); + REQUIRE( + ::connect(fd, reinterpret_cast(&addr), sizeof(addr)) == 0); + ctx = SSL_CTX_new(TLS_client_method()); + REQUIRE(ctx != nullptr); + ssl = SSL_new(ctx); + REQUIRE(ssl != nullptr); + REQUIRE(SSL_set_fd(ssl, fd) == 1); + SSL_set_connect_state(ssl); + }; + + // Two silent clients: they complete the TLS handshake and then send nothing + // at all, so on_data is never invoked for them. + int fds[cap] = {-1, -1}; + SSL_CTX* ctxs[cap] = {nullptr, nullptr}; + SSL* ssls[cap] = {nullptr, nullptr}; + for (size_t i = 0; i < cap; ++i) + { + connect_tls(fds[i], ctxs[i], ssls[i]); + REQUIRE(SSL_connect(ssls[i]) == 1); + } + + { + std::unique_lock lock(m); + REQUIRE(cv.wait_for(lock, std::chrono::seconds(10), [&]() { + return admitted.size() == cap; + })); + // Nothing was sent, so nothing reached the session layer - which is + // precisely why counting there would have missed these connections. + REQUIRE(data_callbacks == 0); + REQUIRE(closed.empty()); + } + + // A third connection is refused at accept. It is closed without a TLS + // handshake, and must not be reported through on_close since it was never + // admitted. + { + int fd = -1; + SSL_CTX* ctx = nullptr; + SSL* ssl = nullptr; + connect_tls(fd, ctx, ssl); + REQUIRE(SSL_connect(ssl) != 1); + SSL_free(ssl); + SSL_CTX_free(ctx); + ::close(fd); + } + + // Now drop the two admitted connections; each must be released exactly once. + for (size_t i = 0; i < cap; ++i) + { + SSL_free(ssls[i]); + SSL_CTX_free(ctxs[i]); + ::close(fds[i]); + } + + { + std::unique_lock lock(m); + REQUIRE(cv.wait_for( + lock, std::chrono::seconds(10), [&]() { return closed.size() >= cap; })); + REQUIRE(closed.size() == cap); + REQUIRE( + std::set<::tcp::ConnID>(closed.begin(), closed.end()) == + std::set<::tcp::ConnID>(admitted.begin(), admitted.end())); + } + + server->stop(OpenSSLServer::LoopState::Running); +} + +// Node startup can fail after the interfaces are listening but before the +// event loop is ever entered, so stopping without a running loop has to be +// safe. The servers do not drive the loop themselves: they hand their handles +// to uv_close and return. Each handle owns itself, so the servers can then be +// destroyed immediately, and the loop reclaims the handles whenever it next +// runs - or never, for a process on its way out. +TEST_CASE("Transports stop and are destroyed before the libuv loop ever runs") +{ + auto [cert, key] = make_server_cert(); + { + auto tcp_server = std::make_shared( + OpenSSLServer::Config{ + .host = "127.0.0.1", .cert_pem = cert, .key_pem = key}, + []( + ::tcp::ConnID, + std::vector, + const std::vector&, + bool) {}); + tcp_server->start(); + tcp_server->stop(); + + DatagramServer udp_server( + "127.0.0.1", + 0, + [](const uint8_t*, size_t, const sockaddr_storage&, socklen_t) {}); + udp_server.start(); + udp_server.stop(); + + // Both servers go out of scope here, while their handles are still + // registered with the loop and awaiting their close callbacks. + } + + // Which the loop can now run without touching the destroyed servers. This + // mirrors the drain run.cpp performs after its event loop exits. + constexpr size_t max_iterations = 100; + size_t iterations = 0; + while (uv_loop_alive(uv_default_loop()) != 0 && iterations < max_iterations) + { + uv_run(uv_default_loop(), UV_RUN_NOWAIT); + ++iterations; + } + REQUIRE(uv_loop_alive(uv_default_loop()) == 0); +} + +TEST_CASE("Transport shutdown drains TLS tasks with no background workers") +{ + struct RestoreWorkers + { + ~RestoreWorkers() + { + ccf::tasks::set_task_threads(4); + } + } restore_workers; + ccf::tasks::set_task_threads(0); + + auto [cert, key] = make_server_cert(); + auto server = std::make_shared( + OpenSSLServer::Config{ + .host = "127.0.0.1", .cert_pem = cert, .key_pem = key}, + [](::tcp::ConnID, std::vector, const std::vector&, bool) { + }); + UVLoopRunner loop; + server->start(); + loop.start(); + + const int fd = ::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0); + REQUIRE(fd >= 0); + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_port = htons(server->port()); + REQUIRE(inet_pton(AF_INET, "127.0.0.1", &addr.sin_addr) == 1); + REQUIRE(::connect(fd, reinterpret_cast(&addr), sizeof(addr)) == 0); + const uint8_t byte = 0; + REQUIRE(::send(fd, &byte, sizeof(byte), MSG_NOSIGNAL) == sizeof(byte)); + + const auto deadline = + std::chrono::steady_clock::now() + std::chrono::seconds(5); + while (ccf::tasks::get_main_job_board().get_summary().pending_tasks == 0 && + std::chrono::steady_clock::now() < deadline) + { + std::this_thread::yield(); + } + REQUIRE(ccf::tasks::get_main_job_board().get_summary().pending_tasks > 0); + + server->stop(OpenSSLServer::LoopState::Running); + ::close(fd); +} + +// Shutdown must not declare itself complete on a transient lull in the +// pending-close count. Connections close one at a time, and libuv runs close +// callbacks at the end of each loop iteration, so a connection which tears +// down promptly can look like the last one while others are still open. +// Reporting "stopped" there lets the caller destroy the server while the loop +// is still going to use its handles. +// +// The stagger is produced by holding some connections' workers inside on_data +// while one connection is left idle and therefore closes immediately. +TEST_CASE("Shutdown with staggered connection closes releases every uv handle") +{ + auto [cert, key] = make_server_cert(); + + constexpr size_t num_conns = 3; + constexpr size_t num_blocked = 2; + + std::mutex m; + std::condition_variable cv; + std::map<::tcp::ConnID, bool> block_decision; + size_t arrived = 0; + bool release = false; + + auto server = std::make_shared( + OpenSSLServer::Config{ + .host = "127.0.0.1", + .cert_pem = cert, + .key_pem = key, + // Configure an idle timeout, so the timer handle is in play as well. + .idle_timeout = std::chrono::milliseconds(60000)}, + [&]( + ::tcp::ConnID id, + std::vector, + const std::vector&, + bool) { + bool should_block = false; + { + std::unique_lock lock(m); + auto it = block_decision.find(id); + if (it == block_decision.end()) + { + it = block_decision.emplace(id, block_decision.size() < num_blocked) + .first; + ++arrived; + cv.notify_all(); + } + should_block = it->second; + if (should_block) + { + cv.wait(lock, [&]() { return release; }); + } + } + }); + UVLoopRunner loop; + server->start(); + const auto port = server->port(); + loop.start(); + + std::vector clients; + clients.reserve(num_conns); + for (size_t i = 0; i < num_conns; ++i) + { + clients.emplace_back([port, &m, &cv, &release]() { + const int fd = ::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0); + REQUIRE(fd >= 0); + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_port = htons(port); + REQUIRE(inet_pton(AF_INET, "127.0.0.1", &addr.sin_addr) == 1); + REQUIRE( + ::connect(fd, reinterpret_cast(&addr), sizeof(addr)) == 0); + + SSL_CTX* cctx = SSL_CTX_new(TLS_client_method()); + REQUIRE(cctx != nullptr); + SSL* ssl = SSL_new(cctx); + REQUIRE(ssl != nullptr); + REQUIRE(SSL_set_fd(ssl, fd) == 1); + SSL_set_connect_state(ssl); + REQUIRE(SSL_connect(ssl) == 1); + const uint8_t byte = 'x'; + REQUIRE(SSL_write(ssl, &byte, 1) == 1); + + // Hold the connection open until the blocked workers are released, so + // that shutdown genuinely has several live connections to tear down. + { + std::unique_lock lock(m); + cv.wait(lock, [&]() { return release; }); + } + SSL_free(ssl); + SSL_CTX_free(cctx); + ::close(fd); + }); + } + + { + std::unique_lock lock(m); + REQUIRE(cv.wait_for( + lock, std::chrono::seconds(10), [&]() { return arrived == num_conns; })); + } + + // Release the held workers only after shutdown has had time to close the + // one connection which is not blocked, which is the window in which the + // pending-close count transiently reaches zero. + std::thread releaser([&]() { + std::this_thread::sleep_for(std::chrono::milliseconds(250)); + std::lock_guard guard(m); + release = true; + cv.notify_all(); + }); + + server->stop(OpenSSLServer::LoopState::Running); + + releaser.join(); + for (auto& t : clients) + { + t.join(); + } + + // stop() has returned, so nothing refers to the handles any more and the + // server can be destroyed even though the loop has not yet reclaimed them. + server.reset(); + + // The loop then finishes of its own accord, which it can only do once every + // handle has been closed - so if any had been missed, or if a close callback + // touched the destroyed server, this would hang or crash rather than pass. + loop.thread.join(); + REQUIRE(uv_loop_alive(uv_default_loop()) == 0); +} + +// Shutdown must not be hostage to a peer which has stopped reading. A +// connection with output it cannot flush is closed after a single further +// pass, rather than being re-dispatched until the socket becomes writable - +// which, for a client advertising a zero window, is never. +TEST_CASE("Shutdown completes while a connection cannot flush its output") +{ + auto [cert, key] = make_server_cert(); + + // Far larger than any socket buffer, so the write is guaranteed to stall + // with most of the payload still queued. + constexpr size_t payload_size = 32 * 1024 * 1024; + const std::vector payload(payload_size, 'z'); + + std::shared_ptr server; + server = std::make_shared( + OpenSSLServer::Config{ + .host = "127.0.0.1", .cert_pem = cert, .key_pem = key}, + [&]( + ::tcp::ConnID id, + std::vector, + const std::vector&, + bool) { server->send(id, std::vector(payload)); }); + UVLoopRunner loop; + server->start(); + const auto port = server->port(); + loop.start(); + + const int fd = ::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0); + REQUIRE(fd >= 0); + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_port = htons(port); + REQUIRE(inet_pton(AF_INET, "127.0.0.1", &addr.sin_addr) == 1); + REQUIRE(::connect(fd, reinterpret_cast(&addr), sizeof(addr)) == 0); + + SSL_CTX* cctx = SSL_CTX_new(TLS_client_method()); + REQUIRE(cctx != nullptr); + SSL* ssl = SSL_new(cctx); + REQUIRE(ssl != nullptr); + REQUIRE(SSL_set_fd(ssl, fd) == 1); + SSL_set_connect_state(ssl); + REQUIRE(SSL_connect(ssl) == 1); + + const uint8_t request = 'x'; + REQUIRE(SSL_write(ssl, &request, 1) == 1); + + // Read just enough to know the response has started, then stop reading. The + // server's remaining output has nowhere to go from here on. + std::vector chunk(1024); + REQUIRE(SSL_read(ssl, chunk.data(), static_cast(chunk.size())) > 0); + + auto stopped = std::async(std::launch::async, [&]() { + server->stop(OpenSSLServer::LoopState::Running); + }); + const bool completed = + stopped.wait_for(std::chrono::seconds(10)) == std::future_status::ready; + + // Unblock the server if it did not stop, so that the failure is reported + // rather than hanging the test binary. + SSL_free(ssl); + SSL_CTX_free(cctx); + ::close(fd); + + stopped.get(); + REQUIRE(completed); + + server.reset(); + loop.thread.join(); + REQUIRE(uv_loop_alive(uv_default_loop()) == 0); +} + +// An interface binds before its certificate is necessarily known - a joining +// node only receives the service certificate once its join has been accepted. +// Until then it must refuse connections outright rather than accept them and +// drop them, so that a client fails to connect (and retries) rather than +// completing a TCP handshake and then failing mid-TLS. +TEST_CASE("A TLS interface without a certificate refuses connections") +{ + auto [cert, key] = make_server_cert(); + + std::mutex m; + std::condition_variable cv; + size_t data_callbacks = 0; + + // No cert_pem/key_pem: the certificate arrives later, via set_server_cert. + std::shared_ptr server; + server = std::make_shared( + OpenSSLServer::Config{.host = "127.0.0.1"}, + [&]( + ::tcp::ConnID id, + std::vector d, + const std::vector&, + bool) { + server->send(id, std::move(d)); + std::lock_guard guard(m); + ++data_callbacks; + cv.notify_all(); + }); + UVLoopRunner loop; + server->start(); + + // The port is fixed by the bind, so it is reportable even before the + // interface is serving. + const auto port = server->port(); + REQUIRE(port != 0); + loop.start(); + + const auto try_connect = [port]() { + const int fd = ::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0); + REQUIRE(fd >= 0); + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_port = htons(port); + REQUIRE(inet_pton(AF_INET, "127.0.0.1", &addr.sin_addr) == 1); + const int rc = + ::connect(fd, reinterpret_cast(&addr), sizeof(addr)); + const int err = errno; + ::close(fd); + return rc == 0 ? 0 : err; + }; + + // Bound but not listening, so the kernel answers the SYN with a reset. + REQUIRE(try_connect() == ECONNREFUSED); + + // Supplying the certificate is what puts the interface into service. + server->set_server_cert(cert, key); + + const auto deadline = + std::chrono::steady_clock::now() + std::chrono::seconds(10); + while (try_connect() != 0 && std::chrono::steady_clock::now() < deadline) + { + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + } + REQUIRE(try_connect() == 0); + + // And the interface now serves a complete TLS session. + { + const int fd = ::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0); + REQUIRE(fd >= 0); + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_port = htons(port); + REQUIRE(inet_pton(AF_INET, "127.0.0.1", &addr.sin_addr) == 1); + REQUIRE( + ::connect(fd, reinterpret_cast(&addr), sizeof(addr)) == 0); + + SSL_CTX* cctx = SSL_CTX_new(TLS_client_method()); + REQUIRE(cctx != nullptr); + SSL* ssl = SSL_new(cctx); + REQUIRE(ssl != nullptr); + REQUIRE(SSL_set_fd(ssl, fd) == 1); + SSL_set_connect_state(ssl); + REQUIRE(SSL_connect(ssl) == 1); + + const std::string request = "hello"; + REQUIRE( + SSL_write(ssl, request.data(), static_cast(request.size())) == + static_cast(request.size())); + + std::vector echoed(request.size()); + REQUIRE( + SSL_read(ssl, echoed.data(), static_cast(echoed.size())) == + static_cast(request.size())); + REQUIRE(std::string(echoed.begin(), echoed.end()) == request); + + SSL_free(ssl); + SSL_CTX_free(cctx); + ::close(fd); + } + + { + std::unique_lock lock(m); + REQUIRE(cv.wait_for( + lock, std::chrono::seconds(10), [&]() { return data_callbacks > 0; })); + } + + server->stop(OpenSSLServer::LoopState::Running); + server.reset(); + loop.thread.join(); + REQUIRE(uv_loop_alive(uv_default_loop()) == 0); +} + +// Idle connections are closed so that a client which connects, handshakes and +// then goes quiet cannot hold a file descriptor and TLS state indefinitely. +TEST_CASE("Idle connections are closed after the configured timeout") +{ + auto [cert, key] = make_server_cert(); + + std::mutex m; + std::condition_variable cv; + std::set<::tcp::ConnID> opened; + std::set<::tcp::ConnID> closed; + + // Shorter than the sweep interval, so the first sweep after the connection + // goes quiet closes it. + std::shared_ptr server; + server = std::make_shared( + OpenSSLServer::Config{ + .host = "127.0.0.1", + .cert_pem = cert, + .key_pem = key, + .idle_timeout = std::chrono::milliseconds(100)}, + [&]( + ::tcp::ConnID id, + std::vector d, + const std::vector&, + bool) { + server->send(id, std::move(d)); + std::lock_guard guard(m); + opened.insert(id); + cv.notify_all(); + }, + [&](::tcp::ConnID id) { + std::lock_guard guard(m); + closed.insert(id); + cv.notify_all(); + }); + UVLoopRunner loop; + server->start(); + const auto port = server->port(); + loop.start(); + + const int fd = ::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0); + REQUIRE(fd >= 0); + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_port = htons(port); + REQUIRE(inet_pton(AF_INET, "127.0.0.1", &addr.sin_addr) == 1); + REQUIRE(::connect(fd, reinterpret_cast(&addr), sizeof(addr)) == 0); + + SSL_CTX* cctx = SSL_CTX_new(TLS_client_method()); + REQUIRE(cctx != nullptr); + SSL* ssl = SSL_new(cctx); + REQUIRE(ssl != nullptr); + REQUIRE(SSL_set_fd(ssl, fd) == 1); + SSL_set_connect_state(ssl); + REQUIRE(SSL_connect(ssl) == 1); + + const uint8_t request = 'x'; + REQUIRE(SSL_write(ssl, &request, 1) == 1); + uint8_t echoed = 0; + REQUIRE(SSL_read(ssl, &echoed, 1) == 1); + + // The connection is established and served. Now go quiet, and the server + // should close it of its own accord. + { + std::unique_lock lock(m); + REQUIRE(cv.wait_for( + lock, std::chrono::seconds(10), [&]() { return !closed.empty(); })); + REQUIRE(closed == opened); + } + + // The client observes the close rather than the connection simply stalling. + uint8_t after = 0; + REQUIRE(SSL_read(ssl, &after, 1) <= 0); + + SSL_free(ssl); + SSL_CTX_free(cctx); + ::close(fd); + + server->stop(OpenSSLServer::LoopState::Running); + server.reset(); + loop.thread.join(); + REQUIRE(uv_loop_alive(uv_default_loop()) == 0); +} + +TEST_CASE("TCP connections use the legacy latency and keepalive options") +{ + const int fd = ::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0); + REQUIRE(fd >= 0); + REQUIRE_FALSE(asynchost::details::configure_tcp_connection(fd).has_value()); + + const auto get_option = [fd](int level, int option) { + int value = 0; + socklen_t value_size = sizeof(value); + REQUIRE(getsockopt(fd, level, option, &value, &value_size) == 0); + REQUIRE(value_size == sizeof(value)); + return value; + }; + + REQUIRE(get_option(IPPROTO_TCP, TCP_NODELAY) == 1); + REQUIRE(get_option(SOL_SOCKET, SO_KEEPALIVE) == 1); + REQUIRE(get_option(IPPROTO_TCP, TCP_KEEPIDLE) == 30); + REQUIRE(get_option(IPPROTO_TCP, TCP_KEEPINTVL) == 1); + REQUIRE(get_option(IPPROTO_TCP, TCP_KEEPCNT) == 10); + REQUIRE((fcntl(fd, F_GETFD) & FD_CLOEXEC) != 0); + + ::close(fd); + + const auto error = asynchost::details::configure_tcp_connection(-1); + REQUIRE(error.has_value()); + REQUIRE(std::string(error->option) == "TCP_NODELAY"); + REQUIRE(error->error == EBADF); +} + +TEST_CASE("TLS handshake and small round-trip") +{ + auto [cert, key] = make_server_cert(); + EchoServer s(cert, key); + REQUIRE(s.port() != 0); + + const std::vector msg = {'h', 'e', 'l', 'l', 'o'}; + REQUIRE(tls_client_exchange(s.port(), msg, msg.size()) == msg); +} + +TEST_CASE("Coalesced pipelined HTTP responses are preserved") +{ + auto [cert, key] = make_server_cert(); + + std::shared_ptr server; + server = std::make_shared( + OpenSSLServer::Config{ + .host = "127.0.0.1", .cert_pem = cert, .key_pem = key}, + [&]( + ::tcp::ConnID id, + std::vector, + const std::vector&, + bool) { + const std::string responses = + "HTTP/1.1 200 OK\r\nContent-Length: 1\r\n\r\na" + "HTTP/1.1 201 Created\r\nContent-Length: 1\r\n\r\nb"; + server->send( + id, std::vector(responses.begin(), responses.end())); + }); + UVLoopRunner loop; + server->start(); + loop.start(); + + { + auto ca = std::make_shared<::tls::CA>(cert); + client::RpcTlsClient client( + "127.0.0.1", std::to_string(server->port()), ca); + const std::array request = {'x'}; + client.write(request); + + const auto first = client.read_response(); + const auto second = client.read_response(); + REQUIRE(first.status == HTTP_STATUS_OK); + REQUIRE(std::string(first.body.begin(), first.body.end()) == "a"); + REQUIRE(second.status == HTTP_STATUS_CREATED); + REQUIRE(std::string(second.body.begin(), second.body.end()) == "b"); + } + + server->stop(OpenSSLServer::LoopState::Running); +} + +TEST_CASE("TLS processing runs off the libuv thread") +{ + auto [cert, key] = make_server_cert(); + std::mutex callback_mutex; + std::thread::id callback_thread; + OpenSSLServer* server_ptr = nullptr; + auto server = std::make_shared( + OpenSSLServer::Config{ + .host = "127.0.0.1", .cert_pem = cert, .key_pem = key}, + [&]( + ::tcp::ConnID id, + std::vector data, + const std::vector&, + bool) { + { + std::lock_guard guard(callback_mutex); + callback_thread = std::this_thread::get_id(); + } + server_ptr->send(id, data.data(), data.size()); + }); + UVLoopRunner loop; + server_ptr = server.get(); + server->start(); + loop.start(); + + const std::vector msg = {'w', 'o', 'r', 'k', 'e', 'r'}; + REQUIRE(tls_client_exchange(server->port(), msg, msg.size()) == msg); + { + std::lock_guard guard(callback_mutex); + REQUIRE(callback_thread != std::thread::id{}); + REQUIRE(callback_thread != loop.thread.get_id()); + } + + server->stop(OpenSSLServer::LoopState::Running); +} + +TEST_CASE("Large transfer exercises the backpressure path") +{ + auto [cert, key] = make_server_cert(); + EchoServer s(cert, key); + + // 4 MiB forces the socket send buffer to fill, so SSL_write returns + // WANT_WRITE and the server must buffer + re-arm UV_WRITABLE. + const auto payload = random_bytes(4 * 1024 * 1024); + const auto resp = tls_client_exchange(s.port(), payload, payload.size()); + + REQUIRE(resp.size() == payload.size()); + REQUIRE(resp == payload); +} + +// A read pass stops after max_read_per_event (64KiB), so a client which sends +// far more than that in one go and then goes silent must still have all of it +// delivered - either because the socket is still readable, or because the +// server noticed OpenSSL's own buffered data (SSL_has_pending) and scheduled +// another pass. +// +// NB: with OpenSSL's default TLS settings (read_ahead off, one record per +// socket read) the second case is hard to provoke deliberately, so this test +// does not by itself prove the SSL_has_pending() continuation is load-bearing. +// It covers the multi-pass read path, which is the part that regresses easily. +TEST_CASE("A single large write past the per-pass read cap is fully delivered") +{ + auto [cert, key] = make_server_cert(); + + std::mutex m; + std::condition_variable cv; + size_t received = 0; + + auto server = std::make_shared( + OpenSSLServer::Config{ + .host = "127.0.0.1", .cert_pem = cert, .key_pem = key}, + [&]( + ::tcp::ConnID, + std::vector data, + const std::vector&, + bool) { + std::lock_guard guard(m); + received += data.size(); + cv.notify_all(); + }); + UVLoopRunner loop; + server->start(); + loop.start(); + + // Comfortably more than the 64KiB per-pass cap, written in one go and + // followed by no further traffic at all. + const size_t payload_size = 512 * 1024; + const auto payload = random_bytes(payload_size); + + std::thread client([&]() { + const int fd = ::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0); + REQUIRE(fd >= 0); + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_port = htons(server->port()); + REQUIRE(inet_pton(AF_INET, "127.0.0.1", &addr.sin_addr) == 1); + REQUIRE( + ::connect(fd, reinterpret_cast(&addr), sizeof(addr)) == 0); + + SSL_CTX* cctx = SSL_CTX_new(TLS_client_method()); + REQUIRE(cctx != nullptr); + SSL* ssl = SSL_new(cctx); + REQUIRE(ssl != nullptr); + REQUIRE(SSL_set_fd(ssl, fd) == 1); + SSL_set_connect_state(ssl); + REQUIRE(SSL_connect(ssl) == 1); + + size_t off = 0; + while (off < payload.size()) + { + const int n = SSL_write( + ssl, payload.data() + off, static_cast(payload.size() - off)); + REQUIRE(n > 0); + off += static_cast(n); + } + + // Deliberately send nothing more, and hold the connection open, so the + // only way the server can see the rest is by noticing its own buffered + // data rather than waiting for readability. + { + std::unique_lock lock(m); + cv.wait_for(lock, std::chrono::seconds(10), [&]() { + return received >= payload_size; + }); + } + + SSL_free(ssl); + SSL_CTX_free(cctx); + ::close(fd); + }); + + { + std::unique_lock lock(m); + REQUIRE(cv.wait_for(lock, std::chrono::seconds(10), [&]() { + return received >= payload_size; + })); + REQUIRE(received == payload_size); + } + + client.join(); + server->stop(OpenSSLServer::LoopState::Running); +} + +TEST_CASE("Concurrent connections") +{ + auto [cert, key] = make_server_cert(); + EchoServer s(cert, key); + const uint16_t port = s.port(); + + constexpr int num_clients = 16; + std::vector clients; + std::atomic ok{0}; + clients.reserve(num_clients); + for (int i = 0; i < num_clients; ++i) + { + clients.emplace_back([port, i, &ok]() { + const std::vector msg(64, static_cast('A' + (i % 26))); + const auto resp = tls_client_exchange(port, msg, msg.size()); + if (resp == msg) + { + ok.fetch_add(1); + } + }); + } + for (auto& t : clients) + { + t.join(); + } + + REQUIRE(ok.load() == num_clients); +} + +// Models the production dispatch path: the libuv thread hands the request to a +// worker thread, which replies via send() - exercising cross-thread send + +// uv_async_t loop wakeup. +TEST_CASE("Reply from a worker thread") +{ + auto [cert, key] = make_server_cert(); + + OpenSSLServer* sp = nullptr; + std::mutex m; + std::condition_variable cv; + std::deque>> q; + std::atomic stop{false}; + + std::thread worker([&]() { + for (;;) + { + std::unique_lock l(m); + cv.wait(l, [&]() { return stop.load() || !q.empty(); }); + if (stop.load() && q.empty()) + { + return; + } + auto item = std::move(q.front()); + q.pop_front(); + l.unlock(); + sp->send(item.first, item.second.data(), item.second.size()); + } + }); + + auto server = std::make_shared( + OpenSSLServer::Config{ + .host = "127.0.0.1", .cert_pem = cert, .key_pem = key}, + [&]( + uint64_t id, std::vector d, const std::vector&, bool) { + { + std::lock_guard l(m); + q.emplace_back(id, std::move(d)); + } + cv.notify_one(); + }); + UVLoopRunner loop; + sp = server.get(); + server->start(); + loop.start(); + + const std::vector msg = {'w', 'o', 'r', 'k', 'e', 'r'}; + REQUIRE(tls_client_exchange(server->port(), msg, msg.size()) == msg); + + server->stop(OpenSSLServer::LoopState::Running); + { + std::lock_guard l(m); + stop.store(true); + } + cv.notify_one(); + worker.join(); +} + +TEST_CASE("Datagram server round-trip on the libuv reactor") +{ + DatagramServer* server_ptr = nullptr; + DatagramServer server( + "127.0.0.1", + 0, + [&]( + const uint8_t* data, + size_t len, + const sockaddr_storage& peer, + socklen_t peerlen) { + REQUIRE(server_ptr->send_to(peer, peerlen, data, len)); + }); + UVLoopRunner loop; + server_ptr = &server; + server.start(); + loop.start(); + + const int fd = ::socket(AF_INET, SOCK_DGRAM, 0); + REQUIRE(fd >= 0); + timeval timeout{1, 0}; + REQUIRE( + setsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, &timeout, sizeof(timeout)) == 0); + sockaddr_in address{}; + address.sin_family = AF_INET; + address.sin_port = htons(server.port()); + REQUIRE(inet_pton(AF_INET, "127.0.0.1", &address.sin_addr) == 1); + + const std::vector message = {'u', 'd', 'p'}; + REQUIRE( + ::sendto( + fd, + message.data(), + message.size(), + 0, + reinterpret_cast(&address), + sizeof(address)) == static_cast(message.size())); + std::vector response(message.size()); + REQUIRE(::recv(fd, response.data(), response.size(), 0) == 3); + REQUIRE(response == message); + ::close(fd); + + server.stop(DatagramServer::LoopState::Running); +} + +// The datagram handler runs inline on the libuv thread and, in the real +// manager, takes the same lock that shutdown needs. Shutting down while +// datagrams are still arriving must therefore not deadlock: shutdown waits on +// the loop, and the loop must never end up waiting on shutdown. +TEST_CASE("Datagram server stops cleanly while datagrams are still arriving") +{ + std::atomic handled{0}; + std::mutex handler_mutex; + + DatagramServer* server_ptr = nullptr; + DatagramServer server( + "127.0.0.1", + 0, + [&]( + const uint8_t* data, + size_t len, + const sockaddr_storage& peer, + socklen_t peerlen) { + // Stands in for RPCConnectionManager's interfaces_mutex, which the + // datagram path takes on the loop thread while stop() may be running on + // another thread. + std::lock_guard guard(handler_mutex); + handled.fetch_add(1); + (void)server_ptr->send_to(peer, peerlen, data, len); + }); + UVLoopRunner loop; + server_ptr = &server; + server.start(); + const auto port = server.port(); + loop.start(); + + std::atomic sending{true}; + std::thread flooder([port, &sending]() { + const int fd = ::socket(AF_INET, SOCK_DGRAM, 0); + REQUIRE(fd >= 0); + sockaddr_in address{}; + address.sin_family = AF_INET; + address.sin_port = htons(port); + REQUIRE(inet_pton(AF_INET, "127.0.0.1", &address.sin_addr) == 1); + const std::vector message = {'u', 'd', 'p'}; + while (sending.load()) + { + (void)::sendto( + fd, + message.data(), + message.size(), + 0, + reinterpret_cast(&address), + sizeof(address)); + } + ::close(fd); + }); + + const auto deadline = + std::chrono::steady_clock::now() + std::chrono::seconds(10); + while (handled.load() == 0 && std::chrono::steady_clock::now() < deadline) + { + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + } + REQUIRE(handled.load() > 0); + + // Stops while the flood is still in flight. If this deadlocks, the test + // hangs rather than failing, which is the intended signal. + server.stop(DatagramServer::LoopState::Running); + + sending.store(false); + flooder.join(); + + REQUIRE(uv_loop_alive(uv_default_loop()) == 0); +} + +TEST_CASE("Session bridge: round-trip via ccf::Session + SessionWriter") +{ + auto [cert, key] = make_server_cert(); + OpenSSLSessionManager mgr( + {.host = "127.0.0.1", .cert_pem = cert, .key_pem = key}, + [](::tcp::ConnID id, ccf::SessionWriter& w, std::vector, bool) { + return std::make_shared(id, w); + }); + UVLoopRunner loop; + mgr.start(); + loop.start(); + REQUIRE(mgr.port() != 0); + + const std::vector msg = {'b', 'r', 'i', 'd', 'g', 'e'}; + REQUIRE(tls_client_exchange(mgr.port(), msg, msg.size()) == msg); + + mgr.stop(OpenSSLServer::LoopState::Running); +} + +TEST_CASE("Session bridge: large transfer via ccf::Session + SessionWriter") +{ + auto [cert, key] = make_server_cert(); + OpenSSLSessionManager mgr( + {.host = "127.0.0.1", .cert_pem = cert, .key_pem = key}, + [](::tcp::ConnID id, ccf::SessionWriter& w, std::vector, bool) { + return std::make_shared(id, w); + }); + UVLoopRunner loop; + mgr.start(); + loop.start(); + + const auto payload = random_bytes(2 * 1024 * 1024); + const auto resp = tls_client_exchange(mgr.port(), payload, payload.size()); + REQUIRE(resp == payload); + + mgr.stop(OpenSSLServer::LoopState::Running); +} + +// The server must request the client certificate during the handshake so it is +// available for application-level caller authentication (user/member cert +// auth). Verifies the cert presented by the client reaches the session factory. +TEST_CASE("Peer certificate is captured for inbound connections") +{ + auto [cert, key] = make_server_cert(); + auto [client_cert, client_key] = make_server_cert(); + + std::mutex m; + std::vector captured; + std::atomic got{false}; + + OpenSSLSessionManager mgr( + {.host = "127.0.0.1", .cert_pem = cert, .key_pem = key}, + [&]( + ::tcp::ConnID id, ccf::SessionWriter& w, std::vector pc, bool) { + { + std::lock_guard l(m); + captured = std::move(pc); + } + got.store(true); + return std::make_shared(id, w); + }); + UVLoopRunner loop; + mgr.start(); + loop.start(); + + const std::vector msg = {'m', 't', 'l', 's'}; + REQUIRE( + tls_client_exchange(mgr.port(), msg, msg.size(), client_cert, client_key) == + msg); + + REQUIRE(got.load()); + std::lock_guard l(m); + REQUIRE(!captured.empty()); + + mgr.stop(OpenSSLServer::LoopState::Running); +} + +// The server deliberately does not enforce client certificate validity: it +// requests one and hands whatever arrives to the application, which decides. +// A client presenting no certificate at all must therefore still connect. +TEST_CASE("Client certificate is requested but not enforced") +{ + auto [cert, key] = make_server_cert(); + + std::mutex m; + std::vector captured; + std::atomic got{false}; + + OpenSSLSessionManager mgr( + {.host = "127.0.0.1", .cert_pem = cert, .key_pem = key}, + [&]( + ::tcp::ConnID id, ccf::SessionWriter& w, std::vector pc, bool) { + { + std::lock_guard l(m); + captured = std::move(pc); + } + got.store(true); + return std::make_shared(id, w); + }); + UVLoopRunner loop; + mgr.start(); + loop.start(); + + const std::vector msg = {'n', 'o', 'c', 'e', 'r', 't'}; + REQUIRE(tls_client_exchange(mgr.port(), msg, msg.size()) == msg); + + REQUIRE(got.load()); + std::lock_guard l(m); + REQUIRE(captured.empty()); + + mgr.stop(OpenSSLServer::LoopState::Running); +} + +// The server certificate must be verifiable by a client that trusts the CA +// which endorsed it, and rejected by one that does not. +TEST_CASE("Server certificate is verified by the client") +{ + auto ca = make_ca(); + + SUBCASE("a client trusting the endorsing CA completes the handshake") + { + auto [cert, key] = make_endorsed_server_cert(ca); + EchoServer s(cert, key); + REQUIRE(verifying_client_handshake(s.port(), ca.cert)); + } + + SUBCASE("a client trusting a different CA rejects the server") + { + auto [cert, key] = make_endorsed_server_cert(ca); + EchoServer s(cert, key); + REQUIRE_FALSE(verifying_client_handshake(s.port(), make_ca().cert)); + } + + SUBCASE("a verifying client rejects a self-signed server certificate") + { + auto [cert, key] = make_server_cert(); + EchoServer s(cert, key); + REQUIRE_FALSE(verifying_client_handshake(s.port(), ca.cert)); + } +} + +namespace +{ + // Connect to host:port (resolved via getaddrinfo, any family), TLS + // round-trip. + std::vector tls_echo_roundtrip( + const std::string& host, uint16_t port, const std::vector& msg) + { + addrinfo hints{}; + hints.ai_socktype = SOCK_STREAM; + addrinfo* res = nullptr; + REQUIRE( + getaddrinfo(host.c_str(), std::to_string(port).c_str(), &hints, &res) == + 0); + int fd = -1; + for (addrinfo* ai = res; ai != nullptr; ai = ai->ai_next) + { + fd = ::socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol); + if (fd < 0) + { + continue; + } + if (::connect(fd, ai->ai_addr, ai->ai_addrlen) == 0) + { + break; + } + ::close(fd); + fd = -1; + } + freeaddrinfo(res); + REQUIRE(fd >= 0); + + SSL_CTX* cctx = SSL_CTX_new(TLS_client_method()); + REQUIRE(cctx != nullptr); + SSL* ssl = SSL_new(cctx); + REQUIRE(SSL_set_fd(ssl, fd) == 1); + SSL_set_connect_state(ssl); + REQUIRE(SSL_connect(ssl) == 1); + + size_t off = 0; + while (off < msg.size()) + { + const int n = + SSL_write(ssl, msg.data() + off, static_cast(msg.size() - off)); + REQUIRE(n > 0); + off += static_cast(n); + } + + std::vector resp(msg.size()); + size_t roff = 0; + while (roff < resp.size()) + { + const int n = + SSL_read(ssl, resp.data() + roff, static_cast(resp.size() - roff)); + if (n <= 0) + { + break; + } + roff += static_cast(n); + } + + SSL_shutdown(ssl); + SSL_free(ssl); + SSL_CTX_free(cctx); + ::close(fd); + return resp; + } +} + +TEST_CASE("Listener binds a hostname (localhost)") +{ + auto [cert, key] = make_server_cert(); + EchoServer s(cert, key, "localhost"); + REQUIRE(s.port() != 0); + + const std::vector msg = {'l', 'o', 'c', 'a', 'l'}; + REQUIRE(tls_echo_roundtrip("localhost", s.port(), msg) == msg); +} + +TEST_CASE("Listener binds IPv6 loopback when available") +{ + auto [cert, key] = make_server_cert(); + std::unique_ptr s; + try + { + s = std::make_unique(cert, key, "::1"); + } + catch (const std::exception&) + { + MESSAGE("IPv6 loopback unavailable in this environment - skipping"); + return; + } + + const std::vector msg = {'v', '6'}; + REQUIRE(tls_echo_roundtrip("::1", s->port(), msg) == msg); +} + +TEST_CASE("Graceful close flushes buffered response without truncation") +{ + auto [cert, key] = make_server_cert(); + const auto payload = random_bytes(4 * 1024 * 1024); + + OpenSSLSessionManager mgr( + {.host = "127.0.0.1", .cert_pem = cert, .key_pem = key}, + [&payload]( + ::tcp::ConnID id, ccf::SessionWriter& w, std::vector, bool) { + return std::make_shared(id, w, payload); + }); + UVLoopRunner loop; + mgr.start(); + loop.start(); + + const std::vector req = {'g', 'o'}; + const auto resp = tls_client_exchange(mgr.port(), req, payload.size()); + REQUIRE(resp.size() == payload.size()); + REQUIRE(resp == payload); + + mgr.stop(OpenSSLServer::LoopState::Running); +} + +// Multiple sequential requests on a single kept-alive TLS connection - the node +// must not drop the connection between requests (regression for the e2e +// "Server disconnected" after a few requests on an idle keep-alive connection). +TEST_CASE("Persistent connection survives many sequential round-trips") +{ + auto [cert, key] = make_server_cert(); + EchoServer s(cert, key); + REQUIRE(s.port() != 0); + + const int fd = ::socket(AF_INET, SOCK_STREAM, 0); + REQUIRE(fd >= 0); + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_port = htons(s.port()); + REQUIRE(inet_pton(AF_INET, "127.0.0.1", &addr.sin_addr) == 1); + REQUIRE(::connect(fd, reinterpret_cast(&addr), sizeof(addr)) == 0); + + SSL_CTX* cctx = SSL_CTX_new(TLS_client_method()); + SSL* ssl = SSL_new(cctx); + REQUIRE(SSL_set_fd(ssl, fd) == 1); + SSL_set_connect_state(ssl); + REQUIRE(SSL_connect(ssl) == 1); + + for (int i = 0; i < 10; ++i) + { + const std::vector msg = { + 'r', static_cast('0' + (i % 10))}; + REQUIRE(SSL_write(ssl, msg.data(), static_cast(msg.size())) == 2); + + std::vector resp(msg.size()); + size_t off = 0; + while (off < resp.size()) + { + const int n = + SSL_read(ssl, resp.data() + off, static_cast(resp.size() - off)); + REQUIRE(n > 0); + off += static_cast(n); + } + REQUIRE(resp == msg); + + // Idle a moment between requests, as the e2e client does (sleep(0.5)). + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + + SSL_shutdown(ssl); + SSL_free(ssl); + SSL_CTX_free(cctx); + ::close(fd); +} + +// The server's cipher, ciphersuite and group policy is defined in +// build_server_ctx(). These assert it from the wire. + +TEST_CASE("Server restricts TLS 1.3 ciphersuites to the configured list") +{ + auto [cert, key] = make_server_cert(); + EchoServer s(cert, key); + + SUBCASE("a configured ciphersuite is accepted") + { + const auto r = + handshake_and_inspect(s.port(), {}, "TLS_AES_256_GCM_SHA384"); + REQUIRE(r.succeeded); + REQUIRE(r.cipher == "TLS_AES_256_GCM_SHA384"); + } + + SUBCASE("the other configured ciphersuite is accepted") + { + const auto r = + handshake_and_inspect(s.port(), {}, "TLS_AES_128_GCM_SHA256"); + REQUIRE(r.succeeded); + REQUIRE(r.cipher == "TLS_AES_128_GCM_SHA256"); + } + + SUBCASE("an unconfigured ciphersuite is refused") + { + // ChaCha20-Poly1305 is a valid TLS 1.3 ciphersuite that CCF does not offer. + const auto r = + handshake_and_inspect(s.port(), {}, "TLS_CHACHA20_POLY1305_SHA256"); + REQUIRE_FALSE(r.succeeded); + } +} + +TEST_CASE("Server restricts key exchange groups to the configured list") +{ + auto [cert, key] = make_server_cert(); + EchoServer s(cert, key); + + SUBCASE("an approved classical group is accepted") + { + const auto r = handshake_and_inspect(s.port(), "P-256"); + REQUIRE(r.succeeded); + REQUIRE(r.group == "secp256r1"); + } + + SUBCASE("the client order decides among approved groups") + { + // In TLS 1.3 OpenSSL selects the first client-offered group the server + // also supports, so the server order is only a filter. + REQUIRE( + handshake_and_inspect(s.port(), "P-521:P-256").group == "secp521r1"); + REQUIRE( + handshake_and_inspect(s.port(), "P-256:P-521").group == "secp256r1"); + } + + SUBCASE("a group the server does not offer is refused") + { + const auto r = handshake_and_inspect(s.port(), "X448"); + REQUIRE_FALSE(r.succeeded); + } + + SUBCASE("an unoffered group falls back to the first shared approved one") + { + const auto r = handshake_and_inspect(s.port(), "X448:P-384"); + REQUIRE(r.succeeded); + REQUIRE(r.group == "secp384r1"); + } +} + +TEST_CASE("Server prefers the strongest hybrid post-quantum group") +{ + if (!supports_hybrid_groups()) + { + return; + } + + auto [cert, key] = make_server_cert(); + EchoServer s(cert, key); + + SUBCASE("a client offering all configured groups gets the strongest hybrid") + { + const auto r = handshake_and_inspect( + s.port(), + "SecP384r1MLKEM1024:SecP256r1MLKEM768:X25519MLKEM768:P-521:P-384:P-256"); + REQUIRE(r.succeeded); + REQUIRE(r.group == "SecP384r1MLKEM1024"); + } + + SUBCASE("each configured hybrid group can be negotiated") + { + for (const auto* group : + {"SecP384r1MLKEM1024", "SecP256r1MLKEM768", "X25519MLKEM768"}) + { + const auto r = handshake_and_inspect(s.port(), group); + REQUIRE(r.succeeded); + REQUIRE(r.group == group); + } + } +} + +// A client which sends faster than the node can execute must not be able to +// make it queue unbounded work. Reads pause once the node-wide budget is +// exhausted and resume once it is released, without dropping or truncating +// anything. +TEST_CASE("Reads pause while the node-wide inbound budget is exhausted") +{ + auto [cert, key] = make_server_cert(); + + constexpr size_t limit = 64 * 1024; + constexpr size_t to_send = 4 * 1024 * 1024; + auto admission = std::make_shared(limit); + + std::atomic received{0}; + + // Stands in for the bridge, which charges the budget as it hands data to a + // session and releases it once the session reports the data processed. This + // session never reports, so the budget stays charged until the test frees it. + auto server = std::make_shared( + OpenSSLServer::Config{ + .host = "127.0.0.1", + .cert_pem = cert, + .key_pem = key, + .inbound_admission = admission}, + [&]( + ::tcp::ConnID, + std::vector d, + const std::vector&, + bool) { + received += d.size(); + admission->queued(d.size()); + }); + + UVLoopRunner loop; + server->start(); + const auto port = server->port(); + loop.start(); + + // The client has to run on its own thread: once the server stops reading, + // the socket buffers fill and SSL_write blocks. It must also stay connected + // until the server has read everything - closing a socket which still has + // unread data resets the connection and discards it. + std::atomic client_may_close{false}; + std::thread client([port, &client_may_close]() { + const int fd = ::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0); + REQUIRE(fd >= 0); + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_port = htons(port); + REQUIRE(inet_pton(AF_INET, "127.0.0.1", &addr.sin_addr) == 1); + REQUIRE( + ::connect(fd, reinterpret_cast(&addr), sizeof(addr)) == 0); + + SSL_CTX* cctx = SSL_CTX_new(TLS_client_method()); + REQUIRE(cctx != nullptr); + SSL* ssl = SSL_new(cctx); + REQUIRE(ssl != nullptr); + REQUIRE(SSL_set_fd(ssl, fd) == 1); + SSL_set_connect_state(ssl); + REQUIRE(SSL_connect(ssl) == 1); + + const std::vector chunk(16 * 1024, 'x'); + size_t sent = 0; + while (sent < to_send) + { + const int n = + SSL_write(ssl, chunk.data(), static_cast(chunk.size())); + if (n <= 0) + { + break; + } + sent += static_cast(n); + } + + while (!client_may_close.load()) + { + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + } + + SSL_free(ssl); + SSL_CTX_free(cctx); + ::close(fd); + }); + + // Wait for the gate to engage. + const auto deadline = + std::chrono::steady_clock::now() + std::chrono::seconds(10); + while (!admission->saturated() && std::chrono::steady_clock::now() < deadline) + { + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + } + REQUIRE(admission->saturated()); + + // Give the server every chance to keep reading if the gate does not hold. + std::this_thread::sleep_for(std::chrono::milliseconds(500)); + + // A pass which was already in flight when the gate engaged may read up to + // its per-pass cap, and the kernel and OpenSSL hold buffers of their own, so + // some overshoot is expected. What must not happen is the whole 4MiB + // arriving: without the gate this reaches to_send almost immediately. + const size_t stalled_at = received.load(); + REQUIRE(stalled_at < to_send); + REQUIRE(stalled_at <= limit + (1024 * 1024)); + + // Releasing the budget must wake the transport and let the rest through - + // the gate pauses reads, it does not drop or truncate anything. + admission->consumed(admission->bytes_pending()); + REQUIRE_FALSE(admission->saturated()); + + const auto resume_deadline = + std::chrono::steady_clock::now() + std::chrono::seconds(20); + while (received.load() < to_send && + std::chrono::steady_clock::now() < resume_deadline) + { + // The test is standing in for the bridge, so it also has to keep releasing + // as the server reads more. + admission->consumed(admission->bytes_pending()); + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + } + REQUIRE(received.load() == to_send); + + client_may_close.store(true); + client.join(); + server->stop(OpenSSLServer::LoopState::Running); + server.reset(); + loop.thread.join(); +} + +// A session which never reports the data it was given - a custom protocol, or +// one whose queued work is cancelled as it is destroyed - must not be able to +// strand part of the node-wide budget. The bridge releases whatever is still +// outstanding when the connection is torn down. +TEST_CASE("Inbound budget is released when a connection closes unreported") +{ + auto [cert, key] = make_server_cert(); + + constexpr size_t limit = 1024 * 1024; + auto admission = std::make_shared(limit); + + OpenSSLSessionManager bridge( + OpenSSLServer::Config{ + .host = "127.0.0.1", + .cert_pem = cert, + .key_pem = key, + .inbound_admission = admission}, + [&](::tcp::ConnID id, ccf::SessionWriter& w, std::vector, bool) + -> std::shared_ptr { + // EchoSession is a plain ccf::Session, so it never calls + // inbound_consumed - exactly the case this test is about. + return std::make_shared(id, w); + }); + + UVLoopRunner loop; + bridge.start(); + const auto port = bridge.port(); + loop.start(); + + { + const int fd = ::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0); + REQUIRE(fd >= 0); + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_port = htons(port); + REQUIRE(inet_pton(AF_INET, "127.0.0.1", &addr.sin_addr) == 1); + REQUIRE( + ::connect(fd, reinterpret_cast(&addr), sizeof(addr)) == 0); + + SSL_CTX* cctx = SSL_CTX_new(TLS_client_method()); + REQUIRE(cctx != nullptr); + SSL* ssl = SSL_new(cctx); + REQUIRE(ssl != nullptr); + REQUIRE(SSL_set_fd(ssl, fd) == 1); + SSL_set_connect_state(ssl); + REQUIRE(SSL_connect(ssl) == 1); + + const std::vector payload(32 * 1024, 'y'); + REQUIRE( + SSL_write(ssl, payload.data(), static_cast(payload.size())) == + static_cast(payload.size())); + + // Wait for the bytes to be charged to the budget. + const auto deadline = + std::chrono::steady_clock::now() + std::chrono::seconds(10); + while (admission->bytes_pending() < payload.size() && + std::chrono::steady_clock::now() < deadline) + { + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + } + REQUIRE(admission->bytes_pending() >= payload.size()); + + SSL_free(ssl); + SSL_CTX_free(cctx); + ::close(fd); + } + + // Closing the connection must hand the whole charge back, or a node would + // leak budget on every connection which used a non-reporting session and + // would eventually stop reading altogether. + const auto deadline = + std::chrono::steady_clock::now() + std::chrono::seconds(10); + while (admission->bytes_pending() != 0 && + std::chrono::steady_clock::now() < deadline) + { + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + } + REQUIRE(admission->bytes_pending() == 0); + + bridge.stop(OpenSSLServer::LoopState::Running); + loop.thread.join(); +} + +int main(int argc, char** argv) +{ + ccf::tasks::set_task_threads(4); + + doctest::Context context; + context.applyCommandLine(argc, argv); + const auto result = context.run(); + + ccf::tasks::set_task_threads(0); + OPENSSL_cleanup(); + return result; +} diff --git a/src/host/test/rpc_connections.cpp b/src/host/test/rpc_connections.cpp deleted file mode 100644 index 8a6544a4053c..000000000000 --- a/src/host/test/rpc_connections.cpp +++ /dev/null @@ -1,40 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the Apache 2.0 License. - -#include "host/rpc_connections.h" - -#include "ds/ring_buffer.h" - -#define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN -#include -#include -#include -#include - -using namespace std::chrono_literals; - -TEST_CASE("RPC connections retain their ID generator until UV close") -{ - constexpr size_t ringbuffer_size = 4096; - ringbuffer::TestBuffer to_inside(ringbuffer_size); - ringbuffer::TestBuffer from_inside(ringbuffer_size); - ringbuffer::Circuit circuit(to_inside.bd, from_inside.bd); - ringbuffer::WriterFactory writer_factory(circuit); - - auto id_gen = std::make_shared(); - std::weak_ptr weak_id_gen = id_gen; - - { - asynchost::RPCConnections rpc(1s, writer_factory, id_gen); - asynchost::RPCConnections rpc_udp( - 1s, writer_factory, id_gen); - } - - // Destroying the proxies only schedules their timer close callbacks. - id_gen.reset(); - CHECK_FALSE(weak_id_gen.expired()); - - CHECK(uv_run(uv_default_loop(), UV_RUN_DEFAULT) == 0); - CHECK(weak_id_gen.expired()); - CHECK(uv_loop_close(uv_default_loop()) == 0); -} diff --git a/src/host/tls/inbound_admission.h b/src/host/tls/inbound_admission.h new file mode 100644 index 000000000000..80880f528516 --- /dev/null +++ b/src/host/tls/inbound_admission.h @@ -0,0 +1,135 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the Apache 2.0 License. +#pragma once + +#include "ds/internal_logger.h" + +#include +#include +#include +#include +#include +#include + +namespace asynchost +{ + // A node-wide bound on inbound data which has been read off sockets but not + // yet processed by the sessions it was handed to. + // + // Without such a bound, a client which sends faster than the node can + // execute grows node memory without limit: nothing else in the read path + // consults how far behind the application is. + // + // The bound is node-wide rather than per-connection, so a single connection + // can exhaust the budget and pause the others. That is deliberate: a + // per-connection limit would have to be smaller than one maximum-sized + // request before it could bound total node memory to the same figure. + // + // Accounting is in bytes rather than in queued items because bytes are the + // resource being defended. The two happen to be within a constant factor + // today (the transport hands over at most one read chunk at a time), but + // that is an artefact of the current read loop rather than a property to + // rely on. + class InboundAdmission + { + public: + // Called to ask a transport to re-examine its read interest. + using Waker = std::function; + + explicit InboundAdmission(size_t limit_) : limit(limit_) {} + + // True while the node has more unprocessed inbound data than it is willing + // to hold. Transports stop reading (but do not close anything) until this + // goes false again. + [[nodiscard]] bool saturated() const + { + return pending.load(std::memory_order_relaxed) >= limit; + } + + [[nodiscard]] size_t bytes_pending() const + { + return pending.load(std::memory_order_relaxed); + } + + // Bytes have been handed to a session and are not yet processed. + void queued(size_t n) + { + const size_t after = pending.fetch_add(n) + n; + if (after >= limit && !reported_saturated.exchange(true)) + { + LOG_INFO_FMT( + "Inbound queue limit reached ({} bytes queued, limit {}); pausing " + "reads until the node catches up", + after, + limit); + } + } + + // A session has finished with bytes previously passed to queued(). If this + // brings the node back under the limit, every registered transport is + // woken, not just the one which happened to make room - otherwise an + // interface with no traffic of its own would stay paused indefinitely. + void consumed(size_t n) + { + if (n == 0) + { + return; + } + const size_t before = pending.fetch_sub(n); + if (before >= limit && (before - n) < limit) + { + reported_saturated.store(false); + LOG_INFO_FMT("Inbound queue back under limit; resuming reads"); + wake_all(); + } + } + + // Transports register while they are running and unregister as they stop, + // so that wake_all() never touches a server which is tearing down. Returns + // a token to pass to unregister_waker(). + size_t register_waker(Waker waker) + { + const std::lock_guard guard(wakers_mutex); + const size_t token = next_token++; + wakers.emplace_back(token, std::move(waker)); + return token; + } + + void unregister_waker(size_t token) + { + const std::lock_guard guard(wakers_mutex); + std::erase_if( + wakers, [token](const auto& entry) { return entry.first == token; }); + } + + private: + void wake_all() + { + // Copied out so that no transport's own locks are taken while holding + // this one - consumed() is called from session workers, and unregister + // happens during a transport's shutdown. + std::vector to_wake; + { + const std::lock_guard guard(wakers_mutex); + to_wake.reserve(wakers.size()); + for (const auto& [token, waker] : wakers) + { + to_wake.push_back(waker); + } + } + for (const auto& waker : to_wake) + { + waker(); + } + } + + std::atomic pending{0}; + const size_t limit; + // Only so that a sustained overload does not log on every crossing. + std::atomic reported_saturated{false}; + + std::mutex wakers_mutex; + std::vector> wakers; + size_t next_token = 0; + }; +} diff --git a/src/host/tls/openssl_server.h b/src/host/tls/openssl_server.h new file mode 100644 index 000000000000..b0da31d1e62a --- /dev/null +++ b/src/host/tls/openssl_server.h @@ -0,0 +1,1867 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the Apache 2.0 License. +#pragma once + +// OpenSSL-native TLS/plaintext TCP server for RPC interfaces. OpenSSL owns the +// socket fd directly. The host libuv loop reports readiness, while +// per-connection OrderedTasks drive non-blocking handshake, reads, writes, and +// graceful close away from the loop thread. + +#include "ccf/crypto/openssl/openssl_wrappers.h" +#include "ds/internal_logger.h" +#include "host/tls/inbound_admission.h" +#include "tasks/ordered_tasks.h" +#include "tasks/task_system.h" +#include "tasks/worker.h" +#include "tcp/msg_types.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace asynchost +{ + namespace details + { + struct SocketOptionError + { + const char* option = nullptr; + int error = 0; + }; + + inline std::optional configure_tcp_connection(int fd) + { + const auto set_option = [fd]( + int level, + int option, + const char* option_name, + int value) -> std::optional { + if (setsockopt(fd, level, option, &value, sizeof(value)) != 0) + { + return SocketOptionError{option_name, errno}; + } + return std::nullopt; + }; + + const int one = 1; + if (auto error = set_option(IPPROTO_TCP, TCP_NODELAY, "TCP_NODELAY", one)) + { + return error; + } + if ( + auto error = set_option(SOL_SOCKET, SO_KEEPALIVE, "SO_KEEPALIVE", one)) + { + return error; + } + if ( + auto error = set_option(IPPROTO_TCP, TCP_KEEPIDLE, "TCP_KEEPIDLE", 30)) + { + return error; + } + if ( + auto error = set_option(IPPROTO_TCP, TCP_KEEPINTVL, "TCP_KEEPINTVL", 1)) + { + return error; + } + return set_option(IPPROTO_TCP, TCP_KEEPCNT, "TCP_KEEPCNT", 10); + } + } + + class OpenSSLServer : public std::enable_shared_from_this + { + public: + // Invoked on a worker with a complete chunk of decrypted bytes, the + // certificate captured during this connection's handshake, and the + // soft-limit decision taken when the connection was admitted. + using OnData = std::function data, + const std::vector& peer_cert, + bool soft_limited)>; + + // Invoked on the libuv thread when a connection is torn down (peer + // disconnect, error, or close_connection()). Lets an owner drop per- + // connection state. + using OnClose = std::function; + + // Invoked on the libuv thread for each newly accepted connection, before + // any TLS state is created. Returning nullopt refuses the connection: the + // socket is closed immediately and OnClose is *not* invoked for it. + // + // Otherwise the returned bool is the "admitted above a soft limit" + // decision, recorded on the connection and handed back to OnData. It is + // decided here rather than when the first request arrives so that it + // reflects the connection's position in the admission order, and does not + // drift for a client which is slow to send. + // + // If this returns a value, OnClose is guaranteed to be invoked exactly + // once for that connection id. Admission control can therefore reserve a + // resource here and release it in OnClose. + using OnAccept = std::function(::tcp::ConnID conn_id)>; + + // Whether anything is running the libuv loop when stop() is called. See + // stop() for why this has to be stated rather than inferred. + enum class LoopState : uint8_t + { + Running, + NotRunning, + }; + + // Everything about a server which is not a callback. Aggregate-initialised + // at the call site so each setting is named, rather than being a run of + // positional arguments where a transposition would compile silently. + // + // Every field has a default member initialiser, so a call site can name + // only the settings it cares about without tripping + // -Wmissing-field-initializers. That is also why the `= {}` on the string + // members cannot be dropped, despite looking redundant on its own. + // NOLINTBEGIN(readability-redundant-member-init) + struct Config + { + // Address to bind. Resolved with getaddrinfo, so hostnames ("localhost") + // and IPv6 literals ("::1") work, not just IPv4 literals. Port 0 + // requests an ephemeral port, which port() reports back once bound. + std::string host = {}; + uint16_t port = 0; + + // Server certificate and key. May be empty for a TLS interface whose + // certificate is not yet known - the interface then refuses connections + // until set_server_cert() supplies one. Ignored when `plaintext` is set. + std::string cert_pem = {}; + std::string key_pem = {}; + + // ALPN protocol to advertise, e.g. "h2" or "http/1.1". Empty disables + // ALPN. + std::string alpn = {}; + + // UNSECURED interface: no TLS at all, raw socket reads and writes. + bool plaintext = false; + + // Close a connection after this much inactivity. nullopt keeps idle + // connections open indefinitely. + std::optional idle_timeout = std::nullopt; + + // Shared connection-id source, so that servers on different interfaces + // allocate ids from a single space - required for a global session + // registry and reply routing. Null uses a per-server counter. + std::atomic<::tcp::ConnID>* shared_next_id = nullptr; + + // Node-wide bound on inbound data which has been read but not yet + // processed, shared with every other interface's transport. While it is + // saturated this server stops reading. Null disables the gate. + std::shared_ptr inbound_admission = nullptr; + + // Loop to register this server's handles on. + uv_loop_t* loop = uv_default_loop(); + }; + // NOLINTEND(readability-redundant-member-init) + + private: + static constexpr size_t read_chunk = 16384; + static constexpr size_t max_read_per_event = read_chunk * 4; + + // Heap-allocate a libuv handle whose lifetime is independent of this + // server, and close it so that it frees itself. + // + // uv_close() is asynchronous: the close callback only runs when the loop + // next runs, which may be long after the server has been destroyed - or + // never, if node startup failed before the event loop was entered. If + // handles were members, destruction would have to block until the loop had + // drained them, which in turn would force the server to drive the loop + // itself. Owning each handle separately means shutdown is fire-and-forget: + // request the close, drop the handle, and let the loop reclaim it whenever + // it next runs. + // + // Nothing dereferences handle->data after uv_close(), because libuv + // guarantees no further callbacks for a handle beyond its close callback, + // and that callback does nothing but free the handle. + template + static THandle* new_handle() + { + // NOLINTNEXTLINE(cppcoreguidelines-owning-memory) + return new THandle{}; + } + + template + static void close_handle(THandle*& handle) + { + if (handle == nullptr) + { + return; + } + auto* as_handle = reinterpret_cast(handle); + handle = nullptr; + if (uv_is_closing(as_handle) != 0) + { + return; + } + uv_close(as_handle, [](uv_handle_t* h) { + // NOLINTNEXTLINE(cppcoreguidelines-owning-memory) + delete reinterpret_cast(h); + }); + } + + struct OutItem; + + struct Conn + { + OpenSSLServer* owner = nullptr; + int fd = -1; + uv_poll_t* poll = nullptr; + std::shared_ptr tls_tasks; + std::shared_ptr accepted_ctx; + SSL* ssl = nullptr; + ::tcp::ConnID id = 0; + // Admission decision from OnAccept, passed to OnData. + bool soft_limited = false; + enum State : uint8_t + { + Handshaking, + Ready + } state = Handshaking; + std::vector peer_cert; + // Pending plaintext to be encrypted/written; out_off bytes already sent. + std::vector outbuf; + size_t out_off = 0; + // True when progress needs the socket to become writable (handshake + // wants write, or there is buffered outbound data). + bool want_write = false; + // A close was requested but is deferred until the buffered output has + // been fully written, so a response queued just before close is not + // truncated. + bool close_after_flush = false; + // Last time any I/O happened on this connection; used for idle timeout. + std::chrono::steady_clock::time_point last_active = + std::chrono::steady_clock::now(); + + // The fields below are accessed only on the libuv thread. Polling stays + // stopped while worker_active is true. + std::vector pending_commands; + int pending_events = 0; + bool worker_active = false; + bool close_requested = false; + // Set once, when the server is being torn down. Unlike close_requested + // it is sticky, and it does not wait for buffered output to drain: a + // peer which has stopped reading must not be able to keep the connection + // (and so the shutdown) alive indefinitely. + bool force_close = false; + }; + + struct OutItem + { + ::tcp::ConnID id = 0; + std::vector data; + bool close = false; + }; + + struct DriveInput + { + int events = 0; + std::vector commands; + bool close_requested = false; + bool force_close = false; + }; + + struct DriveResult + { + std::shared_ptr conn; + bool alive = true; + // The pass stopped at the per-pass read cap with bytes still available + // that will not produce a new readability event. The loop must schedule + // another pass rather than re-arm polling. + bool more_to_read = false; + std::chrono::steady_clock::time_point last_active; + }; + + std::shared_ptr ctx; + uv_loop_t* loop = nullptr; + ::tcp::ConnID next_id = 1; + // Optional shared id source so multiple servers (one per interface) + // allocate connection ids from a single global space - required for a + // global session registry and reply routing. + std::atomic<::tcp::ConnID>* shared_next_id = nullptr; + std::chrono::steady_clock::time_point last_idle_sweep = + std::chrono::steady_clock::now(); + + // Close a connection after this much inactivity (no I/O); nullopt disables + // idle closure. A libuv timer wakes every idle_sweep_interval_ms to check. + static constexpr int idle_sweep_interval_ms = 1000; + std::optional idle_timeout; + + // Node-wide inbound budget, shared with every other interface's transport. + // Null disables the gate. + std::shared_ptr inbound_admission; + std::optional admission_token; + + [[nodiscard]] bool inbound_saturated() const + { + return inbound_admission != nullptr && inbound_admission->saturated(); + } + + // The subset of a connection's pending events which may be acted on now. + // Read interest is withheld while the node holds more unprocessed inbound + // data than it is willing to. + [[nodiscard]] int actionable_events(const Conn& c) const + { + if (!inbound_saturated()) + { + return c.pending_events; + } + return c.pending_events & ~UV_READABLE; + } + + // Cross-thread outbound queue: send()/close_connection() append here from + // any thread and wake the loop, which drains it on the libuv thread. + std::vector pending_out; + std::vector completed_drives; + + // Cross-thread server-cert (re)load requests (deferred cert / rotation), + // applied on the loop thread so `ctx` is only ever touched there. + std::vector> pending_certs; + + // ALPN protocol advertised by the server (wire format, length-prefixed), + // e.g. "\x02h2" or "\x08http/1.1". Empty disables ALPN. + std::string alpn_wire; + OnData on_data; + OnClose on_close; + OnAccept on_accept; + + std::mutex out_mutex; + std::mutex lifecycle_mutex; + std::condition_variable teardown_cv; + std::unordered_map> conns; + std::unordered_map<::tcp::ConnID, int> id_to_fd; + // Owned by the loop once closed - see new_handle()/close_handle(). + uv_async_t* wake_handle = nullptr; + uv_timer_t* idle_timer = nullptr; + uv_poll_t* listen_poll = nullptr; + int listen_fd = -1; + uint16_t bound_port = 0; + // Whether the bound socket has been placed in the LISTEN state and its + // poll handle armed. A TLS interface with no certificate stays bound but + // not listening, so inbound connections are refused by the kernel rather + // than accepted and dropped. Loop-thread only after start(). + bool listening = false; + // Plaintext (UNSECURED) interface: no TLS, raw socket I/O. + bool plaintext = false; + bool started = false; + bool stopping = false; + // Set once the teardown has run: every connection dropped, every handle + // handed to uv_close(), and the listening socket closed. The handles may + // not have been reclaimed by the loop yet, but nothing here refers to + // them any more, so the server is safe to destroy. + bool torn_down = false; + + // Describe a failed SSL operation. SSL_get_error() only gives the + // category: for SSL_ERROR_SSL the detail is in the (thread-local) error + // queue, and for SSL_ERROR_SYSCALL it may be in errno instead. Consuming + // the queue entry here also keeps it from being misattributed to the next + // operation this worker performs. + static std::string ssl_error_string(int ssl_error) + { + switch (ssl_error) + { + case SSL_ERROR_ZERO_RETURN: + return "peer closed the TLS connection"; + + case SSL_ERROR_SSL: + case SSL_ERROR_SYSCALL: + { + const auto err = ERR_get_error(); + if (err != 0) + { + return ccf::crypto::OpenSSL::error_string(err); + } + if (ssl_error == SSL_ERROR_SYSCALL) + { + return fmt::format( + "syscall failed: {}", std::generic_category().message(errno)); + } + return "protocol error"; + } + + default: + return fmt::format("SSL_get_error {}", ssl_error); + } + } + + static bool set_nonblocking(int fd) + { + int flags = fcntl(fd, F_GETFL, 0); + if (flags < 0) + { + return false; + } + return fcntl(fd, F_SETFL, flags | O_NONBLOCK) == 0; + } + + static bool load_cert_key( + SSL_CTX* ctx, const std::string& cert_pem, const std::string& key_pem) + { + namespace OpenSSL = ccf::crypto::OpenSSL; + + OpenSSL::Unique_BIO cbio( + cert_pem.data(), static_cast(cert_pem.size())); + // check_null defaults to false for this overload: a malformed PEM yields + // a null pointer rather than an exception. + OpenSSL::Unique_X509 cert(cbio, true); + if (cert == nullptr) + { + return false; + } + if (SSL_CTX_use_certificate(ctx, cert) != 1) + { + return false; + } + + OpenSSL::Unique_BIO kbio( + key_pem.data(), static_cast(key_pem.size())); + OpenSSL::Unique_PKEY pkey( + PEM_read_bio_PrivateKey(kbio, nullptr, nullptr, nullptr), + EVP_PKEY_free, + false); + if (pkey == nullptr) + { + return false; + } + if (SSL_CTX_use_PrivateKey(ctx, pkey) != 1) + { + return false; + } + + return SSL_CTX_check_private_key(ctx) == 1; + } + + // Build a server SSL_CTX (min TLS 1.2, ALPN if configured) and load the + // cert/key. Returns nullptr on failure, having logged which step failed and + // why. Called on the loop thread. + // + // This is the only place CCF's inbound TLS policy is defined. It is + // asserted from the wire by src/host/test/openssl_server_test.cpp and, for + // a running service, by tests/tls_groups.py. + std::shared_ptr build_server_ctx( + const std::string& cert_pem, const std::string& key_pem) + { + // So that anything reported below comes from this function rather than + // from unrelated work previously done on this thread. + ERR_clear_error(); + + const auto fail = [](const char* step) { + LOG_FAIL_FMT( + "Failed to build TLS context ({}): {}", + step, + ccf::crypto::OpenSSL::error_string(ERR_get_error())); + return std::shared_ptr{}; + }; + + ccf::crypto::OpenSSL::Unique_SSL_CTX c(TLS_server_method()); + + // Require at least TLS 1.2, support up to 1.3 + if (SSL_CTX_set_min_proto_version(c, TLS1_2_VERSION) != 1) + { + return fail("SSL_CTX_set_min_proto_version"); + } + + // Disable renegotiation to avoid DoS + SSL_CTX_set_options( + c, + SSL_OP_CIPHER_SERVER_PREFERENCE | + SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION | + SSL_OP_NO_RENEGOTIATION); + + // Set cipher for TLS 1.2 + const auto* const cipher_list = + "ECDHE-ECDSA-AES256-GCM-SHA384:" + "ECDHE-ECDSA-AES128-GCM-SHA256:" + "ECDHE-RSA-AES256-GCM-SHA384:" + "ECDHE-RSA-AES128-GCM-SHA256"; + if (SSL_CTX_set_cipher_list(c, cipher_list) != 1) + { + return fail("SSL_CTX_set_cipher_list"); + } + + // Set cipher for TLS 1.3 + const auto* const ciphersuites = + "TLS_AES_256_GCM_SHA384:" + "TLS_AES_128_GCM_SHA256"; + if (SSL_CTX_set_ciphersuites(c, ciphersuites) != 1) + { + return fail("SSL_CTX_set_ciphersuites"); + } + + // Prefer hybrid post-quantum groups when available, while retaining the + // approved classical groups as fallbacks + if ( + SSL_CTX_set1_groups_list( + c, + "?SecP384r1MLKEM1024:?SecP256r1MLKEM768:?X25519MLKEM768:" + "P-521:P-384:P-256") != 1) + { + return fail("SSL_CTX_set1_groups_list"); + } + + // Allow buffer to be relocated between WANT_WRITE retries, and do partial + // writes if possible. do_write() retries SSL_write() from a std::vector + // which drive_connection() may have appended to (and so reallocated) + // since the previous attempt, so both are required. + SSL_CTX_set_mode( + c, SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER | SSL_MODE_ENABLE_PARTIAL_WRITE); + + // Request the client certificate during the handshake so it can be used + // for application-level caller authentication (user/member cert auth). + // Verification is not enforced here - the application decides. + SSL_CTX_set_verify( + c, SSL_VERIFY_PEER, [](int, X509_STORE_CTX*) { return 1; }); + if (!alpn_wire.empty()) + { + SSL_CTX_set_alpn_select_cb(c, alpn_select_cb, this); + } + if (!load_cert_key(c, cert_pem, key_pem)) + { + return fail("loading certificate and key"); + } + return {c.release(), SSL_CTX_free}; + } + + void update_interest(Conn& c) const + { + if (c.poll == nullptr) + { + return; + } + int events = c.want_write ? UV_WRITABLE : 0; + if (!inbound_saturated()) + { + events |= UV_READABLE; + } + if (events == 0) + { + // uv_poll_start rejects an empty mask, and there is genuinely nothing + // to wait for: this connection resumes when the node drops back under + // its inbound budget, at which point every transport is woken. + (void)uv_poll_stop(c.poll); + return; + } + const int rc = uv_poll_start(c.poll, events, on_connection_poll); + if (rc != 0) + { + // Should not happen. If it does the connection will make no further + // progress, so it is worth reporting loudly rather than hiding. + LOG_FAIL_FMT( + "uv_poll_start failed for connection {}: {}", c.id, uv_strerror(rc)); + } + } + + static int alpn_select_cb( + SSL* /*ssl*/, + const unsigned char** out, + unsigned char* outlen, + const unsigned char* in, + unsigned int inlen, + void* arg) + { + auto* self = static_cast(arg); + const auto& wire = self->alpn_wire; + if (wire.empty()) + { + return SSL_TLSEXT_ERR_NOACK; + } + // NOLINTNEXTLINE(cppcoreguidelines-pro-type-reinterpret-cast) + const auto* protos = reinterpret_cast(wire.data()); + if ( + SSL_select_next_proto( + const_cast(out), + outlen, + protos, + static_cast(wire.size()), + in, + inlen) != OPENSSL_NPN_NEGOTIATED) + { + return SSL_TLSEXT_ERR_NOACK; + } + return SSL_TLSEXT_ERR_OK; + } + + // Returns false if the connection should be closed. + bool do_handshake(Conn& c, bool& more_to_read) + { + // The error queue is thread-local and SSL_get_error() consults it. Clear + // it before each operation because successive actions for this + // connection may execute on different workers. + ERR_clear_error(); + const int r = SSL_accept(c.ssl); + if (r == 1) + { + c.state = Conn::Ready; + c.want_write = false; + X509* cert = SSL_get1_peer_certificate(c.ssl); + if (cert != nullptr) + { + const int len = i2d_X509(cert, nullptr); + if (len > 0) + { + c.peer_cert.resize(static_cast(len)); + unsigned char* p = c.peer_cert.data(); + if (i2d_X509(cert, &p) != len) + { + c.peer_cert.clear(); + } + } + X509_free(cert); + } + LOG_TRACE_FMT("Connection {}: handshake complete", c.id); + return do_read(c, more_to_read) && do_write(c); + } + + const int e = SSL_get_error(c.ssl, r); + if (e == SSL_ERROR_WANT_READ) + { + c.want_write = false; + return true; + } + if (e == SSL_ERROR_WANT_WRITE) + { + c.want_write = true; + return true; + } + // Entirely client-controlled (an unsupported cipher, a rejected + // certificate, or simply a port scan), so this must stay at a level + // which cannot be used to flood the log. + LOG_DEBUG_FMT( + "Connection {}: handshake failed: {}", c.id, ssl_error_string(e)); + return false; + } + + // Returns false if the connection should be closed. + bool do_read_plaintext(Conn& c, bool& more_to_read) + { + size_t total_read = 0; + while (total_read < max_read_per_event) + { + uint8_t buf[read_chunk]; + const ssize_t n = ::recv(c.fd, buf, sizeof(buf), 0); + if (n > 0) + { + total_read += static_cast(n); + if (on_data) + { + on_data( + c.id, + std::vector(buf, buf + static_cast(n)), + c.peer_cert, + c.soft_limited); + } + continue; + } + if (n == 0) + { + return false; // peer closed + } + if (errno == EAGAIN || errno == EWOULDBLOCK) + { + return true; + } + if (errno == EINTR) + { + continue; + } + return false; + } + // Stopped at the cap rather than at EAGAIN, so there may be more in the + // socket buffer. Level-triggered polling would report it again, but ask + // for another pass directly rather than depend on that. + more_to_read = true; + return true; + } + + // Returns false if the connection should be closed. + bool do_write_plaintext(Conn& c) + { + while (c.out_off < c.outbuf.size()) + { + const ssize_t n = ::send( + c.fd, + c.outbuf.data() + c.out_off, + c.outbuf.size() - c.out_off, + MSG_NOSIGNAL); + if (n > 0) + { + c.out_off += static_cast(n); + continue; + } + if (n < 0 && (errno == EAGAIN || errno == EWOULDBLOCK)) + { + c.want_write = true; + compact_outbuf(c); + return true; + } + if (n < 0 && errno == EINTR) + { + continue; + } + return false; + } + c.outbuf.clear(); + c.out_off = 0; + c.want_write = false; + return true; + } + + // Returns false if the connection should be closed. + bool do_read(Conn& c, bool& more_to_read) + { + if (c.ssl == nullptr) + { + return do_read_plaintext(c, more_to_read); + } + size_t total_read = 0; + while (total_read < max_read_per_event) + { + uint8_t buf[read_chunk]; + ERR_clear_error(); + const int n = SSL_read(c.ssl, buf, static_cast(sizeof(buf))); + if (n > 0) + { + total_read += static_cast(n); + if (on_data) + { + on_data( + c.id, + std::vector(buf, buf + n), + c.peer_cert, + c.soft_limited); + } + continue; + } + + const int e = SSL_get_error(c.ssl, n); + if (e == SSL_ERROR_WANT_READ) + { + return true; + } + if (e == SSL_ERROR_WANT_WRITE) + { + // A renegotiation needs the socket to become writable. + c.want_write = true; + return true; + } + // SSL_ERROR_ZERO_RETURN (clean close), an unclean EOF from the peer + // (no close_notify), or a fatal error - in all cases close. + return false; + } + + // Stopped at the per-pass cap rather than at WANT_READ. OpenSSL may + // still be holding bytes it has already taken off the socket - either + // decrypted plaintext or a buffered record. Those bytes will never + // produce a readability event, so waiting for one here would stall the + // connection until the peer happened to send more. Ask for another pass + // instead. SSL_has_pending() covers both processed and unprocessed + // buffered data; if it is only a partial record, the next pass returns + // WANT_READ and polling resumes normally. + more_to_read = SSL_has_pending(c.ssl) == 1; + return true; + } + + // Release the already-written prefix of a partially flushed buffer, so a + // large response which is draining slowly does not keep holding the bytes + // the peer has already received. erase() would retain the original + // capacity, so rebuild into a right-sized buffer instead. Called only when + // a write has just stalled and the remainder is about to be held until the + // socket becomes writable again - compacting a buffer we are about to + // flush anyway would be pure overhead. Only worth the copy once most of + // the buffer is consumed, which also bounds the total copying for one + // response to O(its size). + static void compact_outbuf(Conn& c) + { + if (c.out_off == 0 || c.out_off <= c.outbuf.size() / 2) + { + return; + } + std::vector remaining( + c.outbuf.data() + c.out_off, c.outbuf.data() + c.outbuf.size()); + c.outbuf = std::move(remaining); + c.out_off = 0; + } + + // Returns false if the connection should be closed. Implements + // backpressure: a WANT_WRITE leaves the remaining plaintext buffered and + // arms UV_WRITABLE. + bool do_write(Conn& c) + { + if (c.ssl == nullptr) + { + return do_write_plaintext(c); + } + while (c.out_off < c.outbuf.size()) + { + ERR_clear_error(); + const int n = SSL_write( + c.ssl, + c.outbuf.data() + c.out_off, + static_cast(c.outbuf.size() - c.out_off)); + if (n > 0) + { + c.out_off += static_cast(n); + continue; + } + + const int e = SSL_get_error(c.ssl, n); + if (e == SSL_ERROR_WANT_WRITE) + { + c.want_write = true; + compact_outbuf(c); + return true; + } + if (e == SSL_ERROR_WANT_READ) + { + // A renegotiation needs to read before we can write more. + return true; + } + // Usually just the peer having gone away mid-response, so again a + // level which a client cannot use to flood the log. + LOG_DEBUG_FMT( + "Connection {}: write failed: {}", c.id, ssl_error_string(e)); + return false; + } + + // Fully flushed. + c.outbuf.clear(); + c.out_off = 0; + c.want_write = false; + return true; + } + + void complete_drive( + std::shared_ptr conn, bool alive, bool more_to_read) + { + { + std::lock_guard guard(out_mutex); + completed_drives.push_back( + {std::move(conn), + alive, + more_to_read, + std::chrono::steady_clock::now()}); + } + // Unlike wake(), this must signal even while stopping: shutdown only + // completes once the loop has observed every outstanding completion and + // closed the corresponding connection. + std::lock_guard guard(lifecycle_mutex); + if (wake_handle != nullptr) + { + (void)uv_async_send(wake_handle); + } + } + + void drive_connection(std::shared_ptr conn, DriveInput input) + { + bool alive = true; + bool more_to_read = false; + + if (conn->ssl == nullptr && conn->accepted_ctx != nullptr) + { + ERR_clear_error(); + conn->ssl = SSL_new(conn->accepted_ctx.get()); + if (conn->ssl == nullptr || SSL_set_fd(conn->ssl, conn->fd) != 1) + { + LOG_FAIL_FMT( + "Connection {}: failed to create SSL state: {}", + conn->id, + ccf::crypto::OpenSSL::error_string(ERR_get_error())); + if (conn->ssl != nullptr) + { + SSL_free(conn->ssl); + conn->ssl = nullptr; + } + complete_drive(std::move(conn), false, false); + return; + } + SSL_set_accept_state(conn->ssl); + } + + for (auto& command : input.commands) + { + if (command.close) + { + conn->close_after_flush = true; + } + else if (conn->outbuf.empty()) + { + // The common case is a single response with nothing still draining, + // so take ownership of the buffer instead of copying it again. + // outbuf is only ever emptied alongside out_off being reset, so + // there is no consumed prefix to preserve here. + conn->outbuf = std::move(command.data); + } + else + { + conn->outbuf.insert( + conn->outbuf.end(), command.data.begin(), command.data.end()); + } + } + if (input.close_requested) + { + conn->close_after_flush = true; + } + + if (conn->state == Conn::Handshaking) + { + alive = do_handshake(*conn, more_to_read); + } + else + { + if ((input.events & (UV_READABLE | UV_DISCONNECT)) != 0) + { + alive = do_read(*conn, more_to_read); + } + if (alive) + { + alive = do_write(*conn); + } + } + + if ( + alive && conn->close_after_flush && + conn->out_off >= conn->outbuf.size()) + { + alive = false; + } + if (input.force_close) + { + // The server is going away. Whatever could be written above has been, + // but the connection is not held open waiting for more room. + alive = false; + } + if (!alive && conn->ssl != nullptr) + { + ERR_clear_error(); + (void)SSL_shutdown(conn->ssl); + SSL_free(conn->ssl); + conn->ssl = nullptr; + } + complete_drive(std::move(conn), alive, more_to_read); + } + + void dispatch_connection(const std::shared_ptr& conn) + { + if (conn->worker_active) + { + return; + } + if (conn->poll != nullptr) + { + (void)uv_poll_stop(conn->poll); + } + conn->worker_active = true; + DriveInput input; + input.events = actionable_events(*conn); + // Read interest which the inbound budget is currently withholding stays + // pending rather than being discarded: it may be the only record that + // OpenSSL is holding buffered data which will never produce another + // readability event. + conn->pending_events &= ~input.events; + input.commands.swap(conn->pending_commands); + input.close_requested = std::exchange(conn->close_requested, false); + input.force_close = conn->force_close; + // The worker keeps the server alive for the whole pass. complete_drive() + // posts its result and only then touches lifecycle_mutex and + // wake_handle; without this the loop could consume that result, finish + // the teardown and let stop() return in between, destroying the server + // underneath the worker. + conn->tls_tasks->add_action(ccf::tasks::make_basic_action( + [self = shared_from_this(), conn, input = std::move(input)]() mutable { + self->drive_connection(conn, std::move(input)); + }, + "OpenSSLServer::drive_connection")); + } + + void close_conn(int fd) + { + auto it = conns.find(fd); + if (it == conns.end()) + { + return; + } + auto conn = it->second; + if (on_close) + { + on_close(conn->id); + } + id_to_fd.erase(conn->id); + conns.erase(it); + + if (conn->poll != nullptr) + { + (void)uv_poll_stop(conn->poll); + } + assert(conn->ssl == nullptr); + // The poll handle frees itself, so the Conn can be dropped here rather + // than being parked until the close callback runs. + close_handle(conn->poll); + ::close(fd); + conn->fd = -1; + } + + void accept_all() + { + for (;;) + { + sockaddr_storage peer{}; + socklen_t plen = sizeof(peer); + const int cfd = accept4( + listen_fd, + reinterpret_cast(&peer), + &plen, + SOCK_NONBLOCK | SOCK_CLOEXEC); + if (cfd < 0) + { + if (errno == EAGAIN || errno == EWOULDBLOCK) + { + break; + } + if (errno == EINTR) + { + continue; + } + const auto err = errno; + // The listening socket is level-triggered, so a persistent failure + // (notably EMFILE once the process is out of file descriptors) is + // re-reported on every loop iteration. Keep it out of the default + // log for that reason. + LOG_DEBUG_FMT( + "accept4 failed: {}", std::generic_category().message(err)); + break; + } + + if (const auto error = details::configure_tcp_connection(cfd)) + { + LOG_FAIL_FMT( + "setsockopt({}) failed for accepted RPC socket: {}", + error->option, + std::generic_category().message(error->error)); + ::close(cfd); + continue; + } + + const ::tcp::ConnID cid = (shared_next_id != nullptr) ? + shared_next_id->fetch_add(1) : + next_id++; + + // Admission control runs here, before any TLS context, SSL object or + // task queue is created for the connection. A refused connection + // therefore costs nothing beyond the accept itself, and - crucially - + // an admitted connection is counted from the moment it exists, not + // from the moment it first sends a request. A client which completes + // the TCP and TLS handshakes and then goes silent still holds a file + // descriptor and TLS state, and must count against the caps. + bool soft_limited = false; + if (on_accept) + { + const auto admission = on_accept(cid); + if (!admission.has_value()) + { + ::close(cfd); + continue; + } + soft_limited = *admission; + } + + // From here on the connection has been admitted, so every failure path + // must release it by invoking on_close. + const auto release_admitted = [this, cid]() { + if (on_accept && on_close) + { + on_close(cid); + } + }; + + auto c = std::make_unique(); + c->owner = this; + c->fd = cfd; + c->id = cid; + c->soft_limited = soft_limited; + c->tls_tasks = ccf::tasks::OrderedTasks::create( + ccf::tasks::get_main_job_board(), + "TLS connection " + std::to_string(c->id)); + + if (plaintext) + { + // No TLS: ready to read/write raw bytes immediately. + c->state = Conn::Ready; + } + else + { + if (ctx == nullptr) + { + // No server certificate yet - refuse the connection until one is + // supplied (see set_server_cert). Expected while a joining node + // waits for the service certificate, so this is not an error, but + // it is otherwise invisible, hence the log. + LOG_DEBUG_FMT( + "Refusing connection {}: no server certificate yet", cid); + ::close(cfd); + release_admitted(); + continue; + } + c->accepted_ctx = ctx; + } + + c->poll = new_handle(); + const int poll_rc = uv_poll_init_socket(loop, c->poll, cfd); + if (poll_rc != 0) + { + LOG_FAIL_FMT( + "uv_poll_init_socket failed for connection {}: {}", + cid, + uv_strerror(poll_rc)); + // Never registered with the loop, so free it directly rather than + // going through uv_close. + // NOLINTNEXTLINE(cppcoreguidelines-owning-memory) + delete c->poll; + c->poll = nullptr; + ::close(cfd); + release_admitted(); + continue; + } + c->poll->data = c.get(); + const int start_rc = + uv_poll_start(c->poll, UV_READABLE, on_connection_poll); + if (start_rc != 0) + { + LOG_FAIL_FMT( + "uv_poll_start failed for connection {}: {}", + cid, + uv_strerror(start_rc)); + close_handle(c->poll); + ::close(cfd); + release_admitted(); + continue; + } + conns.emplace(cfd, std::move(c)); + id_to_fd.emplace(cid, cfd); + LOG_TRACE_FMT("Accepted connection {} on fd {}", cid, cfd); + } + } + + void on_conn_event(int fd, int events) + { + auto it = conns.find(fd); + if (it == conns.end()) + { + return; + } + auto& c = it->second; + c->last_active = std::chrono::steady_clock::now(); + c->pending_events |= events; + dispatch_connection(c); + } + + static void on_connection_poll(uv_poll_t* handle, int status, int events) + { + auto* conn = static_cast(handle->data); + auto* self = conn->owner; + if (status < 0) + { + auto it = self->conns.find(conn->fd); + if (it != self->conns.end()) + { + it->second->close_requested = true; + self->dispatch_connection(it->second); + } + return; + } + self->on_conn_event(conn->fd, events); + } + + void wake() + { + std::lock_guard guard(lifecycle_mutex); + if (wake_handle != nullptr && !stopping) + { + (void)uv_async_send(wake_handle); + } + } + + // Place the bound socket in the LISTEN state and arm its poll handle, if + // the interface is ready to serve connections and is not already doing so. + // Idempotent. Called from start() and, for an interface whose certificate + // arrived later, from the loop thread once it has been applied. + void begin_listening() + { + if ( + listening || listen_poll == nullptr || listen_fd < 0 || + (!plaintext && ctx == nullptr)) + { + return; + } + if (listen(listen_fd, SOMAXCONN) != 0) + { + LOG_FAIL_FMT( + "listen() failed on port {}: {}", + bound_port, + std::generic_category().message(errno)); + return; + } + + const int rc = uv_poll_start(listen_poll, UV_READABLE, on_listen_poll); + if (rc != 0) + { + LOG_FAIL_FMT( + "uv_poll_start(listen) failed on port {}: {}", + bound_port, + uv_strerror(rc)); + return; + } + listening = true; + } + + // Apply worker completions and cross-thread commands on the libuv thread. + void drain_pending_out() + { + std::vector items; + std::vector completions; + std::vector> certs; + { + std::lock_guard g(out_mutex); + std::swap(items, pending_out); + std::swap(completions, completed_drives); + std::swap(certs, pending_certs); + } + + for (auto& [cert_pem, key_pem] : certs) + { + // This runs inside a libuv callback, so nothing may escape: an + // operator-supplied cert which fails to parse must not terminate the + // process, it must leave the previous context in place. + try + { + // build_server_ctx() reports why it failed. + auto nc = build_server_ctx(cert_pem, key_pem); + if (nc == nullptr) + { + continue; + } + ctx = std::move(nc); + } + catch (const std::exception& e) + { + LOG_FAIL_FMT( + "set_server_cert: failed to build TLS context: {}", e.what()); + } + } + + if (!certs.empty()) + { + // A cert-deferred interface has been waiting for exactly this. + begin_listening(); + } + + for (auto& item : items) + { + auto fit = id_to_fd.find(item.id); + if (fit == id_to_fd.end()) + { + continue; + } + auto cit = conns.find(fit->second); + if (cit == conns.end()) + { + continue; + } + if (item.close) + { + cit->second->close_requested = true; + } + else + { + cit->second->last_active = std::chrono::steady_clock::now(); + cit->second->pending_commands.push_back(std::move(item)); + } + } + + for (auto& completion : completions) + { + const auto& conn = completion.conn; + auto it = conns.find(conn->fd); + if (it == conns.end() || it->second.get() != conn.get()) + { + continue; + } + conn->worker_active = false; + conn->last_active = completion.last_active; + if (!completion.alive) + { + close_conn(conn->fd); + } + else if (completion.more_to_read) + { + // Data is buffered where polling cannot see it, so ask for another + // pass explicitly. The loop below dispatches on pending_events. + conn->pending_events |= UV_READABLE; + } + } + + for (auto& [fd, conn] : conns) + { + if (conn->worker_active) + { + continue; + } + if ( + conn->force_close || conn->close_requested || + !conn->pending_commands.empty() || actionable_events(*conn) != 0) + { + dispatch_connection(conn); + } + else + { + update_interest(*conn); + } + } + + bool shutting_down = false; + { + std::lock_guard guard(lifecycle_mutex); + shutting_down = stopping && !torn_down; + } + if (shutting_down) + { + // Some of those connections may have just gone, which may be the last + // thing shutdown was waiting for. + tear_down_on_loop(); + } + } + + // Close connections idle longer than idle_timeout (loop thread). + void sweep_idle() + { + if (!idle_timeout.has_value()) + { + return; + } + const auto now = std::chrono::steady_clock::now(); + std::vector to_close; + for (const auto& [fd, c] : conns) + { + if (!c->worker_active && now - c->last_active > *idle_timeout) + { + to_close.push_back(fd); + } + } + for (const int fd : to_close) + { + auto it = conns.find(fd); + if (it != conns.end()) + { + LOG_DEBUG_FMT("Closing idle connection {}", it->second->id); + it->second->close_requested = true; + dispatch_connection(it->second); + } + } + } + + static void on_listen_poll(uv_poll_t* handle, int status, int events) + { + auto* self = static_cast(handle->data); + if (status < 0) + { + self->tear_down_on_loop(); + return; + } + if ((events & UV_READABLE) != 0) + { + self->accept_all(); + } + } + + static void on_wake(uv_async_t* handle) + { + auto* self = static_cast(handle->data); + // Always drain, including while shutting down: shutdown completes only + // once every connection worker's completion has been processed and its + // connection closed, and drain_pending_out() resumes the teardown once + // it has done so. + self->drain_pending_out(); + } + + static void on_idle_timer(uv_timer_t* handle) + { + auto* self = static_cast(handle->data); + self->sweep_idle(); + } + + // Begin, or resume, shutdown on the loop thread. + // + // The listener closes immediately and every live connection is marked for + // forced close, but the server's own handles can only go once those + // connections have actually gone: a connection whose worker is still + // running is still reading and writing its socket, so neither its fd nor + // its poll handle may be touched here. drain_pending_out() calls back in + // as each worker completes, and the last call finishes the job. + // + // The close is forced rather than deferred until buffered output has + // flushed, so that each connection takes exactly one further worker pass + // and shutdown cannot be held up by a peer which has stopped reading. + // + // Nothing here waits for the uv close callbacks. Each handle owns itself + // (see new_handle()), so once torn_down is set nothing refers to them any + // more and the server is safe to destroy, whether or not the loop ever + // runs again. + void tear_down_on_loop() + { + { + std::lock_guard guard(lifecycle_mutex); + if (torn_down) + { + return; + } + stopping = true; + } + + if (listen_poll != nullptr) + { + (void)uv_poll_stop(listen_poll); + close_handle(listen_poll); + } + listening = false; + if (listen_fd >= 0) + { + ::close(listen_fd); + listen_fd = -1; + } + + for (auto& [fd, conn] : conns) + { + // force_close is sticky, so a connection already on its way out is not + // dispatched again. Without that, a connection whose output cannot + // drain would be re-dispatched on every completion, spinning the loop + // thread and a worker for as long as its peer declined to read. + if (conn->force_close) + { + continue; + } + conn->force_close = true; + dispatch_connection(conn); + } + if (!conns.empty()) + { + // Workers still own these connections. Finish when they report back. + return; + } + + if (idle_timer != nullptr) + { + (void)uv_timer_stop(idle_timer); + close_handle(idle_timer); + } + + { + // Cleared under the lock, because send()/close_connection() and + // complete_drive() consult it from other threads. Clearing it before + // the uv_close() means no other thread can observe the handle as + // usable once it is closing. + std::lock_guard guard(lifecycle_mutex); + auto* wake = wake_handle; + wake_handle = nullptr; + close_handle(wake); + + torn_down = true; + teardown_cv.notify_all(); + } + } + + // Drive shutdown to completion when nothing is running the loop. Only this + // server's own queues need servicing - no libuv work is required, and no + // other thread can be inside the loop - so this never runs the loop + // itself. + void tear_down_without_loop() + { + for (;;) + { + tear_down_on_loop(); + { + std::lock_guard guard(lifecycle_mutex); + if (torn_down) + { + return; + } + } + + // A connection worker is still running. Service the task board so it + // can finish, then pick up its completion. + auto task = ccf::tasks::get_main_job_board().get_task(); + if (task != nullptr) + { + ccf::tasks::try_do_task(*task); + } + else + { + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + } + drain_pending_out(); + } + } + + public: + // `on_data` is required; `on_close` and `on_accept` are optional. These + // stay as explicit parameters rather than joining Config so that on_data + // cannot be omitted, and so a transposition is a compile error (their + // signatures differ). + OpenSSLServer( + Config config, + OnData on_data_, + OnClose on_close_ = {}, + OnAccept on_accept_ = {}) : + loop(config.loop), + shared_next_id(config.shared_next_id), + idle_timeout(config.idle_timeout), + inbound_admission(std::move(config.inbound_admission)), + on_data(std::move(on_data_)), + on_close(std::move(on_close_)), + on_accept(std::move(on_accept_)), + plaintext(config.plaintext) + { + if (!config.alpn.empty()) + { + alpn_wire.push_back(static_cast(config.alpn.size())); + alpn_wire.append(config.alpn); + } + + // Plaintext interfaces have no TLS context. TLS interfaces build their + // context now if the cert is already available, or defer until + // set_server_cert() (e.g. a joining node receiving the service cert). + if (!plaintext && !config.cert_pem.empty()) + { + ctx = build_server_ctx(config.cert_pem, config.key_pem); + if (ctx == nullptr) + { + throw std::runtime_error("Failed to load server cert/key"); + } + } + + // Resolve and bind the listening address. Using getaddrinfo (rather than + // inet_pton) supports hostnames (e.g. "localhost") and IPv6 (e.g. "::1"), + // not just IPv4 literals. + addrinfo hints{}; + hints.ai_family = AF_UNSPEC; + hints.ai_socktype = SOCK_STREAM; + hints.ai_flags = AI_PASSIVE; + addrinfo* res = nullptr; + const std::string port_str = std::to_string(config.port); + if (getaddrinfo(config.host.c_str(), port_str.c_str(), &hints, &res) != 0) + { + cleanup(); + throw std::runtime_error("getaddrinfo failed for " + config.host); + } + + const int one = 1; + bool bound_ok = false; + for (addrinfo* ai = res; ai != nullptr; ai = ai->ai_next) + { + listen_fd = socket( + ai->ai_family, ai->ai_socktype | SOCK_CLOEXEC, ai->ai_protocol); + if (listen_fd < 0) + { + continue; + } + // SO_REUSEADDR permits rebinding a port left in TIME_WAIT by a + // previous process. Note that SO_REUSEPORT is deliberately *not* set: + // it would suppress EADDRINUSE, so two nodes misconfigured onto the + // same port would both bind successfully and have connections split + // between them at random, and it would let any process with the same + // effective UID siphon off a share of inbound connections. + if ( + setsockopt(listen_fd, SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one)) != + 0) + { + ::close(listen_fd); + listen_fd = -1; + continue; + } + if (bind(listen_fd, ai->ai_addr, ai->ai_addrlen) == 0) + { + bound_ok = true; + break; + } + ::close(listen_fd); + listen_fd = -1; + } + freeaddrinfo(res); + if (!bound_ok) + { + cleanup(); + throw std::runtime_error("bind() failed for " + config.host); + } + // Deliberately no listen() here. Binding reserves the port (and fixes an + // ephemeral one, which port() reports), but the socket only enters the + // LISTEN state once this interface can actually serve a connection - see + // begin_listening(). Until then the kernel refuses inbound SYNs, so a + // client fails to connect rather than completing a TCP handshake against + // an interface which will immediately drop it. + if (!set_nonblocking(listen_fd)) + { + cleanup(); + throw std::runtime_error("set_nonblocking(listen) failed"); + } + + // Read back the actual bound port (supports ephemeral port 0, v4 and v6). + sockaddr_storage bound{}; + socklen_t blen = sizeof(bound); + if ( + getsockname(listen_fd, reinterpret_cast(&bound), &blen) == 0) + { + if (bound.ss_family == AF_INET6) + { + bound_port = + ntohs(reinterpret_cast(&bound)->sin6_port); + } + else + { + bound_port = ntohs(reinterpret_cast(&bound)->sin_port); + } + } + } + + OpenSSLServer(const OpenSSLServer&) = delete; + OpenSSLServer& operator=(const OpenSSLServer&) = delete; + OpenSSLServer(OpenSSLServer&&) = delete; + OpenSSLServer& operator=(OpenSSLServer&&) = delete; + + ~OpenSSLServer() + { + stop(); + cleanup(); + } + + uint16_t port() const + { + return bound_port; + } + + void start() + { + std::lock_guard guard(lifecycle_mutex); + if (started) + { + return; + } + + // Mark the server started before any handle is created. If one of the + // steps below throws, the handles which were already created are still + // registered with the loop and must be closed, and stop() is a no-op + // unless `started` is set. + torn_down = false; + stopping = false; + started = true; + listening = false; + + listen_poll = new_handle(); + listen_poll->data = this; + int rc = uv_poll_init_socket(loop, listen_poll, listen_fd); + if (rc != 0) + { + throw std::runtime_error( + std::string("uv_poll_init_socket(listen) failed: ") + + uv_strerror(rc)); + } + + wake_handle = new_handle(); + wake_handle->data = this; + rc = uv_async_init(loop, wake_handle, on_wake); + if (rc != 0) + { + throw std::runtime_error( + std::string("uv_async_init failed: ") + uv_strerror(rc)); + } + + if (idle_timeout.has_value()) + { + idle_timer = new_handle(); + idle_timer->data = this; + rc = uv_timer_init(loop, idle_timer); + if (rc != 0) + { + throw std::runtime_error( + std::string("uv_timer_init failed: ") + uv_strerror(rc)); + } + rc = uv_timer_start( + idle_timer, + on_idle_timer, + idle_sweep_interval_ms, + idle_sweep_interval_ms); + if (rc != 0) + { + throw std::runtime_error( + std::string("uv_timer_start failed: ") + uv_strerror(rc)); + } + } + + // Only actually listens if this interface can serve a connection now; a + // TLS interface still waiting for its certificate stays bound but + // unlistening until set_server_cert() supplies one. + begin_listening(); + + if (inbound_admission != nullptr) + { + // Woken when the node drops back under its inbound budget, so that + // this interface re-arms its reads even if the bytes which freed the + // budget belonged to a different one. + admission_token = + inbound_admission->register_waker([weak = weak_from_this()]() { + if (auto self = weak.lock()) + { + self->wake(); + } + }); + } + } + + // Tear the server down. Idempotent, and safe to call from the destructor. + // + // The teardown itself must run where it cannot race the loop, and that is + // not something this class can work out for itself: a loop which is + // running only reveals itself when it first invokes a callback, and for an + // interface which has seen no connections that may never happen. So the + // caller states it. + // + // `loop_state` == Running: another thread is running the loop, so the + // teardown is posted to it and this blocks until it has run. + // + // `loop_state` == NotRunning: nothing is running the loop and nothing + // will - node startup failed before the event loop was entered, or this is + // a test which never started one. The teardown runs inline, which is safe + // precisely because no other thread can be inside the loop, and returns + // without waiting: the handles own themselves, so their close callbacks + // simply never run. + void stop(LoopState loop_state = LoopState::NotRunning) + { + std::unique_lock lock(lifecycle_mutex); + if (!started || torn_down) + { + return; + } + + if (admission_token.has_value()) + { + inbound_admission->unregister_waker(*admission_token); + admission_token.reset(); + } + + if (loop_state == LoopState::NotRunning) + { + lock.unlock(); + tear_down_without_loop(); + return; + } + + if (!stopping) + { + stopping = true; + // tear_down_on_loop() clears wake_handle under this same lock before + // closing it, so this cannot signal a handle which is already closing. + if (wake_handle != nullptr) + { + (void)uv_async_send(wake_handle); + } + } + + // The loop performs the teardown. Keep servicing the task board while + // waiting, because an in-flight connection worker may need to complete + // before the loop can finish with it, and this thread may be one of the + // few able to run it. + while (!torn_down) + { + lock.unlock(); + auto task = ccf::tasks::get_main_job_board().get_task(); + if (task != nullptr) + { + ccf::tasks::try_do_task(*task); + } + lock.lock(); + if (!torn_down && task == nullptr) + { + teardown_cv.wait_for(lock, std::chrono::milliseconds(1)); + } + } + } + + // Thread-safe. Queue plaintext to be encrypted and written to `conn_id`. + // The buffer is taken rather than copied, so a response of any size crosses + // this boundary without another allocation. + void send(::tcp::ConnID conn_id, std::vector&& data) + { + { + std::lock_guard g(out_mutex); + pending_out.push_back({conn_id, std::move(data), false}); + } + wake(); + } + + // Thread-safe. Queue a copy of plaintext to be encrypted and written to + // `conn_id`. + void send(::tcp::ConnID conn_id, const uint8_t* data, size_t len) + { + send(conn_id, std::vector(data, data + len)); + } + + // Thread-safe. Request that `conn_id` be torn down. + void close_connection(::tcp::ConnID conn_id) + { + { + std::lock_guard g(out_mutex); + pending_out.push_back({conn_id, {}, true}); + } + wake(); + } + + // Thread-safe. (Re)load the server certificate/key. Used for deferred cert + // (a node that learns the service cert after binding) and rotation; applies + // to connections accepted after it takes effect on the loop thread. + void set_server_cert( + const std::string& cert_pem, const std::string& key_pem) + { + { + std::lock_guard g(out_mutex); + pending_certs.emplace_back(cert_pem, key_pem); + } + wake(); + } + + private: + void cleanup() + { + if (listen_fd >= 0) + { + ::close(listen_fd); + listen_fd = -1; + } + ctx.reset(); + } + }; +} diff --git a/src/host/tls/openssl_session_manager.h b/src/host/tls/openssl_session_manager.h new file mode 100644 index 000000000000..1de7ffa17385 --- /dev/null +++ b/src/host/tls/openssl_session_manager.h @@ -0,0 +1,273 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the Apache 2.0 License. +#pragma once + +// Bridges the OpenSSL-native transport (OpenSSLServer) to ccf::Session objects: +// +// * inbound plaintext from a connection -> ccf::Session::handle_incoming_data +// * ccf::Session output (via ccf::SessionWriter) -> OpenSSLServer::send, +// which encrypts and writes with backpressure +// * connection teardown -> the owning session is dropped +// +// One ccf::Session is created per connection by a caller-supplied factory (e.g. +// "make an HTTPServerSession for this interface"). Sessions are created lazily +// on first inbound data and removed on close. +// +// Note that admission control is deliberately *not* tied to the session: a +// connection is admitted (and counted) by the transport's OnAccept before any +// session exists, and released here in on_close, once per admitted connection. +// Tying it to the session would leave connections which complete the TLS +// handshake but never send a request entirely uncounted. +// +// Threading: OpenSSLServer invokes on_data on its TLS OrderedTasks worker and +// on_close on its loop thread. The session may dispatch again to its own +// OrderedTasks and reply via write_outbound from any worker. Every public +// method is therefore safe to call from any thread, and the connection map is +// guarded by a mutex. + +#include "ccf/node/session.h" +#include "host/tls/openssl_server.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace asynchost +{ + class OpenSSLSessionManager : public ccf::SessionWriter + { + public: + // Creates the protocol session for a freshly seen connection. `writer` is + // this manager - the session emits its (plaintext) output through it. + // `peer_cert` is the client certificate (DER) captured at handshake, for + // caller authentication. `soft_limited` is the admission decision taken + // when the connection was accepted. + using SessionFactory = std::function( + ::tcp::ConnID conn_id, + ccf::SessionWriter& writer, + std::vector peer_cert, + bool soft_limited)>; + + private: + std::shared_ptr server; + SessionFactory factory; + // Node-wide budget for inbound data which has been delivered to a session + // but not yet processed. Charged here rather than in the transport so that + // every path which drops data instead of delivering it is visible in one + // function. Null disables the accounting. + std::shared_ptr inbound_admission; + // Invoked when an admitted connection is torn down, so an owner can + // release whatever it reserved at accept time. Called on the loop thread + // from on_close, exactly once per admitted connection. + std::function on_connection_closed; + + struct ConnState + { + std::shared_ptr session; + // The session asked for the connection to be closed. The transport + // teardown is asynchronous, so bytes already in flight may still arrive; + // they are dropped rather than being used to build a replacement + // session for a connection which is going away. + bool closing = false; + // Bytes delivered to the session which it has not yet reported as + // processed. Released to the node-wide budget when this connection is + // torn down, so that a session which never reports (or whose queued work + // is cancelled) cannot strand part of the budget forever. + size_t inbound_outstanding = 0; + }; + + std::mutex conns_mutex; + std::unordered_map<::tcp::ConnID, ConnState> conns; + + void on_data( + ::tcp::ConnID conn_id, + std::vector data, + const std::vector& peer_cert, + bool soft_limited) + { + const size_t size = data.size(); + std::shared_ptr session; + { + std::lock_guard guard(conns_mutex); + // find() then insert, rather than operator[], so that first sight of a + // connection is an explicit case. An entry is only ever removed by + // on_close, and the transport delivers nothing for a connection after + // that, so an insert here is always a genuinely new connection and + // never a zombie whose inbound budget nothing would release. + auto it = conns.find(conn_id); + if (it == conns.end()) + { + it = conns.emplace(conn_id, ConnState{}).first; + } + auto& state = it->second; + if (state.closing) + { + return; + } + if (state.session == nullptr) + { + state.session = factory(conn_id, *this, peer_cert, soft_limited); + if (state.session == nullptr) + { + // Factory refused - tear the connection down. + state.closing = true; + server->close_connection(conn_id); + return; + } + } + session = state.session; + // Charged only now that the data is definitely being delivered. + state.inbound_outstanding += size; + } + + if (inbound_admission != nullptr) + { + inbound_admission->queued(size); + } + + session->handle_incoming_data(std::move(data)); + } + + void on_close(::tcp::ConnID conn_id) + { + size_t to_release = 0; + { + std::lock_guard guard(conns_mutex); + auto it = conns.find(conn_id); + if (it != conns.end()) + { + to_release = it->second.inbound_outstanding; + conns.erase(it); + } + } + + // The transport only tears a connection down once its worker has + // finished, so no further data can be delivered for it and anything + // still outstanding will never be reported as processed. + if (inbound_admission != nullptr) + { + inbound_admission->consumed(to_release); + } + + // Unconditional: the connection, not the session, is what was reserved + // at accept time, and a connection which never sent a request has no + // session to key off. + if (on_connection_closed) + { + on_connection_closed(conn_id); + } + } + + public: + // Takes the transport's own Config verbatim, so there is a single place + // where a listening interface is described. + OpenSSLSessionManager( + OpenSSLServer::Config config, + SessionFactory factory_, + std::function on_connection_closed_ = {}, + OpenSSLServer::OnAccept on_accept = {}) : + factory(std::move(factory_)), + inbound_admission(config.inbound_admission), + on_connection_closed(std::move(on_connection_closed_)) + { + server = std::make_shared( + std::move(config), + [this]( + ::tcp::ConnID id, + std::vector data, + const std::vector& peer_cert, + bool soft_limited) { + on_data(id, std::move(data), peer_cert, soft_limited); + }, + [this](::tcp::ConnID id) { on_close(id); }, + std::move(on_accept)); + } + + // The session for `id`, or nullptr. Thread-safe. + std::shared_ptr get_session(::tcp::ConnID id) + { + std::lock_guard guard(conns_mutex); + auto it = conns.find(id); + return it == conns.end() ? nullptr : it->second.session; + } + + // (Re)load this interface's server certificate (deferred cert / rotation). + void set_server_cert( + const std::string& cert_pem, const std::string& key_pem) + { + server->set_server_cert(cert_pem, key_pem); + } + + void start() + { + server->start(); + } + + void stop( + OpenSSLServer::LoopState loop_state = + OpenSSLServer::LoopState::NotRunning) + { + server->stop(loop_state); + } + + uint16_t port() const + { + return server->port(); + } + + // ccf::SessionWriter (callable from any thread). + + void write_outbound(::tcp::ConnID id, std::vector&& data) override + { + server->send(id, std::move(data)); + } + + void close_socket(::tcp::ConnID id) override + { + { + std::lock_guard guard(conns_mutex); + auto it = conns.find(id); + if (it != conns.end()) + { + it->second.closing = true; + it->second.session.reset(); + } + } + // No release here: the connection's reservation is released by on_close + // when the transport has actually torn it down, exactly once. + server->close_connection(id); + } + + void inbound_consumed(::tcp::ConnID id, size_t bytes) override + { + size_t to_release = 0; + { + std::lock_guard guard(conns_mutex); + auto it = conns.find(id); + if (it == conns.end()) + { + // Already torn down, and on_close released everything which was + // outstanding for it. + return; + } + // Clamped rather than trusted: releasing more than was charged would + // underflow the node-wide counter and pause every interface's reads + // permanently. + to_release = std::min(bytes, it->second.inbound_outstanding); + it->second.inbound_outstanding -= to_release; + } + + if (inbound_admission != nullptr) + { + inbound_admission->consumed(to_release); + } + } + }; +} diff --git a/src/host/udp.h b/src/host/udp.h deleted file mode 100644 index 134593f29927..000000000000 --- a/src/host/udp.h +++ /dev/null @@ -1,543 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the Apache 2.0 License. -#pragma once - -#include "before_io.h" -#include "ccf/pal/locking.h" -#include "dns.h" -#include "ds/internal_logger.h" -#include "ds/pending_io.h" -#include "proxy.h" -#include "socket.h" - -#include - -namespace asynchost -{ - // NOLINTBEGIN(cppcoreguidelines-virtual-class-destructor) - class UDPImpl; - using UDP = proxy_ptr; - - /// For now this is server only, as we have no immediate plans to - /// create node-to-node UDP channels or use UDP for REST between nodes - class UDPImpl : public with_uv_handle - { - private: - friend class close_ptr; - - static constexpr int backlog = 128; - static constexpr size_t max_read_size = 16384; - - // Each uv iteration, read only a capped amount from all sockets. - static constexpr auto max_read_quota = max_read_size * 4; - static size_t remaining_read_quota; - - // This is a simplified version of the state machine for QUIC that - // mostly follows plain UDP state. We should add more when we need - // for QUIC, not predict complexity prematurely. - enum Status : uint8_t - { - // Starting state + failure recovery (if any) - FRESH, - // DNS::resolve - RESOLVING, - RESOLVING_FAILED, - // uv_udp_recv_start <-> on_read - READING, - READING_FAILED, - // uv_udp_send has no state (it's synchronous) - WRITING_FAILED, - // There is no connected/reconnect/disconnect - }; - - /// Current status - Status status{FRESH}; - /// Callback behaviour from user - std::unique_ptr> behaviour; - - using PendingWrites = std::vector>; - /// Writes sent before writing socket is read - PendingWrites pending_writes; - - /// Host to bind the server - std::string host; - /// Port to bind the server - std::string port; - /// Listening name - std::optional listen_name = std::nullopt; - - /// Base address (head of linked list) - addrinfo* addr_base = nullptr; - /// Current address (node in the list that resolved first) - addrinfo* addr_current = nullptr; - - [[nodiscard]] bool port_assigned() const - { - return port != "0"; - } - - [[nodiscard]] std::string get_address_name() const - { - const std::string port_suffix = - port_assigned() ? fmt::format(":{}", port) : ""; - - if (addr_current != nullptr && addr_current->ai_family == AF_INET6) - { - return fmt::format("[{}]{}", host, port_suffix); - } - - return fmt::format("{}{}", host, port_suffix); - } - - UDPImpl() - { - if (!init()) - { - throw std::logic_error("uv UDP initialization failed"); - } - - uv_handle.data = this; - } - - ~UDPImpl() override - { - { - std::unique_lock guard(pending_resolve_requests_mtx); - for (const auto& req : pending_resolve_requests) - { - // The UV request objects can stay, but if there are any references - // to `this` left, we need to remove them. - if (req->data == this) - { - req->data = nullptr; - } - } - } - if (addr_base != nullptr) - { - uv_freeaddrinfo(addr_base); - } - } - - public: - static void reset_read_quota() - { - remaining_read_quota = max_read_quota; - } - - void set_behaviour(std::unique_ptr> b) - { - behaviour = std::move(b); - } - - [[nodiscard]] std::string get_host() const - { - return host; - } - - [[nodiscard]] std::string get_port() const - { - return port; - } - - [[nodiscard]] std::optional get_listen_name() const - { - return listen_name; - } - - /// Listen to packets on host:port - bool listen( - const std::string& host_, - const std::string& port_, - const std::optional& name = std::nullopt) - { - listen_name = name; - auto name_str = name.has_value() ? name.value() : ""; - LOG_TRACE_FMT("UDP listen on {}:{} [{}]", host_, port_, name_str); - return resolve(host_, port_, false); - } - - /// Start the service via behaviour (register on ringbuffer, etc) - void start(int64_t id) - { - behaviour->on_start(id); - } - - bool connect(const std::string& /*host_*/, const std::string& /*port_*/) - { - LOG_TRACE_FMT("UDP dummy connect to {}:{}", host, port); - return true; - } - - bool write(size_t len, const uint8_t* data, sockaddr addr) - { - auto* req = new uv_udp_send_t; // NOLINT(cppcoreguidelines-owning-memory) - auto* copy = new char[len]; // NOLINT(cppcoreguidelines-owning-memory) - if (data != nullptr) - { - memcpy(copy, data, len); - } - req->data = copy; - - switch (status) - { - // Handles unbound or in unknown state - case RESOLVING: - case RESOLVING_FAILED: - case READING_FAILED: - case WRITING_FAILED: - { - pending_writes.emplace_back(req, len, addr, free_write); - break; - } - - // Both read and write handles have been bound here - case READING: - { - auto [h, p] = addr_to_str(&addr); - LOG_TRACE_FMT("UDP write addr: {}:{}", h, p); - return send_write(req, len, &addr); - } - - case FRESH: - default: - { - free_write(req); - throw std::logic_error( - fmt::format("Unexpected status during write: {}", status)); - } - } - - return true; - } - - private: - /// Initializes both handles (recv/send) - bool init() - { - assert_status(FRESH, FRESH); - - int rc = 0; - LOG_TRACE_FMT("UDP init"); - if ((rc = uv_udp_init(uv_default_loop(), &uv_handle)) < 0) - { - LOG_FAIL_FMT("uv_udp_init failed on recv handle: {}", uv_strerror(rc)); - return false; - } - - return true; - } - - bool send_write(uv_udp_send_t* req, size_t len, const struct sockaddr* addr) - { - auto* copy = static_cast(req->data); - - uv_buf_t buf; - buf.base = copy; - buf.len = len; - - int rc = 0; - - auto [h, p] = addr_to_str(addr); - LOG_TRACE_FMT("UDP send_write addr: {}:{}", h, p); - std::string data(copy, len); - LOG_TRACE_FMT("UDP send_write [{}]", data); - if ((rc = uv_udp_send(req, &uv_handle, &buf, 1, addr, on_write)) < 0) - { - free_write(req); - LOG_FAIL_FMT("uv_write failed: {}", uv_strerror(rc)); - status = WRITING_FAILED; - behaviour->on_disconnect(); - return false; - } - - return true; - } - - void update_resolved_address(int address_family, sockaddr* sa) - { - auto [h, p] = addr_to_str(sa, address_family); - host = h; - port = p; - LOG_TRACE_FMT("UDP update address to {}:{}", host, port); - } - - void resolved() - { - int rc = 0; - - LOG_TRACE_FMT("UDP bind to {}:{}", host, port); - while (addr_current != nullptr) - { - update_resolved_address(addr_current->ai_family, addr_current->ai_addr); - - if ((rc = uv_udp_bind(&uv_handle, addr_current->ai_addr, 0)) < 0) - { - addr_current = addr_current->ai_next; - LOG_FAIL_FMT( - "uv_udp_bind failed on {}: {}", - get_address_name(), - uv_strerror(rc)); - continue; - } - - // If bound on port 0 (ie - asking the OS to assign a port), then we - // need to call uv_udp_getsockname to retrieve the bound port - // (addr_current will not contain it) - if (!port_assigned()) - { - sockaddr_storage sa_storage{}; - auto* const sa = reinterpret_cast(&sa_storage); - int sa_len = sizeof(sa_storage); - if ((rc = uv_udp_getsockname(&uv_handle, sa, &sa_len)) != 0) - { - LOG_FAIL_FMT("uv_udp_getsockname failed: {}", uv_strerror(rc)); - } - update_resolved_address(addr_current->ai_family, sa); - } - - LOG_TRACE_FMT("UDP to call on_listening"); - - behaviour->on_listening(host, port); - - assert_status(RESOLVING, READING); - read_start(); - return; - } - - status = RESOLVING_FAILED; - - // This should show even when verbose logs are off - LOG_INFO_FMT( - "Unable to connect: all resolved addresses failed: {}:{}", host, port); - } - - void assert_status(Status from, Status to) - { - if (status != from) - { - throw std::logic_error(fmt::format( - "Trying to transition from {} to {} but current status is {}", - from, - to, - status)); - } - - status = to; - } - - bool resolve( - const std::string& host_, const std::string& port_, bool async = true) - { - host = host_; - port = port_; - - LOG_TRACE_FMT("UDP resolve {}:{}", host, port); - if (addr_base != nullptr) - { - uv_freeaddrinfo(addr_base); - addr_base = nullptr; - addr_current = nullptr; - } - - assert_status(FRESH, RESOLVING); - - if (!DNS::resolve(host, port, this, on_resolved, async)) - { - LOG_DEBUG_FMT("Resolving '{}' failed", host); - status = RESOLVING_FAILED; - return false; - } - - return true; - } - - static void on_resolved(uv_getaddrinfo_t* req, int rc, struct addrinfo* res) - { - std::unique_lock guard(pending_resolve_requests_mtx); - pending_resolve_requests.erase(req); - - LOG_TRACE_FMT("UDP on_resolve static"); - if (req->data != nullptr) - { - static_cast(req->data)->on_resolved(req, rc); - } - else - { - // The UDPImpl that submitted the request has been destroyed, but we - // need to clean up the request object. - uv_freeaddrinfo(res); - delete req; // NOLINT(cppcoreguidelines-owning-memory) - } - } - - void on_resolved(uv_getaddrinfo_t* req, int rc) - { - LOG_TRACE_FMT("UDP on_resolve dynamic"); - // It is possible that on_resolved is triggered after there has been a - // request to close uv_handle. In this scenario, we should not try to - // do anything with the handle and return immediately (otherwise, - // uv_close cb will abort). - if (uv_is_closing(reinterpret_cast(&uv_handle)) != 0) - { - LOG_DEBUG_FMT("on_resolved: closing"); - uv_freeaddrinfo(req->addrinfo); - delete req; // NOLINT(cppcoreguidelines-owning-memory) - return; - } - - if (rc < 0) - { - status = RESOLVING_FAILED; - LOG_DEBUG_FMT("UDP resolve failed: {}", uv_strerror(rc)); - behaviour->on_resolve_failed(); - } - else - { - addr_base = req->addrinfo; - addr_current = addr_base; - - LOG_TRACE_FMT("UDP to call resolved"); - resolved(); - } - - delete req; // NOLINT(cppcoreguidelines-owning-memory) - } - - void push_pending_writes() - { - for (auto& w : pending_writes) - { - auto [h, p] = addr_to_str(&w.addr); - LOG_TRACE_FMT("UDP pending_writes addr: {}:{}", h, p); - send_write(w.req, w.len, &w.addr); - w.req = nullptr; - } - - PendingWrites().swap(pending_writes); - } - - void read_start() - { - int rc = 0; - - LOG_TRACE_FMT("UDP read start"); - if ((rc = uv_udp_recv_start(&uv_handle, on_alloc, on_read)) < 0) - { - status = READING_FAILED; - LOG_FAIL_FMT("uv_udp_read_start failed: {}", uv_strerror(rc)); - behaviour->on_disconnect(); - } - } - - static void on_alloc( - uv_handle_t* handle, size_t suggested_size, uv_buf_t* buf) - { - static_cast(handle->data)->on_alloc(suggested_size, buf); - } - - void on_alloc(size_t suggested_size, uv_buf_t* buf) - { - auto alloc_size = std::min(suggested_size, max_read_size); - - alloc_size = std::min(alloc_size, remaining_read_quota); - remaining_read_quota -= alloc_size; - LOG_TRACE_FMT( - "Allocating {} bytes for UDP read ({} of quota remaining)", - alloc_size, - remaining_read_quota); - - // NOLINTNEXTLINE(cppcoreguidelines-owning-memory) - buf->base = new char[alloc_size]; - buf->len = alloc_size; - } - - void on_free(const uv_buf_t* buf) - { - delete[] buf->base; // NOLINT(cppcoreguidelines-owning-memory) - } - - static void on_read( - uv_udp_t* handle, - ssize_t sz, - const uv_buf_t* buf, - const struct sockaddr* addr, - unsigned flags) - { - static_cast(handle->data)->on_read(sz, buf, addr, flags); - } - - void on_read( - ssize_t sz, - const uv_buf_t* buf, - const struct sockaddr* addr, - unsigned /*flags*/) - { - if (sz == 0) - { - on_free(buf); - return; - } - - if (sz == UV_ENOBUFS) - { - LOG_DEBUG_FMT("UDP on_read reached allocation quota"); - on_free(buf); - return; - } - - if (sz < 0) - { - on_free(buf); - LOG_DEBUG_FMT("UDP on_read: {}", uv_strerror(static_cast(sz))); - behaviour->on_disconnect(); - return; - } - - auto [h, p] = addr_to_str(addr); - LOG_TRACE_FMT("UDP on_read addr: {}:{}", h, p); - - auto* b = reinterpret_cast(buf->base); - std::string data(reinterpret_cast(b), sz); - LOG_TRACE_FMT("UDP on_read [{}]", data); - behaviour->on_read(static_cast(sz), b, *addr); - - if (b != nullptr) - { - on_free(buf); - } - } - - static void on_write(uv_udp_send_t* req, int /*status*/) - { - free_write(req); - } - - static void free_write(uv_udp_send_t* req) - { - if (req == nullptr) - { - return; - } - - auto* copy = static_cast(req->data); - delete[] copy; // NOLINT(cppcoreguidelines-owning-memory) - delete req; // NOLINT(cppcoreguidelines-owning-memory) - } - }; - - // NOLINTEND(cppcoreguidelines-virtual-class-destructor) - - class ResetUDPReadQuotaImpl - { - public: - ResetUDPReadQuotaImpl() = default; - - void before_io() - { - UDPImpl::reset_read_quota(); - } - }; - - using ResetUDPReadQuota = proxy_ptr>; -} diff --git a/src/http/curl.h b/src/http/curl.h index ba5af07efcfd..1e03f3da1de2 100644 --- a/src/http/curl.h +++ b/src/http/curl.h @@ -82,6 +82,19 @@ namespace ccf::curl code == CURLE_HTTP2 || code == CURLE_HTTP2_STREAM; } + inline bool is_retryable_join_error( + CURLcode code, bool has_received_pending_join_response) + { + // CURLE_SSL_CONNECT_ERROR is intentionally not generally transient: it + // covers permanent TLS configuration and protocol errors as well as a peer + // disappearing during the handshake. Once this joiner has received a + // PENDING response, the same TLS configuration and pinned service identity + // have already succeeded, so a later handshake failure may be retried while + // that target changes role. + return is_transient_transport_error(code) || + (has_received_pending_join_response && code == CURLE_SSL_CONNECT_ERROR); + } + class UniqueCURL { private: diff --git a/src/http/http2_session.h b/src/http/http2_session.h index bffa330d4bfb..71e542314d09 100644 --- a/src/http/http2_session.h +++ b/src/http/http2_session.h @@ -12,8 +12,6 @@ namespace http { - using HTTP2Session = ccf::EncryptedSession; - struct HTTP2SessionContext : public ccf::SessionContext { int32_t stream_id; @@ -172,7 +170,7 @@ namespace http } }; - class HTTP2ServerSession : public HTTP2Session, + class HTTP2ServerSession : public ccf::PlaintextSession, public http::RequestProcessor, public ccf::http::HTTPResponder { @@ -200,7 +198,7 @@ namespace http it, stream_id, std::make_shared( - session_id, tls_io->peer_cert(), interface_id, stream_id)); + session_id, peer_cert(), interface_id, stream_id)); } return it->second; @@ -241,11 +239,11 @@ namespace http std::shared_ptr rpc_map_, int64_t session_id_, ccf::ListenInterfaceID interface_id_, - ringbuffer::AbstractWriterFactory& writer_factory, - std::unique_ptr ctx, + ccf::SessionWriter& writer, + std::vector peer_cert, const ccf::http::ParserConfiguration& configuration, const std::shared_ptr& error_reporter_) : - HTTP2Session(session_id_, writer_factory, std::move(ctx)), + ccf::PlaintextSession(session_id_, writer, std::move(peer_cert)), server_parser( std::make_shared(*this, configuration)), rpc_map(std::move(rpc_map_)), @@ -433,4 +431,5 @@ namespace http ->set_on_stream_close_callback(cb); } }; + } diff --git a/src/http/http_parser.h b/src/http/http_parser.h index 1bfa37afe2aa..ae182397e6e7 100644 --- a/src/http/http_parser.h +++ b/src/http/http_parser.h @@ -5,7 +5,7 @@ #include "ccf/ds/hex.h" #include "ccf/http_configuration.h" #include "ccf/http_query.h" -#include "enclave/tls_session.h" +#include "ds/internal_logger.h" #include "http/http_exceptions.h" #include "http_builder.h" #include "http_proc.h" diff --git a/src/http/http_proc.h b/src/http/http_proc.h index 74e93e3f2c4e..d067d3672848 100644 --- a/src/http/http_proc.h +++ b/src/http/http_proc.h @@ -2,7 +2,7 @@ // Licensed under the Apache 2.0 License. #pragma once -#include "enclave/tls_session.h" +#include "ds/internal_logger.h" #include "http2_types.h" #include "http_builder.h" diff --git a/src/http/http_session.h b/src/http/http_session.h index 0b22cd89096f..6a1da6a72171 100644 --- a/src/http/http_session.h +++ b/src/http/http_session.h @@ -9,12 +9,11 @@ #include "http_parser.h" #include "http_responder.h" #include "http_rpc_context.h" +#include "node/commit_callback_subsystem.h" namespace http { - using HTTPSession = ccf::EncryptedSession; - - class HTTPServerSession : public HTTPSession, + class HTTPServerSession : public ccf::PlaintextSession, public http::RequestProcessor, public ccf::http::HTTPResponder { @@ -33,12 +32,12 @@ namespace http std::shared_ptr rpc_map_, ::tcp::ConnID session_id_, ccf::ListenInterfaceID interface_id_, - ringbuffer::AbstractWriterFactory& writer_factory, - std::unique_ptr ctx, + ccf::SessionWriter& writer, + std::vector peer_cert, const ccf::http::ParserConfiguration& configuration, const std::shared_ptr& error_reporter_, const std::shared_ptr& commit_callbacks_) : - HTTPSession(session_id_, writer_factory, std::move(ctx)), + ccf::PlaintextSession(session_id_, writer, std::move(peer_cert)), request_parser(*this, configuration), rpc_map(std::move(rpc_map_)), error_reporter(error_reporter_), @@ -139,7 +138,7 @@ namespace http if (session_ctx == nullptr) { session_ctx = std::make_shared( - session_id, tls_io->peer_cert(), interface_id); + session_id, peer_cert(), interface_id); } std::shared_ptr rpc_ctx = nullptr; diff --git a/src/http/test/curl_test.cpp b/src/http/test/curl_test.cpp index f9ef7993880b..2939a513e52c 100644 --- a/src/http/test/curl_test.cpp +++ b/src/http/test/curl_test.cpp @@ -58,10 +58,10 @@ TEST_CASE("is_transient_transport_error classifies curl errors") CHECK(ccf::curl::is_transient_transport_error(code)); } - // Errors that must be treated as fatal (never retried): TLS/certificate - // failures, application-level errors, and our own response size-cap - // rejection (CURLE_WRITE_ERROR). CURLE_OK and CURLE_ABORTED_BY_CALLBACK are - // not transport errors either. + // Errors that must be treated as fatal (never retried): explicit certificate + // or local TLS configuration failures, application-level errors, and our own + // response size-cap rejection (CURLE_WRITE_ERROR). CURLE_OK and + // CURLE_ABORTED_BY_CALLBACK are not transport errors either. const std::vector fatal = { CURLE_OK, CURLE_PEER_FAILED_VERIFICATION, @@ -81,6 +81,20 @@ TEST_CASE("is_transient_transport_error classifies curl errors") } } +TEST_CASE("is_retryable_join_error narrows generic TLS handshake failures") +{ + CHECK_FALSE( + ccf::curl::is_retryable_join_error(CURLE_SSL_CONNECT_ERROR, false)); + CHECK(ccf::curl::is_retryable_join_error(CURLE_SSL_CONNECT_ERROR, true)); + + CHECK(ccf::curl::is_retryable_join_error(CURLE_COULDNT_CONNECT, false)); + CHECK_FALSE( + ccf::curl::is_retryable_join_error(CURLE_PEER_FAILED_VERIFICATION, true)); + CHECK_FALSE( + ccf::curl::is_retryable_join_error(CURLE_SSL_CACERT_BADFILE, true)); + CHECK_FALSE(ccf::curl::is_retryable_join_error(CURLE_SSL_CERTPROBLEM, true)); +} + TEST_CASE("RequestBody supports replay") { const std::vector expected = {1, 2, 3, 4}; diff --git a/src/js/extensions/ccf/crypto.cpp b/src/js/extensions/ccf/crypto.cpp index 02eeb642f1ca..f4ed73ced590 100644 --- a/src/js/extensions/ccf/crypto.cpp +++ b/src/js/extensions/ccf/crypto.cpp @@ -9,6 +9,7 @@ #include "ccf/crypto/entropy.h" #include "ccf/crypto/hmac.h" #include "ccf/crypto/key_wrap.h" +#include "ccf/crypto/openssl/openssl_wrappers.h" #include "ccf/crypto/rsa_key_pair.h" #include "ccf/crypto/sha256.h" #include "ccf/crypto/verifier.h" @@ -16,7 +17,6 @@ #include "ccf/js/core/context.h" #include "ds/internal_logger.h" #include "js/checks.h" -#include "tls/ca.h" #include @@ -310,7 +310,16 @@ namespace ccf::js::extensions try { - ::tls::CA ca(pem.value()); + ccf::crypto::OpenSSL::Unique_BIO bio( + pem.value().data(), pem.value().size()); + ccf::crypto::OpenSSL::Unique_X509 cert(bio, true); + if (cert == nullptr) + { + LOG_DEBUG_FMT( + "isValidX509Bundle: {}", + ccf::crypto::OpenSSL::error_string(ERR_get_error())); + return ccf::js::core::constants::False; + } } catch (const std::runtime_error& e) { diff --git a/src/node/http_node_client.h b/src/node/http_node_client.h index 84e43c2cb0c6..b2edae4182d3 100644 --- a/src/node/http_node_client.h +++ b/src/node/http_node_client.h @@ -2,6 +2,7 @@ // Licensed under the Apache 2.0 License. #pragma once +#include "http/http_rpc_context.h" #include "node/node_client.h" #include diff --git a/src/node/node_state.h b/src/node/node_state.h index eb4f64e7a911..3d8da7959f51 100644 --- a/src/node/node_state.h +++ b/src/node/node_state.h @@ -27,7 +27,7 @@ #include "ds/files.h" #include "ds/internal_logger.h" #include "ds/state_machine.h" -#include "enclave/rpc_sessions.h" +#include "enclave/abstract_rpc_sessions.h" #include "encryptor.h" #include "history.h" #include "http/curl.h" @@ -448,7 +448,7 @@ namespace ccf std::shared_ptr> cmd_forwarder; std::shared_ptr commit_callbacks = nullptr; std::shared_ptr signature_cache = nullptr; - std::shared_ptr rpcsessions; + std::shared_ptr rpcsessions; std::shared_ptr history; std::shared_ptr encryptor; @@ -506,6 +506,11 @@ namespace ccf // and so must not take NodeState::lock. std::atomic join_request_in_flight = false; + // A successful PENDING response proves that this joiner's TLS settings and + // pinned service identity are valid. A later generic TLS handshake failure + // can then be retried safely while the target changes role. + bool has_received_pending_join_response = false; + // Number of times we have fetched the latest snapshot from the primary size_t join_fetch_count = 0; @@ -785,7 +790,7 @@ namespace ccf NodeState( ringbuffer::AbstractWriterFactory& writer_factory, NetworkState& network, - std::shared_ptr rpcsessions, + std::shared_ptr rpcsessions, ccf::crypto::CurveID curve_id_) : sm("NodeState", NodeStartupState::uninitialized), curve_id(curve_id_), @@ -1316,12 +1321,10 @@ namespace ccf // The service certificate is the sole trust anchor for the join // connection. CURLOPT_CAINFO_BLOB installs it and CURLOPT_CAPATH=nullptr - // prevents any fallback to the system CA store, so the set of accepted - // certificate authorities is identical to the legacy tls::CA path. The - // joining node presents its self-signed node certificate for mutual TLS - // (it is not yet endorsed at join time). CURLOPT_SSL_VERIFYHOST=2 - // additionally checks that the target certificate matches the address we - // connected to. + // prevents any fallback to the system CA store. The joining node presents + // its self-signed node certificate for mutual TLS (it is not yet endorsed + // at join time). CURLOPT_SSL_VERIFYHOST=2 additionally checks that the + // target certificate matches the address we connected to. ccf::curl::UniqueCURL curl_handle; curl_handle.set_opt(CURLOPT_SSL_VERIFYPEER, 1L); curl_handle.set_opt(CURLOPT_SSL_VERIFYHOST, 2L); @@ -1431,10 +1434,11 @@ namespace ccf // The legacy httpclient path silently dropped a failed // connection and relied on the periodic join timer to retry // when the target could not yet be reached, while treating TLS - // handshake failures (e.g. an untrusted service certificate) as - // fatal. Preserve both behaviours: transient transport errors - // are retried, everything else is fatal. - if (ccf::curl::is_transient_transport_error(curl_response)) + // explicit certificate verification/loading failures as fatal. + // Preserve both behaviours: transient transport errors are + // retried, everything else is fatal. + if (ccf::curl::is_retryable_join_error( + curl_response, has_received_pending_join_response)) { LOG_INFO_FMT( "Transient error contacting {} to join: {} ({}). The join " @@ -1761,6 +1765,7 @@ namespace ccf } else if (resp.node_status == NodeStatus::PENDING) { + has_received_pending_join_response = true; LOG_INFO_FMT( "Node {} is waiting for votes of members to be trusted", self); diff --git a/src/node/rpc/custom_protocol_subsystem.h b/src/node/rpc/custom_protocol_subsystem.h index ef93a25d1d5c..892552026f4a 100644 --- a/src/node/rpc/custom_protocol_subsystem.h +++ b/src/node/rpc/custom_protocol_subsystem.h @@ -40,12 +40,12 @@ namespace ccf std::shared_ptr create_session( const std::string& protocol_name, ccf::tls::ConnID conn_id, - const std::unique_ptr&& ctx) override + ccf::SessionWriter& writer) override { auto it = session_creation_functions.find(protocol_name); if (it != session_creation_functions.end()) { - return it->second(conn_id, std::move(ctx)); + return it->second(conn_id, writer); } throw std::logic_error(fmt::format( "Session creation function for protocol '{}' has not been installed", diff --git a/src/quic/quic_session.h b/src/quic/quic_session.h deleted file mode 100644 index 3d49d92ab040..000000000000 --- a/src/quic/quic_session.h +++ /dev/null @@ -1,456 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the Apache 2.0 License. -#pragma once - -#include "ds/internal_logger.h" -#include "ds/messaging.h" -#include "ds/pending_io.h" -#include "ds/ring_buffer.h" -#include "enclave/session.h" -#include "udp/msg_types.h" - -#include - -namespace quic -{ - class QUICSession : public ccf::Session, - public std::enable_shared_from_this - { - protected: - ringbuffer::WriterPtr to_host; - ccf::tls::ConnID session_id; - - std::shared_ptr task_scheduler; - - enum Status : std::uint8_t - { - handshake, - ready, - closed, - authfail, - error - }; - - Status get_status() const - { - return status; - } - - using PendingBuffer = PendingIO; - using PendingList = std::vector; - PendingList pending_writes; - PendingList pending_reads; - - private: - // Decrypted data - std::vector read_buffer; - - Status status = handshake; - - public: - QUICSession( - int64_t session_id_, ringbuffer::AbstractWriterFactory& writer_factory_) : - to_host(writer_factory_.create_writer_to_outside()), - session_id(session_id_) - { - task_scheduler = ccf::tasks::OrderedTasks::create( - ccf::tasks::get_main_job_board(), - fmt::format("Session {}", session_id)); - } - - ~QUICSession() override - { - task_scheduler->cancel_task(); - // RINGBUFFER_WRITE_MESSAGE(quic::quic_closed, to_host, session_id); - } - - std::string hostname() - { - return {}; - } - - std::vector peer_cert() - { - return {}; - } - - // Returns count N of bytes read, which will be the first N bytes of data, - // up to a maximum of size. If exact is true, will only return either size - // or 0 (when size bytes are not currently available). data may be accessed - // beyond N during operation, up to size, but only the first N should be - // used by caller. - size_t read(uint8_t* data, size_t size, sockaddr addr, bool exact = false) - { - LOG_TRACE_FMT("Requesting up to {} bytes", size); - - // This will return empty if the connection isn't - // ready, but it will not block on the handshake. - do_handshake(); - - if (status != ready) - { - return 0; - } - - // Send pending writes. - flush(); - - size_t offset = 0; - - if (!read_buffer.empty()) - { - LOG_TRACE_FMT( - "Have existing read_buffer of size: {}", read_buffer.size()); - offset = std::min(size, read_buffer.size()); - ::memcpy(data, read_buffer.data(), offset); - - if (offset < read_buffer.size()) - { - read_buffer.erase(read_buffer.begin(), read_buffer.begin() + offset); - } - else - { - read_buffer.clear(); - } - - if (offset == size) - { - return size; - } - - // NB: If we continue past here, read_buffer is empty - } - - // This will need to be handled by the actual QUIC stack - auto r = handle_recv(data + offset, size - offset, addr); - LOG_TRACE_FMT("quic read returned: {}", r); - - if (r < 0) - { - LOG_TRACE_FMT("QUIC {} error on read", session_id); - stop(error); - return 0; - } - - auto total = r + offset; - - // We read _some_ data but not enough, and didn't get - // WANT_READ. Probably hit an internal size limit - try - // again - if (exact && (total < size)) - { - LOG_TRACE_FMT( - "Asked for exactly {}, received {}, retrying", size, total); - read_buffer.insert(read_buffer.end(), data, data + total); - return read(data, size, addr, exact); - } - - return total; - } - - void recv_buffered(const uint8_t* data, size_t size, sockaddr addr) - { - LOG_TRACE_FMT("QUIC Session recv_buffered with {} bytes", size); - pending_reads.emplace_back(const_cast(data), size, addr); - do_handshake(); - } - - struct SessionDataTask : public ccf::tasks::ITaskAction - { - std::shared_ptr self; - std::vector data; - sockaddr addr{}; - - SessionDataTask( - std::shared_ptr s, - std::span d, - sockaddr sa) : - self(std::move(s)), - addr(sa) - { - data.assign(d.begin(), d.end()); - } - }; - - struct SendDataTask : public SessionDataTask - { - using SessionDataTask::SessionDataTask; - - void do_action() override - { - self->send_raw_thread(data, addr); - } - - [[nodiscard]] const std::string& get_name() const override - { - static const std::string name = "quic::SendDataTask"; - return name; - } - }; - - struct RecvDataTask : public SessionDataTask - { - using SessionDataTask::SessionDataTask; - - void do_action() override - { - self->recv(data.data(), data.size(), addr); - } - - [[nodiscard]] const std::string& get_name() const override - { - static const std::string name = "quic::RecvDataTask"; - return name; - } - }; - - void send_raw(const uint8_t* data, size_t size, sockaddr addr) - { - task_scheduler->add_action(std::make_shared( - shared_from_this(), std::span{data, size}, addr)); - } - - void send_raw_thread(const std::vector& data, sockaddr addr) - { - // Writes as much of the data as possible. If the data cannot all - // be written now, we store the remainder. We - // will try to send pending writes again whenever write() is called. - do_handshake(); - - if (status == handshake) - { - pending_writes.emplace_back( - const_cast(data.data()), data.size(), addr); - return; - } - - if (status != ready) - { - return; - } - - pending_writes.emplace_back( - const_cast(data.data()), data.size(), addr); - - flush(); - } - - void send_buffered(const std::vector& data, sockaddr addr) - { - pending_writes.emplace_back( - const_cast(data.data()), data.size(), addr); - } - - void handle_incoming_data(std::span data) override - { - auto [_, addr_family, addr_data, body] = - ringbuffer::read_message(data); - - task_scheduler->add_action(std::make_shared( - shared_from_this(), - body, - udp::sockaddr_decode(addr_family, addr_data))); - } - - virtual void recv(const uint8_t* data_, size_t size_, sockaddr addr_) = 0; - - void flush() - { - do_handshake(); - - if (status != ready) - { - return; - } - - for (auto& write : pending_writes) - { - LOG_TRACE_FMT("QUIC write_some {} bytes", write.len); - - // This will need to be handled by the actual QUIC stack - int rc = handle_send(write.req, write.len, write.addr); - if (rc < 0) - { - LOG_TRACE_FMT("QUIC {} error on flush", session_id); - stop(error); - return; - } - - // Mark for deletion (avoiding invalidating iterator) - write.clear = true; - } - - // Clear all marked for deletion - PendingBuffer::clear_empty(pending_writes); - } - - void close_session() override - { - auto self = shared_from_this(); - task_scheduler->add_action( - ccf::tasks::make_basic_action([self]() { self->close_thread(); })); - } - - void close_thread() - { - switch (status) - { - case handshake: - { - LOG_TRACE_FMT("QUIC {} closed during handshake", session_id); - stop(closed); - break; - } - - case ready: - { - LOG_TRACE_FMT("QUIC {} closed", session_id); - stop(closed); - break; - } - - case closed: - case authfail: - case error: - default: - { - } - } - } - - private: - void do_handshake() - { - // This should be called when additional data is written to the - // input buffer, until the handshake is complete. - if (status != handshake) - { - return; - } - - // This will need to be handled by the actual QUIC stack - LOG_TRACE_FMT("QUIC do_handshake unimplemented"); - status = ready; - } - - void stop(Status status_) - { - switch (status) - { - case closed: - case authfail: - case error: - return; - - case handshake: - case ready: - default: - { - } - } - - status = status_; - } - - int handle_send(const uint8_t* buf, size_t len, sockaddr addr) - { - auto [addr_family, addr_data] = udp::sockaddr_encode(addr); - - // Either write all of the data or none of it. - auto wrote = RINGBUFFER_TRY_WRITE_MESSAGE( - udp::udp_outbound, - to_host, - session_id, - addr_family, - addr_data, - serializer::ByteRange{buf, len}); - - if (!wrote) - { - return -1; - } - - return (int)len; - } - - int handle_recv(uint8_t* buf, size_t len, sockaddr addr) - { - size_t len_read = 0; - for (auto& read : pending_reads) - { - // Only handle pending reads that belong to the same address - if (memcmp((void*)&addr, (void*)&read.addr, sizeof(addr)) != 0) - { - continue; - } - - size_t rd = std::min(len, read.len); - ::memcpy(buf, read.req, rd); - read.clear = true; - - // UDP packets are datagrams, so it's either whole or nothing - len_read += rd; - if (len_read >= len) - { - break; - } - } - - // Clear all marked for deletion - PendingBuffer::clear_empty(pending_reads); - - if (len_read > 0) - { - return len_read; - } - return -1; - } - }; - - // This is a wrapper for the QUICSession so we can use in rpc_sessions - // Ultimately, this needs to be an HTTP3ServerSession : HTTP3Session : - // QUICSession - class QUICEchoSession : public QUICSession - { - std::shared_ptr rpc_map; - std::shared_ptr handler; - std::shared_ptr session_ctx; - ccf::ListenInterfaceID interface_id; - sockaddr addr; - - /// Move all reads into the writes and push back to the client - void echo() - { - pending_reads.swap(pending_writes); - flush(); - } - - public: - QUICEchoSession( - std::shared_ptr rpc_map_, - int64_t session_id_, - ccf::ListenInterfaceID interface_id_, - ringbuffer::AbstractWriterFactory& writer_factory) : - QUICSession(session_id_, writer_factory), - rpc_map(std::move(rpc_map_)), - interface_id(std::move(interface_id_)), - addr{} - {} - - void send_data(std::vector&& data) override - { - send_raw(data.data(), data.size(), addr); - } - - void recv(const uint8_t* data_, size_t size_, sockaddr addr_) override - { - recv_buffered(data_, size_, addr_); - addr = addr_; - - LOG_TRACE_FMT("recv called with {} bytes", size_); - - // ECHO SERVER - echo(); - } - }; -} diff --git a/src/quic/test/main.cpp b/src/quic/test/main.cpp deleted file mode 100644 index 6d248d1ee62e..000000000000 --- a/src/quic/test/main.cpp +++ /dev/null @@ -1,19 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the Apache 2.0 License. - -#define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN -#include "tls/tls.h" - -#include -#include - -using namespace std; -using namespace ccf::crypto; - -TEST_CASE("check QUIC OpenSSL library call") -{ - OpenSSL::Unique_SSL_CTX cfg(TLS_client_method()); - OpenSSL::Unique_SSL ssl(cfg); - SSL_QUIC_METHOD* quic; - SSL_set_quic_method(ssl, quic); -} \ No newline at end of file diff --git a/src/tasks/job_board.cpp b/src/tasks/job_board.cpp index 3ffdcd2ee285..e51da58c20c4 100644 --- a/src/tasks/job_board.cpp +++ b/src/tasks/job_board.cpp @@ -70,33 +70,66 @@ namespace ccf::tasks std::shared_ptr> waiting_worker_threads = std::make_shared>(); + ccf::ds::WorkBeaconPtr work_beacon = nullptr; + // Collection of delayed tasks, that may be ready for execution on a future // tick Delayed delayed; - void add_task(Task&& task) + void set_work_beacon(ccf::ds::WorkBeaconPtr work_beacon_) { - // Under lock - std::unique_lock lock(mutex); + ccf::ds::WorkBeaconPtr beacon; + { + std::lock_guard lock(mutex); + work_beacon = std::move(work_beacon_); + if (work_beacon != nullptr && !pending_tasks.empty()) + { + beacon = work_beacon; + } + } - // First check if there is an idle worker waiting for a task - for (WorkerThreadPtr& worker : *waiting_worker_threads) + if (beacon != nullptr) { - // NB: Although waiting_worker_threads is modified under lock, it is - // possible that a second call to add_task arrives before the notified - // thread wakes up and removes itself from this collection. In this case - // we must avoid overwriting a previously-assigned task. - if (worker->assigned_task == nullptr) + beacon->notify_work_available_coalesced(); + } + } + + void add_task(Task&& task) + { + ccf::ds::WorkBeaconPtr beacon; + { + // Under lock + std::unique_lock lock(mutex); + + // First check if there is an idle worker waiting for a task + for (WorkerThreadPtr& worker : *waiting_worker_threads) { - worker->assigned_task = std::move(task); - worker->cv.notify_one(); - return; + // NB: Although waiting_worker_threads is modified under lock, it is + // possible that a second call to add_task arrives before the notified + // thread wakes up and removes itself from this collection. In this + // case we must avoid overwriting a previously-assigned task. + if (worker->assigned_task == nullptr) + { + worker->assigned_task = std::move(task); + worker->cv.notify_one(); + return; + } } + + // There are no waiting_worker_threads currently, or none waiting for a + // task, so enqueue this task for later execution. Wake the external + // consumer only when the pending queue becomes non-empty. + if (pending_tasks.empty()) + { + beacon = work_beacon; + } + pending_tasks.emplace(std::move(task)); } - // There are no waiting_worker_threads currently, or none waiting for a - // task, so enqueue this task for later execution - pending_tasks.emplace(std::move(task)); + if (beacon != nullptr) + { + beacon->notify_work_available_coalesced(); + } } Task get_task() @@ -226,6 +259,11 @@ namespace ccf::tasks JobBoard::~JobBoard() = default; + void JobBoard::set_work_beacon(ccf::ds::WorkBeaconPtr work_beacon) + { + pimpl->set_work_beacon(std::move(work_beacon)); + } + void JobBoard::add_task(Task task) { pimpl->add_task(std::move(task)); diff --git a/src/tasks/job_board.h b/src/tasks/job_board.h index c2a5cb349ba3..5cfb28b23c28 100644 --- a/src/tasks/job_board.h +++ b/src/tasks/job_board.h @@ -25,6 +25,8 @@ namespace ccf::tasks JobBoard(); ~JobBoard(); + void set_work_beacon(ccf::ds::WorkBeaconPtr work_beacon); + void add_task(Task t); Task get_task(); diff --git a/src/tasks/test/basic_tasks.cpp b/src/tasks/test/basic_tasks.cpp index eb79a19fd714..9bc990a2f014 100644 --- a/src/tasks/test/basic_tasks.cpp +++ b/src/tasks/test/basic_tasks.cpp @@ -97,6 +97,102 @@ TEST_CASE("JobBoard" * doctest::test_suite("basic_tasks")) REQUIRE(b.load()); } +TEST_CASE("JobBoard external work beacon" * doctest::test_suite("basic_tasks")) +{ + constexpr auto short_wait = std::chrono::milliseconds(10); + + auto work_beacon = std::make_shared(); + ccf::tasks::JobBoard job_board; + job_board.set_work_beacon(work_beacon); + + const auto first = ccf::tasks::make_basic_task([]() {}); + const auto second = ccf::tasks::make_basic_task([]() {}); + + SUBCASE("Pending work notifications are coalesced") + { + job_board.add_task(first); + REQUIRE(work_beacon->wait_for_work_with_timeout(short_wait)); + + job_board.add_task(second); + REQUIRE_FALSE(work_beacon->wait_for_work_with_timeout(short_wait)); + + REQUIRE(job_board.get_task() == first); + REQUIRE(job_board.get_task() == second); + + job_board.add_task(first); + REQUIRE(job_board.get_task() == first); + job_board.add_task(second); + + REQUIRE(work_beacon->wait_for_work_with_timeout(short_wait)); + REQUIRE_FALSE(work_beacon->wait_for_work_with_timeout(short_wait)); + } + + SUBCASE("Direct worker handoff does not notify the external consumer") + { + ccf::tasks::Task received = nullptr; + std::thread worker( + [&]() { received = job_board.wait_for_task(std::chrono::seconds(1)); }); + + while (job_board.get_summary().idle_workers == 0) + { + std::this_thread::yield(); + } + + job_board.add_task(first); + worker.join(); + + REQUIRE(received == first); + REQUIRE_FALSE(work_beacon->wait_for_work_with_timeout(short_wait)); + } + + SUBCASE("Existing pending work is notified when a beacon is registered") + { + job_board.set_work_beacon(nullptr); + job_board.add_task(first); + job_board.set_work_beacon(work_beacon); + + REQUIRE(work_beacon->wait_for_work_with_timeout(short_wait)); + REQUIRE(job_board.get_task() == first); + } + + SUBCASE("A bounded consumer continues without waiting through a backlog") + { + constexpr size_t max_tasks_per_iteration = 256; + constexpr size_t num_tasks = max_tasks_per_iteration + 100; + for (size_t i = 0; i < num_tasks; ++i) + { + job_board.add_task(ccf::tasks::make_basic_task([]() {})); + } + + size_t completed = 0; + bool should_wait_for_work = true; + while (completed < num_tasks) + { + if (should_wait_for_work) + { + REQUIRE(work_beacon->wait_for_work_with_timeout(short_wait)); + } + + size_t completed_this_iteration = 0; + while (completed_this_iteration < max_tasks_per_iteration) + { + auto task = job_board.get_task(); + if (task == nullptr) + { + break; + } + + ++completed; + ++completed_this_iteration; + } + + should_wait_for_work = completed_this_iteration < max_tasks_per_iteration; + } + + REQUIRE(job_board.get_summary().pending_tasks == 0); + } +} + TEST_CASE("Cancellation" * doctest::test_suite("basic_tasks")) { ccf::tasks::JobBoard job_board; diff --git a/src/tcp/msg_types.h b/src/tcp/msg_types.h index de46467e328c..5fca13c17b44 100644 --- a/src/tcp/msg_types.h +++ b/src/tcp/msg_types.h @@ -2,51 +2,10 @@ // Licensed under the Apache 2.0 License. #pragma once -#include "ds/ring_buffer_types.h" +#include namespace tcp { + // Connection identifier used throughout the RPC session/transport layer. using ConnID = int64_t; - - /// TCP-related ringbuffer messages - /// The body of each message will begin with a connection ID - enum : ringbuffer::Message - { - /// New connection has been opened. This will always be the first message - /// sent regarding a connection. Host -> Enclave - DEFINE_RINGBUFFER_MSG_TYPE(tcp_start), - - /// Request for a new connection to a remote peer. Enclave -> Host - DEFINE_RINGBUFFER_MSG_TYPE(tcp_connect), - - /// Data read from socket, to be read inside enclave. Host -> Enclave - DEFINE_RINGBUFFER_MSG_TYPE(tcp_inbound), - - /// Data sent from the enclave, to be written to socket. Enclave -> Host - DEFINE_RINGBUFFER_MSG_TYPE(tcp_outbound), - - /// While processing data, the enclave decided this connection is stopped. - /// Enclave -> Host - DEFINE_RINGBUFFER_MSG_TYPE(tcp_stop), - - /// Connection has been invalidated. No more messages will be sent regarding - /// this connection. Host -> Enclave - DEFINE_RINGBUFFER_MSG_TYPE(tcp_close), - - /// Enclave session has been deleted. Host can now safely remove the - /// corresponding connection. Enclave -> Host - DEFINE_RINGBUFFER_MSG_TYPE(tcp_closed), - }; } - -DECLARE_RINGBUFFER_MESSAGE_PAYLOAD( - ::tcp::tcp_start, ::tcp::ConnID, std::string); -DECLARE_RINGBUFFER_MESSAGE_PAYLOAD( - ::tcp::tcp_connect, ::tcp::ConnID, std::string, std::string); -DECLARE_RINGBUFFER_MESSAGE_PAYLOAD( - ::tcp::tcp_inbound, ::tcp::ConnID, serializer::ByteRange); -DECLARE_RINGBUFFER_MESSAGE_PAYLOAD( - ::tcp::tcp_outbound, ::tcp::ConnID, serializer::ByteRange); -DECLARE_RINGBUFFER_MESSAGE_PAYLOAD(::tcp::tcp_stop, ::tcp::ConnID, std::string); -DECLARE_RINGBUFFER_MESSAGE_PAYLOAD(::tcp::tcp_close, ::tcp::ConnID); -DECLARE_RINGBUFFER_MESSAGE_PAYLOAD(::tcp::tcp_closed, ::tcp::ConnID); diff --git a/src/tls/README.md b/src/tls/README.md deleted file mode 100644 index 0585b0535085..000000000000 --- a/src/tls/README.md +++ /dev/null @@ -1,67 +0,0 @@ -# OpenSSL TLS Implementation - -This is a TLS implementation using OpenSSL that mimics the existing MbedTLS -one in a similar fashion. Because of that, some structures and call backs -look odd and have some work-arounds to make it fit the current workflow. - -Once we completely deprecate the MbedTLS implementation from CCF, we should -re-write the TLS implementation to fit the OpenSSL coding flow, which would -make it much simpler and easier to use. - -## CAs and Certificates - -In the MbedTLS world, certificates can be null and have methods to change -some configurations in the TLS config/session objects. There isn't a lot of -cross-over, so updating the config does the trick. - -However, in OpenSSL, session objects (ssl) are created from config objects -(cfg) and inherit all its properties. Therefore, to emulate MbedTLS, we need -to do to the session object every action we do to the config object, which is -not only redundant, but could be unsafe, if the calls are slightly different. - -### Validation - -Certificate validation can be complex to handle if you can accept connections -with certificates or not, and if they come, when and how to validate. - -MbedTLS is a lot more lenient on checks. For example, CAs are not tested for -validity of actually signing other certificates, while OpenSSL has extensive -checks, which can fail functionality that was previously passing. - -For this reason, a number of extra checks in the OpenSSL side were disabled. -Once we get rid of MbedTLS we should revisit those checks again and improve -CCF's usage of TLS, and perhaps also creating weaker checks for non-CA -certificates, etc. - -## Context - -### BIOs - -MbedTLS operates reads and writes solely via callbacks, with a buffer in the -session object acting as async I/O. This is in stark contrast with OpenSSL -which uses BIO objects to pass information back and forth, and only have -callbacks for debug or very specialized cases. - -We had to implement callbacks and specialize our case, but it could really be -just done with BIOs between the ring buffer and the context, but we'd have to -change a lot of code outside of the TLS implementation to add that. - -### Reads and Writes - -Reading and writing in MbedTLS returns a positive value for success (number of -bytes written) or a negative value for error (pre-defined error codes) including -WANTS_READ and WANTS_WRITE. - -In OpenSSL, those methods return 1 for success and 0 or -1 for errors (depending -on the version), with all errors, including WANTS_READ and WANTS_WRITE -accessible through `SSL_get_error`. This imposes a number of hacks needed to -mimic the MbedTLS implementation, including: - -- Multiple `#define`s with common error messages in `tls.h` -- Having to negate the error code to match -- Multiple checks to `SSL_want_read` and `SSL_want_write` - -### Error Handling - -As discussed above, the error handling is slightly different and promotes -verbose code in OpenSSL's side. diff --git a/src/tls/client.h b/src/tls/client.h deleted file mode 100644 index c3e6a2ac718c..000000000000 --- a/src/tls/client.h +++ /dev/null @@ -1,22 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the Apache 2.0 License. -#pragma once - -#include "context.h" - -namespace tls -{ - class Client : public ccf::tls::Context - { - private: - std::shared_ptr cert; - - public: - Client(std::shared_ptr cert_) : Context(true), cert(std::move(cert_)) - { - cert->configure_context(cfg); - create_ssl(); - cert->configure_connection(get_ssl()); - } - }; -} diff --git a/src/tls/context.h b/src/tls/context.h index 69be4fe3ff31..6b6905be67cd 100644 --- a/src/tls/context.h +++ b/src/tls/context.h @@ -77,7 +77,7 @@ namespace ccf::tls // approved classical groups as fallbacks CHECK1(SSL_CTX_set1_groups_list( cfg, - "?SecP384r1MLKEM1024:?SecP256r1MLKEM768:?X25519MLKEM768:" + "?X25519MLKEM768:?SecP256r1MLKEM768:?SecP384r1MLKEM1024:" "P-521:P-384:P-256")); // Allow buffer to be relocated between WANT_WRITE retries, and do partial diff --git a/src/tls/plaintext_server.h b/src/tls/plaintext_server.h deleted file mode 100644 index 60648e091984..000000000000 --- a/src/tls/plaintext_server.h +++ /dev/null @@ -1,87 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the Apache 2.0 License. -#pragma once - -#include "context.h" - -namespace nontls -{ - class PlaintextServer : public ccf::tls::Context - { - public: - PlaintextServer() : Context(false) {} - - protected: - Unique_BIO read_bio; - Unique_BIO write_bio; - - public: - void set_bio( - void* cb_obj, BIO_callback_fn_ex send, BIO_callback_fn_ex recv) override - { - // Read/Write BIOs will be used by TLS - BIO_set_mem_eof_return(read_bio, -1); - BIO_set_callback_arg(read_bio, static_cast(cb_obj)); - BIO_set_callback_ex(read_bio, recv); - - BIO_set_mem_eof_return(write_bio, -1); - BIO_set_callback_arg(write_bio, static_cast(cb_obj)); - BIO_set_callback_ex(write_bio, send); - } - - int handshake() override - { - return 0; - } - - int read(uint8_t* buf, size_t len) override - { - if (len == 0) - { - return 0; - } - size_t readbytes = 0; - int rc = BIO_read_ex(read_bio, buf, len, &readbytes); - if (rc > 0) - { - return readbytes; - } - return -rc; - } - - int write(const uint8_t* buf, size_t len) override - { - if (len == 0) - { - return 0; - } - size_t written = 0; - int rc = BIO_write_ex(write_bio, buf, len, &written); - if (rc > 0) - { - return written; - } - return -rc; - } - - int close() override - { - return 0; - } - - bool peer_cert_ok() override - { - return true; - } - - std::string get_verify_error() override - { - return "no error"; - } - - std::vector peer_cert() override - { - return {}; - } - }; -} \ No newline at end of file diff --git a/src/tls/server.h b/src/tls/server.h deleted file mode 100644 index eb777603538e..000000000000 --- a/src/tls/server.h +++ /dev/null @@ -1,74 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the Apache 2.0 License. -#pragma once - -#include "context.h" - -namespace tls -{ - struct AlpnProtocols - { - const unsigned char* data; - unsigned int size; - }; - - static int alpn_select_cb( - SSL* /*ssl*/, - const unsigned char** out, - unsigned char* outlen, - const unsigned char* in, - unsigned int inlen, - void* arg) - { - auto* protos = static_cast(arg); - - if ( - SSL_select_next_proto( - const_cast(out), - outlen, - protos->data, - protos->size, - in, - inlen) != OPENSSL_NPN_NEGOTIATED) - { - return SSL_TLSEXT_ERR_NOACK; - } - - return SSL_TLSEXT_ERR_OK; - } - - class Server : public ccf::tls::Context - { - private: - std::shared_ptr cert; - - public: - Server(const std::shared_ptr& cert_, bool http2 = false) : - Context(false), - cert(cert_) - { - cert->configure_context(cfg); - - // Configure protocols negotiated by ALPN - // See https://nghttp2.org/documentation/tutorial-server.html and use of - // nghttp2_select_next_protocol for better example - if (http2) - { - static unsigned char alpn_protos_data[] = {2, 'h', '2'}; - static AlpnProtocols alpn_protos{ - alpn_protos_data, sizeof(alpn_protos_data)}; - SSL_CTX_set_alpn_select_cb(cfg, alpn_select_cb, &alpn_protos); - } - else - { - static unsigned char alpn_protos_data[] = { - 8, 'h', 't', 't', 'p', '/', '1', '.', '1'}; - static AlpnProtocols alpn_protos{ - alpn_protos_data, sizeof(alpn_protos_data)}; - SSL_CTX_set_alpn_select_cb(cfg, alpn_select_cb, &alpn_protos); - } - - create_ssl(); - } - }; -} diff --git a/src/tls/test/main.cpp b/src/tls/test/main.cpp deleted file mode 100644 index 3436ca481890..000000000000 --- a/src/tls/test/main.cpp +++ /dev/null @@ -1,925 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the Apache 2.0 License. -#include "ccf/crypto/ec_key_pair.h" -#include "ccf/crypto/verifier.h" -#include "ccf/ds/nonstd.h" -#include "crypto/certs.h" -#include "ds/internal_logger.h" -#include "tcp/msg_types.h" -#include "tls/client.h" -#include "tls/server.h" -#include "tls/tls.h" - -#include -#include -#include -#include -#define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN -#include -#include -#include -#include -#include -#include - -using namespace std; -using namespace ccf::crypto; -using namespace tls; - -/// Server uses one pipe while client uses the other. -/// Writes always to one side, reads always from the other. -/// Use the send/recv template wrappers below as callbacks. -class TestPipe -{ - int pfd[2]; - -public: - static const int SERVER = 0; - static const int CLIENT = 1; - - TestPipe() - { - if (socketpair(PF_LOCAL, SOCK_STREAM, 0, pfd) == -1) - { - throw runtime_error( - "Failed to create socketpair: " + string(ccf::nonstd::strerror(errno))); - } - } - ~TestPipe() - { - close(pfd[0]); - close(pfd[1]); - } - - size_t send(int id, const uint8_t* buf, size_t len) - { - int rc = write(pfd[id], buf, len); - if (rc == -1) - LOG_FAIL_FMT("Error while reading: {}", ccf::nonstd::strerror(errno)); - return rc; - } - - size_t recv(int id, uint8_t* buf, size_t len) - { - int rc = read(pfd[id], buf, len); - if (rc == -1) - LOG_FAIL_FMT("Error while reading: {}", ccf::nonstd::strerror(errno)); - return rc; - } -}; - -/// Callback wrapper around TestPipe->send(). -template -int send(void* ctx, const uint8_t* buf, size_t len) -{ - auto pipe = reinterpret_cast(ctx); - int rc = pipe->send(end, buf, len); - REQUIRE(rc == len); - return rc; -} - -/// Callback wrapper around TestPipe->recv(). -template -int recv(void* ctx, uint8_t* buf, size_t len) -{ - auto pipe = reinterpret_cast(ctx); - int rc = pipe->recv(end, buf, len); - REQUIRE(rc == len); - return rc; -} - -// OpenSSL callbacks that call onto the pipe's ones -template -long send( - BIO* b, - int oper, - const char* argp, - size_t len, - int argi, - long argl, - int ret, - size_t* processed) -{ - // Unused arguments - (void)argi; - (void)argl; - (void)processed; - - if (ret && oper == (BIO_CB_WRITE | BIO_CB_RETURN)) - { - // Flush the BIO so the "pipe doesn't clog", but we don't use the - // data here, because 'argp' already has it. - BIO_flush(b); - size_t pending = BIO_pending(b); - if (pending) - BIO_reset(b); - - // Pipe object - auto pipe = reinterpret_cast(BIO_get_callback_arg(b)); - size_t put = send(pipe, (const uint8_t*)argp, len); - REQUIRE(put == len); - } - - // Unless we detected an error, the return value is always the same as the - // original operation. - return ret; -} - -template -long recv( - BIO* b, - int oper, - const char* argp, - size_t len, - int argi, - long argl, - int ret, - size_t* processed) -{ - // Unused arguments - (void)argi; - (void)argl; - - if (ret && oper == (BIO_CB_READ | BIO_CB_RETURN)) - { - // Pipe object - auto pipe = reinterpret_cast(BIO_get_callback_arg(b)); - size_t got = recv(pipe, (uint8_t*)argp, len); - - // Got nothing, return "WANTS READ" - if (got <= 0) - return ret; - - // Write to the actual BIO so SSL can use it - BIO_write_ex(b, argp, got, processed); - - // If original return was -1 because it didn't find anything to read, return - // 1 to say we actually read something - if (got > 0 && ret < 0) - return 1; - } - - // Unless we detected an error, the return value is always the same as the - // original operation. - return ret; -} - -/// Performs a TLS handshake, looping until there's nothing more to read/write. -/// Returns 0 on success, throws a runtime error with SSL error str on failure. -int handshake(ccf::tls::Context* ctx, std::atomic& keep_going) -{ - while (keep_going) - { - int rc = ctx->handshake(); - - switch (rc) - { - case 0: - return 0; - - case TLS_ERR_WANT_READ: - case TLS_ERR_WANT_WRITE: - // Continue calling handshake until finished - LOG_DEBUG_FMT("Handshake wants data"); - break; - - case TLS_ERR_NEED_CERT: - { - LOG_FAIL_FMT("Handshake error: {}", ::tls::error_string(rc)); - return 1; - } - - case TLS_ERR_CONN_CLOSE_NOTIFY: - { - LOG_FAIL_FMT("Handshake error: {}", ::tls::error_string(rc)); - return 1; - } - - case TLS_ERR_X509_VERIFY: - { - auto err = ctx->get_verify_error(); - LOG_FAIL_FMT("Handshake error: {} [{}]", err, ::tls::error_string(rc)); - return 1; - } - - default: - { - LOG_FAIL_FMT("Handshake error: {}", ::tls::error_string(rc)); - return 1; - } - } - } - - return 0; -} - -struct NetworkCA -{ - shared_ptr kp; - ccf::crypto::Pem cert; -}; - -static ccf::crypto::Pem generate_self_signed_cert( - const ccf::crypto::ECKeyPairPtr& kp, const std::string& name) -{ - using namespace std::literals; - constexpr size_t certificate_validity_period_days = 365; - auto valid_from = - ccf::ds::to_x509_time_string(std::chrono::system_clock::now() - 24h); - - return ccf::crypto::create_self_signed_cert( - kp, name, {}, valid_from, certificate_validity_period_days); -} - -static ccf::crypto::Pem generate_endorsed_cert( - const ccf::crypto::ECKeyPairPtr& kp, - const std::string& name, - const ccf::crypto::ECKeyPairPtr& issuer_kp, - const ccf::crypto::Pem& issuer_cert) -{ - constexpr size_t certificate_validity_period_days = 365; - - using namespace std::literals; - auto valid_from = - ccf::ds::to_x509_time_string(std::chrono::system_clock::now() - 24h); - - return ccf::crypto::create_endorsed_cert( - kp, - name, - {}, - valid_from, - certificate_validity_period_days, - issuer_kp->private_key_pem(), - issuer_cert); -} - -/// Get self-signed CA certificate. -NetworkCA get_ca() -{ - // Create a CA with a self-signed certificate - auto kp = ccf::crypto::make_ec_key_pair(); - auto crt = generate_self_signed_cert(kp, "CN=issuer"); - LOG_DEBUG_FMT("New self-signed CA certificate:\n{}", crt.str()); - return {kp, crt}; -} - -/// Creates a ::tls::Cert with a new CA using a new self-signed Pem certificate. -unique_ptr<::tls::Cert> get_dummy_cert( - NetworkCA& net_ca, string name, bool auth_required = true) -{ - // Create a CA with a self-signed certificate - auto ca = make_unique<::tls::CA>(net_ca.cert.str()); - - // Create a signing request and sign with the CA - auto kp = ccf::crypto::make_ec_key_pair(); - auto crt = generate_endorsed_cert(kp, "CN=" + name, net_ca.kp, net_ca.cert); - LOG_DEBUG_FMT("New CA-signed certificate:\n{}", crt.str()); - - // Verify node certificate with the CA's certificate - auto v = ccf::crypto::make_verifier(crt); - REQUIRE(v->verify_certificate({&net_ca.cert})); - - // Create a ::tls::Cert with the CA, the signed certificate and the private - // key - auto pk = kp->private_key_pem(); - return make_unique(std::move(ca), crt, pk, std::nullopt, auth_required); -} - -TEST_CASE("CA configures trusted certificate store") -{ - auto ca = get_ca(); - ::tls::CA trusted_ca(ca.cert.str(), true); - ccf::crypto::OpenSSL::Unique_SSL_CTX ctx(TLS_method()); - - trusted_ca.configure_trusted_cert_store(ctx); - - auto* store = SSL_CTX_get_cert_store(ctx); - REQUIRE(store != nullptr); - auto* params = X509_STORE_get0_param(store); - REQUIRE(params != nullptr); - REQUIRE( - (X509_VERIFY_PARAM_get_flags(params) & X509_V_FLAG_PARTIAL_CHAIN) != 0); -} - -TEST_CASE("Cert configures TLS verification and own certificate") -{ - auto ca = get_ca(); - auto cert = get_dummy_cert(ca, "server"); - ccf::crypto::OpenSSL::Unique_SSL_CTX ctx(TLS_method()); - - cert->configure_context(ctx); - ccf::crypto::OpenSSL::Unique_SSL ssl(ctx); - cert->configure_connection(ssl); - - constexpr auto expected_verify_mode = - SSL_VERIFY_PEER | SSL_VERIFY_FAIL_IF_NO_PEER_CERT; - REQUIRE(SSL_CTX_get_verify_mode(ctx) == expected_verify_mode); - REQUIRE(SSL_get_verify_mode(ssl) == expected_verify_mode); - REQUIRE(SSL_CTX_get0_certificate(ctx) != nullptr); - REQUIRE(SSL_get_certificate(ssl) != nullptr); -} - -/// Helper to write past the maximum buffer (16k) -int write_helper(ccf::tls::Context& handler, const uint8_t* buf, size_t len) -{ - LOG_DEBUG_FMT("WRITE {} bytes", len); - int rc = handler.write(buf, len); - if (rc <= 0 || (size_t)rc == len) - return rc; - return rc + write_helper(handler, buf + rc, len - rc); -} - -/// Helper to read past the maximum buffer (16k) -int read_helper(ccf::tls::Context& handler, uint8_t* buf, size_t len) -{ - LOG_DEBUG_FMT("READ {} bytes", len); - int rc = handler.read(buf, len); - if (rc <= 0 || (size_t)rc == len) - return rc; - return rc + read_helper(handler, buf + rc, len - rc); -} - -/// Helper to truncate long messages to make logs more readable -std::string truncate_message(const uint8_t* msg, size_t len) -{ - const size_t MAX_LEN = 32; - if (len < MAX_LEN) - return std::string((const char*)msg); - std::string str((const char*)msg, MAX_LEN); - str += "... + " + std::to_string(len - MAX_LEN); - return str; -} - -void run_handshake(tls::Server& server, tls::Client& client) -{ - std::atomic keep_going = true; - std::optional client_exception, server_exception; - - thread client_thread([&client, &keep_going, &client_exception]() { - LOG_INFO_FMT("Client handshake"); - try - { - if (handshake(&client, keep_going)) - throw runtime_error("Client handshake error"); - } - catch (std::runtime_error& ex) - { - keep_going = false; - client_exception = ex; - } - }); - - thread server_thread([&server, &keep_going, &server_exception]() { - LOG_INFO_FMT("Server handshake"); - try - { - if (handshake(&server, keep_going)) - throw runtime_error("Server handshake error"); - } - catch (std::runtime_error& ex) - { - keep_going = false; - server_exception = ex; - } - }); - - client_thread.join(); - server_thread.join(); - LOG_INFO_FMT("Handshake completed"); - - if (client_exception) - { - throw *client_exception; - } - if (server_exception) - { - throw *server_exception; - } -} - -/// Test runner, with various options for different kinds of tests. -void run_test_case( - const uint8_t* message, - size_t message_length, - const uint8_t* response, - size_t response_length, - unique_ptr<::tls::Cert> server_cert, - unique_ptr<::tls::Cert> client_cert) -{ - std::vector buf(max(message_length, response_length) + 1); - - // Create a pair of client/server - tls::Server server(std::move(server_cert)); - tls::Client client(std::move(client_cert)); - - // Connect BIOs together - TestPipe pipe; - server.set_bio(&pipe, send, recv); - client.set_bio(&pipe, send, recv); - - run_handshake(server, client); - - // The rest of the communication is deterministic and easy to simulate - // so we take them out of the thread, to guarantee there will be bytes - // to read at the right time. - if (message_length == 0) - { - LOG_INFO_FMT("Empty message. Ignoring communication test"); - LOG_INFO_FMT("Closing connection"); - client.close(); - server.close(); - return; - } - - // Send the first message - LOG_INFO_FMT( - "Client sending message [{}]", truncate_message(message, message_length)); - int written = write_helper(client, message, message_length); - REQUIRE(written == message_length); - - // Receive the first message - int read = read_helper(server, buf.data(), message_length); - REQUIRE(read == message_length); - buf[message_length] = '\0'; - LOG_INFO_FMT( - "Server message received [{}]", - truncate_message(buf.data(), message_length)); - REQUIRE( - strncmp((const char*)buf.data(), (const char*)message, message_length) == - 0); - - // Send the response - LOG_INFO_FMT( - "Server sending message [{}]", truncate_message(response, message_length)); - written = write_helper(server, response, response_length); - REQUIRE(written == response_length); - - // Receive the response - read = read_helper(client, buf.data(), response_length); - REQUIRE(read == response_length); - buf[response_length] = '\0'; - LOG_INFO_FMT( - "Client message received [{}]", - truncate_message(buf.data(), message_length)); - REQUIRE( - strncmp((const char*)buf.data(), (const char*)response, response_length) == - 0); - - LOG_INFO_FMT("Closing connection"); - client.close(); - server.close(); -} - -std::string negotiated_group_name(SSL* ssl) -{ - const auto group_id = SSL_get_negotiated_group(ssl); - if (group_id == NID_undef) - { - return {}; - } - - const auto* group_name = SSL_group_to_name(ssl, group_id); - if (group_name != nullptr) - { - return group_name; - } - - return std::to_string(group_id); -} - -class InspectableClient : public tls::Client -{ -public: - using tls::Client::Client; - - InspectableClient( - std::shared_ptr<::tls::Cert> cert, const std::string& groups) : - tls::Client(std::move(cert)) - { - REQUIRE(SSL_set1_groups_list(get_ssl(), groups.c_str()) == 1); - } - - int verify_mode() - { - return SSL_get_verify_mode(get_ssl()); - } - - std::string negotiated_group() - { - return negotiated_group_name(get_ssl()); - } -}; - -class InspectableServer : public tls::Server -{ -public: - InspectableServer( - const std::shared_ptr<::tls::Cert>& cert, const std::string& groups) : - tls::Server(cert) - { - REQUIRE(SSL_set1_groups_list(get_ssl(), groups.c_str()) == 1); - } - - std::string negotiated_group() - { - return negotiated_group_name(get_ssl()); - } -}; - -// Hybrid groups offered by src/tls/context.h, in the order they are offered -constexpr auto secp384r1_mlkem1024 = "SecP384r1MLKEM1024"; -constexpr auto secp256r1_mlkem768 = "SecP256r1MLKEM768"; -constexpr auto x25519_mlkem768 = "X25519MLKEM768"; - -// Classical groups offered by src/tls/context.h, in the same order -constexpr auto classical_groups = "P-521:P-384:P-256"; - -/// Handshakes a client offering client_groups against a server offering -/// server_groups, and returns the group they agreed on. Throws if they cannot -/// agree. -std::string negotiate_group( - const std::string& client_groups, const std::string& server_groups) -{ - INFO("client groups: ", client_groups); - INFO("server groups: ", server_groups); - - auto ca = get_ca(); - InspectableServer server(get_dummy_cert(ca, "server", false), server_groups); - InspectableClient client(get_dummy_cert(ca, "client", false), client_groups); - - TestPipe pipe; - server.set_bio(&pipe, send, recv); - client.set_bio(&pipe, send, recv); - - run_handshake(server, client); - - const auto group = client.negotiated_group(); - REQUIRE(group == server.negotiated_group()); - return group; -} - -TEST_CASE("connection inherits verification mode from context") -{ - auto ca = get_ca(); - - InspectableClient verified_client(get_dummy_cert(ca, "verified")); - REQUIRE( - (verified_client.verify_mode() & - (SSL_VERIFY_PEER | SSL_VERIFY_FAIL_IF_NO_PEER_CERT)) == - (SSL_VERIFY_PEER | SSL_VERIFY_FAIL_IF_NO_PEER_CERT)); - - InspectableClient request_only_client( - get_dummy_cert(ca, "request_only", false)); - // auth_required=false still requests a peer certificate, but does not fail - // the handshake if the peer certificate is missing. - REQUIRE((request_only_client.verify_mode() & SSL_VERIFY_PEER) != 0); - REQUIRE( - (request_only_client.verify_mode() & SSL_VERIFY_FAIL_IF_NO_PEER_CERT) == 0); -} - -TEST_CASE("group negotiation") -{ - SUBCASE("the first group offered by the client wins") - { - REQUIRE( - negotiate_group("P-384:P-256:P-521", "P-384:P-256:P-521") == "secp384r1"); - REQUIRE( - negotiate_group("P-256:P-384:P-521", "P-256:P-384:P-521") == "secp256r1"); - } - - SUBCASE("the client order wins over the server order") - { - // In TLS 1.3 the server selects the group, and OpenSSL selects the first - // group the client offered that the server also supports. The server order - // is only a filter, so the client dictates the outcome. - REQUIRE(negotiate_group("P-256:P-521", "P-521:P-256") == "secp256r1"); - REQUIRE(negotiate_group("P-521:P-256", "P-256:P-521") == "secp521r1"); - } - - SUBCASE("disjoint groups fail the handshake") - { - REQUIRE_THROWS_AS( - negotiate_group("P-521", "P-256"), const std::runtime_error&); - } -} - -TEST_CASE("hybrid group negotiation") -{ - const std::vector hybrid_groups = { - secp384r1_mlkem1024, secp256r1_mlkem768, x25519_mlkem768}; - - SUBCASE("the configured groups negotiate the strongest hybrid group") - { - auto ca = get_ca(); - tls::Server server(get_dummy_cert(ca, "server", false)); - InspectableClient client(get_dummy_cert(ca, "client", false)); - - TestPipe pipe; - server.set_bio(&pipe, send, recv); - client.set_bio(&pipe, send, recv); - - run_handshake(server, client); - - REQUIRE(client.negotiated_group() == secp384r1_mlkem1024); - } - - SUBCASE("disjoint hybrid groups with no classical fallback fail") - { - REQUIRE_THROWS_AS( - negotiate_group(secp384r1_mlkem1024, secp256r1_mlkem768), - const std::runtime_error&); - } - - SUBCASE("disjoint hybrid groups fall back to the first shared classical one") - { - REQUIRE( - negotiate_group( - fmt::format("{}:{}", secp384r1_mlkem1024, classical_groups), - fmt::format("{}:{}", secp256r1_mlkem768, classical_groups)) == - "secp521r1"); - } - - SUBCASE("disjoint hybrid and disjoint classical groups fail") - { - REQUIRE_THROWS_AS( - negotiate_group( - fmt::format("{}:P-521", secp384r1_mlkem1024), - fmt::format("{}:P-256", secp256r1_mlkem768)), - const std::runtime_error&); - } - - SUBCASE("the client order decides which shared hybrid group is used") - { - // As for classical groups, the client's order is what matters - REQUIRE( - negotiate_group( - fmt::format("{}:{}", secp384r1_mlkem1024, secp256r1_mlkem768), - fmt::format("{}:{}", secp256r1_mlkem768, secp384r1_mlkem1024)) == - secp384r1_mlkem1024); - REQUIRE( - negotiate_group( - fmt::format("{}:{}", secp256r1_mlkem768, secp384r1_mlkem1024), - fmt::format("{}:{}", secp384r1_mlkem1024, secp256r1_mlkem768)) == - secp256r1_mlkem768); - } - - SUBCASE("each hybrid group can be negotiated on its own") - { - for (const auto& group : hybrid_groups) - { - INFO("group: ", group); - REQUIRE(negotiate_group(group, group) == group); - } - } - - SUBCASE("a shared hybrid group is preferred over classical fallbacks") - { - for (const auto& group : hybrid_groups) - { - INFO("group: ", group); - const auto groups = fmt::format("{}:{}", group, classical_groups); - REQUIRE(negotiate_group(groups, groups) == group); - } - } -} - -TEST_CASE("unverified handshake") -{ - // Create a CA - auto ca = get_ca(); - - // Create bogus certificate - auto server_cert = get_dummy_cert(ca, "server", false); - auto client_cert = get_dummy_cert(ca, "client", false); - - LOG_INFO_FMT("TEST: unverified handshake"); - - // Just testing handshake, does not verify certificates, no communication. - run_test_case( - (const uint8_t*)"", - 0, - (const uint8_t*)"", - 0, - std::move(server_cert), - std::move(client_cert)); -} - -TEST_CASE("unverified communication") -{ - const uint8_t message[] = "Hello World!"; - size_t message_length = strlen((const char*)message); - const uint8_t response[] = "Hi back!"; - size_t response_length = strlen((const char*)response); - - // Create a CA - auto ca = get_ca(); - - // Create bogus certificate - auto server_cert = get_dummy_cert(ca, "server", false); - auto client_cert = get_dummy_cert(ca, "client", false); - - LOG_INFO_FMT("TEST: unverified communication"); - - // Just testing communication channel, does not verify certificates. - run_test_case( - message, - message_length, - response, - response_length, - std::move(server_cert), - std::move(client_cert)); -} - -TEST_CASE("verified handshake") -{ - // Create a CA - auto ca = get_ca(); - - // Create bogus certificate - auto server_cert = get_dummy_cert(ca, "server"); - auto client_cert = get_dummy_cert(ca, "client"); - - LOG_INFO_FMT("TEST: verified handshake"); - - // Just testing handshake, no communication, but verifies certificates. - run_test_case( - (const uint8_t*)"", - 0, - (const uint8_t*)"", - 0, - std::move(server_cert), - std::move(client_cert)); -} - -TEST_CASE("self-signed server certificate") -{ - auto kp = ccf::crypto::make_ec_key_pair(); - auto pk = kp->private_key_pem(); - auto crt = generate_self_signed_cert(kp, "CN=server"); - auto server_cert = make_unique(nullptr, crt, pk); - - // Create a CA - auto ca = get_ca(); - auto client_cert = get_dummy_cert(ca, "client"); - - // Client expected to complain about self-signedness. - REQUIRE_THROWS_WITH_AS( - run_test_case( - (const uint8_t*)"", - 0, - (const uint8_t*)"", - 0, - std::move(server_cert), - std::move(client_cert)), - "Client handshake error", - std::runtime_error); -} - -TEST_CASE("server certificate from different CA") -{ - auto server_ca = get_ca(); - auto server_cert = get_dummy_cert(server_ca, "server"); - - auto client_ca = get_ca(); - auto client_cert = get_dummy_cert(client_ca, "client"); - - // Client expected to complain - REQUIRE_THROWS_WITH_AS( - run_test_case( - (const uint8_t*)"", - 0, - (const uint8_t*)"", - 0, - std::move(server_cert), - std::move(client_cert)), - "Client handshake error", - std::runtime_error); -} - -TEST_CASE("self-signed client certificate") -{ - auto server_ca = get_ca(); - auto server_cert = get_dummy_cert(server_ca, "server", false); - - auto kp = ccf::crypto::make_ec_key_pair(); - auto pk = kp->private_key_pem(); - auto crt = generate_self_signed_cert(kp, "CN=server"); - - // With verification enabled, the client is expected to complain. - auto client_cert = make_unique(nullptr, crt, pk); - - REQUIRE_THROWS_WITH_AS( - run_test_case( - (const uint8_t*)"", - 0, - (const uint8_t*)"", - 0, - std::move(server_cert), - std::move(client_cert)), - "Client handshake error", - std::runtime_error); - - // Without verification enabled on the client, the server should complain. - server_cert = get_dummy_cert(server_ca, "server"); - client_cert = make_unique(nullptr, crt, pk, std::nullopt, false); - - REQUIRE_THROWS_WITH_AS( - run_test_case( - (const uint8_t*)"", - 0, - (const uint8_t*)"", - 0, - std::move(server_cert), - std::move(client_cert)), - "Server handshake error", - std::runtime_error); - - // Neither, neither. - server_cert = get_dummy_cert(server_ca, "server", false); - client_cert = make_unique(nullptr, crt, pk, std::nullopt, false); - REQUIRE_NOTHROW(run_test_case( - (const uint8_t*)"", - 0, - (const uint8_t*)"", - 0, - std::move(server_cert), - std::move(client_cert))); -} - -TEST_CASE("verified communication") -{ - const uint8_t message[] = "Hello World!"; - size_t message_length = strlen((const char*)message); - const uint8_t response[] = "Hi back!"; - size_t response_length = strlen((const char*)response); - - // Create a CA - auto ca = get_ca(); - - // Create bogus certificate - auto server_cert = get_dummy_cert(ca, "server"); - auto client_cert = get_dummy_cert(ca, "client"); - - LOG_INFO_FMT("TEST: verified communication"); - - // Testing communication channel, verifying certificates. - run_test_case( - message, - message_length, - response, - response_length, - std::move(server_cert), - std::move(client_cert)); -} - -TEST_CASE("large message") -{ - // Uninitialised on purpose, we don't care what's in here - size_t len = 8192; - std::vector buf(len); - auto message = ccf::crypto::b64_from_raw(buf.data(), len); - - // Create a CA - auto ca = get_ca(); - - // Create bogus certificate - auto server_cert = get_dummy_cert(ca, "server"); - auto client_cert = get_dummy_cert(ca, "client"); - - LOG_INFO_FMT("TEST: large message"); - - // Testing communication channel, verifying certificates. - run_test_case( - (const uint8_t*)message.data(), - message.size(), - (const uint8_t*)message.data(), - message.size(), - std::move(server_cert), - std::move(client_cert)); -} - -TEST_CASE("very large message") -{ - // Uninitialised on purpose, we don't care what's in here - size_t len = 16 * 1024; // 16k, base64 will be more - std::vector buf(len); - auto message = ccf::crypto::b64_from_raw(buf.data(), len); - - // Create a CA - auto ca = get_ca(); - - // Create bogus certificate - auto server_cert = get_dummy_cert(ca, "server"); - auto client_cert = get_dummy_cert(ca, "client"); - - LOG_INFO_FMT("TEST: very large message"); - - // Testing communication channel, verifying certificates. - run_test_case( - (const uint8_t*)message.data(), - message.size(), - (const uint8_t*)message.data(), - message.size(), - std::move(server_cert), - std::move(client_cert)); -} diff --git a/src/tls/tls.h b/src/tls/tls.h deleted file mode 100644 index e293f793c848..000000000000 --- a/src/tls/tls.h +++ /dev/null @@ -1,36 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the Apache 2.0 License. -#pragma once - -// These macros setup return values for when the connection is reading/writing -// and needs more data or has an error. -// -// In OpenSSL, the return is -1/0/1 and the error code depends on what -// SSL_want() returns. So we need to return some distinct negative number -// and then handle WANT_READ/WANT_WRITE and errors. -// -// Depending on the error, the connection needs to close with success, failure -// or auth-failure. -#define TLS_READING -SSL_READING -#define TLS_WRITING -SSL_WRITING -#define TLS_ERR_WANT_READ -SSL_ERROR_WANT_READ -#define TLS_ERR_WANT_WRITE -SSL_ERROR_WANT_WRITE -#define TLS_ERR_CONN_CLOSE_NOTIFY -SSL_ERROR_ZERO_RETURN -#define TLS_ERR_NEED_CERT -SSL_ERROR_WANT_X509_LOOKUP -// Specific error to check validity of certificate, not emitted by OpenSSL, but -// by Context. We set to a bogus negative value that won't match any OpenSSL -// error code. -// Once we refactor the code to match the OpenSSL style we may not need this. -#define TLS_ERR_X509_VERIFY INT_MIN - -#include "ccf/crypto/openssl/openssl_wrappers.h" - -#include - -namespace tls -{ - inline std::string error_string(int ec) - { - return ccf::crypto::OpenSSL::error_string(ec); - } -} diff --git a/src/udp/msg_types.h b/src/udp/msg_types.h deleted file mode 100644 index b27fdf25fe27..000000000000 --- a/src/udp/msg_types.h +++ /dev/null @@ -1,50 +0,0 @@ -// Copyright (c) Microsoft Corporation. All rights reserved. -// Licensed under the Apache 2.0 License. -#pragma once - -#include "ds/ring_buffer_types.h" - -#include - -namespace udp -{ - using ConnID = int64_t; - - enum : ringbuffer::Message - { - DEFINE_RINGBUFFER_MSG_TYPE(udp_start), - DEFINE_RINGBUFFER_MSG_TYPE(udp_inbound), - DEFINE_RINGBUFFER_MSG_TYPE(udp_outbound), - }; - - static std::tuple> sockaddr_encode(sockaddr& addr) - { - short family = addr.sa_family; - std::vector data(14, '\0'); - memcpy(data.data(), &addr.sa_data, 14); - return std::make_pair(family, data); - } - - static sockaddr sockaddr_decode( - short family, const std::vector& data) - { - sockaddr addr{}; - addr.sa_family = family; - memcpy(&addr.sa_data, data.data(), 14); - return addr; - } -} - -DECLARE_RINGBUFFER_MESSAGE_PAYLOAD(udp::udp_start, udp::ConnID, std::string); -DECLARE_RINGBUFFER_MESSAGE_PAYLOAD( - udp::udp_inbound, - int64_t, - short, - std::vector, - serializer::ByteRange); -DECLARE_RINGBUFFER_MESSAGE_PAYLOAD( - udp::udp_outbound, - int64_t, - short, - std::vector, - serializer::ByteRange); diff --git a/tests/connections.py b/tests/connections.py index 4e8c8529c1ac..16bee0ec7e47 100644 --- a/tests/connections.py +++ b/tests/connections.py @@ -340,8 +340,10 @@ def node_tcp_socket(node): interface = node.n2n_interface s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect((interface.host, interface.port)) - yield s - s.close() + try: + yield s + finally: + s.close() # NB: This does rudimentary smoke testing. See fuzzing.py for more thorough test diff --git a/tests/e2e_operations.py b/tests/e2e_operations.py index 67e7e43d5b63..bcea9bcc41f6 100644 --- a/tests/e2e_operations.py +++ b/tests/e2e_operations.py @@ -2582,7 +2582,9 @@ def run_initial_uvm_descriptor_checks(const_args): ) for chunk in ledger: _, chunk_end_seqno = chunk.get_seqnos() - if chunk_end_seqno < recovery_seqno: + # Open chunks have no end seqno, so they may contain the + # recovery transaction. Only skip chunks known to end earlier. + if chunk_end_seqno is not None and chunk_end_seqno < recovery_seqno: continue for tx in chunk: tables = tx.get_public_domain().get_tables() @@ -2665,7 +2667,9 @@ def run_initial_tcb_version_checks(const_args): ) for chunk in ledger: _, chunk_end_seqno = chunk.get_seqnos() - if chunk_end_seqno < recovery_seqno: + # Open chunks have no end seqno, so they may contain the + # recovery transaction. Only skip chunks known to end earlier. + if chunk_end_seqno is not None and chunk_end_seqno < recovery_seqno: continue for tx in chunk: tables = tx.get_public_domain().get_tables() diff --git a/tests/infra/clients.py b/tests/infra/clients.py index 9be9fa6c5df0..680211f3cc44 100644 --- a/tests/infra/clients.py +++ b/tests/infra/clients.py @@ -1334,8 +1334,10 @@ def close(self): @contextlib.contextmanager def client(*args, **kwargs): c = CCFClient(*args, **kwargs) - yield c - c.close() + try: + yield c + finally: + c.close() class APIVersionedCCFClient(CCFClient): @@ -1368,5 +1370,7 @@ def call(self, path: str, *args, **kwargs): @contextlib.contextmanager def api_versioned_client(*args, api_version=None, **kwargs): c = APIVersionedCCFClient(*args, api_version=api_version, **kwargs) - yield c - c.close() + try: + yield c + finally: + c.close() diff --git a/tests/tls_groups.py b/tests/tls_groups.py index 7cba3c9006c0..0fac8195556a 100644 --- a/tests/tls_groups.py +++ b/tests/tls_groups.py @@ -8,7 +8,8 @@ import suite.test_requirements as reqs from loguru import logger as LOG -# Hybrid groups offered by src/tls/context.h, in the order they are offered +# Hybrid groups offered by src/host/tls/openssl_server.h, in the order they are +# offered HYBRID_GROUPS = ["SecP384r1MLKEM1024", "SecP256r1MLKEM768", "X25519MLKEM768"] # Weakest classical fallback, and the name OpenSSL reports for it