diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..2e60f62 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,45 @@ +# AGENTS.md + +## Repo Snapshot + +- This repo is an archived Ruby gem that wraps the Apache Pulsar C++ client through a native extension built with Rice. +- The main Ruby entrypoint is `lib/pulsar/client.rb`; it loads `pulsar/bindings`, so most meaningful changes either touch the Ruby wrappers under `lib/pulsar/` or the extension under `ext/bindings/`. +- `rake` is the real CI entrypoint. In `Rakefile`, the default task is `[:compile, :spec]`. + +## Setup And Build + +- This gem links against `libpulsar`; local work needs both the runtime library and C++ headers installed before `bundle install` or `rake compile` will succeed. +- `ext/bindings/extconf.rb` links with `-lpulsar` and forces `-std=c++11`. +- The README calls out one Ruby-specific prerequisite: if Ruby was not built with `--enable-shared`, native extension loading can fail. The documented example is `CONFIGURE_OPTS="--enable-shared" rbenv install `. +- `bin/setup` only runs `bundle install`; it does not install system dependencies. +- The lockfile is old on purpose: `bundler ~> 1.16`, `rake ~> 10.0`, `rspec ~> 3.0`. + +## Commands + +- Install gem deps: `bin/setup` +- Compile the extension: `bundle exec rake compile` +- Run all specs: `bundle exec rake spec` +- Run the CI-equivalent local flow: `bundle exec rake` +- Open a local console with the library loaded: `bin/console` +- Install the gem locally: `bundle exec rake install` +- Run one spec file directly: `bundle exec rspec spec/pulsar/producer_spec.rb` + +## Testing Notes + +- `spec/spec_helper.rb` requires `pulsar/client`, so even Ruby-only specs expect the native extension to be built and loadable. +- Live broker coverage is in `spec/pulsar/client_spec.rb`. Those examples skip unless both `PULSAR_BROKER_URI` and `PULSAR_CLIENT_RUBY_TEST_NAMESPACE` are set. +- CI (`.travis.yml`) starts a Pulsar broker in Docker, creates tenant `ruby-client` and namespace `ruby-client/tests`, then runs `rake` with: + - `PULSAR_BROKER_URI=pulsar://localhost:6650` + - `PULSAR_CLIENT_RUBY_TEST_NAMESPACE=ruby-client/tests` +- `spec/pulsar/ext_spec.rb` separately recompiles the fixture extension under `spec/pulsar/ext/` with `extconf.rb` + `make clean all`; changes to error wrapping or Rice bindings should be verified there as well. + +## Codebase Conventions + +- The Ruby wrappers use `prepend ...::RubySideTweaks` to smooth the generated/native API instead of replacing it outright. Preserve that pattern when adapting extension-backed classes such as `Pulsar::Client` and `Pulsar::Producer`. +- Environment-based client setup lives in `Pulsar::Client.from_environment` and `Pulsar::ClientConfiguration.from_environment`. If behavior depends on env vars or Pulsar `client.conf`, verify it in those two files first. +- Topic/subscription integration tests generate random non-persistent topics to avoid collisions; follow that pattern instead of hard-coding shared topic names. + +## Workflow Guardrails + +- There is no repo-local lint, formatter, or typecheck config. Do not invent new verification steps in this repo; use compile/spec tasks and any focused spec you changed. +- README usage examples are incomplete by its own admission (`TODO.md`), so prefer `Rakefile`, specs, and the wrapper classes as the source of truth when docs and behavior diverge. diff --git a/README.md b/README.md index 7db3c02..12f2e93 100644 --- a/README.md +++ b/README.md @@ -40,6 +40,10 @@ Setup and basic `consumer.receive` example: # export PULSAR_AUTH_TOKEN=your-auth-token # export PULSAR_AUTH_OAUTH2_PARAMS=your-oauth2-params +# TLS is handled by the broker URL and TLS-specific options such as +# PULSAR_CERT_PATH. The client no longer exposes a separate `use_tls` +# toggle. + # create client using values from environment client = Pulsar::Client.from_environment @@ -88,6 +92,10 @@ listenerThread.join # wait for the thread to finish ## Development +If you are using `mise` on current macOS, this repo includes a repo-local +Ruby 3.1.7 patch for the missing `socket` extension issue discussed in +`jdx/mise#9703`. Run `mise install` from the repo root before `bundle install`. + If your ruby is not already compiled with `--enable-shared`, you'll need to rebuild it. Example for rbenv: @@ -102,6 +110,29 @@ automake for the compilation and linking to work. Example with brew: brew install libpulsar automake ``` +On Ubuntu, install the Pulsar client runtime and headers plus build +tools before running Bundler. For example: + +``` +sudo apt-get update +sudo apt-get install -y automake libpulsar libpulsar-dev +``` + +If your distro installs `libpulsar` outside standard system paths, pass +the location through extconf options such as +`--with-pulsar-dir=/custom/prefix`. + +To verify the Linux build in the provided Ubuntu-based container image, +run: + +``` +docker run --rm --entrypoint /bin/bash \ + -v "$PWD:/workspace" \ + -w /workspace \ + 127178877223.dkr.ecr.us-east-2.amazonaws.com/learn-test/learn-base-image:1774966705 \ + -lc 'bundle install && bundle exec rake compile && bundle exec ruby -e "require "'"'pulsar/client'"'"'; puts Pulsar::Client::VERSION"' +``` + Next, run `bin/setup` to install dependencies -- Rice in particular. Once that successfully completes, you can `rake compile` to build the extension. It is then ready to use locally. diff --git a/Rakefile b/Rakefile index 4055a85..8634dd5 100644 --- a/Rakefile +++ b/Rakefile @@ -1,6 +1,30 @@ require "rspec/core/rake_task" require "rake/extensiontask" +# Rake 10 forwards FileUtils options positionally, but Ruby 3 expects +# keyword arguments for methods like mkdir_p. +if RUBY_VERSION >= "3.0" + module Rake + module FileUtilsExt + def mkdir_p(*args, **kwargs, &block) + super(*args, **kwargs, &block) + end + + def chdir(*args, **kwargs, &block) + super(*args, **kwargs, &block) + end + + def cp(*args, **kwargs, &block) + super(*args, **kwargs, &block) + end + + def install(*args, **kwargs, &block) + super(*args, **kwargs, &block) + end + end + end +end + RSpec::Core::RakeTask.new(:spec) task :default => [:compile, :spec] diff --git a/docs/superpowers/plans/2026-06-19-ubuntu-compile-plan.md b/docs/superpowers/plans/2026-06-19-ubuntu-compile-plan.md new file mode 100644 index 0000000..eb2a427 --- /dev/null +++ b/docs/superpowers/plans/2026-06-19-ubuntu-compile-plan.md @@ -0,0 +1,156 @@ +# Ubuntu Compile Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make the native extension build discovery work on modern Ubuntu as well as the currently fixed macOS setup. + +**Architecture:** Keep the current Ruby 3, Bundler, and binding fixes intact. Only refactor the extension configuration entrypoints so they discover `libpulsar` through explicit overrides first, then platform-appropriate defaults instead of a Homebrew-only path. + +**Tech Stack:** Ruby 3.1.7, Bundler 1.17.3 lockfile, Rake, rake-compiler, Rice, native C++ extension, libpulsar. + +--- + +### Task 1: Refactor Main Extension Discovery + +**Files:** +- Modify: `ext/bindings/extconf.rb` +- Test: `ext/bindings/extconf.rb` via `ruby ext/bindings/extconf.rb --with-pulsar-*` or `bundle exec rake compile` + +- [ ] **Step 1: Replace the Homebrew-only path logic with override-first discovery** + +Use this structure in `ext/bindings/extconf.rb`: + +```ruby +require 'mkmf-rice' +require 'rbconfig' + +DEFAULT_PULSAR_DIRS = if RbConfig::CONFIG['host_os'] =~ /darwin/ + ['/opt/homebrew/opt/libpulsar'] +else + ['/usr', '/usr/local'] +end.freeze + +def existing_dir(paths) + paths.find { |path| path && File.directory?(path) } +end + +pulsar_dir = with_config('pulsar-dir') || existing_dir(DEFAULT_PULSAR_DIRS) +include_dir, lib_dir = dir_config( + 'pulsar', + pulsar_dir && File.join(pulsar_dir, 'include'), + pulsar_dir && File.join(pulsar_dir, 'lib') +) + +client_header = File.join(include_dir.to_s, 'pulsar', 'Client.h') + +abort 'libpulsar headers not found' unless File.exist?(client_header) +abort 'libpulsar library not found' unless have_library('pulsar') + +$CXXFLAGS += ' -std=c++17 ' + +create_makefile('pulsar/bindings') +``` + +- [ ] **Step 2: Confirm explicit overrides still win over defaults** + +Run: + +```bash +ruby -e 'require "mkmf"; p with_config("pulsar-dir")' +``` + +Expected: `nil` without args, and a string value if extconf is run with `--with-pulsar-dir=...`. + +- [ ] **Step 3: Verify macOS extconf still resolves the current Homebrew install** + +Run: + +```bash +mise exec -- bundle _1.17.3_ exec ruby ext/bindings/extconf.rb +``` + +Expected: successful header/library checks and `creating Makefile`. + + +### Task 2: Refactor Fixture Extension Discovery + +**Files:** +- Modify: `spec/pulsar/ext/extconf.rb` +- Test: `spec/pulsar/ext/extconf.rb` + +- [ ] **Step 1: Mirror the same discovery logic in the fixture extension** + +Use the same code pattern as Task 1, but keep the final makefile target unchanged: + +```ruby +create_makefile('bindings') +``` + +- [ ] **Step 2: Verify the fixture extconf still succeeds on the current machine** + +Run: + +```bash +mise exec -- bundle _1.17.3_ exec ruby spec/pulsar/ext/extconf.rb +``` + +Expected: successful header/library checks and `creating Makefile`. + + +### Task 3: Document Ubuntu Compile Setup + +**Files:** +- Modify: `README.md` + +- [ ] **Step 1: Add a short Ubuntu note to the Development section** + +Add a compact note near the existing dependency/setup section such as: + +```markdown +On Ubuntu, install the Pulsar client runtime and headers plus build tools before running Bundler. For example: + +```bash +sudo apt-get update +sudo apt-get install -y automake libpulsar-dev libpulsar +``` + +If your distro installs `libpulsar` outside standard system paths, pass the location through extconf options such as `--with-pulsar-dir=/custom/prefix`. +``` + +- [ ] **Step 2: Keep the macOS `mise` note intact** + +Do not remove the existing macOS-specific `mise` workaround; just extend the section so both setups are documented. + + +### Task 4: Re-verify Current Compile Flow + +**Files:** +- Modify: none +- Test: native compile flow + +- [ ] **Step 1: Run the current compile flow end-to-end** + +Run: + +```bash +mise exec -- bundle _1.17.3_ exec rake compile +``` + +Expected: compile completes and produces `lib/pulsar/bindings.bundle`. + +- [ ] **Step 2: Verify the built extension still loads** + +Run: + +```bash +mise exec -- bundle _1.17.3_ exec ruby -e 'require "pulsar/client"; puts Pulsar::Client::VERSION' +``` + +Expected: prints the gem version. + +- [ ] **Step 3: Commit** + +```bash +git add ext/bindings/extconf.rb spec/pulsar/ext/extconf.rb README.md docs/superpowers/specs/2026-06-19-ubuntu-compile-design.md docs/superpowers/plans/2026-06-19-ubuntu-compile-plan.md +git commit -m "build: make libpulsar discovery cross-platform" +``` diff --git a/docs/superpowers/specs/2026-06-19-ubuntu-compile-design.md b/docs/superpowers/specs/2026-06-19-ubuntu-compile-design.md new file mode 100644 index 0000000..b52fa57 --- /dev/null +++ b/docs/superpowers/specs/2026-06-19-ubuntu-compile-design.md @@ -0,0 +1,82 @@ +# Ubuntu Compile Design + +## Goal + +Make this gem compile reasonably well on a modern Ubuntu Linux system with a packaged `libpulsar`, without changing gem versions or proving the live broker specs on Linux. + +## Current Problem + +The current `extconf.rb` logic is macOS/Homebrew-specific: + +- it hard-codes `/opt/homebrew/opt/libpulsar` +- it checks headers only in that prefix +- it does not provide a Linux-friendly fallback path + +That means the repo now builds on the patched macOS setup, but it is less likely to compile on Ubuntu where `libpulsar` headers and libraries normally live under system paths. + +## Chosen Approach + +Use a cross-platform `extconf.rb` cleanup. + +This keeps the recent Ruby 3 / Rake compatibility fixes and the `libpulsar` API fixes, and only changes build discovery so Linux can find `libpulsar` without Homebrew-specific assumptions. + +## Design + +Update both extension entrypoints: + +- `ext/bindings/extconf.rb` +- `spec/pulsar/ext/extconf.rb` + +Both files should: + +1. Honor explicit user overrides first: + - `--with-pulsar-dir` + - `--with-pulsar-include` + - `--with-pulsar-lib` + +2. Choose platform defaults only when overrides are absent: + - macOS: prefer `/opt/homebrew/opt/libpulsar` + - Linux: prefer system include/lib locations + +3. Validate the chosen paths: + - confirm `pulsar/Client.h` exists in the selected include path + - confirm `have_library("pulsar")` succeeds with the selected library path + +4. Keep the C++ standard requirement at `-std=c++17` + +## Linux Assumptions + +For Ubuntu compile-only support, assume: + +- `libpulsar` runtime and headers are installed separately from Ruby gems +- common install locations are under `/usr/include` and `/usr/lib*` +- users can still override locations explicitly if their distro or local install differs + +## README Changes + +Add a short Ubuntu-oriented setup note in `README.md`: + +- install `libpulsar` runtime/dev packages and `automake` +- run `bundle install` +- run `bundle exec rake compile` + +Keep the existing macOS `mise` workaround note unchanged. + +## Verification Plan + +On the current machine: + +1. Verify the refactored extconf still works with the current macOS/Homebrew setup. +2. Verify the existing successful compile flow still works: + - `mise exec -- bundle _1.17.3_ exec rake compile` + +For Linux-readiness in repo logic: + +1. Confirm the new fallback logic no longer hard-codes Homebrew-only paths. +2. Confirm explicit `--with-pulsar-*` overrides still take precedence. + +## Non-Goals + +- Upgrading Bundler, Rake, Rice, or other gems +- Proving the full spec suite on Ubuntu +- Adding or updating CI in this pass diff --git a/ext/bindings/client.cpp b/ext/bindings/client.cpp index ab1fbd9..4b11541 100644 --- a/ext/bindings/client.cpp +++ b/ext/bindings/client.cpp @@ -77,19 +77,12 @@ bool ClientConfiguration::getSilentLogging() { return silentLogging; } -bool ClientConfiguration::isUseTls() { - return _config.isUseTls(); -} - -void ClientConfiguration::setUseTls(bool enable) { - _config.setUseTls(enable); -} - std::string ClientConfiguration::getTlsTrustCertsFilePath() { - return _config.getTlsTrustCertsFilePath(); + return tlsTrustCertsFilePath; } void ClientConfiguration::setTlsTrustCertsFilePath(const std::string& path) { + tlsTrustCertsFilePath = path; _config.setTlsTrustCertsFilePath(path); } @@ -214,8 +207,6 @@ void bind_client(Module& module) { .define_method("log_conf_file_path=", &pulsar_rb::ClientConfiguration::setLogConfFilePath) .define_method("silent_logging?", &pulsar_rb::ClientConfiguration::getSilentLogging) .define_method("silent_logging=", &pulsar_rb::ClientConfiguration::setSilentLogging) - .define_method("use_tls?", &pulsar_rb::ClientConfiguration::isUseTls) - .define_method("use_tls=", &pulsar_rb::ClientConfiguration::setUseTls) .define_method("tls_trust_certs_file_path", &pulsar_rb::ClientConfiguration::getTlsTrustCertsFilePath) .define_method("tls_trust_certs_file_path=", &pulsar_rb::ClientConfiguration::setTlsTrustCertsFilePath) .define_method("tls_allow_insecure_connection?", &pulsar_rb::ClientConfiguration::isTlsAllowInsecureConnection) diff --git a/ext/bindings/client.hpp b/ext/bindings/client.hpp index 3bfa148..563a26c 100644 --- a/ext/bindings/client.hpp +++ b/ext/bindings/client.hpp @@ -14,6 +14,7 @@ namespace pulsar_rb { public: pulsar::ClientConfiguration _config; bool silentLogging = false; + std::string tlsTrustCertsFilePath; ClientConfiguration(); void setAuthFromToken(const std::string &token); @@ -30,8 +31,6 @@ namespace pulsar_rb { void setLogConfFilePath(const std::string& path); void setSilentLogging(bool); bool getSilentLogging(); - bool isUseTls(); - void setUseTls(bool enable); std::string getTlsTrustCertsFilePath(); void setTlsTrustCertsFilePath(const std::string& path); bool isTlsAllowInsecureConnection(); diff --git a/ext/bindings/extconf.rb b/ext/bindings/extconf.rb index 6533b54..278230f 100644 --- a/ext/bindings/extconf.rb +++ b/ext/bindings/extconf.rb @@ -1,4 +1,29 @@ require 'mkmf-rice' -$LOCAL_LIBS << "-lpulsar" -$CXXFLAGS += " -std=c++11 " +require 'rbconfig' + +DEFAULT_PULSAR_DIRS = if RbConfig::CONFIG['host_os'] =~ /darwin/ + ['/opt/homebrew/opt/libpulsar'] +else + ['/usr', '/usr/local'] +end.freeze + +def existing_dir(paths) + paths.find { |path| path && File.directory?(path) } +end + +pulsar_dir = with_config('pulsar-dir') || existing_dir(DEFAULT_PULSAR_DIRS) + +include_dir, = dir_config( + 'pulsar', + pulsar_dir && File.join(pulsar_dir, 'include'), + pulsar_dir && File.join(pulsar_dir, 'lib') +) + +client_header = File.join(include_dir.to_s, 'pulsar', 'Client.h') + +abort 'libpulsar headers not found' unless File.exist?(client_header) +abort 'libpulsar library not found' unless have_library('pulsar') + +$CXXFLAGS += ' -std=c++17 ' unless $CXXFLAGS.include?('-std=') + create_makefile('pulsar/bindings') diff --git a/lib/pulsar/client_configuration.rb b/lib/pulsar/client_configuration.rb index 9762c2c..ce22b08 100644 --- a/lib/pulsar/client_configuration.rb +++ b/lib/pulsar/client_configuration.rb @@ -39,7 +39,6 @@ def self.from(config) def self.from_environment(config={}, environment=ENV.to_h) environment_config = {} if environment.has_key?('PULSAR_CERT_PATH') - environment_config[:use_tls] = true environment_config[:tls_allow_insecure_connection] = false environment_config[:tls_validate_hostname] = false environment_config[:tls_trust_certs_file_path] = environment['PULSAR_CERT_PATH'] @@ -66,7 +65,6 @@ def self.read_from_client_conf(pulsar_client_conf_file) # If a TLS certificate had been given, use it if pulsar_config.has_key? 'tlsTrustCertsFilePath' - client_config[:use_tls] = true client_config[:tls_trust_certs_file_path] = pulsar_config['tlsTrustCertsFilePath'] end # If 'TLS enable hostname verification' is false, then switch it off in config @@ -139,7 +137,6 @@ def populate(config={}) populate_one(config, :concurrent_lookup_requests) populate_one(config, :log_conf_file_path) populate_one(config, :silent_logging) - populate_one(config, :use_tls) populate_one(config, :tls_trust_certs_file_path) populate_one(config, :tls_allow_insecure_connection) populate_one(config, :tls_validate_hostname) diff --git a/mise.toml b/mise.toml new file mode 100644 index 0000000..a458ff7 --- /dev/null +++ b/mise.toml @@ -0,0 +1,5 @@ +[tools] +ruby = "3.1.7" + +[settings.ruby] +apply_patches = "./patches/ruby-3.1.7-darwin-socket.patch" diff --git a/patches/ruby-3.1.7-darwin-socket.patch b/patches/ruby-3.1.7-darwin-socket.patch new file mode 100644 index 0000000..3bdf193 --- /dev/null +++ b/patches/ruby-3.1.7-darwin-socket.patch @@ -0,0 +1,17 @@ +diff --git a/ext/socket/extconf.rb b/ext/socket/extconf.rb +--- a/ext/socket/extconf.rb ++++ b/ext/socket/extconf.rb +@@ -660,5 +660,12 @@ + int t(struct in6_addr *addr) {return IN6_IS_ADDR_UNSPECIFIED(addr);} + SRC + print "fixing apple's netinet6/in6.h ..."; $stdout.flush +- in6 = File.read("/usr/include/#{hdr}") ++ file = xpopen(%w"clang -include netinet/in.h -E -xc -", in: IO::NULL) do |f| ++ re = %r[^# *\d+ *"(.*/netinet/in\.h)"] ++ Logging.message " grep(#{re})\n" ++ f.read[re, 1] ++ end ++ Logging.message "Substitute from #{file}\n" ++ ++ in6 = File.read(file) + if in6.gsub!(/\*\(const\s+__uint32_t\s+\*\)\(const\s+void\s+\*\)\(&\(\(\w+\)\)->s6_addr\[(\d+)\]\)/) do diff --git a/spec/pulsar/client_configuration_spec.rb b/spec/pulsar/client_configuration_spec.rb index 94853da..a80c256 100644 --- a/spec/pulsar/client_configuration_spec.rb +++ b/spec/pulsar/client_configuration_spec.rb @@ -32,7 +32,6 @@ 'PULSAR_CERT_PATH' => '/path/to/cert.pem' } config = Pulsar::ClientConfiguration.from_environment({}, test_env) - expect(config[:use_tls]).to eq(true) expect(config[:tls_allow_insecure_connection]).to eq(false) expect(config[:tls_validate_hostname]).to eq(false) expect(config[:tls_trust_certs_file_path]).to eq('/path/to/cert.pem') @@ -116,7 +115,6 @@ expect(config[:tls_allow_insecure_connection]).to eq(false) expect(config[:tls_validate_hostname]).to eq(false) expect(config[:tls_trust_certs_file_path]).to eq('/test/cert/file/ca.pem') - expect(config[:use_tls]).to eq(true) ensure test_config.unlink test_token.unlink @@ -133,7 +131,6 @@ expect(config[:tls_allow_insecure_connection]).to eq(nil) expect(config[:tls_validate_hostname]).to eq(nil) expect(config[:tls_trust_certs_file_path]).to eq(nil) - expect(config[:use_tls]).to eq(nil) end it 'handle when token file is not found without exceptions' do diff --git a/spec/pulsar/ext/extconf.rb b/spec/pulsar/ext/extconf.rb index 0aa5276..3085b81 100644 --- a/spec/pulsar/ext/extconf.rb +++ b/spec/pulsar/ext/extconf.rb @@ -1,3 +1,29 @@ require 'mkmf-rice' -$CXXFLAGS += ' -std=c++11 ' +require 'rbconfig' + +DEFAULT_PULSAR_DIRS = if RbConfig::CONFIG['host_os'] =~ /darwin/ + ['/opt/homebrew/opt/libpulsar'] +else + ['/usr', '/usr/local'] +end.freeze + +def existing_dir(paths) + paths.find { |path| path && File.directory?(path) } +end + +pulsar_dir = with_config('pulsar-dir') || existing_dir(DEFAULT_PULSAR_DIRS) + +include_dir, = dir_config( + 'pulsar', + pulsar_dir && File.join(pulsar_dir, 'include'), + pulsar_dir && File.join(pulsar_dir, 'lib') +) + +client_header = File.join(include_dir.to_s, 'pulsar', 'Client.h') + +abort 'libpulsar headers not found' unless File.exist?(client_header) +abort 'libpulsar library not found' unless have_library('pulsar') + +$CXXFLAGS += ' -std=c++17 ' unless $CXXFLAGS.include?('-std=') + create_makefile('bindings')