Feat: Prototype Static Big-O Analyzer for Espalier#143
Merged
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
| def load_registry(language) | ||
| path = File.join(__dir__, "stdlib_complexity_#{language}.yml") | ||
| return {} unless File.exist?(path) |
| def initialize(language: :ruby, nil_kill_evidence: {}) | ||
| @language = language | ||
| @nil_kill_evidence = nil_kill_evidence | ||
| @registry = load_registry(language) |
- Fix type inference wrapper in type_profile.rb to prevent TypeError on generic strings. - Enable NilKillEvidence to load dynamic loop iteration counts from runtime trace logs. - Update Espalier CLI/Aggregator to run BigOAnalyzer and attach big_o metrics to methods. - Enhance BigOAnalyzer type resolution (dotted chains, self calls, ivars) and complexity math. - Fix Nil-kill test failures in hash shape merging and Lua table normalization. - Add big_o_test.rb unit tests for analyzer correctness. Co-authored-by: gemini-cli <218195315+gemini-cli@users.noreply.github.com>
- Report unknown method operations when receiver type is known but complexity is missing from stdlib_complexity_ruby.yml. - Output warnings and unknowns list inside quality_metrics in the manifest. - Exclude internal warnings/unknowns metadata from the markdown report to keep tables clean. - Create tools/espalier_diagnostic.rb to show unexpected poorly performing functions, uncertainty stats, and suggested YAML config expansions. Co-authored-by: gemini-cli <218195315+gemini-cli@users.noreply.github.com>
- Allows developers to run 'ruby tools/espalier_diagnostic.rb --explain <MethodName>' to see a step-by-step breakdown of how the complexity was derived. - Helps identify if loops are sequential or nested, and highlights when a fallback method boundary causes trailing loops to be counted. Co-authored-by: gemini-cli <218195315+gemini-cli@users.noreply.github.com>
|
Bencher Report| Branch | espalier-big-o |
| Testbed | ubuntu-latest |
⚠️ WARNING: No Threshold found!Without a Threshold, no Alerts will ever be generated.
Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the--ci-only-thresholdsflag.
Click to view all benchmark results
| Benchmark | leak-build-ms | Measure (units) x 1e3 | leak-count | Measure (units) | leak-run-ms | Measure (units) |
|---|---|---|---|---|---|---|
| benchmarks/concurrent/01_socket_throughput/bench | 📈 view plot | 6.34 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 7.74 units |
| benchmarks/concurrent/06_dynamic_spawn/bench | 📈 view plot | 6.05 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 38.43 units |
| benchmarks/concurrent/11_parallel_aggregation/bench | 📈 view plot | 6.19 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 18.32 units |
| benchmarks/concurrent/18_atomic_counter/bench | 📈 view plot | 6.24 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 5.86 units |
| benchmarks/inter-clear/04_concurrent_mvcc_fat_struct/bench | 📈 view plot | 6.33 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 165.24 units |
| benchmarks/sequential/03_alloc_throughput/bench | 📈 view plot | 5.88 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 24.22 units |
| benchmarks/sequential/08_sort/bench | 📈 view plot | 6.02 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 5.09 units |
| benchmarks/sequential/13_soa_layout/bench | 📈 view plot | 5.84 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 743.91 units |
Comment on lines
+95
to
+96
| elsif signature =~ /->\s*([A-Za-z0-9_:]+)/ | ||
| $1.strip |
|
|
||
| big_o = quality[:big_o] | ||
| warnings = quality[:big_o_warnings] || [] | ||
| unknowns = quality[:big_o_unknowns] || [] |
Diff Coverage BucketsDiff base:
Src Ruby Visibility BreakdownScope: Files: 163 Counts are nonblank, non-comment Ruby source lines. Protected methods are grouped into
Zig Special Coverage AlertsNo added production Zig lines require missing Loom/VOPR/wait-loop coverage alerts. |
| @@ -0,0 +1,297 @@ | |||
| #!/usr/bin/env ruby | |||
There was a problem hiding this comment.
SlopCop found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
Comment on lines
+117
to
+145
| def resolve_simple_type(receiver_name, line) | ||
| if receiver_name == "self" | ||
| return clean_type_name(@class_name) | ||
| end | ||
|
|
||
| if receiver_name.start_with?("@") | ||
| type = @ivar_types[receiver_name] || @ivar_types[receiver_name[1..]] | ||
| return clean_type_name(type) if type | ||
| end | ||
|
|
||
| if (type = @ivar_types["@#{receiver_name}"] || @ivar_types[receiver_name]) | ||
| return clean_type_name(type) | ||
| end | ||
|
|
||
| if @class_name && @nil_kill | ||
| sig = @nil_kill.method_signatures["#{@class_name}##{receiver_name}"] | ||
| if sig | ||
| ret = extract_return_type(sig) | ||
| return clean_type_name(ret) if ret | ||
| end | ||
| end | ||
|
|
||
| if @nil_kill_evidence | ||
| type = @nil_kill_evidence.dig(line.to_s, receiver_name) || @nil_kill_evidence.dig(line.to_s, "@#{receiver_name}") | ||
| return clean_type_name(type) if type | ||
| end | ||
|
|
||
| nil | ||
| end |
Comment on lines
+124
to
139
| dir = File.dirname(evidence_path) | ||
| runtime_dir = File.join(dir, "runtime") | ||
| return unless File.directory?(runtime_dir) | ||
|
|
||
| Dir.glob(File.join(runtime_dir, "loops-*.jsonl")).each do |loop_file| | ||
| File.readlines(loop_file).each do |line| | ||
| data = JSON.parse(line) rescue next | ||
| path = data["path"] | ||
| line_num = data["line"]&.to_i | ||
| if path && line_num | ||
| @loop_counts[path][line_num] += 1 | ||
| end | ||
| end | ||
| end | ||
| end | ||
| end |
Comment on lines
+1396
to
+1399
| def self.record_loop_iteration(path, line) | ||
| @loop_file ||= File.open(File.join(OUT_DIR, "loops-#{Process.pid}.jsonl"), "a") | ||
| @loop_file.puts %({"path":#{path.inspect},"line":#{line}}) | ||
| end |
| @decomplex_data = decomplex_data | ||
| @nil_kill_data = nil_kill_data | ||
| @risk_data = risk_data | ||
| @nil_kill_loops = nil_kill_loops |
| @nil_kill_data = nil_kill_data | ||
| @risk_data = risk_data | ||
| @nil_kill_loops = nil_kill_loops | ||
| @nil_kill_evidence = nil_kill_evidence |
| attr_reader :registry, :nil_kill_evidence | ||
|
|
||
| def initialize(language: :ruby, nil_kill_evidence: {}, class_name: nil, ivar_types: {}, nil_kill: nil) | ||
| @language = language |
| @nil_kill_evidence = nil_kill_evidence | ||
| @class_name = class_name | ||
| @ivar_types = ivar_types || {} | ||
| @nil_kill = nil_kill |
| @state_types = nested_state_map(facts["state_types"] || {}) | ||
| @state_param_origins = nested_state_map(facts["state_param_origins"] || facts["ivar_param_origins"] || {}) | ||
| @state_protocols = nested_state_map(facts["state_protocols"] || facts["ivar_protocols"] || {}) | ||
| @loop_counts = Hash.new { |h, k| h[k] = Hash.new(0) } |
| end | ||
|
|
||
| def self.record_loop_iteration(path, line) | ||
| @loop_file ||= File.open(File.join(OUT_DIR, "loops-#{Process.pid}.jsonl"), "a") |
| end | ||
|
|
||
| def state_accessor_return_type(class_name, method_name) | ||
| return nil unless @nil_kill&.respond_to?(:state_types) |
Comment on lines
+38
to
+46
| def initialize(language: :ruby, nil_kill_evidence: {}, class_name: nil, ivar_types: {}, nil_kill: nil, local_types: {}) | ||
| @language = language | ||
| @nil_kill_evidence = nil_kill_evidence | ||
| @class_name = class_name | ||
| @ivar_types = ivar_types || {} | ||
| @local_types = local_types || {} | ||
| @nil_kill = nil_kill | ||
| @registry = load_registry(language) | ||
| end |
Comment on lines
+57
to
+108
| def analyze_method(method_name, ast_nodes, local_types: nil) | ||
| previous_local_types = @local_types | ||
| @local_types = local_types if local_types | ||
| complexity = "O(1)" | ||
| unknown_operations = [] | ||
| warnings = [] | ||
|
|
||
| # Lower bound calculation | ||
| # For a prototype, we just scan for the most complex operation in the flat AST. | ||
| # In reality, this would recursively multiply nested loops. | ||
|
|
||
| ast_nodes.each do |node| | ||
| if node[:type] == :call | ||
| receiver_type = resolve_type(node[:receiver], node[:line]) | ||
| method_called = node[:method].to_s | ||
|
|
||
| if receiver_type | ||
| known_complexity = @registry.dig(receiver_type, method_called) | ||
| if known_complexity | ||
| # If it's sequential, we just take the max of what we've seen so far. | ||
| complexity = max_complexity(complexity, known_complexity) | ||
| elsif (chained_complexity = flattened_chain_complexity(node, ast_nodes)) | ||
| complexity = max_complexity(complexity, chained_complexity) | ||
| elsif state_accessor_return_type(receiver_type, method_called) | ||
| complexity = max_complexity(complexity, "O(1)") | ||
| else | ||
| unknown_operations << "#{receiver_type}##{method_called}" | ||
| warnings << "Missing method complexity for `#{receiver_type}##{method_called}` in stdlib_complexity_ruby.yml at line #{node[:line]}." | ||
| end | ||
| else | ||
| unknown_operations << "#{node[:receiver]}.#{method_called}" | ||
| warnings << "Unknown receiver type for `#{node[:receiver]}` at line #{node[:line]}. Defaulting to O(1) for `.#{method_called}`, but this could be worse." | ||
| end | ||
| elsif node[:type] == :loop | ||
| # Runtime loop evidence is currently flat by source line. Without a | ||
| # nesting tree, multiple observed loops in one method are a sequential | ||
| # lower bound, not proof of nested O(N^k) behavior. | ||
| complexity = max_complexity(complexity, "O(N)") | ||
| elsif node[:type] == :callback || node[:type] == :yield | ||
| warnings << "Function pointer / callback executed at line #{node[:line]}. This could execute arbitrary O(N^x) code, meaning our calculation is strictly a LOWER BOUND." | ||
| end | ||
| end | ||
|
|
||
| { | ||
| method: method_name, | ||
| lower_bound_complexity: complexity, | ||
| unknown_operations: unknown_operations.uniq, | ||
| warnings: warnings.uniq | ||
| } | ||
| ensure | ||
| @local_types = previous_local_types if local_types | ||
| end |
Comment on lines
+188
to
+206
| def resolve_method_return_type(class_name, method_name) | ||
| return nil unless class_name | ||
|
|
||
| if @nil_kill | ||
| sig = @nil_kill.method_signatures["#{class_name}##{method_name}"] | ||
| if sig | ||
| ret = extract_return_type(sig) | ||
| return clean_type_name(ret) | ||
| end | ||
|
|
||
| state_type = state_accessor_return_type(class_name, method_name) | ||
| return state_type if state_type | ||
| end | ||
|
|
||
| stdlib_type = STDLIB_RETURN_TYPES.dig(class_name, method_name) | ||
| return clean_type_name(stdlib_type) if stdlib_type | ||
|
|
||
| nil | ||
| end |
Comment on lines
+208
to
+212
| def state_accessor_return_type(class_name, method_name) | ||
| return nil unless @nil_kill&.respond_to?(:state_types) | ||
|
|
||
| clean_type_name(@nil_kill.state_types.dig(class_name, "@#{method_name}")) | ||
| end |
Comment on lines
+214
to
+232
| def flattened_chain_complexity(node, ast_nodes) | ||
| receiver = node[:receiver].to_s | ||
| method_called = node[:method].to_s | ||
| line = node[:line] | ||
|
|
||
| Array(ast_nodes).each do |candidate| | ||
| next unless candidate[:type] == :call | ||
| next unless candidate[:receiver].to_s == receiver | ||
| next unless candidate[:line] == line | ||
| accessor = candidate[:method].to_s | ||
| next if accessor == method_called | ||
|
|
||
| chained_type = resolve_type("#{receiver}.#{accessor}", line) | ||
| known_complexity = @registry.dig(chained_type, method_called) | ||
| return known_complexity if known_complexity | ||
| end | ||
|
|
||
| nil | ||
| end |
|
|
||
| def initialize(language: :ruby, nil_kill_evidence: {}, class_name: nil, ivar_types: {}, nil_kill: nil, local_types: {}) | ||
| @language = language | ||
| @nil_kill_evidence = nil_kill_evidence |
| def initialize(language: :ruby, nil_kill_evidence: {}, class_name: nil, ivar_types: {}, nil_kill: nil, local_types: {}) | ||
| @language = language | ||
| @nil_kill_evidence = nil_kill_evidence | ||
| @class_name = class_name |
| @language = language | ||
| @nil_kill_evidence = nil_kill_evidence | ||
| @class_name = class_name | ||
| @ivar_types = ivar_types || {} |
| @nil_kill_evidence = nil_kill_evidence | ||
| @class_name = class_name | ||
| @ivar_types = ivar_types || {} | ||
| @local_types = local_types || {} |
| 10 + ($1.to_i * 2) + ($2 ? 1 : 0) | ||
| when "O(2^N)" then 100 | ||
| when "O(N!)" then 200 | ||
| else |
| 10 + ($1.to_i * 2) + ($2 ? 1 : 0) | ||
| when "O(2^N)" then 100 | ||
| when "O(N!)" then 200 | ||
| else |
| 10 + ($1.to_i * 2) + ($2 ? 1 : 0) | ||
| when "O(2^N)" then 100 | ||
| when "O(N!)" then 200 | ||
| else |
Comment on lines
+209
to
+228
| def preliminary_method_complexities(modules) | ||
| analyzer = Espalier::BigOAnalyzer.new( | ||
| language: :ruby, | ||
| nil_kill: @nil_kill_evidence | ||
| ) | ||
| modules.each_with_object(Hash.new { |h, k| h[k] = {} }) do |mod, complexities| | ||
| Array(mod[:methods]).each do |method| | ||
| analyzer.instance_variable_set(:@class_name, mod[:name]) | ||
| analyzer.instance_variable_set(:@ivar_types, mod[:ivar_types] || {}) | ||
| key = "#{mod[:name]}##{method[:name]}" | ||
| sig = @nil_kill_data[key] || method[:signature] | ||
| result = analyzer.analyze_method( | ||
| key, | ||
| big_o_nodes_for(mod, method), | ||
| local_types: local_types_for_signature(sig) | ||
| ) | ||
| complexities[mod[:name]][method[:name].to_s] = result[:lower_bound_complexity] | ||
| end | ||
| end | ||
| end |
Comment on lines
+230
to
+260
| def structural_method_complexities(modules) | ||
| complexities = preliminary_method_complexities(modules) | ||
| analyzer = Espalier::BigOAnalyzer.new( | ||
| language: :ruby, | ||
| nil_kill: @nil_kill_evidence | ||
| ) | ||
|
|
||
| 8.times do | ||
| changed = false | ||
| structural_big_o = Espalier::StructuralBigO.new(method_complexities: complexities) | ||
| modules.each do |mod| | ||
| Array(mod[:methods]).each do |method| | ||
| analyzer.instance_variable_set(:@class_name, mod[:name]) | ||
| analyzer.instance_variable_set(:@ivar_types, mod[:ivar_types] || {}) | ||
| key = "#{mod[:name]}##{method[:name]}" | ||
| sig = @nil_kill_data[key] || method[:signature] | ||
| nodes = big_o_nodes_for(mod, method) | ||
| nodes.concat(structural_big_o.hints_for(mod[:file], method, mod[:name])) | ||
| result = analyzer.analyze_method(key, nodes, local_types: local_types_for_signature(sig)) | ||
| current = complexities[mod[:name]][method[:name].to_s] || "O(1)" | ||
| next unless complexity_rank(result[:lower_bound_complexity]) > complexity_rank(current) | ||
|
|
||
| complexities[mod[:name]][method[:name].to_s] = result[:lower_bound_complexity] | ||
| changed = true | ||
| end | ||
| end | ||
| break unless changed | ||
| end | ||
|
|
||
| complexities | ||
| end |
Comment on lines
+278
to
+299
| def big_o_nodes_for(mod, method) | ||
| nodes = Array(method[:delegations]).map do |delegation| | ||
| { | ||
| type: :call, | ||
| receiver: delegation[:receiver], | ||
| method: delegation[:message], | ||
| line: delegation[:line] || method[:line] || 0 | ||
| } | ||
| end | ||
|
|
||
| meth_line, end_line, end_inclusive = method_line_bounds(mod[:methods], method) | ||
| file = mod[:file] | ||
| if file && @nil_kill_loops && @nil_kill_loops[file] | ||
| @nil_kill_loops[file].each do |line, calls| | ||
| if line_in_method_bounds?(line, meth_line, end_line, end_inclusive) && calls > 0 | ||
| nodes << { type: :loop, line: line, calls: calls } | ||
| end | ||
| end | ||
| end | ||
|
|
||
| nodes | ||
| end |
Comment on lines
+17
to
+22
| def initialize(source_cache: {}, method_complexities: {}) | ||
| @source_cache = source_cache | ||
| @method_complexities = method_complexities.transform_values do |methods| | ||
| methods.select { |_name, complexity| complexity_rank(complexity) >= complexity_rank("O(N)") } | ||
| end | ||
| end |
Comment on lines
+280
to
+297
| def project_call_hints(lines, loop_range, owner) | ||
| return [] unless owner | ||
|
|
||
| linear_methods = @method_complexities.fetch(owner, {}) | ||
| return [] if linear_methods.empty? | ||
|
|
||
| body_lines(lines, loop_range).filter_map do |line_no, line| | ||
| method_name = linear_methods.keys.find { |name| bare_method_call?(line, name) } | ||
| next unless method_name | ||
|
|
||
| structural_hint( | ||
| line: line_no, | ||
| operation: method_name, | ||
| reason: "known linear project call inside fixpoint loop", | ||
| detail: line.strip | ||
| ) | ||
| end | ||
| end |
Comment on lines
+299
to
+324
| def expensive_project_call_hints(lines, loop_range, owner, current_method_name) | ||
| return [] unless owner | ||
|
|
||
| current_method_name = current_method_name.sub(/\Aself\./, "") | ||
| expensive_methods = @method_complexities.fetch(owner, {}).select do |_name, complexity| | ||
| complexity_rank(complexity) >= complexity_rank("O(N^2)") | ||
| end | ||
| return [] if expensive_methods.empty? | ||
|
|
||
| body_lines(lines, loop_range).filter_map do |line_no, line| | ||
| method_name, method_complexity = expensive_methods.find do |name, _| | ||
| next false if name.sub(/\Aself\./, "") == current_method_name | ||
|
|
||
| bare_method_call?(line, name) | ||
| end | ||
| next unless method_name && method_complexity | ||
|
|
||
| structural_hint( | ||
| line: line_no, | ||
| complexity: multiply_complexity("O(N)", method_complexity), | ||
| operation: method_name, | ||
| reason: "known expensive project call inside loop", | ||
| detail: line.strip | ||
| ) | ||
| end | ||
| end |
| /x.freeze | ||
|
|
||
| def initialize(source_cache: {}, method_complexities: {}) | ||
| @source_cache = source_cache |
|
|
||
| def initialize(source_cache: {}, method_complexities: {}) | ||
| @source_cache = source_cache | ||
| @method_complexities = method_complexities.transform_values do |methods| |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.