Add Sofa RPC instrumentation (Bolt, H2C, REST, Triple)#11135
Add Sofa RPC instrumentation (Bolt, H2C, REST, Triple)#11135ValentinZakharov wants to merge 8 commits intomasterfrom
Conversation
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 60 metrics, 11 unstable metrics. Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.62.0-SNAPSHOT~055530140f, baseline=1.62.0-SNAPSHOT~95de525c87
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.058 s) : 0, 1058232
Total [baseline] (8.821 s) : 0, 8820576
Agent [candidate] (1.052 s) : 0, 1052253
Total [candidate] (8.831 s) : 0, 8830725
section iast
Agent [baseline] (1.23 s) : 0, 1230062
Total [baseline] (9.59 s) : 0, 9590214
Agent [candidate] (1.223 s) : 0, 1223121
Total [candidate] (9.53 s) : 0, 9530101
gantt
title insecure-bank - break down per module: candidate=1.62.0-SNAPSHOT~055530140f, baseline=1.62.0-SNAPSHOT~95de525c87
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.251 ms) : 0, 1251
crashtracking [candidate] (1.218 ms) : 0, 1218
BytebuddyAgent [baseline] (634.486 ms) : 0, 634486
BytebuddyAgent [candidate] (632.429 ms) : 0, 632429
AgentMeter [baseline] (29.43 ms) : 0, 29430
AgentMeter [candidate] (29.244 ms) : 0, 29244
GlobalTracer [baseline] (249.635 ms) : 0, 249635
GlobalTracer [candidate] (248.336 ms) : 0, 248336
AppSec [baseline] (32.493 ms) : 0, 32493
AppSec [candidate] (32.253 ms) : 0, 32253
Debugger [baseline] (59.449 ms) : 0, 59449
Debugger [candidate] (58.982 ms) : 0, 58982
Remote Config [baseline] (603.282 µs) : 0, 603
Remote Config [candidate] (586.177 µs) : 0, 586
Telemetry [baseline] (8.107 ms) : 0, 8107
Telemetry [candidate] (8.791 ms) : 0, 8791
Flare Poller [baseline] (6.573 ms) : 0, 6573
Flare Poller [candidate] (4.267 ms) : 0, 4267
section iast
crashtracking [baseline] (1.247 ms) : 0, 1247
crashtracking [candidate] (1.231 ms) : 0, 1231
BytebuddyAgent [baseline] (806.133 ms) : 0, 806133
BytebuddyAgent [candidate] (801.007 ms) : 0, 801007
AgentMeter [baseline] (11.514 ms) : 0, 11514
AgentMeter [candidate] (11.475 ms) : 0, 11475
GlobalTracer [baseline] (239.816 ms) : 0, 239816
GlobalTracer [candidate] (238.528 ms) : 0, 238528
AppSec [baseline] (30.533 ms) : 0, 30533
AppSec [candidate] (31.332 ms) : 0, 31332
Debugger [baseline] (63.387 ms) : 0, 63387
Debugger [candidate] (64.224 ms) : 0, 64224
Remote Config [baseline] (551.4 µs) : 0, 551
Remote Config [candidate] (560.961 µs) : 0, 561
Telemetry [baseline] (11.127 ms) : 0, 11127
Telemetry [candidate] (9.422 ms) : 0, 9422
Flare Poller [baseline] (3.383 ms) : 0, 3383
Flare Poller [candidate] (3.421 ms) : 0, 3421
IAST [baseline] (25.937 ms) : 0, 25937
IAST [candidate] (25.758 ms) : 0, 25758
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.62.0-SNAPSHOT~055530140f, baseline=1.62.0-SNAPSHOT~95de525c87
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.071 s) : 0, 1070709
Total [baseline] (11.15 s) : 0, 11149982
Agent [candidate] (1.072 s) : 0, 1072126
Total [candidate] (11.092 s) : 0, 11091643
section appsec
Agent [baseline] (1.254 s) : 0, 1253802
Total [baseline] (11.138 s) : 0, 11138024
Agent [candidate] (1.25 s) : 0, 1249515
Total [candidate] (11.07 s) : 0, 11069515
section iast
Agent [baseline] (1.232 s) : 0, 1232453
Total [baseline] (11.386 s) : 0, 11385601
Agent [candidate] (1.235 s) : 0, 1234974
Total [candidate] (11.417 s) : 0, 11417023
section profiling
Agent [baseline] (1.192 s) : 0, 1192362
Total [baseline] (10.96 s) : 0, 10960423
Agent [candidate] (1.19 s) : 0, 1190043
Total [candidate] (11.145 s) : 0, 11145152
gantt
title petclinic - break down per module: candidate=1.62.0-SNAPSHOT~055530140f, baseline=1.62.0-SNAPSHOT~95de525c87
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.247 ms) : 0, 1247
crashtracking [candidate] (1.229 ms) : 0, 1229
BytebuddyAgent [baseline] (638.738 ms) : 0, 638738
BytebuddyAgent [candidate] (641.04 ms) : 0, 641040
AgentMeter [baseline] (29.815 ms) : 0, 29815
AgentMeter [candidate] (29.884 ms) : 0, 29884
GlobalTracer [baseline] (252.237 ms) : 0, 252237
GlobalTracer [candidate] (251.633 ms) : 0, 251633
AppSec [baseline] (33.053 ms) : 0, 33053
AppSec [candidate] (32.74 ms) : 0, 32740
Debugger [baseline] (61.153 ms) : 0, 61153
Debugger [candidate] (60.549 ms) : 0, 60549
Remote Config [baseline] (614.266 µs) : 0, 614
Remote Config [candidate] (603.892 µs) : 0, 604
Telemetry [baseline] (8.32 ms) : 0, 8320
Telemetry [candidate] (9.082 ms) : 0, 9082
Flare Poller [baseline] (9.194 ms) : 0, 9194
Flare Poller [candidate] (9.077 ms) : 0, 9077
section appsec
crashtracking [baseline] (1.23 ms) : 0, 1230
crashtracking [candidate] (1.231 ms) : 0, 1231
BytebuddyAgent [baseline] (665.121 ms) : 0, 665121
BytebuddyAgent [candidate] (662.833 ms) : 0, 662833
AgentMeter [baseline] (12.091 ms) : 0, 12091
AgentMeter [candidate] (12.103 ms) : 0, 12103
GlobalTracer [baseline] (250.1 ms) : 0, 250100
GlobalTracer [candidate] (249.042 ms) : 0, 249042
AppSec [baseline] (185.14 ms) : 0, 185140
AppSec [candidate] (185.007 ms) : 0, 185007
Debugger [baseline] (66.176 ms) : 0, 66176
Debugger [candidate] (65.847 ms) : 0, 65847
Remote Config [baseline] (598.734 µs) : 0, 599
Remote Config [candidate] (611.432 µs) : 0, 611
Telemetry [baseline] (8.524 ms) : 0, 8524
Telemetry [candidate] (8.397 ms) : 0, 8397
Flare Poller [baseline] (3.582 ms) : 0, 3582
Flare Poller [candidate] (3.514 ms) : 0, 3514
IAST [baseline] (24.707 ms) : 0, 24707
IAST [candidate] (24.608 ms) : 0, 24608
section iast
crashtracking [baseline] (1.235 ms) : 0, 1235
crashtracking [candidate] (1.229 ms) : 0, 1229
BytebuddyAgent [baseline] (806.636 ms) : 0, 806636
BytebuddyAgent [candidate] (808.95 ms) : 0, 808950
AgentMeter [baseline] (11.541 ms) : 0, 11541
AgentMeter [candidate] (11.568 ms) : 0, 11568
GlobalTracer [baseline] (240.705 ms) : 0, 240705
GlobalTracer [candidate] (240.601 ms) : 0, 240601
AppSec [baseline] (31.903 ms) : 0, 31903
AppSec [candidate] (32.226 ms) : 0, 32226
Debugger [baseline] (60.478 ms) : 0, 60478
Debugger [candidate] (64.833 ms) : 0, 64833
Remote Config [baseline] (551.433 µs) : 0, 551
Remote Config [candidate] (555.971 µs) : 0, 556
Telemetry [baseline] (13.601 ms) : 0, 13601
Telemetry [candidate] (9.422 ms) : 0, 9422
Flare Poller [baseline] (3.522 ms) : 0, 3522
Flare Poller [candidate] (3.487 ms) : 0, 3487
IAST [baseline] (26.046 ms) : 0, 26046
IAST [candidate] (25.872 ms) : 0, 25872
section profiling
crashtracking [baseline] (1.196 ms) : 0, 1196
crashtracking [candidate] (1.188 ms) : 0, 1188
BytebuddyAgent [baseline] (696.762 ms) : 0, 696762
BytebuddyAgent [candidate] (694.017 ms) : 0, 694017
AgentMeter [baseline] (9.216 ms) : 0, 9216
AgentMeter [candidate] (8.966 ms) : 0, 8966
GlobalTracer [baseline] (208.595 ms) : 0, 208595
GlobalTracer [candidate] (208.545 ms) : 0, 208545
AppSec [baseline] (33.214 ms) : 0, 33214
AppSec [candidate] (33.183 ms) : 0, 33183
Debugger [baseline] (65.897 ms) : 0, 65897
Debugger [candidate] (65.946 ms) : 0, 65946
Remote Config [baseline] (572.101 µs) : 0, 572
Remote Config [candidate] (577.683 µs) : 0, 578
Telemetry [baseline] (7.81 ms) : 0, 7810
Telemetry [candidate] (7.9 ms) : 0, 7900
Flare Poller [baseline] (3.531 ms) : 0, 3531
Flare Poller [candidate] (3.562 ms) : 0, 3562
ProfilingAgent [baseline] (93.92 ms) : 0, 93920
ProfilingAgent [candidate] (94.892 ms) : 0, 94892
Profiling [baseline] (94.505 ms) : 0, 94505
Profiling [candidate] (95.474 ms) : 0, 95474
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 3 performance regressions! Performance is the same for 17 metrics, 16 unstable metrics.
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.62.0-SNAPSHOT~055530140f, baseline=1.62.0-SNAPSHOT~95de525c87
dateFormat X
axisFormat %s
section baseline
no_agent (1.243 ms) : 1231, 1256
. : milestone, 1243,
iast (3.347 ms) : 3297, 3397
. : milestone, 3347,
iast_FULL (5.95 ms) : 5889, 6010
. : milestone, 5950,
iast_GLOBAL (3.643 ms) : 3582, 3703
. : milestone, 3643,
profiling (2.346 ms) : 2323, 2368
. : milestone, 2346,
tracing (2.082 ms) : 2062, 2102
. : milestone, 2082,
section candidate
no_agent (1.245 ms) : 1233, 1256
. : milestone, 1245,
iast (3.334 ms) : 3287, 3382
. : milestone, 3334,
iast_FULL (6.132 ms) : 6070, 6195
. : milestone, 6132,
iast_GLOBAL (3.579 ms) : 3519, 3640
. : milestone, 3579,
profiling (2.373 ms) : 2352, 2394
. : milestone, 2373,
tracing (1.941 ms) : 1925, 1957
. : milestone, 1941,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.62.0-SNAPSHOT~055530140f, baseline=1.62.0-SNAPSHOT~95de525c87
dateFormat X
axisFormat %s
section baseline
no_agent (18.047 ms) : 17867, 18227
. : milestone, 18047,
appsec (18.786 ms) : 18594, 18978
. : milestone, 18786,
code_origins (18.063 ms) : 17884, 18242
. : milestone, 18063,
iast (18.44 ms) : 18250, 18630
. : milestone, 18440,
profiling (18.648 ms) : 18462, 18834
. : milestone, 18648,
tracing (17.836 ms) : 17662, 18011
. : milestone, 17836,
section candidate
no_agent (19.568 ms) : 19369, 19767
. : milestone, 19568,
appsec (18.654 ms) : 18468, 18841
. : milestone, 18654,
code_origins (17.969 ms) : 17791, 18147
. : milestone, 17969,
iast (17.668 ms) : 17494, 17841
. : milestone, 17668,
profiling (18.364 ms) : 18184, 18544
. : milestone, 18364,
tracing (19.075 ms) : 18882, 19267
. : milestone, 19075,
DacapoParameters
See matching parameters
SummaryFound 1 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 0 unstable metrics.
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.62.0-SNAPSHOT~055530140f, baseline=1.62.0-SNAPSHOT~95de525c87
dateFormat X
axisFormat %s
section baseline
no_agent (1.499 ms) : 1488, 1511
. : milestone, 1499,
appsec (3.85 ms) : 3626, 4075
. : milestone, 3850,
iast (2.297 ms) : 2227, 2367
. : milestone, 2297,
iast_GLOBAL (2.341 ms) : 2270, 2412
. : milestone, 2341,
profiling (2.128 ms) : 2072, 2184
. : milestone, 2128,
tracing (2.095 ms) : 2040, 2149
. : milestone, 2095,
section candidate
no_agent (1.488 ms) : 1477, 1500
. : milestone, 1488,
appsec (2.554 ms) : 2499, 2609
. : milestone, 2554,
iast (2.288 ms) : 2218, 2358
. : milestone, 2288,
iast_GLOBAL (2.336 ms) : 2265, 2407
. : milestone, 2336,
profiling (2.111 ms) : 2055, 2167
. : milestone, 2111,
tracing (2.086 ms) : 2032, 2140
. : milestone, 2086,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.62.0-SNAPSHOT~055530140f, baseline=1.62.0-SNAPSHOT~95de525c87
dateFormat X
axisFormat %s
section baseline
no_agent (15.582 s) : 15582000, 15582000
. : milestone, 15582000,
appsec (14.878 s) : 14878000, 14878000
. : milestone, 14878000,
iast (18.354 s) : 18354000, 18354000
. : milestone, 18354000,
iast_GLOBAL (18.102 s) : 18102000, 18102000
. : milestone, 18102000,
profiling (14.984 s) : 14984000, 14984000
. : milestone, 14984000,
tracing (14.736 s) : 14736000, 14736000
. : milestone, 14736000,
section candidate
no_agent (15.305 s) : 15305000, 15305000
. : milestone, 15305000,
appsec (14.751 s) : 14751000, 14751000
. : milestone, 14751000,
iast (18.32 s) : 18320000, 18320000
. : milestone, 18320000,
iast_GLOBAL (18.247 s) : 18247000, 18247000
. : milestone, 18247000,
profiling (14.856 s) : 14856000, 14856000
. : milestone, 14856000,
tracing (15.03 s) : 15030000, 15030000
. : milestone, 15030000,
|
ca28620 to
972244b
Compare
…n, scope/span ordering
…span hierarchy, add REST and Triple tests
…er spans, rpc.method tag
96c7d44 to
ea98882
Compare
|
For info, I clean up the duplicate "inst:sofa rpc" (without the space after |
b3f3860 to
0555301
Compare
| @@ -0,0 +1 @@ | |||
| apply from: "$rootDir/gradle/java.gradle" | |||
There was a problem hiding this comment.
Note you can delete dd-java-agent/instrumentation/sofarpc/build.gradle and omit the ":dd-java-agent:instrumentation:sofarpc", entry from settings.gradle if this parent directory doesn't contain any source / resources.
mcculls
left a comment
There was a problem hiding this comment.
LGTM, thanks for the comprehensive tests
| class SofaRpcRestTest extends InstrumentationSpecification { | ||
|
|
||
| @Shared | ||
| int port = 12205 | ||
|
|
||
| @Shared | ||
| ProviderBootstrap restProviderBootstrap | ||
|
|
There was a problem hiding this comment.
Just curious if tests can be implemented with JUnit?
Recently we added a lot of functionality and base classes to simplify conversion from Groovy to JUnit.
Assuming this is new code it would benefit to have JUnit tests if possible
What Does This Do
Adds auto-instrumentation for Sofa RPC, a high-perfomance RPC framework. Produces
sofarpc.requestclient and server spans with distributed trace propagation for all four supported transports: Bolt, H2C, REST and Triple(gRPC)Motivation
Sofa RPC is widely used in Ant Financial's ecosystem and it's not covered by any existing dd-trace-java integration. WIthout this instrumentaiton, traces break at the service boundary - downstream PRC calls are unlinked from traces
Additional Notes
Module structure
A single module
sofarpc-5.0covers all protocols and all public version of sofa-rpc. The framework was open-sources at 5.0.0; no earlier versions exists in public repositoriesHow server spans are coordinated
ProviderProxyInvoker.invoke()is a single entry point for all protocols on the server side - but it has noo info about the transport delivered the request. Transport specific instrumentations write the protocol name into ThreadLocal (SofaRpcProtocolContext) beforeinvoke()is called.ProviderProxuInvokerInstrumentaitonreads it to created the span and tag with the protocol. If the ThreadLocal is empty (uninstrumented transport), no span is created.Trace propagation by protocol
Bolt and H2C - context travels inside SofaRequest.requestProps (the framework's own key-value bag)

REST -

SofaRequest.requestPropsis not serialized as HTTP headers by the REST transport. Context instead travels as standard HTTP headers, injected and extracted by HTTP layer (Apache HttpClient on the client side, Netty on the server side). TheRestServerHandlerInstumentationsets the protocol context so thatProviderProxyInvoketInstrumentationactivated and creates asofarpc.request[server] span that naturally attaches to the activenetty.request` spanTriple - uses gRPC unses the hood. The grpc-1.5 instrumentation handles trace propagation via gRPC Metadata. The

TripleServerInstrumentationsetsprotocol="tri"so thatProviderProxyInvokerInstrumentaitoncreates asofarpc.request[server]span with child of the activegrpc.serverspanSpan attributes
Resource name format:
{rpc.service}/{rpc.method}(e.g.com.example.HelloService:1.0/sayHello)Known limitations
SofaRequest.getTaargetServiceUniqueName()returns null on the server side - the service name is not propagated thought the JAX-RS layer.Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]
Note: Once your PR is ready to merge, add it to the merge queue by commenting
/merge./merge -ccancels the queue request./merge -f --reason "reason"skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.