Commit 3e97fc9
committed
Improve localTaint predicate performance in STLContainer.
Previous implementation was using nomagic to prevent bad joins from
being introduced in localTaint. However, this forces the evaluator to
construct the entire graph of local taint paths, which is about 25
million tuples on pandas.
This blocks updating to new dataflow where the performance issues get
worse. The evaluator creates `additionalTaintStep+`, a relation with
6.7 billion rows. This is done via a higher-order-predicate and is
surprisingly fast (8sec). However, the next step takes the `+`-style
transitive closure and tries to make the `*` version, by unioning the
`+` version with a scan of all dataflow nodes n into tuples (n, n)`. I
don't know why this union OOMs, to be honest, but the crash stack
trace includes an "unsorted relation writer," which maybe indicates
that the OOM comes from trying to sort the 6.7 billion values. In any
case, I'm satisfied that we shouldn't be constructing a 6.7 billion
relation here.
Fixed by identifying the root candidate set of expressions that come
from container function calls that we're interested in, which is 799
nodes on pandas, and then the transitive relation `containerTaint` is
just 1765 rows on old dataflow. On new dataflow this is 701 sources
and 5779 nodes in the transitive closure.1 parent aaac22d commit 3e97fc9
1 file changed
Lines changed: 26 additions & 16 deletions
Lines changed: 26 additions & 16 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
312 | 312 | | |
313 | 313 | | |
314 | 314 | | |
315 | | - | |
316 | | - | |
317 | | - | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
318 | 339 | | |
319 | 340 | | |
320 | 341 | | |
| |||
326 | 347 | | |
327 | 348 | | |
328 | 349 | | |
329 | | - | |
330 | | - | |
331 | | - | |
332 | | - | |
333 | | - | |
334 | | - | |
335 | | - | |
336 | | - | |
337 | | - | |
338 | | - | |
339 | | - | |
340 | | - | |
341 | | - | |
| 350 | + | |
| 351 | + | |
342 | 352 | | |
343 | 353 | | |
344 | 354 | | |
| |||
0 commit comments