Skip to content

[fix](join) Restrict auto salt join rewrite#64518

Open
feiniaofeiafei wants to merge 3 commits into
apache:masterfrom
feiniaofeiafei:fix_salt_join
Open

[fix](join) Restrict auto salt join rewrite#64518
feiniaofeiafei wants to merge 3 commits into
apache:masterfrom
feiniaofeiafei:fix_salt_join

Conversation

@feiniaofeiafei

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Problem Summary:

This PR restricts the auto salt join rewrite in two cases.

First, ASOF joins are excluded from automatic salt join optimization because the current rewrite may introduce large build-side expansion and is not cost guarded for ASOF semantics.

Second, SaltJoin now marks internally generated helper joins with isSaltJoinGenerated in JoinReorderContext. SkewJoin skips these generated joins to avoid applying salt rewrite recursively on SaltJoin's own expansion plan.

Release note

None

Check List (For Author)

  • Test: Manual test
  • Behavior changed: Yes. The optimizer no longer applies automatic salt join rewrite to ASOF joins or SaltJoin-generated helper joins.
  • Does this need documentation: No

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@feiniaofeiafei

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29052 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f96bcb1411017af0736a7ede61535d16d7808f91, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17970	4141	4027	4027
q2	q3	10902	1401	838	838
q4	4687	478	344	344
q5	7537	872	580	580
q6	182	171	135	135
q7	808	828	643	643
q8	9369	1647	1642	1642
q9	5840	4539	4525	4525
q10	6767	1786	1535	1535
q11	430	278	248	248
q12	628	415	290	290
q13	18122	3355	2804	2804
q14	264	249	245	245
q15	q16	821	771	710	710
q17	932	974	1031	974
q18	6906	5914	5720	5720
q19	1314	1358	956	956
q20	492	404	264	264
q21	5970	2642	2273	2273
q22	443	362	299	299
Total cold run time: 100384 ms
Total hot run time: 29052 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4307	4246	4224	4224
q2	q3	4552	4987	4340	4340
q4	2109	2182	1381	1381
q5	4475	4361	4308	4308
q6	227	174	127	127
q7	1739	1659	1987	1659
q8	2640	2215	2160	2160
q9	8052	8294	7949	7949
q10	4792	4774	4327	4327
q11	574	414	394	394
q12	768	783	542	542
q13	3439	3649	2978	2978
q14	302	301	272	272
q15	q16	745	753	645	645
q17	1311	1323	1458	1323
q18	7996	7365	7267	7267
q19	1133	1067	1092	1067
q20	2195	2202	1938	1938
q21	5260	4554	4398	4398
q22	519	446	399	399
Total cold run time: 57135 ms
Total hot run time: 51698 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 168401 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f96bcb1411017af0736a7ede61535d16d7808f91, data reload: false

query5	4326	620	485	485
query6	430	183	177	177
query7	4919	552	292	292
query8	390	210	192	192
query9	8754	4010	4005	4005
query10	439	298	254	254
query11	5943	2413	2182	2182
query12	160	104	96	96
query13	1256	575	437	437
query14	6358	5379	5056	5056
query14_1	4363	4368	4373	4368
query15	206	200	180	180
query16	976	462	393	393
query17	1123	715	585	585
query18	2520	479	360	360
query19	202	182	142	142
query20	113	106	119	106
query21	212	137	119	119
query22	13633	13619	13413	13413
query23	17446	16405	16161	16161
query23_1	16341	16338	16301	16301
query24	7627	1801	1309	1309
query24_1	1305	1283	1288	1283
query25	527	433	363	363
query26	1308	324	161	161
query27	2659	556	332	332
query28	4418	2003	1988	1988
query29	1060	595	467	467
query30	305	236	198	198
query31	1123	1058	940	940
query32	105	57	58	57
query33	497	310	246	246
query34	1151	1123	667	667
query35	752	776	683	683
query36	1417	1420	1258	1258
query37	153	103	91	91
query38	3236	3155	3034	3034
query39	941	959	896	896
query39_1	883	894	883	883
query40	219	120	95	95
query41	64	63	60	60
query42	94	90	97	90
query43	321	325	280	280
query44	
query45	199	187	177	177
query46	1162	1191	762	762
query47	2334	2410	2259	2259
query48	423	415	294	294
query49	620	488	346	346
query50	958	351	269	269
query51	4342	4256	4174	4174
query52	87	86	75	75
query53	240	264	184	184
query54	268	220	202	202
query55	75	72	72	72
query56	235	219	209	209
query57	1455	1400	1341	1341
query58	248	218	205	205
query59	1561	1702	1415	1415
query60	280	253	229	229
query61	154	157	152	152
query62	684	656	588	588
query63	229	187	185	185
query64	2534	768	634	634
query65	
query66	1777	453	337	337
query67	29786	29679	29358	29358
query68	
query69	422	309	259	259
query70	990	951	955	951
query71	297	226	211	211
query72	3189	2632	2330	2330
query73	831	775	462	462
query74	5090	4944	4764	4764
query75	2642	2562	2258	2258
query76	2359	1131	766	766
query77	354	377	287	287
query78	12393	12228	11913	11913
query79	1285	1015	742	742
query80	523	466	402	402
query81	450	282	239	239
query82	240	155	124	124
query83	267	275	249	249
query84	
query85	843	517	424	424
query86	321	302	263	263
query87	3359	3356	3201	3201
query88	3580	2733	2714	2714
query89	403	374	333	333
query90	2188	182	175	175
query91	171	157	127	127
query92	64	60	55	55
query93	1442	1476	884	884
query94	516	369	303	303
query95	676	375	414	375
query96	1059	788	362	362
query97	2711	2689	2578	2578
query98	211	210	205	205
query99	1135	1168	1031	1031
Total cold run time: 250344 ms
Total hot run time: 168401 ms

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 90.91% (10/11) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 1.15% (10/873) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants