Add configuration mechanism for custom list that works via Splunk Lookup

[dfederschmidt]

Currently, supplying a custom list requires editing bin/suffix_list_custom.dat. As an administrator, I'd like to supply my custom list via a Splunk Lookup that does not require me to create a fork of the app.

[pirxthepilot]

Hi @dfederschmidt , I might get some free cycles to work on a PR, but I am not sure where do start. Do you have some docs you can point me out to? Thanks!

[dfederschmidt]

Hi @pirxthepilot - Thanks for considering to contribute!

In general, Splunk lookup table files can simply be CSV files in the lookups folder of an App.
What we could do is we add a new type of TLD list. Currently, we support the values mozilla, iana and custom.

Specifically, we could add a new mechanism inside this function:

utbox/utbox/bin/ut_parse_lib.py

Lines 102 to 110 in c156de5

	def loadTLDFile(tldlist="iana"):
	if tldlist == "mozilla":
	return _loadMozillaList()
	elif tldlist == "custom":
	return _loadIANAList(
	"suffix_list_custom.dat") #dirty hack, but works ;)
	elif tldlist == "*":
	# load all lists

What I'm thinking is that we could add a pattern such as lookup:my_custom_list.csv which would load a list located inside lookups/my_custom_list.csv. Probably we would need another function eg. def _loadLookupList() to load the file, similar to the other methods in https://github.com/splunk/utbox/blob/main/utbox/bin/ut_parse_lib.py

A limitation of this approach would be is that the lookup would have to be located inside of the app directory, so it's not possible to pass a lookup from another app context.

On how to develop on this app specifically, there is not really a whole lot of documentation right now - there is a brief mention on https://github.com/splunk/utbox#development-setup to get started. https://dev.splunk.com/ has the whole wealth of knowledge around how Splunk Apps work, but I think in this case, we don't need to dive too deep into that.