Data Leak - Core Algorithm

Watcher.data_leak.core.check_keywords(keywords)

Check keywords in Searx Instance & Pastebin.

Parameters:

keywords – Keywords stored in database.

Watcher.data_leak.core.check_pastebin(keywords)

Check Pastebin for keyword list.

Parameters:

keywords – Keywords stored in database.

Returns:

Matched urls & Corresponding keyword.

Return type:

dictionary

Watcher.data_leak.core.check_searx(keyword)

Pull Searx instance for keyword.

Parameters:

keyword – Keyword stored in database.

Returns:

Matched urls.

Return type:

list

Watcher.data_leak.core.check_urls(keyword, urls)

Check if the URL is new.

Parameters:

• keyword – Keyword stored in database.

• urls – Fresh searx urls.

Returns:

Urls not already in alert database column.

Return type:

list

Watcher.data_leak.core.cleanup()

Remove 2 hours old, useless, pasteIDs.

Watcher.data_leak.core.main_data_leak()

Main function:

• close_old_connections()

• read in our list of keywords

• check_keywords(keywords)

Watcher.data_leak.core.send_data_leak_notifications(alert)

Sends notifications to Slack, Citadel, TheHive or Email based on Data Leak.

Parameters:

alert – Alert Object.

Watcher.data_leak.core.send_data_leak_notifications_group(keyword, alerts_number, alerts)

Sends grouped notifications to Slack, Citadel, TheHive or Email based on data_leak_group. If the application is TheHive, individual notifications are sent for each alert.

Parameters:

• keyword – The keyword or term associated with the data leak.

• alerts_number – The total number of alerts in the group.

• alerts – The list of individual alerts to be processed and sent to TheHive.

Watcher.data_leak.core.start_scheduler()

Launch multiple planning tasks in background:

• Fire main every 5 minutes from Monday to Sunday

• Fire cleanup every 2 hours