This talk is about the benefits of building custom OSINT APIs and key lessons learned through visualizing scope. There will be a short tutorial on how to use the powerful VivaGraph library to literally "connect the dots" while mining large OSINT data sources. By visualizing our targets, it often possible to bypass certain whois privacy protections, find useful runaway SPF records, gain insights on key vendors used by the target organization, and find previously overlooked attack paths. The talk then shifts to a real-world example of how we built a MongoDB database that allows us to quickly search every forward DNS record known to Rapid7's Project Sonar (hundreds of gigs of text data) and expose it through an API for tooling. We will cover key hurdles we hit while working with such a huge dataset, techniques we used to mine big data on a shoestring budget, and key takeaways for pen testers. There will be plenty of demos with "Scope Creep":
https://github.com/fkasler/scope_creep
