(NOTE: The Nebulous API was retired on 4/15/2020. The command line tool will remain available on GitHub. Thank you to everyone who made use of NebulousAD and sent us valuable feedback!)
Storing passwords is a problem. And while we’re working on eliminating the need for passwords to be shared or stored anywhere, we know this problem isn’t going away overnight.
In the meantime, one of the most effective ways we can break the cycle of hackers using stolen passwords to breach accounts and companies—thereby arming themselves with even more stolen passwords—is to prevent people from using passwords that are already in the hands of attackers.
NIST recommended exactly that in their long-awaited update to their digital identity guidelines. Released in 2017, the 800-63B publication recommends doing away with all complexity rules, and opting instead for just a minimum length requirement (eight characters), and the use of password blacklists.
"When processing requests to establish and change memorized secrets, verifiers SHALL compare the prospective secrets against a list that contains values known to be commonly-used, expected, or compromised. For example, the list MAY include, but is not limited to:
- Passwords obtained from previous breach corpuses.
- Dictionary words.
- Repetitive or sequential characters (e.g. ‘aaaaaa’, ‘1234abcd’).
- Context-specific words, such as the name of the service, the username, and derivatives thereof."
Last week we released a beta version of NebulousAD at the BSides Las Vegas security conference. This command line tool is meant to help system administrators implement breached password blacklisting with the most common enterprise user management system: Microsoft’s Active Directory.
NebulousAD makes it easy to securely check AD users’ passwords against a database of over 2.5 billion unique passwords that have been found in data breaches over the past 10 years.
You can watch Robert’s talk at BSides about the problem of breached credentials and the NebulousAD tool here.
At a high level, NebulousAD has three functions:
- Extract user passwords from an AD domain in their native NTLM hash format and output them into a csv or json file (for this step we use the popular Impacket tool).
- Wrap the NTLM hashes in a more secure SHA-2 hash.
- Submit the SHA-2(NTLM(password)) hashes to our API, which queries our database of breached passwords and returns a YES/NO status for each hash depending on if a match was found.
The result is a list of the users in your domain that are using passwords that have shown up in other data breaches, allowing you to inform those users and/or force a password reset.
We require users to register an account and API key before hitting our API. This is done primarily to enforce rate limiting and to keep an eye on any potentially abusive behavior. While we do collect aggregate performance metrics, we do not store or log the hashed passwords sent to the API.
You can read more about the tool and take a look at the code on GitHub. The current v1.0.0 is a pre-release, but we will be working to further polish up the tool to make it more accessible and expand functionality. We would love to hear feedback and requests from early adopters, so please do go take a look and let us know what you think!
The Nebulous data
NebulousAD was built to take advantage of a breached credential database we have been working on for over a year.
We started collecting breached credentials from public and dark web sources with the intention of conducting research on password usage, and integrating a password blacklisting functionality into our authentication product, NuLogin. Leaning on the offensive security experience of our team, we found that we were able to put together a collection of breached credentials that was larger than any other single source we have encountered.
The database, which we refer to as Nebulous, currently contains over 8 billion records from more than 7,500 individual breaches going back to 2009. There are almost 4 billion individual plaintext passwords in Nebulous, of which about 2.5 billion are unique. There are also around 3 billion hashed passwords that we have not yet attempted to crack.
We continually search for new data as it becomes available and add it to Nebulous. Our approach is to focus on the latest breaches that include new and unique passwords. These credentials are the most valuable to attackers since they are more likely to still be in use, which makes them the most important ones to be blocking.
Why we built this
Other commercial and free tools exist to help filter user credentials, with the most well-known example being Troy Hunt’s Pwned Passwords. These are great options, and we highly encourage people to take advantage of whatever credential blacklisting methods are available to them.
We decided to build our own partly because we found that our Nebulous database far exceeded the record count of other available options. For example, we have about five times more searchable unique passwords than the current Pwned Passwords version. As our CTO explained in a recent Forbes article, we think breached data needs to be more widely shared and used by the security community—a “fight fire with fire” approach. So, naturally we wanted to make our massive Nebulous database useful to people defending enterprise networks.
We also felt that there was room for improvement in credential blacklisting tools built specifically for use with Active Directory. AD system administrators have a different set of threats, priorities, and restrictions compared to developers of consumer-facing applications. Plus, even a single set of compromised employee credentials can have an outsized impact if they lead to an entire network being breached.
NebulousAD is easily integrated with SIEM and syslog tools, and it can be run with Windows Task Scheduler for a “set it and forget it” flow. This would allow a system admin to automate periodic checks with NebulousAD to make sure that employee credentials are not compromised on an on-going basis. And of course since the tool is open source, users can create their own extensions and scripts that expand the usefulness for their company.
As mentioned above, the current version of NebulousAD is a pre-release that we made available as part of Robert’s presentation at BSides. It is fully functioning and has been tested by our team for bugs, but it is also in need of further usability testing and feedback, as well as a bit of polish. We encourage people to go take a look, grab a key, and let us know how your experience is—we appreciate any comments or questions! (firstname.lastname@example.org)
Some items on the immediate release plan include comprehensive API and user wiki documentation, the ability to redact specific users or groups from what is sent to the API, and a facelift for our API key registration portal.
Another important update that will be coming is the addition of a more private method for checking credentials with the NebulousAD API. Even though we do not log the contents of requests, we understand that some people may not want to send hashes over the network to us. We plan to adopt the k-anonymity protocol that is used in Troy Hunt’s project so that we never see full hashes. We’ll get into this further with that update, but for now you can also check out a great post from Junade Ali at Cloudflare, who proposed the k-anon system for Pwned Passwords.
Finally, we decided to release this as a free open source tool to help advance the practice of password blacklisting and make our Nebulous database useful to the security community. While we want to preserve that goal, we are leaving open the possibility that a paid version or functionality could be introduced in the future if there is demand for further growth of the tool. This would most likely take the form of a usage cap for the free version, or premium functionality like automated remediation. In any scenario, we are dedicated to making this tool available to as many people as possible, including those who would be unable to pay for it.