Content Discovery: Finding Hidden Directories and Files

Most websites have more files and folders than their links show. Old admin panels, backup files, test pages and forgotten API versions often sit on the server with no link pointing to them. Content discovery is the process of finding these hidden paths by sending many requests and watching how the server responds. In this post I’ll show how I approach content discovery for bug bounty: picking wordlists, running ffuf, feroxbuster and dirsearch, filtering noise, and deciding what is worth a closer look.

Before you start, one short note. Content discovery sends a lot of requests. Run it only on assets that are in scope for a program you are allowed to test, and follow the program’s rate limits. Many programs ask for a specific User-Agent or header, so read the rules first.

What Content Discovery Finds and Why It Matters

When you visit a site, you only see the pages the developers chose to link. The server may still answer for many other paths. Some common finds are:

• Admin or staff panels such as /admin/, /manage/ or /internal/
• Backup and old files like config.php.bak, backup.zip or db.sql
• Version control folders such as /.git/ left on the server
• Old API versions like /api/v1/ when the app now uses /api/v3/
• Debug and status pages, log files and default install pages

A hidden path is not a bug by itself. It becomes a bug when it exposes something it should not, such as data without login, a working admin function, or source code. So the goal is to find paths, then check what each one really does.

Content discovery fits after you have a list of live hosts. If you have not built that list yet, start with my post on subdomain enumeration with passive and active techniques, then probe the results with httpx.

Start With Paths You Already Know

Brute forcing is not the first step. You can collect many real paths for free before sending a single guess.

Archived URLs

Tools like gau and waybackurls pull URLs that were seen in the past. These often include paths that are no longer linked. I covered this in detail in Wayback Machine recon for mining archived URLs.

gau example.com | uro > archived.txt

Crawling and JavaScript files

A crawler like katana follows links and can parse JavaScript to find endpoints. JS files often mention API routes that you will never see in the HTML.

katana -u https://target.example -d 3 -jc -o crawl.txt

Once you have these real paths, take the folder names and file names out of them. They make a small custom wordlist that matches how this target names things. This list often gives better results than a huge generic one.

Choosing the Right Wordlist

Your results depend more on the wordlist than on the tool. A few simple rules help:

1. Match the technology. If httpx shows the site runs PHP, use a list with PHP files. For a Java app, look for lists with .jsp and .do paths. For an API, use an API route list.
2. Start small, then grow. Run a short, high-quality list first. Go bigger only on hosts that look interesting.
3. Add your custom words. Combine target-specific words from archives and JS files with a general list.

SecLists is the most common collection of wordlists and has folders for web content, APIs and many technologies. Assetnote also publishes wordlists built from real-world data. Keep a few lists you trust instead of downloading hundreds you never read.

Content Discovery With ffuf

ffuf is fast and flexible. You put the word FUZZ where you want the wordlist entries to go.

ffuf -u https://target.example/FUZZ -w common.txt -mc 200,204,301,302,307,401,403 -rate 50

Here -mc matches only these status codes, and -rate limits requests per second. Keep the rate low enough to respect the program rules.

Filtering noise

Many sites return the same page for every path, often with status 200. This gives you thousands of false results. ffuf has filters for this:

• -fs filters responses by size in bytes
• -fw filters by number of words
• -fc filters by status code
• -ac turns on auto-calibration, which tries to learn the “not found” response and hide it

ffuf -u https://target.example/FUZZ -w common.txt -ac -rate 50

Looking for files with extensions

Use -e to try each word with extra extensions. This is useful for backup files.

ffuf -u https://target.example/FUZZ -w words.txt -e .bak,.old,.zip,.txt -ac -rate 50 -o ffuf.json -of json

Recursive Scans With feroxbuster and dirsearch

When you find a folder, you usually want to look inside it too. feroxbuster does recursion by default, which saves time.

feroxbuster -u https://target.example -w common.txt -x php,txt -d 2 --rate-limit 50 -o ferox.txt

The -d flag sets how deep it goes. Keep the depth small. Deep recursion on a large site can send a huge number of requests very quickly.

dirsearch is another good option, especially for beginners. It ships with its own wordlist and handles extensions well.

dirsearch -u https://target.example -e php,bak,zip

You do not need all three tools. I like ffuf for control and feroxbuster for quick recursive passes. Pick one, learn its filters well, and use it often.

Checking Your Results by Hand

The tool gives you a list. The real work starts after that. For each interesting path, open it in Burp Suite and look closely.

• 401 and 403 pages: Note them, but do not assume a bypass exists. Check if the same content is reachable on another host or path that is not protected.
• Redirects: Follow them. A redirect to a login page may still leak data in the response body before the redirect.
• Backup or config files: If you find one with secrets, stop. Do not use the credentials. Report what you found with a short proof and let the team rotate them.
• Admin panels: Check if functions work without login. Do not change real data. Show impact in a safe way, such as viewing a settings page.

Write down every interesting result with the host, path, status and size. You will want these notes later, especially when the site changes and you scan again.

Common Mistakes in Content Discovery

• Using one giant wordlist on every host. It is slow, noisy and can get your IP blocked. Use smaller, targeted lists.
• Ignoring the “not found” pattern. If you do not filter the default response, you will waste hours reading fake results.
• Sending too many requests. High thread counts can break program rules and hurt the target. Always set a rate limit.
• Reporting a path with no impact. A 403 on /admin/ or a public robots.txt is not a vulnerability. Show what is exposed and why it matters.
• Scanning only the main domain. Staging, dev and old subdomains are often less protected. Run discovery on every live host that is in scope.
• Forgetting the trailing slash. Some servers treat /admin and /admin/ differently. Try both on interesting paths.

Conclusion

Content discovery is simple to start and hard to do well. Collect real paths from archives and crawling first, then use a wordlist that matches the target’s technology. Filter the noise, keep your rate low, and check each result by hand before you report it. Done this way, it often leads you to pages that other hunters never see.

If you want to automate more of your recon on small targets, take a look at my small scope reconnaissance automation script, so you can spend more of your time on manual testing.