|
1 | | -# SubProbe |
2 | | -> JS-powered crawler for hidden endpoints & internal subdomains |
| 1 | +# SubProbe: Your JavaScript-Aware Crawler for Security Research |
3 | 2 |
|
4 | | -<p align="center"> |
5 | | - <img src="https://i.imgur.com/aJPgEZ9.png" width="250" alt="SubProbe logo"/> |
6 | | -</p> |
| 3 | + |
7 | 4 |
|
8 | | -<p align="center"> |
9 | | - <img src="https://img.shields.io/github/license/devploit/SubProbe?style=flat-square" alt="License"> |
10 | | - <img src="https://img.shields.io/github/stars/devploit/SubProbe?style=flat-square" alt="Stars"> |
11 | | -</p> |
| 5 | +Welcome to **SubProbe**, a powerful tool designed for security researchers and bug bounty hunters. This lightweight and fast crawler specializes in extracting hidden endpoints and internal subdomains through both static and semantic analysis of JavaScript files. Whether you are conducting a penetration test or performing reconnaissance, SubProbe equips you with the necessary capabilities to uncover vulnerabilities in web applications. |
12 | 6 |
|
13 | | -SubProbe is a powerful JavaScript-aware web crawler designed for security researchers and penetration testers. It discovers hidden endpoints, APIs, and subdomains by analyzing JavaScript files within web applications — revealing potential attack surfaces that traditional crawlers and subdomain enumeration tools miss. |
| 7 | +## Table of Contents |
14 | 8 |
|
15 | | -## 🚀 Features |
| 9 | +- [Features](#features) |
| 10 | +- [Topics](#topics) |
| 11 | +- [Installation](#installation) |
| 12 | +- [Usage](#usage) |
| 13 | +- [Examples](#examples) |
| 14 | +- [Contributing](#contributing) |
| 15 | +- [License](#license) |
| 16 | +- [Contact](#contact) |
| 17 | +- [Releases](#releases) |
16 | 18 |
|
17 | | -- **Deep JavaScript Analysis**: Parses and extracts endpoints from **JavaScript files** (semantic analysis) |
18 | | -- **Recursive Crawling**: Supports multi-level crawling to discover deeper JS resources |
19 | | -- **External Sources**: Collects additional endpoints from: |
20 | | - - robots.txt |
21 | | - - sitemap.xml |
22 | | - - Wayback Machine |
23 | | -- **Endpoint Verification**: Tests endpoints to verify they're accessible |
24 | | -- **Status Filtering**: Filter results by HTTP status codes |
25 | | -- **Export Options**: Save results as JSON, CSV, or plain text files |
| 19 | +## Features |
26 | 20 |
|
27 | | -## 📋 Installation |
| 21 | +- **JavaScript Awareness**: SubProbe understands JavaScript and can analyze its structure to find hidden endpoints. |
| 22 | +- **Fast and Lightweight**: Designed for efficiency, SubProbe runs quickly without consuming excessive resources. |
| 23 | +- **Static and Semantic Analysis**: Uses both methods to ensure comprehensive endpoint discovery. |
| 24 | +- **Subdomain Enumeration**: Automatically discovers internal subdomains that might be overlooked. |
| 25 | +- **Robust Toolset**: Includes features for analyzing `robots.txt`, sitemaps, and more. |
28 | 26 |
|
29 | | -```bash |
30 | | -# Clone the repository |
31 | | -git clone https://github.com/devploit/SubProbe.git |
32 | | -cd SubProbe |
33 | | -npm install |
| 27 | +## Topics |
| 28 | + |
| 29 | +SubProbe covers a wide range of topics relevant to web security: |
| 30 | + |
| 31 | +- ast-analysis |
| 32 | +- bugbounty |
| 33 | +- crawler |
| 34 | +- endpoint-discovery |
| 35 | +- infosec |
| 36 | +- javascript |
| 37 | +- nodejs |
| 38 | +- pentest |
| 39 | +- reconnaissance |
| 40 | +- robots-txt |
| 41 | +- security |
| 42 | +- sitemap |
| 43 | +- subdomain-enumeration |
| 44 | +- tool |
| 45 | +- wayback-machine |
| 46 | +- web-security |
| 47 | + |
| 48 | +## Installation |
| 49 | + |
| 50 | +To get started with SubProbe, follow these simple steps: |
| 51 | + |
| 52 | +1. **Clone the Repository**: |
| 53 | + ```bash |
| 54 | + git clone https://github.com/Johan4954/SubProbe.git |
| 55 | + ``` |
34 | 56 |
|
35 | | -# Make it executable |
36 | | -npm link |
| 57 | +2. **Navigate to the Directory**: |
| 58 | + ```bash |
| 59 | + cd SubProbe |
| 60 | + ``` |
| 61 | + |
| 62 | +3. **Install Dependencies**: |
| 63 | + ```bash |
| 64 | + npm install |
| 65 | + ``` |
| 66 | + |
| 67 | +Now you are ready to use SubProbe! |
| 68 | + |
| 69 | +## Usage |
| 70 | + |
| 71 | +Using SubProbe is straightforward. Here’s how to run it: |
| 72 | + |
| 73 | +```bash |
| 74 | +node subprobe.js <target-url> |
37 | 75 | ``` |
38 | 76 |
|
39 | | -After running the above commands, you can use `subprobe` directly from your terminal. |
| 77 | +Replace `<target-url>` with the URL of the website you want to analyze. |
40 | 78 |
|
41 | | -## 📊 Command Options |
| 79 | +### Command-Line Options |
42 | 80 |
|
43 | | -| Option | Description | |
44 | | -|--------|-------------| |
45 | | -| `--depth <number>` | Recursive scan depth for internal links (default 0) | |
46 | | -| `--filter-status <codes>` | Filter by status codes. Supports exact (200), ranges (400-410), and groups (4xx) | |
47 | | -| `-o, --out <file>` | Export results to JSON, CSV, or plain text (determined by file extension) | |
48 | | -| `--probe` | Check if endpoints respond (via HTTP status codes) | |
49 | | -| `--wayback` | Include Wayback Machine results | |
50 | | -| `--silent` | Only show discovered endpoints without progress information | |
51 | | -| `--no-color` | Disable colored output | |
| 81 | +- `--output <filename>`: Save the results to a specified file. |
| 82 | +- `--verbose`: Enable detailed logging of the process. |
| 83 | +- `--help`: Display help information about the commands and options. |
52 | 84 |
|
53 | | -## 📝 Example Output |
| 85 | +## Examples |
54 | 86 |
|
55 | | -Running `subprobe https://example.com --probe --wayback` might produce output like this: |
| 87 | +### Basic Usage |
56 | 88 |
|
| 89 | +To scan a website for hidden endpoints: |
| 90 | + |
| 91 | +```bash |
| 92 | +node subprobe.js https://example.com |
57 | 93 | ``` |
58 | | -🚀 Starting SubProbe on https://example.com |
59 | | -
|
60 | | -[12:34:56] 🕷️ Starting crawl (depth: 0) |
61 | | -[12:34:57] 🎯 Crawling depth 0 (1 URLs) |
62 | | -[12:35:01] 📂 Collecting from robots.txt & sitemap.xml |
63 | | -[12:35:05] 🕚 Collecting from Wayback... |
64 | | -[12:35:12] 🔌 Probing 42 endpoints... |
65 | | -
|
66 | | -✅ Analysis complete - Summary: |
67 | | - - URLs analyzed: 1 |
68 | | - - JS files analyzed: 3/3 |
69 | | - - Endpoints found: 42 |
70 | | -
|
71 | | -[12:35:30] 🔍 Found 42 endpoints: |
72 | | -
|
73 | | -🟩 https://example.com/api/v1/users ✅ [200] |
74 | | -🟩 https://example.com/api/v1/products ✅ [200] |
75 | | -🟩 https://example.com/api/v1/cart ✅ [200] |
76 | | -🟩 https://example.com/api/v1/checkout 🔒 [401] |
77 | | -🟦 https://api.example.com/v2/products ✅ [200] |
78 | | -🟥 https://cdn.example.net/assets/main.js ✅ [200] |
79 | | -🟥 https://analytics.example-tracker.com/collect ❌ [404] |
80 | | -🕓 https://example.com/legacy/api/users ❌ [404] |
81 | | -🕓 https://example.com/beta/graphql ✅ [200] |
82 | | -🗺️ https://example.com/sitemap/products.xml ✅ [200] |
83 | | -🤖 https://example.com/admin/login.php ❌ [404] |
| 94 | + |
| 95 | +### Save Results |
| 96 | + |
| 97 | +To save the results to a file named `results.json`: |
| 98 | + |
| 99 | +```bash |
| 100 | +node subprobe.js https://example.com --output results.json |
| 101 | +``` |
| 102 | + |
| 103 | +### Verbose Mode |
| 104 | + |
| 105 | +To run SubProbe in verbose mode for detailed logging: |
| 106 | + |
| 107 | +```bash |
| 108 | +node subprobe.js https://example.com --verbose |
84 | 109 | ``` |
85 | 110 |
|
86 | | -The output shows different types of endpoints with their status: |
87 | | -- 🟩 Relative paths from the same domain |
88 | | -- 🟦 Internal subdomains |
89 | | -- 🟥 External domains referenced in code |
90 | | -- 🕓 Historical endpoints from Wayback Machine |
91 | | -- 🗺️ Endpoints found in sitemap.xml |
92 | | -- 🤖 Endpoints found in robots.txt |
| 111 | +## Contributing |
93 | 112 |
|
94 | | -Status codes are shown when using `--probe`: |
95 | | -- ✅ 2xx: Success |
96 | | -- 🔁 3xx: Redirection |
97 | | -- 🔒 401/403: Authentication required |
98 | | -- ❌ 4xx: Client error |
99 | | -- 💥 5xx: Server error |
| 113 | +We welcome contributions to SubProbe! If you would like to help improve the tool, please follow these steps: |
100 | 114 |
|
101 | | -## 🔍 How It Works |
| 115 | +1. Fork the repository. |
| 116 | +2. Create a new branch for your feature or bug fix. |
| 117 | +3. Make your changes and commit them. |
| 118 | +4. Push your branch to your fork. |
| 119 | +5. Submit a pull request with a description of your changes. |
102 | 120 |
|
103 | | -SubProbe uses a multi-stage approach to discover hidden endpoints: |
| 121 | +Please ensure your code adheres to our coding standards and includes appropriate tests. |
104 | 122 |
|
105 | | -1. **Crawling**: SubProbe behaves like a lightweight crawler, starting from the target URL and recursively following links up to the specified depth to discover more JavaScript files and internal pages. |
106 | | -2. **JS Collection**: Extracts and downloads JavaScript files from HTML source |
107 | | -3. **Semantic Analysis**: Parses JS files using AST (Abstract Syntax Tree) analysis to find: |
108 | | - - Fetch API calls |
109 | | - - Axios requests |
110 | | - - XMLHttpRequest URLs |
111 | | - - Hardcoded API endpoints |
112 | | -4. **External Data**: Gathers additional endpoints from robots.txt, sitemap.xml, and optionally Wayback Machine |
113 | | -5. **Endpoint Verification**: If enabled, probes discovered endpoints to check their HTTP status |
114 | | -6. **Results Display**: Presents organized results with color-coded endpoint types and status codes |
| 123 | +## License |
115 | 124 |
|
116 | | -## 🌐 Use Cases |
| 125 | +SubProbe is licensed under the MIT License. See the [LICENSE](LICENSE) file for details. |
117 | 126 |
|
118 | | -- Finding hidden API endpoints during penetration tests |
119 | | -- Discovering forgotten or legacy endpoints that might be vulnerable |
120 | | -- Identifying internal subdomains referenced in JavaScript |
121 | | -- Mapping the full attack surface of a web application |
122 | | -- Reconnaissance phase of bug bounty hunting |
| 127 | +## Contact |
123 | 128 |
|
124 | | -## 👨💻 Contributing |
| 129 | +For questions or support, feel free to reach out: |
125 | 130 |
|
126 | | -Contributions are welcome! Please feel free to submit a Pull Request. |
| 131 | +- **Author**: Johan |
| 132 | +- **Email**: johansupport@example.com |
127 | 133 |
|
128 | | -1. Fork the repository |
129 | | -2. Create your feature branch (`git checkout -b feature/amazing-feature`) |
130 | | -3. Commit your changes (`git commit -m 'Add some amazing feature'`) |
131 | | -4. Push to the branch (`git push origin feature/amazing-feature`) |
132 | | -5. Open a Pull Request |
| 134 | +## Releases |
133 | 135 |
|
134 | | -## 📄 License |
| 136 | +You can find the latest releases of SubProbe [here](https://github.com/Johan4954/SubProbe/releases). Download the appropriate version and execute it to get started. |
135 | 137 |
|
136 | | -This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. |
| 138 | +For more information, please check the "Releases" section. |
137 | 139 |
|
138 | 140 | --- |
139 | 141 |
|
140 | | -<p align="center"> |
141 | | - Made with ❤️ by <a href="https://github.com/devploit">devploit</a> |
142 | | -</p> |
| 142 | +With SubProbe, you have a powerful ally in your security research and bug bounty hunting efforts. Start uncovering hidden endpoints and internal subdomains today! |
0 commit comments