Hi! This is a big project from the topic Concurrency from my Software Architect and DevSecOps Engineer road map.
For the concurrency project, I'm creating a CLI (Command-line Interface) Concurrent Downloader to download data from such websites concurrently!
Follow instructions at the bottom or check how_to_use.txt in the repo.
Check out my 2nd concurrency project which is basically still CLI Concurrent Downloader but all coded from scratch with lack external help from AI or other code refeences that improves and make more features of this Project 1 as well as fixing edge cases and better error handling in code logic:
Check out my benchmark and analysis comparing downloading using threading, asyncio, and multiprocessing of 10 URLS and 100 URLS in my Road Map Repo:
Below is high level design of my CLI Concurrent Downloader!
📘 HOW TO USE - Concurrent Downloader CLI Tool This project allows you to download multiple files concurrently using threading, asyncio, or multiprocessing in Python.
✅ Requirements
Python 3.10+
Visual Studio Code (VS Code) is recommended
🧠 Step-by-Step Instructions
🔧 Step 0: Clone the Repository
Open VS Code Press Ctrl + ~ to open the terminal
Navigate to a folder where you want to store the project: cd path/to/your/folder
Clone the repo: git clone https://github.com/WillyPhan06/Concurrency-Project-CLI-Concurrent-Downloader
Navigate into the project folder: cd Concurrency-Project-CLI-Concurrent-Downloader
🐍 Step 1: Set Up Virtual Environment
Run the following command to create a virtual environment: python -m venv venv
Activate the virtual environment:
On Windows (CMD): venv\Scripts\activate
On Windows PowerShell (recommended if you are using terminal of VSCode in Window): .\venv\Scripts\Activate.ps1
On Mac/Linux: source venv/bin/activate
Install dependencies (if any are added in future): pip install -r requirements.txt
🌐 Step 2: Edit urls.txt
Open urls.txt in VS Code. Add one URL per line that you want to download content from.
Example: https://example.com/file1.jpg https://example.com/file2.jpg
📥 Step 3: Run the Downloader Make sure you're in the main project folder (Concurrency-Project-CLI-Concurrent-Downloader).
Choose one of the following methods:
⏳ Step 4: Wait for Completion
The terminal will show a "Done" message along with the total time taken.
Progress and status will be printed while downloading.
📁 Step 5: Check Your Downloads
All downloaded files will be saved inside the downloads/ folder.
If you used threading or asyncio, a log.txt file will also be generated inside downloads/, containing detailed logs of the download process.
💡 Tips
Make sure the URLs in urls.txt are valid and accessible.
To restart, just clear downloads/ and re-run a method.