Merge pull request #38 from DataScientest-Studio/develop

Develop

Author: fdayde

.gitignore

@@ -5,7 +5,6 @@ __pycache__/
 
 # C extensions
 *.so
-
 # Distribution / packaging
 .Python
 build/
@@ -165,5 +164,5 @@ cython_debug/
 data/raw/*
 data/processed/*
 
-# models folder
-models/*
+# weights for del model
+src/streamlit/models/model_densenet_masked.weights.h5

README.md

@@ -1,8 +1,39 @@
-Project Name
+Data Science Project: COVID Lung X-Rays Classification
 ==============================
 
-This repo is a Starting Pack for DS projects. You can rearrange the structure to make it fits your project.
+View the streamlit app on [Huggingface](https://huggingface.co/spaces/fdayde/streamlit-dl-radio) 🤗
 
+------------
+This project was made during the Data Scientist course of [Datascientest](https://datascientest.com/), and uses the COVID-QU-Ex dataset available on Kaggle: https://www.kaggle.com/datasets/anasmohammedtahir/covidqu
+
+
+[1] A. M. Tahir, M. E. H. Chowdhury, A. Khandakar, Y. Qiblawey, U. Khurshid, S. Kiranyaz, N. Ibtehaz, M. S. Rahman, S. Al-Madeed, S. Mahmud, M. Ezeddin, K. Hameed, and T. Hamid, “COVID-19 Infection Localization and Severity Grading from Chest X-ray Images”, Computers in Biology and Medicine, vol. 139, p. 105002, 2021, https://doi.org/10.1016/j.compbiomed.2021.105002.  
+[2] Anas M. Tahir, Muhammad E. H. Chowdhury, Yazan Qiblawey, Amith Khandakar, Tawsifur Rahman, Serkan Kiranyaz, Uzair Khurshid, Nabil Ibtehaz, Sakib Mahmud, and Maymouna Ezeddin, “COVID-QU-Ex .” Kaggle, 2021, https://doi.org/10.34740/kaggle/dsv/3122958.  
+[3] T. Rahman, A. Khandakar, Y. Qiblawey A. Tahir S. Kiranyaz, S. Abul Kashem, M. Islam, S. Al Maadeed, S. Zughaier, M. Khan, M. Chowdhury, "Exploring the Effect of Image Enhancement Techniques on COVID-19 Detection using Chest X-rays Images," Computers in Biology and Medicine, p. 104319, 2021, https://doi.org/10.1016/j.compbiomed.2021.104319.  
+[4] A. Degerli, M. Ahishali, M. Yamac, S. Kiranyaz, M. E. H. Chowdhury, K. Hameed, T. Hamid, R. Mazhar, and M. Gabbouj, "Covid-19 infection map generation and detection from chest X-ray images," Health Inf Sci Syst 9, 15 (2021), https://doi.org/10.1007/s13755-021-00146-8.  
+[5] M. E. H. Chowdhury, T. Rahman, A. Khandakar, R. Mazhar, M. A. Kadir, Z. B. Mahbub, K. R. Islam, M. S. Khan, A. Iqbal, N. A. Emadi, M. B. I. Reaz, M. T. Islam, "Can AI Help in Screening Viral and COVID-19 Pneumonia?," IEEE Access, vol. 8, pp. 132665-132676, 2020, https://doi.org/10.1109/ACCESS.2020.3010287.
+
+------------
+Team: 
+- Thomas Baret [linkedin](https://linkedin.com/in/thomas-baret-080050107) [github](https://github.com/tom-b974)
+- Nicolas Bouzinbi [linkedin](https://linkedin.com/in/nicolas-bouzinbi-7916481b4) [github](https://github.com/NicolasBouzinbi)
+- Florent Daydé [linkedin](https://linkedin.com/in/florent-daydé-16431469) [github](https://github.com/fdayde)
+- Nicolas Fenassile [linkedin](https://linkedin.com/in/nicolasfenassile) [github](https://github.com/NicoFena)
+
+supervised by: Gaël Penessot
+
+------------
+How to deploy the streamlit app on Huggingface: 
+
+- Create a new space on Huggingface and clone the repository
+- Push the content of the `src/streamlit` directory
+- Add the model's weights file to the `models` folder
+- Store the model weights in Git LFS by adding the following line to the `.gitattributes` file:  
+```*.h5 filter=lfs diff=lfs merge=lfs -text```
+- Push to Huggingface
+- Do not modify or delete the `REAMDE.md` file created by Huggingface during the initialization on the space.
+
+------------
 Project Organization
 ------------
 
@@ -12,16 +43,17 @@ Project Organization
     │   ├── processed      <- The final, canonical data sets for modeling.
     │   └── raw            <- The original, immutable data dump.
     │
-    ├── models             <- Trained and serialized models, model predictions, or model summaries
+    ├── demo               <- Samples from the dataset for demonstration in streamlit
+    │
+    ├── models             <- Trained and serialized models, model predictions, or model summaries, not on Github for size reasons
     │
     ├── notebooks          <- Jupyter notebooks. Naming convention is a number (for ordering),
     │                         the creator's name, and a short `-` delimited description, e.g.
     │                         `1.0-alban-data-exploration`.
     │
-    ├── references         <- Data dictionaries, manuals, links, and all other explanatory materials.
     │
-    ├── reports            <- The reports that you'll make during this project as PDF
-    │   └── figures        <- Generated graphics and figures to be used in reporting
+    ├── reports            <- The final report made during this project (PDF)
+    │
     │
     ├── requirements.txt   <- The requirements file for reproducing the analysis environment, e.g.
     │                         generated with `pip freeze > requirements.txt`
@@ -36,6 +68,8 @@ Project Organization
     │   │   │                 predictions
     │   │   ├── predict_model.py
     │   │   └── train_model.py
+    │   │   
+    │   │── streamlit      <- Scripts for the Streamlit app
     │   │
     │   ├── visualization  <- Scripts to create exploratory and results oriented visualizations
     │   │   └── visualize.py

demo/Normal (14).png

@@ -245,6 +245,89 @@
     "normalization_HistEgal(normal_path)"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def apply_clahe_normalization(img):\n",
+    "    \"\"\"\n",
+    "    Apply CLAHE normalization to an input image.\n",
+    "\n",
+    "    Args:\n",
+    "        img (numpy.ndarray): Input image, either in grayscale or RGB format.\n",
+    "\n",
+    "    Returns:\n",
+    "        numpy.ndarray: CLAHE normalized image.\n",
+    "\n",
+    "    Raises:\n",
+    "        ValueError: If the input image is None or not a valid image.\n",
+    "    \"\"\"\n",
+    "    if img is None:\n",
+    "        raise ValueError(\"No image data received!\")\n",
+    "\n",
+    "    if not isinstance(img, np.ndarray):\n",
+    "        raise ValueError(\"Input must be a numpy.ndarray\")\n",
+    "\n",
+    "    # Convert image to grayscale if it's not already\n",
+    "    if len(img.shape) == 3 and img.shape[2] == 3:  # RGB image\n",
+    "        img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n",
+    "    elif len(img.shape) != 2:  # Not a grayscale or RGB image\n",
+    "        raise ValueError(\"Input image must be either grayscale or RGB\")\n",
+    "\n",
+    "    # Ensure image is of type uint8 (required for CLAHE)\n",
+    "    if img.dtype != np.uint8:\n",
+    "        img = img.astype('uint8')\n",
+    "\n",
+    "    # Apply CLAHE transformation\n",
+    "    clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8))\n",
+    "    img = clahe.apply(img)\n",
+    "\n",
+    "    return img\n",
+    "\n",
+    "def normalization_CLAHE(path):\n",
+    "    '''Cette fonction crée un nouveau dossier nommé \"CLAHE images\", et y stocke toutes les images après CLAHE'''\n",
+    "\n",
+    "    CLAHE_images = \"CLAHE images\"\n",
+    "    CLAHE_images_path = os.path.join(path, CLAHE_images)\n",
+    "\n",
+    "    if not os.path.exists(CLAHE_images_path):\n",
+    "        os.mkdir(CLAHE_images_path) # Création du nouveau dossier pour stocker la normalisation si il n'existe pas déjà\n",
+    "    else :\n",
+    "        print(\"Le dossier existe déjà, rien n'a été fait\")\n",
+    "        return\n",
+    "\n",
+    "    images_path = os.path.join(path, \"images\") # Chemin vers le dossier contenant les images\n",
+    "\n",
+    "    for image in os.listdir(images_path):\n",
+    "        image_path = os.path.join(images_path, image) # Chemin de l'image\n",
+    "        image_read = cv2.imread(image_path, cv2.IMREAD_GRAYSCALE)\n",
+    "        normalized_image = apply_clahe_normalization(image_read) # On crée l'image normalisée\n",
+    "        normalized_image_path = os.path.join(CLAHE_images_path, image) # On récupère le chemin de l'image\n",
+    "\n",
+    "        cv2.imwrite(normalized_image_path, normalized_image) # On met l'image dans le nouveau dossier"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "normalization_CLAHE(covid_path)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "normalization_CLAHE(noncovid_path)\n",
+    "normalization_CLAHE(normal_path)"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 43,

demo/Normal (24).png

@@ -3629,7 +3629,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.3"
+   "version": "3.12.2"
   }
  },
  "nbformat": 4,