feat: add VinBigData CXR Kaggle pipeline notebooks

VINBIGDATA_ARCHITECTURE.md

@@ -0,0 +1,85 @@
+# VinBigData 结构图
+
+这份结构图描述当前仓库的主流程、各 notebook 职责，以及它们之间的输入输出关系。
+
+## 1. 总体流水线
+
+```mermaid
+flowchart LR
+    A[官方比赛数据<br/>train.csv / test dicom] --> B[图像预处理数据集<br/>256 / 512 / 1024 PNG]
+    B --> C[yolov5-chest-512.ipynb<br/>主检测流程]
+    A --> C
+    B --> D[vinbigdata-2-class-classifier-complete-pipeline.ipynb<br/>normal / abnormal 二分类]
+    A --> D
+
+    C --> E[14 类异常检测结果]
+    D --> F[test_pred.csv / valid_pred.csv<br/>整图 normal 概率]
+
+    E --> G[后处理逻辑<br/>Keep / Add / Replace]
+    F --> G
+    G --> H[submission.csv]
+
+    H --> I[ensemble-of-best-public-notebooks.ipynb<br/>轻量 2-class 再过滤]
+    F --> I
+    I --> J[postprocessed submission.csv]
+
+    H --> K[ensembling-approach.ipynb<br/>多 submission 融合]
+    J --> K
+    K --> L[final submission.csv]
+```
+
+## 2. Notebook 职责图
+
+```mermaid
+flowchart TD
+    A[yolov5-chest-512.ipynb] --> A1[读取比赛数据]
+    A --> A2[MultilabelStratifiedKFold 分 fold]
+    A --> A3[生成 YOLO 标签]
+    A --> A4[调用检测模型推理]
+    A --> A5[结合整图分类做后处理]
+    A --> A6[导出 submission.csv]
+
+    B[vinbigdata-cxr-ad-yolov5-14-class-infer.ipynb] --> B1[加载 best.pt]
+    B --> B2[运行 detect.py]
+    B --> B3[YOLO 坐标转回比赛格式]
+    B --> B4[纯检测 baseline submission]
+
+    C[vinbigdata-2-class-classifier-complete-pipeline.ipynb] --> C1[构造 normal / abnormal 标签]
+    C --> C2[StratifiedKFold 训练]
+    C --> C3[导出 valid_pred.csv]
+    C --> C4[导出 test_pred.csv]
+    C --> C5[按阈值修正检测 submission]
+
+    D[ensemble-of-best-public-notebooks.ipynb] --> D1[读取已有 submission]
+    D --> D2[读取 2-class 概率]
+    D --> D3[Keep / Add / Replace]
+
+    E[ensembling-approach.ipynb] --> E1[读取多个 submission]
+    E --> E2[按图片拆分 PredictionString]
+    E --> E3[top-n 保留]
+    E --> E4[同类框平均融合]
+```
+
+## 3. 关键后处理决策
+
+```mermaid
+flowchart TD
+    A[输入: 一张图的检测结果 + class0 概率] --> B{normal 概率阈值}
+    B -->|p < low_threshold| C[Keep<br/>保留检测结果]
+    B -->|low_threshold <= p < high_threshold| D[Add<br/>保留检测结果并追加 class 14]
+    B -->|p >= high_threshold| E[Replace<br/>替换为 14 1 0 0 1 1]
+```
+
+## 4. 代码层面的对应关系
+
+- 主检测流程：`yolov5-chest-512.ipynb`
+- 纯检测推理：`vinbigdata-cxr-ad-yolov5-14-class-infer.ipynb`
+- 二分类与后处理：`vinbigdata-2-class-classifier-complete-pipeline.ipynb`
+- 轻量再过滤：`ensemble-of-best-public-notebooks.ipynb`
+- 多结果融合：`ensembling-approach.ipynb`
+
+## 5. 适合怎么用
+
+- 想先理解全局：先看“总体流水线”
+- 想找每个 notebook 的职责：看“Notebook 职责图”
+- 想单独复现提分逻辑：看“关键后处理决策”

VINBIGDATA_REVIEW.md

@@ -0,0 +1,442 @@
+# VinBigData 项目说明
+
+## 1. 文档目的
+
+本文档用于说明本仓库中 `VinBigData Chest X-ray Abnormalities Detection` Kaggle 比赛项目的目标、组成、运行链路和复现注意事项。
+
+该仓库本质上是一个 **以 Kaggle Notebook 为中心的比赛归档项目**，重点在于提交策略验证，而不是可直接安装和运行的标准工程包。因此，本文档的定位是：
+
+- 说明项目主线与各 notebook 的职责
+- 记录关键输入、输出和依赖关系
+- 为后续回看、迁移或局部复现提供导航
+
+## 2. 项目概述
+
+### 2.1 比赛任务
+
+比赛输入为胸部 X-ray 影像，输出为 Kaggle 要求的 `PredictionString`，每 6 个值表示一条预测：
+
+`class_id score x_min y_min x_max y_max`
+
+其中：
+
+- `0` 到 `13` 表示 14 类胸片异常
+- `14` 表示 `No finding`
+
+因此，该任务不仅是目标检测任务，还包含明显的图像级正常/异常判别需求。仓库中的方案围绕这一点构建，核心并非单一模型，而是以下组合：
+
+1. 14 类异常检测
+2. normal / abnormal 二分类
+3. `No finding` 的规则化后处理
+4. 多个 submission 的结果级融合
+
+### 2.2 仓库定位
+
+从当前文件内容看，该项目的主要价值不在于“训练脚本工程化”，而在于保留一套完整的比赛思路：
+
+- 检测模型负责输出异常框
+- 二分类模型负责判断整图是否更接近正常
+- 后处理阶段决定保留框、追加 `class 14`，或直接替换为 `No finding`
+- 在此基础上尝试进一步 ensemble
+
+## 3. 仓库文件说明
+
+| 文件 | 作用 | 说明 |
+| --- | --- | --- |
+| `yolov5-chest-512.ipynb` | 主流程 notebook | 包含数据整理、fold 划分、YOLO 标签生成、检测推理、图像级分类辅助后处理 |
+| `vinbigdata-cxr-ad-yolov5-14-class-infer.ipynb` | 纯检测推理 | 使用训练好的 YOLOv5 权重生成检测结果并导出 `submission.csv` |
+| `vinbigdata-2-class-classifier-complete-pipeline.ipynb` | 二分类完整流程 | 训练 normal / abnormal 分类器，并导出 `valid_pred.csv`、`test_pred.csv` |
+| `ensembling-approach.ipynb` | submission 级融合 | 对多个检测 submission 做启发式融合与平均 |
+| `ensemble-of-best-public-notebooks.ipynb` | 轻量后处理 | 对已有 submission 再应用一次 2-class filter |
+| `VINBIGDATA_REVIEW.md` | 项目说明文档 | 当前文档 |
+
+## 4. 方案架构
+
+### 4.1 总体流程
+
+该项目可以概括为以下流水线：
+
+1. 使用检测模型生成 14 类异常框
+2. 使用二分类模型估计图像为 normal 的概率
+3. 根据 normal 概率对检测结果做后处理
+4. 必要时对多个 submission 再进行融合
+
+可简化表示为：
+
+`Detection -> 2-class probability -> Post-process No finding -> Ensemble`
+
+为了便于快速把握整体关系，下面补充三张结构图，分别对应总体流水线、各 notebook 职责，以及后处理决策逻辑。
+
+**总体流水线**
+
+```mermaid
+flowchart LR
+    A[官方比赛数据<br/>train.csv / test dicom] --> B[图像预处理数据集<br/>256 / 512 / 1024 PNG]
+    B --> C[yolov5-chest-512.ipynb<br/>主检测流程]
+    A --> C
+    B --> D[vinbigdata-2-class-classifier-complete-pipeline.ipynb<br/>normal / abnormal 二分类]
+    A --> D
+
+    C --> E[14 类异常检测结果]
+    D --> F[test_pred.csv / valid_pred.csv<br/>整图 normal 概率]
+
+    E --> G[后处理逻辑<br/>Keep / Add / Replace]
+    F --> G
+    G --> H[submission.csv]
+
+    H --> I[ensemble-of-best-public-notebooks.ipynb<br/>轻量 2-class 再过滤]
+    F --> I
+    I --> J[postprocessed submission.csv]
+
+    H --> K[ensembling-approach.ipynb<br/>多 submission 融合]
+    J --> K
+    K --> L[final submission.csv]
+```
+
+**Notebook 职责图**
+
+```mermaid
+flowchart TD
+    A[yolov5-chest-512.ipynb] --> A1[读取比赛数据]
+    A --> A2[MultilabelStratifiedKFold 分 fold]
+    A --> A3[生成 YOLO 标签]
+    A --> A4[调用检测模型推理]
+    A --> A5[结合整图分类做后处理]
+    A --> A6[导出 submission.csv]
+
+    B[vinbigdata-cxr-ad-yolov5-14-class-infer.ipynb] --> B1[加载 best.pt]
+    B --> B2[运行 detect.py]
+    B --> B3[YOLO 坐标转回比赛格式]
+    B --> B4[纯检测 baseline submission]
+
+    C[vinbigdata-2-class-classifier-complete-pipeline.ipynb] --> C1[构造 normal / abnormal 标签]
+    C --> C2[StratifiedKFold 训练]
+    C --> C3[导出 valid_pred.csv]
+    C --> C4[导出 test_pred.csv]
+    C --> C5[按阈值修正检测 submission]
+
+    D[ensemble-of-best-public-notebooks.ipynb] --> D1[读取已有 submission]
+    D --> D2[读取 2-class 概率]
+    D --> D3[Keep / Add / Replace]
+
+    E[ensembling-approach.ipynb] --> E1[读取多个 submission]
+    E --> E2[按图片拆分 PredictionString]
+    E --> E3[top-n 保留]
+    E --> E4[同类框平均融合]
+```
+
+**后处理决策图**
+
+```mermaid
+flowchart TD
+    A[输入: 一张图的检测结果 + class0 概率] --> B{normal 概率阈值}
+    B -->|p < low_threshold| C[Keep<br/>保留检测结果]
+    B -->|low_threshold <= p < high_threshold| D[Add<br/>保留检测结果并追加 class 14]
+    B -->|p >= high_threshold| E[Replace<br/>替换为 14 1 0 0 1 1]
+```
+
+### 4.2 检测主线
+
+`yolov5-chest-512.ipynb` 是最接近“总控台”的 notebook。根据 notebook 中保留的代码，其主线包括：
+
+- 读取官方比赛数据与预处理后的 PNG 数据
+- 将原始标注缩放到训练尺寸
+- 使用 `MultilabelStratifiedKFold` 划分 5-fold
+- 生成 YOLO 训练目录及标签文件
+- 以 512 尺度作为主要检测输入
+- 调用 YOLOv5 `detect.py` 做推理
+- 结合图像级分类结果生成最终 `submission.csv`
+
+需要说明的是，该 notebook 中保留了部分历史实验痕迹，例如 EfficientDet / EfficientNet 相关配置、硬编码阈值和样本过滤列表。它更像比赛过程中的工作 notebook，而不是整理后的单一职责脚本。
+
+### 4.3 二分类补偿分支
+
+`vinbigdata-2-class-classifier-complete-pipeline.ipynb` 负责训练整图二分类器，用于回答“这张图是否正常”。
+
+其主要配置和行为包括：
+
+- 输入图像目录：`vinbigdata-chest-xray-resized-png-256x256`
+- 默认 backbone：`resnet18`
+- 5-fold `StratifiedKFold`
+- 训练轮数：`epoch=15`
+- 调度器：`CosineAnnealingWarmRestarts`
+- 支持 `mixup`、`label_smoothing`、`EMA`
+- 输出：
+  - `valid_pred.csv`
+  - `test_pred.csv`
+
+该分支不负责检测框位置，只输出 normal / abnormal 概率，供后处理阶段使用。
+
+### 4.4 后处理策略
+
+当前仓库中的后处理逻辑高度一致，核心思想为：
+
+1. 若 normal 概率较低，则保留检测结果
+2. 若 normal 概率处于中间区间，则保留检测结果并追加 `class 14`
+3. 若 normal 概率较高，则直接替换为 `14 1 0 0 1 1`
+
+在 `vinbigdata-2-class-classifier-complete-pipeline.ipynb` 中，保存下来的阈值示例为：
+
+- `low_threshold = 0.0`
+- `high_threshold = 0.976`
+
+在 `ensemble-of-best-public-notebooks.ipynb` 中，轻量后处理版本使用：
+
+- `low_threshold = 0.0`
+- `high_threshold = 0.90`
+
+这说明项目后期的主要增分手段之一，是利用二分类模型修正检测器在正常样本上的误报。
+
+### 4.5 多模型融合
+
+`ensembling-approach.ipynb` 不是模型层面的联合训练，而是 **submission 级别** 的结果融合。
+
+它的做法是：
+
+1. 读取多个模型导出的 `PredictionString`
+2. 将每个 submission 拆分成按图像组织的预测字典
+3. 每个模型仅保留 top-n 预测，默认 `n=3`
+4. 找出相同类别的重复预测
+5. 对重复类别的分数和框坐标做平均
+6. 重新拼接为新的 `submission.csv`
+
+该方法更接近启发式 averaging，而不是标准训练期融合。
+
+## 5. 各 Notebook 的正式说明
+
+### 5.1 `yolov5-chest-512.ipynb`
+
+用途：
+
+- 项目主流程梳理
+- 训练前数据准备
+- YOLO 标签生成
+- 检测结果读取与后处理
+- 结合图像级分类器生成最终提交
+
+关键实现：
+
+- `split_df(...)`
+  - 使用 `MultilabelStratifiedKFold` 做检测任务分层划分
+- `Preprocess_wbf(...)`
+  - 对训练标注做 WBF 风格整理
+- `create_file(...)`
+  - 生成 YOLO 所需目录结构和标签文件
+- `Predict_process.fit(...)`
+  - 读取 YOLO 检测结果
+  - 执行 NMS
+  - 调用 `EfficientnetCus` 进行整图分类
+  - 根据分类概率决定最终 `PredictionString`
+
+说明：
+
+- notebook 中提供了 YOLOv5 训练命令示例，但训练命令本身是注释状态
+- 检测推理阶段调用的是外部 `best.pt`
+- 部分阈值和过滤列表属于比赛调参产物，保留了明显经验化特征
+
+### 5.2 `vinbigdata-cxr-ad-yolov5-14-class-infer.ipynb`
+
+用途：
+
+- 以最小流程生成纯检测提交
+
+关键行为：
+
+- 读取已训练好的 `best.pt`
+- 调用 `detect.py`
+- 使用 `img=640`、`conf=0.15`、`iou=0.4`
+- 保存 `txt` 和置信度
+- 将 YOLO 输出转换回比赛要求的原图坐标
+- 对无检测框图像填充 `14 1 0 0 1 1`
+
+适用场景：
+
+- 快速恢复纯检测 baseline
+- 对比二分类后处理前后的差异
+
+### 5.3 `vinbigdata-2-class-classifier-complete-pipeline.ipynb`
+
+用途：
+
+- 训练二分类器
+- 导出验证集和测试集概率
+- 演示如何将 2-class 结果作用到检测 submission
+
+关键行为：
+
+- 数据标签定义为：
+  - 无异常标注 -> normal
+  - 有异常标注 -> abnormal
+- 使用 `StratifiedKFold` 保持 normal / abnormal 比例
+- 导出：
+  - `valid_pred.csv`
+  - `test_pred.csv`
+- 支持最后一步 2-class filter 后处理
+
+适用场景：
+
+- 单独复现 normal / abnormal 分支
+- 为现有检测 submission 追加后处理信号
+
+### 5.4 `ensembling-approach.ipynb`
+
+用途：
+
+- 对多个已有 submission 做结果融合
+
+关键行为：
+
+- 按图像读取多个 `PredictionString`
+- 对每个模型保留高置信度 top-n 结果
+- 对重复类别做概率和框坐标平均
+- 导出新的融合 submission
+
+适用场景：
+
+- 比较不同检测方案或不同公开 notebook 的互补性
+
+### 5.5 `ensemble-of-best-public-notebooks.ipynb`
+
+用途：
+
+- 对已有 submission 快速再施加一层 2-class filter
+
+关键行为：
+
+- 读取检测 submission
+- 读取 2-class 预测 csv
+- 按阈值执行 `Keep / Add / Replace`
+- 导出新的 `submission.csv`
+
+适用场景：
+
+- 不重新训练模型，只做最后一步规则修正
+
+## 6. 输入、输出与依赖
+
+### 6.1 运行环境
+
+该项目基于 Kaggle Notebook 环境编写，当前仓库 **不包含完整可离线执行的依赖清单和脚本封装**。大部分 notebook 默认假设以下前提成立：
+
+- 运行环境为 Kaggle
+- 数据、模型权重和外部代码位于 `../input/...`
+- 图像已提前转换为 PNG 或已存在预处理版本
+
+### 6.2 主要外部输入
+
+从 notebook 中能确认的主要依赖包括：
+
+- 官方比赛数据：
+  - `vinbigdata-chest-xray-abnormalities-detection`
+- 预处理图像数据：
+  - `vinbigdata-chest-xray-resized-png-256x256`
+  - `vinbigdata-chest-xray-resized-png-1024x1024`
+  - `vinbigdata-512-image-dataset` 或同类 512 PNG 数据
+- 测试元数据：
+  - `vinbigdata-testmeta/test_meta.csv`
+- 2-class 预测结果：
+  - `vinbigdata-2class-prediction/2-cls test pred.csv`
+  - 或 `vinbigdata2classpred/test_pred.csv`
+- 外部模型/代码：
+  - `yolov5`
+  - 已训练检测权重 `best.pt`
+  - 二分类模型权重
+
+### 6.3 主要输出
+
+| Notebook | 主要输出 |
+| --- | --- |
+| `yolov5-chest-512.ipynb` | `submission.csv`，YOLO 标签目录 |
+| `vinbigdata-cxr-ad-yolov5-14-class-infer.ipynb` | `submission.csv` |
+| `vinbigdata-2-class-classifier-complete-pipeline.ipynb` | `valid_pred.csv`、`test_pred.csv`、后处理后的 `submission.csv` |
+| `ensembling-approach.ipynb` | 融合后的 `submission.csv` |
+| `ensemble-of-best-public-notebooks.ipynb` | 追加 2-class filter 后的 `submission.csv` |
+
+## 7. 推荐阅读顺序
+
+### 7.1 理解项目主线
+
+建议顺序：
+
+1. `VINBIGDATA_REVIEW.md`
+2. `yolov5-chest-512.ipynb`
+3. `vinbigdata-2-class-classifier-complete-pipeline.ipynb`
+
+目标：
+
+- 先理解最终提交的构成方式
+- 再理解检测和二分类分别提供什么信号
+
+### 7.2 只恢复纯检测 baseline
+
+建议优先阅读：
+
+1. `vinbigdata-cxr-ad-yolov5-14-class-infer.ipynb`
+2. `yolov5-chest-512.ipynb` 中的检测后处理部分
+
+### 7.3 只恢复后处理与提分思路
+
+建议优先阅读：
+
+1. `vinbigdata-2-class-classifier-complete-pipeline.ipynb`
+2. `ensemble-of-best-public-notebooks.ipynb`
+3. `ensembling-approach.ipynb`
+
+## 8. 复现注意事项
+
+### 8.1 本仓库不是完整的本地工程
+
+当前仓库仅保留 notebook 与说明文档，没有：
+
+- 独立的 `requirements.txt`
+- 统一的训练入口脚本
+- 完整的本地路径配置
+- 明确的最终线上提交版本记录
+
+因此，仓库更适合做以下用途：
+
+- 回顾比赛策略
+- 迁移部分 notebook 逻辑
+- 抽取后处理与 ensemble 思路
+
+不适合直接视为“一键复现实验”的标准工程。
+
+### 8.2 路径和参数存在 Kaggle 绑定
+
+多数 notebook 直接写死了 `../input/...` 路径，说明原始运行环境依赖 Kaggle Dataset / Notebook Output。若迁移到本地，需要自行重建：
+
+- 原始数据目录
+- 预处理 PNG 目录
+- 测试元数据
+- 模型权重目录
+- YOLOv5 代码目录
+
+### 8.3 存在比赛期经验化规则
+
+项目中保留了较多比赛调参痕迹，例如：
+
+- `list_remove`
+- `image_remove`
+- 类别特定阈值
+- `No finding` 的分段规则
+
+这些逻辑对理解历史提交有价值，但不应直接视为可泛化的医学影像检测最佳实践。
+
+### 8.4 最终最佳提交版本无法仅凭仓库唯一确认
+
+从当前仓库可以清楚恢复整体方案，但无法仅凭现有文件 100% 唯一确认最终线上提交是哪一个 notebook 产物或哪个 ensemble 版本。原因包括：
+
+- 缺少 leaderboard 版本记录
+- 缺少 commit 级说明
+- 仓库中同时保留了多条尝试路线
+
+## 9. 结论
+
+本项目的核心不是单一检测模型，而是一套围绕 VinBigData 比赛构建的提交工程：
+
+1. 用检测模型识别 14 类异常
+2. 用二分类模型判断 normal / abnormal
+3. 用后处理规则显式处理 `No finding`
+4. 视情况再对多个 submission 进行融合
+
+如果只保留一个最值得回看的文件，应优先查看 `yolov5-chest-512.ipynb`；如果要理解本项目为什么能进一步提分，应重点查看 `vinbigdata-2-class-classifier-complete-pipeline.ipynb` 中的 2-class filter 流程。

ensemble-of-best-public-notebooks.ipynb

@@ -0,0 +1,303 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "papermill": {
+     "duration": 0.005268,
+     "end_time": "2021-03-21T01:43:26.398126",
+     "exception": false,
+     "start_time": "2021-03-21T01:43:26.392858",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "### Credit of this notebook goes entirely to below public notebook, kindly upvote and appreciate the original author\n",
+    "\n",
+    "* https://www.kaggle.com/muhammad4hmed/lets-overfit-together"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
+    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
+    "execution": {
+     "iopub.execute_input": "2021-03-21T01:43:26.411073Z",
+     "iopub.status.busy": "2021-03-21T01:43:26.410014Z",
+     "iopub.status.idle": "2021-03-21T01:43:26.414755Z",
+     "shell.execute_reply": "2021-03-21T01:43:26.415196Z"
+    },
+    "papermill": {
+     "duration": 0.012965,
+     "end_time": "2021-03-21T01:43:26.415505",
+     "exception": false,
+     "start_time": "2021-03-21T01:43:26.402540",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "\n",
+    "import numpy as np # linear algebra\n",
+    "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
+    "\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2021-03-21T01:43:26.427333Z",
+     "iopub.status.busy": "2021-03-21T01:43:26.426414Z",
+     "iopub.status.idle": "2021-03-21T01:43:26.475008Z",
+     "shell.execute_reply": "2021-03-21T01:43:26.475504Z"
+    },
+    "papermill": {
+     "duration": 0.055753,
+     "end_time": "2021-03-21T01:43:26.475682",
+     "exception": false,
+     "start_time": "2021-03-21T01:43:26.419929",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>image_id</th>\n",
+       "      <th>target</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>002a34c58c5b758217ed1f584ccbcfe9</td>\n",
+       "      <td>0.013326</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>004f33259ee4aef671c2b95d54e4be68</td>\n",
+       "      <td>0.037235</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>008bdde2af2462e86fd373a445d0f4cd</td>\n",
+       "      <td>0.939700</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>009bc039326338823ca3aa84381f17f1</td>\n",
+       "      <td>0.123799</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>00a2145de1886cb9eb88869c85d74080</td>\n",
+       "      <td>0.654006</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>...</th>\n",
+       "      <td>...</td>\n",
+       "      <td>...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2995</th>\n",
+       "      <td>ff91fb82429a27521bbec8569b041f02</td>\n",
+       "      <td>0.936325</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2996</th>\n",
+       "      <td>ff9fcc4087ed5e941209aa3fa948e364</td>\n",
+       "      <td>0.963583</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2997</th>\n",
+       "      <td>ffaa288c8abca300974f043b57d81521</td>\n",
+       "      <td>0.178720</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2998</th>\n",
+       "      <td>ffc441e0c8b7153844047483a577e7c3</td>\n",
+       "      <td>0.225196</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2999</th>\n",
+       "      <td>ffccf1709d0081d122a1d1f9edbefdf1</td>\n",
+       "      <td>0.987406</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "<p>3000 rows × 2 columns</p>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "                              image_id    target\n",
+       "0     002a34c58c5b758217ed1f584ccbcfe9  0.013326\n",
+       "1     004f33259ee4aef671c2b95d54e4be68  0.037235\n",
+       "2     008bdde2af2462e86fd373a445d0f4cd  0.939700\n",
+       "3     009bc039326338823ca3aa84381f17f1  0.123799\n",
+       "4     00a2145de1886cb9eb88869c85d74080  0.654006\n",
+       "...                                ...       ...\n",
+       "2995  ff91fb82429a27521bbec8569b041f02  0.936325\n",
+       "2996  ff9fcc4087ed5e941209aa3fa948e364  0.963583\n",
+       "2997  ffaa288c8abca300974f043b57d81521  0.178720\n",
+       "2998  ffc441e0c8b7153844047483a577e7c3  0.225196\n",
+       "2999  ffccf1709d0081d122a1d1f9edbefdf1  0.987406\n",
+       "\n",
+       "[3000 rows x 2 columns]"
+      ]
+     },
+     "execution_count": 2,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "pred_2class = pd.read_csv(\"../input/vinbigdata-2class-prediction/2-cls test pred.csv\")\n",
+    "low_threshold = 0.0\n",
+    "high_threshold = 0.90\n",
+    "pred_2class"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "papermill": {
+     "duration": 0.004459,
+     "end_time": "2021-03-21T01:43:26.485337",
+     "exception": false,
+     "start_time": "2021-03-21T01:43:26.480878",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "## Apply 2class filter"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2021-03-21T01:43:26.504745Z",
+     "iopub.status.busy": "2021-03-21T01:43:26.503972Z",
+     "iopub.status.idle": "2021-03-21T01:43:27.617091Z",
+     "shell.execute_reply": "2021-03-21T01:43:27.616495Z"
+    },
+    "papermill": {
+     "duration": 1.127304,
+     "end_time": "2021-03-21T01:43:27.617254",
+     "exception": false,
+     "start_time": "2021-03-21T01:43:26.489950",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "n_normal: 1586 -> 1974 with threshold 0.0 & 0.9\n",
+      "Keep 0 Add 1026 Replace 1974\n",
+      "Saved to submission.csv\n"
+     ]
+    }
+   ],
+   "source": [
+    "NORMAL = \"14 1 0 0 1 1\"\n",
+    "\n",
+    "pred_det_df = pd.read_csv(\"../input/vinbigdatastack/submission_postprocessed.csv\")\n",
+    "n_normal_before = len(pred_det_df.query(\"PredictionString == @NORMAL\"))\n",
+    "merged_df = pd.merge(pred_det_df, pred_2class, on=\"image_id\", how=\"left\")\n",
+    "\n",
+    "\n",
+    "if \"target\" in merged_df.columns:\n",
+    "    merged_df[\"class0\"] = 1 - merged_df[\"target\"]\n",
+    "\n",
+    "c0, c1, c2 = 0, 0, 0\n",
+    "for i in range(len(merged_df)):\n",
+    "    p0 = merged_df.loc[i, \"class0\"]\n",
+    "    if p0 < low_threshold:\n",
+    "\n",
+    "        c0 += 1\n",
+    "    elif low_threshold <= p0 and p0 < high_threshold:\n",
+    "\n",
+    "        merged_df.loc[i, \"PredictionString\"] += f\" 14 {p0} 0 0 1 1\"\n",
+    "        c1 += 1\n",
+    "    else:\n",
+    "\n",
+    "        merged_df.loc[i, \"PredictionString\"] = NORMAL\n",
+    "        c2 += 1\n",
+    "\n",
+    "n_normal_after = len(merged_df.query(\"PredictionString == @NORMAL\"))\n",
+    "print(\n",
+    "    f\"n_normal: {n_normal_before} -> {n_normal_after} with threshold {low_threshold} & {high_threshold}\"\n",
+    ")\n",
+    "print(f\"Keep {c0} Add {c1} Replace {c2}\")\n",
+    "submission_filepath = str(\"submission.csv\")\n",
+    "submission_df = merged_df[[\"image_id\", \"PredictionString\"]]\n",
+    "submission_df.to_csv(submission_filepath, index=False)\n",
+    "print(f\"Saved to {submission_filepath}\")\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.9"
+  },
+  "papermill": {
+   "default_parameters": {},
+   "duration": 7.332473,
+   "end_time": "2021-03-21T01:43:28.131726",
+   "environment_variables": {},
+   "exception": null,
+   "input_path": "__notebook__.ipynb",
+   "output_path": "__notebook__.ipynb",
+   "parameters": {},
+   "start_time": "2021-03-21T01:43:20.799253",
+   "version": "2.2.2"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

ensembling-approach.ipynb

@@ -0,0 +1,697 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "papermill": {
+     "duration": 0.009537,
+     "end_time": "2021-03-19T01:19:31.724839",
+     "exception": false,
+     "start_time": "2021-03-19T01:19:31.715302",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "<div align='center'><font size=\"5\" color='#353B47'>Chest-X-ray</font></div>\n",
+    "<div align='center'><font size=\"4\" color=\"#353B47\">How to deal with models averaging ?</font></div>\n",
+    "<br>\n",
+    "<hr>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "papermill": {
+     "duration": 0.007744,
+     "end_time": "2021-03-19T01:19:31.740829",
+     "exception": false,
+     "start_time": "2021-03-19T01:19:31.733085",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "The objective of this notebook is to aproach different methods for Ensembling\n",
+    "\n",
+    "* OR method (Affirmative): A box is considered if it’s generated by at least one of the models.\n",
+    "* AND method (Unanimous): A box is considered if all of the models generate the same box (the box is considered the same if IOU > 0.5).\n",
+    "* Consensus method: A box is considered if the majority of the models generate the same box (ie) if there are m models and (m/2 +1) models generate the same box, that box is considered as valid.\n",
+    "* Weighted Fusion: This is a novel method which was created to replace NMS and it’s shortcomings."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "papermill": {
+     "duration": 0.007818,
+     "end_time": "2021-03-19T01:19:31.756653",
+     "exception": false,
+     "start_time": "2021-03-19T01:19:31.748835",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "# <div id=\"summary\">Summary</div>\n",
+    "\n",
+    "**<font size=\"2\"><a href=\"#chap1\">1. Load libraries and dataframes with predictions</a></font>**\n",
+    "**<br><font size=\"2\"><a href=\"#chap2\">2. Helper functions</a></font>**\n",
+    "**<br><font size=\"2\"><a href=\"#chap3\">3. Run ensembling with appropriate strategy</a></font>**\n",
+    "**<br><font size=\"2\"><a href=\"#chap4\">4. Save results</a></font>**"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "papermill": {
+     "duration": 0.00764,
+     "end_time": "2021-03-19T01:19:31.772375",
+     "exception": false,
+     "start_time": "2021-03-19T01:19:31.764735",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "# <div id=\"chap1\">1. Load libraries and dataframes with predictions"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
+    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
+    "execution": {
+     "iopub.execute_input": "2021-03-19T01:19:31.792647Z",
+     "iopub.status.busy": "2021-03-19T01:19:31.791596Z",
+     "iopub.status.idle": "2021-03-19T01:19:31.797664Z",
+     "shell.execute_reply": "2021-03-19T01:19:31.796907Z"
+    },
+    "papermill": {
+     "duration": 0.017416,
+     "end_time": "2021-03-19T01:19:31.797873",
+     "exception": false,
+     "start_time": "2021-03-19T01:19:31.780457",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import warnings\n",
+    "warnings.filterwarnings('ignore')\n",
+    "\n",
+    "import numpy as np\n",
+    "import pandas as pd\n",
+    "import os\n",
+    "from tqdm import tqdm"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2021-03-19T01:19:31.820376Z",
+     "iopub.status.busy": "2021-03-19T01:19:31.819460Z",
+     "iopub.status.idle": "2021-03-19T01:19:31.859156Z",
+     "shell.execute_reply": "2021-03-19T01:19:31.859859Z"
+    },
+    "papermill": {
+     "duration": 0.053733,
+     "end_time": "2021-03-19T01:19:31.860047",
+     "exception": false,
+     "start_time": "2021-03-19T01:19:31.806314",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "ename": "FileNotFoundError",
+     "evalue": "[Errno 2] No such file or directory: '../input/ensample/submission0.175.csv'",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+      "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
+      "\u001b[0;32m<ipython-input-2-7dcdf4de33b8>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;31m# detectron = pd.read_csv('../input/vinbigdatastack/detectron2.csv')\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;31m# fasterrcnn = pd.read_csv('../input/vinbigdatastack/fasterrcnn.csv')\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0myolo\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_csv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'../input/ensample/submission0.175.csv'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m \u001b[0mdetectron\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_csv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'../input/ensample/submission_0.2.csv'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0mfasterrcnn\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_csv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'../input/ensample/submission_2class filter.csv'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36mread_csv\u001b[0;34m(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, dialect, error_bad_lines, warn_bad_lines, delim_whitespace, low_memory, memory_map, float_precision, storage_options)\u001b[0m\n\u001b[1;32m    603\u001b[0m     \u001b[0mkwds\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkwds_defaults\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    604\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 605\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0m_read\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilepath_or_buffer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    606\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    607\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
+      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36m_read\u001b[0;34m(filepath_or_buffer, kwds)\u001b[0m\n\u001b[1;32m    455\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    456\u001b[0m     \u001b[0;31m# Create the parser.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 457\u001b[0;31m     \u001b[0mparser\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mTextFileReader\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilepath_or_buffer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    458\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    459\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mchunksize\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0miterator\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, f, engine, **kwds)\u001b[0m\n\u001b[1;32m    812\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"has_index_names\"\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mkwds\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"has_index_names\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    813\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 814\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_engine\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_make_engine\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mengine\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    815\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    816\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36m_make_engine\u001b[0;34m(self, engine)\u001b[0m\n\u001b[1;32m   1043\u001b[0m             )\n\u001b[1;32m   1044\u001b[0m         \u001b[0;31m# error: Too many arguments for \"ParserBase\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1045\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mmapping\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mengine\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# type: ignore[call-arg]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1046\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1047\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_failover_to_python\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, src, **kwds)\u001b[0m\n\u001b[1;32m   1860\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1861\u001b[0m         \u001b[0;31m# open handles\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1862\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_open_handles\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msrc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1863\u001b[0m         \u001b[0;32massert\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhandles\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1864\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mkey\u001b[0m \u001b[0;32min\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;34m\"storage_options\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"encoding\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"memory_map\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"compression\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36m_open_handles\u001b[0;34m(self, src, kwds)\u001b[0m\n\u001b[1;32m   1361\u001b[0m             \u001b[0mcompression\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"compression\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1362\u001b[0m             \u001b[0mmemory_map\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"memory_map\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1363\u001b[0;31m             \u001b[0mstorage_options\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"storage_options\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1364\u001b[0m         )\n\u001b[1;32m   1365\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
+      "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/pandas/io/common.py\u001b[0m in \u001b[0;36mget_handle\u001b[0;34m(path_or_buf, mode, encoding, compression, memory_map, is_text, errors, storage_options)\u001b[0m\n\u001b[1;32m    642\u001b[0m                 \u001b[0mencoding\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mioargs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mencoding\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    643\u001b[0m                 \u001b[0merrors\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0merrors\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 644\u001b[0;31m                 \u001b[0mnewline\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    645\u001b[0m             )\n\u001b[1;32m    646\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+      "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: '../input/ensample/submission0.175.csv'"
+     ]
+    }
+   ],
+   "source": [
+    "# Import outputs of each selected models\n",
+    "# yolo = pd.read_csv('../input/vinbigdatastack/yolov5.csv')\n",
+    "# detectron = pd.read_csv('../input/vinbigdatastack/detectron2.csv')\n",
+    "# fasterrcnn = pd.read_csv('../input/vinbigdatastack/fasterrcnn.csv')\n",
+    "yolo = pd.read_csv('../input/ensample/submission0.175.csv')\n",
+    "detectron = pd.read_csv('../input/ensample/submission_0.2.csv')\n",
+    "fasterrcnn = pd.read_csv('../input/ensample/submission_2class filter.csv')\n",
+    "image_ids = yolo.image_id.values"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "papermill": {
+     "duration": 0.008483,
+     "end_time": "2021-03-19T01:19:31.877751",
+     "exception": false,
+     "start_time": "2021-03-19T01:19:31.869268",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "# <div id=\"chap2\">2. Helper functions"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2021-03-19T01:19:31.899085Z",
+     "iopub.status.busy": "2021-03-19T01:19:31.898351Z",
+     "iopub.status.idle": "2021-03-19T01:19:31.915873Z",
+     "shell.execute_reply": "2021-03-19T01:19:31.916363Z"
+    },
+    "papermill": {
+     "duration": 0.029965,
+     "end_time": "2021-03-19T01:19:31.916530",
+     "exception": false,
+     "start_time": "2021-03-19T01:19:31.886565",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "def getitem(dataframe, img_id):\n",
+    "    \n",
+    "    \"\"\"\n",
+    "    Parameters\n",
+    "    ----------\n",
+    "    dataframe : pd.DataFrame\n",
+    "    img_id : str\n",
+    "        \n",
+    "    Returns\n",
+    "    -------\n",
+    "    Dictionary of radiographic observations\n",
+    "    \"\"\"\n",
+    "\n",
+    "    pred = list(dataframe.loc[dataframe.image_id == img_id, \"PredictionString\"])[0].split(' ')\n",
+    "    nb_elm = len(pred)//6\n",
+    "    output = {}\n",
+    "    \n",
+    "    for elm in range(nb_elm):\n",
+    "        output[f'elm_{elm}'] = pred[elm*6 : (elm+1)*6]\n",
+    "        \n",
+    "    return output\n",
+    "\n",
+    "\n",
+    "def sortDictByProba(dict_):\n",
+    "    \n",
+    "    \"\"\"\n",
+    "    Parameters\n",
+    "    ----------\n",
+    "    dict_ : dict, Dictionary of radiographic observations\n",
+    "        \n",
+    "    Returns\n",
+    "    -------\n",
+    "    Dictionary of radiographic observations sorted by probabilities \n",
+    "    \"\"\"\n",
+    "    \n",
+    "    for key in dict_.keys():\n",
+    "        dict_[key] = list(map(lambda x: float(x), dict_[key]))\n",
+    "    \n",
+    "    # item[1][1] corresponds to the second element of the value (the confidence of the class identified)\n",
+    "    return {k: v for k, v in sorted(dict_.items(), key=lambda item: item[1][1], reverse = True)}\n",
+    "\n",
+    "\n",
+    "def getHighestProba(*list_of_dicts, n=3):\n",
+    "    \n",
+    "    \"\"\"\n",
+    "    Parameters\n",
+    "    ----------\n",
+    "    list_of_dicts : list[dict], List of dictionaries containing radiographic observations\n",
+    "    n : int, keep n highest elements of each list_of_dicts at most\n",
+    "    \n",
+    "    Returns\n",
+    "    -------\n",
+    "    Dict of merged top3 confidence interval in each dict of list_of_dicts\n",
+    "    \"\"\"\n",
+    "    \n",
+    "    output = {}\n",
+    "    for index, dict_ in enumerate(list_of_dicts):\n",
+    "        dict_length = len(dict_)\n",
+    "        for i in range(dict_length):\n",
+    "            if i < n:\n",
+    "                output[f\"elm_{i}_dict_{index}\"] =list(dict_.values())[i]\n",
+    "                \n",
+    "    return output\n",
+    "\n",
+    "\n",
+    "def getUnique(dict_):\n",
+    "    \n",
+    "    \"\"\"\n",
+    "    Parameters\n",
+    "    ----------\n",
+    "    dict_ : dict, Dictionary of radiographic observations\n",
+    "        \n",
+    "    Returns\n",
+    "    -------\n",
+    "    List of unique class_id, list of duplicates class_id\n",
+    "    \"\"\"\n",
+    "    \n",
+    "    dict_length = len(dict_)\n",
+    "    \n",
+    "    classes_non_unique = [list(dict_.values())[index][0] for index in range(dict_length)]\n",
+    "    classes_unique = list(set(classes_non_unique))\n",
+    "    \n",
+    "    uniques, counts = np.unique(classes_non_unique, return_counts=True)\n",
+    "    duplicates = uniques[counts > 1]\n",
+    "    singles = np.setdiff1d(classes_unique, duplicates)\n",
+    "    \n",
+    "    return singles, duplicates\n",
+    "\n",
+    "\n",
+    "def getKeysByValue(dictOfElements, valueToFind):\n",
+    "    \n",
+    "    \"\"\"\n",
+    "    Parameters\n",
+    "    ----------\n",
+    "    dictOfElements : dict, Dictionary of radiographic observations\n",
+    "    valueToFind : int, corresponds to class_id\n",
+    "    \n",
+    "    Returns\n",
+    "    -------\n",
+    "    List of keys of dictOfElements that contain valueToFind\n",
+    "    \"\"\"\n",
+    "    \n",
+    "    output = list()\n",
+    "    listOfItems = dictOfElements.items()\n",
+    "    \n",
+    "    for item  in listOfItems:\n",
+    "        if item[1][0] == valueToFind:\n",
+    "            output.append(item[0])\n",
+    "            \n",
+    "    return  output\n",
+    "\n",
+    "\n",
+    "def getListKeysByValue(dictOfElements, valuesToFind):\n",
+    "    \n",
+    "    \"\"\"\n",
+    "    Parameters\n",
+    "    ----------\n",
+    "    dictOfElements : dict, Dictionary of radiographic observations\n",
+    "    valuesToFind : list[int], list of class_id\n",
+    "    \n",
+    "    Returns\n",
+    "    -------\n",
+    "    List of lists of keys of dictOfElements for each value in valuesToFind\n",
+    "    \"\"\"\n",
+    "    \n",
+    "    output = []\n",
+    "    \n",
+    "    for value in valuesToFind:\n",
+    "        output.append(getKeysByValue(dictOfElements, value))\n",
+    "        \n",
+    "    return output\n",
+    "\n",
+    "\n",
+    "def averaging(from_dict, single_keys, dupl_keys):\n",
+    "    \n",
+    "    \"\"\"\n",
+    "    Parameters\n",
+    "    ----------\n",
+    "    from_dict : dict, dictionary to be filtered\n",
+    "    single_keys : list[str], list of keys that should be infered\n",
+    "    dupl_keys : list[str], list of class_id\n",
+    "    \n",
+    "    Returns\n",
+    "    -------\n",
+    "    A filtered dictionary with averaged probs and boxes\n",
+    "    \"\"\"\n",
+    "    \n",
+    "    output = {}\n",
+    "    \n",
+    "    # Infer single keys\n",
+    "    if len(np.ravel(single_keys)) != 0:\n",
+    "        for single in np.ravel(single_keys):\n",
+    "            output[single] = from_dict[single]\n",
+    "\n",
+    "    # For each duplicates, get index of all occurences and average boxing\n",
+    "    if len(np.ravel(dupl_keys)) != 0:\n",
+    "        for index, list_of_duplicate_class in enumerate(dupl_keys):\n",
+    "            probs = [] \n",
+    "            boxing1 = []\n",
+    "            boxing2 = []\n",
+    "            boxing3 = []\n",
+    "            boxing4 = []\n",
+    "            \n",
+    "            for elm in list_of_duplicate_class:\n",
+    "                probs.append(from_dict[elm][1])\n",
+    "                boxing1.append(from_dict[elm][2])\n",
+    "                boxing2.append(from_dict[elm][3])\n",
+    "                boxing3.append(from_dict[elm][4])\n",
+    "                boxing4.append(from_dict[elm][5])\n",
+    "            \n",
+    "            output[f\"elm_{index}\"] = [from_dict[list_of_duplicate_class[0]][0],\n",
+    "                                      np.mean(probs),\n",
+    "                                      np.mean(boxing1),\n",
+    "                                      np.mean(boxing2),\n",
+    "                                      np.mean(boxing3),\n",
+    "                                      np.mean(boxing4)]\n",
+    "            \n",
+    "    return output\n",
+    "\n",
+    "\n",
+    "def toString(pred_list):\n",
+    "    \n",
+    "    \"\"\"\n",
+    "    Parameters\n",
+    "    ----------\n",
+    "    list_final : list[int], list of all radiographic observations\n",
+    "    \n",
+    "    Returns\n",
+    "    -------\n",
+    "    A string which fits with the expected output\n",
+    "    \"\"\"\n",
+    "    \n",
+    "    castedList = []\n",
+    "    for index, elm in enumerate(pred_list):\n",
+    "        if index%6 == 0:\n",
+    "            castedList.append(str(int(elm)))\n",
+    "        else:\n",
+    "            castedList.append(str(elm))\n",
+    "            \n",
+    "    output = \" \".join(castedList)\n",
+    "    \n",
+    "    return output"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "papermill": {
+     "duration": 0.008647,
+     "end_time": "2021-03-19T01:19:31.934073",
+     "exception": false,
+     "start_time": "2021-03-19T01:19:31.925426",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "--------\n",
+    "\n",
+    "**<font size=\"2\"><a href=\"#summary\">Back to summary</a></font>**"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "papermill": {
+     "duration": 0.008675,
+     "end_time": "2021-03-19T01:19:31.951695",
+     "exception": false,
+     "start_time": "2021-03-19T01:19:31.943020",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "# <div id=\"chap3\">3. Run ensembling with appropriate strategy"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "papermill": {
+     "duration": 0.008744,
+     "end_time": "2021-03-19T01:19:31.969329",
+     "exception": false,
+     "start_time": "2021-03-19T01:19:31.960585",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "My strategy here consists in averaging observations that have at least one dupplicate among all models. Some filtering about boxing areas should be added. This will come in a future release"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2021-03-19T01:19:31.992666Z",
+     "iopub.status.busy": "2021-03-19T01:19:31.992043Z",
+     "iopub.status.idle": "2021-03-19T01:19:31.999240Z",
+     "shell.execute_reply": "2021-03-19T01:19:31.999739Z"
+    },
+    "papermill": {
+     "duration": 0.021755,
+     "end_time": "2021-03-19T01:19:31.999915",
+     "exception": false,
+     "start_time": "2021-03-19T01:19:31.978160",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "def main():\n",
+    "    \n",
+    "    output = pd.DataFrame(columns = [\"image_id\", \"PredictionString\"])\n",
+    "    \n",
+    "    for image_id in tqdm(image_ids):\n",
+    "        \n",
+    "        # For each model, get PredictionString of image_id as a dict\n",
+    "        fasterrcnn_pred = getitem(fasterrcnn, image_id)\n",
+    "        detectron_pred = getitem(detectron, image_id)\n",
+    "        yolo_pred = getitem(yolo, image_id)  \n",
+    "        \n",
+    "        # Sort dicts by proba\n",
+    "        sorted_fasterrcnn = sortDictByProba(fasterrcnn_pred)\n",
+    "        sorted_detectron = sortDictByProba(detectron_pred)\n",
+    "        sorted_yolo = sortDictByProba(yolo_pred)\n",
+    "\n",
+    "        # Filter dicts into one dict with at most top n probs\n",
+    "        highest_probs = getHighestProba(sorted_fasterrcnn, \n",
+    "                                        sorted_detectron, \n",
+    "                                        sorted_yolo,\n",
+    "                                        n = 3)\n",
+    "        \n",
+    "        # Get keys of unique and duplicates values in the filtered dict\n",
+    "        singles, duplicates = getUnique(highest_probs)\n",
+    "        single_keys = getListKeysByValue(highest_probs, singles)\n",
+    "        dupl_keys = getListKeysByValue(highest_probs, duplicates)\n",
+    "        \n",
+    "        # Apply averaging strategy\n",
+    "        stacked_dict = averaging(highest_probs, single_keys, dupl_keys)\n",
+    "        \n",
+    "        # Put string in right format\n",
+    "        prediction_int = np.ravel(list(stacked_dict.values()))\n",
+    "        prediction_string = toString(prediction_int)\n",
+    "        \n",
+    "        output = output.append({\"image_id\": image_id, \n",
+    "                                \"PredictionString\": prediction_string},\n",
+    "                               ignore_index=True)\n",
+    "        \n",
+    "    return output"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "papermill": {
+     "duration": 0.008891,
+     "end_time": "2021-03-19T01:19:32.018555",
+     "exception": false,
+     "start_time": "2021-03-19T01:19:32.009664",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "Some other strategies will be tested in a future release:\n",
+    "* OR method \n",
+    "* AND method\n",
+    "* Consensus method\n",
+    "* Weighted Fusion"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "papermill": {
+     "duration": 0.009073,
+     "end_time": "2021-03-19T01:19:32.036791",
+     "exception": false,
+     "start_time": "2021-03-19T01:19:32.027718",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "In the meantime, if you found this notebook usefull and you do have some suggestions on how this could be better implemented, do not hesitate to contribute, i'd really appreciate !"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "papermill": {
+     "duration": 0.008758,
+     "end_time": "2021-03-19T01:19:32.054697",
+     "exception": false,
+     "start_time": "2021-03-19T01:19:32.045939",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "--------\n",
+    "\n",
+    "**<font size=\"2\"><a href=\"#summary\">Back to summary</a></font>**"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "papermill": {
+     "duration": 0.00879,
+     "end_time": "2021-03-19T01:19:32.072694",
+     "exception": false,
+     "start_time": "2021-03-19T01:19:32.063904",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "# <div id=\"chap4\">4. Save results"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2021-03-19T01:19:32.094093Z",
+     "iopub.status.busy": "2021-03-19T01:19:32.093456Z",
+     "iopub.status.idle": "2021-03-19T01:19:32.119598Z",
+     "shell.execute_reply": "2021-03-19T01:19:32.119006Z"
+    },
+    "papermill": {
+     "duration": 0.038032,
+     "end_time": "2021-03-19T01:19:32.119771",
+     "exception": false,
+     "start_time": "2021-03-19T01:19:32.081739",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "ename": "NameError",
+     "evalue": "name 'image_ids' is not defined",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
+      "\u001b[0;32m<ipython-input-5-27cf38391b15>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mfinal_sub\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mfinal_sub\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto_csv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"submission.csv\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mindex\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+      "\u001b[0;32m<ipython-input-4-dec6ed9bcdff>\u001b[0m in \u001b[0;36mmain\u001b[0;34m()\u001b[0m\n\u001b[1;32m      3\u001b[0m     \u001b[0moutput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDataFrame\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcolumns\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m\"image_id\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"PredictionString\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m     \u001b[0;32mfor\u001b[0m \u001b[0mimage_id\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mtqdm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mimage_ids\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m         \u001b[0;31m# For each model, get PredictionString of image_id as a dict\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+      "\u001b[0;31mNameError\u001b[0m: name 'image_ids' is not defined"
+     ]
+    }
+   ],
+   "source": [
+    "final_sub = main()\n",
+    "final_sub.to_csv(\"submission.csv\", index=False)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "papermill": {
+     "duration": 0.009203,
+     "end_time": "2021-03-19T01:19:32.138645",
+     "exception": false,
+     "start_time": "2021-03-19T01:19:32.129442",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "# References\n",
+    "\n",
+    "* <a href = \"https://medium.com/inspiredbrilliance/object-detection-through-ensemble-of-models-fed015bc1ee0\">Article on object detection through ensemble of models</a>\n",
+    "* detectron2 : https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection/code?competitionId=24800&sortBy=scoreDescending\n",
+    "* fasterrcnn : https://www.kaggle.com/awsaf49/vinbigdata-cxr-ad-yolov5-14-class-infer\n",
+    "* yolov5 : https://www.kaggle.com/basu369victor/chest-x-ray-abnormalities-detection-submission"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "papermill": {
+     "duration": 0.009058,
+     "end_time": "2021-03-19T01:19:32.157045",
+     "exception": false,
+     "start_time": "2021-03-19T01:19:32.147987",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "<hr>\n",
+    "<div align='justify'><font color=\"#353B47\" size=\"4\">Thank you for taking the time to read this notebook. I hope that I was able to answer your questions or your curiosity and that it was quite understandable. <u>any constructive comments are welcome</u>. They help me progress and motivate me to share better quality content. I am above all a passionate person who tries to advance my knowledge but also that of others. If you liked it, feel free to <u>upvote and share my work.</u> </font></div>\n",
+    "<br>\n",
+    "<div align='center'><font color=\"#353B47\" size=\"3\">Thank you and may passion guide you.</font></div>"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.9"
+  },
+  "papermill": {
+   "default_parameters": {},
+   "duration": 6.790273,
+   "end_time": "2021-03-19T01:19:32.779060",
+   "environment_variables": {},
+   "exception": null,
+   "input_path": "__notebook__.ipynb",
+   "output_path": "__notebook__.ipynb",
+   "parameters": {},
+   "start_time": "2021-03-19T01:19:25.988787",
+   "version": "2.2.2"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}