||Doctors conventionally analyzed echocardiographic images for diagnosing congenital heart diseases (CHDs). However, this process is laborious and depends on the experience of the doctors. This study investigated the use of deep learning algorithms for the image detection of the ventricular septal defect (VSD), the most common type. Color Doppler echocardiographic images containing three types of VSDs were tested with color doppler ultrasound medical images. To the best of our knowledge, this study is the first one to solve this object detection problem by using a modified YOLOv4–DenseNet framework. Because some techniques of YOLOv4 are not suitable for echocardiographic object detection, we revised the algorithm for this problem. The results revealed that the YOLOv4–DenseNet outperformed YOLOv4, YOLOv3, YOLOv3–SPP, and YOLOv3–DenseNet in terms of metric mAP-50. The F1-score of YOLOv4-DenseNet and YOLOv3-DenseNet were better than those of others. Hence, the contribution of this study establishes the feasibility of using deep learning for echocardiographic image detection of VSD investigation and a better YOLOv4-DenseNet framework could be employed for the VSD detection.