The main aim of the article is intended to design an effective Plate-Fin Heat Exchanger (PFHX) with composite materials such as SS316 + copper and SS304 + Flyash in a counterflow type. These are brazed together with the method of salt bath brazing and vacuum brazing. The study presents the analytical formulation of heat transfer and fluid flow in PFHX design to predict the enhancement of heat transfer and overall heat transfer coefficient, finite difference method is suggested for analyzing the hot and cold fluids. Moreover, Quantum particle swarm optimization with the radial basis function is proposed for accurate prediction of heat transfer enhancement. The findings of the research demonstrate that the design structure of PFHX with the composite materials is analyzed using a microscopic approach and eroded test. The proposed study is performed using various types of coolants namely MFC, ECSTAR, and TFC anti-freeze coolants with water, and the thermophysical properties of the coolants are also analyzed. The findings demonstrate that the variation between the experimental and theoretical results is less than 3.26%, this indicates that the proposed method is effective for heat exchanger design and the optimization algorithm is more feasible than the analytical results. Hence, the outcome of this study offers a better prediction analysis of heat transfer enhancement using PFHX.