Real-time guidance of robotic swarms in dynamic environments containing static and mobile obstacles is a complex challenge in autonomous robotic planning. This research presents an advanced reinforcement learning framework that compares and enhances the performance of two algorithms, Q-learning and Double Q-learning, for cooperative guidance of aerial robot swarms. The study employs a simulation involving 10 aerial robots, 12 intelligently dispersed static obstacles, and 2 randomly moving mobile obstacles. Two motion structures were utilized: a 4-directional system (cardinal directions) and a 6-directional system (with added diagonal movements). Innovative reward mechanisms, including a progress reward (based on reduced Euclidean distance to the target) and a group success reward, were implemented. Results demonstrated that Double Q-learning with the 6-directional structure achieved the highest performance, with a positional accuracy of 78.2% and an average reward of 156.18. The presented Monte Carlo-based framework provides a reliable method for evaluating reinforcement learning algorithms in complex scenarios. This framework can be applied in practical applications such as autonomous delivery and search and rescue operations.