Reinforcement learning (RL) has been used in combination with cooperative caching to deal with growing traffic in mobile networks, but the performance of RL based caching policies depends heavily on network settings. This paper investigates the impact of access delays within network infrastructures and popularity and similarity properties of the contents requested on network performance. A deep Q-network based caching framework is established in both basic and extended cooperative edge networks. Our simulation results reveal explicit relationships between the performance and influential parameters, which can provide a guidance and benchmark for the design of effective caching polices with RL and cooperation technologies.