Reinforcement learning (RL)-based automatic web GUI testing techniques have gained significant attraction in both academia and industry due to their ability to facilitate intelligent exploration in the websites under testing. However, the testing performance of a single RL agent often reaches its limit in testing large-scale websites. To address this limitation, we propose to utilize Multi-Agent RL (MARL) algorithms for automatic web GUI testing, aiming to improve test efficiency and coverage. The use of multiple agents introduces challenges of state explosion. How to sharing data for effective cooperation among the agents should also be addressed. In this paper, we propose the first \textbf{\underline{MAR}}L-based web \textbf{\underline{G}}UI testing system (MARG) that allows multiple testing agents to perform automatic web GUI testing simultaneously. It adopts a client-server architecture, in which the agents perform web testing and a controller is responsible for coordination. To share testing experience among the agents, two data sharing schemes are implemented on the controller: one is centralized with a shared Q-table to facilitate efficient communication, and the other is distributed with data exchange to decrease overhead. We evaluated the performance of MARG on nine popular real-world websites. When configure the system with five agents, MARG achieves an average increase of 4.34 and 3.89 times in the number of explored states, and the number of detected failures reached 4.03 and 3.76 times, respectively, compared to two state-of-the-art approaches. Compared to independently running the same number of agents, MARG can explore 36.42% more unique web states. These results demonstrate the potential of MARL in enhancing efficiency and performance in large-scale web GUI testing tasks.