Recently, I conducted experiments on Google Cloud using the A100-40GB GPU. Initially, I faced numerous program crashes, some of which occurred without any error messages, making it challenging to identify the root cause. Occasionally, there were specific error messages such as:

1. No space left
2. Bus Error
3. Generation of numerous core.python files.

Possible Solutions

I explored various potential causes, including issues with my image, PyTorch version, CUDA, and DeepSpeed version. However, none of these turned out to be the actual problem.

After extensive searching online, I discovered that the issue was related to insufficient space on /dev/shm. It turned out that my /dev/shm had only 64MB of space.

To address this problem, there are two solutions:

If you are using Docker, you can set the shm size using the following command:

docker run -it --shm-size 40G

If you are not using Docker, you need to mount a larger size to /dev/shm.

Note: Sometimes setting num_workers = 0 for the dataloader might work, but it can result in slower performance.

Related Links

1. https://discuss.huggingface.co/t/importing-datacollator-gives-me-a-bus-error/25489
2. https://github.com/facebookresearch/mmf/issues/877
3. https://github.com/pytorch/pytorch/issues/5247
4. https://blog.csdn.net/ybdesire/article/details/134604587