Training accuracy randomly dropping while training U-Net based convolutional neural network for image segmentation

I am having a weird problem while training an image segmentation neural network based on the U-Net architecture. The training accuracy suddenly drops to almost zero and the validation accuracy drops to 0.5. Here's the training graph.

I am using the Adam optimizer with initial learning rate 0.0001 and Dice based loss function. The mini-batch size I am using is 8. The net takes an input image with size 256x256 and outputs binary segmentation mask.

What is really weird to me is the fact, that the training seems to go well and then the accuracy drops so suddenly after 1 iteration.

I am using Python 3.6.5 and Keras with Tensorflow-GPU.

I can work around this problem by running the training multiple times and sometimes it sticks, but I would really like to know what can cause this behavior.

Thank you.