Detection of number plate and digits/alphabets in one deep learning model

I tried to make a caffe-based deep learning model trained with number plate, digits and alphabet altogether.

But in testing, I detect only number plate. I used both py-faster-rcnn and SSD.

Is it because of overlapping between number plate and those digits/alphabets?