The person believes they have committed some sort of crime. A sense of guilt seems to permeate their conscious mind. The person feels deserving of some sort of punishment.
Similar to other symptoms, the potential of delusion is not far, and they can extend from a common distortion in self.
For instance, as Beck points out in one of his books, someone with low self-esteem may progress to the point in which they believe they are the devil. Further, and perhaps one of the most unique ways you can see this developing or at-least present in someones mind without being a professional in that filed, specifically, is in depression memes. It is no surprise hundreds of likes / upvotes go to memes that allude to some sort of punishment for being in / while going through some sort of depression.
If the intensity of some cognitive distortions reach levels intense enough, one cab warrant it a delusion.