It's a mask applied to a layer above all the rest. And that paper texture fills the masked out layer. As ron stated specific types of brushes can create that jagged edge effect.
And for that multi-layered look, just double up on the masked paper layer, and apply a different shaped mask to the 2nd layer.
And give the topmost paper layer a slight drop shadow to enhance the depth effect.
That title frame is opened-ended. Could be a photo, could be lineart filled with gradients, or a metal effect from a plugin. That's not the toughest element in that artwork.