Once upon a time...
When you first opened an image, you have a history "state" in your History palette (at the top). You then add a brush stroke, type some text, apply a filter, etc. etc. Each of these adds to the image's history.
You set the number of history states you want PS to remember in Preferences (the default is 20). After 20 "things" you do to the image, the first "thing" disappears. So, if you're doing a lot, like most of us do, and want PS to remember a specific time in the image's history, you create a new history "state" in your history palette. No matter what else you do to the image, if you click on that state, it'll go back to whatever that was. Apply the History Brush now, and you'll get there.
If you click on one of the things in the History Palette, you'll get back to where you were when you did whatever that thing was. Same thing, apply the History Brush and you'll paint in whatever the image looked like at that point in your history's time.
If you don't click on anything in the history palette, PS assumes you want to go back to the beginning when you paint with the History Brush.
Think of the History Brush as an undo applied to a brush. (The Art History brush works exactly the same way, except it applies artistic effects while it goes back in time.)
Actually, the History and Art History brushes are way underused. You can do some cool effects with them, like apply a very distorting filter like twirl, then paint in a history to the original in the middle of it.
Hope that helps, and didn't confuse you
(Edited to add caution)
Caution: The history states only stay as long as you have the image opened. They disappear when you close it.
Black and White images with colour parts 
Linear Mode
