I find that often in low stakes, these are silly players who read one snippet of a strategy in a forum somewhere and think it's the best way to play.
In situations like the one you describe, I have two possible ways that you can play it to share with you.
The first, is a longer-than-one-hand-play to trap your foolhardy and predictable opponent.
Strategy #1 - "Enjoy my blinds, I'll be taking every last one of yours soon"
If you can afford to lose a few blinds, flat call or min-raise out of position against your opponent, while they are seated somewhere near the button (regardless of your cards, just look for a good moment when there isn't lots of drama).
If they shove all in and you have air, take a moment to imply it's a bit of a decision and then fold. If they don't shove but do call, play the hand as you normally would, perhaps a bit weaker (check, check, ?). Sometimes you can take these pots down anyway through cards or actions.
After you've let them have maybe 3 or 4 pots this way, they'll be ripe for the taking. All you have to do is find a decent hand (KQ+), play it weakly, and then come over the top - by checking or calling and then re-raising their bet, preflop or on later streets - or just call their all in when they aren't expecting it.
If they're bad/tilting they'll call or reraise with a poor hand; if they just happened to get lucky with nice cards this time they'll do the same. If they're a bit better they'll either call [a less than all in raise] or just plain fold. After doing this to them a few times they'll soon either learn a lesson and change their play, get knocked out by you, or in unlucky circumstances, knock you out/down.
My second strategy is sometimes useful when you want to see the flop with a decent hand like JT, but you don't want to inflate the pot too early:
Strategy #2 - Fight Fire With Fire
Raise to an arbitrarily odd amount (like 2.85 BBs) preflop and out of position, before it gets to them. Often they'll fold, sometimes they'll call, yet rarely will they re-raise over the top.
If they do re-raise over the top, and it's -EV to call, give it up and wait a few rounds. This will contribute to them gaining a false sense of security against you. You will be looking weak; but isn't that quite when you're actually quite strong?
In conclusion, my attitude and strategy towards these types of players is make them think you're weak, and then look for an opportunity to punish them. Often the weaker they think you are, the more outrageously bold they'll get, and it soon becomes obvious that you can continuously shove back over them or call their shoves with a wider range of cards. Enable them to become over confident, and then pounce on their newly revealed weaknesses.
P.S. As a side note, you should mostly be analysing the play with at least crude mathematics, regardless of stack sizes or closeness to victory i.e. if they've just shoved all in four times in six hands, and it's not the first flurry, it's about time to make the brave QJ/JT/QT/etc. call.