Family and Borrowed Money?

With my sister and mother, if they need it, they have it because they are the same with me. My brother had been out of work for a few weeks and asked to borrow $100. My brother has never been there when I needed him but I felt bad for him so I loaned him the money. He now has a good job (got his first check this week) and just got his tax return. Should I ask him for the money back? I just think if I do not ask now, I will never get it back.