[14f] older brother of kid I babysit [13m] realized i'm there to sit for him too and now it's awkward

I've been babysitting a 5f and his older brother 13m who is in a grade lower than I am. I'm mostly there for the 5f but still make dinner for both of them and help him with homework. I was over a few days ago and the 5f was out with her grandma so it was just the two of us and he realized I was there to babysit him. I could tell when he realized it and he went to his room and didn't come out the rest of the time I was there. We ride the same bus to school and now he won't even look at me. What should I do? Stop sitting for the family? Tell the mom I only want to sit the 5f? Ask the mom to talk with him?