WIBTA for insisting we alter the custody agreement after my ex brought me back the wrong baby?

So this guy, "Sean", and I have a... complex... history, but the important part of that history is that Sean got me and another woman, "Lydia", pregnant at roughly the same time. Our sons are only about 6 weeks apart in age. Lydia's son is currently just over 15 months, while mine is just under 17 months.

Lydia and I have identical custody agreements. We both had sole custody for the entire first year, with Sean only having visitation, and when both boys were a year old, it became 50/50 custody. Both myself and Lydia have our sons for the odd weeks, while Sean gets the even weeks. This means Sean has both boys at the same time. He pushed for this, as he wanted a week on/week off from looking after kids. He also requested this so his sons could bond, but his primary motive was getting time off.

Sean had both boys this last week. He brought my son back earlier today. I was on a work call and the baby was sleeping, so Sean just sort of placed the baby, carrier and all, down on the floor and left silently. When the baby woke up and started crying, I got off the work call and immediately noticed that this was not my son, as my son has blue eyes, and this baby had green eyes. Taking off the baby's hat also showed dark blond hair, and not my son's light brown hair.

I immediately called Sean, in full panic mode, telling him to bring me the correct child right this second. It had only been about 10 minutes so Sean wasn't far away and hadn't gone by Lydia's yet, but Jesus fucking Christ it was the worst 10 minutes of my life. Sean came back smiling, carrying my son, and says in the most condescending tone of voice "see? He's fine. No harm done."

I told him that we are fixing this custody agreement. He is not having both boys alone again. He couldn't tell his own sons apart, so the solution here is for him to not be in a position to confuse them. I feel that this whole situation is already a total fucking farce, and I should never have been in that position. I want to know that I have my child, not Lydia's. I do not feel that this is a lot to ask for.

Sean has said that I'm being unreasonable. Says it's an easy mistake to make, and that there were times his parents brought home the wrong baby from nursery without a half sibling to excuse it, and that this was the first time it happened and the issue only lasted 10 minutes. He also says that I'll be causing undue harm to all 3 guys in this situation, as the kids won't get to bond with their half brother, and Sean won't get any time to himself if he has Lydia's son on even weeks and mine on odds, meaning that he'll be more exhausted and it will decrease his income as he won't be able to work overtime (which he currently does in his weeks without the boys).

I said I'd be contacting the people who drew up our agreement to amend it, and Sean called me selfish/crazy.

I am worried that I might be TA as I could easily be acting irrationally and if I am doing that, then there is a risk that the main person this will harm is my son, by affecting his relationship with his father and half brother, which I have no intention of doing.

WIBTA for insisting on making the necessary changes so Sean doesn't have both boys simultaneously again?