Aero Precision M4E1 Lower - Front Pivot Question

I just got a complete aero precision m4e1 lower in. I was going to get the upper attached but I cant seem to get the front pin removed. Is there some trick with these I am not seeing?

The rear pin was easily able to be pushed out with my fingers basically. But Ive tried oiling and trying to hammer out the front pin, but it doesnt go anywhere... Am I missing something haha