Lessons learned - always do gauge swatch in pattern. Turns out I knit so much looser when doing 3 colour knitting, oops.

Initially I did this (After the colourwork nd short row shaping I realised this was massive and decreased from 192 stitches to 132 using 4 decrease points in the round over a few inches. This has massively changed the shape but I kinda love it.)
But then I realised it looked good from the front, but I had a shelf at the back which was not filled by boobs which looked very odd, so I undid back to the colourwork, scrapped the short rows so the arm holes were slightly higher, and did slight side shaping only. This has made for a looser but still slightly fitted look which works.
I then combined this pattern with a baby jumper to make a matching one for my niece.