Added a few decreases to the sleeves as well as switching to 5mm needles before starting the ribbing (about 5cm above), did the ribbing with 4mm needles to avoid the super large cuffs. Like so:

Switched to 5mm after two and a half chart repeats, decreased 2 stitches (p2tg at the two purl stitches closest to before & after the BOR)
knit 8 rounds then decrease two more stitches
knit 7 more rounds, then decrease 3 more stitches so I had 68 stitches for the two by two ribbing, switch to 4mm and knit the ribbing