Changed a bit design. Used Italian cast on and double knitting technique.
Because of that beginning was loose (gauge 20sts in 10cm), but later got more tense (gauge 22-24 in 10cm).

Maybe solution would be to make 1st 5-10 rows with 3.5mm needles and later use 4mm. Also skip garter stitch on sides to release decrease tension.