How in the world am I supposed to keep out the enemy when I cannot add more men to the outer wall where the blue armies attack? Some tips would be greatly appreciated

Mass up archers on the tower in the middle of the castle for when they break through (this will also stop all green attacks without endangering the inner sanctum and keep) ~40 should be more than enough.

On the outer walls where blue attacks, I would suggest just building spike traps to slow down catapults and hurt enemy units and to use pitch on strategic positions, as it is probably the most deadly weapon in the game, especially against armored units.

So yea... really just bother producing bows and some armor + swords for swordsmen when they break the outer walls.