Why does the captain always sit on the left side?

Because most early rotary engines turned counterclockwise (from the front, clockwise from the pilot’s view).  The physics of rotating objects resulted in left turns being easier than right turns.  That led to standard maneuvers using left turns. And that resulted in the prime/most experienced pilot sitting with a view out of the left window, once airplanes grew large enough to have side-by-side pilots.

The propeller blade turning one direction results in an equal and opposite force attempting to move the aircraft in the opposite direction, resulting in a tendency to bank left.

Precession causes the aircraft to yaw left while the nose is changing its pitch in a downward direction and to yaw right when the nose is changing its pitch in an upwards direction.

If the propeller is not perfectly orthogonal to the direction of travel, there will be a difference in thrust on opposing sides of the propeller blades.  This can result in a yaw to port (left) while nose up and a yaw to starboard (right) while nose down.

The air that passes the propeller is affected by the rotation of the propeller, resulting in a slipstream that spirals about the body of the aircraft.  This results in a leftward yaw.