Can I use CSS in a form that is running in an iFrame?

Yes, you can. 

You could also echo the link rel inclusion as a BEFORE FORM piece.
Form Properties > Advanced Tab > More Options Link > Form Pieces
Then in the BEFORE FORM section, click on the CUSTOM button and add the following code:

$this->execPieceByName('ff_InitLib');
echo '<link rel="stylesheet" type="text/css" media="all" href="'.JURI::root().'path/to/stylesheet.css">';

Then click on the SAVE icon at the bottom of that screen, and then, once back at the form properties screen, click on the SAVE icon in the upper right-hand corner of the screen.