The audio codec has a built in headphone amplifier that drives the headphones out, that's why headphones (which are speakers) can be directly connected. The line outs are not amplified, and cannot be directly connected to any kind of speaker, not even tiny headphones.
As Paul already stated, you cannot hook up any kind of speaker directly to a line out, it must go through an amplifier first.
You should connect the line-out to your amplifier. Technically you could connect the headphone out to the amp input, it would "work" if the volume is safely low enough, but is a bad idea because the headphone output is 'amplified' and could be high enough to damage the input to the amp.