Heart of Crown

When I use a Refuge card as an Action card, can I add a card from the bottom of my draw pile that was put there by a Library to my hand?

No, you cannot. The card refers specifically to groups of cards that have been revealed on the top of the draw pile. It has no effect on revealed cards that are under the cards unrevealed.