Mercedes ended third practice for the Malaysia GP fastest overall with Nico Rosberg edging out Lewis Hamilton at Sepang.