Nevermind, the code part I used was using get posts which does not seem to work for the order. I switched over to WP Query. Any reason why the order does not work for get posts? The relationship query works as intended but not the order.
Using get_posts is risky, since it auto applies suppress_filters=true, which is used to filter the results (the extension changes the query by applying new WHERE, JOIN clauses). If you want to make it work like WP_Query, I think you need to set it to false.