MathWorks interview question

solve linear algebra problem.