A matrix is a rectangular table of numbers that can represent a system of linear equations. Matrix notation uses row operations like switching rows, multiplying rows by constants, and adding multiples of rows to other rows to transform the matrix into upper triangular form, from which the solution to the system of equations can be determined. The three main row operations are: 1) switching rows, 2) multiplying a row by a constant, and 3) adding a multiple of one row to another row.