What is the relation between Latex and MathJax? I find this reference very useful for writing questions on mathematics site and I also came across this site for Latex symbols which seems very similar to me. So I want to understand the relation here.
As other answers have already explained, LaTeX and MathJax are related markup languages for preparing mathematical documents. In particular, MathJax is designed to be readily usable by people who know LaTeX. However, MathJax does not support the full power of LaTeX because (1) doing so would be very computationally intensive (2) doing so would be incompatible with supporting Markdown and basic HTML (3) doing so would open math.SE to malicious code insertion, since LaTeX is a Turing complete language.
As a first approximation, you can assume that LaTeX constructions you are familiar with will work in MathJax.
As a second approximation, you can assume that mathematical symbols and commands closely related to them will work, but commands related to page layout, document structure and macro programming may not.
For the details, see here.
$\LaTeX$ is typesetting software that's not limited to mathematical notation. $\LaTeX$ keeps track of page numbers, section numbers, etc., and does everything MSWord can do, more efficiently and with less clutter. One types whole articles and books in $\LaTeX$. MathJax is a very stripped-down relative of $\LaTeX$ used for mathematical notation on web pages.
Here is an example of a document written entirely in $\LaTeX$ --- not just the parts that are in mathematical notation. Obviously MathJax could not do this.