You do not have to struggle with all the dependencies synchronization.

The advantage of the Anaconda distribution is that lot of the essential python packages comes in bundled.

We install Continuum’s Anaconda distribution by downloading the install script from the Continuum website.

We also leverage the strengths of Spark including Spark-SQL, Spark-MLLib or ML.

We leverage the power of the Python ecosystem with libraries such as Numpy (scientific computing library of high-level mathematical functions to operate on arrays and matrices), SciPy (SciPy library depends on NumPy, which provides convenient and fast N-dimensional array manipulation), Pandas (high performance data structure and data analysis library to build complex data transformation flows), Scikit-Learn (library that implements a range of machine learning, preprocessing, cross-validation and visualization algorithms), NLTK (Natural Language Tool Kit to process text data, libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning, wrappers for industrial-strength NLP libraries)….

Adding the ability to combine Apache Spark to this will also allow you to prototype ideas and exploratory data pipelines before hitting a Hadoop cluster and paying for Amazon Web Services.

It is often useful to have python with the Jupyter notebook installed on your laptop in order to quickly develop and test some code ideas or to explore some data.

c.Pandas DataFrames, Spark DataSets, DataFrames and RDDs.6-Launching Jupyter Notebook with PySpark 2.0.2.Installing and Exploring Spark 2.0 with Jupyter Notebook and Anaconda Python in your laptop

Install spark linux python