By default, the Container environment is also configured to use the Glue Data Catalog so you can use spark.sql commands against Glue tables.įrom pyspark. The following code snippet shows how to initialize a Spark Session inside the notebook. Use the Create: New Jupyter Notebook command to create a new Jupyter notebook. Inside the VS Code Terminal, you can use the pyspark or spark-shell commands to start a local Spark session.īy default, the EMR Development Container also supports Jupyter. You can use it like any Spark-enabled environment. ![]() The EMR Development Container is configured to run Spark in local mode. devcontainer/aws.env file that you can populate with AWS credentials. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |