Load a different dataset in a Spark dataframe

[ghost]

Hello,
Im not sure if this is possible but wanted to check if I can load a publicly available dataset into a data frame. Im doing this to analyze a different dataset other than tpch. Please let me know if im missing something here. Thanks.

Steps I followed:

• make notebook #to start a notebook

• Ran below commands in notebook:
  from pyspark.sql import SparkSession
  from pyspark.sql.functions import col, sum
  from pyspark.sql.types import *
  from pyspark.sql.functions import *

  Initialize a Spark session

  spark = (
  SparkSession.builder.appName("Spark Anatomy")
  .enableHiveSupport()
  .getOrCreate()
  )

• loading a data from .csv

  fire_df = spark.read
  .csv("/databricks-datasets/learning-spark-v2/sf-fire/sf-fire-calls.csv",
  headers="true",
  inferSchema="true")

• I see an error like, 'AnalysisException: Path does not exist: file:/databricks-datasets/learning-spark-v2/sf-fire/sf-fire-calls.csv'

• I could tell the file is not in the working directory(/opt/spark/work-dir). But not sure how to move the file to the working directory. Also, uploaded the file to a folder called 'Data' in Jupyter notebook. Thank you!

[ghost]

Joseph mentioned this in chat, hope this helps others trying the same.

Hey, Yes you can, put the data you download under the data-processing-spark folder and when reading the data in Spark
(via NB) use the path /opt/spark/work-dir/data-processing-spark/your-data-file