notebook-demos/ingesting-an-image.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ingesting an Image\n",
    "\n",
    "This example shows how to ingest a grayscale image and save the results locally. It is assumed that you have already read through the documentation on GeoPySpark before beginning this tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting the Data\n",
    "\n",
    "Before we can begin with the ingest, we must first download the data from S3. This curl command will download a file from S3 and save it to your `/tmp` direcotry. The file being downloaded comes from the [Shuttle Radar Topography Mission (SRTM)](https://www2.jpl.nasa.gov/srtm/index.html) dataset, and contains elevation data on the east coast of Sri Lanka.\n",
    "\n",
    "**A side note**: Files can be retrieved directly from S3 using the methods shown in this tutorial. However, this could not be done in this instance due to permission requirements needed to access the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!curl -o /tmp/cropped.tif https://s3.amazonaws.com/geopyspark-test/example-files/cropped.tif"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is an Ingest?\n",
    "\n",
    "Before continuing on, it would be best to briefly discuss what an ingest actually is. When data is acquired, it may cover an arbitrary spatial extent in an arbitrary projection. This data needs to be regularized to some expected layout and cut into tiles. After this step, we will possess a `TiledRasterLayer` that can be analyzed and saved for later use. For more information on layers and the data they hold, see the [layers](layers.ipynb) guide."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Code\n",
    "\n",
    "With our file downloaded we can begin the ingest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import geopyspark as gps\n",
    "\n",
    "from pyspark import SparkContext"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting Up the SparkContext\n",
    "\n",
    "The first thing one needs to do when using GeoPySpark is to setup `SparkContext`. Because GeoPySpark is backed by Spark, the `pysc` is needed to initialize our starting classes.\n",
    "\n",
    "For those that are already familiar with Spark, you may already know there are multiple ways to create a `SparkContext`. When working with GeoPySpark, it is advised to create this instance via `SparkConf`. There are numerous settings for `SparkConf`, and some **have** to be set a certain way in order for GeoPySpark to work. Thus, `geopyspark_conf` was created as way for a user to set the basic parameters without having to worry about setting the other, required fields."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "conf = gps.geopyspark_conf(master=\"local[*]\", appName=\"ingest-example\")\n",
    "pysc = SparkContext(conf=conf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading in the Data\n",
    "\n",
    "After the creation of `pysc`, we can now read in the data. For this example, we will be reading in a single GeoTiff that contains spatial data. Hence, why we set the `layer_type` to `LayerType.SPATIAL`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "raster_layer = gps.geotiff.get(layer_type=gps.LayerType.SPATIAL, uri=\"file:///tmp/cropped.tif\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tiling the Data\n",
    "\n",
    "It is now time to format the data within the layer to our desired layout. The aptly named, `tile_to_layout`, method will cut and arrange the rasters in the layer to the layout of our choosing. This results in us getting a new class instance of `TiledRasterLayer`. For this example, we will be tiling to a `GlobalLayout`.\n",
    "\n",
    "With our tiled data, we might like to make a tile server from it and show it in on a map at some point. Therefore, we have to make sure that the tiles within the layer are in the right projection. We can do this by setting the `target_crs` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "tiled_raster_layer = raster_layer.tile_to_layout(gps.GlobalLayout(), target_crs=3857)\n",
    "tiled_raster_layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pyramiding the Data\n",
    "\n",
    "Now it's time to pyramid! With our reprojected data, we will create an instance of `Pyramid` that contains 12 `TiledRasterLayer`s. Each one having it's own `zoom_level` from 11 to 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pyramided_layer = tiled_raster_layer.pyramid()\n",
    "pyramided_layer.max_zoom"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pyramided_layer.levels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving the Pyramid Locally\n",
    "\n",
    "To save all of the `TiledRasterLayer`s within `pyramid_layer`, we just have to loop through values of `pyramid_layer.level` and write each layer locally."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for tiled_layer in pyramided_layer.levels.values():\n",
    "    gps.write(uri=\"file:///tmp/ingested-image\", layer_name=\"ingested-image\", tiled_raster_layer=tiled_layer)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1+"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}