Update readme to include google colab setup + remove problematic packages from requirements.txt

Notebooks/SavtaDepth_Colab.ipynb

@@ -0,0 +1,367 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "name": "SavtaDepth Colab.ipynb",
+      "provenance": [],
+      "collapsed_sections": []
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "accelerator": "GPU"
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "QKUz-9qCbHRH"
+      },
+      "source": [
+        "<center> <img alt=\\\"DAGsHub\\\" width=500px src=https://raw.githubusercontent.com/DAGsHub/client/master/dagshub_github.png> </center>\n",
+        "\n",
+        "# SavtaDepth Colab Environment\n",
+        "\n",
+        "### ***This notebook is meant to be run on Google Colab***\n",
+        "\n",
+        "This notebook is a tool to setup and orchestrate (read: Run) experiments for [SavtaDepth](https://dagshub.com/OperationSavta/SavtaDepth) on Google Colab, while maintaining a relatively clean environment and using version control to promote reproducibility. It is a WIP, but following (and modifying) the cells below should let you git clone a project into colab, download the data, run it, and push it to https://DAGsHub.com which is a free platform for open source data science. \n",
+        "\n",
+        "SavtaDepth is an Open Source Data Science project. We'd love to get help from the community, so if you'd like to contribute head over to the [project page](https://dagshub.com/OperationSavta/SavtaDepth) to get started."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "wJNCkwSIa7To"
+      },
+      "source": [
+        "# General Setup\n",
+        "Run this before anything else"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "2Tq2Dl33XDqw"
+      },
+      "source": [
+        "%reload_ext autoreload\n",
+        "%autoreload 2"
+      ],
+      "execution_count": 1,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "UiFeCt5xXEK1"
+      },
+      "source": [
+        "from google.colab import auth\n",
+        "auth.authenticate_user()"
+      ],
+      "execution_count": 2,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "xn8URgDmWBKm"
+      },
+      "source": [
+        "Clone Git Repo from DAGsHub"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "jxOO9c2hU_zM"
+      },
+      "source": [
+        "!git clone https://dagshub.com/OperationSavta/SavtaDepth.git\n",
+        "%cd SavtaDepth/"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "4bndN7kVagXj"
+      },
+      "source": [
+        "*Note: Currently you can't see the intermediate output so it's completely opaque and long process + you can't set up credentials for google within the shell instance*\n",
+        "\n",
+        "This installs conda, creates a virtual environment and installs all relevant requirements."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "JAsG5scFYjqd"
+      },
+      "source": [
+        "%%bash\n",
+        "MINICONDA_INSTALLER_SCRIPT=Miniconda3-py37_4.8.3-Linux-x86_64.sh\n",
+        "MINICONDA_PREFIX=/usr/local\n",
+        "wget https://repo.continuum.io/miniconda/$MINICONDA_INSTALLER_SCRIPT\n",
+        "chmod +x $MINICONDA_INSTALLER_SCRIPT\n",
+        "./$MINICONDA_INSTALLER_SCRIPT -b -f -p $MINICONDA_PREFIX\n",
+        "rm $MINICONDA_INSTALLER_SCRIPT\n",
+        "make env"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "TR0asdFLaeAV"
+      },
+      "source": [
+        "Since locally we might not be working with a GPU, here we install the pytorch version that should use the GPU provided by Google Colab. After that we install the rest of the requirements from the requirements.txt file."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "n3U6qqVFWYRm"
+      },
+      "source": [
+        "!bash -c \"source activate savta_depth && conda install -y pytorch torchvision cudatoolkit=10.1 -c pytorch && make load_requirements\""
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "WOzZu9GQaaeI"
+      },
+      "source": [
+        "### Pull DVC files from our remote"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "sNmiqub0WbqT"
+      },
+      "source": [
+        "!bash -c \"source activate savta_depth && dvc pull -r dvc-remote\""
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "qNNIfodJc_0g"
+      },
+      "source": [
+        "\n",
+        "\n",
+        "---\n",
+        "\n",
+        "\n",
+        "# Setup is done!\n",
+        "If you've made it here, everything is set up. You have the code and data in the file viewer to the left. You can edit the files located in SavtaDepth/src/code/ as you like. You can see the YAML defining the project pipeline in dvc.yaml. If you change dependencies or outputs (for example, add an addition code file for the training stage), make sure you edit the pipeline to reflect this.\n",
+        "\n",
+        "### Once you are done with you changes run the cell below to run the pipeline end-to-end\n",
+        "* You can run this multiple times if you've made a change and want to test it\n",
+        "* If you want to run only a specific stage you can change the `dvc repro` command to any other command you like.\n",
+        "\n",
+        "▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽▽"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "fwOfOwnrZWMe"
+      },
+      "source": [
+        "!bash -c \"source activate savta_depth && dvc repro\""
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "PAxz-29WhN12"
+      },
+      "source": [
+        "---\n",
+        "# Commiting Your Work and Pushing Back to DAGsHub\n",
+        "\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "kyse-hAuZY9X"
+      },
+      "source": [
+        "!git status"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "Ib12i6aOhbgI"
+      },
+      "source": [
+        "# Add the files you want to commit\n",
+        "!git add {your files here}"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "duM9An0Khr_F"
+      },
+      "source": [
+        "Run the following 2 cells without modifications. They will prompt you for a commit message, and for credentials to push back to DAGsHub"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "V9SYYA9Zh3f9"
+      },
+      "source": [
+        "**Commiting**"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "_fjym_38hhgl"
+      },
+      "source": [
+        "os.environ['COMMIT_MESSAGE'] = input('Enter the commit message for you commit: ')\n",
+        "!git commit -m \"${COMMIT_MESSAGE}\"\n",
+        "os.environ['COMMIT_MESSAGE'] = \"\""
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "mWtgXU0ph6Dv"
+      },
+      "source": [
+        "**Pushing to DAGsHub**"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "z7LoVk9Zh9GC"
+      },
+      "source": [
+        "# If this stage fails, make sure to remove outputs as it will show your password to whoever it is shared with.\n",
+        "from getpass import getpass\n",
+        "import os\n",
+        "\n",
+        "os.environ['USER'] = input('Enter the username of your DAGsHub account: ')\n",
+        "os.environ['PASSWORD'] = getpass('Enter the password of your DAGsHub account: ')\n",
+        "os.environ['REPO_URL'] = input('Enter the url of your DAGsHub project: ').split('https://')[-1]\n",
+        "os.environ['DAGSHUB_AUTH'] = os.environ['USER'] + ':' + os.environ['PASSWORD']\n",
+        "\n",
+        "!git push https://$DAGSHUB_AUTH@$REPO_URL.git"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "MXlzJbuLiFWb"
+      },
+      "source": [
+        "***NOTE: ALWAYS RUN THIS CELL AFTER THE PREVIOUS***, it will delete your DAGsHub password in case you share this notebook with someone"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "6-yZzyeAiGPI"
+      },
+      "source": [
+        "os.environ['PASSWORD'] = os.environ['DAGSHUB_AUTH'] = \"\""
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "FImsqcvYie77"
+      },
+      "source": [
+        "## Push data back to your DVC Remote\n",
+        "For this step you must first create a DVC remote on some cloud provider. We recommend Google Cloud Storage. If you're not sure how to set up a DVC remote, [follow these instructions](https://dagshub.com/docs/getting-started/set-up-remote-storage-for-data-and-models/#create-a-storage-bucket) (you only need to go through creating a storage bucket and adding permissions)."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "L4XWSvPQjfLb"
+      },
+      "source": [
+        "Add your storage bucket to DVC (replace {bucket-name} with the bucket name you chose)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "kGx03PpijiAW"
+      },
+      "source": [
+        "!bash -c \"source activate savta_depth && dvc remote add my-dvc-remote gs://{bucket-name}\""
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "i8uYEa64imDl"
+      },
+      "source": [
+        "from google.colab import auth\n",
+        "auth.authenticate_user()"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "ZsM5epDbiotM"
+      },
+      "source": [
+        "!bash -c \"source activate savta_depth && dvc push -r my-dvc-remote\""
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ne7pU7bMj18m"
+      },
+      "source": [
+        "# That's it, you can now create a PR on DAGsHub."
+      ]
+    }
+  ]
+}

README.md

@@ -11,12 +11,53 @@ If you'd like to take part, please follow the guide.
 
 ### Setting up your environment to contribute
 * To get started, fork the repository on DAGsHub
-* Now, You have 3 way to set up your environment: Google Colab, local or docker. If you're not sure which one to go with, we recommend using Colab.
+* Now, you have 3 way to set up your environment: Google Colab, local or docker. If you're not sure which one to go with, we recommend using Colab.
 #### Google Colab
-We can treat Colab, as your web connected, GPU powered IDE. Here is a link to a well-documented Colab notebook, that will load the code and data from this repository, enabling you to modify the code and re-run training.
+Google Colab can be looked at as your web connected, GPU powered IDE. Below is a link to a well-documented Colab notebook, that will load the code and data from this repository, enabling you to modify the code and re-run training. Notice that you still need to modify the code within the `src/code/` folder, adding cells should be used only for testing things out.
+
+In order to edit code files, you must save the notebook to your drive. You can do this by typing `ctrl+s` or `cmd+s` on mac.
+
+\>\> **[SavtaDepth Colab Environment](https://colab.research.google.com/drive/19027P09jiiN1C99-YGk4FPj0Ol9iXUIU?usp=sharing)** \<\<
+
+**_NOTE: The downside of this method (if you are not familiar with Colab) is that Google Colab will limit the amount of time an instance can be live, so you might be limited in your ability to train models for longer periods of time._**
+
+This notebook is also a part of this project, in case it needs modification, in the `Notebooks` folder. You should not commit your version unless your contribution is an improvement to the environment.
 
-**_NOTE: The downside of this method (if you are not familiar with Colab) is that Google Colab will limit the amount of time an instance can be live, so you might be limited in your ability to train models for longer periods of time. 
 #### Local
+* Create a virtual environment or Conda environment and activate it
+    ```bash
+    # Create the virtual environment
+    $ make env
+  
+    # Activate the virtual environment
+    # VENV
+    $ source env/bin/activate .
+  
+    # or Conda
+    $ source activate savta_depth
+    ```
+* Install the required libraries
+    ```bash
+    $ make load_requirements
+    ```
+  **_NOTE: Here I assume a setup without GPU. Otherwise, you might need to modify requirements, which is outside the scope of this readme (feel free to contribute to this)._**
+* Pull the dvc files to your workspace by typing:
+
+    ```bash
+    $ dvc pull -r dvc-remote
+    $ dvc checkout #use this to get the data, models etc
+    ```
+
+    **Note**: You might need to install and setup the tools to pull from a remote. Read more in [this guide](https://dagshub.com/docs/getting-started/set-up-remote-storage-for-data-and-models/) on how to setup Google Storage or AWS S3 access.
+* After you are finished your modification, make sure to do the following:
+    * If you modified packages, make sure to freeze your virtualenv by typing in the terminal:
+
+        ```bash
+        $ make save_requirements
+        ```
+
+    * Push your code to DAGsHub, and your dvc managed files to your dvc remote. To setup a dvc remote please refer to [this guide](https://dagshub.com/docs/getting-started/set-up-remote-storage-for-data-and-models/).
+    
 #### Docker
 * Next, clone the repository you just forked by typing the following command in your terminal:
   ```bash
@@ -53,7 +94,6 @@ We can treat Colab, as your web connected, GPU powered IDE. Here is a link to a
         ```
 
 
-        
 * Pull the dvc files to your workspace by typing:
 
     ```bash
@@ -70,8 +110,11 @@ We can treat Colab, as your web connected, GPU powered IDE. Here is a link to a
         ```
 
     * Push your code to DAGsHub, and your dvc managed files to your dvc remote. In order to setup a dvc remote please refer to [this guide](https://dagshub.com/docs/getting-started/set-up-remote-storage-for-data-and-models/).
-    * Create a Pull Request on DAGsHub!
-    * 🐶
+    
+---
+### After pushing code and data to DAGsHub
+* Create a Pull Request on DAGsHub!
+* 🐶
 
 ### TODO:
 [ ] Web UI

requirements.txt

@@ -17,7 +17,7 @@ decorator==4.4.2
 dictdiffer==0.8.1
 distro==1.5.0
 dpath==2.0.1
-dvc==1.6.0
+dvc==1.9.1
 fastai==2.0.0
 fastcore==1.0.0
 fastprogress==1.0.0
@@ -38,6 +38,7 @@ grandalf==0.6
 h5py==2.10.0
 idna==2.10
 importlib-metadata==1.7.0
+ipykernel==5.3.4
 ipython==7.17.0
 ipython-genutils==0.2.0
 jedi==0.17.2
@@ -45,8 +46,6 @@ joblib==0.16.0
 jsonpath-ng==1.5.1
 kiwisolver==1.2.0
 matplotlib==3.3.1
-mkl-random==1.1.1
-mkl-service==2.3.0
 murmurhash==1.0.2
 nanotime==0.5.2
 networkx==2.4