Updating Readme, adding data overview (#3)

README.md

@@ -1,6 +1,6 @@
-# Source of truth
+# Brainforest EcoHackathon 2024, Group "" (pronounced empty string)
 
-## welcome and initial setup
+## welcome and minimal setup
 
 hi all,
 i think when we start with the EDA, it suffices if anyone just uses what they are used to (f.e. conda or whatever). However, afterwards, i think it could be helpful that everyone, always has exactly the same environment, same package/python versions, which is why i propose working with docker to minimize headaches and "but it works on my machine" issues. I think with this minimal setup below, we can fully focus on hacking while not having pain with painful stuff.
@@ -15,7 +15,9 @@ please feel free to add / change / challenge things!
 
 essentially, you just have to build the container with the services you want. if you're interested in it i can go into more detail just let me know.
 
-1. navigate to dc/dev and run:
+1. clone the repo
+
+2. navigate to dc/dev and run:
 
 ```
 docker compose up -d --build
@@ -23,61 +25,45 @@ docker compose up -d --build
 
 only use the `--build` flag the first time around, or if you want to rebuild the container (e.g. when having added a package you need in the container). **NOTE:** the `-d` flag stands for `detach` which means that your docker container runs in the background and does not log everything into your console.
 
-2. then, to check whether everything worked hit:
+3. then, to check whether everything worked hit:
 
 ```
 docker ps
 ```
 
-3. for this specific setup, you can head to `localhost:8888` where jupyterlab is running.
+4. for this specific setup, you can head to `localhost:8888` where jupyterlab is running.
 
-4. to create a new file (using jupytext, see below), just create a new .ipynb file, the .py file will be created automatically. all the changes you make in the notebook, will be reflected in the .py files which you then can use for your commits.
+5. to create a new file (using jupytext, see below), just create a new .ipynb file, the .py file will be created automatically. all the changes you make in the notebook, will be reflected in the .py files which you then can use for your commits.
 
 now you shoold see the running docker containers.
 
 ### what about huggingface spaces:
 
-- [here](./docs/huggingface-spaces.md), you can see what huggingface spaces is and how we can complement our github repo with it (credits to chat-gpt)
+- IMO we have two options:
+  1. [connect hugginface space](https://huggingface.co/docs/hub/spaces-github-actions#managing-spaces-with-github-actions) of our group to this repo using github actions
+  2. have separate hugginface space (might be a pain)
 
 ### jupytext - nice versioning of jupyter notebooks
 
-since we are likely be working with jupyter notebooks alot, lets use jupytext. It automatically maps .ipynb to .py files with some magic. The .ipynb are in the gitignore, so we only have .py files nicely versioned in the repo. read more about it [here](https://jupytext.readthedocs.io/en/latest/)
+- we are likely be working with jupyter notebooks alot
+- lets use [jupytext](https://jupytext.readthedocs.io/en/latest/)
+- It automatically maps .ipynb to .py files with some magic
+- The .ipynb are in the gitignore, so we only have .py files nicely versioned in the repo
 
 ### trunk based development
 
-lets stick to trunk based. if you dont know what it is, read all about it [here](https://trunkbaseddevelopment.com/)
-
-key take aways:
-
-#### Trunk-Based Development: Key Points
-
-1. **Single Main Branch**: All developers commit to the trunk or main branch.
-2. **Short-Lived Branches**: Branches, if used, are short-lived and quickly merged back.
-3. **Frequent Integrations**: Code changes are integrated frequently, often multiple times a day.
-4. **Feature Flags**: Incomplete features are managed with feature flags to maintain trunk stability.
-
-#### Benefits
-
-- **Reduced Integration Problems**: Early conflict detection and resolution.
-- **Higher Code Quality**: Continuous testing ensures stable and high-quality code.
-- **Simpler Workflow**: Less overhead managing branches and merges.
-- **Enhanced Collaboration**: Encourages teamwork and code reviews.
+- lets stick to trunk based
+- if you dont know what it is, read all about it [here](https://trunkbaseddevelopment.com/)
+- or read the [key take aways](./docs/key-takeaways-tb.md)
 
-#### Challenges
+### code format
 
-- **Discipline Required**: Developers must write clean, well-tested code.
-- **Handling Incomplete Features**: Requires careful use of feature flags.
+- lets stick to [Black](https://black.readthedocs.io/en/stable/) for python and [Prettier](https://prettier.io/) for .md and other formats
+- using docker for the purpose of formatting is really easy
 
-#### Best Practices
+  1.  `chmod +x format` so that the `format` file is executable
+  2.  then simply use `./format` before adding your changes and all the files will be autoformatted
 
-- **Frequent Commits**: Small, incremental changes reduce integration risks.
-- **Comprehensive Testing**: Automated tests for codebase coverage.
-- **Feature Flags**: Manage incomplete or experimental features.
-- **Code Reviews**: Maintain quality and knowledge sharing.
+  ## awesome data overview
 
-### code format
-
-- lets stick to black for python and prettier for .md and other formats
-- using docker for the purpose of formatting is really easy
-- just `chmod +x format` so that the `format` is executable
-- then simply use `./format` before adding your changes and all the files will be autoformatted
+  ![](./docs/data_overview_yuri.jpg)

docs/huggingface-spaces.md

@@ -1,64 +0,0 @@
-Hugging Face Spaces and GitHub repositories serve different but complementary purposes. Here’s a comparison and how they can be used together:
-
-### Comparison with GitHub Repositories
-
-- **GitHub Repository**:
-
-  - **Purpose**: Primarily used for version control, collaboration, and sharing of code and projects.
-  - **Capabilities**: Stores code, tracks changes, manages issues, and supports CI/CD pipelines.
-  - **Usage**: Developers collaborate on software development projects, manage codebases, and deploy applications.
-
-- **Hugging Face Spaces**:
-  - **Purpose**: Designed specifically for deploying interactive machine learning applications and demos.
-  - **Capabilities**: Hosts and deploys machine learning models and applications using frameworks like Streamlit, Gradio, or custom HTML/CSS/JS.
-  - **Usage**: Users create and share interactive demos and applications, especially in the field of machine learning.
-
-### Integration with GitHub
-
-You can import a GitHub repository into Hugging Face Spaces to deploy an application hosted on GitHub. Here’s how to do it:
-
-1. **Create a Space on Hugging Face**:
-
-   - Go to the Hugging Face Spaces website and create a new Space.
-
-2. **Link to GitHub Repository**:
-
-   - During the setup of the new Space, you can link it to a GitHub repository. This allows Hugging Face Spaces to pull the code from your GitHub repo.
-
-3. **Configure Your Space**:
-
-   - Ensure your repository contains the necessary files for the framework you are using (Streamlit, Gradio, or HTML/CSS/JS).
-   - For example, if you are using Streamlit, ensure you have a `requirements.txt` file for dependencies and a main Python script that runs the Streamlit app.
-
-4. **Deploy the Application**:
-   - Once linked, Hugging Face Spaces will automatically deploy the application from the GitHub repository.
-   - Any updates pushed to the GitHub repository can automatically trigger redeployment of the application on Hugging Face Spaces.
-
-### Example Steps to Import a GitHub Repo into Hugging Face Spaces
-
-1. **Create a New Space**:
-   - Navigate to Hugging Face Spaces and click on “New Space”.
-2. **Set Up Space**:
-
-   - Choose a name for your Space, select the appropriate SDK (e.g., Streamlit, Gradio, or HTML), and choose the visibility (public or private).
-
-3. **Connect GitHub Repository**:
-
-   - In the Space settings, you will find an option to link a GitHub repository. Provide the URL of your GitHub repository.
-   - Hugging Face Spaces will clone your GitHub repository to use it as the source code for your Space.
-
-4. **Configure and Deploy**:
-
-   - Make sure your GitHub repository is set up correctly for the chosen framework. For example, a Streamlit app should have a `requirements.txt` and an entry-point script like `app.py`.
-   - Once everything is set up, your Space will be deployed and can be accessed via a URL provided by Hugging Face.
-
-5. **Update and Maintain**:
-   - Any changes you push to the linked GitHub repository will be reflected in the deployed application after the repository is synced with Hugging Face Spaces.
-
-### Benefits
-
-- **Version Control**: Leveraging GitHub’s version control capabilities ensures that your code is managed effectively.
-- **Collaboration**: Teams can collaborate on the development of the application using GitHub’s collaborative tools.
-- **Easy Deployment**: Hugging Face Spaces simplifies the deployment of interactive machine learning applications without the need for complex infrastructure management.
-
-By combining the strengths of GitHub and Hugging Face Spaces, you can efficiently develop, manage, and deploy machine learning applications.

docs/key-takeaways-tb.md

@@ -0,0 +1,25 @@
+## Trunk-Based Development: Key Points
+
+1. **Single Main Branch**: All developers commit to the trunk or main branch.
+2. **Short-Lived Branches**: Branches, if used, are short-lived and quickly merged back.
+3. **Frequent Integrations**: Code changes are integrated frequently, often multiple times a day.
+4. **Feature Flags**: Incomplete features are managed with feature flags to maintain trunk stability.
+
+## Benefits
+
+- **Reduced Integration Problems**: Early conflict detection and resolution.
+- **Higher Code Quality**: Continuous testing ensures stable and high-quality code.
+- **Simpler Workflow**: Less overhead managing branches and merges.
+- **Enhanced Collaboration**: Encourages teamwork and code reviews.
+
+## Challenges
+
+- **Discipline Required**: Developers must write clean, well-tested code.
+- **Handling Incomplete Features**: Requires careful use of feature flags.
+
+## Best Practices
+
+- **Frequent Commits**: Small, incremental changes reduce integration risks.
+- **Comprehensive Testing**: Automated tests for codebase coverage.
+- **Feature Flags**: Manage incomplete or experimental features.
+- **Code Reviews**: Maintain quality and knowledge sharing.

format

@@ -1,7 +1,7 @@
 #! /bin/sh
 
 # Run as root, otherwise the container cannot modify the mounted files.
-docker run --rm --user root --volume $(pwd):/work tmknom/prettier prettier --loglevel warn --write .
+docker run --rm --user root --volume $(pwd):/work tmknom/prettier prettier --log-level warn --write .
 
 # Format Python files using Black
 docker run --rm --user root --volume $(pwd):/data cytopia/black:latest .