up

README.md

@@ -16,63 +16,29 @@ Warning: this is an experimental, proof-of-concept project made in a few days.
 
 It is not ready for production use by other people! Also, this use models that should only be used for research purposes (no commercial usage).
 
-Note: this won't work on iOS due to an apparent ban on Media Source Extensions (available on iPadOS).
+Note: because the stream uses FLV, it doesn't work on iPhone. There is however a [Twitch mirror here](https://www.twitch.tv/ai_webtv).
 
-It should be possible however to use some other protocol or library.
+The main code of the webtv is located inside the [media-server](https://huggingface.co/spaces/jbilcke-hf/media-server/tree/main) :
 
-# Installation
-## Building and run without Docker
+manual steps:
+- human input to write a short paragraph describing a multi-shot video sequence
+- manual submit it to GPT-4 to generate a list of video captions for each shot (the system instructions are extracts from a stable diffusion guide)
+- commit the captions to the [playlist database](https://huggingface.co/spaces/jbilcke-hf/media-server/raw/main/database.json)
 
-```bash
-nvm use
-npm i
-```
+Inside the `media-server` space (generation process running in the background):
+- for each prompt in the database
+- generate a silent 3 seconds video clip with   Zeroscope V2 576w (hosted on Hugging Face Spaces)
+- upscale the clip with  Zeroscope V2 XL (also a HF Space)
+- perform frame interpolation with FILM (also a HF Space)
+- storage in the Persistent Storage of the media-server Space
 
-First setup some env vars:
-WEBTV_VIDEOPATH="./sandbox/video"
-WEBTV_AUDIOPATH="./sandbox/audio" 
-WEBTV_RTMP_URL="rtmp://localhost:1935/webtv"
+Inside the `media-server` space (streaming process running in the foreground):
+- for each video file in the persistent storage folder
+- add it to a new FFmpeg playlist (it's just a .txt file)
+- broadcast it over the RTMP protocol using FFmpeg (in FLV format)
+- diffusion of the stream using node-media-server
 
+Inside the `AI-WebTV` space:
+- display the stream using `mpegts.js`
+- this doesn't work on iPhone, but now there is also a Twitch mirror
 
-In a terminal, run:
-
-```
-./scripts/init.sh
-```
-
-Then run:
-
-```
-./scripts/audio.sh
-```
-
-In another terminal, run:
-
-```
-./scripts/video.sh
-```
-
-In another terminal, run:
-
-```
-./scripts/stream.sh
-```
-
-In another terminal, run:
-
-```
-npm run start
-```
-
-## Building and running with Docker
-
-```bash
-npm run docker
-```
-
-This script is a shortcut executing the following commands:
-
-```bash
-docker build -t ai-webtv .
-docker run -it -p 7860:7860 ai-webtv
-```