Much base work started
This commit is contained in:
Dan Milne
548
docs/phases/phase_1.md
Normal file
548
docs/phases/phase_1.md
Normal file
@@ -0,0 +1,548 @@
|
||||
# Velour Phase 1: MVP (Local Filesystem)
|
||||
|
||||
Phase 1 delivers a complete video library application for local files with grouping, transcoding, and playback. This is the foundation that provides immediate value.
|
||||
|
||||
**Architecture Note**: This phase implements a extensible MediaFile architecture using Single Table Inheritance (STI). Video inherits from MediaFile, preparing the system for future audio support in Phase 5.
|
||||
|
||||
## Technology Stack
|
||||
|
||||
### Core Components
|
||||
- **Ruby on Rails 8.x** with SQLite3
|
||||
- **Hotwire** (Turbo + Stimulus) for frontend
|
||||
- **Solid Queue** for background jobs
|
||||
- **Video.js** for video playback
|
||||
- **FFmpeg** for video processing
|
||||
- **Active Storage** for thumbnails/assets
|
||||
- **TailwindCSS** for styling
|
||||
|
||||
## Database Schema (Core Models)
|
||||
|
||||
### Work Model
|
||||
```ruby
|
||||
class Work < ApplicationRecord
|
||||
validates :title, presence: true
|
||||
|
||||
has_many :videos, dependent: :destroy
|
||||
has_many :external_ids, dependent: :destroy
|
||||
|
||||
scope :organized, -> { where(organized: true) }
|
||||
scope :unorganized, -> { where(organized: false) }
|
||||
|
||||
def primary_video
|
||||
videos.order(created_at: :desc).first
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
### MediaFile Model (Base Class)
|
||||
```ruby
|
||||
class MediaFile < ApplicationRecord
|
||||
# Base class for all media files using STI
|
||||
include Streamable, Processable
|
||||
|
||||
# Common associations
|
||||
belongs_to :work
|
||||
belongs_to :storage_location
|
||||
has_many :playback_sessions, dependent: :destroy
|
||||
|
||||
# Common metadata stores
|
||||
store :fingerprints, accessors: [:xxhash64, :md5, :oshash, :phash]
|
||||
store :media_metadata, accessors: [:duration, :codec, :bit_rate, :format]
|
||||
|
||||
# Common validations and methods
|
||||
validates :filename, presence: true
|
||||
validates :xxhash64, presence: true, uniqueness: { scope: :storage_location_id }
|
||||
|
||||
scope :web_compatible, -> { where(web_compatible: true) }
|
||||
scope :needs_transcoding, -> { where(web_compatible: false) }
|
||||
|
||||
def display_title
|
||||
work&.display_title || filename
|
||||
end
|
||||
|
||||
def full_file_path
|
||||
File.join(storage_location.path, filename)
|
||||
end
|
||||
|
||||
def format_duration
|
||||
# Duration formatting logic
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
### Video Model (Inherits from MediaFile)
|
||||
```ruby
|
||||
class Video < MediaFile
|
||||
# Video-specific associations
|
||||
has_many :video_assets, dependent: :destroy
|
||||
|
||||
# Video-specific metadata
|
||||
store :video_metadata, accessors: [:width, :height, :video_codec, :audio_codec, :frame_rate]
|
||||
|
||||
# Video-specific methods
|
||||
def resolution_label
|
||||
# Resolution-based quality labeling (SD, 720p, 1080p, 4K, etc.)
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
### StorageLocation Model
|
||||
```ruby
|
||||
class StorageLocation < ApplicationRecord
|
||||
has_many :videos, dependent: :destroy
|
||||
|
||||
validates :name, presence: true
|
||||
validates :path, presence: true, uniqueness: true
|
||||
validates :storage_type, presence: true, inclusion: { in: %w[local] }
|
||||
|
||||
validate :path_must_exist_and_be_readable
|
||||
|
||||
def accessible?
|
||||
File.exist?(path) && File.readable?(path)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def path_must_exist_and_be_readable
|
||||
errors.add(:path, "must exist and be readable") unless accessible?
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
## Storage Architecture (Local Only)
|
||||
|
||||
### Directory Structure
|
||||
```bash
|
||||
# User's original media directories (read-only references)
|
||||
/movies/action/Die Hard (1988).mkv
|
||||
/movies/anime/Your Name (2016).webm
|
||||
/movies/scifi/The Matrix (1999).avi
|
||||
|
||||
# Velour transcoded files (same directories, web-compatible)
|
||||
/movies/action/Die Hard (1988).web.mp4
|
||||
/movies/anime/Your Name (2016).web.mp4
|
||||
/movies/scifi/The Matrix (1999).web.mp4
|
||||
|
||||
# Velour managed directories
|
||||
./velour_data/
|
||||
├── assets/ # Active Storage for generated content
|
||||
│ └── thumbnails/ # Video screenshots
|
||||
└── tmp/ # Temporary processing files
|
||||
└── transcodes/ # Temporary transcode storage
|
||||
```
|
||||
|
||||
### Docker Volume Mounting with Heuristic Discovery
|
||||
|
||||
Users mount their video directories under `/videos` and Velour automatically discovers and categorizes them:
|
||||
|
||||
```yaml
|
||||
# docker-compose.yml
|
||||
volumes:
|
||||
- /path/to/user/movies:/videos/movies:ro
|
||||
- /path/to/user/tv_shows:/videos/tv:ro
|
||||
- /path/to/user/documentaries:/videos/docs:ro
|
||||
- /path/to/user/anime:/videos/anime:ro
|
||||
- ./velour_data:/app/velour_data
|
||||
```
|
||||
|
||||
### Automatic Storage Location Discovery
|
||||
```ruby
|
||||
# app/services/storage_discovery_service.rb
|
||||
class StorageDiscoveryService
|
||||
CATEGORIES = {
|
||||
'movies' => 'Movies',
|
||||
'tv' => 'TV Shows',
|
||||
'tv_shows' => 'TV Shows',
|
||||
'series' => 'TV Shows',
|
||||
'docs' => 'Documentaries',
|
||||
'documentaries' => 'Documentaries',
|
||||
'anime' => 'Anime',
|
||||
'cartoons' => 'Animation',
|
||||
'animation' => 'Animation',
|
||||
'sports' => 'Sports',
|
||||
'music' => 'Music Videos',
|
||||
'music_videos' => 'Music Videos',
|
||||
'kids' => 'Kids Content',
|
||||
'family' => 'Family Content'
|
||||
}.freeze
|
||||
|
||||
def self.discover_and_create
|
||||
base_path = '/videos'
|
||||
return [] unless Dir.exist?(base_path)
|
||||
|
||||
discovered = []
|
||||
|
||||
Dir.children(base_path).each do |subdir|
|
||||
dir_path = File.join(base_path, subdir)
|
||||
next unless Dir.exist?(dir_path)
|
||||
|
||||
category = categorize_directory(subdir)
|
||||
storage = StorageLocation.find_or_create_by!(
|
||||
name: "#{category}: #{subdir.titleize}",
|
||||
path: dir_path,
|
||||
storage_type: 'local'
|
||||
)
|
||||
|
||||
discovered << storage
|
||||
end
|
||||
|
||||
discovered
|
||||
end
|
||||
|
||||
def self.categorize_directory(dirname)
|
||||
downcase = dirname.downcase
|
||||
CATEGORIES[downcase] || CATEGORIES.find { |key, _| downcase.include?(key) }&.last || 'Other'
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
## Video Processing Pipeline
|
||||
|
||||
### Background Job with ActiveJob 8.1 Continuations
|
||||
```ruby
|
||||
class VideoProcessorJob < ApplicationJob
|
||||
include ActiveJob::Statuses
|
||||
|
||||
def perform(video_id)
|
||||
video = Video.find(video_id)
|
||||
|
||||
progress.update(stage: "metadata", total: 100, current: 0)
|
||||
metadata = VideoMetadataExtractor.new(video.full_file_path).extract
|
||||
video.update!(video_metadata: metadata)
|
||||
progress.update(stage: "metadata", total: 100, current: 100)
|
||||
|
||||
progress.update(stage: "thumbnail", total: 100, current: 0)
|
||||
generate_thumbnail(video)
|
||||
progress.update(stage: "thumbnail", total: 100, current: 100)
|
||||
|
||||
unless video.web_compatible?
|
||||
progress.update(stage: "transcode", total: 100, current: 0)
|
||||
transcode_video(video)
|
||||
progress.update(stage: "transcode", total: 100, current: 100)
|
||||
end
|
||||
|
||||
progress.update(stage: "complete", total: 100, current: 100)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def generate_thumbnail(video)
|
||||
# Generate thumbnail at 10% of duration
|
||||
thumbnail_path = VideoTranscoder.new.extract_frame(video.full_file_path, video.duration * 0.1)
|
||||
video.video_assets.create!(asset_type: "thumbnail", file: File.open(thumbnail_path))
|
||||
end
|
||||
|
||||
def transcode_video(video)
|
||||
transcoder = VideoTranscoder.new
|
||||
output_path = video.full_file_path.gsub(/\.[^.]+$/, '.web.mp4')
|
||||
|
||||
transcoder.transcode_for_web(
|
||||
input_path: video.full_file_path,
|
||||
output_path: output_path,
|
||||
on_progress: ->(current, total) { progress.update(current: current) }
|
||||
)
|
||||
|
||||
video.update!(
|
||||
transcoded_path: File.basename(output_path),
|
||||
transcoded_permanently: true,
|
||||
web_compatible: true
|
||||
)
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
## Frontend Architecture
|
||||
|
||||
### Uninterrupted Video Playback
|
||||
```erb
|
||||
<!-- videos/show.html.erb -->
|
||||
<turbo-frame id="video-player-frame" data-turbo-permanent>
|
||||
<div class="video-container">
|
||||
<video
|
||||
id="video-player"
|
||||
data-controller="video-player"
|
||||
data-video-player-video-id-value="<%= @video.id %>"
|
||||
data-video-player-start-position-value="<%= @last_position %>"
|
||||
class="w-full"
|
||||
controls>
|
||||
<source src="<%= stream_video_path(@video) %>" type="video/mp4">
|
||||
</video>
|
||||
</div>
|
||||
</turbo-frame>
|
||||
|
||||
<turbo-frame id="video-info">
|
||||
<h1><%= @work.title %></h1>
|
||||
<p>Duration: <%= format_duration(@video.duration) %></p>
|
||||
</turbo-frame>
|
||||
```
|
||||
|
||||
### Video Player Stimulus Controller
|
||||
```javascript
|
||||
// app/javascript/controllers/video_player_controller.js
|
||||
import { Controller } from "@hotwired/stimulus"
|
||||
|
||||
export default class extends Controller {
|
||||
static values = { videoId: Number, startPosition: Number }
|
||||
|
||||
connect() {
|
||||
this.player = videojs(this.element, {
|
||||
controls: true,
|
||||
responsive: true,
|
||||
fluid: true,
|
||||
playbackRates: [0.5, 1, 1.25, 1.5, 2]
|
||||
})
|
||||
|
||||
this.player.ready(() => {
|
||||
if (this.startPositionValue > 0) {
|
||||
this.player.currentTime(this.startPositionValue)
|
||||
}
|
||||
})
|
||||
|
||||
// Save position every 10 seconds
|
||||
this.interval = setInterval(() => {
|
||||
this.savePosition()
|
||||
}, 10000)
|
||||
|
||||
// Save on pause
|
||||
this.player.on("pause", () => this.savePosition())
|
||||
}
|
||||
|
||||
disconnect() {
|
||||
clearInterval(this.interval)
|
||||
this.player.dispose()
|
||||
}
|
||||
|
||||
savePosition() {
|
||||
const position = Math.floor(this.player.currentTime())
|
||||
|
||||
fetch(`/videos/${this.videoIdValue}/playback-position`, {
|
||||
method: "PATCH",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ position })
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Service Objects
|
||||
|
||||
### File Scanner Service
|
||||
```ruby
|
||||
class FileScannerService
|
||||
def initialize(storage_location)
|
||||
@storage_location = storage_location
|
||||
end
|
||||
|
||||
def scan
|
||||
return failure_result("Storage location not accessible") unless @storage_location.accessible?
|
||||
|
||||
video_files = find_video_files
|
||||
new_videos = process_files(video_files)
|
||||
|
||||
success_result(new_videos)
|
||||
rescue => e
|
||||
failure_result(e.message)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def find_video_files
|
||||
Dir.glob(File.join(@storage_location.path, "**", "*.{mp4,avi,mkv,mov,wmv,flv,webm,m4v}"))
|
||||
end
|
||||
|
||||
def process_files(file_paths)
|
||||
new_videos = []
|
||||
|
||||
file_paths.each do |file_path|
|
||||
filename = File.basename(file_path)
|
||||
|
||||
next if Video.exists?(filename: filename, storage_location: @storage_location)
|
||||
|
||||
video = Video.create!(
|
||||
filename: filename,
|
||||
storage_location: @storage_location,
|
||||
work: Work.find_or_create_by(title: extract_title(filename))
|
||||
)
|
||||
|
||||
new_videos << video
|
||||
VideoProcessorJob.perform_later(video.id)
|
||||
end
|
||||
|
||||
new_videos
|
||||
end
|
||||
|
||||
def extract_title(filename)
|
||||
# Simple title extraction - can be enhanced
|
||||
File.basename(filename, ".*").gsub(/[\[\(].*?[\]\)]/, "").strip
|
||||
end
|
||||
|
||||
def success_result(videos = [])
|
||||
{ success: true, videos: videos, message: "Found #{videos.length} new videos" }
|
||||
end
|
||||
|
||||
def failure_result(message)
|
||||
{ success: false, message: message }
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
### Video Transcoder Service
|
||||
```ruby
|
||||
class VideoTranscoder
|
||||
def transcode_for_web(input_path:, output_path:, on_progress: nil)
|
||||
movie = FFMPEG::Movie.new(input_path)
|
||||
|
||||
# Calculate progress callback
|
||||
progress_callback = ->(progress) {
|
||||
on_progress&.call(progress, 100)
|
||||
}
|
||||
|
||||
movie.transcode(output_path, {
|
||||
video_codec: "libx264",
|
||||
audio_codec: "aac",
|
||||
custom: %w[
|
||||
-pix_fmt yuv420p
|
||||
-preset medium
|
||||
-crf 23
|
||||
-movflags +faststart
|
||||
-tune fastdecode
|
||||
]
|
||||
}, &progress_callback)
|
||||
end
|
||||
|
||||
def extract_frame(input_path, seconds)
|
||||
movie = FFMPEG::Movie.new(input_path)
|
||||
output_path = "#{Rails.root}/tmp/thumbnail_#{SecureRandom.hex(8)}.jpg"
|
||||
|
||||
movie.screenshot(output_path, seek_time: seconds, resolution: "320x240")
|
||||
output_path
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
## Routes
|
||||
|
||||
```ruby
|
||||
Rails.application.routes.draw do
|
||||
root "storage_locations#index"
|
||||
|
||||
resources :storage_locations, only: [:index, :show, :create, :destroy] do
|
||||
member do
|
||||
post :scan
|
||||
end
|
||||
end
|
||||
|
||||
resources :works, only: [:index, :show] do
|
||||
resources :videos, only: [:show]
|
||||
end
|
||||
|
||||
resources :videos, only: [] do
|
||||
member do
|
||||
get :stream
|
||||
patch :playback_position
|
||||
post :retry_processing
|
||||
end
|
||||
|
||||
resources :playback_sessions, only: [:create]
|
||||
end
|
||||
|
||||
# Real-time job progress
|
||||
resources :jobs, only: [:show] do
|
||||
member do
|
||||
get :progress
|
||||
end
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
## Implementation Sub-Phases
|
||||
|
||||
### Phase 1A: Core Foundation (Week 1-2)
|
||||
1. Generate models with migrations
|
||||
2. Implement model validations and associations
|
||||
3. Create storage adapter pattern
|
||||
4. Create FileScannerService
|
||||
5. Basic UI for storage locations and video listing
|
||||
|
||||
### Phase 1B: Video Playback (Week 3)
|
||||
1. Video streaming controller with byte-range support
|
||||
2. Video.js integration with Stimulus controller
|
||||
3. Playback session tracking
|
||||
4. Resume functionality
|
||||
|
||||
### Phase 1C: Processing Pipeline (Week 4)
|
||||
1. Video metadata extraction with FFmpeg
|
||||
2. Background job processing with progress tracking
|
||||
3. Thumbnail generation and storage
|
||||
4. Processing UI with status indicators
|
||||
|
||||
### Phase 1D: Works & Grouping (Week 5)
|
||||
1. Duplicate detection by file hash
|
||||
2. Manual grouping interface
|
||||
3. Works display with version selection
|
||||
4. Search, filtering, and pagination
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
```bash
|
||||
# Required
|
||||
RAILS_ENV=development
|
||||
RAILS_MASTER_KEY=your_master_key
|
||||
|
||||
# Video processing
|
||||
FFMPEG_PATH=/usr/bin/ffmpeg
|
||||
FFPROBE_PATH=/usr/bin/ffprobe
|
||||
|
||||
# Storage
|
||||
VIDEOS_PATH=./velour_data/videos
|
||||
MAX_TRANSCODE_SIZE_GB=50
|
||||
|
||||
# Background jobs (SolidQueue runs with defaults)
|
||||
SOLID_QUEUE_PROCESSES="*:2" # 2 workers for all queues
|
||||
```
|
||||
|
||||
### Docker Configuration
|
||||
```dockerfile
|
||||
FROM ruby:3.3-alpine
|
||||
|
||||
# Install FFmpeg and other dependencies
|
||||
RUN apk add --no-cache \
|
||||
ffmpeg \
|
||||
imagemagick \
|
||||
sqlite-dev \
|
||||
nodejs \
|
||||
npm
|
||||
|
||||
# Install Rails and other gems
|
||||
COPY Gemfile* /app/
|
||||
RUN bundle install
|
||||
|
||||
# Copy application code
|
||||
COPY . /app/
|
||||
|
||||
# Create directories
|
||||
RUN mkdir -p /app/velour_data/{assets,tmp}
|
||||
|
||||
EXPOSE 3000
|
||||
CMD ["rails", "server", "-b", "0.0.0.0"]
|
||||
```
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Model Tests
|
||||
- Model validations and associations
|
||||
- Scopes and class methods
|
||||
- JSON store accessor functionality
|
||||
|
||||
### Service Tests
|
||||
- FileScannerService with sample directory
|
||||
- VideoTranscoder service methods
|
||||
- Background job processing
|
||||
|
||||
### System Tests
|
||||
- Video playback functionality
|
||||
- File scanning workflow
|
||||
- Work grouping interface
|
||||
|
||||
This Phase 1 delivers a complete, useful video library application that provides immediate value while establishing the foundation for future enhancements.
|
||||
Reference in New Issue
Block a user