Concurrent Build Scheduler
A build system, starting from its thread pool.
- Kind
- coursework
- Context
- Collaborative university project
- Domain
- SYSTEMS
Optional. The homepage explains this project with a live dependency graph, so a screenshot is only needed for the detail page. Terminal output of a run with timings would work.
/projects/concurrent-build-scheduler/run.png
Overview
Tasks are declared in a JSON dependency graph. The scheduler works out what can run now, runs those tasks in parallel, and unlocks the next ones as their dependencies complete.
The thread pool underneath is written by hand — worker threads, futures, locks and condition variables — because the point of the project was to build the scheduling machinery rather than call it.
Execution
- JSON graph
- Scheduler
- Thread pool
- Task states
Technical challenge
Correct parallel scheduling is mostly about states and waiting. A task can be pending, ready, running, complete or cancelled, and the transition has to be atomic with respect to every worker asking "is there anything I can do".
Workers block on condition variables rather than polling, cancellation has to propagate without leaving a worker waiting on a task that will never finish, and resource limits mean a task that is ready is not always allowed to start.
What it does
- 01JSON-defined task dependency graph
- 02Executes shell and Python tasks
- 03Hand-written thread pool with worker threads
- 04Futures for task results
- 05Locks and condition variables for coordination
- 06Explicit task state management
- 07Task cancellation
- 08CPU and RAM resource allocation per task
- 09Parallel execution of independent tasks
Technologies
- Language
Python
- Concurrency
Thread pool
Worker threads
Futures
Locks
Condition variables
Cancellation
- Scheduling
Dependency graph
Task state machine
Resource allocation