wathiede/aocsync
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Start rewrite in python w/ cursor

Browse Source
This commit is contained in:
Bill Thiede 2025-12-02 15:30:46 -08:00
parent c3b4638d5f
commit 6960863b24
11 changed files with 1204 additions and 3344 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
.gitignore vendored
View File

@ -1 +1,9 @@
/target
# AOC Sync generated files
/data/
/output/
/repos/
*.db
*.db-journal
__pycache__

3028
Cargo.lock generated
View File

File diff suppressed because it is too large Load Diff

16
Cargo.toml
View File

@ -1,16 +0,0 @@
[package]
name = "aocsync"
version = "0.1.0"
edition = "2021"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
rocket = "0.5.0-rc.2"
glog = {version = "0.1.0", git = "https://github.com/wathiede/glog-rs"}
log = "0.4.17"
serde = { version = "1.0.148", features = ["serde_derive"] }
rocket_anyhow = "0.1.1"
thiserror = "1.0.37"
rocket_dyn_templates = { version = "0.1.0-rc.2", features = ["tera"] }
anyhow = "1.0.66"

166
README.md
View File

@ -1,17 +1,155 @@
# `git-sync` webhook
# AOC Sync
```
POST /-/reload?repo=akramer HTTP/1.1
Host: aocperf.z.xinu.tv
X-Real-IP: 172.17.0.7
X-Forwarded-For: 172.17.0.7
X-Forwarded-Proto: https
X-Forwarded-Host: aocperf.z.xinu.tv
X-Forwarded-Server: aocperf.z.xinu.tv
Connection: close
Content-Length: 0
gitsync-hash: 6f498804bf7203ab69fa7aa8e9f441d20d07ac67
accept-encoding: gzip
user-agent: Go-http-client/2.0
A Python script that polls multiple git repositories containing Advent of Code implementations written in Rust using `cargo-aoc` format. It automatically updates repositories when changes are detected, runs benchmarks, and generates a beautiful HTML comparison page showing performance metrics across users, years, and days.
## Features
- **Automatic Git Polling**: Monitors multiple repositories for changes
- **Flexible Repository Structure**: Supports both single-repo (all years) and multi-repo (one per year) configurations
- **Automatic Benchmarking**: Runs `cargo aoc bench` for all implemented days
- **Performance Parsing**: Extracts timing data from cargo-aoc output
- **Data Storage**: SQLite database for historical performance data
- **HTML Reports**: Beautiful, responsive HTML comparison pages
- **Gap Handling**: Gracefully handles missing years, days, and parts
- **Configurable Comparisons**: Filter by specific years and days
## Requirements
- Python 3.7+
- Git
- Rust and Cargo
- `cargo-aoc` installed (install with `cargo install cargo-aoc`)
## Installation
1. Clone this repository:
```bash
git clone <repository-url>
cd aocsync
```
2. Install Python dependencies:
```bash
pip install -r requirements.txt
```
3. Ensure `cargo-aoc` is installed:
```bash
cargo install cargo-aoc
```
## Configuration
Edit `config.yaml` to configure repositories to monitor:
```yaml
# Poll interval in seconds
poll_interval: 300
# Output directory for generated HTML
output_dir: "output"
# Data storage directory
data_dir: "data"
# Repositories to monitor
repositories:
# Single repository with all years
- name: "user1"
url: "https://github.com/user1/advent-of-code"
type: "single"
local_path: "repos/user1"
# Multiple repositories, one per year
- name: "user2"
type: "multi-year"
years:
- year: 2023
url: "https://github.com/user2/aoc-2023"
local_path: "repos/user2-2023"
- year: 2024
url: "https://github.com/user2/aoc-2024"
local_path: "repos/user2-2024"
# Optional: Filter specific years to compare
compare_years: [2023, 2024]
# Optional: Filter specific days to compare
# compare_days: [1, 2, 3, 4, 5]
```
## Usage
### Run Once
To sync repositories and generate a report once:
```bash
python aocsync.py --once
```
### Continuous Polling
To continuously poll repositories (default):
```bash
python aocsync.py
```
Or with a custom config file:
```bash
python aocsync.py --config myconfig.yaml
```
## Output
- **Database**: Performance data is stored in `data/results.db` (SQLite)
- **HTML Report**: Generated at `output/index.html` (configurable via `output_dir`)
The HTML report includes:
- Performance comparison tables for each year/day/part
- Visual highlighting of fastest and slowest implementations
- Relative speed comparisons (X times faster/slower)
- Responsive design for viewing on any device
## How It Works
1. **Git Polling**: Checks each configured repository for changes by comparing local and remote commits
2. **Repository Update**: Clones new repositories or updates existing ones when changes are detected
3. **Day Detection**: Automatically finds implemented days by scanning for `day*.rs` files and Cargo.toml entries
4. **Benchmarking**: Runs `cargo aoc bench --day X` for each implemented day
5. **Parsing**: Extracts timing data from cargo-aoc output (handles nanoseconds, microseconds, milliseconds)
6. **Storage**: Stores results in SQLite database with timestamps
7. **Report Generation**: Generates HTML comparison page showing latest results
## Repository Structure Detection
The script automatically detects:
- **Year**: From repository path, name, or Cargo.toml
- **Days**: From `src/bin/day*.rs`, `src/day*.rs`, or Cargo.toml entries
- **Parts**: From cargo-aoc benchmark output (Part 1, Part 2)
## Troubleshooting
### Cargo-aoc not found
Ensure `cargo-aoc` is installed and in your PATH:
```bash
cargo install cargo-aoc
```
### Git authentication issues
For private repositories, ensure your git credentials are configured or use SSH URLs.
### Benchmark timeouts
If benchmarks take too long, the script has a 5-minute timeout per day. Adjust in the code if needed.
### Missing performance data
If some users/days/parts don't show up:
- Check that `cargo aoc bench` runs successfully in the repository
- Verify the repository structure matches cargo-aoc conventions
- Check logs for parsing errors
## License
[Your License Here]

14
Rocket.toml
View File

@ -1,14 +0,0 @@
[release]
address = "0.0.0.0"
port = 9346
[debug]
address = "0.0.0.0"
port = 9346
# Uncomment to make it production like.
#log_level = "critical"
build_root = "/tmp/aocsync/"
www_root = "/tmp/aocsync-serve/"
repos = { akramer = { url = "https://github.com/akramer/aoc2022", name = "aoc2022", branch = "main" }, ggriffiniii = { url = "https://github.com/ggriffiniii/aoc2022", name = "aoc2022", branch = "main" }, wathiede = { url = "https://github.com/wathiede/advent", name = "advent/2022", branch = "master", first_commit = "bc3feabb4669b7ad8167398a4d37eefe68ea0565", subdir = "2022" } }

1007
aocsync.py Executable file
View File

File diff suppressed because it is too large Load Diff

36
config.yaml Normal file
View File

@ -0,0 +1,36 @@
# Configuration for AOC Sync
# Poll interval in seconds
poll_interval: 300
# Output directory for generated HTML
output_dir: "output"
# Data storage
data_dir: "data"
# Repositories to monitor
repositories:
# Example: Single repository with all years
- name: "wathiede"
url: "https://github.com/wathiede/advent"
type: "single" # single repo for all years
local_path: "repos/wathiede"
# Optional: specify years if auto-detection fails
# years: [2023, 2024]
# Example: Multiple repositories, one per year
- name: "ggriffiniii"
type: "multi-year" # one repo per year
years:
- year: 2025
url: "https://github.com/ggriffiniii/aoc2025"
local_path: "repos/ggriffiniii-2025"
#- year: 2024
# url: "https://github.com/user2/aoc-2024"
# local_path: "repos/user2-2024"
# Years to compare (optional, if not specified, all years found will be used)
compare_years: [2025]
# Days to compare (optional, if not specified, all days found will be used)
# compare_days: [1, 2, 3, 4, 5]

1
requirements.txt Normal file
View File

@ -0,0 +1 @@
PyYAML>=6.0

34
src/bin/aocpoller.rs
View File

@ -1,34 +0,0 @@
use std::process::Command;
use aocsync::{logging_run, Config};
use glog::Flags;
use log::info;
fn main() -> anyhow::Result<()> {
glog::new()
.init(Flags {
colorlogtostderr: true,
//alsologtostderr: true, // use logtostderr to only write to stderr and not to files
logtostderr: true,
..Default::default()
})
.unwrap();
let config = rocket::Config::figment()
.extract::<Config>()
.expect("Couldn't parse config");
info!("{:#?}", config);
for (name, repo) in config.repos.iter() {
let ts = logging_run(
Command::new("git")
.arg("ls-remote")
.arg(&repo.url)
.arg("-q")
.arg(&repo.name)
.arg("-h")
.arg(&repo.branch),
)?;
let out = String::from_utf8_lossy(&ts.output.stdout);
let head = &out[..out.find('\t').expect("No tab in output")];
info!("HEAD: '{}'", head);
}
Ok(())
}

234
src/lib.rs
View File

@ -1,234 +0,0 @@
use std::{
collections::HashMap,
fs,
fs::File,
io,
path::{Path, PathBuf},
process::{Command, Output},
time::{Duration, Instant},
};
use log::{error, info};
use rocket::response::Responder;
use serde::Deserialize;
use thiserror::Error;
#[derive(Error, Debug, Responder)]
pub enum SyncError {
#[error("IO error")]
IoError(#[from] io::Error),
}
#[derive(Debug, Deserialize)]
pub struct Config {
pub build_root: PathBuf,
pub www_root: PathBuf,
pub repos: HashMap<String, Repo>,
}
#[derive(Debug, Deserialize)]
pub struct Repo {
pub name: String,
pub url: String,
pub branch: String,
pub first_commit: Option<String>,
pub subdir: Option<String>,
}
pub fn indent_output(bytes: &[u8]) -> String {
let s = String::from_utf8_lossy(bytes);
let s = if s.is_empty() { "<NO OUTPUT>" } else { &s };
s.split('\n')
.map(|l| format!("\t{l}"))
.collect::<Vec<_>>()
.join("\n")
}
pub fn format_task_status(ts: &TaskStatus) -> String {
format!(
"Status {} ({}) {}s:\nStdout:\n{}\nStderr:\n{}",
ts.command,
ts.output.status,
ts.duration.as_secs_f32(),
indent_output(&ts.output.stdout),
indent_output(&ts.output.stderr)
)
}
pub struct TaskStatus {
pub command: String,
pub output: Output,
pub duration: Duration,
}
pub fn logging_run(cmd: &mut Command) -> Result<TaskStatus, SyncError> {
info!(
"{}$ {} {}",
cmd.get_current_dir()
.unwrap_or(Path::new("<NO CWD>"))
.display(),
cmd.get_program().to_string_lossy(),
cmd.get_args()
.map(|a| a.to_string_lossy())
.collect::<Vec<_>>()
.join(" "),
);
let start = Instant::now();
let o = cmd.output()?;
let ts = TaskStatus {
output: o,
command: format!("{:?}", cmd),
duration: start.elapsed(),
};
info!("{}", format_task_status(&ts));
Ok(ts)
}
pub fn git_checkout(build_path: &Path, rev: &str) -> Result<TaskStatus, SyncError> {
logging_run(
Command::new("git")
.current_dir(&build_path)
.arg("checkout")
.arg("-q")
.arg(rev),
)
}
// From https://stackoverflow.com/questions/26958489/how-to-copy-a-folder-recursively-in-rust
pub fn copy_dir_all(src: impl AsRef<Path>, dst: impl AsRef<Path>) -> io::Result<()> {
fs::create_dir_all(&dst)?;
for entry in fs::read_dir(src)? {
let entry = entry?;
let ty = entry.file_type()?;
if ty.is_dir() {
copy_dir_all(entry.path(), dst.as_ref().join(entry.file_name()))?;
} else {
fs::copy(entry.path(), dst.as_ref().join(entry.file_name()))?;
}
}
Ok(())
}
fn bench_at_commit(
commit: &str,
build_path: &Path,
target_path: &Path,
) -> Result<Vec<TaskStatus>, SyncError> {
let mut output = Vec::new();
output.push(git_checkout(build_path, commit)?);
// Run `cargo aoc bench`
output.push(logging_run(
Command::new("cargo")
.env("CARGO_TARGET_DIR", target_path)
.current_dir(&build_path)
.arg("aoc")
.arg("bench"),
)?);
output.push(git_checkout(build_path, "HEAD")?);
Ok(output)
}
pub fn reload(config: &Config, name: &str) -> Result<String, SyncError> {
let repo = &config.repos[name];
info!("Need to reload '{}': {:?}\n{:#?}", name, config, repo);
let commits_root = config.build_root.join("commits");
let git_root = config.build_root.join("git").join(name);
let www_root = config.www_root.join(name);
let target_root = config.build_root.join("target");
let checkout_path = git_root.join(&repo.name);
let build_path = if let Some(subdir) = &repo.subdir {
git_root.join(&repo.name).join(subdir)
} else {
git_root.join(&repo.name)
};
let target_path = target_root.join(name);
dbg!(
&build_path,
&target_path,
&commits_root,
&git_root,
&www_root
);
if !commits_root.exists() {
info!("Creating {}", commits_root.display());
fs::create_dir_all(&commits_root)?;
}
if !git_root.exists() {
info!("Creating {}", git_root.display());
fs::create_dir_all(&git_root)?;
}
if !target_root.exists() {
info!("Creating {}", target_root.display());
fs::create_dir_all(&target_root)?;
}
if !www_root.exists() {
info!("Creating {}", www_root.display());
fs::create_dir_all(&www_root)?;
}
let mut output = vec![logging_run(Command::new("git").arg("version"))?];
let needs_clone = !checkout_path.exists();
if needs_clone {
output.push(logging_run(
Command::new("git")
.current_dir(&git_root)
.arg("clone")
.arg(&repo.url)
.arg(&checkout_path),
)?);
}
output.push(logging_run(
Command::new("git")
.current_dir(&checkout_path)
.arg("checkout")
.arg("-f")
.arg(&repo.branch),
)?);
output.push(logging_run(
Command::new("git")
.current_dir(&checkout_path)
.arg("checkout")
.arg("HEAD"),
)?);
dbg!(&checkout_path);
// Make sure buildable clone is up to date
output.push(logging_run(
Command::new("git").current_dir(&checkout_path).arg("pull"),
)?);
let commits = logging_run(Command::new("git").current_dir(&checkout_path).args([
"log",
"--format=%H",
&repo.branch,
]))?;
let binding = String::from_utf8_lossy(&commits.output.stdout).into_owned();
let mut unknown_commits: Vec<_> = binding
.lines()
.filter(|commit| !commits_root.join(commit).exists())
.collect();
unknown_commits.reverse();
output.push(commits);
info!("Need to bench commits: {:?}", unknown_commits);
for commit in unknown_commits {
match bench_at_commit(commit, &build_path, &target_path) {
Ok(outputs) => {
output.extend(outputs);
File::create(commits_root.join(commit))?;
}
Err(err) => error!("Failed to bench {}@{}: {}", name, commit, err),
}
}
// Copy files from `target/` to serving directory
let bench_path = target_path.join("criterion");
info!("Copying {} -> {}", bench_path.display(), www_root.display());
copy_dir_all(bench_path, www_root)?;
let response = output
.iter()
.map(|ts| format!("{}", format_task_status(ts)))
.collect::<Vec<_>>()
.join("\n");
Ok(response)
}

4
templates/index.html.tera
View File

@ -1,4 +0,0 @@
{% for d in dirs %}
<div><a href="/results/{{ d }}/report/">{{ d }}</a></div>
{% endfor %}
<div>Source in <a href="https://git.z.xinu.tv/wathiede/aocsync">git.</a>