Feeding from Outside

An engineer at a coastal observatory sorts a small selection of outside signals.

Outside signals, selected for a closer look. Conceptual illustration.

I couldn’t watch everything

I wanted the radar to help me keep up with useful material outside Rootweaver. Follow the YouTube channels I had selected, flag something worth a look, get the transcript, and work out how it connected to what I was building. Then give me a relevant summary I could read.

The useful part would happen when I read it. A comparison with Rootweaver might suggest a feature, challenge an assumption, or give me an experiment to try. I was looking for an ember that gets the neurons firing. A summary did not need to hand me a finished answer to be useful.

I could watch an individual video. I couldn’t watch everything those channels were putting out, alongside building the platform. Having agents help condense the material made sense, but first I needed to get it into text they could work with. A video title tells you very little about the argument behind it.

That was the problem I was working through over Sunday 14 and Monday 15 June. There was plenty of content. Getting enough of it into usable context was much less straightforward. Caption requests were being blocked, attempts to work around that changed what the Mac was doing, and even the small scripts connecting discovery to fetching had a mismatch to fix.

The research loop I wanted; acquisition was the difficult middle step.

Finding a video was easier than getting its words

The radar was already following 11 YouTube channels. Their Atom feeds supplied metadata such as titles and video identifiers, so finding new uploads was relatively straightforward. The caption body came from a different request. Successfully discovering a video did not mean I could fetch its transcript.

The cluster-side caption enrichment was blocked. The practical approach was to keep the radar finding things and do the transcript work on the Mac. A local script could narrow the list, skip videos already represented in the vault, and leave me a batch to work through.

That narrowed the immediate job to something concrete: could I turn a selected video into text without a failed request turning into a much bigger workload? Until I could do that reliably enough for a small research run, there was little point in polishing the summaries at the other end.

A caption failure turned into audio processing

Our Rootweaver wrapper, yt_fetch.py, tried captions through youtube-transcript-api, a third-party Python library, then subtitles through yt-dlp. If both failed, the old fallback used mlx-whisper, an MLX implementation of OpenAI’s Whisper, to transcribe downloaded audio. FFmpeg handled the audio conversion. Those projects and their maintainers supplied the extraction and transcription capabilities; our code connected them to the radar and controlled the attempts and batches.

I had asked for a transcript and ended up with a different job on the Mac. The incident record describes FFmpeg at roughly 120% CPU during the overheating problem. The caption block had not merely stopped the fetch: the fallback had turned it into sustained audio processing.

An engineer considers an idle audio-reel machine beside two text trays.

Creating text from audio was a different job from fetching existing captions. Conceptual illustration.

The transcript hardening work, VW-816 — yt-ingest: sever auto-whisper + paced batch mode, made Whisper an explicit choice. The agent-assisted fix kept the ordinary path to captions and subtitles; if neither worked, it failed. Audio transcription required --whisper.

The retained implementation makes that change small and visible. The flag selects the attempt list before any of those functions run.

Historical source: yt_fetch.py, get_transcript(), revision 31f447d.

    attempts = (
        [("whisper", transcript_via_whisper)] if force_whisper else [
            ("captions", captions_via_api),
            ("yt-dlp-subs", captions_via_ytdlp),
        ]
    )

Explain this code line by line

Line 1 starts assigning the attempt list.
Line 2 chooses a single Whisper callable when the explicit flag is true; otherwise it begins the ordinary list.
Line 3 pairs the caption label with its callable.
Line 4 adds the subtitle-extraction callable as the second ordinary attempt.
Line 5 closes the ordinary list.
Line 6 closes the conditional expression. These names refer to functions; this expression itself does not fetch or transcribe anything.

I still had the option to transcribe a video without captions. I could decide whether that particular video was worth the extra work, instead of finding out after the machine had started it.

Slowing the requests wasn’t enough

The other problem was getting blocked partway through fetching. The Sunday record describes roughly 90 unthrottled caption requests in under a minute, followed by a block that was still present four and a half hours later.

Pacing was an obvious thing to investigate. The batch tool was changed to fetch serially, with a default 15-second delay plus jitter and a longer pause after each group of ten. That was a better way to run a small job than firing off a burst. It still left the question of whether slowing down would let the batch keep going.

Two tests on fresh cellular paths gave a fairly unhelpful answer:

June 14 test	Spacing	Successful fetches before the block	Elapsed time
First	Reduced from 5 seconds to 0.3 seconds	26	110 seconds
Second	Fixed at 2 seconds	25	85 seconds

A slower fixed spacing had not bought much more room in those runs. I couldn’t treat the delay as a sustainable throughput setting and leave a large list running.

I also investigated the network path. The fixed-line router did not provide a useful IP-rotation control, and rebooting it was not a dependable way to get a different address. The mining record reports a more useful result: after a broadband block, a cellular airplane-mode toggle was followed by seven successful caption fetches out of seven attempts.

That helped for that run, but it didn’t remove the limit. By Monday, the handover described blocks at the eleventh attempt across three cellular paths, rather than the roughly 25 successes seen in Sunday’s two probes. The working batch size had to become more conservative. These were small observations from my research workflow, not a quota I could count on every time I changed connections.

The important code change was to stop when the tool recognized that block. get_transcript() re-raised RateLimited instead of swallowing it as another caption failure. The batch loop caught it, recorded the result, and broke out.

Historical source: yt_fetch.py, run_batch(), revision 31f447d.

        except RateLimited as e:
            log(f"RATE-LIMITED at {vid}: {e}\n"
                f"  Stopping batch after {len(ok)} ok — back off 30+ min before retrying.")
            rate_limited = True
            break

Explain this code line by line

Line 1 handles the recognized rate-limit exception from the active video.
Lines 2–3 log the identifier and completed count. The historical “30+ min” advice is a diagnostic string, not a verified refill guarantee.
Line 4 records that the run stopped for this reason.
Line 5 exits the current batch loop. After the loop the function returns 2 for this condition; it returns 0 otherwise, including runs that skipped ordinary failures.

The tool wrote successful transcripts as it went. A blocked run could therefore leave useful text behind. I could inspect what had landed and rebuild the remaining list, instead of treating the batch as either completely successful or completely wasted.

There was a documentation lesson in this too. The retained function still had a comment describing a safe hourly ceiling after the investigation had moved away from that explanation. Changing how I ran the tool was not enough if the next session could pick up the old assumption from its source comments.

I considered buying help with the acquisition problem too. The content-acquisition vendor review (VW-817 — AI radar content-acquisition: vendor ranking + decision) ended with a decision to defer purchasing and keep using the small Mac-side workflow. For the research I was doing, getting selected transcripts and useful notes was the immediate need. Buying another service was not yet the next step.

Make the temporary bridge repeatable

The first transcript-mining run, VW-796 — Radar YouTube mining run: convergent system gaps from 18 talks, had used a temporary scope_channels.py script. It fetched the channel feeds, checked what was already in the vault, and filtered the remaining titles. Once that was useful, I wanted a repeatable command rather than a script left in a temporary directory.

The work to promote that script into the transcript-ingestion skill, VW-821 — Promote scope_channels.py → /yt-ingest + ADR-076, was recorded in ADR-076, the Radar-to-Mac YouTube Transcript-Extraction Lane decision. The bridge and the fetcher stayed separate. I could prepare a list of useful-looking videos without immediately making all the caption requests.

One detail was where the channel list came from. Copying it into the script would work until I changed the radar’s feeds and forgot to update the Mac. Requiring a live cluster read every time would make the local script less useful when I was offline.

The implementation tried the radar’s live configuration first and kept a vendored list as a fallback. Both paths returned the same dictionary shape, so the rest of the script could carry on with whichever source was available.

Historical source: scope_channels.py, channels(), revision 31f447d.

def channels() -> dict:
    live = live_channels()
    if live:
        print(f"# channel source: LIVE configmap ({len(live)} channels)", file=sys.stderr)
        return live
    print(f"# channel source: VENDORED fallback ({len(VENDORED)} channels) — "
          f"desktop unreachable", file=sys.stderr)
    return VENDORED

Explain this code line by line

Line 1 declares a function returning a dictionary.
Line 2 calls the live configuration reader.
Line 3 tests whether that reader returned a nonempty dictionary.
Line 4 reports the live path and its channel count on stderr.
Line 5 returns the live dictionary immediately.
Lines 6–7 report the vendored fallback; this diagnostic does not distinguish every cause of the failed live read.
Line 8 returns the retained fallback dictionary. It does not merge lists or initiate extraction.

The diagnostic told me which list I was using. The fallback made the tool usable without that live read, at the cost of possibly following an older channel list.

The local relevance filter was deliberately simple. It lowercased each title, rejected configured noise terms, and counted the signal terms it contained. That helped prioritize the videos to fetch without adding another model call to this script. It wasn’t the part that compared a talk’s argument with Rootweaver: it hadn’t seen the transcript yet.

Monday found a smaller bug

Once the scripts were connected, the queue format became a problem. The producer wrote readable lines containing a video identifier followed by the channel and title as a comment. The fetcher wasn’t handling that format correctly.

The fix for annotated queue lines breaking batch ingestion, VW-823 — yt_fetch.py --batch crash on comment-format queue, changed the consumer to take the first whitespace token. It also ignored empty lines and full-line comments. The identifier could then go through the existing video_id_from() parser without dragging the annotation along with it.

Historical source: yt_fetch.py, main() batch branch, revision 31f447d. Related work: VW-823 — yt_fetch.py --batch crash on comment-format queue.

            # First whitespace token only — tolerates "VIDEOID  # channel | title"
            # lines from scope_channels.py mining-queue.txt (VW-823).
            ids = [video_id_from(ln.split()[0]) for ln in src
                   if ln.strip() and not ln.lstrip().startswith("#")]

Explain this code line by line

Lines 1–2 document the producer’s annotated-line format and the fixing task.
Line 3 takes the first whitespace token from each accepted line, then calls the video identifier parser.
Line 4 excludes blank lines and lines whose first non-space character is a comment marker. The filter is evaluated before the expression, so a blank line is never indexed.
The list comprehension does not validate an arbitrary first token by itself. That validation remains in video_id_from().

This was a small fix compared with the caption blocks, but both scripts needed to agree before the research run could continue. The Monday handover records the parser fix as proven live and two further rounds that filed 20 transcripts.

Re-running the scoping script checked the vault again and rebuilt the list. That was a practical way to continue after interruptions: videos already represented in filed notes could be left out of the next queue.

The notes were there to start a comparison

Fetching text was only the first part of what I wanted. The fetcher’s output was a skeleton containing the video’s metadata and transcript. Analysis and filing came afterwards. The useful output for me was a condensed account of the talk, with enough context to see why it might matter to Rootweaver.

The first mining synthesis collected findings from 18 filed notes. On long-context behavior, it cited a context-rot discussion from Yannic Kilcher’s channel, alongside material on hierarchical planning and the problems with adding more context. Several talks pointed toward similar questions:

What the material raised	What it made worth checking in Rootweaver
Long research runs can keep accumulating context	Where should a running research agent compact, stop, or make a decision?
Memory can retain an old belief after the evidence changes	Is retrieval surfacing something current, or just something remembered?
A clean evaluation can hide operational failure	What happens with distractors, state, and a different order of tool calls?

Those comparisons were the reason to build the radar in the first place. Reading a relevant account of the material could reveal a connection I might otherwise have missed.

The retained work also linked the mining to the semantic code-index backlog item (VW-810 — Semantic code index), and the graph-embedding backlog item (VW-811 — Graph embeddings). They were possible directions coming out of the research, rather than completed features. A talk could give me a reason to investigate one of those ideas; the platform work still had to stand up on its own.

By Monday, the handover put the cumulative collection at 58 radar-mined notes, with more candidates still waiting. I had text and notes to read, and specific questions to take back to the platform.

An engineer carries a small set of sheets from the observatory to an archive with separate sorting niches.

Bring the useful material back into the work. Conceptual illustration.

What I learned from building it

I would start with the actual obstacle sooner: how to get useful video content into text I could work with. Discovering titles, choosing candidates, fetching transcripts, and summarizing the material were different steps. A working feed at the beginning didn’t tell me much about whether I could finish the research run.

I would also test the behavior of the fetches before designing around a throughput assumption. Pacing made the tool easier to run, but the small experiments showed that spacing alone wasn’t going to keep a long batch alive. The network changes helped in recorded runs, while later blocks made the limits less predictable than the first results suggested.

The fallback change was worth keeping explicit. If I wanted audio transcription, I could choose it. A failed caption request shouldn’t make that choice for me.

Finally, the relevance of the summary mattered more than the size of the collection. The simple title filter helped choose where to spend fetching effort. The richer comparison needed the transcript and a useful account of how it related to Rootweaver. I still wanted to read that account myself and decide whether there was an ember worth following into a feature, an idea, or an experiment.

Tools and source projects

Original repositories, documentation and model

youtube-transcript-api — the third-party caption library used by the first attempt. Credit belongs to the project’s maintainers and contributors.
yt-dlp — subtitle extraction, metadata and audio download in the wrapper. Its repository includes usage documentation.
mlx-whisper in MLX Examples — the local transcription implementation, based on OpenAI’s Whisper. The retained wrapper configured the MLX Community whisper-large-v3-turbo model.
FFmpeg — the media project doing the audio conversion invoked through yt-dlp.

Rootweaver’s yt_fetch.py and scope_channels.py added the radar connection, local selection, queue parsing, attempt policy and batch handling around these upstream capabilities.

Next in the journey

The next part turns inward: after bringing useful material into the system, how should it look after the memory it already has? The retained next draft follows that memory-curation work.

Glossary

Terms used in this episode

Atom: An XML format for syndicated entries. Here it supplies video metadata for discovery, rather than the caption body. Atom specification.
Captions: Timed text associated with a video. The official download method documents authorization and permission to edit that video. YouTube caption download API.
Automatic speech recognition (ASR): Creating text from audio. In this retained script, local Whisper transcription is a separate opt-in workload. OpenAI Whisper source.
Exception: A Python signal that interrupts ordinary execution until a matching handler deals with it. Here recognized rate limits are re-raised to the batch caller. Python errors and exceptions.
Opt-in flag: A command-line choice that explicitly enables a behavior. This script uses its Whisper flag to choose the audio transcription path. Python command-line parsing.
Carrier-grade NAT: Address sharing by a service provider can place multiple customers behind shared public addressing. It is context for the hypothesis, not a proven cause of these blocks. Shared address space specification.
Queue, as used here: A text file listing videos to fetch, with channel and title comments. The batch script reads its video identifiers.
Title gate, as used here: A cheap substring-based relevance filter over titles. It helps pick what to fetch before the transcript is available.