Computer Vision & Utilities
What if your cameras and dash cams could do the tedious, error-prone reviewing — so your team doesn't have to?
The Problem: Why Computer Vision
Imagery and video have historically been difficult to fold into a utility's technology ecosystem, for two reasons. First, the data volume is large relative to other sensor types. Second, it's unstructured by nature — a fundamentally different animal than temperature sensors, voltage meters, or grid monitoring feeds.
But there's real business value in bridging that gap. The clearest one: reducing tedious, error-prone human review. Picture 10,000 photos or hours of dash-cam footage from around substation sites, with people asked to catch every inconsistency or maintenance need after hours of watching. No human reviewer is going to be 100% — but computers don't get tired, and they can provide a strong first-pass review before a person ever looks at it.
What Computer Vision Actually Covers
Computer vision is a huge field with many subfields, usually broken down by the kind of task being performed:
Image classification
what's in the image (e.g., one or more people, sheep, dogs)
Object localization
where each of those things is in the frame
Semantic segmentation
classifying every pixel (this pixel is "sheep," "person," or "background") with no concept of individual instances
Object segmentation
the synthesis of the above: what's in the image, where it is, and which pixels belong to which specific instance
Object tracking
following objects across video frames
Depth estimation
reconstructing a 3D model of a scene from a 2D photo or video, separating foreground from background
Document analysis
treating a scanned document or old PDF as an image so OCR and document-understanding models can process it
Reference example: the Microsoft COCO dataset (Lin et al., ECCV 2014) is the standard public benchmark used to illustrate and train models across these task types.
A Technique Worth Watching: Eulerian Video Magnification
Published by MIT roughly a decade ago (Wu et al., SIGGRAPH 2012), this technique detects pixel-level changes in video that are invisible to the human eye and amplifies them. The classic demo: a stationary construction crane that appears barely still in the raw footage — magnified, it's clearly swaying in the breeze.
Utility application:
this could be relevant for monitoring trucks, substation equipment, or transmission structures for subtle stability issues in wind, or small changes over time that wouldn't otherwise be visible to an inspector walking the site. Worth exploring as a way to get ahead of physical infrastructure changes before they become failures.
How It Works: The Pipeline
A computer vision pipeline typically runs: input → image processing → AI/computer vision model → output.
Image processing
comes first — low-level work like sharpening, denoising, and adjusting exposure to prepare the image for higher-level analysis.
Supervised learning
is how most detection tasks get built. To detect corroding transformers, for example, you need labeled examples: images marked as "good" or "bad," or annotated with exactly where the rust is.
Pre-trained models
You don't have to start from scratch. Pre-trained models (like the COCO baseline) already understand broad classes of objects, lighting conditions, and detection scenarios. You fine-tune on top of that baseline with utility-specific equipment and conditions — this is often called transfer learning.
Vision Language Models (VLMs)
over the last two to three years, models like Claude, ChatGPT, and Gemini have added image and video understanding alongside natural language. That enables interactive workflows: "find me the photos of transformer type XYZ, throw out the underexposed nighttime shots, and do a rough pass at flagging maintenance issues." It won't be as precise as a custom-trained detector, but it's a strong, flexible first cut — separating the wheat from the chaff.
VLMs aren't limited to frontier cloud models
Smaller models can run locally on a laptop or phone. They're less capable than the latest cloud models, but "good enough" depends heavily on the use case — and running locally can meaningfully lower cost.
Scope matters.
A narrow, well-defined use case — like checking whether workers are wearing the right safety equipment — is much easier to build well, because the training data and the range of expected inputs are both tightly constrained, unlike a general-purpose model meant to handle any image, any question, from anyone.
Considerations
Licensing
a single computer vision system can carry multiple, separate licenses: one for the framework/library, one for the base model you fine-tune, and one for the training dataset. Worth documenting provenance carefully, especially in commercial or regulated environments.
Data privacy
depending on requirements, this may mean running a local or self-hosted model, or putting a formal enterprise data privacy agreement in place with a cloud vendor.
Regulatory / compliance
requirements vary by whether an application is consumer-facing or internal, whether health data is involved, and by utility-sector-specific data classification for sensitive infrastructure (including generation sites). Even something as simple as video of someone walking through a plant could itself be considered sensitive data — separate from anything it's later used to train.
Failure modes
especially in a safety-sensitive environment: does the system fail open or fail closed? What's the cost of a false negative versus a false positive? Sensitivity and specificity are tunable thresholds, and "good enough" has to be explicitly defined for the use case rather than assumed.
Findings: What Ryan Built and Learned
(Honest assessment — matches the candid tone of the other prototype pages.)
Ryan built two working prototypes for this demo:
1. Personal protective equipment (PPE) detector
flags safety vests, hard hats, and boots. This one is in good shape and ready to demo.
2. Utility equipment maintenance detector
power lines, transformers, and transmission line insulators, aimed at flagging corrosion, missing equipment, physical damage, and vegetation encroachment. Built alongside Mark Panthofer, this one is still in progress — the broader dataset and much larger number of categories make it a harder problem, and training hasn't yet hit the same quality bar as the PPE model.
Findings: What Ryan Built and Learned
The lesson: narrower, more specific use cases move faster. If Alliant started with just corrosion detection, or just vegetation-encroachment detection, as a single focused use case, it would likely train faster and produce usable results sooner — additional categories can be layered on over time rather than tackled all at once.
A pleasant surprise: fewer labeled examples were needed than expected. Roughly 1,500 open-license images were enough to fine-tune a usable PPE detector on top of a pre-trained baseline — nowhere near the tens of thousands once assumed necessary. That's because the baseline model already understands general shapes, edges, and patterns (and roughly 80 general object classes); fine-tuning only has to teach it the new, specific classes on top of that.
Model family used: YOLO ("You Only Look Once"), originally created by researcher Joseph Redmon (through versions YOLO, YOLOv2, and YOLOv3), then open-sourced. Since then, the broader computer vision research and academic community has grown it into a large ecosystem of task-specific variants.
Worth noting: licensing complexity has grown noticeably since Ryan's earlier deep computer vision work (on the James Webb Space Telescope program) — there are more layers of framework, model, and dataset licensing to track today than there used to be.
Guiding principle throughout: faithfully represent the technology. The goal was a proof of concept nvisia could stand behind — not overselling what it can do, and not underselling it either.
Another lesson — don't over-build. It's tempting to reach for the newest, biggest model, but bigger models cost more, run slower, and need more data and training time to reach a usable state. Defining minimum requirements up front — and considering whether a model actually needs to run in a data center versus on a laptop, phone, or tablet — can meaningfully simplify deployment, especially anywhere data residency or jurisdiction requirements apply.
What This Means for Alliant
A narrow, well-scoped computer vision pilot — PPE compliance, or one specific maintenance-detection category — is realistic to stand up quickly using openly licensed models and datasets.
Vision language models offer a flexible, lower-effort complement for first-pass triage (sorting or filtering large photo or video sets) without needing a fully custom-trained detector.
Questions worth asking internally: What's the cost of a missed defect versus a false alarm in our environment? Do we have — or can we get — labeled examples of the specific equipment and conditions we care about? Does this need to run in the cloud, or could it run at the edge? What data classification applies to our imagery given critical-infrastructure requirements?
Resources
- Computer Vision: Algorithms and Applications, 2nd ed. — Richard Szeliski
- UCI ML Repository, COCO, Pascal VOC (public datasets)
- OpenCV, Scikit-Learn, Scikit-Image (open-source toolkits)
References:
- T.-Y. Lin et al., "Microsoft COCO: Common Objects in Context," ECCV 2014, pp. 740–755.
- H.-Y. Wu et al., "Eulerian Video Magnification for Revealing Subtle Changes in the World," ACM Transactions on Graphics (SIGGRAPH 2012), vol. 31, no. 4.