Building a trustworthy AI maritime agent turns out to be far more about system design than model selection. That’s the central lesson Skylight drew from developing Shippy, an AI agent built specifically for real-time maritime domain awareness — a domain where a wrong answer doesn’t just frustrate a user, it can redirect a patrol vessel miles off course, drain limited resources, and potentially endanger personnel.
Summary
Key takeaways
- Shippy is an AI agent built by Skylight for real-time maritime domain awareness, powering queries on vessel behavior, EEZ and MPA boundaries, and vessel tracks.
- Its architecture is structured around three components: a soul (system prompt), skills (task handling), and config (runtime settings including the use of Claude Opus 4.6).
- Shippy communicates with Skylight’s API through a purpose-built deterministic CLI, eliminating the error patterns that emerged when the model constructed raw API calls directly.
- Each user session runs in a dedicated Kubernetes sandbox provisioned by Mothership, ensuring full data isolation across Skylight’s hundreds of government and NGO clients.
- Evaluation relies on a task- and rubric-based framework with an LLM judge scoring responses against live data — a deliberate departure from static benchmarks.
Why maritime domain awareness raises the reliability bar
Most AI tools operate in environments where a mediocre answer is merely unhelpful. Maritime operations are different. Skylight serves hundreds of government agencies and NGOs across more than 70 countries, from fisheries enforcement to conservation monitoring. When an analyst in that context asks whether vessels are operating illegally inside a Marine Protected Area, the stakes of a hallucinated or imprecise response are immediate and operational.
That context shaped every architectural decision behind Shippy. The question the Skylight team kept returning to wasn’t “what can the model do?” — it was “how do we build a system we can trust to be correct, stay within its limits, and hold up across a wide range of tasks?” All of it had to be verified against Skylight’s live data, updated continuously as new satellite and vessel signals arrive.
Shippy’s architecture: soul, skills, and config
Skylight describes the Shippy AI maritime agent through three distinct layers: a soul, skills, and config. The distinction matters more than it might initially seem.
The soul is the system prompt — it defines Shippy’s persona and sets hard behavioral limits. What Shippy will do, and crucially, what it won’t. The soul is explicit and auditable: Shippy won’t make legal determinations about whether a vessel is breaking the law, and it won’t speculate beyond what the data supports. These aren’t soft guardrails baked into fine-tuning; they’re written directly into the system prompt, which makes them easy to inspect and revise.
Skills tell Shippy how to handle specific request types. They follow the same agent-skills specification used by coding tools like Claude Code and Codex — plain markdown files with structured frontmatter, keeping each skill versioned and comprehensible. Shippy’s current skill set includes:
- Querying the Skylight API for vessel behaviors and events (fishing, transshipment between vessels)
- Looking up Exclusive Economic Zone (EEZ) and Marine Protected Area (MPA) boundaries
- Interpreting vessel track data and position signals, building on activity classifications already produced by Skylight’s own models
- Generating interactive map links so analysts can jump from a chat response to an exact location on the Skylight map
A single user question can activate several skills simultaneously. Ask whether vessels are operating near the Cordillera de Coiba MPA, and Shippy draws on Skylight’s data query skill, ProtectedSeas’ boundary database, and the vessel track interpretation skill — all within a single dialogue turn.
Config and the model question
The config layer handles everything runtime-related: which agent harness to run, which LLM to use, and injected secrets like API keys. Shippy currently runs on Claude Opus 4.6 as its large language model. Importantly, swapping the underlying model is a config change rather than a rebuild — the soul and skills are baked into a Docker image independently, giving the team flexibility to route different query types to different models as the system evolves.
Deterministic tooling for a nondeterministic agent
One of the more instructive early lessons came from letting Shippy construct raw API calls directly. The result was a steady stream of subtle failures: malformed pagination that silently dropped results, geometry encoding errors, and queries that looked correct but returned wrong data due to misunderstood filter types. The Skylight API has dozens of input types, nested filter objects, pagination cursors, and complex geometry inputs — the kind of surface area that invites model-generated errors.
The solution was a purpose-built deterministic CLI. Instead of constructing raw API calls, Shippy issues a single command — for example, skylight events search with typed filter flags — and the CLI handles authentication, pagination, and structured output. The CLI is self-documenting, with detailed help text and error messages that allow the agent to recover from mistakes without guessing. Results are always written to a local JSON file rather than piped through the shell, sidestepping pipe buffer limits that caused problems with large result sets in early prototypes.
The layering design — typed API, deterministic CLI, agent skills referencing CLI commands — means each component can be tested independently. Each layer narrows what the next layer can get wrong. This is the kind of architectural discipline that separates a research prototype from a production system in a high-stakes domain.
Kubernetes sandboxing and the data isolation challenge
Skylight’s user base spans government agencies with sensitive, jurisdiction-specific data. A fisheries officer in the Philippines has watchlists, Areas of Interest, and alert configurations scoped to their own account. Making sure their data never bleeds into another user’s session — and that their conversation history stays entirely private — was one of the most significant engineering efforts in the project.
The solution is Mothership, Skylight’s agent hosting platform. Mothership provisions a dedicated Kubernetes deployment for each user session. When a conversation opens, the system spins up a set of pods packaging the agent runtime, its skills, and the Skylight CLI. The user’s JWT token is injected at provision time, scoping all API calls to that user’s data. Files the agent writes during a multi-step analysis exist only within that session and are never accessible across users.
Inside the sandbox, the agent retains substantial operational capability — it can write and run code, install dependencies, pull in datasets, and work through multi-step analyses. At the network level, the sandbox is restricted to only the services it needs. The isolation is session-scoped, ephemeral, and enforced at the infrastructure level rather than relying solely on application logic.
Evaluating an agent in live operational contexts
Standard AI benchmarks score models on static question sets. They don’t capture what happens when an agent is wired into a real workflow — how it selects tools, queries live data, acts on the results, and knows when to stop. Skylight built a custom evaluation system for exactly this reason.
A rubric-based framework with an LLM judge
In Skylight’s eval framework, subject-matter experts write scenarios and rubrics, determining which criteria apply to each task and setting the weights. A fishing-events query, for instance, weights data accuracy most heavily, with boundary resolution and timeframe next, and source attribution carrying less weight. Experts also annotate individual responses as correct or incorrect, giving the judge ground truth to score against.
The pipeline runs a natural-language prompt through the live sandbox, and an LLM judge grades each criterion on a 0-to-1 scale with written reasoning explaining why the response did or didn’t meet it. The weighted aggregate is then checked against a fixed pass threshold. Tasks run through Harbor, an open evaluation framework, via a Skylight-built plugin that spins up a real Shippy session against the same live data a user would encounter.
The results from recent evaluation runs point to specific, actionable failure modes. Patrol-planning tasks saw Shippy overstepping into tactical recommendations rather than staying within decision-support bounds. Geometry-sensitive queries exposed missed events caused by boundary simplification. And in one case, the agent generated a CLI command that didn’t exist. Each pattern maps directly to a skill improvement target — which is exactly what a well-designed eval system is supposed to produce.
What comes next for Shippy — and beyond
Skylight is opening Shippy to early adopters on a rolling basis, explicitly inviting stress-testing to surface weak guardrails and poorly handled queries. The next development cycle targets three areas: agent-driven UI control (moving from returning map links to directly controlling the Skylight map, applying filters and adjusting time ranges); model routing (directing simple lookups to smaller, faster models while reserving the full-weight model for complex investigations); and cross-thread memory (carrying persistent facts like an analyst’s jurisdiction or preferred sources across separate conversation threads).
The broader implications extend well past maritime applications. Mothership was designed to be domain-agnostic, and Skylight’s parent organization Ai2 is already applying lessons from Shippy to EarthRanger, its wildlife conservation platform, and OlmoEarth, its open suite of Earth observation tools. The architecture — soul, skills, config, deterministic tooling, session-isolated sandboxing, and live-data evaluation — represents a blueprint for deploying AI agents in any domain where the cost of a wrong answer is measured in something other than user frustration.
FAQ
What is Shippy designed to do?
Shippy is an AI agent developed by Skylight for real-time maritime domain awareness. It helps analysts query vessel behaviors, look up Exclusive Economic Zone and Marine Protected Area boundaries, interpret vessel track data, and generate interactive map links directly tied to Skylight’s live data platform.
How does Shippy ensure reliability in answering queries?
Shippy uses a deterministic CLI to handle all Skylight API calls, avoiding the error patterns that emerged when the model constructed raw API calls directly. Combined with a modular architecture — soul, skills, and config — and strict behavioral boundaries in the system prompt, the design prioritizes predictable, auditable behavior over flexibility.
How is Shippy evaluated for accuracy and reliability?
Skylight built a custom evaluation framework where subject-matter experts write task scenarios and weighted rubrics. An LLM judge scores each agent response against live data on a 0-to-1 scale per criterion, with written reasoning. The weighted aggregate is checked against a fixed pass threshold, and any version of Shippy that regresses on evaluation criteria doesn’t reach end users.
Does Shippy handle legal judgments about vessel activity?
No. Shippy explicitly avoids making legal determinations about whether a vessel is violating any law, and it won’t speculate beyond what the data supports. Those determinations are left to human analysts. These limits are written directly into the system prompt — making them auditable and adjustable — rather than being implicit in model fine-tuning.
Article produced with the assistance of artificial intelligence and reviewed by the editorial team.
