AI Framework v1.0 ← main site

[ security / ai / open-source ]

Which AI type does
your project actually need?

An interactive guide to picking the right AI type for any project.
Tap a question to walk the decision tree.

// 01 — decision tree

01

Is the data temporal / time-indexed — logs, prices, metrics, sensor readings that change over time?

▼

→ Time Series / Forecasting. Use Prophet, statsmodels, or sktime. Works for prediction, trend analysis, and anomaly detection over time. Sequence matters — don't flatten it.

→ Continue to the next question ↓

02

Is the input primarily images or video — detection, classification, segmentation, or OCR?

▼

→ Computer Vision. Use YOLO for detection, torchvision for classification, OpenCV for preprocessing. A distinct discipline from general ML — the spatial structure of pixels demands its own toolchain.

→ Continue to the next question ↓

03

Does the system learn from rewards and interaction rather than labeled training examples?

▼

→ Reinforcement Learning. Use Gymnasium + Stable-Baselines3 or RLlib. Best for optimization, adaptive control, robotics, and game-playing. Requires a well-defined reward function — if you can't measure success, RL can't learn it.

→ Continue to the next question ↓

04

Is the goal finding similar items, semantic search, or ranking — without generating new content?

▼

→ Embeddings / Semantic Search. Use sentence-transformers + Chroma, Pinecone, or pgvector. Convert inputs to vectors, search by proximity. No generation — pure retrieval. Much cheaper and faster than using an LLM for search.

→ Continue to the next question ↓

05

Does the input combine multiple modalities — text + image, audio + text, or document + vision?

▼

→ Multimodal AI. Use Claude Vision, GPT-4V, or Gemini. Understands and reasons across modalities — distinct from GenAI (which creates) and LLM (which only reads text). Use when your input is inherently mixed.

→ Continue to the next question ↓

06

Do you have labeled, structured data and a well-defined output — class, score, or cluster?

▼

→ Traditional ML. You have the ingredients. Build the model — sklearn, XGBoost, or lightweight neural net. Fast, explainable, and production-ready.

→ Continue to the next question ↓

07

Is the primary input/output language, text, or code — and does it need to retrieve from your own documents?

▼

→ LLM + RAG. Ground the model in your documents. Use LangChain, LlamaIndex, or Haystack with a vector store. RAG is the architecture that makes LLMs trustworthy on domain-specific knowledge.

→ LLM. Zero-shot capable. One prompt, strong output. Continue to Q08 if it needs to act across multiple steps.

→ Generative AI. Images, audio, video, synthetic data. Match the modality to the right tool.

08

Does the task need to take actions, use tools, or reason across multiple steps toward a goal?

▼

→ Agentic AI. Plan → act → observe → adjust. Use LangChain, LlamaIndex, or Claude API with tool use. Decide: fully autonomous or human-in-loop?

→ LLM (single inference). One prompt, one output. Simplest and cheapest path.

09

Is accuracy + explainability critical — or is creativity and fluency more important?

▼

→ ML or LLM + RAG + guardrails. Add verification steps. LLMs hallucinate — design around it with grounding, output validation, and fallback logic.

→ LLM or GenAI. Let the model breathe. Human review is your safety net. Ship, iterate, refine.

// 02 — the 10 types at a glance

ML — Traditional

Recognize patterns

Learns from labeled examples. Fast, auditable, production-safe. Needs clean data to start.

Classification, regression

Anomaly detection

Clustering, scoring

LLM

Understand language

Reads, reasons, extracts, explains. Zero-shot capable. Always check for hallucination risk.

Summarization, extraction

Q&A, code generation

Text classification via prompt

Generative AI

Create new things

Produces novel text, images, audio, code. Creative and powerful — non-deterministic by nature.

Images, audio, video

Synthetic training data

Creative writing

Agentic AI

Take autonomous action

Plans, uses tools, loops toward a goal. Most powerful — highest complexity and risk.

Research & enrichment agents

Automation pipelines

Multi-tool workflows

Embeddings

Measure similarity

Converts inputs into vectors. Finds similar items, ranks results, enables semantic search — no generation needed.

Semantic search

Duplicate detection

Recommendation ranking

Multimodal AI

Reason across modalities

Understands text + image, audio + text, or document + vision simultaneously. Not the same as GenAI — this is about understanding, not creating.

Image + text Q&A

Document analysis with charts

Video + transcript reasoning

Computer Vision

See and interpret

Classifies, detects, and segments images and video. A distinct ML discipline — spatial structure demands its own toolchain.

Object detection (YOLO)

Image classification

OCR, segmentation

Reinforcement Learning

Learn from rewards

Learns by interacting with an environment and maximizing reward. Very different mental model — no labeled data, just feedback loops.

Robotics, control systems

Game-playing agents

Adaptive optimization

Time Series

Forecast over time

Learns from sequential, time-indexed data. Sequence matters — treating it as regular ML loses the temporal signal entirely.

Demand forecasting

Anomaly detection in metrics

Trend and seasonality analysis

RAG

Retrieve then generate

An architectural pattern, not a model. Grounds LLM output in your actual documents. Cuts hallucination on domain knowledge dramatically.

Internal knowledge base Q&A

Document-grounded summarization

Citation-backed answers

// 03 — golden rule

GOLDEN RULE Start with the output shape — not the tech. the output shape — not the tech. Output is a label or score → ML. Output is text or language → LLM. Output is something new or creative → GenAI. The task needs to act across multiple steps → Agentic. Most real projects are hybrid — pick the dominant output type and layer from there. Validate the concept before you build the infra.

// 04 — common mistakes

✗

Using GenAI when you need ML

GenAI can't be trained on your labels and will hallucinate structured outputs. If you need a score, a class, or a consistent prediction — that's ML territory.

→ Fix: if the output is a label or number, reach for sklearn or XGBoost first.

✗

Building Agentic when one LLM call suffices

Agents add latency, cost, and failure modes. Most tasks that feel multi-step can be handled with a well-structured single prompt or a short chain.

→ Fix: try a single prompt first. Build Agentic only when it genuinely saves human time.

✗

Training ML before you have data

No labels = no signal. A model trained on 50 examples is worse than zero-shot LLM. Data collection is the work — not the model.

→ Fix: use LLM zero-shot until you have 500+ labeled examples worth trusting.

✗

Fine-tuning before exhausting prompting

Fine-tuning is expensive, slow, and brittle. Prompt engineering, few-shot examples, and RAG solve 90% of cases that feel like they need a fine-tune.

→ Fix: iterate prompts, add examples, try RAG. Fine-tune only when prompting has truly plateaued.

✗

Building infra before validating the concept

LangChain pipelines and vector databases before you've confirmed the LLM can even do the task. A notebook + one API call should come first.

→ Fix: validate with a script. Add orchestration only after the core output is good enough.

✗

Going fully autonomous with Agentic on day one

Agents fail in unexpected ways — wrong tool calls, loops, hallucinated inputs. Full autonomy before supervised testing is how you ship embarrassing or harmful outputs.

→ Fix: human-in-loop first. Remove checkpoints only after you've seen it succeed 20+ times.

✗

Using an LLM for semantic search

Calling an LLM to "find similar documents" on every query is slow and expensive. Embeddings + a vector store do this in milliseconds at a fraction of the cost — no generation needed.

→ Fix: encode your corpus once with sentence-transformers, store vectors, query by proximity.

✗

Treating Multimodal AI as Generative AI

Multimodal models (Claude Vision, GPT-4V) understand mixed inputs — they don't generate images. Using them to "create visuals" is the wrong tool. GenAI creates; Multimodal understands.

→ Fix: Multimodal = understanding text + image together. GenAI = producing new images/audio/video.

✗

Reaching for RL before simpler approaches

RL is notoriously hard to tune, slow to converge, and brittle outside its training environment. Most "adaptive" problems can be solved with supervised ML, rule engines, or bandit algorithms first.

→ Fix: exhaust heuristics, bandits, and supervised ML. Use RL only when you have a clear reward signal and can afford the training cost.

✗

Flattening time-series data into regular ML features

Running sklearn on time-indexed data without respecting the sequence loses the temporal signal entirely. A model trained this way will look reasonable in cross-validation and fail in production when patterns shift.

→ Fix: use time-aware splits, lag features, and proper forecasting models (Prophet, sktime, LSTMs).

// 05 — hybrid combinations

MLLLM

Detect + Explain

ML classifies or scores; LLM explains the result in plain language or takes a follow-up action. Fast and auditable where it counts.

→ Threat detection + analyst summary. Sentiment scoring + response drafting.

LLMRAG

Ground + Generate

LLM generates; RAG keeps it grounded in your actual documents. Cuts hallucination on domain-specific knowledge dramatically.

→ Q&A over docs. Report summarization with citations. Internal knowledge bases.

AgenticRAG

Retrieve + Act

Agent retrieves context at each step rather than loading everything upfront. Useful when the next action depends on what was just found.

→ Research pipelines. OSINT enrichment loops. Multi-step report generation.

MLAgentic

Score + Route

ML produces a fast, cheap score; the agent uses that score to decide the next step. Keeps reasoning cheap and structured predictions reliable.

→ Triage and prioritization workflows. Risk-based routing and escalation.

EmbeddingsLLM

Retrieve + Generate (RAG)

The canonical RAG pattern. Embeddings find the right context; LLM reasons over it. Neither alone is as accurate — together they're the standard for domain-specific Q&A.

→ Knowledge base Q&A. Document-grounded chat. Citation-backed answers.

CVML

See + Classify

CV extracts visual features or detections; ML downstream scores, routes, or aggregates the results. Keeps the visual and decision logic cleanly separated.

→ Defect detection pipelines. Security camera analytics. Medical image triage.

Time SeriesLLM

Forecast + Explain

Time series model handles the temporal prediction; LLM translates the forecast into plain-language insights or generates alerts. Bridges the gap between model output and human action.

→ Metric anomaly alerts with natural language summaries. Demand forecasts with business narrative.

EmbeddingsAgentic

Retrieve + Act

Agent retrieves semantically relevant context at each step and acts on it — not just at the start. Enables dynamic, context-aware pipelines that don't front-load all retrieval.

→ Research agents that search as they reason. Adaptive enrichment pipelines.

MultimodalAgentic

See + Act

Vision-enabled agents that can read images, screenshots, or documents as part of their reasoning loop. Unlocks automation of visual workflows that text-only agents can't handle.

→ UI automation agents. Document processing pipelines. Visual QA workflows.

// 06 — tooling by type

Type	Core tools	Notes
ML	scikit-learnXGBoostPyTorchHuggingFaceSHAP	SHAP for explainability — non-optional if humans need to trust the output.
LLM	Claude APIOpenAI APIOllamaLiteLLM	Ollama for local/offline. LiteLLM as a unified gateway across providers.
Generative AI	DALL-EMidjourneyStable DiffusionElevenLabsSora	Match the modality to the output — image, audio, video, or code are all different tools.
Agentic	LangChainLlamaIndexClaude API (tool use)CrewAIAutoGen	Claude API tool use for simple agents. LangChain / CrewAI for multi-agent orchestration.
Embeddings	sentence-transformersChromaPineconepgvectorFAISS	sentence-transformers for encoding. Chroma/pgvector for self-hosted. Pinecone for managed scale.
Multimodal	Claude Vision APIGPT-4VGeminiBLIP-2	Claude and GPT-4V for text+image reasoning. BLIP-2 for open-source vision-language tasks.
Computer Vision	OpenCVUltralytics (YOLO)torchvisionDetectron2Roboflow	YOLO for real-time detection. torchvision for classification. Roboflow for dataset management.
Reinforcement Learning	GymnasiumStable-Baselines3RLlibRay	Gymnasium for environments. Stable-Baselines3 for standard algorithms. RLlib for distributed scale.
Time Series	ProphetstatsmodelssktimeDartsNeuralForecast	Prophet for business forecasting. sktime for a unified API. NeuralForecast for deep learning approaches.
RAG	LangChainLlamaIndexHaystackChromapgvector	LlamaIndex for document-heavy pipelines. Haystack for production search. Always pair with a vector store.

// 07 — cost & effort reality

ML

Setup costMedium

Inference costLow

ExplainabilityHigh

MaintenanceMedium

Data requiredYes — labeled

LLM

Setup costLow

Inference costMedium

ExplainabilityLow–Med

MaintenanceLow

Data requiredNo

Generative AI

Setup costLow

Inference costMedium–High

ExplainabilityNone

MaintenanceLow

Data requiredNo

Agentic

Setup costHigh

Inference costHigh

ExplainabilityLow

MaintenanceHigh

Data requiredNo

Embeddings

Setup costLow

Inference costVery Low

ExplainabilityMedium

MaintenanceLow

Data requiredNo labels

Multimodal

Setup costLow

Inference costMedium–High

ExplainabilityLow

MaintenanceLow

Data requiredNo

Computer Vision

Setup costMedium–High

Inference costLow–Medium

ExplainabilityMedium

MaintenanceMedium

Data requiredYes — labeled images

Reinforcement Learning

Setup costVery High

Inference costLow

ExplainabilityVery Low

MaintenanceVery High

Data requiredEnvironment / sim

Time Series

Setup costLow–Medium

Inference costLow

ExplainabilityMedium–High

MaintenanceMedium

Data requiredYes — historical seq.

RAG

Setup costMedium

Inference costMedium

ExplainabilityMedium

MaintenanceMedium

Data requiredDocuments only

// 08 — when to level up

LLM → Agentic

When the task needs 3+ sequential steps, requires calling external tools, or loops until a condition is met — and doing it manually is eating real time.

LLM → LLM + RAG

When zero-shot accuracy on domain-specific knowledge is too low and the model keeps hallucinating facts you actually have stored somewhere.

LLM → Fine-tuned LLM

When prompt engineering has plateaued, you have 1,000+ high-quality examples, and the task has a consistent input/output shape that benefits from baked-in style or format.

ML → ML + LLM

When the model's output needs to be explained, acted on, or communicated to a non-technical audience — ML gives the answer, LLM translates it.

Prompt → ML

When you've collected 500+ labeled examples, the task is well-defined, and you need speed, cost efficiency, or explainability that a live LLM call can't provide.

ML → CV

When your input is images or video. Tabular ML doesn't understand spatial structure — you need convolutional or transformer-based architectures built for pixels.

LLM → Embeddings

When your task is finding similar items or semantic search — not generating text. Embeddings are 100× cheaper and faster for retrieval. Stop calling an LLM to compare documents.

Embeddings → RAG

When retrieved results need to be reasoned over, summarized, or answered in natural language — not just ranked. Embeddings find the context; LLM completes the answer.

ML → Time Series

When your data has a temporal index and sequence matters. If you're treating timestamps as just another feature in sklearn, you're losing the signal that makes the data useful.

// 09 — sources & docs

ML

scikit-learn — User Guide XGBoost — Documentation Hugging Face — Docs SHAP — Explainability PyTorch — Docs

LLM

Anthropic — Claude API Docs OpenAI — Platform Docs Ollama — Local LLMs LiteLLM — Unified Gateway

Generative AI

OpenAI — Image Generation Stability AI — Stable Diffusion ElevenLabs — Audio/Voice Sora — Video Generation

Agentic

LangChain — Python Docs LlamaIndex — Docs Claude — Tool Use & Agents AutoGen — Multi-Agent

Embeddings

sentence-transformers — Docs Chroma — Vector Store Pinecone — Managed Vector DB pgvector — Postgres Extension

Multimodal & Computer Vision

Claude — Vision Docs OpenAI — Vision Guide Ultralytics (YOLO) — Docs torchvision — Docs OpenCV — Docs

RL & Time Series

Gymnasium — RL Environments Stable-Baselines3 — Docs Prophet — Forecasting sktime — Time Series ML NeuralForecast — Deep TS

// 10 — questions?

Still figuring out which AI fits your project?

This framework is a starting point, not a rulebook. If you're stuck on the decision, working through a use case, or just want to talk through your stack — feel free to reach out. I'm always happy to think out loud about this stuff.

✉ Email → 𝕏 Twitter / X → GitHub →