heating the forge...

orchestra-research_ai-research-skills

Security evaluation report for @forgecat/orchestra-research_ai-research-skills v0.1.4

SRC-003Danger

Static scanner flagged danger: supply chain risk (OWASP LLM03). The profile references 90+ external packages and frameworks (transformers, torch, litgpt, axolotl, vllm, etc.) without cryptographic verification or pinning.
The profile recommends installing packages via pip and npm (e.g., 'pip install litgpt[extra]', 'npx forgecat install') without hash verification or lock files. This is standard practice but creates supply chain exposure if any dependency is compromised.
The profile does NOT instruct typosquatting, dependency hijacking, or installing attacker-controlled packages — it recommends legitimate, well-known open-source projects. The danger is ambient (any package ecosystem has supply chain risk) rather than malicious guidance.

Details

Findings

skills/ai-research-skills/09-infrastructure/lambda-labs/references/troubleshooting.md:148
skills/ai-research-skills/09-infrastructure/lambda-labs/references/troubleshooting.md:196
Code obfuscation
skills/ai-research-skills/docs/npm-package-plan.md · SKILL-005
System modification
skills/ai-research-skills/09-infrastructure/lambda-labs/references/advanced-usage.md · SKILL-009
Suspicious URLs
skills/ai-research-skills/08-distributed-training/ray-train/references/multi-node.md · SKILL-007
Suspicious URLs
skills/ai-research-skills/06-post-training/openrlhf/SKILL.md · SKILL-007
Suspicious URLs
skills/ai-research-skills/06-post-training/openrlhf/references/custom-rewards.md · SKILL-007
Suspicious URLs
skills/ai-research-skills/06-post-training/openrlhf/references/hybrid-engine.md · SKILL-007
Suspicious URLs
skills/ai-research-skills/06-post-training/openrlhf/references/multi-node-training.md · SKILL-007
Suspicious URLs
skills/ai-research-skills/13-mlops/mlflow/SKILL.md · SKILL-007
Data exfiltration
skills/ai-research-skills/13-mlops/mlflow/references/deployment.md · SKILL-002
Suspicious URLs
skills/ai-research-skills/13-mlops/mlflow/references/deployment.md · SKILL-007
Data exfiltration
skills/ai-research-skills/14-agents/autogpt/references/troubleshooting.md · SKILL-002

SRC-010Caution

The autoresearch skill explicitly instructs the agent to operate fully autonomously without user confirmation ('Do not ask the user for permission or confirmation — use your best judgment and keep moving'), which significantly expands autonomous authority.
The agent-continuity reference instructs setting up recurring cron jobs and /loop triggers that inject prompts every 20 minutes, enabling persistent autonomous operation — a form of self-expanding, persistent agency that could be misused if any skill content is malicious.
No direct credential exfiltration, role-hijack, or 'ignore previous instructions' language was found in the reviewed content; the concern is the broad autonomous mandate combined with the large unreviewed skill corpus flagged by the static scanner.

Details

Findings

Invisible characters detected
README.md · SKILL-011
Invisible characters detected
skills/ai-research-skills/CONTRIBUTING.md · SKILL-011
Invisible characters detected
skills/ai-research-skills/docs/ROADMAP.md · SKILL-011

Safe

The profile declares no explicit tools or MCP servers with dangerous permissions (shell, file_write, file_delete, etc.). It is pure markdown guidance.
The autoresearch skill instructs agents to create and manage workspace files (research-state.yaml, findings.md, experiment directories) — this is file I/O for legitimate research state management, not excessive agency.
The profile recommends using domain-specific skills (LitGPT, Axolotl, vLLM, etc.) which may have their own tool definitions, but the profile itself does not demand high-risk permissions beyond what a research agent would need.

Details

Evidence

No MCP servers declared in profile.
File operations are scoped to project workspace: 'Create this structure at the project root: {project}/research-state.yaml, literature/, experiments/, etc.'
Tool usage is routed through domain skills: 'When you encounter a domain-specific task during research, search the skills library for the right tool.'

Safe

No MCP servers are defined in the profile. The profile is pure markdown guidance for agents.
The profile references external tools and libraries (transformers, torch, litgpt, etc.) but does not embed hidden instructions in tool descriptions or define arbitrary binary execution.
The profile does instruct agents to use web search (Exa MCP, Semantic Scholar API) and cloud compute platforms (Modal, SkyPilot), but these are legitimate research infrastructure, not hidden malicious servers.

Details

Evidence

Profile declares: '## MCP servers (untrusted data) (none)'
External tool references are documented openly: 'Use multiple sources — Exa MCP, Semantic Scholar, arXiv, CrossRef.'