Overview
CTI-Transmute is a web platform that converts threat intelligence data between MISP and STIX 2.x formats. The conversion engine is Pivotick, a separate service running on port 6868. The web application wraps it with persistent storage, social features, a tagging system, and interactive visualizations.
Quick Start
All commands are run via uv from the repo root.
# Clone with submodules
git clone --recurse-submodules <repo>
# First-time: init submodules + create DB + apply migrations
uv run manage init
# Start everything
uv run manage startuv run manage start # start the app
uv run manage update # git pull + deps + migrations
uv run manage backup # pg_dump to website/db_class/backups/
uv run manage db migrate # create a new migration
uv run manage db upgrade # apply pending migrationsEdit config/generic.json:
{
"listen_ip": "0.0.0.0",
"listen_port": 5000
}
Database connection is hardcoded in website/web/__init__.py:
postgresql+psycopg2://cti_user:cti_pass@localhost:5432/cti_dbArchitecture
feature.py (routes),
feature_core.py (DB/logic),
feature_form.py (WTForms).
bin/start_website.py registers all blueprints and calls
application.run().
vendor/pivotick, vendor/misp-taxonomies,
vendor/misp-galaxy — all git submodules.
bin/
manage.py # CLI: start / init / update / backup / db
start_website.py # registers blueprints → application.run()
config/
generic.json # listen_ip, listen_port
cti_transmute/ # Transmute class wrapper
vendor/ # git submodules
website/
web/__init__.py # Flask factory, db/login/session/migrate
web/home.py # home_blueprint
web/convert/ # convert blueprint + core + form
web/evaluate/ # evaluate blueprint + core
web/tags/ # tags blueprint + core
web/account/ # account blueprint
web/utils.py # shared helpers
web/templates/ # Jinja2 (base.html + per-feature folders)
web/static/
css/ # one .css per feature
js/graph/ # convert visualizations
js/misp/ # MISP push modal
js/tags/ # tag components
js/evaluate/ # evaluation charts
db_class/db.py # all SQLAlchemy models
migrations/ # Flask-Migrate / Alembic
api/convert.py # REST API blueprint (:6868)Database Models
All models live in website/db_class/db.py.
Every model exposes a to_json() method.
Soft deletes use is_active = False + optional deleted_at.
Every main model has a uuid column (String 36).
| Field | Type | Notes |
|---|---|---|
| id | Integer PK | Auto-increment primary key |
| uuid | String(36) | UUID4, unique |
| first_name / last_name | String(64) | Display name |
| String(64) | Unique, used for login | |
| password | String(300) | bcrypt hash |
| role | String(36) | "user" or "admin" |
| api_key | String(60) | Bearer token for REST API |
| is_connected | Boolean | Online status flag |
| created_at / last_seen | DateTime | UTC timestamps |
| Field | Type | Notes |
|---|---|---|
| id / uuid | Integer / String(36) | PK + UUID4 |
| name | String(256) | User-defined label |
| description | Text | Optional free text |
| conversion_type | String(32) | "MISP_TO_STIX" or "STIX_TO_MISP" |
| input_text / output_text | Text | Raw JSON strings |
| public | Boolean | Visibility flag |
| share_key | String(64) | Token for sharing private converts |
| is_active | Boolean | Soft-delete flag |
| user_id | FK → User | Owner |
| created_at / updated_at / deleted_at | DateTime | UTC timestamps |
| Field | Type | Notes |
|---|---|---|
| version | Integer | Auto-incremented per convert |
| status | String | "pending" | "accepted" | "rejected" |
| input_text / output_text | Text | Snapshot of the conversion at that version |
| comment | Text | Optional change note |
| convert_id | FK → Convert | Parent convert |
| Field | Type | Notes |
|---|---|---|
| content | Text | Max 2000 chars |
| is_private | Boolean | Only visible to author + admins |
| is_evaluation | Boolean | Shows under Evaluation tab instead of Discussion |
| is_deleted | Boolean | Soft-delete, content replaced with placeholder |
| parent_id | FK → Comment | Self-referential for replies (1 level) |
| convert_id / user_id | FK | Owner and target |
| Field | Type | Notes |
|---|---|---|
| name | String(256) | Unique tag name (e.g. tlp:red) |
| colour | String(7) | Hex color code |
| icon | String(64) | FontAwesome class name |
| source_type | String(32) | "Manual" | "Taxonomy" | "Vulnerability" |
| is_approved / is_active | Boolean | Admin approval workflow |
| is_public | Boolean | Visible to non-admin users |
| is_evaluation_tag | Boolean | Available as reaction in evaluation voting |
| Field | Type | Notes |
|---|---|---|
| convert_id / tag_id | FK | Composite unique constraint |
| added_by | String | "user" or "auto-scan" |
| Field | Type | Notes |
|---|---|---|
| vote_type | String | "like" | "dislike" | "reaction" |
| reaction_key | String | Tag name used as evaluation reaction |
| convert_id / user_id | FK | Unique per (user, convert, vote_type) |
| Field | Type | Notes |
|---|---|---|
| notif_type | String | comment_reply, new_follow_convert, report_submitted, … |
| is_read | Boolean | Unread badge count in navbar |
| target_id / target_type | Integer / String | Polymorphic target (convert, comment, …) |
| user_id | FK → User | Recipient |
| Model | Purpose |
|---|---|
| UserFollow | follower_id → followed_id social graph |
| ConvertFavorite | Bookmarked converts per user |
| ConvertReport | Abuse reports: reason + description + status (pending / reviewed / dismissed) |
| SystemLog | Audit log — event_type, actor_name, target_id/type, details, created_at |
| GraphConfig | Saved Pivotick visualization configs per user |
| PlatformReview | 1–5 star platform rating + optional text per user |
REST API
The conversion REST API runs on port 6868 via the Pivotick service.
It is also proxied internally by Flask — call it from Flask at
http://127.0.0.1:6868/api/convert/....
api_key
in the Authorization header. Some endpoints are public.
/api/convert/list
Returns the list of all available conversion directions and their parameters.
{
"available": {
"misp_to_stix": {
"version": ["2.1"],
"description": "Convert a MISP event to a STIX 2.1 bundle"
},
"stix_to_misp": {
"params": ["distribution", "galaxies_as_tags", "single_event", "..."],
"description": "Convert a STIX bundle to a MISP event"
}
}
}
/api/convert/misp_to_stix?version=2.1
Convert a MISP JSON event to a STIX 2.1 bundle. Body is the raw MISP JSON.
{
"Event": {
"info": "Phishing campaign targeting finance sector",
"uuid": "5e9f1234-1234-1234-1234-1234567890ab",
"date": "2024-01-15",
"threat_level_id": "2",
"Attribute": [
{
"category": "Network activity",
"type": "ip-dst",
"value": "198.51.100.42",
"to_ids": true,
"uuid": "5e9f1234-aaaa-bbbb-cccc-1234567890ab"
},
{
"category": "Payload delivery",
"type": "sha256",
"value": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"to_ids": true
}
],
"Tag": [
{ "name": "tlp:amber" }
]
}
}
{
"type": "bundle",
"id": "bundle--a1b2c3d4-1234-5678-abcd-ef0123456789",
"objects": [
{
"type": "report",
"id": "report--...",
"name": "Phishing campaign targeting finance sector",
"published": "2024-01-15T00:00:00Z",
"object_refs": ["indicator--...", "marking-definition--..."]
},
{
"type": "indicator",
"id": "indicator--...",
"pattern": "[ipv4-addr:value = '198.51.100.42']",
"pattern_type": "stix",
"valid_from": "2024-01-15T00:00:00Z"
},
{
"type": "marking-definition",
"id": "marking-definition--...",
"definition_type": "tlp",
"definition": { "tlp": "amber" }
}
]
}
/api/convert/stix_to_misp
Convert a STIX 2.x bundle to a MISP event. All params are query strings.
| Parameter | Type | Default | Description |
|---|---|---|---|
| distribution | int 0–4 | 0 | MISP distribution level (0=org, 1=community, 2=connected, 3=all, 4=sharing group) |
| sharing_group_id | int | — | Required when distribution=4 |
| galaxies_as_tags | flag | off | Convert STIX galaxies to MISP tags instead of galaxy clusters |
| no_force_contextual_data | flag | off | Disable forced contextual data attachment |
| cluster_distribution | int 0–4 | 0 | Distribution for galaxy clusters |
| organisation_uuid | string | — | UUID of the organisation to attach the event to |
| single_event | flag | off | Force a single MISP event output |
| producer | string | — | Producer name tag |
| title | string | — | Override the event title |
{
"Event": {
"info": "STIX Report: ...",
"uuid": "...",
"distribution": 0,
"threat_level_id": 2,
"Attribute": [ { "type": "ip-dst", "value": "198.51.100.42", "..." : "..." } ],
"Object": [],
"Tag": []
}
}
Web Routes — Convert
All routes under /convert. JSON routes return {"success": bool, "message": "...", "toast_class": "..."}.
{ "success": true, "data": { }, "message": "Done", "toast_class": "success" }
{ "success": false, "message": "Error text", "toast_class": "danger" }
/convert/misp_to_stixSubmit a MISP JSON for conversion. Params: stix_version, name, description, public.
/convert/stix_to_mispSubmit a STIX bundle. Same params plus all STIX→MISP options (distribution, galaxies_as_tags, etc.).
/convert/fetch_misp_eventFetch a MISP event from a remote MISP instance via restSearch. Body: misp_url, misp_key, event_id.
/convert/misp_search_eventsSearch events on a remote MISP by info/tag/date. Returns a list of matching events.
/convert/refresh/<uuid>Re-run a conversion with the original input and show a diff preview.
/convert/refreshAccept or reject a pending history version. Body: history_id, action ("accept" | "reject").
/convert/historyHistory listing page (Vue SPA shell).
/convert/get_convert_page_historyPaginated JSON list. Filters: type, sort, date_from/to, tags[], favorites_only, search, search_scope.
/convert/get_convert?id=<id>Get one convert by DB id or UUID. Returns full object including input/output text.
/convert/detail/<id>Detail page with all tabs (Input, Output, Graph, Sunburst, Table, Comments, Evaluation).
/convert/difference/<id>Side-by-side diff of a history version vs the current output.
/convert/search_in_content?q=<q>Full-text search with snippet extraction. Returns matching convert IDs + context snippets.
/convert/edit_public?id=<id>Toggle public/private. Returns new state.
/convert/get_share_key?id=<id>Retrieve the share token for a private convert.
/convert/regenerate_share_key?id=<id>Rotate the share key, invalidating old share links.
/convert/share?uuid=<u>&share_key=<k>Access a private convert via share link. Redirects to detail if valid.
/convert/delete_itemSoft-delete. Body: {"id": 42}. Moves to admin trash.
/convert/get_comments?convert_id=<id>Fetch all comments for a convert. Filters private/deleted based on requester role.
/convert/commentCreate a comment or reply. Body: convert_id, content, is_private, is_evaluation, optional parent_id.
/convert/edit_commentUpdate comment content. Body: comment_id, content.
/convert/delete_comment?comment_id=<id>Soft-delete a comment. Content replaced with placeholder.
/convert/toggle_comment_private?comment_id=<id>Toggle a comment between public and private.
/convert/reactToggle emoji reaction. Body: comment_id, emoji. Whitelist: 👍 😊 ❤️ 🎯 ⚠️
/convert/favorite/toggleBookmark or un-bookmark a convert. Body: {"convert_id": 42}.
/convert/favorite/status/<id>Check whether the current user has favorited a convert.
/convert/download/<id>/inputDownload the raw input JSON file.
/convert/download/<id>/outputDownload the conversion result JSON.
/convert/download/<id>/misp-pushDownload the full PyMISP push payload including evaluation tags object.
/convert/push_to_mispPush to a remote MISP. Body: misp_url, misp_key, convert_id.
/convert/reportSubmit an abuse report. Body: convert_id, reason (spam/inappropriate/inaccurate/other), description.
/convert/trashAdmin trash page listing soft-deleted converts.
/convert/restoreRestore a soft-deleted convert. Body: {"id": 42}.
/convert/hard_deletePermanently delete a convert. Irreversible.
/convert/bulk_actionRestore or hard-delete multiple converts. Body: {"ids": [...], "action": "restore"}.
/convert/admin/get_reportsPaginated list of abuse reports with filters.
/convert/admin/review_report?id=<id>&action=<a>Mark a report as reviewed or dismissed.
Web Routes — Evaluate
All routes under /evaluate. Provides quality scoring and export for conversions.
/evaluate/summary/<id>Evaluation stats for a convert: like/dislike counts, reaction breakdown, consensus score.
/evaluate/toggle_likeLike a convert. Body: {"convert_id": 42}. Toggle: calling twice removes the like.
/evaluate/toggle_dislikeDislike a convert. Mutually exclusive with like.
/evaluate/toggle_reactionVote with a reaction tag. Body: {"convert_id": 42, "reaction_key": "tlp:red"}.
/evaluate/consensus_tags/<id>Returns tags that reached the vote threshold (default ≥3). Used in MISP push payload.
/evaluate/export/<id>/markdownDownload evaluation report as a .md file.
/evaluate/export/<id>/pdfDownload evaluation report as a PDF.
/evaluate/global_statsPlatform-wide evaluation stats.
/evaluate/activity_timeline?days=<n>Evaluation activity over the last N days (chart data).
/evaluate/platform_reviewSubmit a 1–5 star platform review. Body: {"score": 5, "content": "..."}.
/evaluate/admin/listAdmin: paginated list of all evaluations with filters.
Web Routes — Tags
All routes under /tags. Tags have three sources: Manual, Taxonomy (~12 380 from MISP), and Vulnerability.
/tags/createCreate a custom tag. Auto-approved for admins creating Vulnerability tags.
/tags/listPaginated list of tags accessible to the current user.
/tags/availableTags available for attachment. Filter ?evaluation=1 for eval-only tags.
/tags/for_convert/<id>Tags currently attached to a convert (filtered by source_type if provided).
/tags/save_for_convert/<id>Replace all tags on a convert. Body: {"tag_ids": [1, 2, 3]}.
/tags/extract_from_convert/<id>Auto-extract tag names from the stored JSON and match them against the tag library.
/tags/admin/listPaginated all tags with approval workflow. Filter by source, active, approved, public.
/tags/admin/approve/<id>Toggle approval status of a tag.
/tags/admin/import_taxonomiesImport all tags from vendor/misp-taxonomies submodule (~12 380 tags).
/tags/admin/import_galaxiesImport cluster names from vendor/misp-galaxy submodule.
/tags/admin/bulk_converts/scanBackground job: scan all converts and auto-assign matching tags from their JSON.
/tags/admin/bulk_converts/job/<id>Poll the status of a background bulk-tag job.
Web Routes — Account
All routes under /account.
/account/loginLogin with email + password. Returns a session cookie.
/account/registerCreate an account. Validates email uniqueness and password strength.
/account/logoutDestroy session and redirect to home.
/account/editUpdate first/last name, email, password.
/account/get_notification_countReturns unread notification count for the navbar badge.
/account/get_notificationsPaginated notifications. Filter ?unread_only=1, ?search=.
/account/mark_all_readMark all unread notifications as read.
/account/follow?user_id=<id>Follow or unfollow a user. Toggle endpoint.
/account/is_following?user_id=<id>Check if the current user follows a given user.
/account/get_usersPaginated user list. Filters: connection status, admin role, search.
/account/edit_adminGrant or revoke admin rights. Body: {"user_id": 5, "is_admin": true}.
/account/delete/<id>Delete a user and cascade-delete all their converts and comments.
/account/admin/get_all_notificationsMerged view of notifications + system logs. Filters: type, date range, search.
/account/admin/delete_logs_bulkBulk-delete logs by ID list.
Frontend — Vue 3
Vue 3 is loaded as a global CDN build from /static/js/vue.global.js.
There is no build step — components are plain .js ES modules with inline template strings.
The Jinja2 delimiter conflict is resolved by using [[ ]] instead of {{ }} in Vue templates.
const MyComponent = {
delimiters: ['[[', ']]'], // avoid conflict with Jinja2 {{ }}
props: {
convertData: { type: Object, default: null },
},
components: { 'json-viewer': JsonViewer },
template: `
<div class="mc-wrapper">
[[ convertData?.name ]]
</div>`,
setup(props) {
const { ref, watch } = Vue
// ...
watch(() => props.convertData, v => {
if (v) doWork()
}, { immediate: true }) // fires immediately on mount
return { /* exposed to template */ }
},
}
export default MyComponent
import MyComponent from '/static/js/graph/myComponent.js'
const app = createApp({ delimiters: ['[[',']]'], setup() { /* ... */ } })
app.component('my-component', MyComponent)
app.mount('#main-containers')
const csrfToken = document.getElementById('csrf_token')?.value
const res = await fetch('/convert/comment', {
method: 'POST',
headers: { 'Content-Type': 'application/json', 'X-CSRFToken': csrfToken },
body: JSON.stringify({ convert_id: 42, content: 'Great conversion!' }),
})
const data = await res.json()
if (data.success) create_message(data.message, data.toast_class)
Component Inventory
All reusable Vue components, their file paths, and what they do.
The detail page (templates/convert/detail.html) uses Bootstrap 5 tabs.
Each tab is a nav-link + a matching tab-pane.
The visualization components receive the convert object as :convert-data="convert"
and auto-render when the prop becomes available via watch({ immediate: true }).
Adding a New Visualization
All visualization components follow the same pattern.
They live in website/web/static/js/graph/,
receive convertData as a prop, and parse JSON client-side — no extra API call needed.
website/web/static/js/graph/myViz.js. The component receives convertData (the full convert object with input_text and output_text fields). Use watch({ immediate: true }) so it auto-renders as soon as the prop is populated — even before the user clicks the tab."Event" or "Attribute". STIX bundles contain "type":"bundle". Parse synchronously, defer large work with setTimeout(fn, 0) so the spinner renders first.window resize and document.documentElement themechange. Use a ResizeObserver on the chart container to handle the case where the chart initialised while the tab was hidden (zero dimensions).detail.html<li class="nav-item"> with data-bs-toggle="tab" pointing to #myviz-pane-[[convert.id]]. Add the matching tab-pane below. Place <my-viz :convert-data="convert"></my-viz> inside it. Import and register the component in the <script type="module"> block. Add CSS with a unique class prefix to detail.css.import JsonViewer from '/static/js/graph/jsonViewer.js'
// Singleton CDN loader
let _libPromise = null
function loadLib() {
if (window.myLib) return Promise.resolve(window.myLib)
if (_libPromise) return _libPromise
_libPromise = new Promise((resolve, reject) => {
const s = document.createElement('script')
s.src = 'https://cdn.example.com/mylib.min.js'
s.onload = () => resolve(window.myLib)
s.onerror = () => reject(new Error('CDN load failed'))
document.head.appendChild(s)
})
return _libPromise
}
function detectFormat(text) {
if (!text) return 'unknown'
const s = text.trimStart()
if (s.includes('"type":"bundle"') || s.includes('"type": "bundle"')) return 'stix'
if (s.includes('"Event"') || s.includes('"Attribute"')) return 'misp'
return 'unknown'
}
const MyViz = {
delimiters: ['[[', ']]'],
components: { 'json-viewer': JsonViewer },
props: { convertData: { type: Object, default: null } },
template: `
<div class="mvz-wrapper">
<div class="mvz-toolbar">
<button :class="{active: side==='input'}" @click="setSide('input')">Input</button>
<button :class="{active: side==='output'}" @click="setSide('output')">Output</button>
</div>
<div v-if="loading"><!-- spinner --></div>
<div v-else-if="error">[[ error ]]</div>
<div v-else ref="chartEl" class="mvz-chart"></div>
</div>`,
setup(props) {
const { ref, watch, onUnmounted } = Vue
const side = ref('input')
const loading = ref(false)
const error = ref('')
const chartEl = ref(null)
let inst = null, resizeObs = null
async function render() {
if (!props.convertData) return
loading.value = true
error.value = ''
const text = side.value === 'input'
? props.convertData.input_text
: props.convertData.output_text
if (!text) { loading.value = false; return }
const fmt = detectFormat(text)
// parse...
try { await loadLib() } catch { error.value = 'CDN load failed'; loading.value = false; return }
loading.value = false
await Vue.nextTick()
// init chart in chartEl.value...
if (resizeObs) resizeObs.disconnect()
resizeObs = new ResizeObserver(() => { if (inst && chartEl.value?.offsetWidth > 0) inst.resize() })
resizeObs.observe(chartEl.value)
}
function setSide(s) { side.value = s; render() }
window.addEventListener('resize', () => inst?.resize())
document.documentElement.addEventListener('themechange', render)
onUnmounted(() => {
window.removeEventListener('resize', () => inst?.resize())
document.documentElement.removeEventListener('themechange', render)
resizeObs?.disconnect()
inst?.dispose?.()
})
watch(() => props.convertData, v => { if (v?.input_text || v?.output_text) render() }, { immediate: true })
return { side, loading, error, chartEl, setSide }
},
}
export default MyViz
CSS & Theming
The app supports 4 themes — light, dark, dusk, ocean
— applied as a class on <html> (e.g. html.dark-mode).
All color values are CSS custom properties defined in core.css.
0.18s easeWhen the user switches theme, the app dispatches a custom event on document.documentElement. Use this to re-render charts or swap hljs themes.
document.documentElement.addEventListener('themechange', (e) => {
const theme = e.detail?.theme // 'light' | 'dark' | 'dusk' | 'ocean'
const isDark = !['light', 'ocean'].includes(theme)
// re-render charts, swap hljs stylesheet, etc.
})
Each Vue component uses a unique 3–4 character prefix for all its CSS classes to prevent collisions with Bootstrap and other components.