MidnightShark

Desktop AI Co-Pilot (Intellify 3.0)

A proactive, multi-modal Desktop AI Assistant built during the Intellify 3.0 National-Level Hackathon, securing the Second Runner-Up position.

·completed
AILocal LLMPythonReactHackathon

Project Overview

For the Intellify 3.0 National-Level Hackathon at Marwadi University (August 30–31, 2025), our team built a Desktop AI Assistant. We secured Second Runner-Up against teams from across the country.

Rather than building a standard web-based chatbot, we engineered a universal, proactive co-pilot that lives as an overlay on the user's desktop, capable of seeing the screen, hearing voice commands, and automating repetitive workflows.

Key Features

  • Universal Overlay: Works across all apps and windows natively, untethered from a single ecosystem.
  • Proactive Assistance: Suggests actions without being asked (e.g., identifying a deadline in an email and automatically offering to set a calendar event).
  • Multi-Modal Intelligence: Understands text, voice, and on-screen visuals simultaneously.
  • Live Screen-Sharing Vision: The AI can "see" the user's screen in real time to provide contextual assistance.
  • Smart Clipboard: Automatically processes copied text or images and offers instant, context-aware actions.
  • Task & Workflow Automation: Fills forms, navigates menus, observes repeated actions, and learns to automate them.
  • Voice-Enabled: Seamless, promptless usage via speech-to-text.

Technical Approach

We architected the system to run highly capable AI locally to ensure privacy and speed, falling back to cloud APIs only when absolutely necessary.

Frontend / Overlay Layer

  • Electron.js / Flutter Desktop: Provided the cross-platform, transparent overlay interface.
  • React.js + Tailwind CSS: Powered the modern, responsive UI elements.

Backend & Core Logic

  • Python (FastAPI / Flask): Handled the core orchestration and AI inference requests.
  • Rust / C++ (llama.cpp / Ollama): Deeply optimized for running small, quantized LLMs locally.
  • Ninja Build + CUDA: Leveraged for GPU acceleration to achieve fast inference on an RTX 3050.

AI / ML Stack

  • Local LLMs: LLaMA 2 / Mistral 7B Instruct (quantized to Q4) for fast, context-based replies.
  • Whisper.cpp: Handled high-accuracy, on-device speech-to-text for voice commands.
  • CLIP Models: Used for screen content recognition and multimodal visual inputs.

Data & Integration

  • SQLite / LiteDB: Provided lightweight, local context and memory storage.
  • Windows API / Accessibility API: Enabled the screen overlay capabilities and cross-app automation.
  • Screen-capture API: Captured on-screen content for real-time analysis.

Challenges & Mitigations

Building a heavy AI application during a hackathon came with significant risks. We proactively managed these:

  1. High Resource Usage: Running local LLMs can freeze lower-end PCs. We mitigated this by heavily relying on quantized models (7B Q4) to balance speed and memory efficiency.
  2. Privacy & Security: Because the app reads emails and files, we implemented a Local-First Privacy architecture. All sensitive processing stays strictly on-device.
  3. Accuracy & Automation Risks: An AI misinterpreting a command could automate the wrong task. We implemented a Human-in-the-Loop system, requiring explicit confirmation prompts before executing any sensitive automations.
  4. User Adoption Friction: A constantly watching AI can be overwhelming. We focused on a Simple UI/UX with overlay hints, tutorials, and adaptive learning to reduce user friction.

Results

The project was highly praised by the judges for its technical depth, privacy-first architecture, and ambitious scope, ultimately earning us the Second Runner-Up position at the national level.