SIH26171/Smart Automation

Browser extension with local vision redaction

A browser extension and server that runs a local vision model to read screen states, redacts personal data on the client side, and sends anonymized data to a server for task processing.

Indian Space Research Organisation(ISRO) · Department of Space / Indian Space Research Organisation · Software

Teams already in: 1/500almost everyone who enters · read 1 hour ago
Buildable in 36 hours: HighOur estimate, from their wording
Hard blockers: NoneNothing you must source first
Submissions close: 15 Sept9 days left

Make this one of our two Compare with others

Signing in saves it for your whole team — everyone on your invite link sees the same two entries. Anything you shortlisted while signed out comes with you.

Submit on sih.gov.in Dataset

Whether this suits your team

Why this fits you

This organisation declared winners on 11 problem statement(s) in SIH 2025.

What you would have to build

Decomposed from what the description asks for. Nothing added.

01
Local Vision Processing
Runs a client-side vision model in the browser using WebGPU to evaluate screen states.
02
Privacy Preserving Filter
Sanitizes sensitive data locally through bounding-box redaction, blurring, or masking.
03
Server Side Integration
Transmits anonymized visual context to a centralized LLM or VLM to interpret data and return actions.
04
Action Execution Engine
Executes UI actions returned by the server on the local browser client.

What they ask for, and what they leave open

Both columns are read off the brief's own wording. Nothing here is inferred from the ministry's name.

They state

The evaluators will check the accuracy of the visual context from the screen, precision and recall for detecting sensitive data, precision of the redaction, client-side resource utilization, and overall end-to-end task latency.

They never say

specific benchmarks or datasets to test the accuracy
exact performance thresholds for resource utilization

A jury can still ask about these. Decide them deliberately rather than by accident.

What they'll ask you

Generated from the brief's own wording and the competition's published rules — never from a guess about what this ministry prefers.

Specific benchmarks or datasets to test the accuracy?
The brief never answers this, so a panel will. Whatever you decide, say it the same way twice.
Exact performance thresholds for resource utilization?
The brief never answers this, so a panel will. Whatever you decide, say it the same way twice.
Has any part of this been shown at a previous event, hackathon or college project?
The guidelines are explicit: your solution must not have appeared in any previous event or programme, of any sort. A recycled project is what a team under time pressure reaches for.

7 more— the ones a panel gets to if there is time

The brief asks for local vision processing. How would you build that?
Decoded from SIH26171 itself — Runs a client-side vision model in the browser using WebGPU to evaluate screen states. The brief asks for it by name.

The data

What the organisers attached, and what the brief assumes you can get.

Dataset from the organisers: Provided — open it
Does the brief need data?: Yes, and they gave it to you.

Also from Indian Space Research Organisation(ISRO)

Same organisation, same year. Reading two of theirs tells you more about what they care about than reading one.

Take this to an AI

Pick what you are about to do and copy the prompt. It carries the organisers' own wording, the constraints they never spell out, and an instruction not to invent requirements they never set.

Who has this problem, what already exists, and what you would have to find out.

# Context: India's Smart India Hackathon 2026

I am on a student team entering SIH 2026. Problem statement SIH26171, set by Indian Space Research Organisation(ISRO).
Its official title, word for word: "On-device Visual Perception for Light-weight Browser Agents"
Theme: Smart Automation. Category: software.

In plain terms: A browser extension and server that runs a local vision model to read screen states, redacts personal data on the client side, and sends anonymized data to a server for task processing.

## The organisers' full description, verbatim

Background AI agents are becoming omnipresent in the current era and can play an important role in our digital interactions. If an agentic AI pipeline has access to our visual context, screen states, they can assist users in complex workflows and automate many tasks. Most of the agentic AI pipelines are deployed on server side which limits the type to data that a user can share with it. It would open a new dimension of possibilities, if a local agent is deployed on user machine particularly browser which can eliminate the need to share the sensitive data with the server. Local system generally has fewer resources than server and is unable to host a full-fledged pipeline therefore only the non-sensitive data such as structure of the screen, application fields etc can be sent to server for processing. Modern browser APIs (such as WebGPU and WebAssembly) and local inference libraries (like ONNX Runtime Web and Transformers.js) have unlocked the ability to run lightweight machine learning models directly on the client. The aim is to bridge these two environments: leveraging the reasoning power of cloud or server based AI while strictly enforcing data privacy at the client side. Description Participants are required to build a privacy-preserving vision agent which runs on browser. This involves implementing a client-side architecture where a local Vision Transformer (ViT) or equivalent computer vision model 'reads' the user's screen and takes decision based on that. If it requires the visual context to be sent to server, it shall sanitize the sensitive/PII data using DOM tags or any other method, before any network request is made. It should dynamically detect and redact sensitive elements. For example, blurring faces, blacking out passwords, and masking PII etc. Only this anonymized, unidentifiable data should be transmitted to the central server which should be aware for this redaction scheme and can process data accordingly. The server will then process the sanitized context and return actionable commands for the browser agent to execute. Participants must balance the trade-offs between inference latency and the accuracy. Expected Solution A successful submission should include a working prototype consisting of client side extension and server that demonstrates the following: Client-side (extension/JS) running in popular browsers (chrome, Firefox) components: • Local Vision Processing: Implementation of a client-side vision model running in the browser (e.g., via WebGPU) that evaluates the current screen state. • Privacy Preserving Filter: A mechanism for sanitizing sensitive or personal visual data. This can be achieved through local bounding-box redaction, semantic obfuscation, masking etc. This should be clearly demonstrated. Server-side implementation components: • Server Side Integration: The transmission of the anonymized visual context to a centralized LLM/VLM, which successfully interprets the sanitized data and returns the response which may be processed data to be again ingested by local client or an UI action (e.g., 'click the submit button,' 'scroll down') that the local client executes. • Participants are free to use any offline deployable (open-source/open-weights) model on server side. During SIH they can use cloud hosted version of these. An end-to-end task assisting the user should be demonstrated. Evaluation will be done on the following metrics: 1-Accuracy of visual context from screen â€“ 25% 2-Recall and precision for detection of sensitive/PII data â€“ 20% 3-Precision of redaction â€“ 20% 4-Client side resource utilization â€“ 20% 5-Overall end-to-end latency of the provided task -15%

## What the brief never says

- specific benchmarks or datasets to test the accuracy
- exact performance thresholds for resource utilization
## Constraints that are not negotiable

- The judged build is a prototype demonstrated in roughly 36 hours by students, not a product.
- A jury evaluates it against this problem statement's own wording, not against general merit.
- We submit on sih.gov.in; the problem statement number and title above must match exactly.
- Your solution must not have appeared in any previous event or programme, of any sort.
- The organisers attached a dataset, so build against theirs rather than inventing one.

## What I want from you

1. Who actually has this problem in India, and what they do about it today.
2. What already exists — government systems, open-source projects, published research — that we should not rebuild.
3. For each item under 'what the brief never says', what the realistic answer is likely to be and how I could verify it.
4. The two or three things about this domain that a student team would get wrong without local knowledge.
## How to answer

- Everything above between quotation marks is the organisers' own wording. Treat it as the requirement.
- Do NOT invent requirements the brief does not state. If something matters and is not stated, say so and ask me, rather than assuming.
- Where you are inferring rather than reading, mark it as your inference.
- Do not guess what this ministry wants from its name. Its wording above is the only evidence.

---
Problem statement facts assembled by Wynit from Indian Space Research Organisation(ISRO)'s published brief for SIH 2026.
The plain-English summary, the list of parts and the evaluator read are our interpretation of that text; the quoted title and the full description are theirs.

The government's original description (557 words)

Background AI agents are becoming omnipresent in the current era and can play an important role in our digital interactions. If an agentic AI pipeline has access to our visual context, screen states, they can assist users in complex workflows and automate many tasks. Most of the agentic AI pipelines are deployed on server side which limits the type to data that a user can share with it. It would open a new dimension of possibilities, if a local agent is deployed on user machine particularly browser which can eliminate the need to share the sensitive data with the server. Local system generally has fewer resources than server and is unable to host a full-fledged pipeline therefore only the non-sensitive data such as structure of the screen, application fields etc can be sent to server for processing. Modern browser APIs (such as WebGPU and WebAssembly) and local inference libraries (like ONNX Runtime Web and Transformers.js) have unlocked the ability to run lightweight machine learning models directly on the client. The aim is to bridge these two environments: leveraging the reasoning power of cloud or server based AI while strictly enforcing data privacy at the client side. Description Participants are required to build a privacy-preserving vision agent which runs on browser. This involves implementing a client-side architecture where a local Vision Transformer (ViT) or equivalent computer vision model 'reads' the user's screen and takes decision based on that. If it requires the visual context to be sent to server, it shall sanitize the sensitive/PII data using DOM tags or any other method, before any network request is made. It should dynamically detect and redact sensitive elements. For example, blurring faces, blacking out passwords, and masking PII etc. Only this anonymized, unidentifiable data should be transmitted to the central server which should be aware for this redaction scheme and can process data accordingly. The server will then process the sanitized context and return actionable commands for the browser agent to execute. Participants must balance the trade-offs between inference latency and the accuracy. Expected Solution A successful submission should include a working prototype consisting of client side extension and server that demonstrates the following: Client-side (extension/JS) running in popular browsers (chrome, Firefox) components: • Local Vision Processing: Implementation of a client-side vision model running in the browser (e.g., via WebGPU) that evaluates the current screen state. • Privacy Preserving Filter: A mechanism for sanitizing sensitive or personal visual data. This can be achieved through local bounding-box redaction, semantic obfuscation, masking etc. This should be clearly demonstrated. Server-side implementation components: • Server Side Integration: The transmission of the anonymized visual context to a centralized LLM/VLM, which successfully interprets the sanitized data and returns the response which may be processed data to be again ingested by local client or an UI action (e.g., 'click the submit button,' 'scroll down') that the local client executes. • Participants are free to use any offline deployable (open-source/open-weights) model on server side. During SIH they can use cloud hosted version of these. An end-to-end task assisting the user should be demonstrated. Evaluation will be done on the following metrics: 1-Accuracy of visual context from screen â€“ 25% 2-Recall and precision for detection of sensitive/PII data â€“ 20% 3-Precision of redaction â€“ 20% 4-Client side resource utilization â€“ 20% 5-Overall end-to-end latency of the provided task -15%