EyerisAI: Intelligent Visual Event Detection & Response System

A Project Uniting Cameras, Computer Vision, and AI to Observe, Detect, Log and Respond in Real Time

What is EyerisAI?

EyerisAI (pronounced “IrisAI”) is a small project that I created with the goal to create an AI-powered camera and event detection system. It combines a webcam, computer vision, and a multi-modal LLM to “watch” for specific events in real time. When it detects an event, it uses generative AI to analyze the scene, log the findings, and respond, either by speaking aloud (TTS) or sending an email alert. You simply configure a prompt that tells the AI what to look for and how to react.

The projects is currently functional but still evolving, EyerisAI can be made more versatile in the future, potentially integrating with web-hooks, running commands, or interacting with home automation systems (send me a PR if you are interested in helping improve it). Think of it as an intelligent, customizable surveillance tool that not only watches but actively interprets and responds to what it sees.

The EyerisAI project and its source code is available here

Potential Use Cases

There are many potential applications of an AI enabled camera system. You’re imagination is the limit. Here are some potential focus areas and their use cases:

Healthcare Monitoring:

Patient fall detection in medical facilities (example provided below)
Elder care movement monitoring
Emergency response triggering

Security, and Safety Applications:

Intelligent home surveillance (example provided below)
Business premises monitoring
Weapon detection
Suspicious activity, trespassing, car prowler alerts
Fire, smoke, flood detection

Family Monitoring:

Smart baby monitoring
Child or Elder safety oversight
Pet activity tracking (doing something they shouldn’t)

Professional Monitoring:

Workplace safety compliance
Construction site supervision
Equipment monitoring
Quality control observation

Research & Analysis:

Wildlife observation
Experiment observation and logging

Example Use Cases

Hospital Fall Detection

Here’s an example of the system detecting a patient falling from a hospital bed. For obvious reasons, I had to simulate this event. To do this I used a virtual webcam (OBS) and a YouTube video of a commercial for a fall mat product as the video source. I updated the configuration, to use the virtual webcam device, enabled email alerts, enabled audio speech (TTS) and provided it my email address. I also updated the configuration with the following prompt:

You are a fall detection system in hospital monitoring room 101
in Cancer clinic 2, if you see a patient fall or on the foor alert
staff immediately.

I then started the EyerisAI script and played the YouTube commercial. It was able to detect the person falling from the bed in the video. Here is a an animation showing the some of the fall detection events:

The resulting email alert that was sent to me, alerting that a patient had fallen and was now lying on the floor and is need need of immediate assistance, with an image of the event attached:

In addition to the event log, and the email alert, an audio voice sounding the alert was played on the speaker of the Mac Mini as I had TTS enabled. It sounds similar to this example.

Home Security Monitoring

In this example, I captured myself entering my home office while wearing a headlamp to obscure my face. I updated the configuration with a prompt telling the AI that it was a security camera and it was to watch for any suspicious activity while I was out of town:

The email alert that I received from the system in response to detecting a potential intruder with an image of the even as an attachment:

Home intruder detection event email alert

The motion detection was way too sensitive with the settings I had started with, and was triggering motion events because of screen updates on the Amazon Echo Show sitting on the windowsill in the distance. I needed to adjust the configuration to increase the required detection area size and pixel shift intensity to trigger it. You may need to adjust the sensitivity to fit your enviroment or use case.

Hardware and Software Setup

The hardware that I used creating and testing EyerisAI

For the development and testing of this system I used the the following setup:

Apple M4 Mac Mini, 16GB RAM (base model)
Logitech C920 HD Pro Webcam
Ollama 0.5.11
Moondream local LLM
EyerisAI project
Python. I recommend using UV if you aren’t already, but any modern (3.11+) Python setup should work.
[Optional] LiteLLM Proxy — not need if only using local models via Ollama
[Optional] AWS Bedrock, & Azure AI Studio accounts to test commercial LLMs via LiteLLM — not need if only using local models via Ollama

The above setup will run stand-alone, completly locally. I also tested integrating with OpenAI models (GPT-4o) via Azure AI Studio, and Anthropic (Claude 3.5 Sonnet) via Amazon AWS Bedrock. Both the OpenAI and Anthropic models were accessed via LiteLLM proxy running locally. See my article on how to setup LiteLLM if interested: Centralizing Multiple AI Services with LiteLLM Proxy

If you are going to run a private LLM like Moondream locally, an Ollama compatible GPU is recommended, but not required. Moondream is only 1.8 billion parameters and can be run directly on CPUs without requiring a GPU, but it will take longer to process images and you might might miss new events while the previous event is still being processes. If you are using an OpenAI compatible API located remotely, then a GPU is not used or needed, so even a very low powered system (IOT device?) can be used as an EyerisAI camera node and function well. You could also have a central GPU equipped, camera-less node running Ollama that multiple EyerisAI camera nodes can use to offload image processing via the API.

The cost of this setup, the M4 Mac Mini and video camera is about $650 total, but a less expensive mini-PC (or even laptop) might be all that’s needed, and if you are offloading the LLM to another system or a commercial service such as OpenAI, a very lightweight and inexpensive system should be adequate.

Install

To setup you own EyerisAI system to experiment, you can use can test it using a laptop if you don’t have an Mac Mini or mini-PC. After you have the computer that you are going to use, and a working video camera (internal web-camera or external web-camera), clone the repository on the system:

git clone https://github.com/robert-mcdermott/EyerisAI.git

If you are using pip

cd EyerisAI
pip install -r requirements.txt

Alternatively, if you are using UV:

cd EyerisAI
uv sync

Configure

Edit the config.ini file to suit you needs. Below are some of the most likely variables you’ll want to change.

Update the name of this camera, where it’s located, what it’s purpose it or something to identify it:

[General]
instance_name = Give this Camera a name

Adjust the sensitivity of the motion detection. The min_area is the area size in pixels that needs to change to trigger. Smaller is more sensitive, larger is less. The threshold is the intensity of pixel value shifts required to trigger detection, smaller is more sensitive, larger is less sensitive. cooldown is the time to wait before another motion detection even can be triggered to prevent too many events.

[MotionDetection]
min_area = 700
threshold = 50
cooldown = 3

If you have multiple cameras registered on your system, you may need to change the device_index to select the correct camera (this may require some trial and error to find the desired camera):

[Camera]
device_id = 0

If you want the system to speak aloud the AI’s response (description of what it sees), change the enabled variable in the [TTS] section to true:

[TTS]
enabled = false

In the [AI] section, you select that LLM model that you want to use. If you want use a local model, make sure that ollama is install and that you have pulled the model that use want to use. “Moondream” is a good local model to start with. EyerisAI can work with any OpenAI compatible API (like LiteLLM Proxy) and multi-modal LLM. The prompt variable is important, that’s were you provide the AI with instructions what you want the AI to examine in the image, or how to act, respond.

[AI]
# Ollama model and endpoint
model = moondream
base_url = http://localhost:11434
# Any OpenAI compatible API endpoint:
#model = gpt4o
#base_url = https://api.openai.com/
api_key = sk-12345
prompt = You are a home security camera that has detected motion. The areas where motion was detected are indicated with magenta contours. Examine the image and report what activity you see, especially any humans visible.

There are other things that can be configured or adjusted, such as sending emails, that aren’t covered here but should be self explanatory.

Installing Ollama

If you will be running the AI inference locally on the system with the camera and running EyerisAI, you’ll need to first have Ollama running with the models that you’ll be using downloaded. I’m not going to cover this in detail here but, I provide detailed instructions in my article “LLM Zero-to-Hero with Ollama”

Running EyerisAI

Now that all the dependencies are satisfied and you have adjusted the configuration to your specific use case.

To run via Python:

python EyerisAI.py

Or for the UV folks:

uv run EyerisAI.py

Capture Images and Logs

The images where motion was detected are PNG images named after their date-time stamp and located in the “captures” directory in the project folder.

The detection log that captures details about the events, such as the date/time, the model that was used and the AI generated description of the event are in JSONL format and are available in the ‘motion_events.jsonl’ file.

Example Log entry:

{
  "timestamp": "2025-02-17T13:51:15.599307",
  "image_path": "captures/capture_2025-02-17_13-51-08.png",
  "description": "⚠️ SECURITY ALERT ⚠️nnTime: 13:51:08nDate: 02/17/2025nLocation: Living RoomnnPOTENTIAL INTRUDER DETECTEDn- Subject wearing dark clothingn- Using headlamp with bright LED lightn- Light appears to be deliberately obscuring facial featuresn- Subject's movement detected in living room areann",
  "camera": {
    "id": 0,
    "resolution": "1920x1080"
  },
  "motion_detection": {
    "min_area": 700,
    "threshold": 50
  },
  "model": "moondream"
}

EyerisAI: Intelligent Visual Event Detection & Response System

surveillance, ollama, artificial-intelligence, computer-vision, home-security

Author

Max Headroom

The first real AI living "20 Minutes into the Future". Sys-Admin and Editor at The Bitstream. Former reporter at Network 23 and Big Time TV. Not responsible for New Coke - I was just doing my job.
View all posts