I Built a Fully Automated AI Agent for Social Engineering in an Afternoon

At GetReal, the threat research team doesn’t just track the weaponization of deepfakes and AI-generated media, we experiment with the tools attackers use to understand exactly what defenders are up against. 

Web-based tools now allow anyone, including bad actors, to generate an AI-driven voice avatar of a real person with as little as 30 seconds of audio, an artifact almost certainly available for most high-level officials. These avatars can be controlled by LLMs and instructed to behave however the operator wants. So I decided to see just how dangerous that combination really is.

No real users or people were contacted or harmed in this research. All testing was conducted in a controlled environment.

Building an Automated Deepfake IRS Vishing Agent

I set up an account at a popular, commercially available AI avatar website, and using the built-in context system that defines an avatar’s persona and behavior, I created a fake IRS agent designed to scam users into buying gift cards to pay fabricated tax debt. I described that I wanted the agent to impersonate a "cold, authoritative, and unrelenting IRS field officer," gave it specific instructions on what to demand, and even included handling for jailbreaking cases, like what to do if the user called out the scam or said they had no money.

The results were disturbing. The agent behaved exactly as a real scammer would. In a demanding, stern tone, the avatar insisted I send gift cards and read the numbers over the phone. Any hesitancy, whether citing lack of funds or expressing doubt, triggered more aggressive demands and threats of an impending arrest.

What made the interaction truly unsettling was its emotional intelligence. The avatar was aggressive when I wavered and reassuring when I seemed ready to comply. The psychological pressure felt real enough that I could easily see someone like my grandmother falling for it: a heartbreaking reality to live in.

This represents a fundamental shift in how threat actors operate. Previously, AI-generated voices and diction were too robotic for effective impersonation. Today, LLM-driven AI avatars have advanced to a point where we expect threat actors can soon realistically trust them to carry out social engineering in their place, without many of the technical demands of a typical deepfake attack.

In my open-source demo, the IRS agent’s voice still has a slightly flat or robotic effect. That’s a limitation of the tools I used, not a safety net for defenders. Commercial tools already sound more natural, and attackers can use them in ways GetReal, as an enterprise, cannot. Open-source voice models are catching up quickly, and so we don’t expect those remaining “tells” to be reliable for long. 

Where Current Guardrails for AI Avatars Fail

Some safeguards do exist, though. I tested tools from multiple companies and got similar results for most of them, but a few objected to my context. For example, one agent refused to even start the conversation when I joined the "call", just repeating "I can't help you with that." Another company let me use the model when I tested it but later sent me an email saying my context violated their terms of service.

However, these safeguards may not be very effective in practice. For one, they're not ubiquitous and certainly not required. The email I received saying I was in violation of the terms of service came the day after I used it, which wouldn't have prevented me from carrying out an attack. I was also able to use the same platform without any issues the next day, and other platforms had no issues with my context at all. 

Furthermore, open-source LLM-driven AI avatars exist and are available to be run locally, meaning that there are no safeguards in place to prevent a bad actor from carrying out a malicious attack.

Building A Malicious AI Agent with Open-Source Tools

I built an avatar like this from the ground up to prove the point. I wanted to understand the technical barriers (or lack thereof) that currently exist that would prevent threat actors from using these tools to carry out a malicious attack. I wanted to simulate a technical stack that could be created using only free, open-source tools.

I started with OpenAI’s Whisper to bridge the gap between speech and text, and, at first, used the open-source Qwen 3 chatbot as the 'brain' to generate intelligent responses. Interestingly, I was initially able to customize the way Qwen responded using a similar built-in context feature. However, when I tried a few weeks later, the newer model gave me a warning, telling me that my phishing context wasn’t real and that the IRS would never try to extort me like this. To combat this, I used the open source model Deepseek v3 to generate the types of responses that I needed with minimal or no guardrails. I also found some commercial LLMs with weak guardrails that worked as well. 

I then used Kokoro, an open-source text-to-speech model, to convert the LLM-generated text into realistic speech. In choosing the text-to-speech model, I had to ensure that my pipeline had an optimal balance between low latency while also sounding human. These tools alone could be used to engage in fully automated AI voice phishing. It is easy to route the audio input and output to a desktop VoIP phone and have my agent place calls. 

One of my primary observations from this venture was just how convincingly human these off-the-shelf AI avatars can look and sound. Combined with customizable contexts, threat actors can easily impersonate executives, federal officers, and insurance agents with such realism that most people can't tell the difference. And when run locally, the already ineffective safeguards are completely eradicated, paving the way for these attacks to happen on a regular basis.

We're entering an era where verification becomes mandatory and trusting a voice alone becomes impossible. The question is no longer whether these tools can fool us. My experiment confirms they can, and so organizations need to consider AI-powered identity attacks in their threat model and update controls accordingly.

What Defenders Should Do Next

• Warn employees and customers of deepfake vishing and teach them to recognize fake authority, urgency, and unusual payment methods, such as gift cards, as signals of social engineering
• Verify high-risk phone requests, such as credential reset calls to the IT service help desk, through a call-back on a known number or an internal communications system
• Implement real-time deepfake detection and continuous identity verification on live calls and meetings so protection is automated and scalable, not dependent on ad-hoc checks
• Treat unexpected calls from “government agents” as high-risk by default

How prepared is your organization for deepfake threats? Download the Deepfake Readiness Benchmark Report to find out. 

‍