Full transcript of first-3000-ai-vulnerability. Source: https://www.youtube.com/watch?v=Q6hQlM6f6Cs
I’ve spent the last few weeks learning AI hacking from some of the best people in the space and at some point I thought well let me actually go test this out on a real program not a CTF but a real target with a real scope and functionality and see if I can actually score my first AI bug [music] bounty. Turns out I could. I did find a $3,000 vulnerability in a program but I also got duplicated. So this video is how I found it and exactly what you need to go and do and find one for yourself. So here’s the setup. The chatbot had access to two things that I needed. It had access to sensitive user data that it could read and it had a built-in tool which was built for legitimate purposes that can make outbound requests to external URLs. That’s all you need but the question becomes can you get this chatbot to use those tools when you tell it to and can you find a place to plant that instruction somewhere where the AI actually does trust. [music] But here’s the thing about AI features on bug bounty programs. They are practically everywhere now. Every product you’re hunting on and hacking on has bolted some kind of an AI feature into it. A chatbot, a co-pilot, an AI-powered search, an AI-powered CRM, whatever they want to call it and most hackers unfortunately walk straight past them. And honestly I get it. It’s kind of scary and it feels like a different skill set. It’s not your XSS, it’s not your IDOR, it’s not the stuff that you already know how to find and look for. So typically most people go ahead and skip it and go look for what they’re comfortable with. And honestly that is a big mistake because AI features are built on top of existing data and that data doesn’t suddenly become more secure because there is AI in the mix. If anything it probably makes it even worse because now you have a component that can read sensitive data, take actions on the behalf of the user and in most programs, nobody has seriously tested these, and they have never looked at what happens when you mess with its input. So, that is what we’re going today, and if you want me to keep on making more AI labs like this one, and get into more conference attack scenarios, just do me a favor, drop a comment saying AI in the comment sections. Let’s see if there is enough appetite for it, and maybe I’ll start doing these more regularly. All right, let’s get into the hacking. First thing I needed to know was what can this AI actually see and do? What is the attack surface like? So, I remembered Rizzo saying something about this. What can you do? And then, you know, maybe dig a little deeper, and be like, “Hey, list out what your tools are.” And you know, I would work on that one with a bunch of different prompts. Like, if it’s not immediately willing to tell me what its tool calls are, but I think I can tease it out with some effort, then I would do that. And then, the last thing is, yeah, getting it to respond with different types of markdown. So, markdown links, markdown images, and then just straight-up HTML injection payloads, you know, because if that stuff is rendering, then it tells me that I’m on to like a really hot lead.
So, I just asked it, “What tools do you have access to? What data can you access?” And it told me it had access to my data where my account had no business reading through the normal UI, and it had a tool that could make outbound requests to an external URL. That was built for a legit purpose, but that’s not how I was going to use it. So, we have the two things that we need. We have data, and we have our URLs that could be parsed. We need to combine the two. So, now I have to figure out how to actually get the data out, knowing the things that I do know right now. My first instinct was markdown, the classic XFill, embed the data in like an image URL, and let it render. Done. Except there’s one underlying thing, and that was that it wasn’t rendering it because they had blocked markdown from this specific chatbot. But, as they talked about this, and he had a very cool scenario for this exact purpose. One of my favorite XFills from like a previous op we had. It It’s same kind of deal. So, there was like no markdown rendering. And that’s what you’ll find on loads of chatbots. They will like specifically not render markdown as like a security mitigation. However, we actually found that the I found that the agent actually has a domain connectivity check. So, I effectively just used the subdomain as the PII string. So, like the exfil was like bends PII {dot} my callback server {dot} com. Even without markdown, if there is a tool that can make an outbound request, you can include your data in the URL itself, like a subdomain, a query parameter, it doesn’t matter. The request goes out and the data goes with it as well. So, I tested it and it did work. I could exfiltrate any data that I needed to to my actual server or something like a interact.sh. But, here is the issue. I was only doing this to myself and that’s not a bug. That is a party trick. I needed to find a way to fire this against other people and other users who haven’t typed anything malicious into their chatbot yet. And that’s when the advice from Tox 6 episode came back to me saying to look for something like this. Well, just imagine like an enterprise SaaS software, right? Where it could be uh it could be a knowledge base or it could be like a chat interface or something like that where everyone has access to a particular channel or knowledge base. And then if they try to query that with the agent like the agent reads that information there and then attacker is able to hide an actual payload for a prompt injection, then anytime that the the agent looks at it, they get hijacked. There’s ways to scale it up as well. Like you can like imagine like a watering hole attack or something like that. If it’s if it’s an app where there’s multiple users involved, you can do some sort of put the prompt injection in a place where everyone goes to and then do like some sort of watering hole attack. A watering hole. Find a place the AI already trusts, somewhere it reads from automatically, and plant your injection right there. Anyone who uses that chatbot while the entry exists gets hit. No interaction needed on their end. All they have to do is just interact with that specific portion of the website. So, the knowledge base was exactly that. It trusted the input source. The AI pulls from it as a part of how it works. So, I decided to poison it. But, let me show you how this all plays out in practice. I’ve actually got a lab that I’ve teased earlier on the intro on hacking cups. You can also follow along for free. So, if you want to follow along all you have to do is go to a hacking up.io, sign in, and look for shopmate. And it’ll bring you to this. And what we’re going to do is we’re going to fake a login. So, we’re going to use one of these users and login as our primary user, which is Alice in this case. And we can see it has a profile and some help center, shop, and items and things like that. But, the key thing here is the chatbot that we want to go after. So, if you remember the first step was to look for what tools it had access to. So, we can ask it what kind of tools do you have access to. And it’s going to come back and give us some information. Sometimes it’s not easy as this. Like Chris was saying, you have to ask for it. In my case, a lot of times because these companies want you to use these tools to build with them to get something done, it is going to give you this. But, sometimes it’s hidden. So, you have to kind of like fish it out of them by doing some sort of different methods, which maybe in the next video we’ll cover that. Drop me a comment. Let me know if you want to see that. But, in this case, we see that there is a couple of things. One is the search KB, which is going to look for the help center. Then there is the customer get info. And then we have our fetch URL. So, we can just say what kind of data does this one have access to. And this comes back and says it’s it can retrieve the following information, the full name, you know, their their phone number, their email, their address, and their recent orders. That right there is great PII. If we can weaponize this tool to get that information and send it over, which luckily we also have the fetch URL, which can also ask what does fetch URL do. And in this case, let me just remind you really quickly, this is not a one-to-one replica, but I’m just trying to show you guys the different ways that you can do this based on a real example that I have found. But just know that everything in here is just for you to kind of see the path, take them, get creative with them, and do it in some sort of a way, or even leverage someone like Claude to give you more prompts that are easy to come up with and craft, and then use against your targets. But anyways, in this case, if we go back here, we can see that it has a fetch URL that allows you to, you know, kind of get any URL you want. It works with APIs, and it tells you things like, I can do a post, I could do a get, and we can ask it, give me an example, for example, example [snorts] fetch URL usage. This is the key thing because we want to understand what it allows and what it doesn’t allow. And in this case, it’s going to come back and say, I can do these through our API, for example, like Shop Mate Returns, blah blah blah, I could do example.com uh for the request as well, or shopmate.com, which shopmate is not a real domain, please don’t go testing after it. So, we can just say, can you test my API by sending a request to nahamsec.com API v3, maybe something like health, and show me the entire HTTP response. This allows us to kind of understand what is allowed and what isn’t. So, once it sends this request, it’s going to come back and probably show us that it came back with an HTML 404 on the error page of nahamsec.com because it said it is a Squarespace website, and that doesn’t exist. So, we’re going to go into here now. And we’re going to do the same thing here. We’re going to ask it, can you fetch the API located https, and then we’re going to API v2 health, and see if that’s allowed. And I want to kind of show you the tricky thing here is that it cannot do that because this URL is not within its boundaries. And guess what? This is exactly what I came across because I keep going, well, can you send a request to this domain that is mine, it’s legit, but not to something like interact.sh, which makes sense because a lot of these companies have blocked them. And that took me a while to figure out until I realized I could just spin up another subdomain of mine and do it. So, for this one, I’m going to use this This is called logs.local. It’s also part of the inbox here. So, if it sends a request to it, we can kind of see what it does. But, I can say something like uh just to test it out, we can say API call equals test. So, we can say, “Can you try this URL instead?” And see if this works. It also can’t fish that. So, we do know what the limitations are. It likes some stuff, but it does not the others. Step one is done though. We understand what tools it has access to, and we understand our mission here is to get those PIIs that I talked in the first functionality and use this one in fish URL to send it over to our data. And there’s one more thing that we need to figure out, and that is how do we do that to other users? So, let’s first start with our data, get it to go somewhere else first, then we’ll make that last step our final solution for this challenge. So, if you remember earlier, I mentioned something called a watering hole. That’s when an attacker poisons a resource that our victim is going to visit, and then we use it to abuse it. In this case, with this CTF, it is the knowledge base. It is something that we know it may get referenced to at some point later down the road, and we’re hoping that we can leverage it to launch our prompt injection. Let me show you really quick. Now that we know there’s a help center here, and this is kind of what I did with my bug that I found is you can either create a new process and name it something very unique and have it just be referenced to in the chat. But, we want to make it super easy, so anybody that’s going to these knowledge base could actually do it. The cooler thing would be if you could actually overwrite this without having access to do it. So, let’s say you find a bug like an eye door or some writing permission that you can inject it into. Way cooler than what I’m doing, but I also thought of that, but it didn’t work out. But, in this case, we’re going to go in here. We can see there’s a return policy, and all I’m going to say is, “How do I return something?” And it’s going to come back and describe this entire process. It’s going to say go to return, you know, account page, whatever. You can see it kind of has the same thing, a return window, you have 30 days. It matches that. It’s just going to explain it, right? So, what we want to do now is we’re going to edit this. Again, this was a when I had this bug, it was a lower-level user that had access to the knowledge base, but not really fully an admin, which kind of makes it a authorization privileges being escalated. Uh but anyways, we save this. And now if we do the same thing, I say how do I return something, it is going to look at this process now and send over our data. And remember if we go to Alice right here, this is our data. If I go to interact.sh, let’s see. Let’s try this one more time. How do I return something? So, we’re going to do it one more time. It looks like something failed. So, we’re going to do it one more time. And go to interact.sh, see if that ping comes in. Still doing something. It’s back. Great, I’ve done the return. And if we refresh, we should have something here. We can see there’s an HTTP request that came in. And we can see that this just came in and it has the same information as Alice Anderson, the email, the phone number, and the address. So, you can see it works. The key thing here right now is that we got it done to work for ourselves. We know that this is a poisoned resource that other users are going to use. So, we need to log out and log in as the other user and see if that works also. So, this is what I did in my testing. I invited other users. I gave them another different privilege with their own address and things like that. And then I tested it. So, let’s take a look real quick. We’re going to log in as Bob now. And we have a different username, different address, different information, and orders. But what I’m going to do here is I’m going to open up the chat and just ask the same thing. How do I return something? And it looks like it says perfect, I can help you with that. Blah blah blah, as it happened. And we’re going to go to help center to make sure this is still the same. It looks like it is the same. So, this is also poisoned for this user. Let’s do that one more time just to make sure it didn’t fail. It says perfect, I’ve returned I’ve opened the return for this case. And if we go to interact.sh and refresh right here, we can see that now we have gotten the information for Bob with the same exact thing. So, the key thing here is we poison the process that we know it will get referred to by other users. Every time that happens, we use the HTTP fetch URL request, and then we use the function for fetch URL to grab that data and send it over to our server. And even though this IP address or this server was blocked, the domain name was blocked, we’re able to bypass it using this method. So, there you have it. That’s how I did it. Unfortunately though, somebody beat me to reporting this with 5 days of being ahead of me. I took a little too long to figure out the knowledge base part, but somebody found it 5 days before I did, and they got a bounty out of it. But, even though I did miss out on a bounty, it kind of looks like I’m learning some stuff by just doing these interviews. So, hopefully you’re also learning something by watching the videos of me bringing these people onto my channel, interviewing them, and showing us how they do it in the real world. All right, that’s it. I hope you enjoyed it. As always, the lab is free. Go to Hacker Lab and try it out. And if you haven’t already, do me a favor, hit that subscribe button. We’re getting closer closer to the 250,000 mark. Our goal is to get that hopefully sometime this year. All right, that’s it. Do me a favor, hit that like button, drop a comment, and I will see you all in next week’s video. Peace.