Full transcript of claude-skills-for-bug-bounty. Source: https://www.youtube.com/watch?v=qTX9u-EsjmM
But, as you start talking about these things, I just sit here and I start churning. You know, and and my brain’s like, “This is what you’re going to do right after you get off this freaking podcast, you know?” Like Best possible hacking when you can just, you know, critical thing, right? Some of the most fun I’ve had this year hacking already was with you guys in the CTBB Adobe hack-along that we did a couple weeks ago. And the results were great. We found four figures worth of bugs in just 2 hours. And I just want to shout out Adobe for being such an amazing program, for sponsoring the podcast, for staying involved in the bug bounty community. Um and I just wanted to say, if you guys want to check out a new program, the Adobe program is awesome. They’ve got scope for days, guys. They’ve got binaries, they’ve got open-source projects, they’ve got enterprise products, they’ve got web apps, they got wild cards, they got mobile apps. Anything you would want to hack on, the freaking Adobe program has it. The bounties are competitive. And I mean, just look at the um look at the thanks tab on HackerOne, right? And filter for this year. There’s already uh somebody with 620 reputation uh from Adobe, right? So, this guy’s out here like, “Don’t tell them about Adobe. I’m killing it. I don’t want anybody to know about it.” Um so, yeah. It’s a program ripe for opportunity, especially in the area of AI hacking, cuz they’re shipping a lot of AI-related code right now. And they really value AI submissions coming through the bug bounty program. So, if you want a new program to hack on, I recommend Adobe. They’re great. Sup hackers, we got the This Week in Bug Bounty segment real quick. First item on the docket is Intigriti has launched their ambassador program. So, if you’re deep in the uh Intigriti ecosystem and you want to take that a step further, I have it um you know, firsthand talking to the head of uh community over at Integrity that they’re really trying to bump up uh the amount of community engagement and involvement that they have this upcoming year. So, they’re really going to be supporting the Integrity ambassadors a lot. Um so, you can find out how to apply for that at integrity.com/ambassadors. Um you know, it’s a annual thing. They’ll give you support on, you know, engaging with the Integrity community wherever you’re at in various spots around the world, um and give you access to resources that you’ll need to drive that community. So, um really good opportunity. Uh I I definitely think you guys should take advantage of it if you’re in the Integrity ecosystem. All right. Next up is Adobe and Hack the Bay. Um the Adobe team wanted us to let you know that they’re going to be at Hack the Bay this year. That is March uh 23rd, I want to say. Yep. Uh 11:00 to 5:00 p.m. in San Francisco. So, if you’re around and you’re in the bug bounty world and you’re going to Hack the Bay, your new mission is to go find the Adobe team and say hi. Tell them you heard on the CTBB podcast to come and say hi. That’s a great way to support us and a great way to engage with a a staple program in the community, uh Adobe. So, if you’re going to be at Hack the Bay, definitely go by and say hi to them. They’d love to meet you, and that’s a great way to support the pod. Um last but not least, we have another shout-out for the bug bounty maturity framework. Um for any of you program managers that are listening, if you’re looking to um understand how your bug bounty program is on a maturity scale from emerging to leading and what you need to do to take it to the next level with your program from a hacker’s perspective, but also just from a well-run program perspective, then bug bounty maturity framework is an awesome tool that you guys can use um to make sure your hackers are getting the best experience and you’re getting the most value out of your bug bounty program. I know the guy running it, Steve. He used to work with us at the pod. He’s amazing and knows exactly what he’s doing. He’s like one of the best people to run this in the community that I can even possibly think of. So, shout out to Steve. All right, that’s it for the Twib. Let’s hit the show. All right, look dude, here’s the deal. Last week on the pod, you told me that I need to be like actively training Claude and getting it to to do this stuff.
no, you’re triggering me. No, stop now already. I hate it when people use the word training wrong. Okay, all right. And it’s it No, no, no, and I and I know you know, but listen, every normie out there and yeah, I’m not trying to be offensive or anything, but people who are just not in tech but people people say training and I’m like like, you know, you’re they’re like, “Oh, I trained my chat GPT to do this.” Like, you did no training. You gave it some context, you gave it a prompt, you didn’t train it. Fair. Fair. Let me rephrase. You guide my Claude, inform my Claude, equip my Claude with what I want it to do. Sure. Um and I played around with it a little bit more and, you know, you’ve been tweeting up a storm about Claude finding stuff and I have seen these findings firsthand and they’re legit. Um and yeah, we’re just kind of at a point where you need to be using Claude to hack and that’s why we, you know, we pushed Guido and you helped them build the Claude skill for for um for Guido. Um so, what I want to do today is I want to talk a little bit more about skills, how to understand skills and I know I I know I can see your face right now. I know that you are a little uncomfortable talking about this cuz this is kind of the secret sauce, the last secret sauce that there is. So, let’s talk I think I’m I’m hesitant on multiple fronts. One is I already know there’s some hesitancy around talking about AI on the pod too much and Buzz Factor himself has already been like, “I’m so tired.” Not not about the pod, but just about like the community and and X and everything and I know that tons of tons of people we know are like muting lots of words about Claude code and agents and everything else on on X and, you know, again, I’ll say what we’ve said 100 times. It’s like, this just isn’t important enough for us to tell you. You know, it’s kind of like when your parents keep telling you to pick up your room, it’s like, well, you’re going to have to do it when you get older, right? It’s like Exactly. We’re going to have to use this what they want. We’re here to give the people what they need, you know? And that is that is one of the truest statements Justin has ever said about this podcast. He lives and dies by that mantra. So, Um so, yeah, and then too, man. I mean, we we we we try all the time. There’s the pull of like, oh, wow, we’d get so many more views, so much more distribution if we would just talk about XYZ. Yeah. We we don’t do it, guys. We don’t freaking do it. And every day I get to do more beginner content? Yeah, yeah, I many many many times, you know? And and uh you know, I I won’t do it. So, anyway, this is what you guys need even if it’s not what you want. Oh, I didn’t answer your question. Sorry to cut you off. You You said I’m hesitant to talk about it, and yes, I am. Honestly, Justin, I would not have talked about this or another future episode we have coming up in a in a in a few weeks or maybe maybe it’s more than a month away, but about this sauce, like giving away the secret sauce for so much of this because I do think it’s like it’s it’s an edge, it’s leverage. And because I’m willing to scale to more Cloud Max subscriptions to find more bugs across all bug bounty programs. When we talk about this, it is giving stuff away. But, you know, we’ve we’ve we’ve wrestled with this with the podcast for the last, you know, 2 or 3 years uh with the same stuff, right? You’re often giving away techniques on here, and so do our guests. And so, I think it’s like uh just a part of what we do. And so, yeah, we’ll talk about it. It is. So, we’ll see what we can do, and and we’ll see how far we can get today. Uh and I I just I do want to like give you an a little bit of an out here though. Like, if I’m asking things that you really think are going to hack up the secret sauce for you, you know, like AI’s a little bit different than talking about techniques cuz one of the things we’ve built this pod about on is like coming on here and talking about stuff pretty liberally. And then just betting that, you know, the community doesn’t have the either the grit or the the patience or you know, whatever to actually implement it, right? And that’s why we don’t lose out on our bounties as much. We’ve been burned by that many times. And they it turns out, you know, a lot of you guys, the people that are the high-level hackers that are listening, will take the techniques and go do it, right? And that’s exactly, you know, what we’re doing. We’re exchanging the these concepts for your trust and your ears, right? That being said, AI’s a little bit different because if you just do the thing, AI you know, and tell it to do the thing, then AI will just do it, right? You don’t have need any like grit or endurance, right? So, I don’t know, man. It’s I will give you an out if you if you want to not say some of the stuff. You feel free to like just shush me along a little bit, okay? Sure. Yep, sounds good. All right. So, first up, man, let’s let’s let’s get into this. This is going to be a nuts and bolts episode. I’ve got a, you know, rubber meets the road question for you a little bit here. So, I don’t use a ton of cloud skills. The only cloud skill I really use right now is the Kaito mode cloud skill. Um I kind of feel like cloud skills are limiting cloud a little bit. Like if I tell it, “Hey, here’s my super cool thing to like, you know, grab all the lazy loaded JS files or whatever, right? From from the the JS that I gave you.” Um and then it’s like a little off, then cloud, you know, cloud’s going to try to use the skill and he’s like, “Oh, it doesn’t work.” You know, or whatever. And then it’s going to go and get distracted or whatever. Whereas I feel like cloud is smart enough where it’s like I tell it to go download all the lazy loaded files, it’ll write up a little script in like 30 seconds to pull it down and it’s perfectly tailored to the situation. Yeah. So, are we sort of hampering our cloud skills or our cloud when we are giving it cloud skills versus just telling it to do the thing or are we actually enabling them? Yeah, I think that it’s just it really depends on what you’re asking it to do, right? Like maybe lazy like loading lazy loaded JavaScript files is something that it’s like really good at, right? But you do need skills for things it’s not good at, like Kaito. Like the like you Why do Why don’t you just get rid of your Kaito skill? It’s like if you ask it to start in our Kaito, it’s going to have to go into some deep research. If it can’t find the docs online, it might legitimately not be able to figure it out. But if there’s really good docs online, it would go find the docs, it would download, you know, it would figure out what the SDK is, it would go look at the GitHub open-source code, and it would be able to figure it out. But now you’ve wasted like, you know, half of your 5-hour limit for it figuring out how to use Kaito. And so I So I think that um you know, I’ve kind of categorized when it’s useful to have skills into a couple buckets. The one that we’re talking about right now is basically things it doesn’t know. And that could be because you have a custom setup. Like let’s say you have a server and it runs at a certain IP address, and you have a certain user on there, and you want things to be done a certain way. That’s in your head. There’s no way for Claude to figure it out. And so you probably need a skill or to update your Claude MD to say like, “Hey, when you’re using my VPS, here’s the password, or here’s how you connect to it, or here’s where I save things.” Like, you know, you basically need to give Claude Code information it doesn’t have. And I think that also applies to any kind of like groundbreaking techniques. You know, these models are trained on trillions of tokens, and your uh specific technique, especially if you got it from like last year’s Def Con talk or something, that you’re using to exploit some sort of like, you know, nested GraphQL mutation. GraphQL’s maybe a good a bad example because Claude Code’s so good at GraphQL. But, you know, there are definitely things that it’s not as good at or that it just doesn’t know because it’s not been in the training data. It may very well be able to figure it out given enough time and effort, but why not just give it like a head start? Okay. So, custom But see, but then there’s also the inverse of that, man. I feel like like just to challenge you there a little bit. I mean, that’s that pretty much the exact opposite of what I just said, which is you know, if we give it skills Mhm. that that that are not applicable to all of the situations. Yeah. Then, you know, we’re we’re the the whole technological advance that we have here is that it’s smart. Yeah. You know, like I feel like giving it skills is like just why didn’t we just code it? You know? Uh right? Am I wrong there or is it like is the beauty of AI that it can actually go figure out the more nuanced situation without us having to give it instructions on how to deal with that? I mean, I think it it’s definitely both. Like it has a lot of information baking in. You’re not wrong You’re not wrong about that. But then it clearly has limitations, right? Like using Kaito mode. And then it’s smart enough to figure out some stuff, but it’s not smart enough to figure out everything. I think that uh your concern can mostly be mitigated with like just a Claude MD line, like one line that says like, “Hey, when I ask you to do something, like invoke the skill to do it because sometimes I have certain ways I want things to be done. But if the skill isn’t comprehensive enough or if it fails or if you try with it and it doesn’t work, don’t stop there. Use your own exploration and your own creativity to keep going. You know, try harder. Put the OSCP motto or straight straight in your thing.” You know, I have POC or GTFO and try harder both in my Claude MD cuz it’s like I’m not like you can’t just say, “Oh, this looks like it might be vulnerable.” That’s That’s not valuable to me. I want you to actually have a full POC that I can, you know, completely validate end to end to make sure this is an actual bug. And so I think that um you know, the limitation is mostly mitigated by what you’re talking about, but there have been a lot of studies that have shown that like poorly written agent MD or Claude MD files or poorly written skills actually reduce the quality. And so I I do think people should be really um careful and particular about how they’re like what they’re adding and how they’re adding it. Yeah. Yeah. I agree. It is a little tricky though, man, cuz I do find myself like I definitely just gave Claude a list of like things I wanted to brute force the other day and said like, “Hey, you know, brute force these in in Kaito.” Yeah. You know, and it like made a bunch of replay sessions or whatever. It didn’t actually use automate. I don’t know, maybe I I told it to use PlayStations or whatever. And I was like, “Dang, I could have just like put this straight into automate myself. Like, what am I doing? I’m getting lazy as heck.” Yeah. So, it is a little tricky. Um So, building off of what you just said though, I wanted to like describe something that I uh saw in the Kaito mode skill that you and Kaito built and ask your opinion on this as a framework for Kaito skill. So, what will happen in the Kaito skill, Kaito mode skill is if it can do something with the Kaito skill, you know, documentation that you guys have built and stuff like that, it will invoke your bi- your binaries or your your scripts or whatever that you have in place and it will do it with that, right? If it cannot do that it will um use the actual client JS library that the .ts files that run kind of this Kaito skill um is built off of and it will invoke those directly to accomplish its goal, right? And it will sometimes even do that if it if it’s a more complicated action cuz it’ll be like, “Oh, I’ll just write out the script and chain multiple actions together, right?” Yeah. Um and then if it cannot get it with that, it will then try to use GraphQL to control Kaito directly. Yeah. And I think that sort of fallback nature, I don’t know if you guys you coded this directly into the Kaito skill itself, but I think that sort of fallback iteration nature Mhm. of building a skill it works really well because you give it multiple tiers of flexibility and control while also abstracting away the tasks that you know are going to be the same every single time. Does that make sense? Yeah, I didn’t think there was going to be anything I didn’t want to mention, but there is something that I’m not willing to mention, but I have implemented something very similar for tough problems in bug bounty where basically it’s exactly that. Like, I want you to try to solve the problem with this method, but if it doesn’t work for some reason, then try this method. And if that doesn’t work, then try this one. And I think you’re right. That gives it a lot of uh flexibility and a lot of like a much higher odds of success. And no, I I didn’t build that into the skill. It’s really funny that your Claude code did that. Um, but wonder if that’s a Claude I wonder if that’s a Claude code like concept that that Anthropic built into Claude code. Like, “Hey, you know, if if the skill doesn’t work, look at the primitives that built the skill.” Yeah. And then try to use those primitives to accomplish the same goal. Or not. I don’t know. I think it’s probably just the fine-tuning on lots of data, especially lots of coding things where it tried to run something on the command line and it failed. And normally, you know, a year ago or whatever, the models were like fine-tuned to basically just like stop at that point because like Yeah. all the examples were like one-offs. But where people have been using Claude code over the course of the last year, they probably have a lot of training data for examples of it like failing, trying again, failing, trying again, and that being like the ideal training set because that’s what we want as users. We just want it to work. Like, stop getting the error and make it work. Yeah. Yeah, just force it. All right. So, given that I I understand that. That makes sense. We’re going to create skills and we are going to, you know, give it sort of this fallback architecture, which is good. What things do I create skills for? Is is kind of like my next question where I where I go because a lot of this is just like looking at at the JS files, looking at the HTTP request, and like trying things, right? Which is great, you know, and the kind of mode skill is phenomenal, by the way. It really is super helpful for it to have it like, you know, sending stuff uh through replay and being able to have introspection to it. But you know, what what areas should we be building skills in? Do you do you have any thoughts on I think when there’s something very flexible, like making a request, and you have a way you want it to make a request, that’s a great place for a skill, right? Like, it can use curl, it can use JavaScript, it can use Python, it can use Kaito, it can use Wget, like it can use Chrome DevTools, it can use Playwright. I think um, in general, anytime there is a uh way to do like there’s a many many ways to do something and you want it to do a specific way, it’s a great time to use a skill, right? Like you want it to use Cado so you can co-hack with it so you can see the request, so that you have history for screenshots, for POCs. I want it to do that for the same reason, right? There are probably lots of things like that. Like if it’s going to SSH or SCP things to a certain place, it’s like oh it needs to know what server to do that to. It needs to know where to save that thing. Another great example would be like where do you want it to take notes? Do you want it to save at the target level or at the subdomain level? Do you want it to save off leads and findings or just gadgets, you know, like I think these sort of things where the the total output space, especially across multiple sessions. So actually that’s that’s another great example. Let’s say Justin that you were going to be hacking with Cloud Code a lot over the next week. Do you want each Cloud Code instance to save files in different folder structures? It doesn’t make any sense, right? It’s going to confuse you and your desk is going to or your desktop or like you know, your file system is going to be a total mess. It gets messy, man. Well, yeah, but it doesn’t have to be messy. You actually can just use a skill or a Cloud MD like to actually, you know, a line in your Cloud MD or a paragraph in there or whatever to steer it to behave in the way you want it to. That’s not restricting it. Like telling it where to save it isn’t going to degrade the quality, right? Telling it how to make those requests will hopefully not degrade the quality. Though I do think it’s like much better at like Bun TS stuff or like, you know, I think so. Yeah, and it’s like really really good at cuz they bought Bun, right? You know, it’s like yeah, clearly like they’re they’re, you know, smoking their own supply over there with that, I think. Well, what’s kind of crazy is that whatever they have Cloud Code do is probably going to be by default, like if you don’t steer it, it’s probably what’s going to dominate the market because everyone’s using it and everyone’s going to continue to use it and so they kind of have like really large influence over that, but Yeah. But anyways, back to your question. So I think that that’s one way, right? It’s like when the total problem space is large or the total solution space is large and you want it to find a solution a certain way, it’s a good time to implement a skill or to update your your Cloud MD. The other time is when it’s knowledge that like secret knowledge. Um you know, like I don’t know hidden techniques that you that you have that I don’t, right? Or I or you know, there are probably Google gadgets that are in my files that are not in your files. And so like bundling those into a skill makes sense. It’s not on the internet. It’s not in It’s not in, you know, the public domain where you can go find it. So if it needs this this gadget, this Oracle for ID to username or something for some random bug bounty program, it needs to either be in the notes so it gets out of the context whenever attacking that program or it needs to be in a skill, right? And and you know, that’s not going to limit it. That’s going to make it way better cuz it has access to more tools, right? Um I guess that’s another good example. Skills are a great like there are things that AI could sign up for but is really going to struggle to sign up for. So another place for an example another place for a skill to be implemented would would be like let’s say you want it to use a specific piece of software that requires you going through enterprise sales to buy. Like like by by not having a skill there, it just literally can’t use that product. But if you signed up, you’ve got creds, you’ve got API token and you make a skill and you put the token and how to call it in the skill, now it can use that. So like it’s literally like a skill you’re giving it that it could not have had otherwise. Yeah, that that makes sense. That that I don’t struggle with at all. Like if I need to give it access to, you know, Kido or you know, whatever enterprise thing that I needed to have access to, then that makes total sense for a skill. What I’m kind of struggling with from like conceptual perspective from as an offensive security researcher is stuff like For example, I I thought about Well, totally, you could do that for sure. But but I’m thinking like do I create something like a front-end analysis skill where I where I like outline my methodology for doing front-end analysis, you know, and and doing client-side hacking essentially, right? Mapping out the attack surface, understanding, you know, everything about it. Or do I and and do I give it tools to do that, right? Like, hey, here’s, you know, use P prettier for beautification, use this for source map enumeration, you know, that sort of thing. Uh and and will giving it that be helpful and is or am I just turning it into me and I’m losing the the magic of like, wow, it found something that, you know, like for example, when I talked about the the shift, um, you know, the shift vulnerability that I found. I I told it, hey, try some JWT attacks is what I told it on this target, right? And then it just went through and did all the JWT attacks, you know, that it knew and it found a bug and it was like a 15K crit, you know? So, like So, I I think there are two things here. The The first thing is answered this question, but yeah. No, no, no, no, no. Actually, I No, I All the stuff you said triggered completely new thoughts to me. The The The first one is that tokens are cheap for right now, right? They’re subsidized. So, you always should just do both if you have the opportunity. I think it’s amazing and I think that if you really wanted to know, you should compare it and improve your workflow. So, I think personally, you should actually hardcode your front-end analysis because it gives you more determinism, so you know that it’s not going to miss things that you wouldn’t have missed, right? Like Like because when you just let it explore by itself, it might not have actually loaded source maps. It might not have been able to find them. And unless you’re literally watching it the whole time, which you’re clearly not, you’re doing other things, you’re hacking other stuff. When it says it’s done, you’re just going to be like, oh, okay, I guess it didn’t find that anything. It’s like, no, no, no, actually, it missed this entire workflow that I normally do. And so, I think especially, you know, if you want to be thorough, you should outline your methodology and make it follow that. But then, I think you should do it a completely separate run that’s like that has none of that and that that does its own thing. And then, if you have the opportunity, compare them and be like, hey, was there anything that this like free-roaming agent found that our hardcoded workflow didn’t find, if so, how did it do that? And add that to the workflow. Like, what techniques did it use that we don’t that we didn’t like previously hard-code into the workflow? Mhm. That’s a great point. I think I think that’s some very helpful feedback to give. And so, let’s get into the nuts and bolts of that because I think what’s overwhelming for everybody right now is this is like, you know, such a new world, you know? And and getting defined pathways for like improvement and implementation of AI is really helpful. So, let’s say let’s give an example. Um I’ve got this website site.com that I I want to hack on. So, I spit up, you know, maybe I’ve got my tmux pane. I’ve got two windows side by side. One, I like clone down all of my own, you know, skills and and agent definitions and stuff like that into that that cloud code instance. And I say, “Boom, hack site.com. Here’s the cookies. Here’s whatever you need. Here’s like my my, you know, starter pack or whatever.” Yeah. And then the other in the other window, I have it I say just, “Hey, hack site.com.” You know, very few skills, very few resources. And then do I I mean, do you think I should ask it for like a definitive output? Should I say like, “Okay, both of you guys output a report that contains everything that you found that you think might be of interest. And then compare the two and cross-correlate or Yeah. Just spell that out for me here. Personally, if I was going to do it, I would say, you know, what once I feel like both are kind of done, I would just say or or actually maybe while they’re running cuz it might have to compact most multiple times. So, I would I would tell both when when you when starting them. And mine kind of do this naturally cuz of my CloudMD, but keep notes on what you’re doing and what you tried and what was successful and what wasn’t. And then and then at the end, I would say like, “Hey, give me a list of all the things you tried and, you know, you the workflow you went through and all your findings um to to just one of them. It doesn’t matter which one. And then just paste that into the other and be like, ‘Hey, I had another agent work. This is what it did. Compare it to what you did, and tell me about any gaps. Like, what did it find you didn’t? What did you find it didn’t? What did you try that it didn’t? Vice versa. And then, when, you know, that after after it responds, it might be insightful, it might not. If it’s insightful, say like, “Oh, okay, perfect. Now, add that to our workflow, so you don’t miss that next time.” Mhm. That’s really nice because especially with the agent you’re asking it of, it’s got its context already. Exactly. So, it’s like, “Oh, I I I did try that, but then I like cut it last minute.” And then, you know, but if you added asked third agent to like, you know, I almost suggested that. get output from both of almost said, “Take read the output of both.” But, no, it’s lossy. Like, you definitely want to do it with the context. That’s interesting. I bet you could also improve it even further, though, by saying, “Third agent, look at the context files for both of the other agents Yeah. and see what it tried Mhm. uh and compare that to the outputs of each. Yep. Uh and, you know, compare and contrast.” I’m I’m very often using this skill that I made called session search. It’s not a big deal, but it’s just like a small little CLI wrapper around like a fast, you know, rip grep across all of my session logs. I think I’m up to like 4 GB worth of session logs in Cloud Code, just on my local laptop, not including the stuff that runs in the cloud. And um I’ll often say like, “Hey, I was chatting with you about YAML parsers yesterday or like a few a few days ago. I don’t know where the session is. Can you go find that?” Or another thing I’ve used it for, HackerOne’s API token, it only lets you have one at a time. And a lot of times I’ll like lose my token, I don’t know where it’s at. I should just put it in 1 Password. But, I but I’ll be like, “Hey, go grep for BB scope and grab the token that we used in that command. I need to use it for something else, you know, or what have you.” Do you give it access to 1 Password? No. Yeah, okay. Cuz I know some people that do that for like, uh you know, open claw or whatever, and I’m like, “Oh my gosh.” You know, like, don’t do that. Well, I mean, I do think I’m still running pretty risky. Like, I use dangerous skip permissions on everything. And I And I have it running on my local laptop with access to everything, but I don’t give it access to my email or to one password, so I’m I’m at least a little protected. That’s good. That’s good. Um okay, so yes to my methodology for the purpose of getting to deter Yeah, determinism and confidence that it’s doing what I want it to do. Yes. But also But tell it not to limit itself. Yeah, I would say just even put it in your skill like don’t limit yourself to this workflow. If you find something interesting, go down that rabbit hole and then just come back to the workflow. Or after we’re done, if you run through my whole workflow and there’s stuff that you thought was cool or like that we should have checked that we didn’t, add it back to the skill and keep going. Freaking crazy that we’re saying that you thought was cool to a computer like that’s insane, bro. Yeah, it is. Wow, we you just like authentically recommended that you ask a computer to go look into what it thought was cool. Like that’s nuts, dude. And it’s like you know, I’m obviously conflicted because I don’t think I I I don’t think that these models are in any way conscious, but they definitely emulate humans and like all of their thought patterns and all of their token outputs, so stuff like that just works. It is crazy, dude. It really is. Um okay. So, can you give us a couple So, I really liked the session session search ooh skill that you mentioned. Um do you have any other skills that you want to just lob up there? You don’t you don’t need to release them, just talk Yeah. Yeah, so one skill that is really good and you should not let it take away from buying his book, so go buy Eugene’s book, but JDXSS Doctor created a skill that he and I both use called zero-day research that basically told it to go look at Eugene’s book all of his content that he’s put out online and create and create like a zero-day finder and it’s really good at looking at source code and it’s really good at looking at binaries like executables and like Mac OS DMGs and stuff. It’s really like I don’t know why, but like I think that it’s because, you know, there’s not a lot of the internal monologue of the experts, like like Eugene’s brain, for this sort of thing in the training set. So, this is like another example where it’s like kind of like secret knowledge that, you know, eventually will be baked into the model, but right now it’s not. And so, you can you know, you can have it like search for zero days. That’s a good one. Content creator {slash} report writer. I think everyone should have their own. And personally, I think that I’ll just give you some prompting tips right now. Give it an example of like some of your best written reports, and tell it to keep everything super concise and super technical and just straightforward. Like don’t put any flavor in it. And then give it exact fields that you want it to fill out every single time. And this saves me so much time. And like, you know, my agents at the once they find a finding, like I tell them to go ahead and and write the report and then give me a link to it locally. And they’re just they’re so good. Like I very seldom have to edit them very much at all. I’m usually reading them for accuracy, not for any kind of tone. The tone is just like and then I did this, and then I did this, and then this happened. It doesn’t inject any flavor or over hypeness. It’s just like, you know, I mean, it’s very straightforward. And so, that can be really useful. Yeah, dude. I don’t know, man. Like Richard, you can bleep bleep the name out here, but like I’ve I’ve looked at reports recently, and like that are AI generated, and I I had to give them a talking to about it, to be honest. I’m like, this is not good. And this is why, you know, people are are you know, having issues with AI generated reports. So, I think you really have to you know, I think giving that recommendation to anybody who is, you know, not at a at a higher level of hacking proficiency and hasn’t written a thousand plus reports by hand, you know, it’s a little dangerous. Let me talk about the ways that it falls apart, so people are aware. It falls apart in the fact that it will often blend bugs. Like very frequently, it tries to blend two or three bugs into a single report, which it just doesn’t make sense. And and I’m often having to clear that up. The other thing is it’s like it’s understanding of threat modeling is still kind of bad. Um there was a report that um I put in yesterday or the day before. Um and what happened was it was access to a bunch of paywall uh like um free features or a bunch of paid features. So it was a bunch of paywall bypasses, right? And those normally aren’t great reports, but I mean it was like 30-plus features. Like you could get access to basically any pro pro or enterprise feature. But the the agent was like in the report like saying like complete degradation of security. Like basically a lot of the paywall things allowed you to like change things about your object in this in this app that made it less secure. And so it was convinced that it was like bad because you could like paywall bypass to get like enterprise features to then make the product less secure. secure. Yeah, it’s like it just doesn’t make any sense. So I mean just be really careful and read it, but I will say I I don’t have to edit mine very often. Um Yeah, it’s you probably have good good Now I was about to say training. Holy crap. No, uh you have good guidance. Yeah, good intuition there. Yeah. Um some other good skills I think like um BB scope and um like H1 scope or hacker scope or whatever that Patrick just came out with can be really useful because this is data that the model can’t get on its own. And um basically what is I think Patrick’s is actually an MCP, but uh but what it does is it pulls like the policy page, which is really useful cuz I had previously wasn’t giving that to my agent, so it was sometimes going out of scope and having issues. And so it pulls the policy, it pulled it pulls the scope, and then it pulls like um what are they called? Um disclosed public bug reports. So it kind of like can get a heuristic for like what might be vulnerable. Like What is that called? Uh I think it’s called H1 scope. Um we’ll link it in the show notes, but it’s by Patrick. Uh, it’s a really nice scope. And he’s And he’s And he’s updating. Like he’s even doing like a blog series right now where he’s already posted two blogs in the blog series. Um, so that is a nice one. H1 brain, maybe? Is that Is that the MCP server? Yeah, it is. Yeah. H1 brain. Um, I spoke over you, but yeah, that’s what Justin was saying. It’s called H1 brain. Um, Okay. Yeah, I think that’s mostly, you know, the types of things. If you have any workflows, like I do think skills are really nice for Justin, you probably have actually a couple of these in your hacking workflow where like you very frequently want to like This I don’t love this example, but basically get subdomains, pipe through HTTPX, then automatically fuzz, and then get all of those results. Anytime you have like a full pipeline, it’s great to bake it into a skill so that you’re not always having to like kick it off, especially if there’s like contextual flags or contextual input where the AI can basically get that input for you, run the pipeline for you, and then tell you where the output is. It’s just really nice to kind of like have like a faster automation or like a contextual wrapper for your automation. Yeah. And you’re you’re using I mean, skill just I just want to like clarify this here. Skill is being used kind of loosely here, right? Like it can just be an MD file with information. It almost always is. But yeah, sometimes it includes command line tools. Like Okay, like a command line tool bundled with it. And and in your experience, it’s best to have Claude write that, you know, using like TypeScript files or something like Right. Almost al- almost always that’s what mine are these days. Yeah. And I think it’s just And that’s just because that’s what Claude prefers, and so uh my intuition says that it’s going to be better at writing those and make less mistakes when it writes them. So, that’s what I’ve been letting it do even though I’m not a huge TypeScript guy. So, okay, let’s let’s just think about this for a second. Like I think the highest value thing that somebody can do right now is who’s actively hacking and using Claude uh and wants to use Claude to integrate, you know, into their workflow, is just download the the Kaito mode scale and just let it like you can just tell it, “Hey, make all these replay tabs like look at all the stuff that I’ve got in there.” Yeah, let me let me give you let me give you a pro tip for that. Yeah. I don’t know how you do it, but like let’s say you’re looking at your HTTP history and you’re like, “Oh, I want Cloud code to hack on this request.” What I what I usually do is I’ll copy and paste the top two lines because it has the host and the path and then I give that to Cloud and say, “Hey, look at, you know, this request.” And it then it will go find it based on that. Uh, I don’t know if the ID on the left is like matches the ID in the database and all that. So, that’s like kind of like a really um like a pro tip that people probably need for like helping Cloud Cloud basically it the session a name in Kaito. What do you mean What do you mean the session name? Like like Oh, in replay. Got you. Yeah, I know I I’m I’m just doing straight from my HTTP history a lot of times. Yeah. Okay. Yeah. Yeah, that makes sense. That’s good. Yeah, so but like it What is it What are the other, I guess, like 80/20 pieces of like uh you know, you’re going to get very outsized returns. I I’ve gotten very outsized returns by hooking it into Kaito. I think that’s, you know, amazing. Yeah, so I I think I think the biggest second thing that everyone has to do, I guess it’s two things. One is update your Cloud MD for just like contextual information about you because it’s going to make it not reject you as much. Like you just need to say in your Cloud MD you don’t have to tell it its name, but you can just say like, “Hey, I’m a bug bounty hunter. I do ethical testing. Anything I ask you to look at will be in scope, but make sure you like attempt to stay in scope because it’s been going out of scope sometimes lately.” Non-destructive actions. Yeah, yeah. Don’t do not uh destructive actions unless it’s accounts that you know we own because it’s like, you know, I told you that it’s two accounts that we own that are in Kaito or whatever. And and then the second big thing is just the note structure. And um I was going to do an episode where I talk maybe more deeply about it, but just right now my intuition says that the best way to categorize things is like notes, leads, and leads can be {slash} interesting findings {slash} whatever, and then gadgets or primitives. I think the word primitive might be more specific for what’s in the training set, so maybe call it primitives. Then then findings, and then I have a validator. I think you should probably write a validator. Basically, the information you want to give the validator is things like, “Hey, CORS issues are often false positives. BXSS, unless you actually get a trigger to our actual like trigger output isn’t a valid finding just because it bypasses the WAF.” And when you go to validate this bug, be skeptical. Don’t mark everything as a critical or a high. In general, I think these are lows, these are mediums, these are highs. I give it really give it real examples of these bugs. And then and then you have the validator there maybe as a skill or maybe as a as a sub agent or just as a tool. I was going to say that’s probably a good, you know, we we were talking about the before we got on air, like the difference between skills and agents, and you were saying you don’t use agents actually very much at all. That might be a good use case for an agent because you want it to be unbiased by the other information, you know? I I like that. Yeah. Yeah, and so in the and then so I have a validator and then I have what’s called reports. So my hierarchy is basically notes, leads, gadgets, findings, reports. And it kind of flows in like a really nice way where there’s more at the bottom. It should take notes on all kinds of stuff, and then it should, you know, some of those will be gadgets. Some of those or sorry, some of those will be leads or gadgets, and then some of those will become findings, and then some of those will become reports. And so it’s like a funnel. Nice. I like that, dude. That’s a good That’s a good structure. Yeah, so our so our so our three-pronged attack based on your your question just to re- reframe it for everybody is basically you you you need a Kaito skill. You need You need a Cloud MD that talks about how you’re doing bug bounty hunting, stay in scope, don’t do destructive actions, always take notes, and then you need to tell it where to take notes, right? Write in this database or write in this Notion. I know Grapnel loves Notion, so he has it write to Notion. Some people love Obsidian, so put it in these Obsidian notes so you can go back and reference them. Um or or or actually push it to an API, right? Like so one thing I’ve been thinking about doing is just creating like a API.rez.com and having it right there, so that whether it’s on my VPS or my personal machine, all leads and gadgets go there. And so now, no matter where I’m hacking from, I I have access to those gadgets and Claude has access to those gadgets. That’s pretty freaking good, dude. That’s pretty freaking good. Yeah. Yeah. Okay. All right. So, let’s let’s talk about orchestration a little bit. Um so, we another component of this is like, okay, obviously seriously, listeners, you must be using Claude code to pair hack with you. Like, that is not really something that you can avoid doing at this point. Yeah, I I I’ve I’ve also not posted about many other people who message me and say, “Thank you so much. Just found my first bug. Thank you so much. I just escalated my, you know, blind SSRF into a full read. Thank you so much. Like, I found 10 bugs in the last week.” I’m not joking. Um actually, yeah, well, actually, I won’t make him believe it. Basically, somebody you know really well uh messaged me and was like, “I found 10 highs and crits in the last week by using Claude code.” That’s crazy, man. That’s crazy. Yeah, I think I knew who you were talking about. Well, we’ll compare afterwards. Um but, yeah, that’s I think something everybody must be doing. Um one of the problems that you run into is how do you make it be persistent? Like, how do you force it? I know that you you’ve mentioned some Ralph loop or something like that. Um and I’ve seen I think solution by Karpathy at one point. Uh do you have any best practices you want to shout out there on that? Sure. This isn’t any secret. In fact, I think I’ve even tweeted this. I mean, you can just tell it, “I’m going to bed. Don’t ask me for any questions.” Every time I’ve done that, it’s ran for 4-plus hours. So, it’s like Really? Yeah, there’s there’s no secret here. It’s just you just say, “Hey, I’m walking away. Don’t ask me for any input and don’t stop hacking. Like, I want you to keep going, keep going deeper, keep finding more bugs. Just literally give it that prompt and it won’t it will not stop for hours. That’s very surprising. Really? That’s interesting. Yeah, I I didn’t expect it to be that straightforward cuz I thought that would get like compacted away or something. I don’t know. You know, it does get compacted. So, I guess maybe it is working cuz I have a good Claude MD. But, you should just tell I mean, if you want to, you can be like, “Hey, I’m walking away. You’re going to end up compacting. So, just keep good notes about where you were and what you were doing.” But, in general, I think that’s kind of already built into their compact um script. Or to their compact prompt. it. It might survive it, you know, it might just say like when it’s compacting, it might be like, “Oh, I It writes itself a big new prompt to start. And so, it does survive most of the time. Uh this is actually a really great tip for the listener. I probably also shouldn’t share, but um on those compaction loops, if when it comes out of compaction, it reaches a context limit before there’s any steps in between, it can’t like roll back or compact. That happens most likely when you have sub agents with a like a lot of sub agents. So, my personal fix is to tell it not to use more than two to three sub agents. Anytime it spawned four or more, I end up getting into um a place where it can’t compact. And I can’t go back and have it not compact because whenever you go back to redo that to like jump back a few loops, you have to hit like escape twice and you go up. And then you press up and you go to like a separate chat. But, when it’s running autonomously overnight or whatever, you can’t go up. And when it compacts, there’s no there’s no message to jump back to. And so, then you’re in the situation where you have to detangle it. And for me, I then have to open a new instance and tell it to go read all the context from this other session and get all the context it needs to get started again. So, my in my experience, the best way to limit that, especially if you’re going to run overnight, is to be like, “Don’t use more than two sub agents.” Mhm. Mhm. Okay. All right. That’s a That’s a good That’s a good point. I think uh I haven’t run into that issue, which means I’m probably not using it properly. Yeah, I do No, no, I think we’re I think you you often just like have four separate instances. So, they’re probably not using sub agents that much. And you’re also interacting with it often. You’re not like having it run overnight. So, those are the two reasons. You’re running everything from Discord, right? You don’t really interact with the the like Cloud Code online as often, right? I would say there I have basically three modes. Um I’m I’m okay to share this, I think. One is I code hack with Cloud Code on my desktop, and I do that a lot, and I do that all through iTerm, just through the terminal in the normal Cloud Code CLI. That’s probably, let’s say, 50% of my usage. Um and then the other 50%, yeah, I I have like a Discord bot that basically mimics Cloud Code in in like a Discord thread, and I use that anytime I’m doing stuff on my VPS. Um and then it’s a very similar setup for the automated bot that I’ve created that, you know, I use with JD. And we um yeah, same it’s all through Discord. It’s all managed through Discord. Mhm. I just find it so much nicer. Like, I’m in Kids Car Line. I’m, you know, Yeah. I’m doing my business. that I struggle with, yeah, with my setup, which is just tmux four panes, you know, let’s go. All working on different stuff. By the time I’m done prompting one of them, the other one’s done, and I can just kind of jump jump jump jump jump. Um but yeah, the remote control functionality is is getting better, but it’s not perfect, you know? Um so, it’s crazy. I think I talked about it on the pod last week, and it’s already better, you know? Like, they’re already Yeah, oh yeah. Yeah, it is better. For sure. It’s not perfect, but it is better. So, if anybody, I know you were asking me, and I didn’t have a good answer to this before the pod started. If anyone’s using Cloud Code in the desktop app, I think there’s probably some like really big wins there. Like, I don’t like that when you hit control O, it only shows you the output from the most recent command. You can’t go up and see the output from like previous commands. And then sometimes I hit control O, and my computer will just like lock up because I think it’s trying to load like a bajillion, you know, characters from like a bunch of different output over there. And so, yeah, um the Primeagen, though, is like a you know, if people don’t know, he’s like a software dev influencer, and he’s really funny and really great, but he is always making fun of how bad their TUI is. I’ve heard that open open code is much better, but I but back whenever they like stopped allowing Cloud Code to be used in third-party services or something, I never I never went down that route and figured it out, but yeah. Yeah, the subsidization is OP, man. It really is. That’s one thing that we were even talking about with with Shift. Like, I think Shift is super good in Caido, but the the problem is it you know, isn’t free. And I’m just so like hooked on this crack of like, I don’t even have to think about the tokens cuz Cloud Cloud Max is like, you know, 20 bucks, and then I’m I’m good, you know? Um, so yeah, it is it is uh it is really crazy that Cloud Code is has that market cornered because of the the subsidization. Um, all right, man. Um, let’s just talk Well, first I I think we should go back and summarize everything that what we what we just said here. Okay? So, here’s here’s what I understand about the best practices that you mentioned on this pod, and you can interject whenever you want and give me your thoughts, okay? So, here we go. First, we need to be, you know, pair hacking with with Cloud. We have to. We must. And one of the uh highest value things that you can do to do that is get the Caido mode skill. Get it hooked into Caido. Get it using your proxy so you can see what it’s doing and adding uh value, you know, handing things off to you, right? And making things easier for you to hack with, right? What What I’ll often do with it, hand it a bunch of JavaScript files, recreate all of these HTTP requests in Caido replay sessions. Boom. You know, it’s beautiful. Um, so that that’s one one thing. The other thing is giving it a note structure that it can use. So, we’ve got notes, you got leads, you got um primitives, and then you’ve got reports. And and giving it that structure so it knows how to store information in a way that you can digest well. Yeah, for hacking locally, if you’re hacking locally and not remotely, uh I didn’t even think about this. I definitely should have mentioned it. The Kaito skill actually lets you pipe straight to the findings tab. So, you can just have it go So, when you’re hacking it, you’ll get the little red dot and then you’ll know to look. Yeah. That’s a good That’s a good call as well. And then we’re building skills. Skills are either pieces of information or tools or both that Claude can use and will rag into context as needed. Don’t Yeah, ragging doesn’t use rag there. Rag is almost Interesting. Yeah, I It might if you have above a certain amount, but I think like up to at least 35 or 50 or something, it it basically sees the front matter. So, you actually huge tip, sorry. Huge tip we should have mentioned already. The front matter for skills in the skill.md file has a description and a name. That’s what is auto injected into the context when you launch Claude. Okay. So, if you So, if you have rules like use this skill when, put that in the description of the front matter at the top of the skill.md for your skills. And that is auto injected at execution time into the into the prompt like the system prompt that it uses. That’s not happening at the LLM level, right? Like with rag. It is happening at load of skill level, which makes sense cuz that’s why we see it load of skill, you know, okay, okay. I understand. Usually people say Usually when people say rag, they mean like embeddings based search. And by default Claude code doesn’t do any embeddings based search that I know of. Okay, nice. So, we load these skills up. These skills give us information and tools to do things and we want to try to do this when Claude does not have access to specific pieces of information or specific ways that we want it done. That’s right. You know, we certainly can give it ways to do things if we want assurance that it will actually try those things. But we should also caveat it at the end of whatever skills MD file or whatever with hey, but use your creativity as well. Don’t be like cornered by this skill, right? Does that make Is that Is that accurate? That’s exactly right. Okay. I wonder I wonder what the impact of these cumulative tips will have on everyone’s act bots. Um okay, and then you know, last piece is like give it information about you and and how you need things done. So, you know, creds to your VPS, you know, to a you know, cornered little document root or whatever. Um you know, that that’s sort of thing. And so, it knows how to like host things or um present information in a way that is is good for you, right? It’s sort of aligned with the notes thing, but giving it more giving it access to things that you want it to be able to access for the specific mission. Like, okay, here are my creds to this, you know, app that I want you to hack. Here are these, you know, cookies that I want you to use. That sort of thing. Um and and that could be at runtime, right? Via the prompt or it could be in the skills as well. That’s right. All right, man. All right, let’s do this. Let’s do this [ __ ] All right. Well, so one thing that we didn’t mention yet that I know we talked about potentially mentioning is um agents versus folders. You want to talk about that? Okay, yeah. Yeah, let’s do it. Yep. Mhm. So, in my opinion, you like and in most people’s opinion, No, explain that first. What do you mean agents versus folders? Yeah, so one thing we didn’t really talk about, but some of the major components are like agents, which agents, if anyone doesn’t know, in Claude code are and this is probably true in Codex, I’m not sure, but agents are a specific system prompt with a specific set of skills or even tools like command line tools that it’s like white listed to use. And um because of that, you can make like a, you know, pen tester agent. If if you’re going to be using Cloud Code for lots of stuff, like finances and, you know, PDFs and other other junk, but you’re also going to be use it for pen testing, you could use a pen tester agent or like bug bounty agent for that. Alternatively, and I think this is the method that I like slightly prefer, you can you can launch Cloud Code out of a folder on your computer and in that folder, your dot Cloud folder will only be loaded if you’re in that folder. So, the way that Cloud Code works is you have in your home directory or whatever wherever you saved it off as like your main Cloud directory, you have a dot Cloud folder which includes like a Cloud MD and skills and agents, right? But then if you’re in a sub folder, it looks there first and includes that as well as the parent folder. So, you know, if you put in your home dot Cloud MD, you know, this is my home, and then you put in a, you know, another folder in the dot Cloud Cloud MD, this is the folder, and then you were able to go and you like proxy the traffic, it would have both of those in the context if you launched it from that folder. So, Okay. This has this has like two ways that it can be used positively by our listeners. One is you could have a bug bounty folder where you are a hacking folder where you load all of your skills and your and your custom prompts, and if you launch it out of your home directory, it won’t have all that. So, you can like use it for normal day-to-day stuff that’s not hacking. But if you launch it from inside that folder, it will have those things. Um, so that’s one way. The other thing you could do um, is have a target is like launch Cloud Code from a target specific folder and have its Cloud MD have the information the top level information about that target. So, you could have like a a flow where anytime you say like, “Okay, I want to start hacking on X.” The first thing Cloud does is go and create a target folder for that target and then goes and pulls the policy page from HackerOne or whatever and puts that policy in the Cloud MD in that folder. So, then when you launch it from that folder later, it now has that target specific information loaded in automatically. Dude. Oh gosh, there’s so much possibilities with this. My as you’re talking my brain is just sitting I feel I feel like I kind of you know, normally I’m like guiding the conversation and and you know, leading hosting on this podcast, but as you start talking about these things, I just sit here and I start churning. You know, and and my brain’s like this is what you’re going to do right after you get off this freaking podcast, you know, like I was just like oh [ __ ] Okay. Oh no. So, anyway, thanks for thanks for sharing all that information. Um you know, there’s a couple more things that we could do here, but I actually want the you know, extra 10 minutes back to uh to go and actually implement some of this stuff. So, let’s let’s cut it here and let’s let’s get in the HTTP requests. Perfect. All right. Peace, man. Peace. And that’s a wrap on this episode of Critical Thinking. Thanks so much for watching to the end, y’all. If you want more Critical Thinking content or if you want to support the show, head over to ctbb.show/discord. You can hop in the community. There’s lots of great high-level hacking discussion happening there on top of the masterclasses, hackalongs, exclusive content, and a full-time hunters guild if you’re a full-time hunter. It’s a great time, trust me. All right, I’ll see you there.