How To Build A Production Ready RAG AI - My Full AI Coding Workflow
WDWeb Dev Simplified@WebDevSimplified · 1.8M subscribers
- Views
- 46.7K
- 46,675
- Outlier
- ×1.0
- Median 47.5K
- Length
- 3:07:00
- Published
- Jul 14
- Tuesday
- Comments
- 70
- 1.1K likes
Script
How it's written
- Words
- 43,851
- Words a minute
- 234
- Reading time
- 185 min
- Segments
- 130
The hook
0:00–0:25This video is going to be a master class on how you can create your own rag retrieval augmented generation system where essentially we are ingesting every single YouTube video and blog article that I've ever created. And then you as a user can search for anything you want and it's going to show you every single one of my articles and videos that cover that topic. Also, it's going to bring you to the exact point in that article where I talk about that particular thing. As well as in the video, if you click on it, it's going to bring you to the exact timestamp where I talk about
Structure
This video is going to be a master class on how you can create your own rag retrieval augmented generation system where essentially we are ingesting every single YouTube video and blog article that I've ever created. And then you as a user can search for anything you want and it's going to show you every single one of my articles and videos that cover that topic. Also, it's going to bring you to the exact point in that article where I talk about that particular thing. As well as in the video, if you click on it, it's going to bring you to the exact timestamp where I talk about that particular topic. I'm going to show you how to build all of that. But on top of that, I'm going to show you how I do this in a real world scenario. I'm going to be writing this code as if I write my real code. So instead of just showing you line by line what to do, we're going to be going through and seeing how to handle things with certain agents and when you need to write code by hand, how you can make sure the AI actually writes code that is good quality and to maintain that quality throughout your codebase. And I'm going to be going through the full production pipeline of how you deploy this and make sure all your credentials and so on are secure and how to separate that dev and production environment. Now, to make all of this work properly, we have a few different tools that we're going to be using. We're going to be using Neon for our database of choice. That's because they are a Postgress database with full branching support and we need a vector search database to make this whole retrieval augmented generation system work. So I want to give a huge thank you to Neons for supporting this video by sponsoring. We're also going to be using Verscell workflows. These workflows essentially allow us to do these longunning background tasks without worrying about them failing and breaking everything. They are super nice for ingesting hundreds or even thousands of videos and articles or doing longrunning AI related tasks. So I want to give a huge thank you to Verscell for sponsoring this video. We're also going to be hosting on Verscell since we're writing in Nex.js, but if you want to
Watch from 0:00use workflows, you don't have to be working with Nex.js or hosting on Verscell. You can do it entirely independently. And finally, to make sure our code quality is high quality, we're going to be using Code Rabbit for everything for our code reviews. Now, they also have this agent mode that you can use as well inside of Slack for communicating with it through Slack to make it work more seamlessly if your team uses Slack. But for us, we're going to be mostly focusing on the review portion. And the really nice thing is if you have an open- source public repository, you have access to free code reviews inside that repository, which is what we're going to be taking advantage of in this video. So we can get nice code review to make sure our code quality maintains itself throughout the entire lifetime of our application. And again, huge thank you to Code Rabbit for sponsoring this video as well as Verscell and Neon. Welcome back to WebDev Simplified. My name is Kyle and my job is to simplify the web for you. And instead of starting in the terminal with code, I actually want to start with a GitHub repository since when I create real world projects, the first thing I do is create a repository for them. So we're going to call this video blog suggest. And I'll just put yt at the end so I know this is for the YouTube video specifically. We'll keep this as a public repository. And everything else I'm going to keep the same. No readmes or anything like that. I just want to create essentially a brand new blank repository. That way I have somewhere that I can host and push my code to immediately. Now once that is done, the very next thing I want to do is I want to get our project actually set up. So let's just go ahead. We'll copy this command for actually initializing our repository inside of our project. And now we're going to open up just essentially a blank VS Code terminal. And what I want to do is I want to create our project. And many people make the mistake here of immediately turning to AI to create their project. I think by far the best
Watch from 1:31way to create projects is go through the create command for whatever tool or framework you're using. You're going to get much better results for your actual project. So in our case, we're going to be using Next.js. So I'm going to say npx create next app and we want to get the latest version and we're going to create it in the current folder. So I'll just put a period at the end here. This is going to run through to create that nextjs application for us. Now for this project it really doesn't matter if you use next or any other framework because our front end is incredibly minimal for this application and most of our code is going to be in the back end with these like durable workflows. So I'm going to actually go ahead and I'm going to customize the settings I use. I like to use typescript. We're going to use eslint for our default. I'm going to use the react compiler tailwind CSS. I always like to use a source directory. I'm going to be using the app router and I don't really care about changing the import alias default. And we also here can include an agents.md if we want to. We might as well include that just so it has that code for our agent to be able to work with. But again, it doesn't really matter because the actual Nex.js specific code we're using is incredibly minimal. Since this is essentially a single page application with no other content other than a search bar and a result section and all of the complicated code goes into building out how that search engine actually searches and gets the data that it can use. All of that embedding and chunking is where all the complexities of this application come in. Now, with that done, we actually have our project created. And I like to just immediately commit this project and deploy it to see that we at least have something showing up. So, let's just go ahead. We'll just run this as a dev dependency just to make sure that we have our code working locally.
Watch from 3:02And then I'm going to go ahead and push that up to production. That way, we can see that project working in production as well. And if I open this up, we should hopefully see something show up on our screen in just a couple seconds. And there we go. We have something showing up on our screen. super zoomed in, but you can see we have an application working. Now, the next step is going to be adding that to our Git repository. So, let's just go ahead and we're going to paste in that command that we copied earlier. This is going to essentially push all of this code to our GitHub repository. And now, if we go ahead and we open up that GitHub repository, we should see when we refresh this right here that we actually have all of our different code just like we want. Now, the next step is to actually try to deploy this just to make sure that it's working like we expect. Again, the quicker you can get everything working in production as well as development, the quicker it is to figure out if you have problems. If you build your entire application in development and then you push it to production, you don't know where in the actual process it broke if it doesn't work properly. So by incrementally constantly pushing to production, we can make sure every step of the way our project works like we expect. So since we're going to be hosting on Versell, we're just going to go over to Verscell, we're going to create a brand new project. So let's just add a new project here. And this is where we just want to import from that repository. Now in our case, it's not going to show up on this list. If I were to search for video blog suggest YT, it's not going to show up. And that's because I've configured Verscell to only have access to specific repositories. So if we just configure this GitHub app, we can actually come in here, make sure we choose the correct section that we want to configure. And we're just going to log in real quick.
Watch from 4:23And once that's done, we can now actually choose which repositories we want to give Versel access to. If we scroll down here, you can see I've only given access to a few different projects. We can come in here and add that YouTube or sorry video blog suggest. Make sure we get the YouTube one right there. We can now give that a save. And now we have access to that. And you can see it pops up on our list. And we can import that project directly into Verscell. Now since we're using NexJS, pretty much all these defaults are going to work for us, which is really nice. We can just click deploy. And it's going to deploy the entire project to Verscell for us. It's going to take a little bit of time, but hopefully once this finishes, we'll have a working application. So it just finished deploying our entire application. And if we click, we should be able to view our application. Again, it's just the boilerplate Nex.js. But we at least have now seen that we have code in GitHub. We're able to push our application to production and everything is working like we expect, exactly the same in dev and production, which is the perfect first step before we actually implement any code at all. The next thing that I like to do on pretty much any real world project is inside of GitHub, I like to go to the settings and I like to essentially make security around my branches. Now, to do this is quite easy. We can go to this branches section and we can add a branch rule set. And I essentially want to make it impossible for anyone to push to that main branch. So we can just say block main push. There we go. Make sure that this is active by default. And then what we want to do is we want to add a target branch. So we're just going to include that default branch. And then what we want to do is we want to restrict anybody from pushing to that branch specifically by making changes. We're going to require a pull request before merging. So just like that, require pull request before merging. If you want, you
Watch from 5:44can go ahead and you can actually add certain requirements for code review inside of here. I don't really care about that at all. We just want to make sure that it's going to have a pull request before we can merge directly to that section. Once that's done, we can just go ahead and click create. And this is just going to make it so that we can no longer push directly to our main branch. We have to create a pull request and then merge that pull request in. This just makes sure we don't accidentally push code which gets automatically rolled out because the way that we have our production setup working is anytime we push to our main branch, it's going to automatically roll that out to our production environment. So let's just go ahead and test to make sure that this works like we expect. Let's just change something inside of our code. So we'll just go inside of our page here. Anywhere we can just change this code to say new code. Just like that. Let's try to add this code. So we'll just say add new code. And we'll try to push this particular change. And we should say immediately it failed to push these particular commits. And that's again because changes must be made through a pull request. So we need to create a brand new branch and then we can actually merge that particular change in. And this is again just a safety feature for us. And it's making sure that all of the code that gets pushed into this main branch goes through a set workflow that we can really monitor, make sure we have code reviews, use code rabbit, and all those other things that we care about. Now, the next massive mistake people make when they're building out a project from scratch is that they immediately say, "Okay, I got my project running. I've got it in production. I got my branch system set up. I'm going to go ahead and start working with AI immediately to start building things." This is a massive mistake. AI makes writing code incredibly quick, which means people skip a lot of the thinking part of writing code and understanding what you actually need to build. Oftentimes, that slow aspect of manually writing code
Watch from 7:16means you have to think as you're writing code about what you're writing, why you're writing it, and what it does. When you have AI write your code for you, you kind of skip that part. So, it's very important to spend a lot of upfront time thinking about exactly what you want to build. This is why in pretty much any new project, I like to create a list of everything that I need to create inside that project so I can understand the overall picture of my product. And that'll also better help me communicate with the AI on exactly what I wanted to build. So, this can be anywhere, a whiteboard, a document, it doesn't matter. I'm just going to do it in a document right here. And we're just going to say project list.txt. Doesn't really matter what this particular file is. Let's just go ahead and actually make it a markdown file. That way, we can use markdown inside of here. And all I want to do is really just create a list of all the different things we need inside this project. I'm going to go ahead and do that and come back once I finish. Now, I just finished writing out essentially everything about this app that I can think of off the top of my head. You can see it's a not super short but not super long document. I kind of broke it down in a few sections. Like this overall section just says what the application is. Essentially, it's just a way to ingest all of my videos and articles and then chunk those down and be able to search through those by you when you search for particularly anything that you want. I then kind of describe the exact user flow. This is really, really important when you're thinking about your application. Think about how the user uses your application because that'll help you decide how you want to do different architectural things. So, for example, in our application, they first need to log in to prevent any type of abuse. They then need to search for whatever they want.
Watch from 8:46They're going to get a list of results which they can then choose from that list what they want to view. And if for some reason they send too many requests in a short period of time, they're going to be rate limited again to prevent any type of abuse on our particular system. I then gave a brief overview of what our UI is. Essentially, it's just a single page with a search bar and then there are results below that search bar and I kind of specified what I want in those results. For example, I want an image, a title, a description, match percentage and so on. I then kind of described how I want the ingest process to work. Specifically, I want to use these versell workflows to ingest everything to give me durability and retry systems built in. And I also kind of talked about how I want to chunk up the different information because the articles, for example, blog articles have headings with heading levels. I have very good hierarchical information and I want to make sure I preserve all of that different category and hierarch information inside of how I do my different chunking. Finally, I gave a very brief description of some of the different tables that I think I need inside of my database. This isn't going to be like the final tables or anything. It's just kind of what I think I need off the top of my head. So, once I've thought through my project at a deep level and spent quite a bit of time with it, then I go ahead and I do more thinking, but specifically with an AI companion to help me with that thinking. So, what I like to do is just open up any type of AI that you want to use. It really doesn't matter. What I'm going to do is I'm just going to open up a brand new sandbox with Pi inside of it. So, I can just type in start sandbox. This is a command that I created myself which is going to start up a brand new sandbox for me in this particular environment.
Watch from 10:08If you want to learn more about how you can actually sandbox your AI to make sure that it's more secure and easier to work with. I'm going to link a video up in the cards and description for you. This is just my sandboxing setup. I use Docker sandbox for it. But you can see that now we have a sandboxed AI with all the information we want. And inside of here, I can essentially ask it to ask me a bunch of questions. You can use, for example, some skills like the grill me skill from Matt PCO or you can just write your own very simple command that essentially says, hey, ask me a bunch of questions about this project and then give it a very lengthy description of your project. So in my particular case, I may reference my project. I'm going to say, I am creating a project. Here are the details. And then I would reference that file that I just created just like that. And then I would essentially ask it to ask me a bunch of questions. Ask me questions to further flesh out the plan. Make sure the questions cover all topics and architecture decisions. There we go. And I would just go ahead do that. And I would go through and answer all the different questions that it gives me. And the important thing about doing this is that it's going to help you think of all the things that you didn't already think of. It's going to ask you questions about various different things you probably never even thought of or considered. And this is going to help you flesh out this document even more and really figure out what you want. And often what I like to do is after I'm done answering all these different questions, I like to ask the AI to specifically go through and essentially create a document summarizing all the different things we talked about, explaining exactly all the different things inside of my project that I want to do. So it'll go through
Watch from 11:31it, ask me a bunch of questions. I'm not going to actually go through and actually answer all these different questions. The main reason being that I've already kind of gone through this process the first time I built the project, but this is exactly what I would do when I'm building out the project from scratch. And that gives me all the planning that I need up front to actually start working on the project. And even if you're not using AI for any code writing at all, this is still a really great step to make sure that you're writing the actual code that you need to and that you have essentially a list of tasks of all the things that are important to work on. Now, from here, the next thing that you want to work on is figuring out what your first task is you want to implement. In our case, I think the best first task to start with is just going to be a simple user authentication for log in and log out as a relatively easy and simple thing to get started with and we're going to use it everywhere in our application. Now, again, many people just go to AI and say make me off or something like that. That's obviously not going to give you the results that you actually want. What I like to do is I like to do a lot of my setup manually and then I like to use AI to either enhance it further or to write code on top of that setup that I've created. Now, when it comes to setting up our authentication, we first need to set up Drizzle and then we can set up better off. And I'm going to be using the newest version of Drizzle, the Drizzle 1.0 release. It's not quite stable yet. This is the release candidate version, but hopefully by the time you watch this video, it will actually be stable. So, I'm going to go over to the Drizzle documentation, and we're going to be using Neon for our database. So, I'm just going to select Neon, and I'm going to go through and manually set this up. And the main reason I'm doing this is because we're essentially using a version of this drizzle Drizzle library that's not in a lot of the training data. So often the AI is going to get this wrong. So I'm just going to go through and set this up. And it's also very easy to set up.
Watch from 13:02So I don't see a reason not to do it. So let's go ahead and just make sure we copy over the different things that we need to install. So let's just create a brand new terminal to install everything that we need. Just like that. And then we're going to go through and you can see we need to set up a connection environment variable. That's perfectly fine. So inside of ourv, which it doesn't even look like we have yet. Let's just create one.v. We can create a database section just like that. And we'll paste in our database URL just like that. Next thing I need to do is you'll notice it is using the release candidate version of all of this different software, which is exactly what I particularly want. Next, we need to actually connect Drizzle to our OM. So, let's just copy that file. This is going to be how we access our database. So, inside of our source, I'm going to create a new folder. We can call this whatever we want. Let's just go ahead and call this our DB folder. And inside here, I'm going to create a file called DB.ts. And we're going to paste in that file. And if we just export this variable right here, we now have access to our database directly. Now already you'll notice an error with our process env. You know I could override this by putting an exclamation point here which is probably what AI would do. But in my case I want to set up typed environment variables. We're going to be using T3 env for that. So we can just go ahead and make sure we install the dependency for that. I'll copy that over. Make sure we just install that dependency. Just like that. Now we have that inside of our project. And since we're using Nex.js, this is actually super easy to set up. You can see here we have our create env file with all of our different enviables. And that's pretty much all that we need to do. And I'm actually going to be breaking this up in between a client and a server version. So inside of our source folder here, we'll create a folder called data. And inside of here, we'll create a client env.ts.
Watch from 14:30And we'll also create a server env.ts. Just like that. We'll paste down that code that we got from here. And this is where we set up all of our code. So in our case, database URL is the only URL we have. So let's just go ahead and make sure that's our only section. We're breaking apart our server and client. So we're just going to do this. And since we're doing this inside of only the server, we can actually use the experimental runtime version and just set that to process.env. Just like that. And that's going to be everything that we need to actually set this up inside of our server. Now, I'm going to be calling this variable server env. And I also want to make sure that my empty strings are considered undefined. That way, if I pass along an empty string, it essentially instead of using as a string, just treats it as if the variable was not defined at all. So, there we go. We now have our client or our server env set up. I'm going to do the exact same thing for our client. Just like that. So, our client EMV, we'll call this client. Right now, we don't have any data inside this section. And instead of using the experimental runtime, we just have to use our normal runtime. And we're going to have to pass in each object as we use it. Just like that. So, we'll just leave this blank for now cuz we don't have any client environment variables, but we will in the future. Now, this means that here we can use that server environment variable just like that. Get access to our database URL and everything is typed just like we want. So now with that done, let's go ahead and actually finish out what all of our Drizzle configuration is going to look like. If we scroll down a little bit further, the next step is going to be creating our schema for all of our different code inside of our application. So let's just create a very basic boilerplate schema.
Watch from 16:01So inside of our application where we have our database, I'm going to create a new folder called schemas. And I'm going to create a file inside of here. We'll just call this test.ts. And essentially, I just want to take all the code that's inside of here. And we'll copy this over as a starting point for our application. Now for our particular application, I want all of my different name key value pairs here to be lowercase snake case values. So what I can actually use is I can use the snake case type just like this and I can say snakecase.t and that's going to create a table where all of these names are going to be snake case. So for example, if I had like first name here, it's going to convert that to first name for the actual database table name itself. Now this is just a test table. So we can just leave this as is. Maybe we'll just give it an ID and a name just like that. So now we have a test table we can work with just like that. Now the next thing I want to do is I want to export that schema so I can use it in my database. So I can create a new file called schemas.ts. And all I'm going to do is I'm just going to export the file directly. So I'm going to export that schemas.ts. And I want to make sure I export everything from there. So I'll just say export star from and we can say at slash or actually dot slash schemas slashtest.ts. That should be working properly. And it looks like I just need to remove that.ts from the end there. And there we go. Now that is working like I expect. And I have all of my schema related information available. Now to make sure I have that access to that information in here. So for example, when I type in db.query, I want to be able to have that test table show up inside this list. To make that work properly, we need to pass along essentially a relations property.
Watch from 17:25So if I just do here relations, that is what we need to pass along. So we're going to create a brand new file for our relations as well. Just like that. And this relations is going to be relatively straightforward. We can export a constant variable called relations and I believe it is called define relations just like that. And what we want to do is we want to pass along our schema. So let's just get all of our schema information. So we'll say import star as schema from and we want to get that from that schemas file. Just like that. That's going to be all of our schema information. And then we can define relationships between our tables as we have them. But right now we only have one table. So there are no relationships. So this is all we need to do to define our relations. And in here we can just import that relations. So let's just add an import. And now we can come in and say db.query dot. And you can see we have that users table being shown up. And that's because this is called user table. If I just call it tests table or something else. Now it'll actually show up as test table inside of here. There we go. Just like that. So now we have everything set up for our database. All we need to do is make it so that we can actually push and migrate all of our different code. And we need a drizzle config for that. If we come over here, you can see set up a Drizzle config. Give it this exact name and paste in all of this information. So, let's just go ahead. We'll create a brand new file called drizzle.config.ts. Let's make sure it's at the root of our application. And we'll paste in that file that we just copied over. And you can see we just want to make sure all of our output and schema directories are all in the correct location. So, for example, for our output for all of our migrations, I want that to be in this DB folder here. So, that's going to be dot /source/db/migrations.
Watch from 18:56That's going to give us all of our migration code. And for our schema, that is/source/db/s schema. I believe that is the exact location we have it. Actually, ours is called schemas.ts. I think I'm going to rename that to just schema because that's a little bit easier to work with. There we go. Make sure I save that in all the places it's imported. Our dialect is correct. And for our URL, again, we can get that from our server env. And we just want to make sure that we get the database URL. There we go. That should be everything that we need. Now, let's just make sure our env config is the first thing that loads. Now, really the last thing to configure before we can actually test this is we need to set up an environment variable for our application. So, to do that, we're going to go over to Neon. We're going to log in and we're just going to create a brand new database. So, let me just get this zoomed in so it's easy to see. We're going to create a brand new project. I'm going to call this video blog suggest YouTube just so we know this is the YouTube one. Keep whatever Postgress version we want. We're not going to be using Neon for authentication, but they do have a really nice authentication system. But ours is so simple, we're just going to be using our own O system with better off. Once that's done, we now have a database set up. And by default, we're on the production branch. But the nice thing about Neon is that we can actually branch our database to make it easier to work with for various different uses. So I'm actually going to create a brand new branch from this, which is going to be a child. So let's just go over to our branches. I want to create a brand new branch. It's going to be a child of production. We're going to say it never disappears. And we're just going to call this dev for our local development. And we'll branch over all the different information. Right now, our database is blank, but that's going to copy our information. And here, what we can do is we can just copy over all of this code right here. And now in our application,
Watch from 20:30we can paste that down as our database URL. And we're able to access our database directly using this URL. Now, that should be all the configuration we need for Drizzle. And again, you can kind of see why I did this myself instead of letting the AI do this because there's a lot of opinionated things on where I want my files and how I want my code structured that would require me to essentially change everything that the AI does. So, it's much easier just to do it myself in my opinion. Now what we can do is we can actually go ahead and set up our commands for this. So we're going to have a db migrate command and that's just going to call drizzle kit migrate. We're also going to have commands for generate. So we're going to come in here with generate and we're also going to have a studio command which is just going to make it easy for us to view the data in our database. Just like that. Now the generate command is going to generate a migration file. A migrate command is going to migrate those changes for us. And also let's add in a push command here which is essentially just going to push our database changes directly without doing anything else. So we kind of skip the whole migration process. This is useful for like testing while generate and migrate are very useful for more production style environment. So let's actually test to see if we can push those changes to our database. We just run npm run db push and hopefully it should take those database changes and push them directly to our neon database and we should be able to see those changes inside of neon. So give that a quick second to load. You can see it's reading through everything. It's pulling our schema and it looks like the changes applied properly. Now, if we go ahead and we check inside of Neon, we should be able to click on this table section and we should hopefully see there we go. Our test table is here. So, all of our data was pushed properly.
Watch from 22:00Obviously, there's nothing in our database, but we have that table, which means our connection is working like we expect it to. Now, what we want to do is since we've essentially created a point where we have working software exactly like we want it, I want to commit those changes as like a save point for our particular application. And again, we're on the main branch and I want to swap over to a new branch. So, we're just going to check out a branch. And this branch is going to be called setup off because that's what this branch is all about. And I'm just going to add these changes that we've made. So, we can just say get commit setup drizzle. There we go. So, now we have our drizzle set up. And I can even push this along if I want. And it's going to create a brand new branch for me just like that. I can go ahead and I can even create a pull request for this by clicking on this link. but we're not quite ready for a pull request yet cuz we still need to do the authentication portion. Now, the next thing we want to work on is setting up better off. And this is a much more involved and kind of boring process in my opinion where there's not a lot of opinionated stuff that you need to worry about. This is a perfect task for AI. So, what I'm going to do is just get started by installing the better off library. So, I can say npmi better off. And since we're using the release candidate version of Drizzle, we're going to get a warning. So, I'm just going to type in d-force here. essentially forcing it to override the actual version of Drizzle that better off expects. That's because Better Off already supports this newest version of Drizzle and everything works perfectly fine. We don't have to worry about any errors. I'm just essentially forcing the npm compiler to work properly with installing better off.
Watch from 23:30Now, once that's done, in order to get the best out of AI, one of the best things you can use are skills. So, if we actually take a look at Better Off and we search through the skills directly, you can see that Better Off themselves have actually produced a lot of different skills. And if you're unsure what skills are, I'll link a video in the cards and description for you going over everything related to skills. But these are essentially ways to supercharge our AI with exact knowledge on the things we want it to do. So in our case, we wanted to implement better off properly. So we're going to give it the skills to do that. So we can click on one of these skills. And since I want to look through all of the skills for better off, I can come up to this section right here where it says skills. And you can see I get a command npx skills add better off/skills. This is going to allow me to search through and add all of the skills related to better off that I particularly want. So, we're going to come back over here. I'm going to paste this down. And I want to install these skills essentially directly inside my project because they're useful for my project. So, if we take a look at all these skills, you can see this better off best practices. That's probably a good one to install. The create off skill is going to be for scaffolding out our first authentication. Obviously, that's what we want to do. So, that's perfect. And we even have some other ones. For example, we're not using email password. Organizations are two-factor. So, we can skip those. But, it might be good to have some security best practices as well. So, those are the three skills I want to install. We're going to go through. By default, it's going to work with PI. So, we can just kind of skip this by hitting enter. I want to install these in our project. And we can go ahead and click yes because we want to install all of these. Now, in our agents folder, we have those three skills.
Watch from 24:52We're essentially just markdown files explaining exactly how things work inside of better off. Once that's done, we can actually use our agent to do the code that we want. So, let's go back into our agent. I want to just type in /reload here. There we go. Looks like I'm getting some sizing issues. There we go. Let's try that again. /reload. That's going to reload. I want to get a new session. And we should see here we go. Our skills are properly loaded just like we expect. So now this is where we can essentially prompt the AI to start writing this code for us. So let's just go ahead and we'll say that we want to prompt it with the skill for creating O. So we'll use that skill/create. And now we need to tell it what to do. I want you to set up better off to use GitHub for authentication. You can replace all the content on the main homepage with just a simple login/out system. Also show the users profile image and name. There we go. That's essentially all that we want for this application to do. So we can write that out and it should go ahead and actually implement that for us. Now, obviously AI takes a little while to write the code, especially depending on what models and so on you're using. I'm actually using local AI for everything. So, it's even a little bit slower than if I was using like a Frontier model for all of this.
Watch from 26:11So, while the AI is processing, you could scroll through Tik Tok on your phone or something like that. But, that's a very bad idea in my opinion. What I would like to do is other things in parallel to what the AI is doing. So, in our particular case, something we can do is to set up all of our API keys for GitHub and so on to work properly with our authentication system. Now, while I was getting the GitHub stuff set up, I actually noticed it asked me a bunch of questions. So, let's just go through and answer these questions. Only GitHub. We don't care about any other providers. Email password. No. Email / password. Email verification. No. Let's see here. Additional features. No. We want off pages. No, we don't care about that. We just want it to be as simple as humanly possible. And for the UI style, what style do you want for the off pages? Minimal. Yeah, we'll just say that we want m well there is none since we have no off pages. There are no off pages. There we go. That gives it all the information. Essentially, I want it to be as minimal as possible. Now, with that, we can actually go ahead and start working on our GitHub application. Now, for us, we're just going to be using a simple OOTH app because all we care about is authenticating. You can see I actually have quite a few of these already set up. So, let's go ahead and we're going to create a brand new OOTH app. This is going to be our video blog suggest. This is the YouTube version right here. And we're going to create a separate one for development as well as production to keep our systems entirely separate from each other. So I'm going to call this one our development environment. And I might even just put that in all caps so it's very clear which one is dev and which one is production. Now for our homepage URL, this is just going to be that localhost URL. So we'll just say
Watch from 27:33localhost 3000. And for our callback URL, we're going to use the exact same localhost 3000 as our callback URL. And that should be all we we need to do. So let's just go ahead and we're going to register our application. And that gets our application set up and we can use the client ID and client secret. So let's copy over that client ID and we want to put that directly in our env. And it looks like our AI asked us something. We'll get to that in just a second. Let's just go to our off section. And we have our GitHub client ID. Paste that in. GitHub client secret. And let's copy that over as well. So we can come over here, generate a brand new client secret. Get that information right there. And we're going to paste that in. And now let's actually look at what the AI is asking us to do. Before I proceed, you'll need to create a GitHub oath app if you haven't already. Blah blah blah blah. Um, do you have a go off credentials ready? Use the enviables. And I'll paste in the name of those variables for the credentials. There we go. Hopefully that'll actually go ahead and set everything up. So this is our development version of the application, but we essentially need to create a brand new version of the application that is also going to be for production. So let's go back to our GitHub developer settings and I want to essentially create a brand new OOTH app. This one is going to be video blog suggest for YouTube and this one is our production version. So we'll put prod at the end. For our homepage URL, we can actually go over to our first cell deployment of this. So if we just pull that over, you can see right here is my domain. So let's just copy that over.
Watch from 29:03Obviously, you'd probably use a real domain in a real world project, but this is the domain that we'll use. That is going to be our homepage URL. And our authorization authorization callback URL will be exactly the same as that. We can register that application. And again, now we have our client ID and our client secret. We'll just keep this off to the side. When we're ready to deploy this to production, we'll make sure that we add those different environment variables to our production instance as well. Now, let's go ahead and you can see that this is still working on all of the different code for our application. And if we look at the actual sections that are being generated, it does look like we have some code being generated. We could start reviewing that if we want, but I generally like to wait for the AI to be finished before I start reviewing the code. So, for now, I'm just going to let this finish out because we pretty much have everything set up. Actually, we might as well since we have some time to kill. Let's go to Verscell. We're going to go and find our environment variables, and I want to add some environment variables. So, let's go ahead and copy those values. We have our client ID. That is going to be our GitHub client ID. Make sure we paste in that value just like that. And we're going to add another one which is our GitHub client secret. Just like that. And again, let's go over to here and generate a brand new client secret. Copy that value over as well. And we'll paste that in so that we at least have that set up completely done. We'll give that a quick save. We have those two. We also are going to need to have our environment variable for our database URL set up. Let's just again check our AI is still working on its project. So, let's just go ahead and pull over our neon. Let's go to that production branch of our database. And what we want to do is we want to get the connection for this. So I believe if we just go to our dashboard, we can click connect right there. That's going to be our connection
Watch from 30:33string. We can copy that over. And now we can go back into here, add an environment variable. This one is our database URL. And we can paste in our database connection string. That's another sensitive variable. And we'll save that. And now we at least have all three of those different environment variables in place to use in our application when we're ready to deploy. Let's go ahead and you can see our AI is still working on this. I'm just going to come back to you as soon as this finishes. So, the AI just finished up with configuring everything inside of our application. It really didn't take much longer after I ended that last clip. So, let's go through and actually review the code. This is one of the most important steps is review all of the different code that is being changed. You can see right here, it's actually adding some code to our drizzle config to point to a separate location for our schema for our off section. And that's because by default, the way that our off is being generated, it's not putting it inside of our schemas folder. It should be putting it in this folder, but it's not actually putting it there. That's something that we need to make sure we configure with better off to make sure it works like we expect it to. Now, inside this lib folder, it looks like we have our off and our off client. And this is essentially where I want to configure everything. Also, we are using the old version of the drizzle adapter here, which is probably going to cause some errors inside this section of our application. So, we need to make sure we fix that as well. So, inside this off section, let's actually make sure we use the correct Drizzle adapter. This is going to be at slash better off slashdrizle adapter slash relations v2. That's essentially going to work properly with the newest version of drizzle that we're using. Now to use this adapter, we need to actually be using the release candidate version of better off. So let's just go ahead and make sure we install that as well. So we
Watch from 32:03can say npmi better off at rc and again make sure I put that force in there just to make sure everything works properly. I should have installed the release candidate version to begin with, but I just forgot and didn't actually do that. So, once that's finished installing, it should hopefully have everything that we need inside of our application setup. And we can go through and make sure that all the rest of the code inside these sections looks like we want it to. Now, one immediate thing that I noticed is I like to use UU IDs for all my generations. And by default, it uses just text IDs. So, we're actually going to change that. We're going to say in this advanced section, what I want to do is I want to get my database. I want to change the generate ID to UU ID. That's just again because I like to use UU IDs for every single thing that I do. And we can also pass along our schema inside of here. So we can say schema and just import our schema. Whoops. Make sure we actually import that from the right location. Import star as schema from at /db/s schema. There we go. That's going to make sure it has access to all of our schema information to set everything up properly. I'm also going to slightly change how our off is set up. So we have this lib folder, which is great. I'm going to create a brand new O section inside of here. And inside this O section, I'm going to have this file be called config. And then we can have our off client inside of here, which we'll just call client. This is just going to make our code a little cleaner. So, which we can put everything inside of these folders just like that. Other than that, all the code inside of here looks pretty good and we can keep this as is.
Watch from 33:35Now, the next thing I want to do is I want to regenerate this off schema using all the new proper ways of doing things. So, we can essentially delete this existing O schema. We can go into our package JSON. I'm going to create a brand new command called off generate schema. And this is going to be a command that uses that better off. So we'll say off generate. That's going to generate it. And we just want to say where our output is going to be. In our case, our output is going to be inside of this schemas folder. So I can honestly just copy this path. Make sure it's relative. We can paste that down. And of course, we'll just want to swap the order of these. There we go. So it's going to go inside that schemas folder. Just like that. And let's just call this off.ts. That's going to be all of our off code. Then we can specify where our config is going to be, that's going to be in our slashsource/lib/off/config.ts. And then if we just do d- yes, that's going to essentially force it to go through with the command no matter what. So now if I run npm run and let's see what we call that, offgenerate schema. That should hopefully regenerate our schema using all the brand new stuff for us. Now it looks like we don't have that O command installed. So let's just npmi. This is a dev dependency off and we want to get the release candidate version. That's going to hopefully install the newest version of the command line tool for better off. And again, let's make sure we do that d-force to make sure that it works properly with drizzle kit cuz again it doesn't quite recognize the release candidate version of Drizzle.
Watch from 34:58But once that goes through, it looks like that finished. We now should be able to run our original command of Ogenerate schema and that should generate a brand new O schema file inside the schemas folder that's compatible with the newest version of Drizzle. And if we look inside of here, you can see it's using define relations parts, which is a brand new Drizzle feature. And it has all of the different code inside of here for our relationships right here, as well as all of our different schema files. We just need to make sure we use this off relations in our relations file here. So where we define our different relations, we can pass that along. And actually that goes right in here. So where we have our relations, this is where we want to pass that along. So we can say our relations is a combination of all of our relations as well as all of those O relations as well. So let's just make sure that we import this author relations just like that. And now we have all of our different relations. So I can say db.query and you can see I get all of my different information. So for example, I get my account information and so on which is coming directly from right here inside of Drizzle. So let's continue to review the code to make sure everything looks like we expect it to. And you'll actually notice I like to review the code before I check the implementation because if I check the implementation and it works, I tend to skip the code review or do less code review. So this forces me to actually go through a more thorough code review. So inside of our Drizzle config, we can actually get rid of all the changes because we now have our off schema in the correct location. So let's discard all these different changes. And in our schemas file, which we have right here, let's just make sure we export star from that/s schema/off. There we go. And that's going to set up all of our off related code like we need. Of course, our package JSON had some updates, but we pretty much did all those manual. Our
Watch from 36:18skills right here, those were all generated by us. So we can skip past those. And now we can move on to our actual application itself. So our config, we've already double checked. We know that's fine. Our client, this looks fine as well. So we can keep that as is. We have this file here which is autogenerated. We don't have to mess with that. Our schema, we made that change ourself. Database, again, we made that change ourself. And then our server environment variables, which is exactly what we want. That is great. Next, we can check essentially the page as well as the route to make sure this is all good. You can see here the route section. This is just basic better off code. This is correct. I've written enough Nex.js better off to know that this is exactly what you do. And now we have our page itself. And you can see here essentially all it's done is replace the code. So I'm actually just going to go to the page itself to view the code directly rather than look at the diff because it essentially replaced everything inside this application. And this is actually a great place where it makes sense to actually just test the code yourself. So let's just come in here, make sure all these files are saved properly. I'm going to run this in development and let's just test to see what it looks like and if it works because we don't care too much about the UI at this point because this is more just a boilerplate testing application. So let's just make sure we scroll this back and actually I'll just open this up in a brand new URL and you can see here this is opening up but it's giving us an error. Default migration lock table doesn't exist in target module. So there's definitely something going on interesting with our better off and my guess is it's because we haven't actually pushed our tables to production yet. So, let's just go ahead and we're going to make sure we push those changes to our table. So, we can just go ahead and run npm run db push.
Watch from 37:49And that's going to push all of those different tables directly for better off. And it's going to push it to our development server because that's what we're hosting on right here. And now, if I give this a refresh, it still doesn't look like it's quite working. So, we may need to change something else inside of our code. Let's just reset our application by npm runningdev again. And you can see it's still giving us that warning. One thing that we can try is we can try importing the minimal version of better off since we only need GitHub authentication. This could be part of the reason that we're running into these problems. Let's go ahead and see if that worked. And if we pull this over, you can see welcome sign in with GitHub to continue. If I click sign in with GitHub, it's going to ask me to authorize. So, let's just go ahead and authorize that. We're being redirected back. And it does look like it actually properly signed me in. And it has all of my information here. Now, we can go ahead and change this cuz we don't really care about the email of the user being shown at all. We should also make sure sign out works. So, let's just click sign out. And that does work like we expect it to. So, I can just go back to my AI and I can say, "Please do not show the email of the user." There we go. That is all done. And now, let's just go ahead and make sure the rest of the different files that it modified are correct. And again, that's just that page file. So, I don't think there's really much we need to worry about here. The nice thing is a lot of this code in this page file is going to be overwritten or changed in the future. But if we just kind of glance at the actual code to make sure it's working, you can see it's using that signin social with GitHub. And that does look like that's all it's doing as well as sign out. So it is working just like we expect. Now I do believe in the env section, we need to make sure we have a random string of characters here to work with. As well as I believe we got some warnings when we actually ran our dev
Watch from 39:20environment. If we just go back to our dev environment, warning. Let's scroll back down to that. Better off base URL is not set. Set the base URL option or better off URL environment variable. So we can set that environment variable to just a HTTP localhost 3000. Just like that. So now at least we have that environment variable set. And if we close out of that, that at least should get rid of that warning for us, which is great. Next, we want to make sure we have a better off secret. This is just again a random string of characters that we can generate. Now, we could do this ourselves, but the nice thing is better off just has a system to do that. We can just click generate secret, and that'll generate us a brand new secret. So we can paste that directly into here. That's just going to give us a little bit of security with our better off implementation. And we can also generate a brand new secret. So let's just scroll back up to where that section was. And we'll generate a brand new secret. And this one is going to be for our production environment. So we'll copy that over. And now we will use that inside of production. And inside of our application, let's add an environment variable. This one is going to have that value. And this is our better off secret, I believe, is what it is called. So let's just make sure we save that one. And then we also need our better off URL. So again, let's make sure we copy over what our domain is going to be. And we're going to paste that in as our environment variable. So we'll call this one our better off URL. Just like that. This one's not sensitive, so we don't need to worry about the sensitivity on that. And we can just give that a click save. And now we have all of our different environment variables set for hopefully using this application. Now, kind of the final step since we have essentially a complete feature that we're done with working on, what I want to do is I actually want to
Watch from 40:51generate the migration files so that we can use this inside of our database. Because when we push to production, we're not going to be pushing our changes to our database. We're going to be migrating our database so that all of our data stays the same and doesn't get deleted or wiped. So, we can run the npm run db generate command and that's going to generate some migration files for us that we can run in production. So, if we take a look inside of our DB, we have a now brand new migration file. And all this does is create all of those different tables for us. Now, immediately I see a problem. This test table probably shouldn't exist. It was just there for testing. Let's go ahead and remove the test table from here. We're going to go to our schema. We're going to remove our test table from there. That should clean up all of our test related stuff inside of our code. I then want to regenerate those migrations. So, let's just delete the migrations that we generated. We're going to regenerate our migration. Should completely remove that test table from it. So, now if we look at our migration, you can see all this does is it just creates the account session user verification table as well as any different indexes and so on that need to be on those tables. So now when we push to production, we should hopefully be able to run this migration against our database and it'll update our database to include this exact information so we can log in and so on and test everything inside of production. Now, there is one thing we're going to want to do on our actual production environment. When we do a deployment, we're going to want to slightly change how our deployments work. So we just want to go into our settings and we want to find our deployment settings. And you can see right here we have the build command. By default, it's just going to run npm run build, which makes a lot of sense. But we also want to migrate our database. So let's actually override this particular
Watch from 42:21command. What we want to do is we want to run the command for db migrate. That's going to migrate our database. And then we can run the npm run build command to actually build our actual environment. And then if we just give that a quick save, that's going to be everything we need to do to make everything work. So now we're going to be pushing all of our migrations and migrating our database and then rolling out our code by building that and pushing that to production. That way our database and our production roll out happen at exactly the same time. So now if we just give that a quick save like we've done, we can come over to here and we can hopefully push this to our branch. So let's just add all of these different changes and we'll just say get commit add GitHub off push those changes. Now we have that brand new branch. And if we go ahead and we actually check out that branch, we can create a pull request and so on. But before we do that, I actually want to create code rabbit support. So we can get pull request reviews directly inside of that. So if we just log into code rabbit, we can come over here and click on this repositories button. This allows us to add a brand new repository. And what we want to do is we want to search for that repository in our GitHub that we just created to be able to add that into our application to work with. So you can see here if we scroll down, we want to select the repository that we're working with, which is video blog suggest. And this is the YouTube version. So it's got -yt at the end of it. Whoops. Looks like that disappeared.
Watch from 43:52YouTube or sorry video blog suggesttor YT. There we go. Click on that. And now we have access to that. We can give that a quick save. And now hopefully we should be able to see that show up in our repository list. Give it a second. There we go. And we can click on that. This is a public repository. So again, we have all of those different supports and everything should pretty much just work for us out of the box. We don't really need to change anything at all. So now let's just actually go ahead and make sure that we pull and create that pull request. So right here we can see setup off. That's the branch we just created. We'll compare that and create a pull request. Give that a quick second. We can just call it whatever we want. We can add whatever description we want based on as complicated or so on as we need. Since we're just working with oursel, we can keep this simple. And we'll just create a brand new rep request. Just like that. Give that a quick second. And now we have a pull request that we can go through. We can have other people on our team actually review and make sure all the code inside here looks good before we actually commit those changes directly to our repository. Now immediately looks like we are getting an error when we try to deploy. So let's go ahead and try to debug what that is. We can click on that. You can see the command npm install is what actually failed. And you can see here it's just having issues with our installation because we have kind of two conflicting packages. It doesn't know which one to actually use. So we can change our settings for install to use that force command as well. So what to make sure we come over to here where we have our project settings install command and we can just say npmi-force.
Watch from 45:12That's going to essentially override that to hopefully work like we expect it to. So now what we can do is we can just kind of go back to that deploy where we were at and we should try to redeploy this just like that. And this should hopefully work properly just like we expect. And now we have essentially a deployment that we can access and use and view everything for it. Now that finished deploying so we can actually click on this and hopefully this is going to work like we expect as well. Let's just try a quick signin with GitHub and see if this works. We may have some issues because our actual deployment URL does not match the GitHub URL. And you can actually see just like clicking this, it's actually not working. And I believe the entire reason for that is because this URL does not match the URL that we have inside of our environment variables. For example, if we go to our environment variables, you'll notice that this better off URL is specifically for our production deployment and not the actual deployment here, which is like the deployment preview. So that's probably where we're having issues. Also, you'll notice that we have these environment variables saved for production and preview, which for a database URL is definitely not a good idea. This should be set to just production only. So you can see here this should be a productionon version of this environment variable. The reason for that is because this database URL is our production database and we don't want to make sure we overwrite this inside of our preview environment. You could do the exact same thing with our GitHub stuff as well if you wanted. But for our case, we're just so focusing on this database URL cuz that's the main one we care about and we don't really care too much about these preview builds if we don't really want to. For now, I'm just going to skip this preview build and not really worry too much about it.
Watch from 46:29Now, if we go back to our poll request, we will notice if we scroll down that it's actually created a summary of us of all the different changes. And you can also see here that it's currently going through and reviewing all of these different changes. And once that's done, we will go through and actually look at all the different things from that review. So, our Code Rabbit review just finished. And one thing I do kind of want to mention I really like about Code Rabbit is this review change stack is really nice if you're reviewing the code because it actually breaks it down into sections of where the different changes are. For example, it has all of our environment related changes. Right inside this section, we have a whole section for database schema changes inside here. So, we can really easily see what's changed inside of these different sections. Overall, it just makes it really easy for me to go, okay, these are the different groupings of changes inside of our section, which is nice for when you're doing a review, cuz that's sometimes one of the hardest parts. Now, if we just go back to our actual application, we can see here all the different rest of the stuff related to our Code Rabbit review. Now, another nice thing is it has this whole walkthrough section essentially showing you everything that's changed and all the different stuff you need to know when you're reviewing. Again, it's nice because it makes it really easy for you to know what you want to look at when you're doing the actual review. Now, if we look at Code Rabbit's actual comments, you can see there's really not much that it had to change with this review. You can see there was three nitpick comments. You can see here there's a security and privacy potential recurring pass the validated secret explicitly instead of relying on better off ENV fallback. Again, this is essentially saying to use the value directly instead of relying on that better off secret having this exact specific name. The nice thing is it gives us a really good suggested fix right here. Tells us exactly what we need to change. And I think this is a good change to make and also even has a
Watch from 47:52prompt for the AI if you want to have your AI do this. But for this one, since it's a single line, let's just go ahead and change this ourselves. We'll just come into here. We'll go to our application. We'll find that exact location which was in our off configuration. So, right here. And we just want to make sure we pass along that secret information. So we can just paste that down just like that. And now it's going to be properly using that information instead of relying on the naming being exactly the same convention. Let's go ahead now that we've got that one done. Let's look at the next one which is our server env. Consider enforcing a minimum secret length. This is validated with a string. So say that it must be at least 32 characters. That is again another really good requirement. And again that's super easy to implement. We can just copy that code over. And let's go ahead to our environment variable and do just that. So our server env we should make sure that this is at least 32 long because that is the recommendations by better off. Next we can minimize that down and we can go to our page. You can see here consider next.js or next image for the avatar instead of using the raw image. That I agree is probably something but again it's not super important that we work on that and that's because that we're not going to be keeping this exact UI around forever. So for that I'm just going to completely skip that one. But those other two changes I do think were good things to change. And I will actually just go ahead make sure I save all my code and we'll just add those changes. Get add. We'll say get commit minor fixes.
Watch from 49:22And we'll just push that up. Of course, use better commit names than that. But for video purpose, I think that's perfectly okay. That's going to push up those different changes. And since we know everything else is good for this particular project, we should be able to actually merge this and hopefully everything is going to work just like we expect. So, let's just merge this particular pull request. That's going to push it into our main branch. We can delete our original branch because we no longer need that anymore. And now inside of here, if we go to our deployment section, we should see a brand new deployment for production that's being queued up just like we expect. This minor fixes one we don't really care about. We can ignore that one. It aired out, but hopefully that's not a problem. Oh, it's ering out probably because it can't migrate to the database because we don't actually have a URL for that database at all to be pushed to. That makes sense. So, that's perfectly fine. This one should actually work though because we do have that database URL. And again, the only reason a production or the previews don't work is because we made it so our database URL is only production. You would want to create a brand new environment for your preview and save that and do that. But for our case, we have just a production version. So, it's only going to work in our production deployments. So, I'll come back to you once this finishes deploying to make sure everything works. That just finished deploying. So, let's go ahead and test to make sure everything works. We click sign in with GitHub. And that should actually sign us in. Let's just make sure we refresh. Looks like it's still not properly signing us in. It might be because we're using the wrong URL. I believe if we go to our overview and we click on the URL here, this is our actual full production URL. Now, if I sign in with GitHub, that should work properly, but it looks like it's still not quite working like I expected to.
Watch from 50:40Oh, there we go. It was just a little bit of a delay from slow internet. Now, if I click authorize, it's going to redirect us back. It shows the name. It shows the picture just like I expect. And when I click sign out, you can see it signs me out. So, everything does work as I expect, which is perfect. So, now that we've got all of our O set up, that's kind of like the boilerplate of getting our code in the place we want it. Now we can actually work on some of the more exciting things. We can at least look at our project list to see how far we are. So currently we have step one entirely done. We can now log in with GitHub, which is great. And our UI doesn't have anything yet, but that's okay cuz we don't have anything to display on that UI. The next step is we want to be able to let the user search for a topic. But to do that, we need to ingest all of the content. So we're going to be using those Verscell workflows to ingest all of our data. We're going to chunk it. We're going to embed it inside of vector embeddings using AI for all this embedding. And this is really the more complicated part of our application. All of this rag implementation that's going to be happening directly in this ingest step right here. And we need to ingest both blog articles and YouTube videos. And they're going to have some overlap but also some differences between them. So I want to make sure that we handle all of that properly. So in our case, we're going to be setting up an ingestion system. And we're going to start with our blog articles first cuz it's going to be a little bit easier to work with because there's a lot less of them and they're a little bit more simple. And then we're going to move on to YouTube videos after that because we need to deal with the YouTube API which can be quite complicated and cumbersome to work with. And this is a perfect task for us to send off to the AI to at least get it started working on this implementation.
Watch from 52:02So what we can do is we can go into our application and we can just create a brand new session. So I say / new just like that. And this is where I want to be as detailed as possible explaining exactly what I want the AI to do. And I'm just going to write out what it is and I'll read it to you after I'm done. Now I just finished writing out everything. It's a very detailed prompt. I'll kind of give you the high level. Essentially, I said that I wanted to use Verscel workflows to ingest the blog articles from and I pasted the URL to my blog. And I also mentioned that all the articles are listed in an RSS feed and I posted the URL for that as well. I mentioned that I want the workflow to be specifically two steps where the very first step is to just get all of the articles that are not currently in the database. Then what I wanted to do is to loop through each article and run step two. And I want to batch these requests. So instead of like let's say I have 500 articles instead of sending 500 requests to my blog all at the same time it's going to batch them in sets of 10 just so I don't do myself. Then for step two this is where I wanted to run for each article I wanted to fetch the HTML. I specifically told it where the HTML for the blog content lives. So I mentioned that's in the main element. I said that it should also get the OG image and use that as a thumbnail. And then the HTML content should be chunked by H2. So essentially what I wanted to do is for every H2 heading, I want to get all the content under that heading and put that into one individual chunk. That way each chunk is stored in a hierarchical way where it's essentially better for the AI when it goes through and embeds information. It'll be much better at retrieving the proper information. When you're doing this kind of embedding and chunking for a rag system, you want to try to chunk your data in ways that it
Watch from 53:24is contextually all kind of the same piece of data. So if I'm chunking by H2, I generally will have a lot of the same data in each H2. And since my blog article has lots of H2s with not too much content in them, my chunks are never going to be massively large. So it should hopefully give me good chunk sizes as well as a concrete set of data that's all kind of related in each chunk. So inside of there, I told it what to do and I also told it to use the actual text with no HTML markup. And I also kind of gave it some information on some edge cases as well. I also specifically said do not do any of the embedding. Embedding is kind of a whole another beast where we need AI and we need to embed it and then save that in our database. I want to handle that in a separate task. So I essentially told it not to do that at all. I then gave it a list of what I want my database to look like. I essentially said all of the different tables and kind of the different keys and things for my tables. So I just listed out what my table for my chunks as well as for my content are going to look like. And it should go through and create all that information. Now with that done, we should just be able to click enter and let the AI actually run and prompt through this. And this is going to be a prompt that is going to maybe take a little bit longer because it's quite a bit more involved. And since again I'm running this locally, it's going to be slower than if I was running like a frontier model for example. So while this is running, I'm actually just going to go get lunch myself and then come back and we can take a look at the code. And this is a perfect case for where you have a longunning AI task. Go ahead and do something else in your life while that longunning task is working. I will kind of sit here for just a few seconds just to make sure it doesn't ask me any questions, but then I'm going to go ahead and make lunch and eat that and come back when it's done. So I just finished lunch. It looks like everything here is finished. So now let's go through and actually review the code.
Watch from 54:54And I actually want to do a quick review and then I want to do a more in-depth review. And that's because I want to spawn a process for creating tests for this as well since this is one of the most important parts of our application. So I just want to do a really quick review to make sure that the structure looks correct. And then I'll have it write test and then if I need to make more sweeping changes, I can make those and I'll have test to fall back on to make sure everything still works like I expect it to. So just go ahead and again make sure we have just those sweeping changes still in the correct place. So we have ingest blog articles. You can see here it's got some like type stuff that I'll probably get rid of and so on. But it does look like it's using a workflow. And if we look, we should hopefully have some steps for those. We have a step here for fetching the RSS feed. We have a step here that's going to be saving the article. And that should be the two steps that we have. And it looks like that is correct. And then we have all of our chunking code down here. So it does actually look like for the most part that this code is correct. I'll probably split some things out and structure it slightly differently, but at least we have a starting point right here for all of our different code. So I will actually prompt it to create some test for this particular ingesting process. I think I specifically care about checking on how the actual chunking process works. That's the thing I care the most about. So what I might have it do is actually swap out that chunking myself real quick. So you can see here we have this chunk by H2 function that does all of our chunking. That's kind of the thing I care about testing the most because the rest of it is just like fetching HTML and so on. So let's go ahead and make it so that that is the thing that gets tested. So I'm going to take this entire chunking section. and I'm going to bring it into its brand new own file. So, inside of our app, we're going to have this lib folder. And we'll create a folder for chunking. And we'll come in
Watch from 56:24here with a new file. We'll call this one chunk articles.ts. Paste that down. And we'll call this chunk articles. There we go. Want to make sure that I export that. And of course, we need to import the different libraries that it's using. For example, this Cheerio library that it's using, we want to make sure we get imported. So, let's just come all the way up here and we can copy that import in and we'll paste that down to the top. Of course, it looks like it didn't want to copy. Let's try that again. Copy that over. Paste that down. There we go. And it looks like we have a few other types that we need to have inside here. Element type. We'll get that fixed. So, copy that over. And there we go. And then we just have some NE's that obviously we'll clean up later, but at least we're able to get rid of that import. So now let's go ahead and just ask it to create those tests for us. Create. And actually, I'll probably do it in a new session. There we go. Create test for. And we called this chunk articles. Chunk articles. There we go. Perfect. So now let's go ahead and actually read through the code. I'm going to save this ingest file to last because this is the most complicated. I want to make sure all the auxiliary pieces are where we want them to be. So, we can look in the next config, you can see it added with workflow, which is just going to make everything with workflows work properly.
Watch from 57:55In our package JSON, we have those different libraries that we're using for parsing our HTML and so on. Our TypeScript config is including that workflow section again to give us more type safety. Our routes here, it's just created a route for us for ingesting the article. And I actually noticed that it's at this /ingest route. I actually want to change that. I want this to be API workflows ingest blog articles. There we go. That way this is specifically for our blog article being ingested. Just like that. Now let's go back into here. We have our database. And you can see that it's adding our content relations as its own kind of schema thing. I don't really like how that was structured. So I'm going to change how that is structured. Let's go to that content relations. This I don't want to be a relation part. I think I want it to be directly in our relation. And you'll also notice chunks and content are in the exact same file. I want these to be in separate files. So let's go ahead and change that as well. So inside of our schema section, we're going to create a brand new file. This one is going to be called chunks. That's where all of our chunk related content is going to go. So this entire table here that's going to go in our chunks section. And we'll fix all the different typing and stuff. So on we also want to make sure our relationships here are going to go in our relations file. So let's just copy all of this. There we go. go into our relations and we should just be able to paste that down and then everything should work properly which it looks like it is.
Watch from 59:20That's perfect. We can get rid of this entire content relations and go into our database. Get rid of that here and here and still everything else should work properly. Now we just want to make sure that all of our tables have the content we want. So this table is just going to be our content section. So let's make sure that this works like we want. I want this to be snake case. So I'm going to use that snake case table type. Just like that. Let's make sure that we import this. There we go. Don't need this import since we're not using it. And now here for our UU ID, I'm actually going to create a helper file. So in our database, I'm just going to create a utils.ts export const ID. And I'm going to create my own ID type. So we're going to say UU ID coming from Postgress. Just like that. I want that to be a default random. And that should be all we need for our ID. That way anywhere that I have an ID, I can essentially just use ID and import that directly from our utils. And now we have access to that. And I want to do the same exact thing inside of here. So let's make sure that we import ID directly from our utils. And now we have that information. Now I don't actually need to specify the name of our tables. This is all handled automatically for us. So I'm going to get rid of these just to clean up our code quite a bit because that snake case is taking care of all the naming for us.
Watch from 1:00:41Just like that. There we go. And now what I want to do is I want to clean up this created at and updated at section. Again, I'm going to be using these in many places. So, let's create variables for them. Export const created at is equal to I want this to be a timestamp. Make sure that comes from drizzle. And I specifically want this to be with time zone. We'll set that to true. There we go. Default. Now, that's perfect. I can do the exact same thing for updated at. The only difference is it's going to have an on update at the end of it. that is essentially going to force it to be the current date every single time we do an update. So now I have my timestamps essentially right here and I'm actually going to export a constant variable called timestamps and I'll set it equal to those variables. So what I can do I'll just copy this over, clean this up slightly. There we go. And there we go. Put a comma right there. And now my timestamps have all that information. So I can go into my content everywhere I have my timestamps. I can just add that in like that. And if I do a quick little import of my timestamps up here, you should see that cleans that up. And I'll do the exact same thing for my chunk section. Again, I can get rid of it right there. And I can make sure that I just import the timestamps just like that and import them there. Okay. So now let's actually move to make sure that all the content inside of here is what I want it to be. So we have our title, which is a notnull text. Description not null text. Publish date, that's a time stamp, which is a date, not null. I actually want this to be a date type.
Watch from 1:01:58So, let's change that to the date type. Just like that. And for this, I want the mode to be date. That way, it shows up as a date instead of a string. Not null. URL is going to be not null. Unique. Thumbnail is going to be text. And I want this to be not null as well. Content type, that's not null. And then finally, our content itself is not null. So, it looks like this content table is everything that I want. And we also have our video and article types. That's correct. Let's go over to our chunks table and make sure that this is what I want as well. So we're going to have our UU ID type. Let's make sure we import that from Postgress. We're going to have our integer type which is getting imported. Our vector type which is again coming from that same exact import. And finally our text type which again is coming in. And lastly we're creating an index just like that. So that has all of our content there. I want this to be snakecase.t. And then of course we can get rid of all these names that they added in there because again these are automatically generated with the proper encoding. There we go. Let's make sure we import our content table here so we can reference the ID. Make sure our ondelete is proper. This is not null. So that's perfect and it's a foreign key. Our text here is not null. Our vector right now is null because we're not doing any embedding. But eventually we'll change this to not null. So, I'll just add a to-do change to not null when implementing embedding.
Watch from 1:03:26There we go. And then we have our start position, which can be null because this is only useful for videos. Essentially telling us where to start in that particular video. So, that is all that right there. That's all we should need to do. We have our indexing and everything. So, that looks like it's proper. Now, we should check on our AI to see how it's doing with our testing. And it looks like it's finished up. And it just says to actually set this up, we just need to install vest. So let's just go ahead and install vest, which is the testing library I wanted to use. I should have specified that I wanted to use vest, but it is luckily defaulted to that. And then we can just run what was it? npm test. And we should hopefully see that everything is working. And let me make sure in the package json it actually modified it to include a test script. It did actually include that and a watch script as well. So that should hopefully go through and it will run our test to make sure everything passes. And it does look like all of our tests are passing. So at least it works like we expect. We'll read through these tests to make sure they're good in a little bit, but for now, we at least have a base point where if we change our code and things fail, we now at least know that we have some tests to fix that. So, we have all of this is complete. Package JSON just has those new configs as well as vest as a dependency. Again, we've already looked at the TypeScript config. There's all the chunk article code we'll look at in a little bit. This route seems fine. This is how you supposed to manage workflows. You just call start, pass it in your workflow, and that's going to run your workflow for you with all the information that you could possibly need. Now, inside of our relations, let's take a look at this code because it looks like there are some errors in it now. Probably just some import errors because we moved some things around. So, let's just take a look at this. Chunks does not exist. And that is most likely because inside of here, I need to export chunks just like
Watch from 1:04:46that. There we go. That cleaned up the errors inside this file. So, this file is completely done. Our schema file is done. Our utils file that we created ourselves. So, we can ignore. And otherwise, all those other files are correct for our DB schemas. We've taken a look at both of those. our migration file. Looks like it created a migration file for us. I'm actually going to delete this because we changed how our tables are set up. So, we're going to recreate that migration file once we're done with everything related to embedding. So, that's just deleted for now. Let's continue onward. We have our chunk article and our ingest blog articles. So, these are kind of the two next sections we want to look at. Let's take a look at our ingesting of the blog articles first to see how that works. Let's go ahead and remove those unused imports. And we're going to fix the chunks import as well. Make sure it imports from the correct location. There we go. So, we have some types here. Honestly, I'm hoping to be able to get rid of these types and just do everything automatically. We have our ingest blog articles workflow. That's perfect. So, step one is fetching all the RSS feed and finding the articles. That is what we want. So, that is a good first step. Let's go ahead and actually check out that fetch RSS feed to see what it looks like. It looks like it actually is returning us a promise of that particular RSS item type. It has used step, which is correct. And now we can see it's fetching that response. If there is an error, it's throwing an error. Now, that's perfectly okay if it's having a problem fetching. We want to throw an error and then hopefully retry maybe when the URL is back up and running properly. Next, you can see that it's trying to get the response text as XML and then it's trying to parse that by getting a new parser, which I believe comes from RSS parser. Yes, it does. So, that is what we wanted. And let's go back to where we were on that. So, fetch RSS feed. There we go. So, it's parsing
Watch from 1:06:17out all of our different results. It's casting them with an as type right here. That is actually not something that I like. I really don't like as casting unless you absolutely need it and in our case I don't want that and we don't need that. So you can see here we're now getting our result with all of our different information. So if you say result dot let's see if that comes up you can see here now we can click items and that's going to give us our items and if we get the first one we can get all the information for our item. Now in our particular case we have a description as part of our RSS feed. If we actually just open up this RSS feed URL we'll actually pull this over. I'll zoom this in. You can see we have a title, a description or sorry for each item we have a title, we have a link and we have a description as well as a pub date. Those are the information we want. Now this description is not like a default field that's got by this parser. So we want to just make sure we add that into our parser. So while we're generating a new parser, we can add in our own custom fields and we can say that we want to have that custom field of description. Now this custom field description is going to be on our item. So let's just go ahead and specify it like that. And then we'll say that we want our description. There we go. That's going to give us that description field. And now you can see here we have access to that description property because it's coming in as a custom field. Now we can get rid of this hard-coded return type. I generally don't like to use hard-coded return types unless I absolutely need to. And let's see what it does after it fetches RSS feeds. So now it's going to get all the existing URLs from the database and filter to only new articles. This is something I want to do inside that step.
Watch from 1:07:49So I'm going to take all of this information. I'm going to go into my fetch RSS feed step. And I'm going to paste all that in there. And we're going to change around the code a little bit to make it work. But I'm going to rename this to something like get new articles from RSS feed. There we go. And we'll come up here. Make sure we call that. That way it's just going to get us the brand new articles. And we can just say something like new articles. There we go. That's the brand new articles and only the new articles. That way if we run this in the future, it's only going to get the articles that are posted that are new and not already in our database because we don't want to chunk stuff that we already have chunked. That would be very costly and it's going to be very prohibitive. So now here we have our results just like that. And from our results, I want to get the items specifically. So, we can get our items just like that. Now, another thing we can do to actually clean this up is that the parser actually has a parse URL function. And we can just pass it in that RSS URL. And we can essentially get rid of all this code up here for manually fetching that URL. This just does it for us, which again makes it a little bit easier for us to work with. And then we want to again get those items property directly from there. Now, for getting our existing URLs, we can actually make this a little bit better and more performant as well. We can do a simple DB query right here. And I want to query on that content table. I specifically want to come into this query with a wear clause. And to make sure this works, I actually want to do a find many. There we go. That's going to find many results. And now inside my wear clause, there's a type property. I want to make sure that this is article.
Watch from 1:09:11So first, I'm only getting the articles that I specifically need. I can also specify what columns I want. And in my case, I want just the URL column. So I'll set that to true because that's what I'm using to determine whether or not this is a new article or not. Then all I want to do with this content is I essentially want to get just the existing URLs. So I can just say then that's going to give me my data and I can say data map and I want to take in each result and I want to get just the URL property. So now this is my URLs that are existing. So we'll say existing URLs clean that up and we can just check to see if our existing URLs includes our item whoops item.link link and if it doesn't include it, we want to make sure we filter to just the ones that are not included. So, this is only going to include articles that don't already exist inside of our list. Now, here is where we can just return this result because that gives us essentially just our new items. And this section right here for our new articles, we can make sure we come up here and if, for example, our new article length is zero, we can just essentially return that we processed absolutely nothing because there were no new articles and we don't need to worry about logging that data. So now we've cleaned this up drastically and this code is much easier to read and more performant which is great. So we've come in here, we've created our new articles. That's our step one. Next thing that we're doing is we're going to process in batches of 10. So we have our batch size set to 10 and we're going through a loop and processing all of this information. Now this batching implementation is really not bad, but there's a few different things I want to change about it. First of all, we should be using all settled here instead of all. The reason for that is because if
Watch from 1:10:31we use promise.all, if any of them fail, all of them will fail. So, this just makes sure that they all work properly and it's only going to fail each individual one and let the ones that succeed actually work properly. Also, we're going to be doing batching in multiple places. For example, when we ingest YouTube videos. So, I actually want to create this batching in its own file. I want to create a brand new file. We'll put it inside of a util section and we'll just call batch ze or something along those lines. And here I want to essentially write our own batching implementation. So, we'll just kind of copy over that loop just like that. We're going to export a function called batch exec. This is going to be a generic function because we're going to have a generic return type for this. And that's because it takes in essentially an array of items just like that of type T as well as a function that it's going to run for each individual item. So we're going to call this a batch with type of T. And it's going to return to us a promise just like that. And this promise doesn't have a return type. Doesn't really matter what it returns. We don't care at all for this particular use case. And now we want to do that batching. And we also want to be able to configure what the batch size and so on looks like. So we'll set a batch size by default of 10 and a delay in milliseconds of zero. So we can also delay this if we need to. And by default we'll set that to an empty object. There we go. So now we essentially can specify how we want to batch this and how long we want to wait in between our batches as well. Now also inside of this batching I want to take account of which ones have succeeded and not. So we can say total succeeded is equal to zero and we'll just count this up and we can also get our total
Watch from 1:12:02failed and we'll count that up from zero as well. So we can keep track of what is successful and what is not successful. And then here we can just return a total which is our items.length. We can return our succeeded which is going to be our total succeeded. And we can return our failed which is our total failed. Just like that. If I spell everything properly, this allows us to keep track of how many things were successful or not. So we can easily look at our workflow and see where our failures were. Now is where we actually need to do our looping. And so far, this code is actually really good. So we need to take our items. We need to loop through them for the actual batch size that we're working on. And we essentially need to create a batch from those items. All of that code is perfect and just like what we want. Next, we can get the results of those by calling outcomes, which is await promise. And we want to again use that all settled to make sure that they all execute and finish properly. And then we can just call batch.mmap and pass in our function. So this is going to call that function for every single element inside of our array and wait for it to finish properly. And this is an async function since we're using async await just like that. That'll give us all of our outcomes. And then inside of our outcomes, we can do whatever we want. So all of this code inside here, we can essentially completely get rid of it. And I want to just loop through my outcomes. So for each outcome I want to run the code inside of here and if our outcome status is equal to fulfilled well that means it was successful. So we can take our total succeeded add one to it otherwise what we can do is we can take our total failed and we can add one to that
Watch from 1:13:33instead. This is just going to properly make sure we keep track of our failures and our successes. Then since we have this delay we just need to add in a section here to handle that delay. So if our delay is greater than zero and we have more things to still batch on. So if our i plus batch size is less than our items.length then that means we still have more things to loop through and batch on. So what we can do is we can just call await we can pass it in sleep and we can pass it in our delay of milliseconds and this sleep function comes from our workflow and it'll essentially just wait until this sleep duration is done and then it's going to run our code. So this just makes sure that we have the ability to slow down our execution in case we hit like API rate limits or something along those lines. So now we can use this function in multiple places. It gives us a really good set of results that it's going to return to us. And it just overall is going to be a better experience than what they have written out here which is a one-off scenario. So we can call batch exec just like that. We can pass it in our new articles and then we have a function that is going to take in those new articles and we can just call ingest article just like that. And that's the step that's going to take that article in. Right now, this step only takes in a link, but we'll make it take in all the information from our new article. Just like that. We also can get rid of this check right here because if our length of our articles is zero, well, it's automatically going to make sure that it doesn't return anything.
Watch from 1:15:03So, you can see here if this is zero, it's just going to skip all of this and just return 0 0, which is exactly what we want. So, now we can work on this ingest article step, which right now takes in a URL, but I want to take in essentially the entire item that gets returned by our parser. So if we look at the type for this, you can see we get this like parser do item type. So let's make sure that we use that as our type here. So we can just say that this is our feed item and that is going to take in our parser item type I believe. And if we look at that you can see feed item is declared. So it has that proper type. Now, inside of here, since I have the full feed item instead of just the URL like they passed in before, I want to essentially validate that all the data I expect to be on this item is actually there. Cuz unfortunately, this code is not great when it comes to getting information. For example, it's getting the title and the description directly from the HTML as well as this publish time. It would be much better if it just got that from the RSS feed because that is the source of truth in my opinion and this is just a hack trying to get the correct information. So what we want to do is get all that information from our feed. And to do that, we can very easily just come all the way up to the top here and we want to make sure that we create a schema. So we'll say const feed item schema is equal to z.object. Make sure we use zod for that. Perfect. And then we can come in here with a title which is going to be a string.
Watch from 1:16:20There we go. We can make sure that it has a description. Whoops. Which is zring. Copy this down a couple times because we also have a link which is going to be a URL. And then finally we have our publish date which is called pub date and that we are going to coersse into a date because essentially it returns itself as a string. So this is going to make sure all that data we expect is inside of our feed item. So now for each one of these what we can do is not this section actually this section we can try to get our data and our success. So let's just say data for now is equal to parse and we want to parse this safely just like that. And then we can pass it in our feed item. There we go. That's going to give us this data right here. So let's make sure that we get our success and our data. And we can actually call this item. And instead of the success I actually care more about the error. So, if we have an error, well, then we know that we want to return some type of error. So, I'm actually going to throw a fatal error. A fatal error essentially just means stop retrying. And this means that we didn't have the right data in our RSS feed. It's never going to be correct no matter how many times we retry. So, I'm just going to give it a nice little error. Failed to process RSS feed item. And then I'll put in the URL or error message just like that. And I will also put in the URL. So, let's come in here.
Watch from 1:17:43feed item.link. There we go. That's all the information specifically for that section. And now we can go through and use this item. So here for our URL, let's just pass it in the link. It's going to fetch all that data. And if there was not able to get a response, we'll make sure that it essentially says, hey, failed to fetch that particular data. Then we're going to get the HTML. We're going to use Cheerio to try to get the information directly. Our article main, I don't think they're always nested directly. So, I'm going to get rid of that section right there because it's going to make sure it works properly, even if they're not fully nested. And here, I can just say HTML directly. And that'll give us our main HTML. Just like that. There we go. And if for some reason we don't have any main HTML, that's a that's an error. We need to address that. So, we can say if main HTML is equal to null or does not exist in some way, then we want to make sure that shows up as an error. And this is an unreoverable error because it means our format was not like we expected it to. So we'll just come in here and we'll throw that exact same error. We'll just say failed to load HTML from our item URL. Just like that. There we go. Make sure I get that proper. It is item.link. So there we go. The next thing we need to do is to actually get the image properly so we can get our thumbnail image. I'm actually going to move that up here with the main HTML section. There we go. So that's going to be getting our thumbnail URL. And again, if this is equal to null, then we also were failing to get the information properly because these are
Watch from 1:19:07the only two things we need to get from the HTML. Nothing else matters. This title, this description, this publish date, all of that already comes directly from this item right here, which was our feed from our RSS. There's no point in getting that twice. So now that we have that done, again, our publish date, we already have that done. Here's our chunking. I believe I renamed this to chunk articles. So, let's just pass that in. There we go. Get rid of some of these comments since we don't need them. And now we can pass along our information. For example, our title comes from our item.title. Our description comes from our item.cription. Publish date is our item.pubdate. And the URL is item.link. Our thumbnail URL type and content are all there. I'm actually going to move type to the very top. So, that is exactly what we want. And on conflict, I essentially don't want to do anything because there should be no conflict. So we'll just say on conflict do nothing because again there should be no conflict at all. We can then just do a quick check here if our content row ID is equal to null and this could be null just like that. There we go. If that is equal to null then we're going to throw another fatal error because essentially that means for some reason we inserted something that already existed or we had some type of error. And actually I'm not even going to y'all make it a fatal error because that means we inserted something when it definitely should not have been inserted. And we're just going to say something like duplicate detected and failed to insert. There we go.
Watch from 1:20:37And we'll put our item link just so we know which one the failure was for. And I think if I make this just double equal sign right here, that should fix that particular problem. So it says content row ID. I think it's because this is a string or null. But we should be checking the null aspect of our content row right here. Content row ID is string or null. It definitely shouldn't be string or null. So, there may be an issue with my particular SQL code. So, let me go to my utils. I think this needs to say not null. There we go. That'll clean that up because obviously it can't be null. And that does fix that error right here. And then we can say it ingested the article. Actually, we don't even need any logs. Let's just get rid of those logs. Get rid of that comment. And that did clean this up drastically. So, at least this particular section for all of our workflow stuff, I think, is done. We can kind of get rid of some of these comments here. We already are doing our batch exec. So we can get rid of all of this and just return the result of awaiting the batching process. There we go. That's going to return everything that we need from our articles. We also don't need this type anymore, which is great. So we can completely remove that type. And then this code is much cleaner than what it was before. This is why I always read code, especially something that's as important as the core business logic of your entire application. Read it and rewrite it in a way that is much cleaner and easier to work with. Cuz the previous code was errorprone, difficult to read, and not super extendable. This is a much better formatted code in my opinion. Now the next and final thing that we probably need to look at is to make sure our chunking is working properly because that is kind of the core business logic of this entire application. So let's go into our lib into our chunking section and make sure
Watch from 1:21:57all the code inside here works just like we expect it to cuz I mean there's like anyies and stuff inside here which I obviously hate and we want to try to remove that if possible. Now one thing I notice immediately is it looks like it's reparsing our article main. We're kind of putting the same code in multiple places. I don't actually like that. I'm going to try to clean that up. Let's just close out of everything except for this. Make sure we don't worry about that. Okay, perfect. Now, I want to go back into the file where our workflow was and find where we were calling that particular function, which is right here. Cheerio load. Where's our chunk? Chunk articles. We're passing in the main HTML. Well, the main HTML is just the article main content. So, we don't need to research for that article main content, which obviously doesn't exist because this HTML already contains that content. So, what I want to do here when I load my HTML is I want to load it. I'm going to pass in null and I'm going to pass in false here. So by passing in null that's just essentially skipping this parameter and false right here. This is being passed in if we just hover over this load right here is document right here. This is essentially saying don't load this with a head and a body and so on. It is just essentially a fragment of HTML. So load it just as that particular fragment. So that's what I'm essentially telling the code to do right here. And this main content we don't even need to worry about anymore at all. And we can just call this something else. We'll call it main because that is what the main content actually was being properly used for.
Watch from 1:23:28And actually load we'll still call dollar sign but instead actually we want to get our root element which they called dollar sign main. We'll keep it the same name and that is just going to be calling root directly on this just like that. That's going to actually give us essentially the full entire HTML document but in a way that can be processed by Cheerio which is essentially like a fancy jQuery version. Now if we take a look at the code right here you can see it's getting all of the H2 elements that are within that main which is perfect. It's then going through if there's no HTML element, it's just returning all of the text as a single chunk. So that looks like that is working properly. Next, you can see it's getting the text before the first H2. So it's getting our first H2. Then it's going through and trying to get the contents and filtering only the text types. This code right here is way more complicated and convoluted than it needs to be. We can actually clean this up drastically to make it look more like this code right here by just getting the text property directly. So, I think what we want to do is we want to just create a simple loop. I like some of this code, but this is really, really complicated. Much more complicated than it needs to be, especially cuz it's like going through and it's copying the same code in multiple different places. Let's clean this up drastically. So, let's go ahead at the very top here where we have our main element. What I want to do is I want to just loop through every single one of our main elements. So, we can come in here, main, I want to get the children specifically, and I want to just loop through each one. I'm not caring about H2s or anything like that.
Watch from 1:24:44I'm just going to loop through each child. And if I do find an H2 element, that's when I'm going to go through and I'm going to essentially create my chunk boundaries. So for each element, I'm going to say that I want to get the element specifically, just like that. And then for each one of these, I can just get the text. So I can say text is just Whoops. element.ext. That's going to give me the text. And let's make sure we trim it to get rid of any white space because that's not important for our chunking algorithm. Then I can say, hey, you know what? If my element happens to be an H2 element, then I want to essentially create a brand new chunking boundary. So, here is where I want to essentially take whatever my current chunk is and I want to replace it. I believe they probably do that down here somewhere. Looks like yeah, they have their chunk text that they're keeping track of. That's what I'm going to be keeping track of up here. I'm going to essentially have a variable called our current chunk. Doesn't really matter what it's called. It just defaults to an empty string. And inside of here, anytime that we have an H2, I just want to take my current chunk if it exists. So if my current chunktrim, that means that we have a current piece of text that we want to save. I just want to take all of my chunks that I have, which I think they called result. So we'll just say result just like that. And I want to push in that chunk. So we'll say current chunk.trim. There we go. So all I'm doing is I'm saving whatever my current value is in this current chunk. And every time I reach an H2, I'm taking everything that's currently in my chunk, trimming it right there, and saving that as a current chunk. Then once I've hit an H2, I know, hey, I need to reset what my current chunk is. So I'll say current chunk is equal to an empty string. And
Watch from 1:26:04now this is what's happening for all my batching. Every time I hit an H2, I take all my current text, save it, and then completely flush it and start fresh with a new piece of text. Then doesn't matter if I have an H2 or not, whatever it is, if I have some type of text that I want to save, all I want to do is I want to take my current chunk and I want to add that piece of text on. So we can just come into here. We can add my text and we'll put a new line character at the end to kind of represent that we have paragraphs. I'm actually going to use a double new line character just like that. Since this is more like representing separations of paragraphs and that right there is essentially taking care of all of our code. All we're doing is we're just looping through and saving every single piece of text. But if we hit an H2, we take all the text we've already saved, create that as a single chunk, and then start over with creating a brand new chunk. The only thing we need to do to make sure all of this works properly is if we still have text after our last H2 that's going to be saved in our current chunk, we need to make sure we save that. So if our current chunktrim, then we want to make sure we save that. So we can say result.push current chunk.ch. And then we can just return our result. So you can see this is a much simpler algorithm than all of this crap that the AI generated. So unfortunately, AI generation wasn't really that useful for a lot of the actual complicated code, but it worked good for getting the boilerplate app set set up, getting our routing set up, our TypeScript, and so on set up. That all was really useful. The rest of this, not so much. Let's get rid of these comments cuz I don't know if they're even relevant to what our new code does. And our new code is so simple, it probably doesn't even need a comment to explain
Watch from 1:27:36what it does. Now, let's also make sure our chunk articles.t test works as well. So, let's just rerun our test script. we probably like you can see here have a lot of different failures and that's because most likely you can see it's passing the full HTML body and so on. I wanted to just pass the content that's not inside of the HTML body article. So I'm just going to essentially take all these HTML body articles, get rid of them and then same thing here. Get rid of the bottom half of all of these. And now let's try rerunning that. And that should hopefully be all we need to do to make it pass. And it looks like that is still not quite working like we expect. So it looks like there's actually a test here for content without an article wrapper. That's not even test that needs to exist. So here we can see content without article wrapper. Let's completely get rid of that test. Doesn't do anything. Handles deeply nested content within H2 sections. That looks like it's fine. That shouldn't really matter. That should still work. Handles H2 with inline elements. That should also work fine. Let's just run our test again. And now we can see that everything is passing as we expect. Now I also want to create a real world test case. So we're just going to come in here. We'll create a describe block. We'll call this one real world just like that. And we'll say it works on a real blog article just like that. And I'm just going to paste in an actual real blog article that I have. So we can just say const html equals. And we'll paste in that entire article just like that. And it looks like there are potentially some formatting errors probably because it has some back ticks in it or something along those lines. To make this actually probably a little bit
Watch from 1:29:06better, I'm going to go through. I'm going to actually paste in a much smaller article. So that's this one should work better. That other article was a massive article. This one is a much shorter article. So it'll still give us the exact same test. And then we just want to make sure that it matches everything that we expect inside of here. So let's look at some of the other tests to see how they were formatted. You can see here we just get our result and we expect essentially the result to be what we expect it to be. So we can see expect it to be result zero.trim. And let's make sure that's kind of how the rest of them are working to match. Yeah, it's just matching the text exactly. It looks like it's only matching the very first thing. Uh here it's matching more of them. That's fine. So, let's just go ahead and make sure that it is matching what we expect. So, we can say chunk articles result zero to start with. Or is that actually an option? I believe there Oh, to contain is probably what we want. There we go. We can do to contain. And let's just copy some of the text. We can just copy this entire paragraph. If it contains that, then we're probably doing pretty good. So, let's just paste this down. There we go. And then we can expect this second result to contain some more text. So if we look here, here's our first H2. Let's get the text from there. We're going to paste that in. There we go. Let's copy that down again because we also expected to get some random text from the middle. So let's just randomly choose some text in the middle. And we'll copy all of that. And we'll paste that in. And again, since we want to make sure that none of the HTML is sticking around, we'll get rid of the HTML related things because those should not be in there.
Watch from 1:30:36Get rid of that. There we go. And now let's run our test. And hopefully that should work as well. And there we go. That did test, which means it's working on actual real world HTML directly from one of my blog articles. So now that we have that done, what I want to do is I want to first check out a new branch because I'm still on the old branch. So, we can say checkout-b and this is going to be ingest articles. There we go. I want to add this code because I believe we checked through. Yeah, we checked through absolutely everything that was written. So, we can say get commit. Add article ingesting. There we go. And now we need to work on the actual embedding portion of our chunking for our articles. Now, you could go ahead and you could use like OpenAI for the embedding, which is what we will use in production. But there's no point in using these costly production APIs when we're just testing locally when we can use a really lightweight local model to do our embedding instead. So, what I'm actually going to do is inside of LM Studio, which is where I run all my local models, I'm actually going to load a local embedding model that's going to do the embedding for us. You can see here there's a super lightweight 609 megabyte model that's going to do our embedding using Quen. We can load that model directly. We can just click that we want to load that model. It should load incredibly quickly since it's quite small. And now we have access to this embedding model locally as well as when we want to deploy to production. We'll be able to use that openi AI version of the model as well. Now this is something that I could prompt the AI to do, but it's so simple to do. We're just going to do it ourselves cuz it's going to be way quicker than reviewing code and stuff like that. So I'm going to create a brand new section inside of here
Watch from 1:32:03specifically for embedding. And I'm going to have a way to get our embedding model because in production we'll use a different model than we use in development. And I'm also going to be setting up a way for us to be able to embed all the different chunks we want. So let's just come in here. We're going to get our embedding model.ts just like that. So for this get embedding model, we're going to export a function called get embedding model just like that. And this is essentially going to use our local environment variables to determine which embedding provider we want to use. So let's just come in here. We're going to create a section called embedding. and we're going to have an embedding provider. This is either going to be Quen if we're locally doing it or we're going to have OpenAI if we're using the OpenAI way. So, let's just do Quen for our local embedding. And then what we want to do is we want to specify our local models name. And specifically, actually, we want the URL. So, we're going to say local embedding base URL. And that by default, if you're using LMS Studio, is localhost 1 2 3 4. So, there we go. That is going to be our local embedding base URL. Here is our embedding provider. And now inside of here, we can create a function to handle all of that. So we can say if our server env dot and we want to make sure we add these variables to our server env. So we can come in here. We have our local embedding base URL. I think that's what I called it. Yep. Local embedding base URL and our embedding provider. This one right here is a Z URL. And then this embedding
Watch from 1:33:34provider is a Z enum. And this enum is going to be either Quen or Open AI. There we go. So now we can say here if our local embedding provider is equal to Quen, then we want to do the local version otherwise we want to do the production OpenAI version. Now we're going to be using the Verscell AI SDK. So, we're going to say npmi at AAI SDK slash open AAI since we're using OpenAI for everything. That's going to install that AI SDK for us to be able to use for all of our different code. We're also going to be using the AI library and I'll just make sure I force install those so it installs properly. And now we can actually use those different sections. So, here I want to create a custom provider. So, we're going to say create open AI. That's going to create an OpenAI provider for us which takes in an API key. We don't need this. So we can just put not needed. Any value will work because we're using local AI that doesn't need an API key. And then our base URL is our server env. And that is going to be our local embedding base URL. So let's just make sure we import our create open AI as well. Looks like it's not letting me manually import that or automatically.
Watch from 1:35:06So I'll just create open AI from at AI SDK. Just like that. And I believe this just needs a capital I. There we go. Capital I there. So that is our provider. And then we can just return our provider dot and we want to do embedding. So we're going to say embedding. And then we're going to pass it in our model which we'll create a variable for. We'll call this our Quen model. And up here we can say const quen model is equal to. And if we go into LM Studio, we can just click copy next to the name of our model. Paste that down inside of a string. And that's going to be referencing the model that we want to use. Now we can do the exact same thing for our open AI model. So we can say open AI model and for us for this model we're going to use the text embedding three small model because we don't need really complicated or large models for this relatively simple text. Now in the case that we're doing OpenAI this is just as simple as saying open AI. And passing in the model we want to use and we just need to make sure we supply our API key properly inside of our ENV. So in this embedding section we just need to make sure that we have an open API key just like that. And we can specify some value. We'll just say to-do add value. Just like that. And now in our server env we can make sure our open AI key is just going to be a string.
Watch from 1:36:22There we go. And I believe this is called OpenAI API key. And I think I named it the wrong thing in our ENV as well. So let's just make sure that that is the correct name. That is what's required by default for this embedding model to work. So now we have an ability to get whatever model we want wherever we want to do it. Now, let's just create a brand new function called embed chunks. That's going to take all of our different chunks and embed them for this. So, we can say export async function embed chunks just like that. It's going to take in essentially an array of strings. Each one of our different chunks is what these string arrays are going to be. And then it just needs to embed them for us. So, if our textlength, whoops, text.length, and let me just call that texts. There we go. If that is equal to zero, well, we can just return an empty array because we have nothing to do. There we go. Otherwise, what we want to do is we want to get our model, which is just get embedding model. That's why we created that function so we can use it in other places. And then we can just get the results of those embeddings by calling await embed many. And this comes from that AI. So we can say embed many from AI just like that. And this embed menu just allows us to essentially take in a model. So we'll pass in the model that we're using. It allows us to pass in all the values we want, which is just our text, and how many parallel calls we want to do. Let's just say we max it out at 100. And this is going to essentially allow us to create embeddings for all of those chunks at one time. So we can just say return result.bed. And that's going
Watch from 1:37:42to get all those embeddings for us. Now we can use this inside of our ingest section where we have our embeddings. So here you can see our embeddings are by default set to null. we can just make sure that we get all of those embeddings automatically. So here we create our chunks. Now we can create our embeddings which is just embed chunks and we can pass it in all of our chunk text. There we go. Make sure we await this because this is an asynchronous operation. And then in our embeddings we can scroll all the way down here and when we're doing our insert we can make sure that we do it for all of our things as well. So here we just say I chunk text I just like that. And for our embeddings we'll say embeddings of I. There we go. That's going to make sure it matches up the embedding for our particular chunk text. So now let's go ahead and actually test to make sure that this entire workflow works like we expect it to. So what we can do is I'm going to create a brand new file. I'm just going to call this workflows.rest. And this is going to be using an extension that I have inside of my editor. This extension is called rest client. and essentially just allows you to make REST HTTP request directly from VS Code. So we can say that we want to do a git request to localhost 3000/ API/workflows/ingest blog articles. That is the URL that we created for our actual workflow. And if we click send request, it should hopefully send a request and kick off that workflow. So if we click send, you can see connection was rejected. That maybe means that we're not actually running our local host. Yeah, looks like
Watch from 1:39:13we're not. We'll just say npm rundev. That way at least everything is running on local host. And now we should be able to send our request and hopefully we should get a response. It looks like it's still not coming through. Uh looks like we're getting a 405. Okay, there we go. Let's just try it one more time to see if it works. You can see here 405 method not allowed. Let's just take a quick look at here in our API inside of our workflows. It looks like they're using a post request. I believe you need to use a git request. So we're going to change that to a get. And now we're going to do a quick send here. And you can see blog article ingestion workflow started. And here's our run ID. Now, to be able to view our workflows, we can just run npx workflow web. And that should hopefully pop up a web dashboard where we can view our workflow to see exactly what's going on with every single step of the process. And we can also run npm run db studio. And we should also be able to see our data being inserted directly into here if everything is working like we expect. So, let's just come over here. And you can see right here that I have this working and it looks like we immediately had a failure that is happening for trying to get our new articles from RSS feed. And if we take a look at the actual error, I'll zoom this in so it's a little bit easier to see. Of course, it doesn't actually want me to zoom. There we go. And if we take a look at the error, which is quite difficult to read at this zoom level. We'll zoom out a little more. I'll just tell you. It just says it's failing inside of our database query. That means that we forgot to push our database changes. npm run db push. That should push up all of our different database changes. And once that finishes up, looks like it's saying content table is it created or renamed.
Watch from 1:40:45This is a brand new table. So, let's create it. There we go. And it looks like type vector does not exist. Okay, this is great to actually know. This is because we need to make sure that we're using inside of our database the actual vector extension, which right now is not being used. So, we can create a migration for that. We'll create a brand new custom migration specifically for that. So what we can do is we can say npx drizzle kit. We can call custom just like that. That's going to allow us to generate a brand new custom one. We need to say generate. There we go. That's going to give us a custom migration that we can create. Now if we look inside of our migration folder we should see this completely blank custom migration. And inside this custom migration, this is where we can just say create extension if not exists vector. And now if we just give that a quick save that should be all we need to do to actually work with this particular code. Now I can say npm run db generate because we know that hopefully our actual database is the way that we want it. So we have generated and actually before I generate I already remember that we have a to-do. Let's just search for it. Here we go. The to-do is to change to not null when implementing. So we'll say not null. And also our dimensions must directly exactly match the dimensions that we're using for our database for testing purposes as well.
Watch from 1:42:15Now the Quinn model we're using uses 1024 for its dimensions. So let's make sure we specify our dimensions there. And now if we want we should hopefully be able to run our code. So let's try just running migrate first. This should hopefully work properly, but it may actually fail since we never initially did a migration. You can see here it is failing because we never did an initial migration. So let's go to our Neon database. We can get there by just running npm run db studio. And I essentially just want to completely delete our database to start over completely from scratch. That way we can run our migrations and it should work properly. So give it a quick second to load up everything. I essentially want to completely drop all these tables. Just get rid of absolutely everything. We'll do a quick cascade on that. There we go. That gets rid of everything. And then inside of our drizzle table, let's drop this migration table again. Get rid of absolutely everything inside of our database. That should be all we need to do. And now when we run our migration, it should work because there is nothing at all inside of our database. So we can see it's going through. It ran those migrations and hopefully everything worked. Now we can push up our changes as well. So we'll say npm run db push. This is going to push up our temporary changes for our chunks and content. And if we give that a quick second, looks like there is no unique constraint mentioning give us keys for reference table content. I believe I actually know why this error is happening. This is because in my utils, this is why I was getting an error with a not null. This should be primary key. That way it actually knows that this is the primary key of my table. That should hopefully fix the error that I was running into.
Watch from 1:43:35Let's try that push again. This will actually solve a lot of problems that we were having. There we go. It does look like that actually solved our problem. And now let's go ahead and actually test that workflow. Let's just close out of everything other than that. Give this a quick run. We'll send that request. It did actually send that request. And now let's test to see if it worked directly inside of our workflows. I'll just close out and reopen those to make sure that we have the latest version going. And if I open this up, you can see that it is running. And it looks like it's actually working like we expect. It's a little bit difficult to see when I scroll down. We are getting some errors here. But it did actually get our blog articles proper. And if we look at the output here, it has 147 articles, which is how many articles I have written at the time I'm recording this video. This one has an error though. Invalid JSON response is the error that it is getting when it's trying to do various different API calls. I think I actually know why this is. This is because in my environment variables, I need to make sure that I specify the base URL is slashv1 so that it knows which version of the API to use. That should actually solve all of our problems. So, it actually does look like that is now working. If I look at my steps here, you can see a few of them are succeeding after they do their retry. And a couple of other these will hopefully retry relatively soon, and they should succeed almost immediately. So, you can see that they are succeeding just like I expect. And if I were to check out my Drizzle, if we just come into here, npm run DB Studio, and we pull that up, we should hopefully see that we start to have things showing up inside of our database. So, let's just pull this over real quick. Give it a second. And now you can see in our content section, we have those 10 articles that are inside of here. And if we look at the chunks, we have 32 different chunks between those 10 different articles. Now, the reason the chunk number is so low is because this
Watch from 1:44:56is pulling my first few articles I ever wrote. And my first few articles I ever wrote didn't actually have any H2s in them at all. So, it's just one chunk per article. So that's why this number is relatively low. My newer articles are much better structured. That's just kind of how I was when I was writing way back in the day when I wrote these first few articles. But you can see it's pulling in all of our information. It has the title. It has the description. It has the publish date back in 2019. URL thumbnail URL. The type is article. Has the raw content of everything. It looks like all of this data is being pulled in just like we expected to. So this is working. And you can see here it's now ingesting the next set of articles cuz these 10 finished and it did that batching process. So all of that is working just like we expected, which is really nice. So now with that done, let's kind of minimize out a lot of this, close out of a lot of this. There we go. And I want to make sure we set this up to run as a cron job. That's really easy to do with versel. We can just come in here with a versel.json file. And inside of here, we can specify our crons. So we can say crons, which is going to be an array. And inside of here, each one is going to have a path. This is just the URL to where we want to run this. So in our case it's AI API workflows/ingest blog articles just like that. And then we can specify how often we want this cron job to run. So we can say schedule and I'm essentially going to say that I want this to run at midnight every single Tuesday. That's what this is saying right here. The reason I do that is because I release blog articles on Monday. So if I run this at midnight on Tuesday, it'll hopefully get a new article every single time it's published within about 24 hours. And this is going to make sure that I ingest all the
Watch from 1:46:26articles the first time. And then every other time this runs, it's only going to ingest the brand new articles that I just created. So now that we have all this code done, and we wrote this all ourselves, so we don't need to re-review all the code we just wrote. Let's make sure we commit this. So we'll just say add commit add. And actually before we add and commit, there's one other thing that we want to do. While we're in production, we're going to be using a slightly different API for our embedding. We're using OpenAI as opposed to the Quen model. So that means inside of our schema for our chunks, we need to specify the dimensions here for the OpenAI dimensions. Now, OpenAI uses 1536 for its dimensions and you cannot cross-contaminate. I can't have Quen and OpenAI dimensions in the same one. I must hard commit to one API or the other. And since this is going to be used in production, it won't have access to my local computer. We're going to be using the OpenAI model of 1536 dimensions. That's going to be what we use when we create our migration. So now we can actually create our migration. npm run db generate and that'll create that migration file for us that when we push to production we'll make sure all this data is properly in the correct location just like we expect it to be. So now with that out of the way let's go ahead and actually commit these changes. So we'll do our get add again get commit add embedding support and we'll push these changes up just like that. There we go. That's working great. And now what I want to do is I want to run code rabbit again. Now, I could be running Code Rabbit on this pushed up pull request, which is probably the way I should do it since I've already pushed up the pull request.
Watch from 1:47:56But if, for example, I wanted to test this before I committed my code or before I pushed it. You can also run code rabbit locally. Now, it only works in either a Linux or a Mac environment. So, since I'm on Windows, I need to use Windows Subsystem for Linux. So, I just type WSL. That'll open up this Windows subsystem for Linux. And then what I can do is I can run CR to run Code Rabbit after I've installed it and logged in. Now, I already have the local version running on my computer, so I can just type in CR. And since I want to specifically work off of the main branch, I can type in d-base of main and that's going to compare my code to the main branch. If you leave this off, it'll just take whatever changes you currently have in your application that aren't actually committed, and it'll use those. But since we've already committed our changes, I'm essentially just going to be comparing all the changes on this branch to the ones that are inside of the main branch. Now, while that's running, let's go ahead and actually work on our environment variable setup because we have all these different embedding things that we need to specify inside of our Versell environment variables. So, let's add an environment variable. The first one is our embedding provider. I'm just going to copy that over just like that. So, we have our embedding provider. This is OpenAI, and this is not sensitive, so we can leave that marked off. And then we want to add another one. Well, let's just save that one. We'll do them one by one. The next one is going to be our OpenAI API key. So, we'll copy that. And we're going to make sure that we paste that in. And then we're going to have to get a value for this. And this is going to be sensitive. So let's just go to the OpenAI platform. Make sure that you actually log in and authenticate yourself. And then what you want to do is you want to go down to this API key section. And we're going to create a brand new secret key. Now for this secret key, I want to restrict it. And I specifically only wanted to be able to
Watch from 1:49:25use the embedding models. So we'll just come in here and have request be set for embeddings and nothing else. We can then give this a name if we want. We can also associate it with a particular project. So we should probably create a brand new project to be honest. So, let's create a project for this. We will call this video blog suggesture YouTube. And for this one, we're just going to keep everything the same. Our dev keys and our production keys are going to be in the exact same place because they can be separated from one another. But if you wanted to separate them, you could. For now, I'm just going to keep them in the same place. So now, again, we want to make sure we restrict this to have a model of just embeddings and nothing else. And we will call this our dev key. So, we'll just say dev just like that. Create our secret key. And now we can copy this. And actually, I don't know why I called this dev. We're not even using this in development. This is just our production key. So, we'll rename it to production. And now we have that key. And we can just paste that directly into here. Now, if we give that a quick save, that gives us our open API key right there, just like we wanted. Now, I do automatically have one thing that I think I need to change inside of my data. If we look inside of our server enving provider, I want to be able to allow or not the the local embedding base URL. This should be nullable. So, I'm going to come in here with a just simple nullable. That way I don't have to pass that along in production as well. And I will probably need to change some code where this is being used. Yeah, this is having an error here. So I can just say if my local embedding base URL is equal to null, then we just want to return some type of error. So we can just say throw error embedding URL not provided. And let's
Watch from 1:50:56just give it the full name. There we go. Perfect. That'll actually solve that particular problem. So now let's go ahead and actually check to see what Code Rabbit said for its local review. So we can just pop this open and you can see it's giving me the results directly. For example, I have a redundant index on my content. URL. So if we go ahead and we take a look inside of our database schema for content, it's saying that this is redundant, which makes sense because this is already unique, which adds the index for us. So we can get rid of that. While this one is not redundant, at least it didn't say it was redundant, even though I don't really think that we actually need this. So I'm going to remove this as well since I don't think it's actually needed. Now, let's go ahead and we're going to check the rest of this. Let's scroll down a little bit here. You can see on conflict do nothing. It's saying don't make duplicate conflicts fatal. I actually disagree with this. I think it's perfectly fine to make this fatal because if it already has an item in there, I don't want to read my embeddings. I want to completely fail. So, I'm going to leave that as is. I think this is a bit of a false positive, but otherwise, there's nothing else that showed up, which means our code was relatively well written, which makes sense cuz we actually went through and manually rewrote a lot of the code. Now, since I changed what our actual migrations are going to look like, I'm going to go into where our migration is. I'm going to delete that migration and I'm going to essentially regenerate it. So, we'll just come into here. Let's get a brand new terminal set up. npm run db migrate or actually db generate. That's going to generate us a brand new migration. And this is because anytime that I commit changes, I want them to generally fall into one single migration file. I don't want to have 8 n 10 different migrations for one single commit to my database. They should all be wrapped up into one migration if at
Watch from 1:52:29all possible. Now, you will notice that we have two migrations and that's because one is a custom migration to enable our extension and the other one is the real migration that sets up our actual code. Those two being separate, I think makes perfect sense. So now let's go ahead and actually commit those changes. So we'll just say get add get toit clean up code. Again, use better commit names yourself. We'll get push this to our actual repository. And now we can go over to our GitHub. So, if I just pull this over, you can see here we can compare and pull request this in just articles. Just like that. We can make sure we create a pull request for this. And for this one, we've already pretty much tested everything that we need to test. I'm just going to go ahead and directly merge this because we already ran our code rabbit locally. So, we're just going to click merge on this directly. That way, it can roll out to production and we can hopefully make sure that everything is working like we expect it to. So, we're just going to make sure this branch deletes properly. Give it a refresh to make sure it fully deleted. There we go. That was deleted. And now we can go over to Verscell to make sure that it is actually running like we expect. So inside of Verscell we have our deployment. You can see it's building up our production deployment. It's hopefully going to be migrating all of our database changes to include everything that we just created. And our UI should look exactly the same, but now we're going to have cron jobs and so on that we can work with. It looks like we had some type of failure. Invalid environment variables local embedding base URL expected string received undefined. I thought I made it so that was nullable, but it looks like I must have failed. Let me go ahead and check our code wherever we have that server env.
Watch from 1:54:00And here I did make it nullable. Let's maybe make it optional because I think it's returning undefined. So optional accepts undefined while nullable only accepts null. So let's just come in here. Get add get commit fix undefined variable. And we'll just push that up. It'll probably make us create a brand new pull request because we originally deleted that. So, this will just be a really quick fix. I should have caught this before, but this sometimes just happens. There we go. We'll merge that in real quick before any of that code rabbit stuff runs because there's no point in running that on a oneline fix. There we go. That will delete that. And now, if we go back to our deployments, hopefully this one will actually work properly. We can just cancel this one because we don't need that deployment preview. All we care about here is going to be the actual production deployment. And I'll come back to you once this finishes. It looks like that just finished. Everything deployed successfully. If I come in here, I should still be able to sign in with GitHub, which is just like what I expected to do. Let me just refresh. Make sure I got that button working. And of course, the reason it's not working is I'm on the wrong URL. I'm on the deployment URL. I need to be on the main URL to make that work. There we go. Now I click on that. It looks like it signed me in just like I expect. So that is working. The next step is to make sure that our actual workflows are working like we expect as well. Now the nice thing is if we go into our settings here, we can actually find our cron jobs. So you can see here is our cron job for ingesting blog articles. And I can manually run that if I want. So we can just go ahead and we can click run on that. And that should hopefully run this cron job. And now if we go back and we scroll up to or scroll down to the workflow section, we should see inside of here that we have a workflow. Right
Watch from 1:55:26now it's saying that I need to install the workflow stuff. It's probably just taking a second for that to pop up. It actually just did right before I refreshed. And now if we click on this, you can see I get that exact same UI for everything with my workflow. And you can see it's ingesting my articles and it's going very very quick for getting all these different articles ingested into my system. So now I actually have these directly inside of my database. We can even view this by going over to Neon. So if we just pop into Neon over here, we're in our production. We can go to our tables. And now we can click on for example our content. And you can see here it's already ingested 130 of my 147 articles. So it's pretty much almost entirely done. If we actually just go back over here, it should be done very very quickly. Let's refresh our Neon console to see. Here we go. But we got 100 147. Looks like they are all ingested. And you can see that this has completed. And if we scroll to the top, we can actually click on this to get our output. And our output, if we decrypt that, it specifically says that it has a total of 147 succeeded 147. So we fully ingested and embedded every single piece of article information that I've ever written on my blog. And now we can hopefully implement the UI for querying this information to show you the true power of how amazing creating your own rag system can be. So let's just get started by creating a brand new branch. We'll check out main. I want to pull all the changes from main. And then we're going to check out a new branch. And this new branch is going to be for our UI. So we're going to say get checkout-b UI. Again, terrible name. Name it whatever you want. And now what we can do is we can actually go into our sandbox. Let's just actually close out of our current sandbox. There we go. And I'm going to resume our sandbox. That's
Watch from 1:56:58just a command I have for opening up my sandbox where I was before. So it's just going to be opening up our sandbox. I want to be in this very second sandbox actually. So, let's make sure we go to that one because that's the correct one. And now I'm just going to write out a prompt telling the AI exactly what I want. I'll come back to you once I'm done writing that prompt so you don't have to watch me write it. So, I just wrote it all out and essentially I just told it to create a simple single page URL that has a search bar and a header. And the header should have a login logout button as well as the user's information. The search bar should be disabled if the user is not signed in. I also mentioned that when the user searches, it should call out a server function and it should use Drizzle to match that information using a vector database search. So, first it embeds their query, then it does a vector database search and only returns results with at least a 50% match. I then specifically say that I don't want to return individual chunks. I instead want to get the content associated with those chunks, and I want to get at most 20 pieces of content. I then listed out the information that I want to get, as well as kind of what the URL or the UI should look like. I give it a small description as well as a screenshot of what I want the UI to look like. So, we're just going to run that and hopefully you'll be able to create a relatively simple UI for us. And the nice thing is since this is the least important part of our application, we can kind of get away with not really looking at the code as closely for the generated UI as long as the back end of the search works properly. The actual rendering out of the UI is much less important in this application. So, we can get away with spending much less time reviewing that portion of the code. Now, while this is running, there's one thing I actually want to take care of, and that is securing down our cron jobs. Right now, anybody could access that URL and run our cron job, which is not ideal because it cost us money to run those cron jobs.
Watch from 1:58:29So, what we want to do is we want to lock that down by essentially using a cron secret. We'll just copy over this code cuz it's relatively straightforward what's happening. I'll show you what it's doing once we get into the code, but what we can do is inside of wherever we're calling these. So, we got API, we have our workflows, and here's our routes for our cron job section. Essentially what I want to do is I want to make it so that I'm getting my authorization header. I want to get a cron secret which comes from my environment variables and I'm just checking to make sure those cron secrets match. So essentially I'm going to be setting an environment variable called cron secret. Anytime my cron job is run by versel, it passes that secret variable up in the header. And as long as that cron secret matches what's saved in my environment variable, I know that this is a valid request from Verscell. If they don't match, then it means maybe someone else is trying to maliciously execute code on my behalf. So I can make sure I block that with an unauthorized request. So I can just create a quick little helper function for doing this. So let's just go down into my lib workflows section in my utils section for that. Let's just create a brand new file. I'm going to call it verify and run cron.ts. Just like that. And I'm essentially going to be taking all this code, export a function called verify and run cron. It's going to take in my request as well as the cron that I want to run. So we'll say run cron is just a function that is a promise and I don't really care what that promise returns. So we can just specify it as unknown.
Watch from 2:00:00There we go. Make sure that I import next request just like that. And now I can get my off header which comes from here. I can get my cron secret which we're going to use server env for. And let's go ahead and go in server env. And we'll set our cron secret to a zstring. There we go. And then our env variable. We should also probably set that. So we'll just set a category called cron. There we go. And we can specify this cron secret to be whatever we want it to be. Now in our particular case, we can just copy over our better off secret because this is just for local testing. But when we go ahead and we actually use this in a real environment, we'll want to make sure that this is something more secure. But for now, that gives us our cron secret just like that. And what we can do is we can make sure our check is correct right here. And if it's not, we return that status of 404 or 401, sorry. Otherwise, I want to run my cron. So I can just say run equals await start getting that from the workflow API. And then I can call run cron just like that. Make sure this is an async function. And now I can return my result. So I can just say inside of here message cron started and I can get my run ID. There we go. Just like that. That's all I need to do to be able to verify and run my cron.
Watch from 2:01:17And now inside of my workflows, get rid of all that code that I pasted in. I can just replace start with verify and run cron. Just like that. I can pass it in my request. There we go. Which comes from here is a next request. Just like that. And I don't even need to return anything here because this is my return. There we go. That cleans up my code pretty drastically. And that just takes care of making sure this only will run if I have that cron secret set. Now let's check our AI. It looks like our AI is finished. But before that's done, let's actually go ahead to versel and make sure we set this information in our env variable. So here we go. Let's come over to here, add an environment variable called cron secret. Just like that. And for this, what we're going to do is we're going to go to the better off instructions and just generate another secret here. This is the easiest way for us just to create a really simple secret key. Just like that. Now we can save that and we have that set. So that way our cron jobs are actually going to be secure when we actually run and use them. Now let's go ahead and actually since our AI finished running everything. Looks like here all these files were created. Doesn't look like there's anything we need to do on our point. For this one, what we can do is really the most important part is this action. I want to make sure that this action is correct. And then the AI or the API. I want to make sure that the UI looks like it should. So let's look at the actual changed files. So you can see here verify and run crown. That's something we created. Normally I'd put this in its own pull request, but for our purposes, we'll just keep it in the same one. Same thing with our server environment. We changed that. Otherwise, we have some components here that they changed. And then we have a route here particularly and this is something that we changed. So we can ignore that. Our page is something that they changed.
Watch from 2:02:34Layout again is something they changed. And then their search action itself. So let's first for this one actually just test what it actually looks like when we use this. So let's just come into here. We're going to go to our application. I'm going to minimize it down a little bit. npm rundev. Just like that. Start up our application on this localhost 3000. And I'm going to make it so it's in its own window. There we go. So now we have our little search blog content. I don't really like some of the stuff that I added, but you can see that I have my sign in and I need to click the sign in button. It's going through and it's signing me in, which is great. Now that I'm signed in, I can actually do a quick search. So let's just say how to get started with React Hooks. There we go. And it looks like it is doing a failure when it's trying to query that data. And I think the reason why it's failing to query our data, I'm not 100% sure actually. Let's just go ahead and we'll check the code that it generated to see what's going on. You can see here error, no operator matches the given name and argument type. You might need to add explicit type cast. So there's something with how it's doing its checking. So we have our cosign distance. It's doing some weird stuff with like divide by two and so on. Now the really nice thing is that Drizzle actually has documentation on how to set up all of this automatically. So the first thing that I notice is that we have an index that we're missing. We need to set up this index to make it so that our proper searching is going to be much quicker. So let's go into our chunk table and make sure we add that. So we can find here our DB schema chunks. We want to add in that migration or sorry that index. So we're going to have table just like that. And this should be an array.
Watch from 2:04:04And we'll pass in that index just like that. This is our embedding index. And we're going to use snake case because that's what we use for everything. And just like that, it is on our embedding. So that looks exactly like we want it to. Now if we look at the documentation again and we go a little bit further, this is doing that embedding stuff for us automatically. So we can kind of ignore that. Here is how it's finding out the different similarities. So you can see it's using one minus the cosign distance between our embeddings. So that's where there's a slight difference between what's happening in our code versus their code. So let's go ahead and actually fix that. Let's go to this search. You can see where we're doing our embedding. Let's make it do one minus the coine difference. We don't need to worry about this divide by or this map width or anything like that. None of that is something we have to worry about. This is going to give us exactly what we need. Now, if you notice, it's using this embed many function. It should just be using the normal embed function because I'm only embedding one thing. So, we can get our embedding from our value just like that. And here we can get rid of that. Just like that. So, we can get our embedding. And I don't think we need to worry about mapping this to a number or anything like that. It should already have that type properly done. So we just get rid of that. We can call this our query vector. So at least the name matches everything. There we go. That cleans up all that. Gives us our start position, our raw text, our content ID, and it looks like it's doing a lot of code to get this information, of course. So let me go ahead and I'm probably going to rewrite the majority of this entire section because I really am not a fan of how they did this. So what we're going to do is we're going to just delete all this. Actually, I guess we'll keep that cosign stuff because that's what we're
Watch from 2:05:35going to be doing. We're going to be doing a select distinct on and we want to select distinctly on the content ID just like that because we want to get unique content information. So that's what we're going to be putting. And then we want to get all of our information. So our ID is going to be our content ID. We also need to be getting the title which is the content.title. We need the description. Whoops. Which is content.escription. I'm just going to copy this down a bunch of times because we're going to have a lot of the same stuff. We're going to have our thumbnail URL. We're going to have our normal URL. We're going to have the type. Whoops. Type. There we go. Similarity. That is going to be something we have to actually get. So, we'll just call this similarity. Right there. We're going to get our start position and our raw text, which we'll call actually we'll keep it raw text. That's perfectly fine. And that comes from our text. Now the similarity is just this cosine one. So I'm going to actually take this and put it in its own variable. We'll say db similarity just like that. That way I can reuse this because I'll need it in multiple different places. And now what we can do is we can search from that chunks table where that cosine distance I can just actually use that db similarity. So let's come in here with db similarity where that is greater than 0.5.
Watch from 2:07:05That means it's a greater than 50% match. What I want to do is I want to do a join. Just like that. I want to join on that content table where my content do ID is equal to my chunks.content ID. There we go. Now when it comes to the order by unfortunately I first need to order by this content ID. That's just because that's how select distinct on works. And then I can order by my DB similarity after that. There we go. So that's going to first order by content ID and then it's going to give me the most common DB similarity as my very first option. And then I want to limit this to exactly 20. So the ones that reference my chunks table are going to get whatever the most common or most similar chunk is and it's going to give me the top 20 pieces of content. But my content is still not going to be in the correct order because we're ordering by content ID first. So it's always going to show me the first ID, second ID, and so on. Instead, I need to reorder these particular values. So we have our match chunks, which is what this is. I can get our sorted chunks. We'll call it sorted results. And that is just going to be our match chunks. And I want to sort this pretty simply where the B dots similarity minus my A.Similarity.
Watch from 2:08:32This is just going to sort them by similarity. Just like that. Now then after that I can just return that data sorted results and we can get rid of a lot of this extra code that the AI write. For some reason I don't really like that code or what it was doing. So now we can return our sorted results. And of course it always loves to create types for things that I don't really want types for. So we'll get rid of all those types and get rid of these extra imports that are no longer needed. And our code is again much cleaner and much simpler to use. And this should actually return to me those top 20 results. So let's actually test to see if this works. So we are still getting the exact same error. And when I look, it's because they imported cosign distance from the wrong location. Let's make sure it actually imports from the correct location. Hopefully that'll fix our particular error. Okay, I think I actually know what the error is this time. These parentheses are just in the wrong location. Also, we should probably trim this query. So let's just throw a trim on there just so it is trimmed. And we should also probably convert it to lowercase again to make sure it's normalized as much as possible. So now let's go ahead and test that. We're going to just go back on our page. How to get started with React Hooks. There we go. And it looks like it actually did the search this time, but it says cannot read properties of undefined reading to ISO string. That's probably because I changed what this order of actual return results is like. So, let's just go into our components. I'm guessing we'll have some errors somewhere inside of our components. Maybe in the app page actually is where our errors are going to be. Looks like there's no errors in here that are showing up. So, let's check our search results.
Watch from 2:09:56Make sure there's no errors in here. Looks like import search results has no exported member search result. Oh, that's the type. Yeah, of course, we don't actually have that type anymore. So, we essentially just need to completely clean this up pretty drastically for the stuff that's being returned and imported into this function. So, we pretty much know for our search result, this return for our results here is going to be the exact return value from our action. So, it's whatever the return of this function is. So, let's just actually use that. We're going to say that results is the return type of type of search content. And of course, we need to await this. Just like that. There we go. That should clean up what the return type is. You can see that has everything inside of it. And we're probably running into errors down here because it looks like I never actually got the publish date, which is fine because I don't care about the publish date. We can remove that entirely from here. So that cleaned up that and otherwise it looks like all these pages are fine. One thing that I'm going to prompt the AI to do while I do some more testing is I don't want this to search on like key press. I don't want some debouncing search or anything like that. So let's just come into here. Get this full screen. There we go. Change the search so it only searches when pressing the search button or pressing the enter key to submit the form. No debouncing
Watch from 2:11:23or automatic search. And then we'll just link the page. There we go. So that'll just work on that while we do our testing to see how the rest of this looks. So, let's just bring this over. How to get started with React Hooks. There we go. Do a quick search on that. And you can see it's returning all the top matches. And you can see these are essentially all of my blog articles specifically related to searching or related to hooks inside of React. And they are sorted perfectly by which one is actually related to these. So, you can see that this does look like essentially what we want it to look like. And if I click on one of these, it does link me to that blog article. So, that does work just like I expect it to. Okay. Okay, the AI just finished with that. But what I actually want to focus on before we actually test everything is I want to make it so that our URLs, so if we scroll down to wherever our results are inside this function here, wherever we have our URLs being rendered out, let me just give it a quick look here. Here's our anchor tag. Here's our result URL. I actually want to create a function. And this function is going to be get URL at chunk location. And then we can pass it in whatever our result is. And our result is just this type up here. So let's get this type copied over. Come down here. And I want to get specifically an individual value from there. So that's going to be our individual result.
Watch from 2:12:56And we can just do a switch on the type. So we can say type equals result.ype. There we go. And we can do a switch on that type. In the case that we have article because that's the one that we're focusing on right now. Then what I want to do is I want to get my matched text and I want to split it on a new line. So we can just say split text equals result dot and what do we call it? Raw text. There we go. I want to split that on the new line character. Just like that because we automatically added new lines every single time that we did a different piece of our chunk. So I'm going to split it on new line to give me my first chunk as well as my last chunk. Those are the things that I want to get. So I can say here if my split text.length length is equal to one. Well, then I only have one piece of text. So, I can return my result URL. That's just going to be my URL as is. And at the end, I want to add this what is essentially a pound symbol followed by a colon followed by a till day followed by another colon. This is going to allow me to do automatic text highlighting. And then I can just say text equals. And what I want to do is I want to encode a URI component. And I want to pass it in that match text, which I call split text. and I want to get the very first value from it. So essentially all I'm doing is I'm adding that very first piece of text if there's only one piece of text into this fancy little selector right here. And this fancy little selector is something that you can use in the browser to highlight specific text on a page. Now I can do an else section here for what do I want to do if there's multiple elements. And if
Watch from 2:14:15there's multiple elements I want to get the first one followed up by a comma and then I want to essentially encode the exact same thing but I want to get the last element instead. So, we're going to be encoding the last element by just saying atgative1. That's going to give me the last element. And we'll default it to an empty string if for some reason this is null, which it should be impossible. So, that's going to be our article case. And then we want to add in a case here case. This one is going to be for video. And for now, we can just return the result URL. We'll handle what we want to do with video later on. There we go. And let's also add in default case. And this default case is just going to be catching any errors. So we'll say throw new error unsupported type. And I'll pass it in the type just like that. And I want to make sure that this satisfies never. That way if I ever add or remove types, it'll give me a warning inside of here in Typescript. So now wherever I have that result URL, I can just pass it in this. And now we should hopefully be highlighting the text exactly where we are. So, let's just come along and we'll just say I want to learn how to use React Hooks. There we go. Let's do a quick little search. And you can see here are all my different results. And let's try this one right here. And you can see it's highlighting the text where it thinks is the most relevant. Let's try another one. Let's try this custom hooks one.
Watch from 2:15:46You can see it thinks this is the most relevant piece of text. It may not always be perfect, but for articles that are maybe more segmented on what they actually teach, it could be really useful for bringing you to the exact location you need. Now, the last thing I'm going to do to clean this up is we're going to go to our page because there's a bunch of information at the top that I really don't care about. All that like title text stuff of like search blog content, search through our index articles. Yeah, we're just going to completely remove that section. There we go. Much cleaner right there with that. So, that just cleaned up that code drastically. Makes it easier to work with. Also, in our header, the text that we have for right here, this webdev simplified, I really don't like how that text is set up. So, let's go ahead and fix that. First of all, this anchor tag, don't need that at all. Then what we can do is we go ahead and fix that for the user. So let's see here. It looks like it's doing justify between. We want to justify at the end to push that content to the end. And then wherever that text is for the user's name, let's just change this to something that's a little bit brighter. Maybe do like 300 or let's do 400. Actually, if I bring that over, there we go. That's a much easier piece of text to read. And our sign out button still looks like it works. It still has our search results in here. That's fine enough for our particular use case. It doesn't really matter. And you can see when we sign in, everything is working kind of like we expect it to. So you notice for this, I didn't really look at the code nearly as in-depth. It's probably got a lot of problems, but for the most part, the front end of this application doesn't really matter that much as long as it works. So what I'm going to do for this particular one is we're just going to commit this as is.
Watch from 2:17:09So we can say get add get commit add UI. And then we'll just do a get push to push that information. Just like that. And now let's create a brand new pull request for that. So we'll bring this over. create that brand new pull request. That way, Code Rabbit can run on this and we can see kind of what they say because like I said, this is much less important. So, if I miss a few things, this UI is not going to be the end of the world as long as that backend portion is what is working like we expect. Now, while that Code Rabbit review is running, I just want to come in here. We're just going to check out a brand new branch. This branch is going to be for the YouTube ingest. Just like that. Make sure we put dashb in front of there. Now, we have that YouTube ingest. And I essentially want to go over to my AI and I want to tell it to do that YouTube ingesting using the same exact format that we used for the ingesting of the articles. So I just finished writing up what I want the API to do for the AI. I essentially told it to ingest YouTube videos using the same format that articles are ingested. I told it to use the Google API's mpm package and I actually specifically told it what environment variables to be used because we need to make OOTH requests. So I gave it the hint that we're going to be using specific variables for those OOTH requests. I also made sure to tell it to save the same information and to include the start time of the chunk inside the chunks table unlike articles which don't have a start time. I also mentioned to use the subtitles npm package to parse out the captions and that we need to fetch the video content details and transcript and everything. I also specifically mentioned that I wanted to process them in batches of 10 with a 9-hour wait between each batch because
Watch from 2:18:28YouTube has a very aggressive quota policy unless you have a verified app and this is not a verified app obviously. I also told it how I want to chunk the information and how each segment is going to work and to reference this article ingestion anytime when creating the implementation. Now, one thing is I'm going to change this word transcript to captions because that is what it uses in the YouTube API. So, hopefully it won't get too confused. And again, change transcript here to caption. There we go. Now, let's go ahead and let that run. And while that's running, we might as well go ahead and set up our Google credentials because that's a relatively lengthy and kind of manual process that we'll need to go through. So, what we need to do is we need to go to the Google Cloud Console API. Once you're there, click on this button up here to create a new project. Click on new project. And now we can call this whatever we want. We're going to call this video blog suggest YouTube. And we can create a dev version as well as a production version. In our particular case, since it's so painful to create this for dev and production, I'm just going to be creating one for this video, but I would normally create one of each, one for dev and one production to keep my quotas entirely separate. Then we can go ahead and we can create that. And once that's done, we need to go to APIs and services to enable the YouTube API. So here, if we scroll all the way down, you'll see YouTube data API v3. Click on that. And you can see right now it is enabled by default, which is great. So now we need to go ahead and create credentials for this application. So what we can do is we can click create credentials. We want an OOTH client ID because we need OOTH authentication to get the captions. And from here, we can just specify a web application. We can give this whatever name that we want to give it. We can
Watch from 2:19:59also make sure that we add in different URIs for where this is going to be allowing requests from. All we care about is the authorized redirect URL. So we can say HTTP and we want to get localhost 3000. That's the only thing that we really care about for this. And we're going to be creating these credentials directly inside of our development environment. So this is all that matters for our authorized redirect URI. So now let's go ahead and click create on that for a second. And you can see it's going to create that. And we have all of our information inside of here. and we have our secret and our client ID. So, let's copy over our client ID, go into our ENV, and we'll say Google API Google client ID is equal to that Google secret. Let's copy that over. So, we come over here, client secret. Make sure we copy that over. There we go. And then we're going to have our Google refresh token. That essentially allows us to authenticate as ourselves. So now that once we have that done, that is everything we need inside of here. We don't even need to worry about downloading this JSON or anything like that. Now, you also may get asked to create an OOTH consent screen. So just make sure you go to your OOTH consent screen and set all that up. It'll walk you through the setup of it.
Watch from 2:21:29Essentially, it's very basic. You don't need to worry about much because you're only going to be using this to authenticate yourselves. In our case, I'm using it to authenticate myself and my own particular YouTube channel. Now, once we have that done, we actually need to make sure that we go through and we actually add and create our credentials. So, here you'll notice the scopes that are available for OOTH. We specifically need the force SSL scope right here. This is the scope that is going to force us to be able to view captions specifically. That's the only reason we need this ooth stuff is for viewing captions. Then, we can make sure we select HTTP REST and just kind of go through these steps one by one. So the very first thing we need to do is we need to make a request right here to set up our authentication. So let's just copy over this URL. We'll paste this in right now and we'll see what else we need. We need the client ID as part of this URL. So we'll say client ID and we'll copy over our client ID directly from here. So let me just copy that and we'll paste that into our URL. Next thing we're going to need is our redirect URI. So we can come in add that in. And our redirect URI specifically for this one is going to be localhost 3000. Let's make sure I get the HTTP part in front of it. There we go. Next thing that we need is our response type. So, this response type is going to be code. That's specifically what it tells us to use inside of the documentation. So, we'll say response type equals code.
Watch from 2:22:46Let's see what else we need to specify what scopes we want. So, we can say and scope equals. And we know we want that force SSL. So, let's just copy over that scope. Whoops. Paste that down. Next thing, we can keep going. We want our access type to be offline because we want to be able to access this no matter what. So we'll say access type is equal to offline. There we go. And I think that's probably everything. State we don't need. We don't need the these. Yeah, we don't need any of that additional stuff. So let's just make sure we copy this URL in case we messed it up. Click enter and hopefully it should ask us to authenticate. Now once you click that you want to authenticate, you'll see this warning. Just click continue. That just means that this is an unverified app. And then click continue. And now we have access to essentially that entire YouTube account. And it's going to redirect us to a page, localhost 3000, but specifically, you'll notice this code section. Copy this code directly out of the URL because that's what we're going to need for the next step. I'm just going to paste that down here so we have access to it. So now we need to redirect them essentially. So you can see here we want to sample redirect to Google's OS servers. That is essentially that thing that we just did. Step three, Google prompts the user. That's what we just filled out. Now we're on step four. We're going to get that response back with a code. And now we need to use that particular code. So here is where we make a new request. I'm going to copy this. This is how we actually get our token. So we're going to paste that in. And now if we go over to our token section, we need to pass along our client ID, our client secret as well as the code. So we can say here client ID is equal to. And let's copy that over
Watch from 2:24:02just like that. Copy that there. Client secret. Make sure we copy that as well. There we go. And we need our code, I believe, is all it is called. Let me just make sure. Yep, code. And our code is just this. So, we'll copy that in just like that. Now, hopefully if we make this as an actual response. I believe it needs to be a post request. Let me just see. Uh, yeah, it needs to be a post request. So, let's just do that inside of our editor. So, we can come into here. We can go to that workflows. request file that we created and we can just create a temporary post request to that particular location. Give that a save, send that request and we're getting unsupported grant type. So it looks like we're missing something. So let's see here. Grant type field must be set to authorization code. So let's just add that in there. And grant type is equal to that authorization code. Now if we send it missing parameter redirect URI. So let's just make sure that is in there. redirect URI. That is our local host. There we go. Now, if we send our request, make sure I save that. Still says an invalid grant. Most likely, this just means our code expired. So, I just created a brand new code going through that exact same process. And now, if we send, you can see we get access to our access token and our refresh token. And we specifically want to copy over this refresh token and use that in our application. So, let's just paste that down right here. So, now that we have
Watch from 2:25:33that done and it looks like our AI is still working, let's go ahead and go to our Code Rabbit review for adding the UI and see how that looks. So, you can see we get the nice walkthrough and the review change set. We've already reviewed everything. Let's look at our premerge checks. You can see title's too generic. Yeah, we don't really have to worry about that. Let's actually see what it says we should be able to fix. So, we look inside here. We have some nitpick comments as well as some additional comments on all the things that need to be changed. So, let's just kind of go through these one by one. So, the very first one here, this one is a major change. It says um add serverside session enforcement to search content. The session check in app page only hides the UI. Unauthenticated callers are still invoke the server action trigger embedding generation plus content queries reject unauthenticated request. That makes a lot of sense. I don't know how I didn't catch that. But if we were to look inside of our action for search, you'll notice it doesn't actually check to make sure our user is authenticated anywhere inside here. So that is definitely something that we should be fixing. And we will just probably use this prompt right here to make that an actual fix. Here we have another thing. No timeout on embedding call. Uh calls an extra provider with no timeout. That's not really that big of a deal in our particular case. So we can just ignore that one. So let me just minimize that. I'm going to ignore that. This one right here, limit 20 is applied before relevance ranking. So top matches can be dropped. This is actually a good catch.
Watch from 2:27:04Essentially, if I have more than 20 results being returned since they're first ordered by content ID, it's going to limit it those 20 based on content ID. It'll still only be the results with greater than 50%, but it could miss some of the better results. So, this limit should be removed and instead the limit should be added to this sorted results. That's something super easy for us to be able to fix. I will do that probably ourselves. And then finally, possible stale result overwrite on concurrent searches. Has no request sequencing. Yeah, that's fine. I don't really care too much about that. If it returns some stale data, that's not a big deal and pretty minor result. We're not going to worry about fixing that one. So, we'll minimize that down. Looks like we had one more here. use a constant time secret check for the off header. That is not a huge deal for our particular use case. We'll minimize that one as well. And we can go ahead and we can check the nitpicks if we want to to kind of see some of the smaller stuff. Enforce a minimum length on cron secret. Don't care about that. Duplicate GitHub sign-in button. That's fine enough. It's just the UI. And wrap the block. Yeah, that's also completely fine. I don't really care about that. Split text is declared with cons directly under blah without braces. Yeah, okay. I don't really care about that. I generally kind of care about that, but for this particular use case, doesn't really matter. So, let's check on our AI. See, it's still working on some code. So, let's go ahead and see if we can just implement some of this stuff ourself. For example, this limit 20, that should be an easy thing to fix. And this AI should never mess with that file.
Watch from 2:28:20Normally, again, I would create a new GitHub branch to work on this. But for this particular video, I'm just going to go ahead and work on it in one particular place. So, let's just close out of a lot of these. I can go into our workflow rest, and I can remove this entire post request. There we go. since we're not using that anymore. And then what I want to do is go into that search action and I want to make sure that this limit is no longer being applied here, but instead I can just slice 0 to 20. Just like that. That's going to give me my top 20 results. That's essentially solving that particular problem. Next thing I want to make sure that the user actually exists. So I can say const user equals get user or get current user. Actually it's offi get session. There we go. That's what I was looking for. Let me just make sure I import off. There we go. And I want to get the session. And I can pass in my headers and that comes from await. And we're getting headers from next.js. There we go. That's going to give me my user. And then I can check to see if the user exists. So I can say if user then I can just return some type of error. Doesn't really matter what it is. We'll just return an error or we'll just return empty results. There we go. They should never even get to this point because the search doesn't actually exist. So we'll just return an empty result if for some reason the user doesn't exist. Now, while we're still waiting for the AI to finish, I'm actually going to take all these environment variables. I'm going to bring them directly over into our production environment. So we click add environment variable. We can just paste
Watch from 2:29:35this down. it'll add all of them for us. Again, these are all secret keys. Normally, I wouldn't share these between development and production, but because it's such a pain to create the YouTube credentials, I'm just going to share them particularly just for this video's use case to save you some time. So, now we can click save and we have all of those different environment variables added and we have all the environment variables we need so far inside of our production environment, which is great. We can check on our AI and it looks like it's still finishing up what it should be working on. So, I'll come back to you once this is done. So, it looks like it finished up and we can kind of take a look at everything that it's created for us. So it looks like here it created the workflow file for us. That's great. Added the path here for the for cell. We got all of our different APIs installed or libraries. Sorry. We have our search. That's the stuff that we actually added oursel. Again, I normally would have done it in a different uh request, but that's perfectly okay. We have that route, which is correct. Server env uh that looks like it's correct. Not sure why it's got the YouTube playlist ID in there. Let's see what it's putting for that. Um, yeah, it doesn't have anything in here as a default value. That's perfectly okay for now. Let's actually look at the file though because this is where 99% of the changes are. So, we got our chunk size of five or 15, overlap size of three. Okay, here's our start. So, get new videos from playlist.
Watch from 2:31:07Get playlist items from the YouTube playlist ID. So, if we take a look at that, we can see here it's Google.youtube. That's perfectly okay. So, this isn't 100% correct because I don't want a hard-coded playlist ID. I wanted to just get all the latest uploads. To do that, we actually need to query the YouTube API for that. So, we can just come in here. We can say YouTube dot and we want to get the channels. So, we can say channels.list. And I specifically want to get a part here which is the content details. And I just want to get mine. So we'll say mine true. This is going to be our channel details. Just like that. And then we can take our channel details. I want to get the data specifically. I want to get the items from that. I want to get the very first item because that's we're only getting one item or one channel. So we can just get the very first item from there. Then I want to get the content details from that. I want to get the related playlist and I want to get the uploads. Just like that. There we go. And it looks like we have a potential error. Items is possibly undefined. There we go. Fixes that. Now we have our uploads playlist ID, which is what I'll use here. And we'll get rid of this playlist ID. That's not something we need.
Watch from 2:32:27Not something we need here. Not something we need here. And in our server env. We don't need this playlist ID. Okay, that cleans up all that. Now we can go through. We can see that they are getting a max results of 50. We got our page page token. So this should be doing our pageionation in a dowh loop. That's perfectly fine. Get all the items. And for each item, we're just getting the video ID and pushing that into here. Now I believe we have the ability with this playlist items to get more than just the ID. We can get things like the title, description, and so on. I wonder I'm going to kind of keep going to see where they get the title, description, idea. So this is where they're doing that title description section. I think honestly we should be doing that when we do the playlist items that shouldn't contain the same amount of content. So here in this snippet section we can get the content details as well as the status cuz I only want to get public v videos. I don't want like private or unpublished videos to show up in this list. So that's going to get all that information. And then we don't need to just push the video ID. We might as well just push the entire video into this particular section. So here I can just essentially do that. That gives us our items and I can say all items do let's say push items just like that. And we can get rid of all of that code right there. So that's going to push all of our items into this all items array. And this all items array is essentially going to have a playlist item. So I think if we use YouTube dot and then we have or underscore v3. Let's import that.
Watch from 2:33:48Whoops. There we go. It actually autocorrected for me. And let's just make sure we import this just like that. So that's going to give us the correct type for this information. And then we'll return from here. Again, I don't like hard-coded return types, so we'll remove that. There we go. That should hopefully give us all that information properly. Now, I again only want to get the new ones. So, let's make sure that it is doing that properly. So, we have wherever that was, our get playlist items. Then, we're getting our existing URLs. And I believe that should follow the same pattern we did before where article and columns and get URL. Yep, that's exactly the same. And we're just filtering out to make sure it includes and we're just getting that URL just like that. Okay, that does look like it's working exactly as we expect. So I can at least go ahead. I can minimize this section and I can minimize this section cuz those are the correct thing. Create oath 2 client. That looks like it is probably correct. Yeah, we're setting our credentials. So we can minimize that. That is good. And I believe now that we have the videos, now we have our new video and we are on the invest ingest video step. This should take in a full video. And this is going to be that YouTube v3 playlist item. There we go. Also, in my get playlist items, I want to make sure wherever I'm doing my items, I only want to push in the items that have the status of public. So, I'll say item item status is equal to public. There we go. That way, it's only going to give me those public ones. And we want the privacy status. Perfect. So, now let's go ahead and make sure everything else is correct. So, we have our video coming in here. And if we just do a quick search
Watch from 2:35:19on that, yeah, we get all those different pieces of information from our video. And we essentially want to get a parsed version of that for our schema related stuff. So if we look all the way up here, we have that schema video ID, title, description, published at. This should be a URL. There we go. All of that seems to be correct. So let's see where this is used at. It's used right here. Let's take that. I think we want to move that to the top of this function. There we go. Get our video item and we're going to safe parse. Our video ID is just our video ID. There we go. Title, description, publish at, and so on. And we don't even need default values for these. There we go. And then down here where we originally were doing that, let's get rid of that. Take our error. Move that right there. Then I can get rid of that here. Perfect. thumbnail URL should be coming from those thumbnails. So yeah, this all we can kind of get rid of a lot of this. We don't need to re-query for the video because we've already gotten it. So that can be gone. This entire thing can be gone. There we go. The captions is something obviously we still need to care about. And other than that, this is looking really good. There we go. Caption response. I think I'm incorrectly using my video here. I should be using my video item.
Watch from 2:36:50So, let's make sure everywhere else it's video item. Video item. There we go. And I don't know why this is called video ID. We're just going to call this ID. Scroll up here. Call that ID there. Now, wherever we were, looks like I lost my place. Playlist items. Yeah, we can minimize that one. Ingest the video. Video item. Video item. Perfect. Caption track is just trying to get all the tracks. Then we want to find the first English track. So here we go. We're getting an English track. That's the first one. There we go. We don't want to fall back to anything. So get rid of all this fallback stuff. We definitely don't want that. There we go. Get the caption ID. Whoops. Caption ID. There we go. And as long as we have a caption ID, then we need to do something with it. And then we'll throw a fatal error if we can't find it. That makes a lot of sense. And then we can parse the caption. So let's see what's happening in here. We're going to download the captions as SRT. We don't need this as const. I don't think we can get rid of that response data. Oh, so it's giving us an interesting type being returned as our download response. I think we can clean this up because the library has a response type property we can specify as just text.
Watch from 2:38:11That's going to make sure it always gives us text back. And then we just know that this data type here is a string. So we can essentially clean this up drastically because now we know that this response type is always going to be a string. There we go. And to clean this up even further, we'll say if our caption ID is equal to null, then that's where we're going to throw our error. So we're going to throw our error inside of here. Close that off. Get rid of that. There we go. And now we have our response data, which I called SRT content. Just like that. Don't need the SRT content up here. Perfect. That all looks really good. So now we can move on. This is going to parse it. So this is just calling parse sync from that library that's being installed. Just doing all of it for us and returning us these nodes. And then we want to do some filtering on that. Now this function is much more confusing than it needs to be. We can get rid of this entire is typing and this whole is data thing right here. And essentially it's just going to make sure it only gets the one that are Q type. Q's are essentially the spoken word coming from this library. So that is what we want. Then we're going to map it to give us the text and the start time because those are the only two things we care. And we want to make sure that we actually have text. So that's just filtering out any empty rows as well. If there's no valid captions, throw an error. That's great. And now we need to create those chunks with an overlap. So this is just collecting all of our different chunks with that proper overlapping. We're then just calling embed chunks. So let's just see what that does. Oh yeah, we've already created that function. So that's just embedding that content. And now we're just saving everything. So, we're taking all of that content, saving it. That all
Watch from 2:39:33looks correct to me. Um, then you have duplicate detected. Then we're looping through each one of our chunks, inserting them, and making sure that we get our start position set properly. That also all looks correct. So, I think all of the code for this is correct. And that's everything, I think. So, we can actually kind of test to see if this actually works. Let's just close out of everything. Minimize this down. And let's just do a quick test. Actually, before we test, I want to make sure in our workflow for this, we are doing a batch of 10. Yeah, the batch size is only 10 and there's a 9h hour delay just because of how aggressive the quota is for all these different things. So, let's just go ahead and test to see if that works. We'll come into our workflows, send a request to that connection was rejected. Of course, I don't think I'm actually running our commands. And let's just get our workflow set up. There we go. Now we should be able to run this. Hopefully it'll return a result over here. Server error, of course. Looks like there's a ENV variable missing most likely. Google client secret. I think I just called it something slightly different. Let's see here. There we go. Let's just give that the correct name.
Watch from 2:41:06And now we'll retry that request. Unauthorized is what our request is this time. Let's see what it's giving us as the error. Um, yeah, just a straight up 401. Oh, it's probably because our cron secret is maybe not set. Let's go into our actions or apps. Sorry. API workflows here. Verify and run cron. Verify and run cron. Yeah, we're just not setting our actual cron. So, if we take a look at that verify and run cron, we need to have a bearer authorization header. That's what we need. So inside of here, authorization is bearer. And we want to use that cron secret. And to get this value, we can just do this. That's going to give us that actual cron secret. Make sure this has a dollar sign in front of it. We'll copy that to both of them. And now when we send this, we should actually see it is starting up that cron job. And if we go to our workflows and we pull this over, we can actually see that we have this ingest video step that is running. So it is actually working like we expect. It actually looks like it is cued. It's now executing this. It was able to get the videos from the playlist and the output is 866 videos. So, I have created quite a few videos in my past. But now you can see it is actually properly getting our videos and saving them. And now it's doing a sleep command. And this sleep command is going to last 9 hours. So, I'm just going to cancel this cuz obviously we're not going to wait 9 hours. But we at least should have 10 videos inside of our database. We can very easily test this by just running npm run db studio. That
Watch from 2:42:22should open up real quick. Here we go. And now we should be able to check inside of here. If we go to our content section, we should hopefully be able to filter quickly by a type. There we go. Filter. We want to filter by the type is equal to video. Let's apply that real quick. And now you can see we do have 10 pe 10 videos inside of here with all the different chunks associated with them. So this did work exactly like we expected it to. So now that we have that done, let's go ahead and add these. So we'll just say get add get commit add YouTube ingest. There we go. And actually before we even finish committing all this, we should probably implement the UI for this as well. So let's actually test to see if the UI for this works. So I just did a really quick search. You can see here I just searched for sandbox because I know I have some videos on sandboxing. You can see they're all popping up right here. This second number is obviously incorrect. It's doing some rounding incorrectly. So, we should probably fix that. So, whenever we have I think it's just s and then it's got some text in front of it. So, that's going to be kind of hard to search for. Let's just go to app page. It's probably in the component search results. Here we go. And we specifically want to find it. Here we go. The start position. So, the start position, I'm guessing, is in milliseconds. I don't actually care about rendering out the start position.
Watch from 2:43:54So, we can actually just remove that from our results. But down here, I want to make sure to include that start position in the URL. And I believe when it comes to YouTube URLs, all we need to do at the very end is to add t equals and then whatever our start time is. And we'll divide that by 1,00. Let's just see. Looks like we have an error. Unreachable because the left hand is never nullish. I don't think it's called start time. It is start position. There we go. And we'll just wrap that in parenthesis. And divide that by th00and since it looks like it was in milliseconds. and we need to convert this to seconds. So, let's actually see if this works. We'll just do a refresh. We'll just search for sandbox. We'll click on this and hopefully it brings me to that exact location in the video. And it looks like it did not. There was definitely some type of error going on. Let me just render out that type as well. We'll just render out a span here real quick, which was result. Position, just so we can see what that start position looks like again. And you can see it's just this really long number. It definitely seems way too long. Like even if I divide this by a th00and that is going to be 43 seconds into the video which maybe is correct. So maybe dividing by a th00and and let's just round it actually. So we'll just do a quick math.4. There we go. And we'll make sure that that's divided by a th00and. Let's see if that actually fixes the problem. So we can click on this. You can see we got our time up here and it should hopefully start us at that location. But it looks like it did not. I believe I know what the error is. The error is that we just need to put an amperand here because it already has some query parameters on the end. So now let's get rid of that result
Watch from 2:45:14text that we added. Let's refresh this. Search for sandbox. Click on this. And hopefully it actually brings us to that exact location. And you can see it does bring us to that location in the video. Looks like for some reason it's bringing us to the very end of this video. Maybe not quite where we want. Let's try another one to see if it works properly. This one's bringing us to 1244, which is bringing us to this location in the video, which is probably much more accurate. And let's try this one as well to see if this one is working. And yeah, 43 seconds in the video. That is probably fine. Don't know why this one is at the very very end, but that's probably where I said the word sandbox. So that's probably why it's bringing me to that location. Now, kind of the final thing that I want to do now that we have all of this done, let's push all that up. Do get add, get commit, fix UI. We'll push that up and then we'll be able to merge that in once all of that is done. But we can just do a quick check out and we'll just check out a branch called rate limit. Make sure that we add in a dashb at the front. And I just want to add in some really basic rate limiting. So, we'll just create a prompt for that real quick. So, I just created a rather simple prompt for this one. Essentially, I told that I want to add rate limiting to the search action using specific enviables for how many requests they can have per hour. Also, I kind of explained how that's supposed to work. I said that if it's rate limited to send an error back that the UI handles, I said to track the user queries in a database table that just tracks all their searches. This includes things like the text they sent, the user ID, as well as the results that are returned, and most importantly, the time that this search was created. And then finally, every time a query is made, make sure to save that query in the database in that table and use that table for rate limiting. So we're just
Watch from 2:46:44going to send this along. It shouldn't take too long because this is a relatively small task. And while this is happening, let's go ahead and actually create a pull request for that brand new branch that we just passed up. So here we are. We can just compare and pull request all of our code. Make sure that we want to create a pull request. And this is going to go through. It's going to make sure that code wrap is going to run on this. Forcel is going to run on this. Pretty much everything that we want is going to run on this particular code to make sure that everything works like we expect it to. Again, our deployments aren't working because we don't have anything for that database URL set up, but otherwise, everything else should work just fine. Now, while we're waiting for this to finish up, we might as well go ahead and actually merge in the pull request for this ad UI. This should have been merged a while ago, and that may actually cause some duplicates between our different code rabbits, but this should definitely be merged. So, let's just go ahead make sure that we scroll all the way to the bottom, merge in this pull request, which definitely should have been merged in before. That was my bad on not merging that in. We'll delete our branch for that particular one. And now at least we have that all merged in for our code. And now we only have the one pull request which is just for that YouTube ingest related code right here. And again, it's going through with Code Rabbit. And we also have our AI working here behind the scenes to get all of our different search related information done. So looks like our AI just finished writing up the code for the search results stuff. So let's just take a look at that real quick. Let's go to our schema first cuz that's going to kind of determine how everything else looks. Again, we don't need this text here. So we can get rid of the name of these things and we can replace this with snakecase.t wish it would just do that automatically on its own. Other than that we got our text our user ID which should definitely be a reference. So we're going to say references the users ID table. Make sure
Watch from 2:48:15we get an import on that. Looks like it's not coming in users from and let's see. We want to go at slash. Actually, we should just be able to get it from this schema section. So, there should be an off one. Let's take a look here in our schema. It looks like Oh, of course, when I originally created my schemas, I put the off one in the wrong location. So, let me fix that real quick. Inside of my package. JSON, wherever that is. This should be schema. There we go. I just put that in the wrong location. Now, if we got that done, we should be able to get the off directly from there. And users as well. Let's just make sure. Let's see what this is called. User. There we go. Okay, now we got the user ID being referenced properly. And we can even just throw on like ondee cascade. There we go. Perfect. And our result content IDs is a text. This should be an array though. There we go. That's an array of data. And we don't actually need both the timestamps. We just need timestamps.created at. So let's just make sure we have only the created app at property on there and none of the other ones. And this is called created at just like that.
Watch from 2:49:48And yeah, this is just an index to make the search more efficient for searching specifically those queries, which is what we're going to be using this table 99% of the time for. Now, let's go ahead and take a look at our package. JSON. That's just the change I made. We have our search action, which is where a lot of our actual changes are made. So, let's see exactly what changed inside of here. It's probably going to be easier if I just read the search action. So, let's find our action for search. Looks like there's some stuff that's not even used. So, let's just remove that. There we go. And here we go. We're checking our rate limit. That is just a function that we should probably read through. So, get the max request. That is just a server variable. Server variable. So, not really much point having those as actual functions. I'm just going to clean that up. Get max request is that. Get the window hours. Is that There we go. Clean up the code a little bit. Get the window start. There we go. Count property count does not exist. Yeah, it looks like there's definitely a little bit of an error here with how it's getting the count property. Let's just see what the return values are because this is returning us an array of values. So, I think I just need to wrap this inside of brackets. And now we can get the count out of there. There we go. That fixed that particular typo error. And then this is just saying okay you know rate limit is exceeded here's a particular error that it is throwing. So that makes sense. So we're throwing an error here if there's a particular problem and then we're recording the result and probably I have something being typed wrong inside of here. So let's just take a look at our values.
Watch from 2:51:04Let's see what our error is. Actually I can probably just check created at an ID. Those are autogenerated. Oh this needs to be an array. So just as simple as that. There we go. That cleaned up that file and I think otherwise it looks fine. Really all we did is just add in the result query and we just added in this checking of the rate limit. So this file was simple in our page. We just here are checking our results. So we have our do search. If the error is an instance of rate limit error then we're just setting an error. Otherwise we're setting a generic error. So that way the user knows there's a problem. That looks good to me. Server env has these values. I'm going to get rid of the default on them just so we have to manually set these values to whatever we want them to be. Database, that's fine. We changed that ourselves. Same with the schema. That's our own change. That's the O schema. We just moved the location of it. User search is good. And then this rate limit file we just looked through. So all of that actually looks pretty good. There was pretty minor changes overall that needed to be made. Let's just go ahead make sure our project is running. I'm going to pull open local host right here so we can just test it out and let's see what our default enviables are set to. We see here we have max of 10 requests per hour. So let's just go ahead and test that out. I'm going to make a bunch of different requests. We'll say one and of course failed because we need to push our database changes. So let's go ahead and do that. We can say here npm run or yeah run db push. That's going to push those different changes up. And I also probably should run.
Watch from 2:52:35Oops. Let's see here. Renamed. I didn't rename anything. Chunks embedding index. Okay, we'll just say that that's a new one. Oh, of course, because I have my chunking schema set to the wrong one. So, let's fix that real quick. So, in my chunks here, if you remember correctly, I set this. We need to change it back to the dimensions that we're using specifically with Quen. That should hopefully fix our particular problem. That did look like it applied it. I'm going to change that back to what it was before, though. And I'm going to run mpm run db generate just to generate our migration file as well, which will include all those new changes we made. So, if we open that up, you can see it creates that user searches table and adds in all the stuff that we need for that. So, now with that done, let's actually test our search. We got search one. Let's try search 2 3 4 5 6 7 8 9 10 and this is number 11. And you can see rate limit exceeded. You can only make 10 searches per hour. So that works exactly like we expected to. So this is a relatively simple and easy fix. So let's just go ahead and add this. We'll just say get add get commit add rate limit. And we'll push that up. And now we can actually go ahead and we're going to check out the code for our YouTube ingest. Just like that because our code rabbit review is done. So now we can actually work on this. So the very first thing that we see here is await the session before the off check. This is actually a huge security vulnerability.
Watch from 2:54:06I can't believe I didn't catch this. So in our search content. So if we just go ahead and we just do search content and it was right here. We just need to make sure that we await this off check because otherwise the user is going to return to us a promise. That is a massive potential security vulnerability. So, at least we have that fixed. That is good. Next thing that we need to do right here, uh, the enter key submission is engaged by is loading, allowing overlapping searches. And not really a huge problem. I'm too not too concerned about that. Uh, duplicate orphan search icon markup. Uh, this is also fine with me. It's just a UI thing. I don't really care too much about the UI on this application. Uh, again, this is an error we've already seen before, so we're going to get rid of that one. Filter never excludes already ingested video. ID is a full playlist item object, not a video ID. Interesting. Let's see exactly what this is talking about. Returns the item export async function blah blah blah blah blah. So each ID is a full item object interpolated into the template. Oh, okay. So um that is in the get new videos from playlist function. So let's just go into that function. And if we look here, this ID object is not just an ID. You can see it's a bunch of different stuff. We need to specifically get I think the ID from the ID. Let's just make sure that is exactly what we want. Let's see the proposed fixed here.
Watch from 2:55:35It actually tells us to get the content detail video ID which is probably actually correct. So we can say here that we want to get the uh so video ids is ID.content details video ID. There we go. And I should probably call this like V for video. That actually solves that particular problem. So again a huge problem that we didn't actually realize was a problem. Next thing we have our ingest YouTube videos missing optional chain causing crash on empty items array. Um okay if items results to an empty array is undefined. So essentially let's find where that code was. I believe it was in here. Yeah this essentially should just have a question mark in front of it in case the items is null as well. Solve that particular problem. This one says error message references the wrong ID field. Again that's the exact same thing. So we can say video id. Let's just search for all those locations. video.content video ID. There we go. That solves that problem. Again, it's just a misunderstanding of how the API works. Wrap content puts chunk inserts in a transaction to avoid orphaned content rows. This one I could do, but I'm not too worried about it. We're just going to leave it as is. And that's all of the different things that it says that we should fix for this. So, let's go ahead and actually just commit those.
Watch from 2:56:52We'll just say bug fixes again. Use better naming. We can push that up. And now we have all those different changes done. Let's go ahead and we'll just merge those changes in. Give it a second here for it to load. Confirm that merge. And there we go. We will delete that branch. So now what we can do is we can go to our very last pull request for this. So let's just create a pull request for our rate limit. You can see it's trying to merge it into main. It says it actually can't automatically merge because I have some changes that are being made. Let's make sure it still can't actually merge. Give it a quick second. Yeah, it can't merge because of conflicts. Let's just resolve those conflicts. So, we're just going to get checkout main, get poll, then we're going to check out rate limit and we're just going to merge in main just like that. Let's take a look at our conflicts that we're getting. So, inside of search, we just have a conflict here. This is using an await. This is using an await. So these are actually already being fixed potentially for us. So we're just going to stick with the one that's incoming. Just like that. And actually it should be called session. There we go. Because that's what it's being called other places. There we go. And now we can just add those changes. There we go.
Watch from 2:58:08Perfect. Now let's push those changes up. And hopefully once we do that, we should now be able to merge these changes in as we see fit. Give that a second. Let's just go ahead and merge that right now. Give it a quick second to load. And we can delete that branch as well. And now with that done, we should be able to go to Verscell and actually see that everything is working like we expect. So here we are on our deployments for Versell. You can see right now it's building this. I just can cancel this one. I don't care about these preview ones. Yes, we'll cancel that deployment. I care most about this one right here, which is going to be the pull request specifically that's going to be to production. So let's make sure that this works like we expect. So looks like we have a failure. We're just missing some environment variables for rate limit as well as the Google client secret. So let's go ahead and set our environment variables. We will say that our Google client secret I just need to rename this uh won't let won't actually let me rename it. So that's perfectly okay. We are just going to delete this one. Yep. And then let's just add it again. So we'll say Google client secret. And let's copy that over from our environment variable. So let's come into here our environment variable Google client secret. Copy that and we'll paste that.
Watch from 2:59:25There we go. Give that a quick save. We also need to do the same thing for our rate limit max requests. So we'll add that in there. Let's just say it's 10 for example. This one does not need to be sensitive. And then we have finally our rate limit window in hours. So we'll just add that. again it's not sensitive and we'll just say 10 per hour. So now at least all those different environment variables are there and let's try to redeploy this. So we'll just redeploy. Yep, redeploy that. Give it a quick second to load and now I'll get back to you once this is done. That just finished up. So one thing that I want to immediately do is I want to go to my cron jobs and I want to run the cron job for ingesting the YouTube videos to make sure that this works in production. So now that that's running, we can go over to our deployments just to or not the deployments, the overview. So that way we can deploy this or pull it up. And then let's go to our workflows. We should hopefully see if we just give this a quick refresh. Might take a second for it to show up, but that YouTube one should show up inside of this list. Let's make sure here that we're signed in just as a starting point. There we go. That is correct. And here now you can see the YouTube one is active. And hopefully it looks like it's working. It already is ingesting a bunch of videos and we're on this sleep step that is going to take 9 hours to finish. But if we just search for sandbox, we should hopefully see some YouTube videos that have sandboxing. Looks like actually it didn't bring in my sandbox videos first. Let's see what videos it did bring in first. Let's just decrypt this real quick for our input. The input for this one is the containers. So that is containerization. So we'll just say containers or let's just say agentic AI container.
Watch from 3:00:41There we go. Now it's popping up when I do that. So you can see that it is getting my videos. And if we just search for like React hooks, we should hopefully see we get some hook related stuff showing up as well. We can also test the rate limiting. So I can just do a bunch of searches and eventually once I do too many searches, we should hopefully get rate limited. There you go. Actually an error here. Let's just make sure. Looks like there's a particular error that's being not caught properly and this is probably that rate limit error. So we may need to look into why this is happening as well. I think the reason for this is because in production it doesn't actually give you any information about the error when you have an error in your server component. So we essentially kind of need to rewrite our rate limiting code. So let's just go ahead into our app. We're going to go into our actions for our search and instead of returning a error that it throws wherever this is at. So it's here where it's throwing that particular error. Instead, we need to handle this in a different way. So what we could just do is we could just wrap this in a try catch. This is kind of a messy way to do it, but it's going to work for our particular use case. And here we can just say if we have some type of error then we just want to return error rate limit exceeded. And now anywhere we're using this search content. So let's just see where we're using our search content. We have our search results here. We don't actually need to worry about wrapping this too much in a try catch. But we should see our search results could return to us an error object. So we can just say if search results and it should have an error object as the other option. Yeah, it should have an error. So let's just say if there we go if error is in it then what we want
Watch from 3:02:16to do now let's just swap these around. So if the search results and the error is in the search results then I want to set the error. Oops. Otherwise we will set the results. There we go. And I can get rid of all this try catch code. Don't think we actually need that. There we go. Now let's go ahead and make sure that we push this up or test it. So, we'll just sign in real quick. There we go. Let's just do a quick search. And of course, we get rate limit exceeded. Exactly what I want. So, now let's just do a quick add commit. And we'll push that up. And we'll create a brand new pull request for that once it actually pushes all the way up. We'll just do it this way. There we go. Great pull request. And we're just going to immediately merge that. Generally not a good idea. You should wait for review, but for our particular case, it should be fine. There we go. Delete that branch. And now all we need to do is just wait for it to actually be pushed up to Verscell. And while we're waiting for that, I just want to give a huge shout out to the sponsors of this video. Neon is absolutely amazing. If you need a really simple, easy to use Postgress database, which is perfect for this project. All of their branching is absolutely amazing. Also, Code Rabbit, great for that AI code review. You can see it saved us from some massive security vulnerabilities in our application that I completely glazed over. So this can save you big time on those application errors and save you tons of time as
Watch from 3:03:46well. And if you're doing open source software, you can use it entirely for free. And of course, lastly, Versel, we have them for our hosting as well as that workflow system, which makes working with our application super great because we get those durable retries for any potential errors and bugs. And we can have long running tasks like that 9-h hour wait without having to worry about spinning up denser compute and so on. So, I want to give a huge thank you to Neon, Versel, and Code Rabbit for sponsoring this video. You'll find links to all of them in the description down below for you can try them out. All of them. I'm using the free tier for this project. So, you can see you can do quite a lot with just the free tier of these different services. Now, with that out of the way, we should actually have our deployment hopefully done at this point. You can see it's currently building it right now, and I'll get back to you once this finishes. So, we can finalize checking that rate limit. Looks like we have a little bit of an error in our TypeScript code. This is just from me trying to go too quickly with things. Looks like it's way at the bottom here. Results.length. And this is probably because results could be that error object instead. So we just need to make sure that our results is not the error object. So if results, we'll just say if is array results and results.length is greater than that. There we go. That should hopefully solve that particular error. And now we can actually just push this up. Hopefully that's the last error that we have. Fix error. And I'll get back to you once this is I'll merge it all in. So, the deployment just finished and since we were rate limited before, as soon as I search, I should get that rate limit error. And you can see rate limit exceeded. So, everything is working like we expect it to. Again, I highly recommend that you check out Neon, Versel, and Code Rabbit rabbit. They're going to be linked down in the description below. With that said, thank you very much for watching and have a good day. Also, if you want to check out this application, I'll have it linked down in the description if I get around to posting it. That way you can actually search through my content and find things that you want to
Watch from 3:05:17Full transcript
- 0:00
This video is going to be a master class on how you can create your own rag retrieval augmented generation system where essentially we are ingesting every single YouTube video and blog article that I've ever created. And then you as a user can search for anything you want and it's going to show you every single one of my articles and videos that cover that topic. Also, it's going to bring you to the exact point in that article where I talk about that particular
- 0:20
thing. As well as in the video, if you click on it, it's going to bring you to the exact timestamp where I talk about that particular topic. I'm going to show you how to build all of that. But on top of that, I'm going to show you how I do this in a real world scenario. I'm going to be writing this code as if I write my real code. So instead of just showing you line by line what to do, we're going to be going through and seeing how to handle things with certain agents and when you need to write code by hand, how you can make sure the AI actually writes
- 0:42
code that is good quality and to maintain that quality throughout your codebase. And I'm going to be going through the full production pipeline of how you deploy this and make sure all your credentials and so on are secure and how to separate that dev and production environment. Now, to make all of this work properly, we have a few different tools that we're going to be using. We're going to be using Neon for our database of choice. That's because they are a Postgress database with full branching support and we need a vector
- 1:04
search database to make this whole retrieval augmented generation system work. So I want to give a huge thank you to Neons for supporting this video by sponsoring. We're also going to be using Verscell workflows. These workflows essentially allow us to do these longunning background tasks without worrying about them failing and breaking everything. They are super nice for ingesting hundreds or even thousands of videos and articles or doing longrunning
- 1:24
AI related tasks. So I want to give a huge thank you to Verscell for sponsoring this video. We're also going to be hosting on Verscell since we're writing in Nex.js, but if you want to use workflows, you don't have to be working with Nex.js or hosting on Verscell. You can do it entirely independently. And finally, to make sure our code quality is high quality, we're going to be using Code Rabbit for everything for our code reviews. Now, they also have this agent mode that you can use as well inside of Slack for
- 1:45
communicating with it through Slack to make it work more seamlessly if your team uses Slack. But for us, we're going to be mostly focusing on the review portion. And the really nice thing is if you have an open- source public repository, you have access to free code reviews inside that repository, which is what we're going to be taking advantage of in this video. So we can get nice code review to make sure our code quality maintains itself throughout the entire lifetime of our application. And
- 2:06
again, huge thank you to Code Rabbit for sponsoring this video as well as Verscell and Neon. Welcome back to WebDev Simplified. My name is Kyle and my job is to simplify the web for you. And instead of starting in the terminal with code, I actually want to start with a GitHub repository since when I create real world projects, the first thing I do is create a repository for them. So we're going to
- 2:26
call this video blog suggest. And I'll just put yt at the end so I know this is for the YouTube video specifically. We'll keep this as a public repository. And everything else I'm going to keep the same. No readmes or anything like that. I just want to create essentially a brand new blank repository. That way I have somewhere that I can host and push my code to immediately. Now once that is done, the very next thing I want to do is I want to get our project actually
- 2:47
set up. So let's just go ahead. We'll copy this command for actually initializing our repository inside of our project. And now we're going to open up just essentially a blank VS Code terminal. And what I want to do is I want to create our project. And many people make the mistake here of immediately turning to AI to create their project. I think by far the best way to create projects is go through the create command for whatever tool or framework you're using. You're going to get much better results for your actual
- 3:07
project. So in our case, we're going to be using Next.js. So I'm going to say npx create next app and we want to get the latest version and we're going to create it in the current folder. So I'll just put a period at the end here. This is going to run through to create that nextjs application for us. Now for this project it really doesn't matter if you use next or any other framework because our front end is incredibly minimal for this application and most of our code is
- 3:28
going to be in the back end with these like durable workflows. So I'm going to actually go ahead and I'm going to customize the settings I use. I like to use typescript. We're going to use eslint for our default. I'm going to use the react compiler tailwind CSS. I always like to use a source directory. I'm going to be using the app router and I don't really care about changing the import alias default. And we also here can include an agents.md if we want to. We might as well include that just so it
- 3:49
has that code for our agent to be able to work with. But again, it doesn't really matter because the actual Nex.js specific code we're using is incredibly minimal. Since this is essentially a single page application with no other content other than a search bar and a result section and all of the complicated code goes into building out how that search engine actually searches and gets the data that it can use. All of that embedding and chunking is where all the complexities of this application
- 4:11
come in. Now, with that done, we actually have our project created. And I like to just immediately commit this project and deploy it to see that we at least have something showing up. So, let's just go ahead. We'll just run this as a dev dependency just to make sure that we have our code working locally. And then I'm going to go ahead and push that up to production. That way, we can see that project working in production as well. And if I open this up, we should hopefully see something show up on our screen in just a couple seconds.
- 4:32
And there we go. We have something showing up on our screen. super zoomed in, but you can see we have an application working. Now, the next step is going to be adding that to our Git repository. So, let's just go ahead and we're going to paste in that command that we copied earlier. This is going to essentially push all of this code to our GitHub repository. And now, if we go ahead and we open up that GitHub repository, we should see when we refresh this right here that we actually have all of our different code just like
- 4:53
we want. Now, the next step is to actually try to deploy this just to make sure that it's working like we expect. Again, the quicker you can get everything working in production as well as development, the quicker it is to figure out if you have problems. If you build your entire application in development and then you push it to production, you don't know where in the actual process it broke if it doesn't work properly. So by incrementally constantly pushing to production, we can make sure every step of the way our
- 5:14
project works like we expect. So since we're going to be hosting on Versell, we're just going to go over to Verscell, we're going to create a brand new project. So let's just add a new project here. And this is where we just want to import from that repository. Now in our case, it's not going to show up on this list. If I were to search for video blog suggest YT, it's not going to show up. And that's because I've configured Verscell
- 5:34
to only have access to specific repositories. So if we just configure this GitHub app, we can actually come in here, make sure we choose the correct section that we want to configure. And we're just going to log in real quick. And once that's done, we can now actually choose which repositories we want to give Versel access to. If we scroll down here, you can see I've only given access to a few different projects. We can come in here and add that YouTube or sorry video blog
- 5:56
suggest. Make sure we get the YouTube one right there. We can now give that a save. And now we have access to that. And you can see it pops up on our list. And we can import that project directly into Verscell. Now since we're using NexJS, pretty much all these defaults are going to work for us, which is really nice. We can just click deploy. And it's going to deploy the entire project to Verscell for us. It's going to take a little bit of time, but hopefully once this finishes, we'll have
- 6:16
a working application. So it just finished deploying our entire application. And if we click, we should be able to view our application. Again, it's just the boilerplate Nex.js. But we at least have now seen that we have code in GitHub. We're able to push our application to production and everything is working like we expect, exactly the same in dev and production, which is the perfect first step before we actually implement any code at all. The next thing that I like to do on pretty much
- 6:37
any real world project is inside of GitHub, I like to go to the settings and I like to essentially make security around my branches. Now, to do this is quite easy. We can go to this branches section and we can add a branch rule set. And I essentially want to make it impossible for anyone to push to that main branch. So we can just say block main push. There we go. Make sure that this
- 6:58
is active by default. And then what we want to do is we want to add a target branch. So we're just going to include that default branch. And then what we want to do is we want to restrict anybody from pushing to that branch specifically by making changes. We're going to require a pull request before merging. So just like that, require pull request before merging. If you want, you can go ahead and you can actually add certain requirements for code review
- 7:19
inside of here. I don't really care about that at all. We just want to make sure that it's going to have a pull request before we can merge directly to that section. Once that's done, we can just go ahead and click create. And this is just going to make it so that we can no longer push directly to our main branch. We have to create a pull request and then merge that pull request in. This just makes sure we don't accidentally push code which gets automatically rolled out because the way that we have our production setup
- 7:40
working is anytime we push to our main branch, it's going to automatically roll that out to our production environment. So let's just go ahead and test to make sure that this works like we expect. Let's just change something inside of our code. So we'll just go inside of our page here. Anywhere we can just change this code to say new code. Just like that. Let's try to add this code. So we'll just say add new code. And we'll
- 8:01
try to push this particular change. And we should say immediately it failed to push these particular commits. And that's again because changes must be made through a pull request. So we need to create a brand new branch and then we can actually merge that particular change in. And this is again just a safety feature for us. And it's making sure that all of the code that gets pushed into this main branch goes through a set workflow that we can really monitor, make sure we have code
- 8:21
reviews, use code rabbit, and all those other things that we care about. Now, the next massive mistake people make when they're building out a project from scratch is that they immediately say, "Okay, I got my project running. I've got it in production. I got my branch system set up. I'm going to go ahead and start working with AI immediately to start building things." This is a massive mistake. AI makes writing code incredibly quick, which means people skip a lot of the thinking part of writing code and understanding what you
- 8:42
actually need to build. Oftentimes, that slow aspect of manually writing code means you have to think as you're writing code about what you're writing, why you're writing it, and what it does. When you have AI write your code for you, you kind of skip that part. So, it's very important to spend a lot of upfront time thinking about exactly what you want to build. This is why in pretty much any new project, I like to create a list of everything that I need to create
- 9:03
inside that project so I can understand the overall picture of my product. And that'll also better help me communicate with the AI on exactly what I wanted to build. So, this can be anywhere, a whiteboard, a document, it doesn't matter. I'm just going to do it in a document right here. And we're just going to say project list.txt. Doesn't really matter what this particular file is. Let's just go ahead and actually make it a markdown file.
- 9:23
That way, we can use markdown inside of here. And all I want to do is really just create a list of all the different things we need inside this project. I'm going to go ahead and do that and come back once I finish. Now, I just finished writing out essentially everything about this app that I can think of off the top of my head. You can see it's a not super short but not super long document. I kind of broke it down in a few sections. Like this overall section just says what the application is. Essentially, it's
- 9:43
just a way to ingest all of my videos and articles and then chunk those down and be able to search through those by you when you search for particularly anything that you want. I then kind of describe the exact user flow. This is really, really important when you're thinking about your application. Think about how the user uses your application because that'll help you decide how you want to do different architectural things. So, for example, in our
- 10:03
application, they first need to log in to prevent any type of abuse. They then need to search for whatever they want. They're going to get a list of results which they can then choose from that list what they want to view. And if for some reason they send too many requests in a short period of time, they're going to be rate limited again to prevent any type of abuse on our particular system. I then gave a brief overview of what our UI is. Essentially, it's just a single page with a search bar and then there
- 10:24
are results below that search bar and I kind of specified what I want in those results. For example, I want an image, a title, a description, match percentage and so on. I then kind of described how I want the ingest process to work. Specifically, I want to use these versell workflows to ingest everything to give me durability and retry systems built in. And I also kind of talked about how I want to chunk up the different information because the
- 10:44
articles, for example, blog articles have headings with heading levels. I have very good hierarchical information and I want to make sure I preserve all of that different category and hierarch information inside of how I do my different chunking. Finally, I gave a very brief description of some of the different tables that I think I need inside of my database. This isn't going to be like the final tables or anything. It's just kind of what I think I need off the top of my head. So, once I've
- 11:06
thought through my project at a deep level and spent quite a bit of time with it, then I go ahead and I do more thinking, but specifically with an AI companion to help me with that thinking. So, what I like to do is just open up any type of AI that you want to use. It really doesn't matter. What I'm going to do is I'm just going to open up a brand new sandbox with Pi inside of it. So, I can just type in start sandbox. This is a command that I created myself which is
- 11:27
going to start up a brand new sandbox for me in this particular environment. If you want to learn more about how you can actually sandbox your AI to make sure that it's more secure and easier to work with. I'm going to link a video up in the cards and description for you. This is just my sandboxing setup. I use Docker sandbox for it. But you can see that now we have a sandboxed AI with all the information we want. And inside of here, I can essentially ask it to ask me
- 11:47
a bunch of questions. You can use, for example, some skills like the grill me skill from Matt PCO or you can just write your own very simple command that essentially says, hey, ask me a bunch of questions about this project and then give it a very lengthy description of your project. So in my particular case, I may reference my project. I'm going to say, I am creating a project. Here are the details. And then I would
- 12:09
reference that file that I just created just like that. And then I would essentially ask it to ask me a bunch of questions. Ask me questions to further flesh out the plan. Make sure the questions cover all topics and architecture decisions.
- 12:30
There we go. And I would just go ahead do that. And I would go through and answer all the different questions that it gives me. And the important thing about doing this is that it's going to help you think of all the things that you didn't already think of. It's going to ask you questions about various different things you probably never even thought of or considered. And this is going to help you flesh out this document even more and really figure out what you want. And often what I like to do is after I'm done answering all these
- 12:51
different questions, I like to ask the AI to specifically go through and essentially create a document summarizing all the different things we talked about, explaining exactly all the different things inside of my project that I want to do. So it'll go through it, ask me a bunch of questions. I'm not going to actually go through and actually answer all these different questions. The main reason being that I've already kind of gone through this process the first time I built the project, but this is exactly what I
- 13:11
would do when I'm building out the project from scratch. And that gives me all the planning that I need up front to actually start working on the project. And even if you're not using AI for any code writing at all, this is still a really great step to make sure that you're writing the actual code that you need to and that you have essentially a list of tasks of all the things that are important to work on. Now, from here, the next thing that you want to work on is figuring out what your first task is you want to implement. In our case, I
- 13:32
think the best first task to start with is just going to be a simple user authentication for log in and log out as a relatively easy and simple thing to get started with and we're going to use it everywhere in our application. Now, again, many people just go to AI and say make me off or something like that. That's obviously not going to give you the results that you actually want. What I like to do is I like to do a lot of my setup manually and then I like to use AI
- 13:54
to either enhance it further or to write code on top of that setup that I've created. Now, when it comes to setting up our authentication, we first need to set up Drizzle and then we can set up better off. And I'm going to be using the newest version of Drizzle, the Drizzle 1.0 release. It's not quite stable yet. This is the release candidate version, but hopefully by the time you watch this video, it will actually be stable. So, I'm going to go over to the Drizzle documentation, and
- 14:15
we're going to be using Neon for our database. So, I'm just going to select Neon, and I'm going to go through and manually set this up. And the main reason I'm doing this is because we're essentially using a version of this drizzle Drizzle library that's not in a lot of the training data. So often the AI is going to get this wrong. So I'm just going to go through and set this up. And it's also very easy to set up. So I don't see a reason not to do it. So let's go ahead and just make sure we copy over the different things that we need to install. So let's just create a
- 14:36
brand new terminal to install everything that we need. Just like that. And then we're going to go through and you can see we need to set up a connection environment variable. That's perfectly fine. So inside of ourv, which it doesn't even look like we have yet. Let's just create one.v. We can create a database section just like that. And we'll paste in our database URL just like that. Next thing I need to do is you'll notice it is using the release candidate version of
- 14:57
all of this different software, which is exactly what I particularly want. Next, we need to actually connect Drizzle to our OM. So, let's just copy that file. This is going to be how we access our database. So, inside of our source, I'm going to create a new folder. We can call this whatever we want. Let's just go ahead and call this our DB folder. And inside here, I'm going to create a file called DB.ts. And we're going to paste in that file. And if we just
- 15:18
export this variable right here, we now have access to our database directly. Now already you'll notice an error with our process env. You know I could override this by putting an exclamation point here which is probably what AI would do. But in my case I want to set up typed environment variables. We're going to be using T3 env for that. So we can just go ahead and make sure we install the dependency for that. I'll copy that over. Make sure we just
- 15:38
install that dependency. Just like that. Now we have that inside of our project. And since we're using Nex.js, this is actually super easy to set up. You can see here we have our create env file with all of our different enviables. And that's pretty much all that we need to do. And I'm actually going to be breaking this up in between a client and a server version. So inside of our source folder here, we'll create a folder called data. And inside of here,
- 15:58
we'll create a client env.ts. And we'll also create a server env.ts. Just like that. We'll paste down that code that we got from here. And this is where we set up all of our code. So in our case, database URL is the only URL we have. So let's just go ahead and make sure that's our only section. We're breaking apart our server and client. So we're just going to do this. And since we're doing this inside of only the
- 16:19
server, we can actually use the experimental runtime version and just set that to process.env. Just like that. And that's going to be everything that we need to actually set this up inside of our server. Now, I'm going to be calling this variable server env. And I also want to make sure that my empty strings are considered undefined. That way, if I pass along an empty string, it essentially instead of using as a
- 16:40
string, just treats it as if the variable was not defined at all. So, there we go. We now have our client or our server env set up. I'm going to do the exact same thing for our client. Just like that. So, our client EMV, we'll call this client. Right now, we don't have any data inside this section. And instead of using the experimental runtime, we just have to use our normal runtime. And we're going to have to pass in each object as we use it. Just like
- 17:00
that. So, we'll just leave this blank for now cuz we don't have any client environment variables, but we will in the future. Now, this means that here we can use that server environment variable just like that. Get access to our database URL and everything is typed just like we want. So now with that done, let's go ahead and actually finish out what all of our Drizzle configuration is going to look like. If we scroll down a little bit further, the next step is going to be creating our schema for all of our different code
- 17:21
inside of our application. So let's just create a very basic boilerplate schema. So inside of our application where we have our database, I'm going to create a new folder called schemas. And I'm going to create a file inside of here. We'll just call this test.ts. And essentially, I just want to take all the code that's inside of here. And we'll copy this over as a starting point for our application. Now for our particular application, I want all of my different name key value
- 17:42
pairs here to be lowercase snake case values. So what I can actually use is I can use the snake case type just like this and I can say snakecase.t and that's going to create a table where all of these names are going to be snake case. So for example, if I had like first name here, it's going to convert that to first name for the actual database table name itself. Now this is
- 18:03
just a test table. So we can just leave this as is. Maybe we'll just give it an ID and a name just like that. So now we have a test table we can work with just like that. Now the next thing I want to do is I want to export that schema so I can use it in my database. So I can create a new file called schemas.ts. And all I'm going to do is I'm just going to export the file directly. So I'm going to export that schemas.ts.
- 18:26
And I want to make sure I export everything from there. So I'll just say export star from and we can say at slash or actually dot slash schemas slashtest.ts. That should be working properly. And it looks like I just need to remove that.ts from the end there. And there we go. Now that is working like I expect. And I have all of my schema related information available. Now to make sure I have that access to that information
- 18:46
in here. So for example, when I type in db.query, I want to be able to have that test table show up inside this list. To make that work properly, we need to pass along essentially a relations property. So if I just do here relations, that is what we need to pass along. So we're going to create a brand new file for our relations as well. Just like that. And this relations is going to be relatively straightforward. We can export a
- 19:08
constant variable called relations and I believe it is called define relations just like that. And what we want to do is we want to pass along our schema. So let's just get all of our schema information. So we'll say import star as schema from and we want to get that from that schemas file. Just like that. That's going to be all of our schema information. And then we can define
- 19:28
relationships between our tables as we have them. But right now we only have one table. So there are no relationships. So this is all we need to do to define our relations. And in here we can just import that relations. So let's just add an import. And now we can come in and say db.query dot. And you can see we have that users table being shown up. And that's because this is called user table. If I just call it tests table or something else.
- 19:49
Now it'll actually show up as test table inside of here. There we go. Just like that. So now we have everything set up for our database. All we need to do is make it so that we can actually push and migrate all of our different code. And we need a drizzle config for that. If we come over here, you can see set up a Drizzle config. Give it this exact name and paste in all of this information. So, let's just go ahead. We'll create a brand new file called drizzle.config.ts.
- 20:11
Let's make sure it's at the root of our application. And we'll paste in that file that we just copied over. And you can see we just want to make sure all of our output and schema directories are all in the correct location. So, for example, for our output for all of our migrations, I want that to be in this DB folder here. So, that's going to be dot /source/db/migrations. That's going to give us all of our migration code. And for our schema, that
- 20:33
is/source/db/s schema. I believe that is the exact location we have it. Actually, ours is called schemas.ts. I think I'm going to rename that to just schema because that's a little bit easier to work with. There we go. Make sure I save that in all the places it's imported. Our dialect is correct. And for our URL, again, we can get that from our server env. And we just want to make sure that
- 20:53
we get the database URL. There we go. That should be everything that we need. Now, let's just make sure our env config is the first thing that loads. Now, really the last thing to configure before we can actually test this is we need to set up an environment variable for our application. So, to do that, we're going to go over to Neon. We're going to log in and we're just going to create a brand new database. So, let me just get this zoomed in so it's easy to see. We're going to create a brand new project. I'm going to call this video
- 21:14
blog suggest YouTube just so we know this is the YouTube one. Keep whatever Postgress version we want. We're not going to be using Neon for authentication, but they do have a really nice authentication system. But ours is so simple, we're just going to be using our own O system with better off. Once that's done, we now have a database set up. And by default, we're on the production branch. But the nice thing about Neon is that we can actually
- 21:34
branch our database to make it easier to work with for various different uses. So I'm actually going to create a brand new branch from this, which is going to be a child. So let's just go over to our branches. I want to create a brand new branch. It's going to be a child of production. We're going to say it never disappears. And we're just going to call this dev for our local development. And we'll branch over all the different information. Right now, our database is blank, but that's going to copy our
- 21:55
information. And here, what we can do is we can just copy over all of this code right here. And now in our application, we can paste that down as our database URL. And we're able to access our database directly using this URL. Now, that should be all the configuration we need for Drizzle. And again, you can kind of see why I did this myself instead of letting the AI do this because there's a lot of opinionated things on where I want my files and how I want my code structured that would
- 22:16
require me to essentially change everything that the AI does. So, it's much easier just to do it myself in my opinion. Now what we can do is we can actually go ahead and set up our commands for this. So we're going to have a db migrate command and that's just going to call drizzle kit migrate. We're also going to have commands for generate. So we're going to come in here with generate
- 22:37
and we're also going to have a studio command which is just going to make it easy for us to view the data in our database. Just like that. Now the generate command is going to generate a migration file. A migrate command is going to migrate those changes for us. And also let's add in a push command here which is essentially just going to push our database changes directly without doing anything else. So we kind of skip the whole migration process.
- 22:57
This is useful for like testing while generate and migrate are very useful for more production style environment. So let's actually test to see if we can push those changes to our database. We just run npm run db push and hopefully it should take those database changes and push them directly to our neon database and we should be able to see those changes inside of neon. So give that a quick second to load. You can see
- 23:17
it's reading through everything. It's pulling our schema and it looks like the changes applied properly. Now, if we go ahead and we check inside of Neon, we should be able to click on this table section and we should hopefully see there we go. Our test table is here. So, all of our data was pushed properly. Obviously, there's nothing in our database, but we have that table, which means our connection is working like we expect it to. Now, what we want to do is since we've essentially created a point
- 23:37
where we have working software exactly like we want it, I want to commit those changes as like a save point for our particular application. And again, we're on the main branch and I want to swap over to a new branch. So, we're just going to check out a branch. And this branch is going to be called setup off because that's what this branch is all about. And I'm just going to add these changes that we've made. So, we can just say get commit
- 23:59
setup drizzle. There we go. So, now we have our drizzle set up. And I can even push this along if I want. And it's going to create a brand new branch for me just like that. I can go ahead and I can even create a pull request for this by clicking on this link. but we're not quite ready for a pull request yet cuz we still need to do the authentication portion. Now, the next thing we want to work on is setting up better off. And this is a much more involved and kind of boring process in my opinion where
- 24:20
there's not a lot of opinionated stuff that you need to worry about. This is a perfect task for AI. So, what I'm going to do is just get started by installing the better off library. So, I can say npmi better off. And since we're using the release candidate version of Drizzle, we're going to get a warning. So, I'm just going to type in d-force here. essentially forcing it to override the actual version of Drizzle that
- 24:40
better off expects. That's because Better Off already supports this newest version of Drizzle and everything works perfectly fine. We don't have to worry about any errors. I'm just essentially forcing the npm compiler to work properly with installing better off. Now, once that's done, in order to get the best out of AI, one of the best things you can use are skills. So, if we actually take a look at Better Off and we search through the skills directly, you can see that Better Off themselves
- 25:01
have actually produced a lot of different skills. And if you're unsure what skills are, I'll link a video in the cards and description for you going over everything related to skills. But these are essentially ways to supercharge our AI with exact knowledge on the things we want it to do. So in our case, we wanted to implement better off properly. So we're going to give it the skills to do that. So we can click on one of these skills. And since I want to look through all of the skills for better off, I can come up to this
- 25:21
section right here where it says skills. And you can see I get a command npx skills add better off/skills. This is going to allow me to search through and add all of the skills related to better off that I particularly want. So, we're going to come back over here. I'm going to paste this down. And I want to install these skills essentially directly inside my project because they're useful for my project. So, if we take a look at all these skills, you can see this better off best practices.
- 25:43
That's probably a good one to install. The create off skill is going to be for scaffolding out our first authentication. Obviously, that's what we want to do. So, that's perfect. And we even have some other ones. For example, we're not using email password. Organizations are two-factor. So, we can skip those. But, it might be good to have some security best practices as well. So, those are the three skills I want to install. We're going to go through. By default, it's going to work with PI. So, we can just kind of skip
- 26:04
this by hitting enter. I want to install these in our project. And we can go ahead and click yes because we want to install all of these. Now, in our agents folder, we have those three skills. We're essentially just markdown files explaining exactly how things work inside of better off. Once that's done, we can actually use our agent to do the code that we want. So, let's go back into our agent. I want to just type in /reload here.
- 26:26
There we go. Looks like I'm getting some sizing issues. There we go. Let's try that again. /reload. That's going to reload. I want to get a new session. And we should see here we go. Our skills are properly loaded just like we expect. So now this is where we can essentially prompt the AI to start writing this code for us. So let's just go ahead and we'll say that we want to prompt it with the skill for creating O. So we'll use that skill/create.
- 26:47
And now we need to tell it what to do. I want you to set up better off to use GitHub for authentication. You can replace all the content on the main homepage with just a simple login/out
- 27:08
system. Also show the users profile image and name. There we go. That's essentially all that we want for this application to do. So we can write that out and it should go ahead and actually implement that for us. Now, obviously AI takes a little while to write the code, especially depending on what models and so on you're using. I'm actually using
- 27:28
local AI for everything. So, it's even a little bit slower than if I was using like a Frontier model for all of this. So, while the AI is processing, you could scroll through Tik Tok on your phone or something like that. But, that's a very bad idea in my opinion. What I would like to do is other things in parallel to what the AI is doing. So, in our particular case, something we can do is to set up all of our API keys for GitHub and so on to work properly with our authentication system. Now, while I
- 27:50
was getting the GitHub stuff set up, I actually noticed it asked me a bunch of questions. So, let's just go through and answer these questions. Only GitHub. We don't care about any other providers. Email password. No. Email / password. Email verification. No. Let's see here. Additional features. No.
- 28:10
We want off pages. No, we don't care about that. We just want it to be as simple as humanly possible. And for the UI style, what style do you want for the off pages? Minimal. Yeah, we'll just say that we want m well there is none since we have no off pages. There are no off pages. There we go. That gives it all the information. Essentially, I want it to be as minimal as possible. Now, with that, we can actually go ahead and start working on
- 28:31
our GitHub application. Now, for us, we're just going to be using a simple OOTH app because all we care about is authenticating. You can see I actually have quite a few of these already set up. So, let's go ahead and we're going to create a brand new OOTH app. This is going to be our video blog suggest. This is the YouTube version right here. And we're going to create a separate one for development as well as production to keep our systems entirely separate from
- 28:51
each other. So I'm going to call this one our development environment. And I might even just put that in all caps so it's very clear which one is dev and which one is production. Now for our homepage URL, this is just going to be that localhost URL. So we'll just say localhost 3000. And for our callback URL, we're going to use the exact same localhost 3000 as our callback URL. And that should be all we we need to do. So
- 29:12
let's just go ahead and we're going to register our application. And that gets our application set up and we can use the client ID and client secret. So let's copy over that client ID and we want to put that directly in our env. And it looks like our AI asked us something. We'll get to that in just a second. Let's just go to our off section. And we have our GitHub client ID. Paste that in. GitHub client secret.
- 29:34
And let's copy that over as well. So we can come over here, generate a brand new client secret. Get that information right there. And we're going to paste that in. And now let's actually look at what the AI is asking us to do. Before I proceed, you'll need to create a GitHub oath app if you haven't already. Blah blah blah blah. Um, do you have a go off credentials ready? Use the enviables.
- 29:55
And I'll paste in the name of those variables for the credentials. There we go. Hopefully that'll actually go ahead and set everything up. So this is our development version of the application, but we essentially need to create a brand new version of the application that is also going to be for production. So let's go back to our GitHub developer settings and I want to essentially create a brand new OOTH app.
- 30:16
This one is going to be video blog suggest for YouTube and this one is our production version. So we'll put prod at the end. For our homepage URL, we can actually go over to our first cell deployment of this. So if we just pull that over, you can see right here is my domain. So let's just copy that over. Obviously, you'd probably use a real domain in a real world project, but this
- 30:36
is the domain that we'll use. That is going to be our homepage URL. And our authorization authorization callback URL will be exactly the same as that. We can register that application. And again, now we have our client ID and our client secret. We'll just keep this off to the side. When we're ready to deploy this to production, we'll make sure that we add those different environment variables to our production instance as well. Now, let's go ahead and you can see that this
- 30:57
is still working on all of the different code for our application. And if we look at the actual sections that are being generated, it does look like we have some code being generated. We could start reviewing that if we want, but I generally like to wait for the AI to be finished before I start reviewing the code. So, for now, I'm just going to let this finish out because we pretty much have everything set up. Actually, we might as well since we have some time to kill. Let's go to Verscell. We're going to go and find our environment
- 31:17
variables, and I want to add some environment variables. So, let's go ahead and copy those values. We have our client ID. That is going to be our GitHub client ID. Make sure we paste in that value just like that. And we're going to add another one which is our GitHub client secret. Just like that. And again, let's go over to here and generate a brand new client secret. Copy
- 31:38
that value over as well. And we'll paste that in so that we at least have that set up completely done. We'll give that a quick save. We have those two. We also are going to need to have our environment variable for our database URL set up. Let's just again check our AI is still working on its project. So, let's just go ahead and pull over our neon. Let's go to that production branch of our database. And what we want to do is we want to get the connection for
- 31:59
this. So I believe if we just go to our dashboard, we can click connect right there. That's going to be our connection string. We can copy that over. And now we can go back into here, add an environment variable. This one is our database URL. And we can paste in our database connection string. That's another sensitive variable. And we'll save that. And now we at least have all three of those different environment variables in place to use in our
- 32:19
application when we're ready to deploy. Let's go ahead and you can see our AI is still working on this. I'm just going to come back to you as soon as this finishes. So, the AI just finished up with configuring everything inside of our application. It really didn't take much longer after I ended that last clip. So, let's go through and actually review the code. This is one of the most important steps is review all of the different code that is being changed. You can see right here, it's actually
- 32:39
adding some code to our drizzle config to point to a separate location for our schema for our off section. And that's because by default, the way that our off is being generated, it's not putting it inside of our schemas folder. It should be putting it in this folder, but it's not actually putting it there. That's something that we need to make sure we configure with better off to make sure it works like we expect it to. Now, inside this lib folder, it looks like we have our off and our off client. And
- 33:00
this is essentially where I want to configure everything. Also, we are using the old version of the drizzle adapter here, which is probably going to cause some errors inside this section of our application. So, we need to make sure we fix that as well. So, inside this off section, let's actually make sure we use the correct Drizzle adapter. This is going to be at slash better off slashdrizle adapter
- 33:22
slash relations v2. That's essentially going to work properly with the newest version of drizzle that we're using. Now to use this adapter, we need to actually be using the release candidate version of better off. So let's just go ahead and make sure we install that as well. So we can say npmi better off at rc and again make sure I put that force in there just to make
- 33:42
sure everything works properly. I should have installed the release candidate version to begin with, but I just forgot and didn't actually do that. So, once that's finished installing, it should hopefully have everything that we need inside of our application setup. And we can go through and make sure that all the rest of the code inside these sections looks like we want it to. Now, one immediate thing that I noticed is I like to use UU IDs for all my generations. And by default, it uses
- 34:02
just text IDs. So, we're actually going to change that. We're going to say in this advanced section, what I want to do is I want to get my database. I want to change the generate ID to UU ID. That's just again because I like to use UU IDs for every single thing that I do. And we can also pass along our schema inside of here. So we can say schema and just import our schema. Whoops. Make sure we
- 34:23
actually import that from the right location. Import star as schema from at /db/s schema. There we go. That's going to make sure it has access to all of our schema information to set everything up properly. I'm also going to slightly change how our off is set up. So we have this lib folder, which is great. I'm going to create a brand new O section inside of here. And inside this O section, I'm going to have this file be
- 34:44
called config. And then we can have our off client inside of here, which we'll just call client. This is just going to make our code a little cleaner. So, which we can put everything inside of these folders just like that. Other than that, all the code inside of here looks pretty good and we can keep this as is. Now, the next thing I want to do is I want to regenerate this off schema using all the new proper ways of doing things. So, we can essentially delete this
- 35:04
existing O schema. We can go into our package JSON. I'm going to create a brand new command called off generate schema. And this is going to be a command that uses that better off. So we'll say off generate. That's going to generate it. And we just want to say where our output is going to be. In our case, our output is going to be inside of this schemas folder. So I can honestly just copy this path. Make sure
- 35:25
it's relative. We can paste that down. And of course, we'll just want to swap the order of these. There we go. So it's going to go inside that schemas folder. Just like that. And let's just call this off.ts. That's going to be all of our off code. Then we can specify where our config is going to be, that's going to be in our slashsource/lib/off/config.ts.
- 35:46
And then if we just do d- yes, that's going to essentially force it to go through with the command no matter what. So now if I run npm run and let's see what we call that, offgenerate schema. That should hopefully regenerate our schema using all the brand new stuff for us. Now it looks like we don't have that O command installed. So let's just npmi. This is a dev dependency off and we want to get the release candidate version.
- 36:06
That's going to hopefully install the newest version of the command line tool for better off. And again, let's make sure we do that d-force to make sure that it works properly with drizzle kit cuz again it doesn't quite recognize the release candidate version of Drizzle. But once that goes through, it looks like that finished. We now should be able to run our original command of Ogenerate schema and that should generate a brand new O schema file inside the schemas folder that's
- 36:27
compatible with the newest version of Drizzle. And if we look inside of here, you can see it's using define relations parts, which is a brand new Drizzle feature. And it has all of the different code inside of here for our relationships right here, as well as all of our different schema files. We just need to make sure we use this off relations in our relations file here. So where we define our different relations, we can pass that along. And actually that goes right in here. So where we
- 36:48
have our relations, this is where we want to pass that along. So we can say our relations is a combination of all of our relations as well as all of those O relations as well. So let's just make sure that we import this author relations just like that. And now we have all of our different relations. So I can say db.query and you can see I get all of my different information. So for example, I get my account information and so on which is coming directly from
- 37:08
right here inside of Drizzle. So let's continue to review the code to make sure everything looks like we expect it to. And you'll actually notice I like to review the code before I check the implementation because if I check the implementation and it works, I tend to skip the code review or do less code review. So this forces me to actually go through a more thorough code review. So inside of our Drizzle config, we can actually get rid of all the changes because we now have our off schema in
- 37:30
the correct location. So let's discard all these different changes. And in our schemas file, which we have right here, let's just make sure we export star from that/s schema/off. There we go. And that's going to set up all of our off related code like we need. Of course, our package JSON had some updates, but we pretty much did all those manual. Our skills right here, those were all
- 37:50
generated by us. So we can skip past those. And now we can move on to our actual application itself. So our config, we've already double checked. We know that's fine. Our client, this looks fine as well. So we can keep that as is. We have this file here which is autogenerated. We don't have to mess with that. Our schema, we made that change ourself. Database, again, we made that change ourself. And then our server environment
- 38:11
variables, which is exactly what we want. That is great. Next, we can check essentially the page as well as the route to make sure this is all good. You can see here the route section. This is just basic better off code. This is correct. I've written enough Nex.js better off to know that this is exactly what you do. And now we have our page itself. And you can see here essentially all it's done is replace the code. So I'm actually just going to go to the page itself to view the code directly
- 38:33
rather than look at the diff because it essentially replaced everything inside this application. And this is actually a great place where it makes sense to actually just test the code yourself. So let's just come in here, make sure all these files are saved properly. I'm going to run this in development and let's just test to see what it looks like and if it works because we don't care too much about the UI at this point because this is more just a boilerplate
- 38:54
testing application. So let's just make sure we scroll this back and actually I'll just open this up in a brand new URL and you can see here this is opening up but it's giving us an error. Default migration lock table doesn't exist in target module. So there's definitely something going on interesting with our better off and my guess is it's because we haven't actually pushed our tables to production yet. So, let's just go ahead and we're going to make sure we push
- 39:14
those changes to our table. So, we can just go ahead and run npm run db push. And that's going to push all of those different tables directly for better off. And it's going to push it to our development server because that's what we're hosting on right here. And now, if I give this a refresh, it still doesn't look like it's quite working. So, we may need to change something else inside of our code. Let's just reset our application by npm runningdev again. And
- 39:36
you can see it's still giving us that warning. One thing that we can try is we can try importing the minimal version of better off since we only need GitHub authentication. This could be part of the reason that we're running into these problems. Let's go ahead and see if that worked. And if we pull this over, you can see welcome sign in with GitHub to continue. If I click sign in with GitHub, it's going to ask me to authorize. So, let's just go ahead and authorize that. We're being redirected
- 39:56
back. And it does look like it actually properly signed me in. And it has all of my information here. Now, we can go ahead and change this cuz we don't really care about the email of the user being shown at all. We should also make sure sign out works. So, let's just click sign out. And that does work like we expect it to. So, I can just go back to my AI and I can say, "Please do not show the email of the user."
- 40:19
There we go. That is all done. And now, let's just go ahead and make sure the rest of the different files that it modified are correct. And again, that's just that page file. So, I don't think there's really much we need to worry about here. The nice thing is a lot of this code in this page file is going to be overwritten or changed in the future. But if we just kind of glance at the actual code to make sure it's working, you can see it's using that signin social with GitHub. And that does look like that's all it's doing as well as
- 40:40
sign out. So it is working just like we expect. Now I do believe in the env section, we need to make sure we have a random string of characters here to work with. As well as I believe we got some warnings when we actually ran our dev environment. If we just go back to our dev environment, warning. Let's scroll back down to that. Better off base URL is not set. Set the base URL option or better off URL environment variable. So
- 41:00
we can set that environment variable to just a HTTP localhost 3000. Just like that. So now at least we have that environment variable set. And if we close out of that, that at least should get rid of that warning for us, which is great. Next, we want to make sure we have a better off secret. This is just again a random string of characters that we can generate. Now, we could do this
- 41:21
ourselves, but the nice thing is better off just has a system to do that. We can just click generate secret, and that'll generate us a brand new secret. So we can paste that directly into here. That's just going to give us a little bit of security with our better off implementation. And we can also generate a brand new secret. So let's just scroll back up to where that section was. And we'll generate a brand new secret. And this one is going to be for our production environment. So we'll copy
- 41:41
that over. And now we will use that inside of production. And inside of our application, let's add an environment variable. This one is going to have that value. And this is our better off secret, I believe, is what it is called. So let's just make sure we save that one. And then we also need our better off URL. So again, let's make sure we copy over what our domain is going to be. And we're going to paste that in as our environment variable. So we'll call
- 42:02
this one our better off URL. Just like that. This one's not sensitive, so we don't need to worry about the sensitivity on that. And we can just give that a click save. And now we have all of our different environment variables set for hopefully using this application. Now, kind of the final step since we have essentially a complete feature that we're done with working on, what I want to do is I actually want to generate the migration files so that we
- 42:23
can use this inside of our database. Because when we push to production, we're not going to be pushing our changes to our database. We're going to be migrating our database so that all of our data stays the same and doesn't get deleted or wiped. So, we can run the npm run db generate command and that's going to generate some migration files for us that we can run in production. So, if we
- 42:43
take a look inside of our DB, we have a now brand new migration file. And all this does is create all of those different tables for us. Now, immediately I see a problem. This test table probably shouldn't exist. It was just there for testing. Let's go ahead and remove the test table from here. We're going to go to our schema. We're going to remove our test table from there. That should clean up all of our test related stuff inside of our code. I then want to regenerate those
- 43:04
migrations. So, let's just delete the migrations that we generated. We're going to regenerate our migration. Should completely remove that test table from it. So, now if we look at our migration, you can see all this does is it just creates the account session user verification table as well as any different indexes and so on that need to be on those tables. So now when we push to production, we should hopefully be able to run this migration against our
- 43:24
database and it'll update our database to include this exact information so we can log in and so on and test everything inside of production. Now, there is one thing we're going to want to do on our actual production environment. When we do a deployment, we're going to want to slightly change how our deployments work. So we just want to go into our settings and we want to find our deployment settings. And you can see right here we have the build command. By
- 43:44
default, it's just going to run npm run build, which makes a lot of sense. But we also want to migrate our database. So let's actually override this particular command. What we want to do is we want to run the command for db migrate. That's going to migrate our database. And then we can run the npm run build command to actually build our actual environment. And then if we just give
- 44:05
that a quick save, that's going to be everything we need to do to make everything work. So now we're going to be pushing all of our migrations and migrating our database and then rolling out our code by building that and pushing that to production. That way our database and our production roll out happen at exactly the same time. So now if we just give that a quick save like we've done, we can come over to here and we can hopefully push this to our branch. So let's just add all of these
- 44:25
different changes and we'll just say get commit add GitHub off push those changes. Now we have that brand new branch. And if we go ahead and we actually check out that branch, we can create a pull request and so on. But before we do that, I actually want to create code rabbit support. So we can get pull request reviews directly inside
- 44:46
of that. So if we just log into code rabbit, we can come over here and click on this repositories button. This allows us to add a brand new repository. And what we want to do is we want to search for that repository in our GitHub that we just created to be able to add that into our application to work with. So you can see here if we scroll down, we want to select the repository that we're working with, which is video blog suggest.
- 45:06
And this is the YouTube version. So it's got -yt at the end of it. Whoops. Looks like that disappeared. YouTube or sorry video blog suggesttor YT. There we go. Click on that. And now we have access to that. We can give that a quick save. And now hopefully we should be able to see that show up in our repository list. Give it a second.
- 45:26
There we go. And we can click on that. This is a public repository. So again, we have all of those different supports and everything should pretty much just work for us out of the box. We don't really need to change anything at all. So now let's just actually go ahead and make sure that we pull and create that pull request. So right here we can see setup off. That's the branch we just created. We'll compare that and create a pull request. Give that a quick second. We can just call it whatever we want. We can add whatever description we want
- 45:47
based on as complicated or so on as we need. Since we're just working with oursel, we can keep this simple. And we'll just create a brand new rep request. Just like that. Give that a quick second. And now we have a pull request that we can go through. We can have other people on our team actually review and make sure all the code inside here looks good before we actually commit those changes directly to our repository. Now immediately looks like we are getting an error when we try to deploy. So let's go ahead and try to
- 46:07
debug what that is. We can click on that. You can see the command npm install is what actually failed. And you can see here it's just having issues with our installation because we have kind of two conflicting packages. It doesn't know which one to actually use. So we can change our settings for install to use that force command as well. So what to make sure we come over to here where we have our project settings install command and we can just say npmi-force.
- 46:29
That's going to essentially override that to hopefully work like we expect it to. So now what we can do is we can just kind of go back to that deploy where we were at and we should try to redeploy this just like that. And this should hopefully work properly just like we expect. And now we have essentially a deployment that we can access and use and view everything for it. Now that finished deploying so we can actually click on this and hopefully this is going to work like we expect as well.
- 46:50
Let's just try a quick signin with GitHub and see if this works. We may have some issues because our actual deployment URL does not match the GitHub URL. And you can actually see just like clicking this, it's actually not working. And I believe the entire reason for that is because this URL does not match the URL that we have inside of our environment variables. For example, if we go to our environment variables, you'll notice that this better off URL is specifically for our production
- 47:11
deployment and not the actual deployment here, which is like the deployment preview. So that's probably where we're having issues. Also, you'll notice that we have these environment variables saved for production and preview, which for a database URL is definitely not a good idea. This should be set to just production only. So you can see here this should be a productionon version of this environment variable. The reason for that is because this database URL is
- 47:33
our production database and we don't want to make sure we overwrite this inside of our preview environment. You could do the exact same thing with our GitHub stuff as well if you wanted. But for our case, we're just so focusing on this database URL cuz that's the main one we care about and we don't really care too much about these preview builds if we don't really want to. For now, I'm just going to skip this preview build and not really worry too much about it. Now, if we go back to our poll request,
- 47:53
we will notice if we scroll down that it's actually created a summary of us of all the different changes. And you can also see here that it's currently going through and reviewing all of these different changes. And once that's done, we will go through and actually look at all the different things from that review. So, our Code Rabbit review just finished. And one thing I do kind of want to mention I really like about Code Rabbit is this review change stack is really nice if you're reviewing the code
- 48:13
because it actually breaks it down into sections of where the different changes are. For example, it has all of our environment related changes. Right inside this section, we have a whole section for database schema changes inside here. So, we can really easily see what's changed inside of these different sections. Overall, it just makes it really easy for me to go, okay, these are the different groupings of changes inside of our section, which is nice for when you're doing a review, cuz
- 48:34
that's sometimes one of the hardest parts. Now, if we just go back to our actual application, we can see here all the different rest of the stuff related to our Code Rabbit review. Now, another nice thing is it has this whole walkthrough section essentially showing you everything that's changed and all the different stuff you need to know when you're reviewing. Again, it's nice because it makes it really easy for you to know what you want to look at when you're doing the actual review. Now, if
- 48:54
we look at Code Rabbit's actual comments, you can see there's really not much that it had to change with this review. You can see there was three nitpick comments. You can see here there's a security and privacy potential recurring pass the validated secret explicitly instead of relying on better off ENV fallback. Again, this is essentially saying to use the value directly instead of relying on that better off secret having this exact
- 49:14
specific name. The nice thing is it gives us a really good suggested fix right here. Tells us exactly what we need to change. And I think this is a good change to make and also even has a prompt for the AI if you want to have your AI do this. But for this one, since it's a single line, let's just go ahead and change this ourselves. We'll just come into here. We'll go to our application. We'll find that exact location which was in our off configuration. So, right here. And we
- 49:34
just want to make sure we pass along that secret information. So we can just paste that down just like that. And now it's going to be properly using that information instead of relying on the naming being exactly the same convention. Let's go ahead now that we've got that one done. Let's look at the next one which is our server env. Consider enforcing a minimum secret length. This is validated with a string. So say that it must be at least 32
- 49:55
characters. That is again another really good requirement. And again that's super easy to implement. We can just copy that code over. And let's go ahead to our environment variable and do just that. So our server env we should make sure that this is at least 32 long because that is the recommendations by better off. Next we can minimize that down and we can go to our page. You can see here consider next.js or next image for the
- 50:15
avatar instead of using the raw image. That I agree is probably something but again it's not super important that we work on that and that's because that we're not going to be keeping this exact UI around forever. So for that I'm just going to completely skip that one. But those other two changes I do think were good things to change. And I will actually just go ahead make sure I save all my code and we'll just add those changes. Get add. We'll say get commit minor fixes.
- 50:40
And we'll just push that up. Of course, use better commit names than that. But for video purpose, I think that's perfectly okay. That's going to push up those different changes. And since we know everything else is good for this particular project, we should be able to actually merge this and hopefully everything is going to work just like we expect. So, let's just merge this particular pull request. That's going to push it into our main branch. We can delete our original branch because we no longer need that anymore. And now inside
- 51:02
of here, if we go to our deployment section, we should see a brand new deployment for production that's being queued up just like we expect. This minor fixes one we don't really care about. We can ignore that one. It aired out, but hopefully that's not a problem. Oh, it's ering out probably because it can't migrate to the database because we don't actually have a URL for that database at all to be pushed to. That makes sense. So, that's perfectly fine. This one should actually work though
- 51:22
because we do have that database URL. And again, the only reason a production or the previews don't work is because we made it so our database URL is only production. You would want to create a brand new environment for your preview and save that and do that. But for our case, we have just a production version. So, it's only going to work in our production deployments. So, I'll come back to you once this finishes deploying to make sure everything works. That just finished deploying. So, let's go ahead
- 51:43
and test to make sure everything works. We click sign in with GitHub. And that should actually sign us in. Let's just make sure we refresh. Looks like it's still not properly signing us in. It might be because we're using the wrong URL. I believe if we go to our overview and we click on the URL here, this is our actual full production URL. Now, if I sign in with GitHub, that should work properly, but it looks like it's still not quite working like I expected to. Oh, there we go. It was just a little
- 52:03
bit of a delay from slow internet. Now, if I click authorize, it's going to redirect us back. It shows the name. It shows the picture just like I expect. And when I click sign out, you can see it signs me out. So, everything does work as I expect, which is perfect. So, now that we've got all of our O set up, that's kind of like the boilerplate of getting our code in the place we want it. Now we can actually work on some of the more exciting things. We can at least look at our project list to see
- 52:24
how far we are. So currently we have step one entirely done. We can now log in with GitHub, which is great. And our UI doesn't have anything yet, but that's okay cuz we don't have anything to display on that UI. The next step is we want to be able to let the user search for a topic. But to do that, we need to ingest all of the content. So we're going to be using those Verscell workflows to ingest all of our data. We're going to chunk it. We're going to
- 52:44
embed it inside of vector embeddings using AI for all this embedding. And this is really the more complicated part of our application. All of this rag implementation that's going to be happening directly in this ingest step right here. And we need to ingest both blog articles and YouTube videos. And they're going to have some overlap but also some differences between them. So I want to make sure that we handle all of that properly. So in our case, we're
- 53:04
going to be setting up an ingestion system. And we're going to start with our blog articles first cuz it's going to be a little bit easier to work with because there's a lot less of them and they're a little bit more simple. And then we're going to move on to YouTube videos after that because we need to deal with the YouTube API which can be quite complicated and cumbersome to work with. And this is a perfect task for us to send off to the AI to at least get it started working on this implementation. So what we can do is we can go into our
- 53:26
application and we can just create a brand new session. So I say / new just like that. And this is where I want to be as detailed as possible explaining exactly what I want the AI to do. And I'm just going to write out what it is and I'll read it to you after I'm done. Now I just finished writing out everything. It's a very detailed prompt. I'll kind of give you the high level. Essentially, I said that I wanted to use Verscel workflows to ingest the blog
- 53:46
articles from and I pasted the URL to my blog. And I also mentioned that all the articles are listed in an RSS feed and I posted the URL for that as well. I mentioned that I want the workflow to be specifically two steps where the very first step is to just get all of the articles that are not currently in the database. Then what I wanted to do is to loop through each article and run step two. And I want to batch these requests.
- 54:07
So instead of like let's say I have 500 articles instead of sending 500 requests to my blog all at the same time it's going to batch them in sets of 10 just so I don't do myself. Then for step two this is where I wanted to run for each article I wanted to fetch the HTML. I specifically told it where the HTML for the blog content lives. So I mentioned that's in the main element. I said that it should also get the OG image and use
- 54:29
that as a thumbnail. And then the HTML content should be chunked by H2. So essentially what I wanted to do is for every H2 heading, I want to get all the content under that heading and put that into one individual chunk. That way each chunk is stored in a hierarchical way where it's essentially better for the AI when it goes through and embeds information. It'll be much better at retrieving the proper information. When
- 54:49
you're doing this kind of embedding and chunking for a rag system, you want to try to chunk your data in ways that it is contextually all kind of the same piece of data. So if I'm chunking by H2, I generally will have a lot of the same data in each H2. And since my blog article has lots of H2s with not too much content in them, my chunks are never going to be massively large. So it should hopefully give me good chunk
- 55:09
sizes as well as a concrete set of data that's all kind of related in each chunk. So inside of there, I told it what to do and I also told it to use the actual text with no HTML markup. And I also kind of gave it some information on some edge cases as well. I also specifically said do not do any of the embedding. Embedding is kind of a whole another beast where we need AI and we need to embed it and then save that in
- 55:30
our database. I want to handle that in a separate task. So I essentially told it not to do that at all. I then gave it a list of what I want my database to look like. I essentially said all of the different tables and kind of the different keys and things for my tables. So I just listed out what my table for my chunks as well as for my content are going to look like. And it should go through and create all that information. Now with that done, we should just be able to click enter and let the AI
- 55:50
actually run and prompt through this. And this is going to be a prompt that is going to maybe take a little bit longer because it's quite a bit more involved. And since again I'm running this locally, it's going to be slower than if I was running like a frontier model for example. So while this is running, I'm actually just going to go get lunch myself and then come back and we can take a look at the code. And this is a perfect case for where you have a longunning AI task. Go ahead and do something else in your life while that longunning task is working. I will kind
- 56:12
of sit here for just a few seconds just to make sure it doesn't ask me any questions, but then I'm going to go ahead and make lunch and eat that and come back when it's done. So I just finished lunch. It looks like everything here is finished. So now let's go through and actually review the code. And I actually want to do a quick review and then I want to do a more in-depth review. And that's because I want to spawn a process for creating tests for this as well since this is one of the most important parts of our application.
- 56:34
So I just want to do a really quick review to make sure that the structure looks correct. And then I'll have it write test and then if I need to make more sweeping changes, I can make those and I'll have test to fall back on to make sure everything still works like I expect it to. So just go ahead and again make sure we have just those sweeping changes still in the correct place. So we have ingest blog articles. You can see here it's got some like type stuff that I'll probably get rid of and so on. But it does look like it's using a
- 56:55
workflow. And if we look, we should hopefully have some steps for those. We have a step here for fetching the RSS feed. We have a step here that's going to be saving the article. And that should be the two steps that we have. And it looks like that is correct. And then we have all of our chunking code down here. So it does actually look like for the most part that this code is correct. I'll probably split some things out and structure it slightly differently, but at least we have a
- 57:16
starting point right here for all of our different code. So I will actually prompt it to create some test for this particular ingesting process. I think I specifically care about checking on how the actual chunking process works. That's the thing I care the most about. So what I might have it do is actually swap out that chunking myself real quick. So you can see here we have this chunk by H2 function that does all of our chunking. That's kind of the thing I
- 57:37
care about testing the most because the rest of it is just like fetching HTML and so on. So let's go ahead and make it so that that is the thing that gets tested. So I'm going to take this entire chunking section. and I'm going to bring it into its brand new own file. So, inside of our app, we're going to have this lib folder. And we'll create a folder for chunking. And we'll come in here with a new file. We'll call this one chunk articles.ts.
- 58:00
Paste that down. And we'll call this chunk articles. There we go. Want to make sure that I export that. And of course, we need to import the different libraries that it's using. For example, this Cheerio library that it's using, we want to make sure we get imported. So, let's just come all the way up here and we can copy that import in and we'll paste that down to
- 58:20
the top. Of course, it looks like it didn't want to copy. Let's try that again. Copy that over. Paste that down. There we go. And it looks like we have a few other types that we need to have inside here. Element type. We'll get that fixed. So, copy that over.
- 58:42
And there we go. And then we just have some NE's that obviously we'll clean up later, but at least we're able to get rid of that import. So now let's go ahead and just ask it to create those tests for us. Create. And actually, I'll probably do it in a new session. There we go. Create test for. And we called this chunk articles. Chunk articles. There we go.
- 59:04
Perfect. So now let's go ahead and actually read through the code. I'm going to save this ingest file to last because this is the most complicated. I want to make sure all the auxiliary pieces are where we want them to be. So, we can look in the next config, you can see it added with workflow, which is just going to make everything with workflows work properly. In our package JSON, we have those different libraries that we're using for parsing our HTML and so on. Our
- 59:25
TypeScript config is including that workflow section again to give us more type safety. Our routes here, it's just created a route for us for ingesting the article. And I actually noticed that it's at this /ingest route. I actually want to change that. I want this to be API workflows ingest blog articles. There we go. That way this is
- 59:46
specifically for our blog article being ingested. Just like that. Now let's go back into here. We have our database. And you can see that it's adding our content relations as its own kind of schema thing. I don't really like how that was structured. So I'm going to change how that is structured. Let's go to that content relations. This I don't want to be a relation part. I think I want it to be directly in our relation.
- 1:00:06
And you'll also notice chunks and content are in the exact same file. I want these to be in separate files. So let's go ahead and change that as well. So inside of our schema section, we're going to create a brand new file. This one is going to be called chunks. That's where all of our chunk related content is going to go. So this entire table here that's going to go in our chunks section. And we'll fix all the
- 1:00:26
different typing and stuff. So on we also want to make sure our relationships here are going to go in our relations file. So let's just copy all of this. There we go. go into our relations and we should just be able to paste that down and then everything should work properly which it looks like it is. That's perfect. We can get rid of this entire content relations and go into our database. Get rid of that here and here
- 1:00:47
and still everything else should work properly. Now we just want to make sure that all of our tables have the content we want. So this table is just going to be our content section. So let's make sure that this works like we want. I want this to be snake case. So I'm going to use that snake case table type. Just like that. Let's make sure that we import this. There we go.
- 1:01:08
Don't need this import since we're not using it. And now here for our UU ID, I'm actually going to create a helper file. So in our database, I'm just going to create a utils.ts export const ID. And I'm going to create my own ID type. So we're going to say UU ID coming from Postgress. Just like that. I want that to be a default random. And that should be all we need for our ID. That way anywhere that I
- 1:01:30
have an ID, I can essentially just use ID and import that directly from our utils. And now we have access to that. And I want to do the same exact thing inside of here. So let's make sure that we import ID directly from our utils. And now we have that information. Now I don't actually need to specify the name of our tables. This is all handled
- 1:01:50
automatically for us. So I'm going to get rid of these just to clean up our code quite a bit because that snake case is taking care of all the naming for us. Just like that. There we go. And now what I want to do is I want to clean up this created at and updated at section. Again, I'm going to be using these in many places. So, let's create variables for them. Export const created at is equal to I want this to be a timestamp.
- 1:02:12
Make sure that comes from drizzle. And I specifically want this to be with time zone. We'll set that to true. There we go. Default. Now, that's perfect. I can do the exact same thing for updated at. The only difference is it's going to have an on update at the end of it. that is essentially going to force it to be the current date every single time we do an update. So now I have my timestamps essentially right here and I'm actually
- 1:02:33
going to export a constant variable called timestamps and I'll set it equal to those variables. So what I can do I'll just copy this over, clean this up slightly. There we go. And there we go. Put a comma right there. And now my timestamps have all that information. So I can go into my content everywhere I
- 1:02:54
have my timestamps. I can just add that in like that. And if I do a quick little import of my timestamps up here, you should see that cleans that up. And I'll do the exact same thing for my chunk section. Again, I can get rid of it right there. And I can make sure that I just import the timestamps just like that and import them there. Okay. So now let's actually move to make sure that
- 1:03:15
all the content inside of here is what I want it to be. So we have our title, which is a notnull text. Description not null text. Publish date, that's a time stamp, which is a date, not null. I actually want this to be a date type. So, let's change that to the date type. Just like that. And for this, I want the mode to be date. That way, it shows up as a date instead of a string. Not null. URL is going to be not null. Unique.
- 1:03:36
Thumbnail is going to be text. And I want this to be not null as well. Content type, that's not null. And then finally, our content itself is not null. So, it looks like this content table is everything that I want. And we also have our video and article types. That's correct. Let's go over to our chunks table and make sure that this is what I want as well. So we're going to have our UU ID type. Let's make sure we import that from Postgress. We're going to have
- 1:03:58
our integer type which is getting imported. Our vector type which is again coming from that same exact import. And finally our text type which again is coming in. And lastly we're creating an index just like that. So that has all of our content there. I want this to be snakecase.t. And then of course we can get rid of all these names that they added in there
- 1:04:19
because again these are automatically generated with the proper encoding. There we go. Let's make sure we import our content table here so we can reference the ID. Make sure our ondelete is proper. This is not null. So that's perfect and it's a foreign key. Our text here is not null. Our vector right now is null because we're not doing any embedding. But eventually we'll change this to not null. So, I'll just add a to-do
- 1:04:40
change to not null when implementing embedding. There we go. And then we have our start position, which can be null because this is only useful for videos. Essentially telling us where to start in that particular video. So, that is all that right there. That's all we should need to do. We have our indexing and everything. So, that looks like it's proper. Now, we should check on our AI
- 1:05:01
to see how it's doing with our testing. And it looks like it's finished up. And it just says to actually set this up, we just need to install vest. So let's just go ahead and install vest, which is the testing library I wanted to use. I should have specified that I wanted to use vest, but it is luckily defaulted to that. And then we can just run what was it? npm test. And we should hopefully see that everything is working. And let me make sure in the package json it
- 1:05:21
actually modified it to include a test script. It did actually include that and a watch script as well. So that should hopefully go through and it will run our test to make sure everything passes. And it does look like all of our tests are passing. So at least it works like we expect. We'll read through these tests to make sure they're good in a little bit, but for now, we at least have a base point where if we change our code and things fail, we now at least know that we have some tests to fix that. So,
- 1:05:42
we have all of this is complete. Package JSON just has those new configs as well as vest as a dependency. Again, we've already looked at the TypeScript config. There's all the chunk article code we'll look at in a little bit. This route seems fine. This is how you supposed to manage workflows. You just call start, pass it in your workflow, and that's going to run your workflow for you with all the information that you could possibly need. Now, inside of our
- 1:06:03
relations, let's take a look at this code because it looks like there are some errors in it now. Probably just some import errors because we moved some things around. So, let's just take a look at this. Chunks does not exist. And that is most likely because inside of here, I need to export chunks just like that. There we go. That cleaned up the errors inside this file. So, this file is completely done. Our schema file is done. Our utils file that we created
- 1:06:24
ourselves. So, we can ignore. And otherwise, all those other files are correct for our DB schemas. We've taken a look at both of those. our migration file. Looks like it created a migration file for us. I'm actually going to delete this because we changed how our tables are set up. So, we're going to recreate that migration file once we're done with everything related to embedding. So, that's just deleted for now. Let's continue onward. We have our chunk article and our ingest blog
- 1:06:46
articles. So, these are kind of the two next sections we want to look at. Let's take a look at our ingesting of the blog articles first to see how that works. Let's go ahead and remove those unused imports. And we're going to fix the chunks import as well. Make sure it imports from the correct location. There we go. So, we have some types here. Honestly, I'm hoping to be able to get rid of these types and just do everything automatically. We have our ingest blog articles workflow. That's
- 1:07:07
perfect. So, step one is fetching all the RSS feed and finding the articles. That is what we want. So, that is a good first step. Let's go ahead and actually check out that fetch RSS feed to see what it looks like. It looks like it actually is returning us a promise of that particular RSS item type. It has used step, which is correct. And now we can see it's fetching that response. If there is an error, it's throwing an error. Now, that's perfectly okay if
- 1:07:28
it's having a problem fetching. We want to throw an error and then hopefully retry maybe when the URL is back up and running properly. Next, you can see that it's trying to get the response text as XML and then it's trying to parse that by getting a new parser, which I believe comes from RSS parser. Yes, it does. So, that is what we wanted. And let's go back to where we were on that. So, fetch RSS feed. There we go. So, it's parsing
- 1:07:49
out all of our different results. It's casting them with an as type right here. That is actually not something that I like. I really don't like as casting unless you absolutely need it and in our case I don't want that and we don't need that. So you can see here we're now getting our result with all of our different information. So if you say result dot let's see if that comes up you can see here now we can click items and that's going to give us our items
- 1:08:10
and if we get the first one we can get all the information for our item. Now in our particular case we have a description as part of our RSS feed. If we actually just open up this RSS feed URL we'll actually pull this over. I'll zoom this in. You can see we have a title, a description or sorry for each item we have a title, we have a link and we have a description as well as a pub date. Those are the information we want.
- 1:08:30
Now this description is not like a default field that's got by this parser. So we want to just make sure we add that into our parser. So while we're generating a new parser, we can add in our own custom fields and we can say that we want to have that custom field of description. Now this custom field description is going to be on our item. So let's just go ahead and specify it like that. And then we'll say that we want our description. There we go. That's going to give us that description
- 1:08:51
field. And now you can see here we have access to that description property because it's coming in as a custom field. Now we can get rid of this hard-coded return type. I generally don't like to use hard-coded return types unless I absolutely need to. And let's see what it does after it fetches RSS feeds. So now it's going to get all the existing URLs from the database and filter to only new articles. This is something I want to do inside that step.
- 1:09:11
So I'm going to take all of this information. I'm going to go into my fetch RSS feed step. And I'm going to paste all that in there. And we're going to change around the code a little bit to make it work. But I'm going to rename this to something like get new articles from RSS feed. There we go. And we'll come up here. Make sure we call that. That way it's just going to get us the brand new articles. And we can just say something like new articles. There we
- 1:09:32
go. That's the brand new articles and only the new articles. That way if we run this in the future, it's only going to get the articles that are posted that are new and not already in our database because we don't want to chunk stuff that we already have chunked. That would be very costly and it's going to be very prohibitive. So now here we have our results just like that. And from our results, I want to get the items specifically. So, we can get our items just like that. Now, another thing we
- 1:09:52
can do to actually clean this up is that the parser actually has a parse URL function. And we can just pass it in that RSS URL. And we can essentially get rid of all this code up here for manually fetching that URL. This just does it for us, which again makes it a little bit easier for us to work with. And then we want to again get those items property directly from there. Now, for getting our existing URLs, we can actually make this a little bit better
- 1:10:13
and more performant as well. We can do a simple DB query right here. And I want to query on that content table. I specifically want to come into this query with a wear clause. And to make sure this works, I actually want to do a find many. There we go. That's going to find many results. And now inside my wear clause, there's a type property. I want to make sure that this is article. So first, I'm only getting the articles that I specifically need. I can also
- 1:10:34
specify what columns I want. And in my case, I want just the URL column. So I'll set that to true because that's what I'm using to determine whether or not this is a new article or not. Then all I want to do with this content is I essentially want to get just the existing URLs. So I can just say then that's going to give me my data and I can say data map and I want to take in each result and I want to get
- 1:10:56
just the URL property. So now this is my URLs that are existing. So we'll say existing URLs clean that up and we can just check to see if our existing URLs includes our item whoops item.link link and if it doesn't include it, we want to make sure we filter to just the ones that are not included. So, this is only going to
- 1:11:17
include articles that don't already exist inside of our list. Now, here is where we can just return this result because that gives us essentially just our new items. And this section right here for our new articles, we can make sure we come up here and if, for example, our new article length is zero, we can just essentially return that we processed absolutely nothing because there were no new articles and we don't need to worry about logging that data.
- 1:11:37
So now we've cleaned this up drastically and this code is much easier to read and more performant which is great. So we've come in here, we've created our new articles. That's our step one. Next thing that we're doing is we're going to process in batches of 10. So we have our batch size set to 10 and we're going through a loop and processing all of this information. Now this batching implementation is really not bad, but there's a few different things I want to change about it. First of all, we should
- 1:11:58
be using all settled here instead of all. The reason for that is because if we use promise.all, if any of them fail, all of them will fail. So, this just makes sure that they all work properly and it's only going to fail each individual one and let the ones that succeed actually work properly. Also, we're going to be doing batching in multiple places. For example, when we ingest YouTube videos. So, I actually want to create this batching in its own
- 1:12:18
file. I want to create a brand new file. We'll put it inside of a util section and we'll just call batch ze or something along those lines. And here I want to essentially write our own batching implementation. So, we'll just kind of copy over that loop just like that. We're going to export a function called batch exec. This is going to be a generic function because we're going to have a generic return type for this. And that's because it takes in essentially
- 1:12:40
an array of items just like that of type T as well as a function that it's going to run for each individual item. So we're going to call this a batch with type of T. And it's going to return to us a promise just like that. And this promise doesn't have a return type. Doesn't really matter what it returns. We don't care at all for this particular use case. And
- 1:13:00
now we want to do that batching. And we also want to be able to configure what the batch size and so on looks like. So we'll set a batch size by default of 10 and a delay in milliseconds of zero. So we can also delay this if we need to. And by default we'll set that to an empty object. There we go. So now we essentially can specify how we want to batch this and how long we want to wait in between our batches as well. Now also
- 1:13:22
inside of this batching I want to take account of which ones have succeeded and not. So we can say total succeeded is equal to zero and we'll just count this up and we can also get our total failed and we'll count that up from zero as well. So we can keep track of what is successful and what is not successful. And then here we can just return a total which is our items.length. We can return
- 1:13:43
our succeeded which is going to be our total succeeded. And we can return our failed which is our total failed. Just like that. If I spell everything properly, this allows us to keep track of how many things were successful or not. So we can easily look at our workflow and see where our failures were. Now is where we actually need to do our looping. And so far, this code is actually really good. So we need to take our items. We need to
- 1:14:04
loop through them for the actual batch size that we're working on. And we essentially need to create a batch from those items. All of that code is perfect and just like what we want. Next, we can get the results of those by calling outcomes, which is await promise. And we want to again use that all settled to make sure that they all execute and finish properly. And then we can just call batch.mmap and pass in our
- 1:14:25
function. So this is going to call that function for every single element inside of our array and wait for it to finish properly. And this is an async function since we're using async await just like that. That'll give us all of our outcomes. And then inside of our outcomes, we can do whatever we want. So all of this code inside here, we can essentially completely get rid of it. And I want to just loop through my outcomes. So for each outcome
- 1:14:46
I want to run the code inside of here and if our outcome status is equal to fulfilled well that means it was successful. So we can take our total succeeded add one to it otherwise what we can do is we can take our total failed and we can add one to that instead. This is just going to properly make sure we keep track of our failures
- 1:15:07
and our successes. Then since we have this delay we just need to add in a section here to handle that delay. So if our delay is greater than zero and we have more things to still batch on. So if our i plus batch size is less than our items.length then that means we still have more things to loop through and batch on. So what we can do is we can just call await
- 1:15:28
we can pass it in sleep and we can pass it in our delay of milliseconds and this sleep function comes from our workflow and it'll essentially just wait until this sleep duration is done and then it's going to run our code. So this just makes sure that we have the ability to slow down our execution in case we hit like API rate limits or something along those lines. So now we can use this function in multiple places. It gives us a really good set of results that it's
- 1:15:48
going to return to us. And it just overall is going to be a better experience than what they have written out here which is a one-off scenario. So we can call batch exec just like that. We can pass it in our new articles and then we have a function that is going to take in those new articles and we can just call ingest article just like that. And that's the step that's going to take that article in. Right now, this step
- 1:16:08
only takes in a link, but we'll make it take in all the information from our new article. Just like that. We also can get rid of this check right here because if our length of our articles is zero, well, it's automatically going to make sure that it doesn't return anything. So, you can see here if this is zero, it's just going to skip all of this and just return 0 0, which is exactly what we want. So, now we can work on this ingest article step, which right now
- 1:16:29
takes in a URL, but I want to take in essentially the entire item that gets returned by our parser. So if we look at the type for this, you can see we get this like parser do item type. So let's make sure that we use that as our type here. So we can just say that this is our feed item and that is going to take in our parser item type I believe. And if we look at that you can see feed item is declared. So it has that proper type.
- 1:16:50
Now, inside of here, since I have the full feed item instead of just the URL like they passed in before, I want to essentially validate that all the data I expect to be on this item is actually there. Cuz unfortunately, this code is not great when it comes to getting information. For example, it's getting the title and the description directly from the HTML as well as this publish time. It would be much better if it just got that from the RSS feed because that
- 1:17:12
is the source of truth in my opinion and this is just a hack trying to get the correct information. So what we want to do is get all that information from our feed. And to do that, we can very easily just come all the way up to the top here and we want to make sure that we create a schema. So we'll say const feed item schema is equal to z.object. Make sure we use zod for that.
- 1:17:34
Perfect. And then we can come in here with a title which is going to be a string. There we go. We can make sure that it has a description. Whoops. Which is zring. Copy this down a couple times because we
- 1:17:55
also have a link which is going to be a URL. And then finally we have our publish date which is called pub date and that we are going to coersse into a date because essentially it returns itself as a string. So this is going to make sure all that data we expect is inside of our feed item. So now for each one of these what we can do is not this section actually this section we can try to get our data and our success.
- 1:18:16
So let's just say data for now is equal to parse and we want to parse this safely just like that. And then we can pass it in our feed item. There we go. That's going to give us this data right here. So let's make sure that we get our success and our data. And we can actually call this item. And instead of the success I actually care more about
- 1:18:37
the error. So, if we have an error, well, then we know that we want to return some type of error. So, I'm actually going to throw a fatal error. A fatal error essentially just means stop retrying. And this means that we didn't have the right data in our RSS feed. It's never going to be correct no matter how many times we retry. So, I'm just going to give it a nice little error. Failed to process RSS feed item. And
- 1:18:58
then I'll put in the URL or error message just like that. And I will also put in the URL. So, let's come in here. feed item.link. There we go. That's all the information specifically for that section. And now we can go through and use this item. So here for our URL, let's just pass it in the link. It's going to fetch all that
- 1:19:19
data. And if there was not able to get a response, we'll make sure that it essentially says, hey, failed to fetch that particular data. Then we're going to get the HTML. We're going to use Cheerio to try to get the information directly. Our article main, I don't think they're always nested directly. So, I'm going to get rid of that section right there because it's going to make sure it works properly, even if they're not fully nested. And here, I can just say HTML directly. And that'll give us
- 1:19:41
our main HTML. Just like that. There we go. And if for some reason we don't have any main HTML, that's a that's an error. We need to address that. So, we can say if main HTML is equal to null or does not exist in some way, then we want to make sure that shows up as an error. And this is an unreoverable error because it means our format was not like we expected it
- 1:20:02
to. So we'll just come in here and we'll throw that exact same error. We'll just say failed to load HTML from our item URL. Just like that. There we go. Make sure I get that proper. It is item.link. So there we go. The next thing we need to do is to actually get the image properly so we can get our thumbnail image. I'm actually going to
- 1:20:23
move that up here with the main HTML section. There we go. So that's going to be getting our thumbnail URL. And again, if this is equal to null, then we also were failing to get the information properly because these are the only two things we need to get from the HTML. Nothing else matters. This title, this description, this publish date, all of that already comes directly
- 1:20:45
from this item right here, which was our feed from our RSS. There's no point in getting that twice. So now that we have that done, again, our publish date, we already have that done. Here's our chunking. I believe I renamed this to chunk articles. So, let's just pass that in. There we go. Get rid of some of these comments since we don't need them. And now we can pass along our information. For example, our title comes from our item.title. Our description comes from our
- 1:21:06
item.cription. Publish date is our item.pubdate. And the URL is item.link. Our thumbnail URL type and content are all there. I'm actually going to move type to the very top. So, that is exactly what we want. And on conflict, I essentially don't want to do anything because there should be no conflict. So we'll just say on conflict do nothing because again there should be no conflict at all. We can
- 1:21:27
then just do a quick check here if our content row ID is equal to null and this could be null just like that. There we go. If that is equal to null then we're going to throw another fatal error because essentially that means for some reason we inserted something that already existed or we had some type of error. And actually I'm not even going to y'all make it a fatal error because that means we inserted something when it
- 1:21:48
definitely should not have been inserted. And we're just going to say something like duplicate detected and failed to insert. There we go. And we'll put our item link just so we know which one the failure was for. And I think if I make this just double equal sign right here, that should fix that particular problem. So it says content row ID. I think it's because this is a
- 1:22:10
string or null. But we should be checking the null aspect of our content row right here. Content row ID is string or null. It definitely shouldn't be string or null. So, there may be an issue with my particular SQL code. So, let me go to my utils. I think this needs to say not null. There we go. That'll clean that up because obviously it can't be null. And that does fix that
- 1:22:30
error right here. And then we can say it ingested the article. Actually, we don't even need any logs. Let's just get rid of those logs. Get rid of that comment. And that did clean this up drastically. So, at least this particular section for all of our workflow stuff, I think, is done. We can kind of get rid of some of these comments here. We already are doing our batch exec. So we can get rid of all of this and just return the result of awaiting the batching process.
- 1:22:52
There we go. That's going to return everything that we need from our articles. We also don't need this type anymore, which is great. So we can completely remove that type. And then this code is much cleaner than what it was before. This is why I always read code, especially something that's as important as the core business logic of your entire application. Read it and rewrite it in a way that is much cleaner and easier to work with. Cuz the previous code was errorprone, difficult
- 1:23:13
to read, and not super extendable. This is a much better formatted code in my opinion. Now the next and final thing that we probably need to look at is to make sure our chunking is working properly because that is kind of the core business logic of this entire application. So let's go into our lib into our chunking section and make sure all the code inside here works just like we expect it to cuz I mean there's like anyies and stuff inside here which I
- 1:23:33
obviously hate and we want to try to remove that if possible. Now one thing I notice immediately is it looks like it's reparsing our article main. We're kind of putting the same code in multiple places. I don't actually like that. I'm going to try to clean that up. Let's just close out of everything except for this. Make sure we don't worry about that. Okay, perfect. Now, I want to go back into the file where our workflow was and find where we were calling that particular function, which is right
- 1:23:56
here. Cheerio load. Where's our chunk? Chunk articles. We're passing in the main HTML. Well, the main HTML is just the article main content. So, we don't need to research for that article main content, which obviously doesn't exist because this HTML already contains that content. So, what I want to do here when I load my HTML is I want to load it. I'm going to pass in null and I'm going to pass in false here. So by passing in
- 1:24:17
null that's just essentially skipping this parameter and false right here. This is being passed in if we just hover over this load right here is document right here. This is essentially saying don't load this with a head and a body and so on. It is just essentially a fragment of HTML. So load it just as that particular fragment. So that's what I'm essentially telling the code to do right here. And this main content we don't even need to worry about anymore
- 1:24:38
at all. And we can just call this something else. We'll call it main because that is what the main content actually was being properly used for. And actually load we'll still call dollar sign but instead actually we want to get our root element which they called dollar sign main. We'll keep it the same name and that is just going to be calling root directly on this just like that. That's going to actually give us essentially the full entire HTML
- 1:24:58
document but in a way that can be processed by Cheerio which is essentially like a fancy jQuery version. Now if we take a look at the code right here you can see it's getting all of the H2 elements that are within that main which is perfect. It's then going through if there's no HTML element, it's just returning all of the text as a single chunk. So that looks like that is working properly. Next, you can see it's getting the text before the first H2. So it's getting our first H2. Then it's
- 1:25:21
going through and trying to get the contents and filtering only the text types. This code right here is way more complicated and convoluted than it needs to be. We can actually clean this up drastically to make it look more like this code right here by just getting the text property directly. So, I think what we want to do is we want to just create a simple loop. I like some of this code, but this is really, really complicated.
- 1:25:43
Much more complicated than it needs to be, especially cuz it's like going through and it's copying the same code in multiple different places. Let's clean this up drastically. So, let's go ahead at the very top here where we have our main element. What I want to do is I want to just loop through every single one of our main elements. So, we can come in here, main, I want to get the children specifically, and I want to just loop through each one. I'm not caring about H2s or anything like that.
- 1:26:04
I'm just going to loop through each child. And if I do find an H2 element, that's when I'm going to go through and I'm going to essentially create my chunk boundaries. So for each element, I'm going to say that I want to get the element specifically, just like that. And then for each one of these, I can just get the text. So I can say text is just Whoops. element.ext. That's going to give me the
- 1:26:26
text. And let's make sure we trim it to get rid of any white space because that's not important for our chunking algorithm. Then I can say, hey, you know what? If my element happens to be an H2 element, then I want to essentially create a brand new chunking boundary. So, here is where I want to essentially take whatever my current chunk is and I want to replace it. I believe they probably do that down here somewhere. Looks like yeah, they have their chunk
- 1:26:46
text that they're keeping track of. That's what I'm going to be keeping track of up here. I'm going to essentially have a variable called our current chunk. Doesn't really matter what it's called. It just defaults to an empty string. And inside of here, anytime that we have an H2, I just want to take my current chunk if it exists. So if my current chunktrim, that means that we have a current piece of text that we want to save. I just want to
- 1:27:07
take all of my chunks that I have, which I think they called result. So we'll just say result just like that. And I want to push in that chunk. So we'll say current chunk.trim. There we go. So all I'm doing is I'm saving whatever my current value is in this current chunk. And every time I reach an H2, I'm taking everything that's currently in my chunk, trimming it right there, and saving that as a
- 1:27:27
current chunk. Then once I've hit an H2, I know, hey, I need to reset what my current chunk is. So I'll say current chunk is equal to an empty string. And now this is what's happening for all my batching. Every time I hit an H2, I take all my current text, save it, and then completely flush it and start fresh with a new piece of text. Then doesn't matter if I have an H2 or not, whatever it is,
- 1:27:48
if I have some type of text that I want to save, all I want to do is I want to take my current chunk and I want to add that piece of text on. So we can just come into here. We can add my text and we'll put a new line character at the end to kind of represent that we have paragraphs. I'm actually going to use a double new line character just like that. Since this is more like representing separations of paragraphs and that right there is essentially
- 1:28:08
taking care of all of our code. All we're doing is we're just looping through and saving every single piece of text. But if we hit an H2, we take all the text we've already saved, create that as a single chunk, and then start over with creating a brand new chunk. The only thing we need to do to make sure all of this works properly is if we still have text after our last H2 that's going to be saved in our current chunk,
- 1:28:28
we need to make sure we save that. So if our current chunktrim, then we want to make sure we save that. So we can say result.push current chunk.ch. And then we can just return our result. So you can see this is a much simpler algorithm than all of this crap that the AI generated. So unfortunately, AI generation wasn't
- 1:28:48
really that useful for a lot of the actual complicated code, but it worked good for getting the boilerplate app set set up, getting our routing set up, our TypeScript, and so on set up. That all was really useful. The rest of this, not so much. Let's get rid of these comments cuz I don't know if they're even relevant to what our new code does. And our new code is so simple, it probably doesn't even need a comment to explain what it does. Now, let's also make sure our chunk articles.t test works as well.
- 1:29:10
So, let's just rerun our test script. we probably like you can see here have a lot of different failures and that's because most likely you can see it's passing the full HTML body and so on. I wanted to just pass the content that's not inside of the HTML body article. So I'm just going to essentially take all these HTML body articles, get rid of them and then same thing here. Get rid of the bottom half of all of these. And
- 1:29:30
now let's try rerunning that. And that should hopefully be all we need to do to make it pass. And it looks like that is still not quite working like we expect. So it looks like there's actually a test here for content without an article wrapper. That's not even test that needs to exist. So here we can see content without article wrapper. Let's completely get rid of that test. Doesn't do anything. Handles deeply nested content within H2 sections. That looks
- 1:29:50
like it's fine. That shouldn't really matter. That should still work. Handles H2 with inline elements. That should also work fine. Let's just run our test again. And now we can see that everything is passing as we expect. Now I also want to create a real world test case. So we're just going to come in here. We'll create a describe block. We'll call this one real world just like that. And we'll say it works
- 1:30:13
on a real blog article just like that. And I'm just going to paste in an actual real blog article that I have. So we can just say const html equals. And we'll paste in that entire article just like that. And it looks like there are potentially some formatting errors probably because it has some back ticks in it or something along those lines. To
- 1:30:35
make this actually probably a little bit better, I'm going to go through. I'm going to actually paste in a much smaller article. So that's this one should work better. That other article was a massive article. This one is a much shorter article. So it'll still give us the exact same test. And then we just want to make sure that it matches everything that we expect inside of here. So let's look at some of the other tests to see how they were formatted. You can see here we just get our result and we expect essentially the result to
- 1:30:56
be what we expect it to be. So we can see expect it to be result zero.trim. And let's make sure that's kind of how the rest of them are working to match. Yeah, it's just matching the text exactly. It looks like it's only matching the very first thing. Uh here it's matching more of them. That's fine. So, let's just go ahead and make sure that it is matching what we expect. So, we can say chunk articles result zero to
- 1:31:16
start with. Or is that actually an option? I believe there Oh, to contain is probably what we want. There we go. We can do to contain. And let's just copy some of the text. We can just copy this entire paragraph. If it contains that, then we're probably doing pretty good. So, let's just paste this down. There we go. And then we can expect this second result to contain some more text. So if we look
- 1:31:37
here, here's our first H2. Let's get the text from there. We're going to paste that in. There we go. Let's copy that down again because we also expected to get some random text from the middle. So let's just randomly choose some text in the middle. And we'll copy all of that. And we'll paste that in. And again, since we want to make sure that none of the HTML is sticking around, we'll get rid of the
- 1:31:58
HTML related things because those should not be in there. Get rid of that. There we go. And now let's run our test. And hopefully that should work as well. And there we go. That did test, which means it's working on actual real world HTML directly from one of my blog articles. So now that we have that done, what I want to do is I want to first check out a new branch because I'm still on the old branch. So, we can say checkout-b and this is going
- 1:32:20
to be ingest articles. There we go. I want to add this code because I believe we checked through. Yeah, we checked through absolutely everything that was written. So, we can say get commit. Add article ingesting. There we go. And now we need to work on the actual embedding portion of our
- 1:32:42
chunking for our articles. Now, you could go ahead and you could use like OpenAI for the embedding, which is what we will use in production. But there's no point in using these costly production APIs when we're just testing locally when we can use a really lightweight local model to do our embedding instead. So, what I'm actually going to do is inside of LM Studio, which is where I run all my local models, I'm actually going to load a local embedding model that's going to do
- 1:33:03
the embedding for us. You can see here there's a super lightweight 609 megabyte model that's going to do our embedding using Quen. We can load that model directly. We can just click that we want to load that model. It should load incredibly quickly since it's quite small. And now we have access to this embedding model locally as well as when we want to deploy to production. We'll be able to use that openi AI version of the model as well. Now this is something
- 1:33:24
that I could prompt the AI to do, but it's so simple to do. We're just going to do it ourselves cuz it's going to be way quicker than reviewing code and stuff like that. So I'm going to create a brand new section inside of here specifically for embedding. And I'm going to have a way to get our embedding model because in production we'll use a different model than we use in development. And I'm also going to be setting up a way for us to be able to embed all the different chunks we want.
- 1:33:44
So let's just come in here. We're going to get our embedding model.ts just like that. So for this get embedding model, we're going to export a function called get embedding model just like that. And this is essentially going to use our local environment variables to determine which embedding provider we want to use. So
- 1:34:05
let's just come in here. We're going to create a section called embedding. and we're going to have an embedding provider. This is either going to be Quen if we're locally doing it or we're going to have OpenAI if we're using the OpenAI way. So, let's just do Quen for our local embedding. And then what we want to do is we want to specify our local models name. And specifically, actually, we want the URL. So, we're
- 1:34:26
going to say local embedding base URL. And that by default, if you're using LMS Studio, is localhost 1 2 3 4. So, there we go. That is going to be our local embedding base URL. Here is our embedding provider. And now inside of here, we can create a function to handle all of that. So we can say if our server env
- 1:34:48
dot and we want to make sure we add these variables to our server env. So we can come in here. We have our local embedding base URL. I think that's what I called it. Yep. Local embedding base URL and our embedding provider. This one right here is a Z URL. And then this embedding provider is a Z enum. And this enum is
- 1:35:11
going to be either Quen or Open AI. There we go. So now we can say here if our local embedding provider is equal to Quen, then we want to do the local version otherwise we want to do the production OpenAI version. Now we're going to be using the Verscell AI SDK.
- 1:35:32
So, we're going to say npmi at AAI SDK slash open AAI since we're using OpenAI for everything. That's going to install that AI SDK for us to be able to use for all of our different code. We're also going to be using the AI library and I'll just make sure I force install those so it installs properly. And now we can actually use those different sections. So, here I want to create a
- 1:35:53
custom provider. So, we're going to say create open AI. That's going to create an OpenAI provider for us which takes in an API key. We don't need this. So we can just put not needed. Any value will work because we're using local AI that doesn't need an API key. And then our base URL is our server env. And that is going to be our local
- 1:36:15
embedding base URL. So let's just make sure we import our create open AI as well. Looks like it's not letting me manually import that or automatically. So I'll just create open AI from at AI SDK. Just like that. And I believe this just needs a capital I. There we go. Capital I there. So that is our provider. And then we can just return
- 1:36:35
our provider dot and we want to do embedding. So we're going to say embedding. And then we're going to pass it in our model which we'll create a variable for. We'll call this our Quen model. And up here we can say const quen model is equal to. And if we go into LM Studio, we can just click copy next to the name of our model. Paste that down inside of a string. And that's going to be referencing the model that we want to
- 1:36:56
use. Now we can do the exact same thing for our open AI model. So we can say open AI model and for us for this model we're going to use the text embedding three small model because we don't need really complicated or large models for this relatively simple text. Now in the case that we're doing OpenAI this is just as simple as saying open AI.
- 1:37:16
And passing in the model we want to use and we just need to make sure we supply our API key properly inside of our ENV. So in this embedding section we just need to make sure that we have an open API key just like that. And we can specify some value. We'll just say to-do add value. Just like that. And now in our server env we can make sure our open
- 1:37:38
AI key is just going to be a string. There we go. And I believe this is called OpenAI API key. And I think I named it the wrong thing in our ENV as well. So let's just make sure that that is the correct name. That is what's required by default for this embedding model to work. So now we have an ability to get whatever model we want wherever we want to do it. Now, let's just create a brand new function called embed
- 1:37:59
chunks. That's going to take all of our different chunks and embed them for this. So, we can say export async function embed chunks just like that. It's going to take in essentially an array of strings. Each one of our different chunks is what these string arrays are going to be. And then it just needs to embed them for us. So, if our textlength,
- 1:38:20
whoops, text.length, and let me just call that texts. There we go. If that is equal to zero, well, we can just return an empty array because we have nothing to do. There we go. Otherwise, what we want to do is we want to get our model, which is just get embedding model. That's why we created that function so we can use it in other places. And then we can just get the results of those embeddings by calling
- 1:38:42
await embed many. And this comes from that AI. So we can say embed many from AI just like that. And this embed menu just allows us to essentially take in a model. So we'll pass in the model that we're using. It allows us to pass in all the values we want, which is just our text, and how many parallel calls we
- 1:39:03
want to do. Let's just say we max it out at 100. And this is going to essentially allow us to create embeddings for all of those chunks at one time. So we can just say return result.bed. And that's going to get all those embeddings for us. Now we can use this inside of our ingest section where we have our embeddings. So here you can see our embeddings are by default set to null. we can just make sure that we get all of those embeddings
- 1:39:24
automatically. So here we create our chunks. Now we can create our embeddings which is just embed chunks and we can pass it in all of our chunk text. There we go. Make sure we await this because this is an asynchronous operation. And then in our embeddings we can scroll all the way down here and
- 1:39:45
when we're doing our insert we can make sure that we do it for all of our things as well. So here we just say I chunk text I just like that. And for our embeddings we'll say embeddings of I. There we go. That's going to make sure it matches up the embedding for our particular chunk text. So now let's go ahead and actually test to make sure that this entire workflow works like we expect it to. So what we can do is I'm
- 1:40:06
going to create a brand new file. I'm just going to call this workflows.rest. And this is going to be using an extension that I have inside of my editor. This extension is called rest client. and essentially just allows you to make REST HTTP request directly from VS Code. So we can say that we want to do a git request to localhost 3000/ API/workflows/ingest
- 1:40:30
blog articles. That is the URL that we created for our actual workflow. And if we click send request, it should hopefully send a request and kick off that workflow. So if we click send, you can see connection was rejected. That maybe means that we're not actually running our local host. Yeah, looks like we're not. We'll just say npm rundev. That way at least everything is running on local host. And now we should be able
- 1:40:51
to send our request and hopefully we should get a response. It looks like it's still not coming through. Uh looks like we're getting a 405. Okay, there we go. Let's just try it one more time to see if it works. You can see here 405 method not allowed. Let's just take a quick look at here in our API inside of our workflows. It looks like they're using a post request. I believe you need to use a git request. So we're going to change that to a get. And now we're going to do a quick send here. And you
- 1:41:12
can see blog article ingestion workflow started. And here's our run ID. Now, to be able to view our workflows, we can just run npx workflow web. And that should hopefully pop up a web dashboard where we can view our workflow to see exactly what's going on with every single step of the process. And we can also run npm run db studio.
- 1:41:34
And we should also be able to see our data being inserted directly into here if everything is working like we expect. So, let's just come over here. And you can see right here that I have this working and it looks like we immediately had a failure that is happening for trying to get our new articles from RSS feed. And if we take a look at the actual error, I'll zoom this in so it's a little bit easier to see. Of course, it doesn't actually want me to zoom.
- 1:41:54
There we go. And if we take a look at the error, which is quite difficult to read at this zoom level. We'll zoom out a little more. I'll just tell you. It just says it's failing inside of our database query. That means that we forgot to push our database changes. npm run db push. That should push up all of our different database changes. And once that finishes up, looks like it's saying content table is it created or renamed.
- 1:42:15
This is a brand new table. So, let's create it. There we go. And it looks like type vector does not exist. Okay, this is great to actually know. This is because we need to make sure that we're using inside of our database the actual vector extension, which right now is not being used. So, we can create a migration for that. We'll create a brand new custom migration specifically for
- 1:42:35
that. So what we can do is we can say npx drizzle kit. We can call custom just like that. That's going to allow us to generate a brand new custom one. We need to say generate. There we go. That's going to give us a custom migration that we can create. Now if we look inside of our migration folder we should see this completely blank custom migration. And
- 1:42:56
inside this custom migration, this is where we can just say create extension if not exists vector. And now if we just give that a quick save that should be all we need to do to actually work with this particular code. Now I can say npm run db generate because we know that hopefully our actual database is the way that we want
- 1:43:16
it. So we have generated and actually before I generate I already remember that we have a to-do. Let's just search for it. Here we go. The to-do is to change to not null when implementing. So we'll say not null. And also our dimensions must directly exactly match the dimensions that we're using for our database for testing purposes as well. Now the Quinn model we're using uses
- 1:43:37
1024 for its dimensions. So let's make sure we specify our dimensions there. And now if we want we should hopefully be able to run our code. So let's try just running migrate first. This should hopefully work properly, but it may actually fail since we never initially did a migration. You can see here it is failing because we never did an initial migration. So let's go to our Neon database. We can get there by just running npm run db studio. And I
- 1:43:59
essentially just want to completely delete our database to start over completely from scratch. That way we can run our migrations and it should work properly. So give it a quick second to load up everything. I essentially want to completely drop all these tables. Just get rid of absolutely everything. We'll do a quick cascade on that. There we go. That gets rid of everything. And then inside of our drizzle table, let's drop this migration table again. Get rid
- 1:44:19
of absolutely everything inside of our database. That should be all we need to do. And now when we run our migration, it should work because there is nothing at all inside of our database. So we can see it's going through. It ran those migrations and hopefully everything worked. Now we can push up our changes as well. So we'll say npm run db push. This is going to push up our temporary changes for our chunks and content. And if we give that a quick second, looks
- 1:44:40
like there is no unique constraint mentioning give us keys for reference table content. I believe I actually know why this error is happening. This is because in my utils, this is why I was getting an error with a not null. This should be primary key. That way it actually knows that this is the primary key of my table. That should hopefully fix the error that I was running into. Let's try that push again. This will actually solve a lot of problems that we were having. There we go. It does look
- 1:45:00
like that actually solved our problem. And now let's go ahead and actually test that workflow. Let's just close out of everything other than that. Give this a quick run. We'll send that request. It did actually send that request. And now let's test to see if it worked directly inside of our workflows. I'll just close out and reopen those to make sure that we have the latest version going. And if I open this up, you can see that it is running. And it looks like it's actually working like we expect. It's a little
- 1:45:21
bit difficult to see when I scroll down. We are getting some errors here. But it did actually get our blog articles proper. And if we look at the output here, it has 147 articles, which is how many articles I have written at the time I'm recording this video. This one has an error though. Invalid JSON response is the error that it is getting when it's trying to do various different API calls. I think I actually know why this is. This is because in my environment
- 1:45:42
variables, I need to make sure that I specify the base URL is slashv1 so that it knows which version of the API to use. That should actually solve all of our problems. So, it actually does look like that is now working. If I look at my steps here, you can see a few of them are succeeding after they do their retry. And a couple of other these will hopefully retry relatively soon, and they should succeed almost immediately.
- 1:46:02
So, you can see that they are succeeding just like I expect. And if I were to check out my Drizzle, if we just come into here, npm run DB Studio, and we pull that up, we should hopefully see that we start to have things showing up inside of our database. So, let's just pull this over real quick. Give it a second. And now you can see in our content section, we have those 10 articles that are inside of here. And if we look at the chunks, we have 32 different chunks between those 10
- 1:46:23
different articles. Now, the reason the chunk number is so low is because this is pulling my first few articles I ever wrote. And my first few articles I ever wrote didn't actually have any H2s in them at all. So, it's just one chunk per article. So that's why this number is relatively low. My newer articles are much better structured. That's just kind of how I was when I was writing way back in the day when I wrote these first few articles. But you can see it's pulling in all of our information. It has the
- 1:46:44
title. It has the description. It has the publish date back in 2019. URL thumbnail URL. The type is article. Has the raw content of everything. It looks like all of this data is being pulled in just like we expected to. So this is working. And you can see here it's now ingesting the next set of articles cuz these 10 finished and it did that batching process. So all of that is working just like we expected, which is really nice. So now with that done,
- 1:47:05
let's kind of minimize out a lot of this, close out of a lot of this. There we go. And I want to make sure we set this up to run as a cron job. That's really easy to do with versel. We can just come in here with a versel.json file. And inside of here, we can specify our crons. So we can say crons, which is going to be an array. And inside of here, each one is going to have a path. This is just the URL to where we want to run this. So in our case it's AI API
- 1:47:27
workflows/ingest blog articles just like that. And then we can specify how often we want this cron job to run. So we can say schedule and I'm essentially going to say that I want this to run at midnight every single Tuesday. That's what this is saying right here. The reason I do that is because I release blog articles on Monday. So if I run this at midnight on
- 1:47:48
Tuesday, it'll hopefully get a new article every single time it's published within about 24 hours. And this is going to make sure that I ingest all the articles the first time. And then every other time this runs, it's only going to ingest the brand new articles that I just created. So now that we have all this code done, and we wrote this all ourselves, so we don't need to re-review all the code we just wrote. Let's make sure we commit this. So we'll just say
- 1:48:09
add commit add. And actually before we add and commit, there's one other thing that we want to do. While we're in production, we're going to be using a slightly different API for our embedding. We're using OpenAI as opposed to the Quen model. So that means inside of our schema for our chunks, we need to specify the dimensions here for the OpenAI dimensions. Now, OpenAI uses 1536
- 1:48:31
for its dimensions and you cannot cross-contaminate. I can't have Quen and OpenAI dimensions in the same one. I must hard commit to one API or the other. And since this is going to be used in production, it won't have access to my local computer. We're going to be using the OpenAI model of 1536 dimensions. That's going to be what we use when we create our migration. So now we can actually create our migration. npm run db generate and that'll create
- 1:48:54
that migration file for us that when we push to production we'll make sure all this data is properly in the correct location just like we expect it to be. So now with that out of the way let's go ahead and actually commit these changes. So we'll do our get add again get commit add embedding support and we'll push these changes up just like that. There we go. That's working
- 1:49:16
great. And now what I want to do is I want to run code rabbit again. Now, I could be running Code Rabbit on this pushed up pull request, which is probably the way I should do it since I've already pushed up the pull request. But if, for example, I wanted to test this before I committed my code or before I pushed it. You can also run code rabbit locally. Now, it only works in either a Linux or a Mac environment. So, since I'm on Windows, I need to use Windows Subsystem for Linux. So, I just
- 1:49:37
type WSL. That'll open up this Windows subsystem for Linux. And then what I can do is I can run CR to run Code Rabbit after I've installed it and logged in. Now, I already have the local version running on my computer, so I can just type in CR. And since I want to specifically work off of the main branch, I can type in d-base of main and that's going to compare my code to the main branch. If you leave this off, it'll just take whatever changes you
- 1:49:58
currently have in your application that aren't actually committed, and it'll use those. But since we've already committed our changes, I'm essentially just going to be comparing all the changes on this branch to the ones that are inside of the main branch. Now, while that's running, let's go ahead and actually work on our environment variable setup because we have all these different embedding things that we need to specify inside of our Versell environment variables. So, let's add an environment
- 1:50:19
variable. The first one is our embedding provider. I'm just going to copy that over just like that. So, we have our embedding provider. This is OpenAI, and this is not sensitive, so we can leave that marked off. And then we want to add another one. Well, let's just save that one. We'll do them one by one. The next one is going to be our OpenAI API key. So, we'll copy that. And we're going to make sure that we paste that in. And then we're going to have to get a value
- 1:50:40
for this. And this is going to be sensitive. So let's just go to the OpenAI platform. Make sure that you actually log in and authenticate yourself. And then what you want to do is you want to go down to this API key section. And we're going to create a brand new secret key. Now for this secret key, I want to restrict it. And I specifically only wanted to be able to use the embedding models. So we'll just come in here and have request be set for embeddings and nothing else. We can then
- 1:51:02
give this a name if we want. We can also associate it with a particular project. So we should probably create a brand new project to be honest. So, let's create a project for this. We will call this video blog suggesture YouTube. And for this one, we're just going to keep everything the same. Our dev keys and our production keys are going to be in the exact same place because they can be separated from one another. But if you wanted to separate them, you could. For
- 1:51:23
now, I'm just going to keep them in the same place. So now, again, we want to make sure we restrict this to have a model of just embeddings and nothing else. And we will call this our dev key. So, we'll just say dev just like that. Create our secret key. And now we can copy this. And actually, I don't know why I called this dev. We're not even using this in development. This is just our production key. So, we'll rename it to production. And now we have that key.
- 1:51:43
And we can just paste that directly into here. Now, if we give that a quick save, that gives us our open API key right there, just like we wanted. Now, I do automatically have one thing that I think I need to change inside of my data. If we look inside of our server enving provider, I want to be able to allow or not the the local embedding base URL. This should be nullable. So, I'm going to come in here with a just
- 1:52:04
simple nullable. That way I don't have to pass that along in production as well. And I will probably need to change some code where this is being used. Yeah, this is having an error here. So I can just say if my local embedding base URL is equal to null, then we just want to return some type of error. So we can just say throw error
- 1:52:26
embedding URL not provided. And let's just give it the full name. There we go. Perfect. That'll actually solve that particular problem. So now let's go ahead and actually check to see what Code Rabbit said for its local review. So we can just pop this open and you can see it's giving me the results directly. For example, I have a redundant index on my content. URL. So if we go ahead and we take a look inside of our database
- 1:52:47
schema for content, it's saying that this is redundant, which makes sense because this is already unique, which adds the index for us. So we can get rid of that. While this one is not redundant, at least it didn't say it was redundant, even though I don't really think that we actually need this. So I'm going to remove this as well since I don't think it's actually needed. Now, let's go ahead and we're going to check the rest of this. Let's scroll down a
- 1:53:07
little bit here. You can see on conflict do nothing. It's saying don't make duplicate conflicts fatal. I actually disagree with this. I think it's perfectly fine to make this fatal because if it already has an item in there, I don't want to read my embeddings. I want to completely fail. So, I'm going to leave that as is. I think this is a bit of a false positive, but otherwise, there's nothing else that showed up, which means our code was relatively well written, which makes sense cuz we actually went through and
- 1:53:28
manually rewrote a lot of the code. Now, since I changed what our actual migrations are going to look like, I'm going to go into where our migration is. I'm going to delete that migration and I'm going to essentially regenerate it. So, we'll just come into here. Let's get a brand new terminal set up. npm run db migrate or actually db generate. That's going to generate us a brand new migration. And this is because anytime
- 1:53:49
that I commit changes, I want them to generally fall into one single migration file. I don't want to have 8 n 10 different migrations for one single commit to my database. They should all be wrapped up into one migration if at all possible. Now, you will notice that we have two migrations and that's because one is a custom migration to enable our extension and the other one is the real migration that sets up our actual code. Those two being separate, I
- 1:54:10
think makes perfect sense. So now let's go ahead and actually commit those changes. So we'll just say get add get toit clean up code. Again, use better commit names yourself. We'll get push this to our actual repository. And now we can go over to our GitHub. So, if I just pull this over, you can see here we can compare and pull request this in just articles. Just like that. We can
- 1:54:31
make sure we create a pull request for this. And for this one, we've already pretty much tested everything that we need to test. I'm just going to go ahead and directly merge this because we already ran our code rabbit locally. So, we're just going to click merge on this directly. That way, it can roll out to production and we can hopefully make sure that everything is working like we expect it to. So, we're just going to make sure this branch deletes properly. Give it a refresh to make sure it fully deleted. There we go. That was deleted.
- 1:54:53
And now we can go over to Verscell to make sure that it is actually running like we expect. So inside of Verscell we have our deployment. You can see it's building up our production deployment. It's hopefully going to be migrating all of our database changes to include everything that we just created. And our UI should look exactly the same, but now we're going to have cron jobs and so on that we can work with. It looks like we had some type of failure. Invalid
- 1:55:13
environment variables local embedding base URL expected string received undefined. I thought I made it so that was nullable, but it looks like I must have failed. Let me go ahead and check our code wherever we have that server env. And here I did make it nullable. Let's maybe make it optional because I think it's returning undefined. So optional accepts undefined while nullable only
- 1:55:35
accepts null. So let's just come in here. Get add get commit fix undefined variable. And we'll just push that up. It'll probably make us create a brand new pull request because we originally deleted that. So, this will just be a really quick fix. I should have caught this before, but this sometimes just
- 1:55:55
happens. There we go. We'll merge that in real quick before any of that code rabbit stuff runs because there's no point in running that on a oneline fix. There we go. That will delete that. And now, if we go back to our deployments, hopefully this one will actually work properly. We can just cancel this one because we don't need that deployment preview. All we care about here is going to be the actual production deployment. And I'll come back to you once this finishes. It looks like that just
- 1:56:16
finished. Everything deployed successfully. If I come in here, I should still be able to sign in with GitHub, which is just like what I expected to do. Let me just refresh. Make sure I got that button working. And of course, the reason it's not working is I'm on the wrong URL. I'm on the deployment URL. I need to be on the main URL to make that work. There we go. Now I click on that. It looks like it signed me in just like I expect. So that is working. The next step is to make sure that our actual workflows are working
- 1:56:37
like we expect as well. Now the nice thing is if we go into our settings here, we can actually find our cron jobs. So you can see here is our cron job for ingesting blog articles. And I can manually run that if I want. So we can just go ahead and we can click run on that. And that should hopefully run this cron job. And now if we go back and we scroll up to or scroll down to the workflow section, we should see inside of here that we have a workflow. Right
- 1:56:58
now it's saying that I need to install the workflow stuff. It's probably just taking a second for that to pop up. It actually just did right before I refreshed. And now if we click on this, you can see I get that exact same UI for everything with my workflow. And you can see it's ingesting my articles and it's going very very quick for getting all these different articles ingested into my system. So now I actually have these directly inside of my database. We can even view this by going over to Neon. So
- 1:57:19
if we just pop into Neon over here, we're in our production. We can go to our tables. And now we can click on for example our content. And you can see here it's already ingested 130 of my 147 articles. So it's pretty much almost entirely done. If we actually just go back over here, it should be done very very quickly. Let's refresh our Neon console to see. Here we go. But we got 100 147. Looks like they are all
- 1:57:39
ingested. And you can see that this has completed. And if we scroll to the top, we can actually click on this to get our output. And our output, if we decrypt that, it specifically says that it has a total of 147 succeeded 147. So we fully ingested and embedded every single piece of article information that I've ever written on my blog. And now we can hopefully implement the UI for querying this information to show you the true
- 1:58:00
power of how amazing creating your own rag system can be. So let's just get started by creating a brand new branch. We'll check out main. I want to pull all the changes from main. And then we're going to check out a new branch. And this new branch is going to be for our UI. So we're going to say get checkout-b UI. Again, terrible name. Name it
- 1:58:20
whatever you want. And now what we can do is we can actually go into our sandbox. Let's just actually close out of our current sandbox. There we go. And I'm going to resume our sandbox. That's just a command I have for opening up my sandbox where I was before. So it's just going to be opening up our sandbox. I want to be in this very second sandbox actually. So, let's make sure we go to that one because that's the correct one.
- 1:58:40
And now I'm just going to write out a prompt telling the AI exactly what I want. I'll come back to you once I'm done writing that prompt so you don't have to watch me write it. So, I just wrote it all out and essentially I just told it to create a simple single page URL that has a search bar and a header. And the header should have a login logout button as well as the user's information. The search bar should be disabled if the user is not signed in. I also mentioned that when the user searches, it should call out a server
- 1:59:00
function and it should use Drizzle to match that information using a vector database search. So, first it embeds their query, then it does a vector database search and only returns results with at least a 50% match. I then specifically say that I don't want to return individual chunks. I instead want to get the content associated with those chunks, and I want to get at most 20 pieces of content. I then listed out the information that I want to get, as well
- 1:59:22
as kind of what the URL or the UI should look like. I give it a small description as well as a screenshot of what I want the UI to look like. So, we're just going to run that and hopefully you'll be able to create a relatively simple UI for us. And the nice thing is since this is the least important part of our application, we can kind of get away with not really looking at the code as closely for the generated UI as long as the back end of the search works properly. The actual rendering out of
- 1:59:43
the UI is much less important in this application. So, we can get away with spending much less time reviewing that portion of the code. Now, while this is running, there's one thing I actually want to take care of, and that is securing down our cron jobs. Right now, anybody could access that URL and run our cron job, which is not ideal because it cost us money to run those cron jobs. So, what we want to do is we want to lock that down by essentially using a
- 2:00:03
cron secret. We'll just copy over this code cuz it's relatively straightforward what's happening. I'll show you what it's doing once we get into the code, but what we can do is inside of wherever we're calling these. So, we got API, we have our workflows, and here's our routes for our cron job section. Essentially what I want to do is I want to make it so that I'm getting my authorization header. I want to get a cron secret which comes from my environment variables and I'm just
- 2:00:24
checking to make sure those cron secrets match. So essentially I'm going to be setting an environment variable called cron secret. Anytime my cron job is run by versel, it passes that secret variable up in the header. And as long as that cron secret matches what's saved in my environment variable, I know that this is a valid request from Verscell. If they don't match, then it means maybe someone else is trying to maliciously execute code on my behalf. So I can make
- 2:00:45
sure I block that with an unauthorized request. So I can just create a quick little helper function for doing this. So let's just go down into my lib workflows section in my utils section for that. Let's just create a brand new file. I'm going to call it verify and run cron.ts. Just like that. And I'm essentially going to be taking all this code, export a function called verify and run cron. It's going to take in my
- 2:01:06
request as well as the cron that I want to run. So we'll say run cron is just a function that is a promise and I don't really care what that promise returns. So we can just specify it as unknown. There we go. Make sure that I import next request just like that. And now I can get my off header which comes from here. I can get my cron secret which we're going to use server env for. And
- 2:01:27
let's go ahead and go in server env. And we'll set our cron secret to a zstring. There we go. And then our env variable. We should also probably set that. So we'll just set a category called cron. There we go. And we can specify this cron secret to be whatever we want it to
- 2:01:47
be. Now in our particular case, we can just copy over our better off secret because this is just for local testing. But when we go ahead and we actually use this in a real environment, we'll want to make sure that this is something more secure. But for now, that gives us our cron secret just like that. And what we can do is we can make sure our check is correct right here. And if it's not, we return that status of 404 or 401, sorry. Otherwise, I want to run my cron. So I
- 2:02:09
can just say run equals await start getting that from the workflow API. And then I can call run cron just like that. Make sure this is an async function. And now I can return my result. So I can just say inside of here message cron started and I can get my run ID. There we go.
- 2:02:30
Just like that. That's all I need to do to be able to verify and run my cron. And now inside of my workflows, get rid of all that code that I pasted in. I can just replace start with verify and run cron. Just like that. I can pass it in my request. There we go. Which comes from here is a next request. Just like that. And I
- 2:02:51
don't even need to return anything here because this is my return. There we go. That cleans up my code pretty drastically. And that just takes care of making sure this only will run if I have that cron secret set. Now let's check our AI. It looks like our AI is finished. But before that's done, let's actually go ahead to versel and make sure we set this information in our env variable. So here we go. Let's come over to here, add an environment variable
- 2:03:12
called cron secret. Just like that. And for this, what we're going to do is we're going to go to the better off instructions and just generate another secret here. This is the easiest way for us just to create a really simple secret key. Just like that. Now we can save that and we have that set. So that way our cron jobs are actually going to be secure when we actually run and use them. Now let's go ahead and actually since our AI finished running
- 2:03:32
everything. Looks like here all these files were created. Doesn't look like there's anything we need to do on our point. For this one, what we can do is really the most important part is this action. I want to make sure that this action is correct. And then the AI or the API. I want to make sure that the UI looks like it should. So let's look at the actual changed files. So you can see here verify and run crown. That's something we created. Normally I'd put this in its own pull request, but for our purposes, we'll just keep it in the
- 2:03:53
same one. Same thing with our server environment. We changed that. Otherwise, we have some components here that they changed. And then we have a route here particularly and this is something that we changed. So we can ignore that. Our page is something that they changed. Layout again is something they changed. And then their search action itself. So let's first for this one actually just test what it actually looks like when we use this. So let's just come into here.
- 2:04:13
We're going to go to our application. I'm going to minimize it down a little bit. npm rundev. Just like that. Start up our application on this localhost 3000. And I'm going to make it so it's in its own window. There we go. So now we have our little search blog content. I don't really like some of the stuff that I added, but you can see that I have my sign in and I need to click the sign in button. It's going through and
- 2:04:34
it's signing me in, which is great. Now that I'm signed in, I can actually do a quick search. So let's just say how to get started with React Hooks. There we go. And it looks like it is doing a failure when it's trying to query that data. And I think the reason why it's failing to query our data, I'm not 100% sure actually. Let's just go ahead and
- 2:04:54
we'll check the code that it generated to see what's going on. You can see here error, no operator matches the given name and argument type. You might need to add explicit type cast. So there's something with how it's doing its checking. So we have our cosign distance. It's doing some weird stuff with like divide by two and so on. Now the really nice thing is that Drizzle actually has documentation on how to set up all of this automatically. So the first thing that I notice is that we
- 2:05:14
have an index that we're missing. We need to set up this index to make it so that our proper searching is going to be much quicker. So let's go into our chunk table and make sure we add that. So we can find here our DB schema chunks. We want to add in that migration or sorry that index. So we're going to have table just like that. And this should be an array.
- 2:05:35
And we'll pass in that index just like that. This is our embedding index. And we're going to use snake case because that's what we use for everything. And just like that, it is on our embedding. So that looks exactly like we want it to. Now if we look at the documentation again and we go a little bit further, this is doing that embedding stuff for us automatically. So we can kind of ignore that. Here is how it's finding out the different similarities. So you
- 2:05:56
can see it's using one minus the cosign distance between our embeddings. So that's where there's a slight difference between what's happening in our code versus their code. So let's go ahead and actually fix that. Let's go to this search. You can see where we're doing our embedding. Let's make it do one minus the coine difference. We don't need to worry about this divide by or this map width or anything like that. None of that is something we have to worry about. This is going to give us
- 2:06:16
exactly what we need. Now, if you notice, it's using this embed many function. It should just be using the normal embed function because I'm only embedding one thing. So, we can get our embedding from our value just like that. And here we can get rid of that. Just like that. So, we can get our embedding. And I don't think we need to worry about mapping this to a number or anything like that. It should already have that
- 2:06:37
type properly done. So we just get rid of that. We can call this our query vector. So at least the name matches everything. There we go. That cleans up all that. Gives us our start position, our raw text, our content ID, and it looks like it's doing a lot of code to get this information, of course. So let me go ahead and I'm probably going to rewrite the majority of this entire
- 2:06:58
section because I really am not a fan of how they did this. So what we're going to do is we're going to just delete all this. Actually, I guess we'll keep that cosign stuff because that's what we're going to be doing. We're going to be doing a select distinct on and we want to select distinctly on the content ID just like that because we want to get unique content information. So that's what we're going to be putting. And then we want to get all of our information.
- 2:07:18
So our ID is going to be our content ID. We also need to be getting the title which is the content.title. We need the description. Whoops. Which is content.escription. I'm just going to copy this down a bunch of times because we're going to have a lot of the same stuff. We're going to have
- 2:07:39
our thumbnail URL. We're going to have our normal URL. We're going to have the type. Whoops. Type. There we go. Similarity. That is going to be something we have to actually get. So, we'll just call this similarity. Right there. We're going to get our start position and our raw text,
- 2:08:00
which we'll call actually we'll keep it raw text. That's perfectly fine. And that comes from our text. Now the similarity is just this cosine one. So I'm going to actually take this and put it in its own variable. We'll say db similarity just like that. That way I can reuse this because I'll need it in multiple different places. And now what we can do is we can search from that chunks table
- 2:08:21
where that cosine distance I can just actually use that db similarity. So let's come in here with db similarity where that is greater than 0.5. That means it's a greater than 50% match. What I want to do is I want to do a join. Just like that. I want to join on that
- 2:08:41
content table where my content do ID is equal to my chunks.content ID. There we go. Now when it comes to the order by unfortunately I first need to order by this content ID. That's just because that's how select distinct on works. And then I can order by my DB
- 2:09:02
similarity after that. There we go. So that's going to first order by content ID and then it's going to give me the most common DB similarity as my very first option. And then I want to limit this to exactly 20. So the ones that reference my chunks table are going to get whatever the most common or most similar chunk is and it's going to give me the top 20 pieces of content. But my
- 2:09:23
content is still not going to be in the correct order because we're ordering by content ID first. So it's always going to show me the first ID, second ID, and so on. Instead, I need to reorder these particular values. So we have our match chunks, which is what this is. I can get our sorted chunks. We'll call it sorted results. And that is just going to be our match
- 2:09:44
chunks. And I want to sort this pretty simply where the B dots similarity minus my A.Similarity. This is just going to sort them by similarity. Just like that. Now then after that I can just return that data sorted results and we can get rid of a lot of this extra code that the AI
- 2:10:05
write. For some reason I don't really like that code or what it was doing. So now we can return our sorted results. And of course it always loves to create types for things that I don't really want types for. So we'll get rid of all those types and get rid of these extra imports that are no longer needed. And our code is again much cleaner and much simpler to use. And this should actually return to me those top 20 results. So let's actually test to see if this
- 2:10:25
works. So we are still getting the exact same error. And when I look, it's because they imported cosign distance from the wrong location. Let's make sure it actually imports from the correct location. Hopefully that'll fix our particular error. Okay, I think I actually know what the error is this time. These parentheses are just in the wrong location. Also, we should probably trim this query. So let's just throw a trim on there just so it is trimmed. And
- 2:10:47
we should also probably convert it to lowercase again to make sure it's normalized as much as possible. So now let's go ahead and test that. We're going to just go back on our page. How to get started with React Hooks. There we go. And it looks like it actually did the search this time, but it says cannot read properties of undefined reading to ISO string. That's probably because I changed what this order of actual return
- 2:11:08
results is like. So, let's just go into our components. I'm guessing we'll have some errors somewhere inside of our components. Maybe in the app page actually is where our errors are going to be. Looks like there's no errors in here that are showing up. So, let's check our search results. Make sure there's no errors in here. Looks like import search results has no exported member search result. Oh,
- 2:11:29
that's the type. Yeah, of course, we don't actually have that type anymore. So, we essentially just need to completely clean this up pretty drastically for the stuff that's being returned and imported into this function. So, we pretty much know for our search result, this return for our results here is going to be the exact return value from our action. So, it's whatever the return of this function is. So, let's just actually use that. We're
- 2:11:50
going to say that results is the return type of type of search content. And of course, we need to await this. Just like that. There we go. That should clean up what the return type is. You can see that has everything inside of it. And we're probably running into errors down here because it looks like I never actually
- 2:12:12
got the publish date, which is fine because I don't care about the publish date. We can remove that entirely from here. So that cleaned up that and otherwise it looks like all these pages are fine. One thing that I'm going to prompt the AI to do while I do some more testing is I don't want this to search on like key press. I don't want some debouncing search or anything like that. So let's just come into here.
- 2:12:34
Get this full screen. There we go. Change the search so it only searches when pressing the search button or pressing the enter key to submit the form. No debouncing
- 2:12:56
or automatic search. And then we'll just link the page. There we go. So that'll just work on that while we do our testing to see how the rest of this looks. So, let's just bring this over. How to get started with React Hooks. There we go. Do a quick search on that. And you can see it's returning all the top matches. And you can see these are
- 2:13:17
essentially all of my blog articles specifically related to searching or related to hooks inside of React. And they are sorted perfectly by which one is actually related to these. So, you can see that this does look like essentially what we want it to look like. And if I click on one of these, it does link me to that blog article. So, that does work just like I expect it to. Okay. Okay, the AI just finished with that. But what I actually want to focus
- 2:13:38
on before we actually test everything is I want to make it so that our URLs, so if we scroll down to wherever our results are inside this function here, wherever we have our URLs being rendered out, let me just give it a quick look here. Here's our anchor tag. Here's our result URL. I actually want to create a function. And this function is going to be get URL at chunk location. And then
- 2:13:59
we can pass it in whatever our result is. And our result is just this type up here. So let's get this type copied over. Come down here. And I want to get specifically an individual value from there. So that's going to be our individual result. And we can just do a switch on the type. So we can say type equals result.ype.
- 2:14:23
There we go. And we can do a switch on that type. In the case that we have article because that's the one that we're focusing on right now. Then what I want to do is I want to get my matched text and I want to split it on a new line. So we can just say split text equals result dot and what do we call it? Raw text. There we go. I want to split that on the new
- 2:14:45
line character. Just like that because we automatically added new lines every single time that we did a different piece of our chunk. So I'm going to split it on new line to give me my first chunk as well as my last chunk. Those are the things that I want to get. So I can say here if my split text.length length is equal to one. Well, then I only have one piece of text. So, I can return my result URL.
- 2:15:07
That's just going to be my URL as is. And at the end, I want to add this what is essentially a pound symbol followed by a colon followed by a till day followed by another colon. This is going to allow me to do automatic text highlighting. And then I can just say text equals. And what I want to do is I want to encode a URI component. And I want to pass it in that match text, which I call split text. and I want to
- 2:15:28
get the very first value from it. So essentially all I'm doing is I'm adding that very first piece of text if there's only one piece of text into this fancy little selector right here. And this fancy little selector is something that you can use in the browser to highlight specific text on a page. Now I can do an else section here for what do I want to do if there's multiple elements. And if there's multiple elements I want to get the first one followed up by a comma and
- 2:15:50
then I want to essentially encode the exact same thing but I want to get the last element instead. So, we're going to be encoding the last element by just saying atgative1. That's going to give me the last element. And we'll default it to an empty string if for some reason this is null, which it should be impossible. So, that's going to be our article case. And then we want to add in a case here
- 2:16:11
case. This one is going to be for video. And for now, we can just return the result URL. We'll handle what we want to do with video later on. There we go. And let's also add in default case. And this default case is just going to be catching any errors. So we'll say throw new error unsupported
- 2:16:32
type. And I'll pass it in the type just like that. And I want to make sure that this satisfies never. That way if I ever add or remove types, it'll give me a warning inside of here in Typescript. So now wherever I have that result URL, I can just pass it in this. And now we should hopefully be highlighting the text exactly where we are. So, let's just come along and we'll just say
- 2:16:53
I want to learn how to use React Hooks. There we go. Let's do a quick little search. And you can see here are all my different results. And let's try this one right here. And you can see it's highlighting the text where it thinks is the most relevant. Let's try another one. Let's try this custom hooks one. You can see it thinks this is the most relevant piece of text. It may not always be perfect, but for articles that
- 2:17:14
are maybe more segmented on what they actually teach, it could be really useful for bringing you to the exact location you need. Now, the last thing I'm going to do to clean this up is we're going to go to our page because there's a bunch of information at the top that I really don't care about. All that like title text stuff of like search blog content, search through our index articles. Yeah, we're just going to completely remove that section. There we go. Much cleaner right there with
- 2:17:34
that. So, that just cleaned up that code drastically. Makes it easier to work with. Also, in our header, the text that we have for right here, this webdev simplified, I really don't like how that text is set up. So, let's go ahead and fix that. First of all, this anchor tag, don't need that at all. Then what we can do is we go ahead and fix that for the user. So let's see here. It looks like it's doing justify between. We want to justify at the end to push that content to the end. And then wherever that text
- 2:17:56
is for the user's name, let's just change this to something that's a little bit brighter. Maybe do like 300 or let's do 400. Actually, if I bring that over, there we go. That's a much easier piece of text to read. And our sign out button still looks like it works. It still has our search results in here. That's fine enough for our particular use case. It doesn't really matter. And you can see when we sign in, everything is working kind of like we expect it to. So you
- 2:18:17
notice for this, I didn't really look at the code nearly as in-depth. It's probably got a lot of problems, but for the most part, the front end of this application doesn't really matter that much as long as it works. So what I'm going to do for this particular one is we're just going to commit this as is. So we can say get add get commit add UI. And then we'll just do a get push to push that information. Just like
- 2:18:39
that. And now let's create a brand new pull request for that. So we'll bring this over. create that brand new pull request. That way, Code Rabbit can run on this and we can see kind of what they say because like I said, this is much less important. So, if I miss a few things, this UI is not going to be the end of the world as long as that backend portion is what is working like we expect. Now, while that Code Rabbit review is running, I just want to come in here. We're just going to check out a
- 2:18:59
brand new branch. This branch is going to be for the YouTube ingest. Just like that. Make sure we put dashb in front of there. Now, we have that YouTube ingest. And I essentially want to go over to my AI and I want to tell it to do that YouTube ingesting using the same exact format that we used for the ingesting of the articles. So I just finished writing up
- 2:19:20
what I want the API to do for the AI. I essentially told it to ingest YouTube videos using the same format that articles are ingested. I told it to use the Google API's mpm package and I actually specifically told it what environment variables to be used because we need to make OOTH requests. So I gave it the hint that we're going to be using specific variables for those OOTH requests. I also made sure to tell it to save the same information and to include
- 2:19:41
the start time of the chunk inside the chunks table unlike articles which don't have a start time. I also mentioned to use the subtitles npm package to parse out the captions and that we need to fetch the video content details and transcript and everything. I also specifically mentioned that I wanted to process them in batches of 10 with a 9-hour wait between each batch because YouTube has a very aggressive quota policy unless you have a verified app
- 2:20:03
and this is not a verified app obviously. I also told it how I want to chunk the information and how each segment is going to work and to reference this article ingestion anytime when creating the implementation. Now, one thing is I'm going to change this word transcript to captions because that is what it uses in the YouTube API. So, hopefully it won't get too confused. And again, change transcript here to caption.
- 2:20:24
There we go. Now, let's go ahead and let that run. And while that's running, we might as well go ahead and set up our Google credentials because that's a relatively lengthy and kind of manual process that we'll need to go through. So, what we need to do is we need to go to the Google Cloud Console API. Once you're there, click on this button up here to create a new project. Click on new project. And now we can call this whatever we want. We're going to call this video blog suggest
- 2:20:45
YouTube. And we can create a dev version as well as a production version. In our particular case, since it's so painful to create this for dev and production, I'm just going to be creating one for this video, but I would normally create one of each, one for dev and one production to keep my quotas entirely separate. Then we can go ahead and we can create that. And once that's done, we need to go to APIs and services to enable the YouTube API. So here, if we
- 2:21:07
scroll all the way down, you'll see YouTube data API v3. Click on that. And you can see right now it is enabled by default, which is great. So now we need to go ahead and create credentials for this application. So what we can do is we can click create credentials. We want an OOTH client ID because we need OOTH authentication to get the captions. And from here, we can just specify a web application. We can give this whatever
- 2:21:28
name that we want to give it. We can also make sure that we add in different URIs for where this is going to be allowing requests from. All we care about is the authorized redirect URL. So we can say HTTP and we want to get localhost 3000. That's the only thing that we really care about for this. And we're going to be creating these credentials directly inside of our development environment.
- 2:21:48
So this is all that matters for our authorized redirect URI. So now let's go ahead and click create on that for a second. And you can see it's going to create that. And we have all of our information inside of here. and we have our secret and our client ID. So, let's copy over our client ID, go into our ENV, and we'll say Google API
- 2:22:09
Google client ID is equal to that Google secret. Let's copy that over. So, we come over here, client secret. Make sure we copy that over. There we go. And then we're going to have our Google refresh token. That essentially allows
- 2:22:30
us to authenticate as ourselves. So now that once we have that done, that is everything we need inside of here. We don't even need to worry about downloading this JSON or anything like that. Now, you also may get asked to create an OOTH consent screen. So just make sure you go to your OOTH consent screen and set all that up. It'll walk you through the setup of it. Essentially, it's very basic. You don't need to worry about much because you're only going to be using this to authenticate yourselves. In our case,
- 2:22:52
I'm using it to authenticate myself and my own particular YouTube channel. Now, once we have that done, we actually need to make sure that we go through and we actually add and create our credentials. So, here you'll notice the scopes that are available for OOTH. We specifically need the force SSL scope right here. This is the scope that is going to force us to be able to view captions specifically. That's the only reason we
- 2:23:12
need this ooth stuff is for viewing captions. Then, we can make sure we select HTTP REST and just kind of go through these steps one by one. So the very first thing we need to do is we need to make a request right here to set up our authentication. So let's just copy over this URL. We'll paste this in right now and we'll see what else we need. We need the client ID as part of this URL. So we'll say client ID and we'll copy over our client ID
- 2:23:34
directly from here. So let me just copy that and we'll paste that into our URL. Next thing we're going to need is our redirect URI. So we can come in add that in. And our redirect URI specifically for this one is going to be localhost 3000. Let's make sure I get the HTTP part in front of it. There we go. Next thing that we need is our response type. So, this response type is going to be
- 2:23:56
code. That's specifically what it tells us to use inside of the documentation. So, we'll say response type equals code. Let's see what else we need to specify what scopes we want. So, we can say and scope equals. And we know we want that force SSL. So, let's just copy over that scope. Whoops. Paste that down. Next thing, we can keep going. We want our
- 2:24:16
access type to be offline because we want to be able to access this no matter what. So we'll say access type is equal to offline. There we go. And I think that's probably everything. State we don't need. We don't need the these. Yeah, we don't need any of that additional stuff. So let's just make sure we copy this URL in case we messed it up. Click enter and hopefully it should ask us to authenticate. Now once
- 2:24:37
you click that you want to authenticate, you'll see this warning. Just click continue. That just means that this is an unverified app. And then click continue. And now we have access to essentially that entire YouTube account. And it's going to redirect us to a page, localhost 3000, but specifically, you'll notice this code section. Copy this code directly out of the URL because that's what we're going to need for the next step. I'm just going to paste that down here so we have access to it. So now we
- 2:24:58
need to redirect them essentially. So you can see here we want to sample redirect to Google's OS servers. That is essentially that thing that we just did. Step three, Google prompts the user. That's what we just filled out. Now we're on step four. We're going to get that response back with a code. And now we need to use that particular code. So here is where we make a new request. I'm going to copy this. This is how we actually get our token. So we're going
- 2:25:18
to paste that in. And now if we go over to our token section, we need to pass along our client ID, our client secret as well as the code. So we can say here client ID is equal to. And let's copy that over just like that. Copy that there. Client
- 2:25:39
secret. Make sure we copy that as well. There we go. And we need our code, I believe, is all it is called. Let me just make sure. Yep, code. And our code is just this. So, we'll copy that in just like that. Now, hopefully if we make this as an actual response. I believe it needs to be a post request. Let me just see.
- 2:26:00
Uh, yeah, it needs to be a post request. So, let's just do that inside of our editor. So, we can come into here. We can go to that workflows. request file that we created and we can just create a temporary post request to that particular location. Give that a save, send that request and we're getting unsupported grant type. So it looks like we're missing something. So let's see here. Grant type field must
- 2:26:22
be set to authorization code. So let's just add that in there. And grant type is equal to that authorization code. Now if we send it missing parameter redirect URI. So let's just make sure that is in there. redirect URI. That is our local host.
- 2:26:42
There we go. Now, if we send our request, make sure I save that. Still says an invalid grant. Most likely, this just means our code expired. So, I just created a brand new code going through that exact same process. And now, if we send, you can see we get access to our access token and our refresh token. And we specifically want to copy over this refresh token and use that in our application. So, let's just paste that
- 2:27:03
down right here. So, now that we have that done and it looks like our AI is still working, let's go ahead and go to our Code Rabbit review for adding the UI and see how that looks. So, you can see we get the nice walkthrough and the review change set. We've already reviewed everything. Let's look at our premerge checks. You can see title's too generic. Yeah, we don't really have to worry about that. Let's actually see what it says we should be able to fix. So, we look inside here. We have some nitpick comments as well as some
- 2:27:24
additional comments on all the things that need to be changed. So, let's just kind of go through these one by one. So, the very first one here, this one is a major change. It says um add serverside session enforcement to search content. The session check in app page only hides the UI. Unauthenticated callers are still invoke the server action trigger embedding generation plus content queries reject unauthenticated request.
- 2:27:44
That makes a lot of sense. I don't know how I didn't catch that. But if we were to look inside of our action for search, you'll notice it doesn't actually check to make sure our user is authenticated anywhere inside here. So that is definitely something that we should be fixing. And we will just probably use this prompt right here to make that an actual fix. Here we have another thing. No timeout on embedding call. Uh calls
- 2:28:06
an extra provider with no timeout. That's not really that big of a deal in our particular case. So we can just ignore that one. So let me just minimize that. I'm going to ignore that. This one right here, limit 20 is applied before relevance ranking. So top matches can be dropped. This is actually a good catch. Essentially, if I have more than 20 results being returned since they're first ordered by content ID, it's going to limit it those 20 based on content
- 2:28:26
ID. It'll still only be the results with greater than 50%, but it could miss some of the better results. So, this limit should be removed and instead the limit should be added to this sorted results. That's something super easy for us to be able to fix. I will do that probably ourselves. And then finally, possible stale result overwrite on concurrent searches. Has no request sequencing. Yeah, that's fine. I don't really care too much about that. If it returns some stale data, that's not a big deal and
- 2:28:48
pretty minor result. We're not going to worry about fixing that one. So, we'll minimize that down. Looks like we had one more here. use a constant time secret check for the off header. That is not a huge deal for our particular use case. We'll minimize that one as well. And we can go ahead and we can check the nitpicks if we want to to kind of see some of the smaller stuff. Enforce a minimum length on cron secret. Don't care about that. Duplicate GitHub
- 2:29:09
sign-in button. That's fine enough. It's just the UI. And wrap the block. Yeah, that's also completely fine. I don't really care about that. Split text is declared with cons directly under blah without braces. Yeah, okay. I don't really care about that. I generally kind of care about that, but for this particular use case, doesn't really matter. So, let's check on our AI. See, it's still working on some code. So, let's go ahead and see if we can just implement some of this stuff ourself.
- 2:29:30
For example, this limit 20, that should be an easy thing to fix. And this AI should never mess with that file. Normally, again, I would create a new GitHub branch to work on this. But for this particular video, I'm just going to go ahead and work on it in one particular place. So, let's just close out of a lot of these. I can go into our workflow rest, and I can remove this entire post request. There we go. since we're not using that anymore. And then
- 2:29:51
what I want to do is go into that search action and I want to make sure that this limit is no longer being applied here, but instead I can just slice 0 to 20. Just like that. That's going to give me my top 20 results. That's essentially solving that particular problem. Next thing I want to make sure that the user actually exists. So I can
- 2:30:11
say const user equals get user or get current user. Actually it's offi get session. There we go. That's what I was looking for. Let me just make sure I import off. There we go. And I want to get the session. And I can pass in my headers
- 2:30:32
and that comes from await. And we're getting headers from next.js. There we go. That's going to give me my user. And then I can check to see if the user exists. So I can say if user then I can just return some type of error. Doesn't really matter what it is. We'll just return an error or we'll just return empty results. There we go. They should never even get to this point
- 2:30:52
because the search doesn't actually exist. So we'll just return an empty result if for some reason the user doesn't exist. Now, while we're still waiting for the AI to finish, I'm actually going to take all these environment variables. I'm going to bring them directly over into our production environment. So we click add environment variable. We can just paste this down. it'll add all of them for us. Again, these are all secret keys. Normally, I wouldn't share these between development and production, but because
- 2:31:13
it's such a pain to create the YouTube credentials, I'm just going to share them particularly just for this video's use case to save you some time. So, now we can click save and we have all of those different environment variables added and we have all the environment variables we need so far inside of our production environment, which is great. We can check on our AI and it looks like it's still finishing up what it should be working on. So, I'll come back to you once this is done. So, it looks like it
- 2:31:33
finished up and we can kind of take a look at everything that it's created for us. So it looks like here it created the workflow file for us. That's great. Added the path here for the for cell. We got all of our different APIs installed or libraries. Sorry. We have our search. That's the stuff that we actually added oursel. Again, I normally would have done it in a different uh request, but
- 2:31:55
that's perfectly okay. We have that route, which is correct. Server env uh that looks like it's correct. Not sure why it's got the YouTube playlist ID in there. Let's see what it's putting for that. Um, yeah, it doesn't have anything in here as a default value. That's perfectly okay for now. Let's actually look at the file though because this is where 99% of the changes are.
- 2:32:18
So, we got our chunk size of five or 15, overlap size of three. Okay, here's our start. So, get new videos from playlist. Get playlist items from the YouTube playlist ID. So, if we take a look at that, we can see here it's Google.youtube. That's perfectly okay. So, this isn't 100% correct because I don't want a
- 2:32:39
hard-coded playlist ID. I wanted to just get all the latest uploads. To do that, we actually need to query the YouTube API for that. So, we can just come in here. We can say YouTube dot and we want to get the channels. So, we can say channels.list. And I specifically want to get a part here which is the content details.
- 2:33:00
And I just want to get mine. So we'll say mine true. This is going to be our channel details. Just like that. And then we can take our channel details. I want to get the data specifically. I want to get the items from that. I want to get the very first item because that's we're only getting one item or one channel. So we can just get the very
- 2:33:22
first item from there. Then I want to get the content details from that. I want to get the related playlist and I want to get the uploads. Just like that. There we go. And it looks like we have a potential error. Items is possibly undefined. There we go. Fixes that. Now we have our uploads playlist ID, which is what I'll
- 2:33:43
use here. And we'll get rid of this playlist ID. That's not something we need. Not something we need here. Not something we need here. And in our server env. We don't need this playlist ID. Okay, that cleans up all that. Now we can go through. We can see that they are getting a max results of 50. We
- 2:34:03
got our page page token. So this should be doing our pageionation in a dowh loop. That's perfectly fine. Get all the items. And for each item, we're just getting the video ID and pushing that into here. Now I believe we have the ability with this playlist items to get more than just the ID. We can get things like the title, description, and so on. I wonder I'm going to kind of keep going
- 2:34:23
to see where they get the title, description, idea. So this is where they're doing that title description section. I think honestly we should be doing that when we do the playlist items that shouldn't contain the same amount of content. So here in this snippet section we can get the content details as well as the status cuz I only want to get public v videos. I don't want like private or unpublished videos
- 2:34:44
to show up in this list. So that's going to get all that information. And then we don't need to just push the video ID. We might as well just push the entire video into this particular section. So here I can just essentially do that. That gives us our items and I can say all items do let's say push items just like that. And we can get rid of all of that code right there. So
- 2:35:04
that's going to push all of our items into this all items array. And this all items array is essentially going to have a playlist item. So I think if we use YouTube dot and then we have or underscore v3. Let's import that. Whoops. There we go. It actually autocorrected for me. And let's just make sure we import this just like that. So that's going to give us the correct
- 2:35:25
type for this information. And then we'll return from here. Again, I don't like hard-coded return types, so we'll remove that. There we go. That should hopefully give us all that information properly. Now, I again only want to get the new ones. So, let's make sure that it is doing that properly. So, we have wherever that was, our get playlist items. Then, we're getting our existing URLs. And I believe that should follow the same pattern we did before where
- 2:35:47
article and columns and get URL. Yep, that's exactly the same. And we're just filtering out to make sure it includes and we're just getting that URL just like that. Okay, that does look like it's working exactly as we expect. So I can at least go ahead. I can minimize this section and I can minimize this section cuz those are the correct thing. Create oath 2 client. That looks like it is probably correct. Yeah, we're setting
- 2:36:08
our credentials. So we can minimize that. That is good. And I believe now that we have the videos, now we have our new video and we are on the invest ingest video step. This should take in a full video. And this is going to be that YouTube v3 playlist item. There we go. Also, in my get playlist items, I want to make sure wherever I'm doing my
- 2:36:28
items, I only want to push in the items that have the status of public. So, I'll say item item status is equal to public. There we go. That way, it's only going to give me those public ones. And we want the privacy status. Perfect. So, now let's go ahead and make sure everything else is correct. So, we have our video coming in here. And if we just do a quick search
- 2:36:50
on that, yeah, we get all those different pieces of information from our video. And we essentially want to get a parsed version of that for our schema related stuff. So if we look all the way up here, we have that schema video ID, title, description, published at. This should be a URL. There we go. All of that seems to be correct. So let's see where this is used at. It's
- 2:37:12
used right here. Let's take that. I think we want to move that to the top of this function. There we go. Get our video item and we're going to safe parse. Our video ID is just our video ID. There we go. Title, description, publish at, and so on. And we don't even need default values for these. There we go. And then down here where we
- 2:37:33
originally were doing that, let's get rid of that. Take our error. Move that right there. Then I can get rid of that here. Perfect. thumbnail URL should be coming from those thumbnails. So yeah, this all we can kind of get rid of a lot of this. We don't need to re-query for the video because we've already gotten it. So that can be gone. This entire thing can be gone. There we
- 2:37:55
go. The captions is something obviously we still need to care about. And other than that, this is looking really good. There we go. Caption response. I think I'm incorrectly using my video here. I should be using my video item. So, let's make sure everywhere else it's video item. Video item. There we go. And
- 2:38:16
I don't know why this is called video ID. We're just going to call this ID. Scroll up here. Call that ID there. Now, wherever we were, looks like I lost my place. Playlist items. Yeah, we can minimize that one. Ingest the video. Video item. Video item. Perfect. Caption track is just
- 2:38:37
trying to get all the tracks. Then we want to find the first English track. So here we go. We're getting an English track. That's the first one. There we go. We don't want to fall back to anything. So get rid of all this fallback stuff. We definitely don't want that. There we go. Get the caption ID. Whoops. Caption ID. There we go. And as
- 2:38:59
long as we have a caption ID, then we need to do something with it. And then we'll throw a fatal error if we can't find it. That makes a lot of sense. And then we can parse the caption. So let's see what's happening in here. We're going to download the captions as SRT. We don't need this as const. I don't think we can get rid of that response data.
- 2:39:23
Oh, so it's giving us an interesting type being returned as our download response. I think we can clean this up because the library has a response type property we can specify as just text. That's going to make sure it always gives us text back. And then we just know that this data type here is a string. So we can essentially clean this up drastically because now we know that this response
- 2:39:44
type is always going to be a string. There we go. And to clean this up even further, we'll say if our caption ID is equal to null, then that's where we're going to throw our error. So we're going to throw our error inside of here. Close that off. Get rid of that. There we go. And now we have our response data, which I called SRT content. Just like that. Don't need the
- 2:40:04
SRT content up here. Perfect. That all looks really good. So now we can move on. This is going to parse it. So this is just calling parse sync from that library that's being installed. Just doing all of it for us and returning us these nodes. And then we want to do some filtering on that. Now this function is much more confusing
- 2:40:25
than it needs to be. We can get rid of this entire is typing and this whole is data thing right here. And essentially it's just going to make sure it only gets the one that are Q type. Q's are essentially the spoken word coming from this library. So that is what we want. Then we're going to map it to give us the text and the start time because those are the only two things we care. And we want to make sure that we actually have text. So that's just filtering out any empty rows as well. If
- 2:40:45
there's no valid captions, throw an error. That's great. And now we need to create those chunks with an overlap. So this is just collecting all of our different chunks with that proper overlapping. We're then just calling embed chunks. So let's just see what that does. Oh yeah, we've already created that function. So that's just embedding that content. And now we're just saving everything. So, we're taking all of that content, saving it. That all
- 2:41:06
looks correct to me. Um, then you have duplicate detected. Then we're looping through each one of our chunks, inserting them, and making sure that we get our start position set properly. That also all looks correct. So, I think all of the code for this is correct. And that's everything, I think. So, we can actually kind of test to see if this
- 2:41:27
actually works. Let's just close out of everything. Minimize this down. And let's just do a quick test. Actually, before we test, I want to make sure in our workflow for this, we are doing a batch of 10. Yeah, the batch size is only 10 and there's a 9h hour delay just because of how aggressive the quota is for all these different things. So, let's just go ahead and test to see if that works. We'll come into our
- 2:41:48
workflows, send a request to that connection was rejected. Of course, I don't think I'm actually running our commands. And let's just get our workflow set up. There we go. Now we should be able to run this. Hopefully it'll return a result over
- 2:42:09
here. Server error, of course. Looks like there's a ENV variable missing most likely. Google client secret. I think I just called it something slightly different. Let's see here. There we go. Let's just give that the correct name. And now we'll retry that request. Unauthorized is what our request is this time. Let's see what it's giving us as the error.
- 2:42:30
Um, yeah, just a straight up 401. Oh, it's probably because our cron secret is maybe not set. Let's go into our actions or apps. Sorry. API workflows here. Verify and run cron. Verify and run cron. Yeah, we're just not setting our actual cron. So, if we take a look at that verify and run cron,
- 2:42:51
we need to have a bearer authorization header. That's what we need. So inside of here, authorization is bearer. And we want to use that cron secret. And to get this value, we can just do this. That's going to give us that actual cron secret. Make sure this has a dollar sign in front of it. We'll copy that to both of them. And
- 2:43:12
now when we send this, we should actually see it is starting up that cron job. And if we go to our workflows and we pull this over, we can actually see that we have this ingest video step that is running. So it is actually working like we expect. It actually looks like it is cued. It's now executing this. It was able to get the videos from the playlist and the output is 866 videos. So, I have created quite a few videos in
- 2:43:34
my past. But now you can see it is actually properly getting our videos and saving them. And now it's doing a sleep command. And this sleep command is going to last 9 hours. So, I'm just going to cancel this cuz obviously we're not going to wait 9 hours. But we at least should have 10 videos inside of our database. We can very easily test this by just running npm run db studio. That should open up real quick.
- 2:43:57
Here we go. And now we should be able to check inside of here. If we go to our content section, we should hopefully be able to filter quickly by a type. There we go. Filter. We want to filter by the type is equal to video. Let's apply that real quick. And now you can see we do have 10 pe 10 videos inside of here with all the different chunks
- 2:44:17
associated with them. So this did work exactly like we expected it to. So now that we have that done, let's go ahead and add these. So we'll just say get add get commit add YouTube ingest. There we go. And actually before we even finish committing all this, we should probably implement the UI for
- 2:44:38
this as well. So let's actually test to see if the UI for this works. So I just did a really quick search. You can see here I just searched for sandbox because I know I have some videos on sandboxing. You can see they're all popping up right here. This second number is obviously incorrect. It's doing some rounding incorrectly. So, we should probably fix that. So, whenever we have I think it's just s and then it's got some text in
- 2:44:59
front of it. So, that's going to be kind of hard to search for. Let's just go to app page. It's probably in the component search results. Here we go. And we specifically want to find it. Here we go. The start position. So, the start position, I'm guessing, is in milliseconds. I don't actually care about rendering out the start position. So, we can actually just remove that from our results. But down here, I want to make sure to include that start
- 2:45:19
position in the URL. And I believe when it comes to YouTube URLs, all we need to do at the very end is to add t equals and then whatever our start time is. And we'll divide that by 1,00. Let's just see. Looks like we have an error. Unreachable because the left hand is never nullish. I don't think it's called start time. It is start position. There we go.
- 2:45:40
And we'll just wrap that in parenthesis. And divide that by th00and since it looks like it was in milliseconds. and we need to convert this to seconds. So, let's actually see if this works. We'll just do a refresh. We'll just search for sandbox. We'll click on this and hopefully it brings me to that exact location in the video. And it looks like it did not. There was definitely some type of error going on. Let me just render out that type as well. We'll just
- 2:46:00
render out a span here real quick, which was result. Position, just so we can see what that start position looks like again. And you can see it's just this really long number. It definitely seems way too long. Like even if I divide this by a th00and that is going to be 43 seconds into the video which maybe is correct. So maybe dividing by a th00and and let's just round it actually. So we'll just do a
- 2:46:22
quick math.4. There we go. And we'll make sure that that's divided by a th00and. Let's see if that actually fixes the problem. So we can click on this. You can see we got our time up here and it should hopefully start us at that location. But it looks like it did not. I believe I know what the error is. The error is that we just need to put an amperand here because it already has some query parameters on the
- 2:46:42
end. So now let's get rid of that result text that we added. Let's refresh this. Search for sandbox. Click on this. And hopefully it actually brings us to that exact location. And you can see it does bring us to that location in the video. Looks like for some reason it's bringing us to the very end of this video. Maybe not quite where we want. Let's try another one to see if it works properly. This one's bringing us to 1244, which is
- 2:47:03
bringing us to this location in the video, which is probably much more accurate. And let's try this one as well to see if this one is working. And yeah, 43 seconds in the video. That is probably fine. Don't know why this one is at the very very end, but that's probably where I said the word sandbox. So that's probably why it's bringing me to that location. Now, kind of the final thing that I want to do now that we have all of this done, let's push all that up. Do get add, get commit,
- 2:47:25
fix UI. We'll push that up and then we'll be able to merge that in once all of that is done. But we can just do a quick check out and we'll just check out a branch called rate limit. Make sure that we add in a dashb at the front. And I just want to add in some really basic rate limiting. So, we'll just create a prompt for that real quick. So, I just created a rather simple prompt for this one. Essentially, I told that I want to add rate limiting to the search action
- 2:47:47
using specific enviables for how many requests they can have per hour. Also, I kind of explained how that's supposed to work. I said that if it's rate limited to send an error back that the UI handles, I said to track the user queries in a database table that just tracks all their searches. This includes things like the text they sent, the user ID, as well as the results that are returned, and most importantly, the time that this search was created. And then
- 2:48:09
finally, every time a query is made, make sure to save that query in the database in that table and use that table for rate limiting. So we're just going to send this along. It shouldn't take too long because this is a relatively small task. And while this is happening, let's go ahead and actually create a pull request for that brand new branch that we just passed up. So here we are. We can just compare and pull request all of our code. Make sure that we want to create a pull request. And this is going to go through. It's going
- 2:48:30
to make sure that code wrap is going to run on this. Forcel is going to run on this. Pretty much everything that we want is going to run on this particular code to make sure that everything works like we expect it to. Again, our deployments aren't working because we don't have anything for that database URL set up, but otherwise, everything else should work just fine. Now, while we're waiting for this to finish up, we might as well go ahead and actually merge in the pull request for this ad UI. This should have been merged a while
- 2:48:50
ago, and that may actually cause some duplicates between our different code rabbits, but this should definitely be merged. So, let's just go ahead make sure that we scroll all the way to the bottom, merge in this pull request, which definitely should have been merged in before. That was my bad on not merging that in. We'll delete our branch for that particular one. And now at least we have that all merged in for our code. And now we only have the one pull request which is just for that YouTube
- 2:49:11
ingest related code right here. And again, it's going through with Code Rabbit. And we also have our AI working here behind the scenes to get all of our different search related information done. So looks like our AI just finished writing up the code for the search results stuff. So let's just take a look at that real quick. Let's go to our schema first cuz that's going to kind of determine how everything else looks. Again, we don't need this text here. So we can get rid of the name of these things and we can replace this with
- 2:49:33
snakecase.t wish it would just do that automatically on its own. Other than that we got our text our user ID which should definitely be a reference. So we're going to say references the users ID table. Make sure we get an import on that. Looks like it's not coming in
- 2:49:54
users from and let's see. We want to go at slash. Actually, we should just be able to get it from this schema section. So, there should be an off one. Let's take a look here in our schema. It looks like Oh, of course, when I originally created my schemas, I put the off one in the wrong location. So, let me fix that
- 2:50:14
real quick. Inside of my package. JSON, wherever that is. This should be schema. There we go. I just put that in the wrong location. Now, if we got that done, we should be able to get the off directly from there. And users as well. Let's just make sure. Let's see what this is called. User. There we go. Okay, now we got the user
- 2:50:35
ID being referenced properly. And we can even just throw on like ondee cascade. There we go. Perfect. And our result content IDs is a text. This should be an array though. There we go. That's an array of data. And we don't actually need both the timestamps. We just need timestamps.created
- 2:50:55
at. So let's just make sure we have only the created app at property on there and none of the other ones. And this is called created at just like that. And yeah, this is just an index to make the search more efficient for searching specifically those queries, which is what we're going to be using this table 99% of the time for. Now, let's go ahead and take a look at our package. JSON. That's just the change I made. We have
- 2:51:16
our search action, which is where a lot of our actual changes are made. So, let's see exactly what changed inside of here. It's probably going to be easier if I just read the search action. So, let's find our action for search. Looks like there's some stuff that's not even used. So, let's just remove that. There we go. And here we go. We're checking our rate limit. That is just a function that we should probably read through. So, get the max request. That is just a
- 2:51:39
server variable. Server variable. So, not really much point having those as actual functions. I'm just going to clean that up. Get max request is that. Get the window hours. Is that There we go. Clean up the code a little bit. Get the window start. There we go. Count property count does not exist.
- 2:51:59
Yeah, it looks like there's definitely a little bit of an error here with how it's getting the count property. Let's just see what the return values are because this is returning us an array of values. So, I think I just need to wrap this inside of brackets. And now we can get the count out of there. There we go. That fixed that particular typo error. And then this is just saying okay you know rate limit is exceeded here's a particular error that
- 2:52:20
it is throwing. So that makes sense. So we're throwing an error here if there's a particular problem and then we're recording the result and probably I have something being typed wrong inside of here. So let's just take a look at our values. Let's see what our error is. Actually I can probably just check created at an ID. Those are autogenerated. Oh this
- 2:52:41
needs to be an array. So just as simple as that. There we go. That cleaned up that file and I think otherwise it looks fine. Really all we did is just add in the result query and we just added in this checking of the rate limit. So this file was simple in our page. We just here are checking our results. So we have our do search. If the error is an instance of rate limit
- 2:53:02
error then we're just setting an error. Otherwise we're setting a generic error. So that way the user knows there's a problem. That looks good to me. Server env has these values. I'm going to get rid of the default on them just so we have to manually set these values to whatever we want them to be. Database, that's fine. We changed that ourselves. Same with the schema. That's our own change. That's the O schema. We
- 2:53:24
just moved the location of it. User search is good. And then this rate limit file we just looked through. So all of that actually looks pretty good. There was pretty minor changes overall that needed to be made. Let's just go ahead make sure our project is running. I'm going to pull open local host right here so we can just test it out and let's see what our default enviables are set to. We see here we have max of 10 requests
- 2:53:45
per hour. So let's just go ahead and test that out. I'm going to make a bunch of different requests. We'll say one and of course failed because we need to push our database changes. So let's go ahead and do that. We can say here npm run or yeah run db push. That's going to push those different changes up. And I also probably should run.
- 2:54:06
Oops. Let's see here. Renamed. I didn't rename anything. Chunks embedding index. Okay, we'll just say that that's a new one. Oh, of course, because I have my chunking schema set to the wrong one. So, let's fix that real quick. So, in my chunks here, if you remember correctly, I set this. We need to change it back to
- 2:54:29
the dimensions that we're using specifically with Quen. That should hopefully fix our particular problem. That did look like it applied it. I'm going to change that back to what it was before, though. And I'm going to run mpm run db generate just to generate our migration file as well, which will include all those new changes we made. So, if we open that up, you can see it creates that user searches table and adds in all the stuff that we need for
- 2:54:50
that. So, now with that done, let's actually test our search. We got search one. Let's try search 2 3 4 5 6 7 8 9 10 and this is number 11. And you can see rate limit exceeded. You can only make 10 searches per hour. So that works exactly like we expected to. So this is
- 2:55:10
a relatively simple and easy fix. So let's just go ahead and add this. We'll just say get add get commit add rate limit. And we'll push that up. And now we can actually go ahead and we're going to check out the code for our YouTube ingest. Just like that because our code rabbit review is done. So now we can actually work on this. So the very first
- 2:55:30
thing that we see here is await the session before the off check. This is actually a huge security vulnerability. I can't believe I didn't catch this. So in our search content. So if we just go ahead and we just do search content and it was right here. We just need to make sure that we await this off check because otherwise the user is going to return to us a promise. That is a massive potential security
- 2:55:51
vulnerability. So, at least we have that fixed. That is good. Next thing that we need to do right here, uh, the enter key submission is engaged by is loading, allowing overlapping searches. And not really a huge problem. I'm too not too concerned about that. Uh, duplicate orphan search icon markup. Uh, this is also fine with me. It's just a UI thing. I don't really care too much about the UI on this application.
- 2:56:13
Uh, again, this is an error we've already seen before, so we're going to get rid of that one. Filter never excludes already ingested video. ID is a full playlist item object, not a video ID. Interesting. Let's see exactly what this is talking about. Returns the item export async function blah blah blah blah blah. So each ID is a full item object interpolated into the template. Oh, okay. So um that is in the get new
- 2:56:35
videos from playlist function. So let's just go into that function. And if we look here, this ID object is not just an ID. You can see it's a bunch of different stuff. We need to specifically get I think the ID from the ID. Let's just make sure that is exactly what we want. Let's see the proposed fixed here. It actually tells us to get the content detail video ID which is probably
- 2:56:56
actually correct. So we can say here that we want to get the uh so video ids is ID.content details video ID. There we go. And I should probably call this like V for video. That actually solves that particular problem. So again a huge problem that we didn't actually realize was a problem. Next thing we have our ingest YouTube videos missing optional chain causing
- 2:57:16
crash on empty items array. Um okay if items results to an empty array is undefined. So essentially let's find where that code was. I believe it was in here. Yeah this essentially should just have a question mark in front of it in case the items is null as well. Solve that particular problem. This one says error
- 2:57:37
message references the wrong ID field. Again that's the exact same thing. So we can say video id. Let's just search for all those locations. video.content video ID. There we go. That solves that problem. Again, it's just a misunderstanding of how the API works. Wrap content puts chunk inserts in a transaction to avoid orphaned
- 2:57:57
content rows. This one I could do, but I'm not too worried about it. We're just going to leave it as is. And that's all of the different things that it says that we should fix for this. So, let's go ahead and actually just commit those. We'll just say bug fixes again. Use better naming. We can push that up. And now we have all those different changes done. Let's go ahead and we'll just merge those changes in. Give it a second
- 2:58:19
here for it to load. Confirm that merge. And there we go. We will delete that branch. So now what we can do is we can go to our very last pull request for this. So let's just create a pull request for our rate limit. You can see it's trying to merge it into main. It says it actually can't automatically merge because I have some changes that are being made. Let's make sure it still
- 2:58:40
can't actually merge. Give it a quick second. Yeah, it can't merge because of conflicts. Let's just resolve those conflicts. So, we're just going to get checkout main, get poll, then we're going to check out rate limit and we're just going to merge in main just like that. Let's take a look at our conflicts that we're getting. So, inside
- 2:59:01
of search, we just have a conflict here. This is using an await. This is using an await. So these are actually already being fixed potentially for us. So we're just going to stick with the one that's incoming. Just like that. And actually it should be called session. There we go. Because that's what it's being called other places. There we go. And now we can just add those changes.
- 2:59:23
There we go. Perfect. Now let's push those changes up. And hopefully once we do that, we should now be able to merge these changes in as we see fit. Give that a second. Let's just go ahead and merge that right now. Give it a quick second to load. And we can delete that branch as well. And now with that done, we should be able to go to Verscell and actually see that everything is working like we expect. So here we are on our
- 2:59:45
deployments for Versell. You can see right now it's building this. I just can cancel this one. I don't care about these preview ones. Yes, we'll cancel that deployment. I care most about this one right here, which is going to be the pull request specifically that's going to be to production. So let's make sure that this works like we expect. So looks like we have a failure. We're just missing some environment variables for rate limit as well as the Google client
- 3:00:06
secret. So let's go ahead and set our environment variables. We will say that our Google client secret I just need to rename this uh won't let won't actually let me rename it. So that's perfectly okay. We are just going to delete this one. Yep. And then let's just add it again. So we'll say Google client secret. And let's copy
- 3:00:28
that over from our environment variable. So let's come into here our environment variable Google client secret. Copy that and we'll paste that. There we go. Give that a quick save. We also need to do the same thing for our rate limit max requests.
- 3:00:49
So we'll add that in there. Let's just say it's 10 for example. This one does not need to be sensitive. And then we have finally our rate limit window in hours. So we'll just add that. again it's not sensitive and we'll just say 10 per hour. So now at least all those different environment variables are there and let's try to redeploy this. So we'll just redeploy. Yep, redeploy that.
- 3:01:09
Give it a quick second to load and now I'll get back to you once this is done. That just finished up. So one thing that I want to immediately do is I want to go to my cron jobs and I want to run the cron job for ingesting the YouTube videos to make sure that this works in production. So now that that's running, we can go over to our deployments just to or not the deployments, the overview. So that way we can deploy this or pull
- 3:01:30
it up. And then let's go to our workflows. We should hopefully see if we just give this a quick refresh. Might take a second for it to show up, but that YouTube one should show up inside of this list. Let's make sure here that we're signed in just as a starting point. There we go. That is correct. And here now you can see the YouTube one is active. And hopefully it looks like it's working. It already is ingesting a bunch of videos and we're on this sleep step that is going to take 9 hours to finish.
- 3:01:52
But if we just search for sandbox, we should hopefully see some YouTube videos that have sandboxing. Looks like actually it didn't bring in my sandbox videos first. Let's see what videos it did bring in first. Let's just decrypt this real quick for our input. The input for this one is the containers. So that is containerization. So we'll just say containers or let's just say agentic AI container.
- 3:02:16
There we go. Now it's popping up when I do that. So you can see that it is getting my videos. And if we just search for like React hooks, we should hopefully see we get some hook related stuff showing up as well. We can also test the rate limiting. So I can just do a bunch of searches and eventually once I do too many searches, we should hopefully get rate limited. There you go. Actually an error here. Let's just make sure. Looks like there's a particular error that's being not caught
- 3:02:37
properly and this is probably that rate limit error. So we may need to look into why this is happening as well. I think the reason for this is because in production it doesn't actually give you any information about the error when you have an error in your server component. So we essentially kind of need to rewrite our rate limiting code. So let's just go ahead into our app. We're going to go into our actions for our search and instead of returning a error that it
- 3:02:57
throws wherever this is at. So it's here where it's throwing that particular error. Instead, we need to handle this in a different way. So what we could just do is we could just wrap this in a try catch. This is kind of a messy way to do it, but it's going to work for our particular use case. And here we can just say if we have some type of error then we just want to return error rate limit exceeded. And now anywhere we're
- 3:03:19
using this search content. So let's just see where we're using our search content. We have our search results here. We don't actually need to worry about wrapping this too much in a try catch. But we should see our search results could return to us an error object. So we can just say if search results and it should have an error object as the other option. Yeah, it should have
- 3:03:40
an error. So let's just say if there we go if error is in it then what we want to do now let's just swap these around. So if the search results and the error is in the search results then I want to set the error. Oops. Otherwise we will set the results. There we go. And I can get rid of all this try catch code. Don't think we actually need
- 3:04:01
that. There we go. Now let's go ahead and make sure that we push this up or test it. So, we'll just sign in real quick. There we go. Let's just do a quick search. And of course, we get rate limit exceeded. Exactly what I want. So, now let's just do a quick add
- 3:04:22
commit. And we'll push that up. And we'll create a brand new pull request for that once it actually pushes all the way up. We'll just do it this way. There we go. Great pull request.
- 3:04:42
And we're just going to immediately merge that. Generally not a good idea. You should wait for review, but for our particular case, it should be fine. There we go. Delete that branch. And now all we need to do is just wait for it to actually be pushed up to Verscell. And while we're waiting for that, I just want to give a huge shout out to the sponsors of this video. Neon is absolutely amazing. If you need a really simple, easy to use Postgress database, which is perfect for this project. All
- 3:05:03
of their branching is absolutely amazing. Also, Code Rabbit, great for that AI code review. You can see it saved us from some massive security vulnerabilities in our application that I completely glazed over. So this can save you big time on those application errors and save you tons of time as well. And if you're doing open source software, you can use it entirely for free. And of course, lastly, Versel, we have them for our hosting as well as
- 3:05:23
that workflow system, which makes working with our application super great because we get those durable retries for any potential errors and bugs. And we can have long running tasks like that 9-h hour wait without having to worry about spinning up denser compute and so on. So, I want to give a huge thank you to Neon, Versel, and Code Rabbit for sponsoring this video. You'll find links to all of them in the description down below for you can try them out. All of them. I'm using the free tier for this
- 3:05:44
project. So, you can see you can do quite a lot with just the free tier of these different services. Now, with that out of the way, we should actually have our deployment hopefully done at this point. You can see it's currently building it right now, and I'll get back to you once this finishes. So, we can finalize checking that rate limit. Looks like we have a little bit of an error in our TypeScript code. This is just from me trying to go too quickly with things. Looks like it's way at the bottom here. Results.length. And this is probably
- 3:06:04
because results could be that error object instead. So we just need to make sure that our results is not the error object. So if results, we'll just say if is array results and results.length is greater than that. There we go. That should hopefully solve that particular error. And now we can actually just push this
- 3:06:25
up. Hopefully that's the last error that we have. Fix error. And I'll get back to you once this is I'll merge it all in. So, the deployment just finished and since we were rate limited before, as soon as I search, I should get that rate limit error. And you can see rate limit exceeded. So, everything is working like we expect it to. Again, I highly recommend that you check out Neon, Versel, and Code Rabbit
- 3:06:46
rabbit. They're going to be linked down in the description below. With that said, thank you very much for watching and have a good day. Also, if you want to check out this application, I'll have it linked down in the description if I get around to posting it. That way you can actually search through my content and find things that you want to
Description
SponsorCheckout Neon: https://get.neon.com/HALyXNV
Checkout Vercel Workflows: https://vercel.plug.dev/wMxDguF
SponsorCheckout CodeRabbit: https://www.coderabbit.ai/?utm_source...
RAG (Retrieval Augmented Generation) is crucial to nearly every business. Knowing how to build your own RAG system will put you far ahead of most other job applicants. In this video I will not only show you how to build a complete RAG system, but I will also show you how go about building production ready applications with AI from scratch.
📚 Materials/References:
GitHub Code: https://github.com/WebDevSimplified/v...
AI Sandbox Setup Video: • Containers Don't Make Your AI Agent Safe
AI Skills Video: • How To Use AI Skills Like A Senior Developer
Local AI LM Studio Setup Video: • The Best Local Agentic Coding Workflow (Co...
🌎 Find Me Here:
My Blog: https://blog.webdevsimplified.com
My Courses: https://courses.webdevsimplified.com
Patreon: / webdevsimplified
Twitter: / devsimplified
Discord: / discord
GitHub: https://github.com/WebDevSimplified
CodePen: https://codepen.io/WebDevSimplified
⏱️ Timestamps:
00:00:00 - Introduction
00:02:17 - Project Setup
00:08:23 - Project Planning
00:13:27 - Drizzle Setup
00:20:59 - Neon Setup
00:24:14 - Setup BetterAuth
00:44:46 - CodeRabbit Setup
00:47:51 - CodeRabbit Review
00:52:14 - RAG - Ingesting Blog Articles
01:23:17 - RAG - Article Chunking
01:32:13 - RAG - Embedding Text
01:40:00 - Vercel Workflow Testing
01:49:17 - CodeRabbit Local Review
01:58:03 - Search UI Implementation
01:59:49 - Securing Cron Jobs
02:03:29 - Search UI Continued
02:18:54 - RAG - Ingesting YouTube Videos
02:20:33 - YouTube API Setup
02:27:04 - UI CodeRabbit Review
02:31:32 - RAG - Ingesting YouTube Videos Continued
02:47:17 - Rate Limiting
02:55:25 - YouTube Ingest CodeRabbit Review
02:58:25 - Rate Limit Continued
#AICoding #WDS #RAG
More from Web Dev Simplified
×2244:57The Best Local Agentic Coding Workflow (Complete Guide)
Web Dev Simplified1.1M views
×9.72:10Please… I Need Your Help
Web Dev Simplified459K views
×6.91:15:25The Ultimate MCP Crash Course - Build From Scratch
Web Dev Simplified327K views
×4.925:15I Will Never Use Prettier or ESLint Again
Web Dev Simplified231K views
