It’s been a while but here’s a new post. Probably simpler and more serious than before now. I am now employed so I guess something has shifted to be less whimsical.
Revisiting my FAA/Flight AI app that I was working on. Things are a bit simpler now. I removed the SaaS features I had built in, not necessary and I do not care about that right now. So no users other than myself. I used to save money by testing locally on local models but scrapped this to just have local and ‘prod’ run same cloud models (VoyageAI for embedding, Sonnet for output, GPT5 for evaluation) b/c local model performance is so different from frontier and I can pay a couple bucks a month for cloud models. I previously had started w/ the Pilot Handbook of Aeronautical Knowledge (PHAK) but have chose to switch focus onto Parts 60-109 of the FARs (that is one document in eCFRs which contains parts 61 and 91 which is most regs I care about). I plan to expand to more FARs and to handbooks, ACs, and anything not FARs but making sure FARs are queryable and accurate has been a much smarter decision.
At the end of the day even with theoretical knowledge the law is the law. It would be good to always ensure that an answer is grounded onto the law. Already there’s been good signs that this is working well, which I’m pleasantly surprised by.
Avg Precision: 0.92
Avg Recall: 0.93
Avg Grade (LLM as Judge): 4.49/5
Most questions were not human authored but were compiled by myself from Reddit and then variations created by Claude. Answers were created by claude by using the proper reg and generating an answer. I probably need to make these answers myself so there isn’t a question about if they are correct but for now I’m okay. I’ll probably want to tune the output as well to not be as verbose, again something for later.
The only real difference I’ve done between now and a few months ago was switch embedding models from text-embedding-3-large to VoyageAI’s voyage-4. From a vibes based perspective voyage-4 is significantly better at findings sources than text-embedding-3-large. I’ve also added logic to find more regs, most of it basic like regex’ing for regs in query and top k results and adding them to output model’s context. All of this seems to have made searching over regs pretty solid.
I’m gonna pause on embeddings other than running evaluation over voyage-4 just for fun. I want to think about how to add in more sources, e.g. PHAK so we can go beyond what’s the ‘letter of the law’. Biggest thing I’m thinking about is how to properly get and feed this information to output model. I’m not certain that a raw embedding + semantic search is going to cut it, but even if it is I’m worried about how a model understands the info presented. If there’s a query about thunderstorms, it might say to stay 20 nm away and give some information about a thunderstorm but does it understand a thunderstorm and the dangers? I can’t quite phrase it and probably need examples but I don’t want an AI that just regurgitates information. I love that it can do that now w/ regs but I’m certain if I poked at it w/ 2-3 follow ups it would be clear how surface level its information is.
My end goal is a knowledge graph. An incredible knowledge graph of general aviation. So if it gets a question about airspace it can tell you ANYTHING about airspaces, not just the letters on a page but a real deep knowledge of airspaces. And then a UI. A beautiful UI that lets you look at linked information and a visualization of the graph and how different parts of aviation knowledge link. Maybe like wikipedia a little in terms of linking sources on text but the graph will be more unique.
Bunch of UI improvements I want to make too. Have in mind a way sleeker design, obviously “chats” will come but only once source finding is accurate, and showing sources outside of PDFs. I did PDFs so users can see theirs but man if I want to link a reg I don’t actually want to link an 800 page PDF that already looks weirdly formatted.
Excited to go and continue to work on this. It’s a bit slow at times and hard to focus because it’s uncertain at times but progress is being made and I have the vision, it’s just about experimenting, roughing out the edges, and then executing.