Just looking at the README, I'm very excited as I have a Hackathon coming up and I am looking to bring more type love to a legacy Django codebase, and was going to utilise MonkeyType again, but this time I'll use this!
You can adjust the code on the page easily (it’s open source javascript) to determine the question you are after, which is a valid one: if an adversary starts today and adds x nodes to the existing network, what is their success rate?
BUT the author asked a different (but valid) question: assuming the adversary controls x out of N existing nodes, what is the success rate? I am unclear: is the assertion that everyone’s relay is honest today? From a privacy standpoint, that’s not a great assumption.
No, the author is presenting an idea that $25 a month can buy you a node. That fits adding a new node to the network, not taking over an existing node.
The Tor Project web site makes a bold claim to its users:
"Tor Browser prevents someone watching your connection from knowing what websites you visit. All anyone monitoring your browsing habits can see is that you're using Tor."
Don't misinterpret this claim. It's not true that Tor protects you against "anyone monitoring your browsing".
On this web page, I provide you with the critical information missing from the Tor Project's website: if you estimate your adversary's resources in dollars, I'll estimated the probability that Tor will fail to protect you.
You should check out ChatDBG project - which AFAICT goes much further than this work, though in a different direction, and which, among other things, lets the LLM drive the debugging process - has been out since early 2023. We initially did a WinDBG integration but have since focused on lldb/gdb and pdb (the Python debugger), especially for Python notebooks. In particular, for native code, it integrates a language server to let the LLM easily find declarations and references to variables, for example. We spent considerable time developing an API that enabled the LLM to make the best use of the debugger’s capabilities. (It also is not limited to post mortem debugging). ChatDBG’s been out since 2023, though it has of course evolved since that time. Code is here [1] with some videos; it’s been downloaded north of 80K times to date. Our technical paper [2] will be presented at FSE (top software engineering conference) in June. Our evaluation shows that ChatDBG is on its own able to resolve many issues, and that with some slight nudging from humans, it is even more effective.
Is the benefit of using a language server as opposed to just giving access to the codebase simply a reduction in the amount of tokens used? Or are there other benefits?
Beyond saving tokens, this greatly improved the quality and speed of answers: the language server (most notably used to find the declaration/definition of an identifier) gives the LLM
1. a shorter path to relevant information by querying for specific variables or functions rather than longer investigation of source code. LLMs are typically trained/instructed to keep their answers within a range of tokens, so keeping shorter conversations when possible extends the search space the LLM will be "willing" to explore before outputting a final answer.
2. a good starting point in some cases by immediately inspecting suspicious variables or function calls. In my experience this happens a lot in our Python implementation, where the first function calls are typically `info` calls to gather background on the variables and functions in frame.
Yes. It lets the LLM immediately obtain precise information rather than having to reason across the entire source code of the code base (which ChatDBG also enables). For example (from the paper, Section 4.6):
The second command, `definition`, prints the location and source
code for the definition corresponding to the first occurrence of a symbol on a
given line of code. For example, `definition polymorph.c:118` target prints the
location and source for the declaration of target corresponding to its use on
that line. The definition implementation
leverages the `clangd` language server, which supports source code queries via
JSON-RPC and Microsoft’s Language Server Protocol.
This is in the works :) In the past and still today, OpenAI has much better and easier function call support, something we rely on.
We currently are running through LiteLLM, so while undocumented in theory other LLMs could work (in my experience they don't). I’m working on updating and fixing this.
Nice UI. We started on a project that does this about 2 years ago. ChatDBG (https://github.com/plasma-umass/ChatDBG), downloaded about 70K times to date. It integrates into debuggers like `lldb`, `gdb`, and `pdb` (the Python debugger). For C/C++, it also leverages a language server, which makes a huge difference. You can also chat with it. We wrote a paper about it, should be published shortly in a major conference near you (https://arxiv.org/abs/2403.16354). One of the coolest things we found is that the LLM can leverage real-world knowledge to diagnose errors; for example, it successfully debugged a problem where the number of bootstrap samples was too low.
One major limitation the LLM has is that it can't run a profiler on the code,
but we can. (This would be a fun thing to do in the future - feed the output
of perf to the LLM and say 'now optimize').
This has been a feature of the Scalene Python profiler (https://github.com/plasma-umass/scalene) for some time (at this point, about 1.5 years) - bring your own API key for OpenAI / Azure / Bedrock, also works with Ollama. Optimizing Python code to use NumPy or other similar native libraries can easily yield multiple order of magnitude improvements in real-world settings. We tried it on several of the success stories of Scalene (before the integration with LLMs); see https://github.com/plasma-umass/scalene/issues/58 - and found that it often automatically yielded the same or better optimizations - see https://github.com/plasma-umass/scalene/issues/554. (Full disclosure: I am one of the principal designers of Scalene.)
I think with new agent capabilities of AI IDEs like Cursor and Cline, you can simply instruct the agent to run a profiler in the shell and make optimizations based on the output of the profiler. You can even instruct the agent to try to make the program run faster than a certain time limit and will try to reach that goal iteratively.
This same approach can work for eliminating errors in LLM generated code. We've had good luck doing it with Lattice [1] going from 5% to 90% success rate of creating a fully functional application.
I'm drafting a blog post that talks about how but for now the documentation will have to do.
There are many LLMs that have tools (function calling) available in their API. Some time ago, I was building AI agent that has profiler connected by function calling, and it works exactly as you described. First produce the code, then call profiler, and last step was to optimize code.
I agree that writing type hints can be painful, especially if you are starting with a large code base that is mostly untyped. You might consider using RightTyper (https://github.com/RightTyper/RightTyper) - basically run your Python 3.12+ program with it, and it will add type hints to your code. It’s fast, basically automatic, and by design RightTyper avoids overfitting to your types, letting a type checker like MyPy surface edge cases. In effect, the type checker becomes an anomaly detector (Full disclosure, I am one of the authors of RightTyper.)
From the GitHub page:
RightTyper is a Python tool that generates types for your function arguments and return values. RightTyper lets your code run at nearly full speed with almost no memory overhead. As a result, you won't experience slow downs in your code or large memory consumption while using it, allowing you to integrate it with your standard tests and development process. By virtue of its design, and in a significant departure from previous approaches, RightTyper only captures the most commonly used types, letting a type checker like mypy detect possibly incorrect type mismatches in your code.
Dealing with this currently in a giant old legacy python 2.7 codebase that was migrated to 3.10 in the past year. I do see the requirement @ 3.12, is there a specific reason for this that wouldn't be available to 3.10?
I was part of a skunkworks project at eBay a decade ago. eBay's OSPO was pretty loose at the time. We made our own eBay org to release our stuff under, with their blessing. They linked to our projects from their site.
We've all since left, and eBay's OSPO has churned too. Some years ago, I ended up with a giant undismissible banner on every GitHub page that the org we made has been "flagged." Apparently someone at modern eBay saw that we used the eBay logo and threw a trademark fit about it.
GitHub's support is perhaps the most useless I've ever encountered. They take months to reply to an issue, and there's no notification system. They expect you to constantly check your issue to see if they've replied, and they'll close it if you don't reply to their arbitrarily timed messages promptly.
GitHub Support basically told me to fuck myself. They didn't care that it was sanctioned by eBay at the time we made it. They didn't care that I showed them the obsolete eBay OSPO repo that linked to our org. They gave me no avenue to talk to anyone to get it resolved, nor did they give me any way to dismiss the banner.
Unless I want to write a Chrome extension to dismiss it, my GH session will forever have a black mark that I took a contract at vintage eBay, that modern eBay forgot about and sicked itself on.
From Fil Pizlo (http://www.filpizlo.com/), major contributor to WebKit's JavaScript engine, among other things:
> Fil-C is already powerful enough to run a memory-safe curl and a memory-safe OpenSSH (both client and server) on top of a memory-safe OpenSSL, memory-safe zlib, memory-safe pcre (which required no changes), memory-safe CPython (which required some changes and even found a bug), memory-safe SQLite, memory-safe libcxx and libcxxabi, and memory-safe musl (Fil-C's current libc).
https://github.com/RightTyper/RightTyper
(full disclosure, I am one of its authors).