software sessions

Rust in Production with Armin Ronacher

Armin Ronacher talks about getting into Rust, when to use it, writing Rust extensions for Python, building web applications with actix, creating debugging libraries, and the Rust ecosystem.

Armin is the creator of the Flask web framework, the Jinja2 template engine, and many other open source projects. He's currently the Director of Engineering at Sentry.

Topics

If you're interested in helping Armin build an open source debugging community, reach out to him via e-mail or twitter.

This episode is part of the Rustacean Station feed. Check it out if you're interested in hearing more about Rust.

Armin Ronacher

Sentry

Timestamps


Transcript

Click here to help me correct the transcript on GitHub!

Armin Thank you for having me.

Jeremy The first thing I want to start with is you've been a big part of the Python Community for over a decade and you've mentioned how it left a profound impact on your life and it even connected you with your wife. With such a strong relationship to Python. What got you interested in Rust?

Armin So what got me interested in Rust in the first place was the language itself very independent of I would say sort of a professional environment. So I found the language long before 1.0 I don't know exactly when. And I also don't really recollect anymore what the original motivation was behind me being interested in it, but it had a lot to do with it being an exciting time to look at language development. There was one of the few open source languages that were happening at the time and it had some features that I found really interesting. So that was the initial way for me to become aware of the language, but then relatively soon after it also became pretty obvious that some of the features of the language are really quite exciting and in particular what made me excited was that you could at least on paper right really good abstractions where you didn't have to sacrifice performance and so a lot of what limited my ability to make make nice apis in Python was that you had to pay pretty much everything at one time. So that's I guess what got me excited in the first place, but then there was a lot more afterwards like the community aspect that it was actually useful for problems that I had at work. So all of that came after.

Jeremy And could you give an example of say an abstraction that you said would be very the performance would not be good in Python, but you would be able to kind of use the same or similar abstraction in Rust.

Armin Sure, so a good example for this was what got me excited originally, which was that I wanted to build sort of larger not necessarily distributed systems, but systems where you could serialize some sort of message packing potentially over the wire. So I actually wanted to build a system which internally was based on sort of Highly concurrent functionality where you would do a lot of message passing. And so the at the time in the python community there was a lot of talk about aync asynchronous I/O async I/O and what was happening there was a system called ASGI, which is sort of a asynchronous version of WSGI, which is a web server standard of sorts where applications can have a standardized interface to talk to an HTTP server and this async system always looked like at least on the python side of things that you could almost only write this with a huge overhead. And so I was excited about this like some potential setups for how do you do asynchronous applications in Python? But it also looks like you can almost impossibly write something that would perform in the real world and so on the Rough Side of Things it looked like I don't actually have to deal with async at all. I can just do threads. It has really cool concurrent Primitives. So it looks like you can build systems like this in Rust and and they would work and I think for the most part that's true. And so that's what got me in this even though I didn't actually end up building a system like this. That's what got me excited.

Jeremy And you were saying that you found that within Rust you felt that a sink was maybe not needed but was kind of the difference in Rust we're just using threads with sufficient whereas in Python it wasn't?

Armin So in Python, you basically can't use threads. Except for very Corner Casey situations where you can actually so in Python you had the the global interpreter lock. So the only way really to get concurrency out of it where threads don't punish you for it is if you manage to have execution running in a thread that's effectively spawned outside The Interpreter so a good example for this would be if you have linear algebra or some some Mathy kind of situations where you could kick off a concurrent calculation on the C side and then you don't really get anything out of threads and in Rust that's totally not the case. There is nothing that prevents you from spawning a lot of threads. It turns out they can spawn a ton of threads on most operating systems and it's just fine. So I think for the most part the how you program in Rust and python is just as a result very different because doing lots of threads in Rust is is obvious to do it's simple to do but none of that is true for python.

Jeremy And that's primarily because of the global interpreter lock.

Armin That's a big part of it. But the other one is just that building thread safe Python applications is as hard as in any other language. And Rust is really good at teaching you how to do this better.

Jeremy I see and that's kind of related to the fact that Rust has sort of the ownership and borrowing Concepts that it catches at compile time.

Armin Yes, so that the fact that so rust makes you a better programmer or at least that was my experience with it it teaches you a bunch about how to better build concurrent applications. Yeah, and so that was really exciting and it's still very exciting. And I think overall it almost every person that I've exposed to rust has felt that something about their mental model of programming has fundamentally changed.

Jeremy So I guess would you say that maybe the thing that excited you most about Rust is the concept of being able to catch concurrency problems at compile time.

Armin I think it's the it's the combination of so many things that made me excited about Rust. I don't think like concurrency in itself would have made me appreciate the language. I think it's just the sum of all the parts that did it. I would say like in order of importance probably the borrowing part is number one because it is the most fundamental the ownership model and the borrowing system is the most fundamental thing, but that in itself would not have made me use the language the ecosystem that developed around it like the package manager and in particular the community. That is I think the most important immediately following this but that in itself would still not have made me pick it for something that I would use professionally the fact that we actually have some really strong ecosystems on the Rust side was the ultimate reason I got excited enough to actually use it in a commercial environment. So this is the case for both in our case, we do a lot of essentially we do a lot of debug working with debug files working with things that are roughly related to compilers and there is actually really good ecosystem and that's ultimately what made me and us stick to it. So it's just the sum of all the things I don't think this the ownership alone would have been enough to be excited about the language.

Jeremy Right. It's kind of the whole package sort of you had all these things come together and all those things combined was enough to make you really take notice and really want to use it professionally. You were talking about how you're doing some work with things related to compilers and debugging and things like that. At Sentry you joined Sentry about I believe four years ago. Could you explain a little bit about what Sentry does what their product does?

Armin Yeah, so I joined Sentry more than four years ago now, but I also knew many of the people and in particular David the founder for a very long time and I use Sentry before that. So I've been exposed to Sentry the product long before trying to company and its what it does is it effectively is an error reporting tool and it's an open source product, which also happens to have a quite well functioning business built around it and it consists of multiple components. The most important component is the server and the server receives crash reports from the client SDKs and what the server then does is it takes this crash report and it does some processing on it to produce an individual error report and then it's aggregates these together into a form so that you can see how many individual errors are happening and it does support multiple languages and the one where Rust comes in the most is effectively JavaScript and all kinds of Native Native languages like C C++ rust go. Every language where you want to extract a stack Trace out of either raw memory or where you want to take some memory addresses that you have for the stack trace and you want to find the correct functioning to it at the location when the file the error happened and this is where a lot of our Rust code comes in.

Jeremy And if someone wanted to use Sentry, is this something we're in their own application they would import a Sentry library and that would kind of hook into the memory of the application or hook into the exception handling of the application. Is that sort of how somebody would use Sentry?

Armin Yeah, pretty much. So depending on the language you either start with a high-level SDK like in JavaScript you pull in one of our clients. For Native crash reporting you have more options and one of which is to just use a standardized crash handling library. On Windows there's a built in function function to write a mini dump, which you can upload to us, or you can use brake pad Crash Pad libraries like that. And so it depends like if you already have a stack Trace that comes out of some system then you can probably throw it to us and we will do something with it.

Jeremy And then once you receive it, I believe you create some kind of dashboard that shows people where all their errors are occurring and sort of the frequency and sort of allow them to track. Okay. So how did you first introduce Rust to Sentry?

Armin So the initial, there were two approaches in which rust appeared in Sentry. The first one where we actually use Sentry use Rust was a small project called Sentry-CLI, which was a command line client to the API and that client was just written for fun. It was just an experiment to see how Rust works the second part where Rust actually ended up being quite crucial to how we do things was that we had a Curious Case of debugging where in order to process JavaScript stack traces. We need to work with Source maps and Source maps are fundamentally quite big and ever-growing JSON documents that tell you more or less at which line in the minified file. You would find the same token in a non minified Javascript file. So we'll tell you basically from this minified stack trace where's the original thing. And this was written originally in Python and we look at various different ways to just cache the source map so that they're faster to process and I already had for some other purpose written a Rust implementation of the of the algorithm needed to parse the source map and to map it and for fun we've just plug this in as a module exported to Python and it's tremendously improve the performance of the sourcemap handling in Sentry, and that also turn out to be relatively easy to expose to Sentry. Because Sentry was written in Python. We already had this Rust library all we had to do was write the C wrapper around it and then get it exposed to wire a Python wrapper that we wrote called milksnake to effectively the existing Sentry server and that was the gateway drug basically to get Rust into our code base.

Jeremy So you mentioned how performance was a big motivator here. Were there other things that in terms of building the part that parsed the source maps that was easier in Rust than it was in Python?

Armin I would say it was cleaner. I wouldn't say that it was easier to write it in Rust than in Python, I can tell you why it wasn't another language. It was basically rust because already had an implementation for this and I wrote The implementation because it was easy and fun. And the reason was fun because since a lot of this is based on JSON, I just used Serde the already existing serialization deserialization library to work with the base of the format and then as we started doing a little bit more correct handling of the source Maps it also also the source code was relatively clean because Source Maps if you want to do them correctly, they have some really bizarre behaviors. Which require you to count in utf-16 offsets in a utf-8 stream and that also was quite easy to do with the functionality you have out-of-the-box in in Rust.

Jeremy So you mentioned Serde helped with the sterilization process the source map implementation was that also something that existed as a library or was that something you created from scratch?

Armin No, so that Library we wrote, but the format is not very complicated so that was that was pretty pretty simple to do.

Jeremy Hmm and what things did you find were more difficult in Rust when building this implementation?

Armin For the source Maps. I don't think there was anything harder. It was just overall the original Python module was obviously a lot less code than the Rust one just by it not doing error handling particularly. Well, it was just parsing everything into a really not very efficient data structure. So this is this is a case of second system syndrome. I can't really tell you how to compare this tool because we already knew how to write the source on pending library in Rust so we obviously did it different the second time around so they are not very comparable.

Jeremy Hmm, I see so you mentioned you started with the Sentry CLI, and that was kind of just because just for fun just to kind of get more comfortable with Rust and see how it would work and then that sort of moved into parsing source maps with rust which really gave you a big performance benefit and it kind of proved out that this would be an effective tool, you know to use with Sentry, I guess in particular because of Serde and because of sort of the sort of capabilities of rust. What I kind of wanted to go into next is you you gave a conference talk and it kind of went into how you wrote a python extension in Rust and I believe it is this the the same project that we're referring to here.

Armin Yeah I believe that was the same. There was the first Library we exposed but we also expose some others.

Jeremy So in the conference talk you mentioned that some people had come to you and proposed that services should be written in Rust and that it would be better to have the Python communicate with Rust with the rust application being written as a service and you kind of push back on that a little bit in terms of saying that perhaps it's better to actually write python extensions in Rust rather than building a separate application. Do you kind of still feel that way or have your opinions changed?

Armin I don't think my opinions have changed I think there. Times when it's better to write the service and that times when it's better to expose something to already existing language. Ultimately. I'm a strong believer of only making Individual Services when there is no other way to do it or when the cost of doing the extra service outweigh the what the down side of it I don't know if that makes sense, but overall the there were some reasons why we wrote this as a separate service. But at the same time it's important to know that even though it is an independent service almost all the codes that runs in that service also independently can be accessed in Sentry itself. So to make this more specific symbolicator is an independent service which takes a mini dump or individual stack Trace addresses and it resolves them in debug files and then gives you back a proper stack Trace. Now in order to do that, it needs to open debug files. And in order to work with these debug files. It also has an internal sort of efficient cache format, which we call simcache and it produces this on the symbolicator. But for most people the source of the debug file will have to be uploaded to Sentry in the first place. And so this uploading of the debug file to Sentry actually goes to the main Sentry server, which is written in Python. And so for you to get feedback on if that file is actually valid or not. We do run the same Library which Symbolicator uses also on the side of Sentry to open the debug file to validate it to make sure it's well formed to index it correctly. So a lot of the code that is in symbolicator, which is the base Library called symbolic. Also runs in Python exposed as a python module. So even though a lot of it runs in an independent service almost all the code also gets imported into a regular python process for some functionality, which is better than on the Sentry site. So basically the answer here is like some of the things looked better to put an independent service, but we still wrote it in a way where we actually have it embedded in. The main Sentry system. I don't know if that answers the question.

Jeremy Yeah, think so. You sort of are doing both. I guess you you wrote the Rust code to act as a extension that you could call from Python. I guess you found was symbolicator if I understand correctly, you're uploading the mini dumps directly to symbolicator?

Armin So yeah the way the flow works is that the client uploads the the minidumps to Sentry. Sentry then immediately uploads it into well depending on how the flow goes and this might change at the moment. It goes into Redis and then from Redis it's being picked up by another python process that shows it to symbolicator. And then from this Redis to symbolicator communication by HTTP it goes into some sort of polling system. And so it will try to poll symbolicator until the response is ready and then it will go forward and do other things I would say from an architectural point of view. There is no reason this really absolutely has to be a service we could have built it as a not service. The Reason why symbolicator makes sense for us to be a service is that we have a lot of we see a potential lot of value in independent people using symbolicator even without Sentry. And the reason we hope that's the case is because the total number of people that actually contribute and find any enjoyment in debug information files, like pdb dwarf files, maybe Source Maps is very limited, but the complexity in those files is huge. Pdb is a is a perfect example pdb is a Microsoft proprietary debug format. And a lot of companies that have been attempting to reverse engineer it over the years to build better debugging tools and Microsoft sort of Open Sourced part of the specification a couple of years ago, but there haven't been good open source implementation for it until very recently so our hope in a way is that by making symbolicator a useful service independently, we can actually encourage people to contribute to an open source implementation of these debug formats. We already have these pretty useful crates that the do stuff for Rust. But unless you use Rust yourself. There's a high chance. You would not find them particularly useful. So by making symbolicator itself useful. We're hoping to encourage more development going into these crates. So if it would have built Symbolicator into Sentry itself, then you would have to run Sentry for that to be useful but not everybody will find Sentry in itself useful. Maybe someone just wants to do something else with the debug files and then Symbolicator by itself might be useful for them.

Jeremy So building Symbolicator as a service was not so much because it was not not because like from a technical perspective. You couldn't write it as a python extension, but it was more about being able to hopefully build a community out of an isolated service something that like you were saying people who don't necessarily use Sentry could still contribute to symbolicator if they were particularly interested in dealing with pdbs or other types of debugging

Armin so there's also another thing that plays into this which is that we are effectively fighting for well, we're fighting for a world that doesn't exist currently, which is a world where companies and open source people open source communities will find that there's value. In debug files and to make this concrete. Is that even though we absolutely maybe despises too strong of a word but working with pdb files is not a lot of fun. Their format was written with motivations that are very hard to to understand. It appears to be a format that has been added more and more bloat over the years. So as much as pdbs are not fun to work with. The other system that Microsoft built is actually really good which is the the symbol server protocol the ability to if you have a crash and you can see what modules are referenced. You can go to the Microsoft symbol server and you can ask give me please the executable which contains the unwind information. Please give me the debug file which contains source code file name. Well filename location maybe even embedded source code. That doesn't exist in the open source community. It doesn't even exist on Android where you would expect it. It exists very little on Apple systems. And so. We're also hoping in a way that if there's a system like symbolicator that there will be an appetite for building these types of debug information indexes for other platforms so that you can eventually maybe debug your Android phone or debug your Linux Ubuntu.

Jeremy So that yeah, that would be exciting. So trying to really build a new sort of community. That's that's interested in working on this problem. And that's something that could really benefit basically developers everywhere. And sort of symbolicator is maybe the start of trying to build that community.

Armin Yeah, pretty much. That's that's a hope and so symbolicator doesn't just do simplification. It also assists you in in abstracting over multiple different potential storage systems for sample files so it can connect to AWS. It can connect to a Microsoft symbol server compatible system it tries to help you unpack different types of compressed debug files. So theoretically even if all you care about is getting a debug file for your debugger you can just hit the URL and it will proxy to some remote source. So maybe someone could build a plug-in file a DB or GDB to automatically download remote debug files through it.

Jeremy Another thing. I'd like to talk a little bit about is you had mentioned you used something called milksnake to interact with Rust from Python. Is that something that somebody who's interested in Python and Rust interop is that something that they should look into or have things changed from when you did that?

Armin So milksnake is a library we wrote which helps you do. Just two things it helps you kick off cargo from a Python setup file and it helps you load binary shared object a Dylib or something like that with the pythons CFFI system. It doesn't do much around that it doesn't automatically expose any rust functions to python or anything like that. So it's a little bit more cumbersome to use than some of the alternatives which directly linking in cpython. Our motivation for milksnake was twofold one of which was that we wanted to build wheels, which don't link against lib against libpython. So we effectively compile our rust modules to regular shared object. Then we build c binary. Sorry C ABI into that binary and we make a C header to it, so you can actually load our rust libraries from a C environment. And then we you see CFFI via milksnake to load these into python. So that's maybe not the most convenient workflow and maybe also not the most high performant one, but it gives you the ability to pick and choose python versions. So for instance, it doesn't matter if you load this from python 2.7 or Python 3.5 or Python 3.6 or 3.8 it doesn't really matter and it also just generally forces you to think a little bit about how you approach a sensible layer of abstraction. And so that was our motivation for doing this and I think for a lot of people that model makes sense. Maybe it's not the most sensible model for all things that you would do between Python and rust, but it definitely has its upsides.

Jeremy And that's mainly like you were saying when you don't compile against libpython, then you can use any python distribution. And so you're sort of.

Armin Yes so it does user either PyPy or C python for as long as it supports the ffi. You can go to this extension module.

There are some alternative approaches. I'm toying around with one of which is there's a project called wasmer where you could theoretically compile your rust code into wasm module and then have this wasm module be loaded via wasmer into python. Now that obviously is easier to do if you have like an algorithm that should run. It's less easy to do if you also want to do issue. I don't know file open or something like that where the wasm module would have to have an ABI that it can call.

Jeremy In your conference talk before where you talked about using Python and Rust together. You had mentioned that are handling was very painful has that improved since then?

Armin So we have a solution for that where we basically have some macros which translate errors. So we basically catch the result or panics at the boundary and then translate them into. Error codes and then on the python side we have a wrapper that catches those error codes and reraises them as python exceptions and we have some code that auto generates these mappings. We haven't really built anything that makes this reusable because we didn't have to but in a perfect world there would be some way to standardize this. Now, there is some there's a lot of work going into this sort of stuff in the wasm community. So there are systems like this the to automatic code generation between JavaScript and Rust this could theoretically also exists between Python and Rust. It's just that nobody built this and systems that are currently existing for wasm are not too reusable unfortunately, so that's why not too much has changed between them but a little bit.

Jeremy The next thing I'd like to go a little more into is the web side of symbolicator. So why did you choose actix-web versus Rocket or any of the other Rust web framework projects?

Armin so symbolic ETA was built on actix because. What's a good reason I mean fundamentally there are not that many systems currently in Rust that are stable. Actix at the time was the most stable system. And so we built this on Actix 0.7. Now with the async awaiting coming up. It's actually not entirely clear what's going to be the most sensible unstable solution going forward. Internally symbolicator does a lot of things it can download from external sources. You can produce cache files it can symbolicate based on Cache files it can Stack core minidumps. It does a lot of things. So the actor design made a lot of sense. So that was the biggest reason so actors seem sensible the platform seems stable. That was the decision for us to go with Actix 0.7. Now. We recently attempted to upgrade this to actix 1.0 Oh, we didn't really succeed in the upgrade because of us running into memory leaks. What we believe is to be in the actix core itself. And since there are. This is quite a big of change going on in the rust async/await stabilization phase. This might actually be something which we reconsider. Ultimately the actix design for actors makes a ton of sense for systems like Symbolicator maybe even the service design from actix 1.0 or the service design from Tokyo Tower. Feed well into this design like this the problem with all of this at the moment is that async await is going to throw all of this into like a little bit of a chaos again, and it doesn't necessarily seem like the community is. Is moving into one clear Direction in particularly Actix is turning a little bit into a silo and the rest of the community is also building new silos. So it's hard to say what this is going to look like in a year from now. It might even turn out that we want to build something like Actix on top of something else, which just does what we need to be a little bit more stable and well-guarded for our use because this is not the only thing we built on Actix. There is also a second system called relay. So it's unclear right now that the async await thing is going to change a lot and the reason it's going to change a lot is because once async/await stabilizes certain things are going to work but other things that currently work really well will not work at all. So in particular both actix 1.0 and 0.7 is based on manual futures a lot. So you have to do async/await async await is only going to work in functions and and simple method implementations for now. But if you look into how the service traits work, they actually require you to also Define the sort of this type hierarchy. For from how you return futures from a service and that's not working at all with with impl impl future. And so for that reason, I'm actually expecting that initially at least some other designs will emerge as being fitting better this design of the new features. Yeah, it's unclear. It's very unclear right now. It feels again very unstable. Overall. Where does all this whole thing is moving.

Jeremy Yeah, so if someone was interested in building a web application in Rust now it sounds like the public API of these Frameworks. Maybe it's going. end up start changing a lot due to this all these changes being made in the async await ecosystem. Does that does that sound correct?

Armin Yeah, that's that's how we feel right now where we're waiting and seeing where what this whole thing is moving.

Jeremy For someone who is interested in making a new web application in rust would your advice maybe be to kind of wait and see a little bit and see how things turn out with async await or what would your sort of thought be?

Armin My thoughts on this are very complex. I think it very much depends on the type of application. So the only reason async/await even plays a role in symbolicator is because symbolicator does something we normally don't really do it does a lot of HTTP downloads most of what it spends its time doing other than stack walking which is CPU bound is going to different web services and web servers and downloading stuff from there. So async made some sense to us overall, but I'm not sure. I know that many systems with actually need async like most people will be fine with threads. So I'm not sure if async should be the most motivating factor.

Jeremy And what would you say that async is that most important with IO operations like you were giving the example of symbolicator making a lot of HTTP calls or you might have an application that's doing a lot of file reads or writes. Is that kind of the primary use case for it?

Armin I honestly have to say I'm not sure I think async ultimately right now at least is a question of preference so composing things out of Futures. It doesn't matter if they're CPU bound or if they are IO bound is something that some parts of the Rust Community decided is a good thing to do and that's why in certain environments that appears to be the most logical thing to do and in particular actix definitely has some situations where they say. Well this is how you do stuff. So. That's probably more. The reason why we ended up using async code rather than it actually being the most sensible thing to do. I'm pretty sure you could still use Futures with mostly blocking code just to compose the high-level workflow that we have we could also have used some other things. That would have probably been more sensible. But this is just how we started with this this seems overall to be quite okay as a design. I don't know if it's the best design that we could have done. Like with the experience of hindsight. There are a lot of things we did incorrectly with this design where we just probably didn't fully comprehend how painful async would actually be.

Jeremy Could you maybe give some examples of things that you think might have been done differently in hindsight?

Armin I mean this requires quite a bit of knowledge about how symbolicator works internally but to give you like a basic overview of how symbolicator way that works is the API consumer send some requests into symbolicator and symbolicator will attempt to work this down for some period of time and if it's if it can't fulfill your request in that period of time it will give you back a request ID and you can start polling on the result. That's sort of. The base form of communication that symbolicator has with Sentry. Sentry will send requests to symbolicator. It will try to wait on them. If it takes too long. It will put it into a queue and then we'll start pulling it until the request is done. And that abstraction that is existing internally is basically a lot of futures. the problem with all of these futures is that back pressure management is not clean at all in symbolicator. So the way we do this at the moment is not nice and it actually causes some problems because if we overload the system it gets to a point where it performs in ways in which well behaving system shouldn't perform. And so we definitely want to change some parts about this design to make this back pressure management more sensible and more unified across this pipeline, but the problem you end up with if you have this is that you can. You have a future somewhere and the future does something but from the outside, it's not clear where this future should actually run like does this future do cpu-bound tasks. Then you have to spawn it into some executor. It doesn't block the other futures or is this feature actually just doing lots of IO and it's fine for that future to share it's execution with where all the other futures are running. And so because these Futures look the same on the outside, you can easily spawn a future into the wrong thing. So you could spawn a future that looks like it's I/O bound but someone appended and add then on to the Future, which is actually CPU bound and because it's so easy to get this wrong. We did some mistakes in the symbolicator design where some futures actually ended up running on the wrong executor and this lack of it because it's so easy to do this mistakes. I'm not sure if that's the right abstraction to actually use. Ultimately, all the problems are when you have async code that's running somewhere. And then it has to do some CPU bound task and you would really like that CPU bound task to just be sync code, but what if that code again has to do some async stuff and then it gets all weird and then no nice abstraction currently existing in our code base. And I'm not even sure if there are some good abstractions in the wider ecosystem right now about how to design systems like this. So yeah, I think there probably needs to be a better overall framework to build something like this Actix I think in 0.7And both in 1.0 has some really good ideas, but it falls short in others and because it doesn't establish itself as the standard system. I'm not sure if it's going to actually have the answers that it needs. For people to build good systems on top. This is too complex of a project to be just one or two people.

Jeremy Actix specifically you mentioned you're talking about?

Armin And most projects in Rust are too few people in that ecosystem. It doesn't matter if it's Tower or if it's Actix there would have to be a community that understands. Hey, this is this is how the pattern should be and this is this is a framework that we're going to build that supports that.

Jeremy In other languages are there are there ways that they deal with it whether that's in C or C++ or dotnet Java that sort of thing.

Armin I mean there are definitely frameworks that are based on actor systems or or something like that in various different programming languages I think rust is at the moment not the language of choice for Building Services and that's why there's less demand for doing this and I think that's why we have not such a stable environment here, but I also don't think it's fundamentally a problem right now. Even if even if our ecosystem isn't that strong in actually building web services most web services don't actually need a lot of surface area. So I think in the next one or two years some standard framework will emerge that can be a standardized framework that people build web services on top.

Jeremy It sounds like what you're thinking is that whatever that web framework is it's going to need a much larger Community than the existing ones have where it seems like the existing crates in the space. You said there's only one or two main contributors. There's no Django or no rails of Rust at this point.

Armin So I think the problem is nobody wants a Django or nobody wants a rails in Rust people don't need another HTTP framework what people actually need is a system that answers how to do more complex applications. Because there is rocket. And Rocket is fine for what you do in Django, what you doing rails, which is you can get an HTTP request and you reply with a response. But the things that people actually have some legitimate reasons for building in rust need more than that. They need they effectively need something that gets closer to an erlang type of service where you don't just have HTTP dispatching. You also have internal message queues you have maybe AMQP is involved maybe some distribution is involved. That is all more complex than Django is I think you can build Django like applications just fine in Rust with the Frameworks that exist. But the ones that goes into more complex situations, you also don't have a good answer in Python. You just also would never have probably built a system like this in Python so that's why that's specifically a problem in Rust at the moment.

Jeremy In your opinion for an application that doesn't have these sort of complex requirements of requiring internal message passing or things like that somebody who is making an application that they would normally make in flask or rails or Django that sort of thing. Do you think at this point or even in the future that Rust may not be the best choice for that type of application?

Armin Rust is fine for that. It really depends on what the application does. Like. If the application has. I don't know like most applications have a reason to exist and if the reason is for instance well I need to do some audio processing, I don't know you could build as web service if that's what you're processing then it would be absolutely sensible to build this in Rust. If you have I don't know a web service which purpose it is to write to a single postgres database maybe Rust is not necessarily choice here and because Python gives you better debu ability. There's a more Rich ecosystem. So it really depends on what you're doing for us. It was obvious to build our system in rust because there is no other community right now of or there's no other ecosystem where there is such a good support for working with debug information files. Its rust is pretty unique in that it has a really good ecosystem here. The only alternative C++ and I wouldn't know how to build a web service in C++ either and feel happy about it. So Rust is definitely better choice here. So it really depends on what is it that you're building if Rust is the choice or not.

Jeremy So maybe the sort of web aspect in terms of you know, being an HTTP server that fields JSON requests and responses. That part is maybe not so important but it's really what the application does like in your case you said something in the compiler space. There's a big Rust Community or something. That's heavily CPU bound that you might have to normally write in C or something like that. Maybe something like that would be a good fit for Rust.

Armin Yeah, I would say that there are already some ecosystems where Rust seems to be pretty obvious to use it if you have something where you have a legitimate reason to expecting to run this in a browser. So if you want to have your stuff compiled to wasm rust is a pretty obvious choice. If you have anything in the crypto space rust is a pretty good choice anything in the compiler space Rust is a very good choice right now both in handling debug information files as well as there's cranelift. So if you're actually into language design at this point, you might really consider doing this in Rust but I also think that rust has already a pretty good overall establishment in in anything where you would built. something that is very CPU bound something where you'll want to run over multiple CPU cores and some of these things you actually do want to expose web service. And in that case, why wouldn't you do it in Rust? I mean, you could technically probably do it in Rust and then expose it as a library and then consume it in go and build a web service in go and there are definitely some companies doing that. But you can also already do this in Rust itself. So it really depends on what you're doing.

Jeremy And in the current Rest ecosystem are there I mean we talked about sort of a more robust actor system or being able to to determine whether the thing that you're running is I/O bound or CPU bound something improved for that. Is there anything else within the rust ecosystem either a missing crate or a framework that you really wish rust had right now?

Armin Hmm, I have definitely one thing I wish was there. I wish standard error would carry a backtrace. The fact that the standard error type doesn't carry a backtrace right now is very frustrating. So that's that's the biggest thing that's missing. I wish async/await would be stabilized not because I actually want to use async await but because I want the lower-level HTTP and IO libraries to to stabilize and until async await is stable. I don't think they will stabilize it's also pretty obvious that even if you want to use sync IO the base libraries are most likely be written in async fashion these days just because of how much more complex these Protocols are becoming. HTTP HTTP2 is a pretty good example where even if you only want to make an HTTP request and getting the HTTP response, Internally the protocol has evolved in complexity to the point where you actually need to write it in async style. This is only going to get worse with HTTP3 while you basically get get rid of all of TCP. So I really want that ecosystem to stabilize. I also think that in terms of what's missing in the ecosystem. There's some pretty I think there's some pretty obvious shortcomings still in editor support. It's just like the entire slowness of the compiler is a pretty big showstopper still. The fact that you don't have like rust needs intellisense or something like this and the fact that RLS is still this slow is pretty annoying. I think once that gets better. There will be some more obvious communities automatically picking up Rust.

Jeremy One of the last things I'd like to go into is. In the past you've wrote a lot of back-end code for games and I believe was that in Python and c and C++.

Armin Uh, yeah, mostly python

Jeremy Just got out of curiosity. Are there any projects you worked on in that space in the past where you could see rust fitting in or do you think that that type of work mostly makes sense in Python?

Armin I think game backends make a lot of sense in Rust now the question is. Where where where would use python versus where would you use rust? So in particular when you look into something like like a computer game that actually simulates a world what it has a very connected environment people currently do this in Java quite a lot. They do this in Go quite a lot. I think rust has a pretty good realistic expectation on being a language of choice just because rust makes a lot of sense in other environments. If you're going to have to run code both on the server of the game. And in the client you want that to be the same code and that why you have a lot of C++ code in the space at the moment. You have some other games where the simulation is actually quite detached between client and server and then it might be in different environments or different languages. But where it's about game simulation rust has a future where it's more about well you want to store some inventory or in-game currency or like statistic backends. I think there is no good reason why you would do this in Rust at the moment.

Jeremy Finally, is there anything else that you think I should have asked or anything else you'd like to add?

Armin It's a good question. I think it would be interesting to to explore in the upcoming space time will be to see are there more communities where Rust becomes a language of choice over people would like to use rust but currently can't for some reason or another. So when is. I'm not surprised that the cryptocurrency community embraces rust, but I'm actually still quite surprised that some communities are not picking up rust where I would expect that. I would have expected that there is generally a bigger push for web development in for rust. Then they're actually turns out to be. I was surprised about that. And I think the big reason why there are fewer that projects going to be built in Rust is just a fact of the compiler speed. It probably takes you too long or it's too hard to expose protocols to rust and other languages like. You already have something like thrift or protobufs, but it's actually not that convenient to build this in Rust compared to other languages. So I think it would be interesting for all of us to to specifically talk to some of these communities to find out what actually the reasons are for not backing rust because I really I would have expected at this point that there is more push for web development in the rust space than there is.

Jeremy Have you talked with other companies that are using rust for web development? And you know, if you have have you got any perspective on them on why they chose it or why, you know, they think maybe other people aren't using it.

Armin The biggest reason I've noticed by people have very little interest in Rust for web development is because they already have other choices. And because they feel like the ecosystem is not stable enough. So the chicken and egg situation. Nobody has enough reason to to work on stabilizing a web development environment for rust because of other alternatives. At the same time. There is just not enough development happening by itself to stabilize this so if you like once it stabilizes. By itself by all the people that are already working on it. There's going to be some reason for people to to reevaluate it?

Jeremy Do you think perhaps that might come after async/await become stabilized and maybe that will bring a lot of people off of the fence and get involved with web development?

Armin I. I don't think async/await is going to be what kicks this off because once async/await stabilizes, there's still so many other things that needs to be stabilized as well async/await stabilizing in combination with impl impl future impl all of this stuff is just part of it. We will still need. I'm afraid for web frameworks to be really comfortable to use in Rust we will still also need to compose services and composing Services means that I can write if I return an impl future. I need to name this type somewhere in an associated type. So I will need to say like hey, this is a service which produces a future. But I need to write what type of future it is into the into the associated type. And currently I can't do that and I'm afraid that just stabilizing async/await is not going to it's going to help us write async code on a low level but it's still not going to help us compose services. And without that ability. I don't think web services are going to be easier to write.

Jeremy So that's more of the type of problem that say erlang's VM or the Akka actor framework in Java that something something like that. I guess for rust you're thinking.

Armin I think you will need something like this for people to be excited about web development in Rust. Yeah, I do hope that the community forms that says like hey, this is actually we have an idea of how it should look like and let's all try and forces because right now it feels like they're separate. Just too many separate attempts of doing something. And too many different ideas of how it should work.

Jeremy Yeah, so everyone's sort of has their own idea and we haven't yet found a solution that everybody I guess thinks is good enough to kind of gather around.

Armin Yeah, I believe so.

Jeremy If someone's interested in learning more about how to do web development in Rust what are sort of the the resources that you would suggest they use?

Armin Web development in Rust resources. Hmm. GitHub code search I think that's my answer right now. It's surprisingly good to use GitHub code search to understand how people structure code if you already have a framework that that you want to use like Actix or Tokyo Tower or warp or whatever or Rocket to search on GitHub. For examples of how other people structure code is insanely valuable because for the most part right now people don't have this written down. It's not like there's a book that you can buy which says like this is how you develop a web app in Rust but what you can find are huge amounts of Open Source libraries or software that try to do different kinds of web services in Rust and you can you can find them if you search it.

Jeremy And I suppose symbolicator would be an example of that. Are there any other specific projects that you think would be interesting for people to take a look at?

Armin We have a second service internally. It's called semaphore external is called relay. This is also an example is also an actix 0.7 Project, which I think has a pretty good design. We what else to I know exists. I'm not gonna say I'm not looking too much around what other people are doing, but every once in a while, I'm looking for like specific patterns. Yeah, I'm not sure what else is there.

Jeremy With that we can start wrapping up. So how can people follow you and kind of learn more about what you're doing and follow the symbolicator project and can maybe get involved?

Armin So actually let me Pitch some of my ideas here. I really want the open source Community to embrace debugging. Debug files. Everything that looks like a symbolserver. The project itself is on github.com/getsentry/symbolicator. We are also very actively contributing not to our own libraries, but to the rust ecosystem as a whole in that space for the pdb crate the dwarf and gimli crates. I I want to assemble some sort of community of people that have an interest in improving debug experience for people. So right now we have. We're kind of starting a slack channel for people to talk about sort of the experiences with this. We would also really encourage Linux communities to start pushing for public symbol sources. I find it so disappointing that right now, I can't use a build ID even though they're compiled into every library on Debian and Ubuntu and Red Hat Linux, but I can't use a build ID to look up a debug file. That's so disappointing. So I really wish that would happen. So I think it will hear more from us actually pushing people to do this to give to give them incentives to host debug files because everybody's experience with tremendously improve. Right now if you use Sentry you can put your debug server into the list of sources and you crash and we will search on your servers for debug files. And now there is an open source implementation actually supports this that's to reach out to me if your interest in this kind of stuff on Twitter. My handle is Mitsuhiko. You can also tweet to getsentry or just send me an email anyone who has any sort of interest in debug experience, especially for Native C++ doesn't matter which platform. Please write me I want to start making a community of people that care about this sort of thing because it's really disappointing that there is not that there wasn't enough reason for people to to invest in the sort of thing. I think there's a huge development happiness improvement that could be there if we have this in the open source community.

Jeremy Okay, great. So, hopefully anyone out there who is interested about improving the debugging ecosystem can reach out to you and really get this started within the rust community. So that's awesome.

Armin And it's working with rust out of the box. So rust makes perfect debug files. You can send them up to Sentry and Symbolicator and you can debug production builds. Just fine. Thank you for having me.