Does Python make sense for large projects?
NNeetCodeIO@NeetCodeIO · 352K subscribers
- Views
- 17.7K
- 17,698
- Outlier
- ×1.9
- Median 9.2K
- Length
- 11:08
- Published
- Jun 17
- Wednesday
- Comments
- 60
- 402 likes
Script
How it's written
- Words
- 1,734
- Words a minute
- 156
- Reading time
- 8 min
- Segments
- 8
The hook
0:00–0:26Back then, Envoy was quite new. Um, but they still had the Go control plane library, which is like a framework for creating control planes. That we could have used that at the time, but there's there's always details there. And one of those details is the fact that the team was comfortable with Python, so I opted not to go for something that was going, even though maybe it was better off the shelf.
Structure
Back then, Envoy was quite new. Um, but they still had the Go control plane library, which is like a framework for creating control planes. That we could have used that at the time, but there's there's always details there. And one of those details is the fact that the team was comfortable with Python, so I opted not to go for something that was going, even though maybe it was better off the shelf. Um, and some of it was, you know, the things that I'm familiar with and comfortable with, so I'm you know, I most of my Python experience was building Flask web apps. So, I'm very familiar with with creating Flask apps and doing a lot of uh Ginger templating and that kind of thing. And so, the the initial version was like this really basic um, a web app running threads for polling data and uh YAML Ginger templates, which is considered blasphemy these days, I think, or it should be. But maybe it isn't. I think Kubernetes re- revived that, so that's kind of um insane. Yeah, I was going to ask about, I guess, the in terms of using Python, I don't think I've ever personally used it on like an actual a very large project before or ever even, I think, professionally. And I think a lot of people, like when they hear about using Python and sometimes JavaScript, and nowadays we have TypeScript, uh they they
Watch from 0:00think of like a language that is, you know, sometimes considered a toy language, right? That like, okay, you wouldn't build anything serious with this. Um, not just in terms of like performance, uh but in terms of like um like I think the type system is usually like the biggest knock that it gets. Uh, what's your opinion on Python and how did you guys use it in a way that it was effective for you? And how big were these Python projects that we're talking about? Were they like relatively small like microservices or were they like projects that grew larger and larger and you kind of wish like you had a different language that you picked? They definitely grew to the size where I felt that Python was a mistake. I think that Python is fine if your architecture is really strong. But I still feel like it's not a great choice. I feel like it if you make Python as a choice for a large app, then you're making it a little bit hard for yourself in my opinion. Um and some of the reasons why I think that is that when you write Python code you know, there's all kinds of runtime errors that can happen and it's it's the potential that makes things difficult because you can't just wrap everything in a try except block.
Watch from 1:30You know, you have to be reasonable. Um also try except blocks can be they can ruin your performance if you're not careful with them. I I can't recall exactly how many lines of code um Sovereign is open source, so anyone can see that um and the code in there. I think it might be I don't know. 10,000 or less. Uh but we wrote extensions and templates that uh were given as configuration to Sovereign and that was where most of the complexity lived. And that that was a the complexity there was essentially taking uh one format of data putting it into like a essentially a Pydantic model, validating everything, and then having it produce um a lot of other structures and then feeding that into a template. And Python is not so friendly with uh a lot of iteration in my opinion. Like if you have to run a big complicated for loop, uh it's not so great in Python.
Watch from 2:51And that's essentially what we were doing in those in the in that template code was taking all of the configurations from every uh service that we were running in Atlassian and producing individual Envoy configuration for for each. And I think we we added quite a lot of validation to make sure that those uh structures came out correct. And that was another thing that sort of hurt because all of the type checking validation for Python is done at runtime. So it it I think slowed things down quite a lot more um in the for the benefit of safety. Um but I I wouldn't say that these apps were comparable to some really large Python apps like Bitbucket and Instagram I think is Python or was. I think it's a Django app of some kind. Yeah, I can't say that it's that you shouldn't build something large in Python could because people have clearly been very productive um and effective at doing that and their apps perform just fine. I just Yeah, I I think I stand by my statement where it just makes things a little bit harder than they should be maybe. Uh I think these days I'd probably recommend Golang.
Watch from 4:06Um that's the safe recommendation in my opinion. I personally like Rust and um I find that the two relatively similar. Golang to me feels like Rust that just lets you do a little bit more than than what you you can normally do without checking. Like, you can easily create globally uh shared global a shared mutable global variables and stuff like that. That's really interesting to me because it makes me remember that engineering is is pretty much just trade-offs, I feel. And I think Yeah, you kind of mentioned something similar. And to me, that kind of means that uh unlike like a math problem or something or even like a very small scoped algorithm where there's like a correct solution, there's not really a correct solution when it comes to building things. I find that uh another part of engineering is about sort of creating patterns and systems where there's just less of a risk for things to go wrong. Even if technically they can go wrong in any sort of situation. And for example, like uh C++ when you're manually managing memory, um the everybody will say, well, it's a skill issue. Like, I know C++ fans will say it's a skill issue if you're if you have it bugs or something rather than using a language like Rust. Um but at the same time, if there's just a a way of doing things where with a language or something like TypeScript where you
Watch from 5:36won't have those runtime type errors, it it just seems like it's a no-brainer to kind of structure things in that sort of way. And I feel like this is one of the skills that is so uh so dynamic in that you can kind of apply it to any sort of problem where you're you kind of set up like this system and how you want to architect it in a microservices system, you like there's a million things to go wrong. So, if you can avoid it, people always say, of course, avoid microservices. Um So, I mean, that's kind of just a bit of rambling, but I'm curious like what your thoughts are on that in terms of just trade-offs and setting up systems in a way where you kind of just there's less of a risk for things to go wrong. So, there's a couple things there. Uh you So, you Yeah, so about the trade-offs I recently read a book called The Philosophy of Software Design by Professor John Osterhout at Osterhout at Stanford University. And the book goes through these dichotomies of like for example, general-purpose code is more simple than special-case code, for example. And it's got a lot of dichotomies like that that it goes into. And I felt like that book boil boiled down essentially to when you do have these trade-offs that you have to consider you're trying to balance them against the thing you're building, like which
Watch from 7:07trade-offs are going to be more preferable. And it's essentially an art. Yeah, it's not scientific. Um and I just think that's that's a a huge part of software engineering, and it was a part that confused me for a long time because I I felt like when I was making decisions that I couldn't go to my colleagues and say this is objectively what we should do. I had to take my subjective opinion of why it's a good design and then argue for perhaps the trade-offs. Like it does it does this, but it doesn't do this, and we don't care about the the con, but we do like the the pro. And I I suppose in that way you can try to make it less of an art form and maybe try to logically defend it. But I'm Yeah, I'm still growing in that area. Um that's I see that as the software architecture track, I suppose. Which I'm kind of interested in. And then you you mentioned a few other things that I I thought was interesting. You said like that you can boil down like if you're using a a language that gives you full autonomy and relies on you having uh the skill and the care.
Watch from 8:37I think that's it's interesting because there's an element of skill, but there's also an element of cognitive of capacity that I think people don't think about. You can be a perfectly capable person at solving that problem, but if you have like 20 things to do this week and you only have an hour on you know, next Thursday you've got 1 hour to solve this thing, then your capacity to to solve that is way less. So, if you've got to carefully make sure that all your memory is managed and that you didn't make any logic errors or anything like that, then why not just erase one of those things and work solely on a smaller set of problems. That's how I uh see that. And yeah, so Python is great for it's it's making trade-offs essentially. You're writing in this uh language that's elegant and productive and tactical, but which lacks this uh long-term protection against maintenance burdens and like uh ambiguous types and things like that. I think you're making that trade-off for sure.
Watch from 9:58Full transcript
- 0:00
Back then, Envoy was quite new. Um, but they still had the Go control plane library, which is like a framework for creating control planes. That we could have used that at the time, but there's there's always details there. And one of those details is the fact that the team was comfortable with Python, so I opted
- 0:20
not to go for something that was going, even though maybe it was better off the shelf. Um, and some of it was, you know, the things that I'm familiar with and comfortable with, so I'm you know, I most of my Python experience was building Flask web apps. So, I'm very familiar with with creating
- 0:42
Flask apps and doing a lot of uh Ginger templating and that kind of thing. And so, the the initial version was like this really basic um, a web app running threads for polling data and uh YAML Ginger templates, which is considered blasphemy these days, I
- 1:03
think, or it should be. But maybe it isn't. I think Kubernetes re- revived that, so that's kind of um insane. Yeah, I was going to ask about, I guess, the in terms of using Python, I don't think I've ever personally used it on like an actual a very large project before or ever even, I think, professionally. And I think a lot of people, like when they hear about using Python and sometimes
- 1:26
JavaScript, and nowadays we have TypeScript, uh they they think of like a language that is, you know, sometimes considered a toy language, right? That like, okay, you wouldn't build anything serious with this. Um, not just in terms of like performance, uh but in terms of like um like I think the type system is usually like the biggest knock that it gets.
- 1:47
Uh, what's your opinion on Python and how did you guys use it in a way that it was effective for you? And how big were these Python projects that we're talking about? Were they like relatively small like microservices or were they like projects that grew larger and larger and you kind of wish like you had a different language that you picked?
- 2:07
They definitely grew to the size where I felt that Python was a mistake. I think that Python is fine if your architecture is really strong. But I still feel like it's not a great choice. I feel like it if you make Python as a choice for a large app, then you're making it a little bit hard for
- 2:27
yourself in my opinion. Um and some of the reasons why I think that is that when you write Python code you know, there's all kinds of runtime errors that can happen and it's it's the potential that makes things difficult because you can't just wrap everything in a try except block.
- 2:51
You know, you have to be reasonable. Um also try except blocks can be they can ruin your performance if you're not careful with them. I I can't recall exactly how many lines of code um Sovereign is open source, so anyone can see that um and the code in there. I think it might be
- 3:11
I don't know. 10,000 or less. Uh but we wrote extensions and templates that uh were given as configuration to Sovereign and that was where most of the complexity lived. And that that was a the complexity there was essentially
- 3:32
taking uh one format of data putting it into like a essentially a Pydantic model, validating everything, and then having it produce um a lot of other structures and then feeding that into a template. And Python is
- 3:54
not so friendly with uh a lot of iteration in my opinion. Like if you have to run a big complicated for loop, uh it's not so great in Python. And that's essentially what we were doing in those in the in that template code was taking all of the configurations from every uh service that we were running in
- 4:15
Atlassian and producing individual Envoy configuration for for each. And I think we we added quite a lot of validation to make sure that those uh structures came out correct. And that was another thing that sort of
- 4:37
hurt because all of the type checking validation for Python is done at runtime. So it it I think slowed things down quite a lot more um in the for the benefit of safety. Um but I I wouldn't say that these apps were comparable to some really large
- 4:58
Python apps like Bitbucket and Instagram I think is Python or was. I think it's a Django app of some kind. Yeah, I can't say that it's that you shouldn't build something large in Python could because people have clearly been very productive um
- 5:19
and effective at doing that and their apps perform just fine. I just Yeah, I I think I stand by my statement where it just makes things a little bit harder than they should be maybe. Uh I think these days I'd probably recommend Golang. Um that's the safe recommendation in my opinion.
- 5:40
I personally like Rust and um I find that the two relatively similar. Golang to me feels like Rust that just lets you do a little bit more than than what you you can normally do without checking. Like, you can easily create globally
- 6:01
uh shared global a shared mutable global variables and stuff like that. That's really interesting to me because it makes me remember that engineering is is pretty much just trade-offs, I feel. And I think Yeah, you kind of mentioned something similar. And to me, that kind of means that uh unlike like a math problem or
- 6:22
something or even like a very small scoped algorithm where there's like a correct solution, there's not really a correct solution when it comes to building things. I find that uh another part of engineering is about sort of creating patterns and systems where there's just less of a risk for things to go wrong. Even if technically they can go wrong in any sort of situation.
- 6:44
And for example, like uh C++ when you're manually managing memory, um the everybody will say, well, it's a skill issue. Like, I know C++ fans will say it's a skill issue if you're if you have it bugs or something rather than using a language like Rust. Um but at the same time, if there's just a a way of doing things where with a language or
- 7:06
something like TypeScript where you won't have those runtime type errors, it it just seems like it's a no-brainer to kind of structure things in that sort of way. And I feel like this is one of the skills that is so uh so dynamic in that you can kind of apply it to any sort of problem where you're you kind of set up like this system and how you want to architect it in a microservices system,
- 7:28
you like there's a million things to go wrong. So, if you can avoid it, people always say, of course, avoid microservices. Um So, I mean, that's kind of just a bit of rambling, but I'm curious like what your thoughts are on that in terms of just trade-offs and setting up systems in a way where you kind of just there's less of a risk for things to go wrong. So, there's a couple things there.
- 7:50
Uh you So, you Yeah, so about the trade-offs I recently read a book called The Philosophy of Software Design by Professor John Osterhout at Osterhout at Stanford University. And the book goes through these
- 8:11
dichotomies of like for example, general-purpose code is more simple than special-case code, for example. And it's got a lot of dichotomies like that that it goes into. And I felt like that book boil boiled down essentially to when you do have these trade-offs that you have
- 8:32
to consider you're trying to balance them against the thing you're building, like which trade-offs are going to be more preferable. And it's essentially an art. Yeah, it's not scientific. Um and I just think that's that's a a huge part of software engineering, and it was a part that confused me for a long time
- 8:52
because I I felt like when I was making decisions that I couldn't go to my colleagues and say this is objectively what we should do. I had to take my subjective opinion of why it's a good design and then argue for perhaps the trade-offs. Like it does it does this,
- 9:13
but it doesn't do this, and we don't care about the the con, but we do like the the pro. And I I suppose in that way you can try to make it less of an art form and maybe try to logically defend it. But I'm Yeah, I'm still growing in that area. Um that's I see that as the
- 9:35
software architecture track, I suppose. Which I'm kind of interested in. And then you you mentioned a few other things that I I thought was interesting. You said like that you can boil down like if you're using a a language that gives you full autonomy and relies on you having
- 9:56
uh the skill and the care. I think that's it's interesting because there's an element of skill, but there's also an element of cognitive of capacity that I think people don't think about. You can be a perfectly capable person at solving that problem, but if you have like 20 things to do this week and you only have an hour
- 10:17
on you know, next Thursday you've got 1 hour to solve this thing, then your capacity to to solve that is way less. So, if you've got to carefully make sure that all your memory is managed and that you didn't make any logic errors or anything like that, then why not just erase one of those things and work solely on a smaller set of
- 10:37
problems. That's how I uh see that. And yeah, so Python is great for it's it's making trade-offs essentially. You're writing in this uh language that's elegant and productive and tactical, but which lacks this uh long-term protection against maintenance burdens
- 11:00
and like uh ambiguous types and things like that. I think you're making that trade-off for sure.
Description
The Full Episode:
Spotify - https://open.spotify.com/episode/3P1A...
Apple - https://podcasts.apple.com/us/podcast...
You can find Vasilios here:
Youtube - / @vsyrakis
Linkedin - / vsyrakis
Twitch - / vsyrakis
For sponsorship & business inquiries sponsor-neetcode@10xn.dev
🚀 https://neetcode.io/ - A better way to prepare for technical interviews
🧑💼 LinkedIn: / navdeep-singh-3aaa14161
🐦 Twitter: / neetcode1
#coding #leetcode #python
