A Tale of Five Python SSH Libraries Commentary

The one and only Dinesh Dutt recently wrote a great post titled “A Tale of Five Python SSH Libraries”. Selfishly, this was a particularly interesting post to me because one of those five Python libraries is my library scrapli!

Dinesh’s post is basically about his selection criteria for determining what SSH library to use for his project suzieq, he covers a few primary points:

  • Performance
  • Scale
  • Python 3 Support
  • General Network Device Support

The five libraries referenced by the article title are:

  • paramiko
  • netmiko
  • ssh2-python
  • asyncssh
  • scrapli

He ended up selecting asyncssh as the library of choice for his project. Given what I know about the project and the folks involved, I think this is probably the most correct decision for suzeiq – even if that means not selecting my baby scrapli!

I did have a few feels/thoughts about the article that I thought warranted a follow up post though, so here we are!

Firstly there are a few scrapli things to clear up! Rather than be my normal verbose self, I’ll try to summarize briefly in bullets:

  • scrapli not only supports using paramiko, ssh2-python, and asyncssh as a transport plugin, it also supports (critically! and as the default transport plugin!) the system transport. This system transport plugin essentially wraps your local /bin/ssh and executes commands via this ssh binary – this is not terribly efficient from a CPU perspective (more on that later), however I believe it is incredibly powerful as it automagically provides 100% support for all “normal” OpenSSH directives/config files.
  • The post indicates that scrapli does not provide support for linux devices – scrapli's GenericDriver is actively tested (with a small subset of commands) against linux containers during functional testing.
  • There is a comment that there is no auto-detect mode for scrapli (regarding automatically selecting the appropriate “driver”/“platform” type) – this is a relatively new addition to scrapli, but the Scrapli and AsyncScrapli factories address this by allowing you to provide a device type string (same idea as with ConnectHandler in netmiko).

Before moving on to any of the other thoughts about the testing/requirements I think it is also worth noting that scrapli actually supports three out of four of the libraries being tested! This does of course mean that there is additional overhead with scrapli (hence why it is never faster than asyncssh, which of course just makes sense), however that flexibility to use whatever transport you wish was one of the primary design goals of scrapli. I think this is worth bringing up as one of the complaints (that I 1000% agree with) about ssh2-python was that the api was much more verbose/less friendly than the other tested libraries, but with scrapli you don’t need to worry about the underlying ssh2-python bits, you just write scrapli code. This is really critical in my opinion because as we’ll discuss shortly ssh2-python is obscenely fast…

With all of my personal scrapli bias hopefully out of the way now, there are a few other points I wanted to make more generally with respect to the testing criteria/methodologies.

I know that Kirk Byers has already discussed this with Dinesh and a follow up post may be in the works, however it bears repeating here: comparing asyncio to single threaded things is a bit “unfair”. I put “unfair” in quotes there because it may be actually very fair! Comparing asyncio to connections running concurrently in threads is a much more apples-to-apples comparison… at least from a speed perspective. This however is also sort of unfair as this means that there is of course the overhead of threads and the only way that the synchronous libraries can compete with asyncio libraries is through “cheating” by using threads/processes!

A tricky comparison no doubt!

In general I believe that threads are more approachable for most folks that do not have any background with asynchronous programming. That said, there is of course the downside of additional overhead from threads (or worse (from an overhead perspective), processes) – David Barroso has written about this previously, and it is very much worth a read.

I don’t have enough insight into the suzieq project to have any opinion on whether or not asyncio is “necessary” for that project (though I am 1000% positive Dinesh has thought this through and it is the right choice!), but my general take on asycnio is: if you need it you need it, and if you don’t you don’t (but can still use it if that is your thing), and further that comparing the two is really not particularly useful or productive.

As for scrapli - the API for synchronous and asynchronous is exactly the same. Obviously as a user of the library you would need to write your code a bit differently (await vs “normal” synchronous things), however my hope is that this allows you to test out scrapli very quickly and easily with normal synchronous code, and then move into using it with threads (or perhaps with nornir via the nornir scrapli plugin), or flip over to asyncio and continue using scrapli more or less as you already were! I’m obviously biased, but this flexibility (again with the F word!!) is a core piece of the scrapli library!

Another topic that was not brought up in Dinesh’s post was about actual resources required to run the SSH clients – for most folks this is almost certainly not something to bother with, however when spawning many many many instances, or running things in tiny containers, resources may start to actually matter! In this regard scrapli's default transport (system – the wrapper around /bin/ssh) is almost certainly the worst of the bunch here! This is a trade off that has to be made in order to get that 100% OpenSSH support – for me personally, most of the time this is totally worth it, but is worth knowing.

As far as scrapli's other transports, and the other libraries in that Dinesh covered goes, ssh2-python is almost certainly always going to be the most efficient (as far as the libraries themselves go – if you add threading this may be a different story!). The reason for this is that this library is a very thin wrapper around the libssh2 C library. In the case of scrapli there is of course a small overhead vs using ssh2-python directly, however that should be fairly minimal. Adding threads on top of this will of course add additional overhead… paramiko is fairly efficient with respect to CPU and Memory, and because netmiko is built using paramiko, netmiko is as well.

The resource discussion of course then gets a bit tricky once again when thinking about asyncio. There is of course a benefit to not having to spawn threads, but this is a trade off compared to the simplicity of synchronous programming and threads – no fair way to apples-to-apples compare this in my opinion, just things to think about!

The final two points I wanted to bring up are sort of joined in my mind, and they are documentation and typing. For me these are two of the most important things to look for in a Python library. Typing is obviously a bit of a newer thing in the Python world, and a bit of an additional overhead for library developers/maintainers, however the benefits in IDE auto-completion and in code becoming more explicit and clear (and saving yourself from doing some dumb things due to mypy's constant nagging) is quite nice! While typing does not change anything from a runtime perspective it is something worth considering when evaluating libraries. Once again, selfishly, I have been quite a zealot with respect to typing in scrapli and as such it is fully typed and compliant with mypy “strict” type checking.

Lastly, documentation! Self-explanatory of course! Again, selfishly, I think the docs for scrapli are pretty great :) – asyncssh docs are also very very good.

To sum up – I think the performance comparison in Dinesh’s article is… misleading is the wrong word to use, but perhaps “deceiving” is better(? still not quite right, but hopefully you take my meaning) – in any case I don’t really believe there is any truly fair way to compare asynchronous code with synchronous code – they are simply different things. If there is a follow up article that does a comparison with threads I think that would be just as “deceiving” as the initial comparison.

I would encourage folks to use the most correct paradigm for whatever they are working on, and for the team of folks who will be using it – if threads are maybe a bit more overhead but are plenty fast and easier to write/debug, then that seems like a great option. If concurrent non-blocking things are a requirement then perhaps asyncio is the way to go. I would further encourage folks to seriously take documentation and typing into account when assessing libraries to use as this can be a huge help to speeding up development time.

In the end, I am a huge fan of the asyncssh library and think that Dinesh’s decision to go with that library was a sound one, but I would encourage anyone reading this to take scrapli for a spin – I’m biased but it is pretty great and ticks all the boxes for me!