How to shop for new data tools

It’s pretty difficult to find a tool in the data space that has no competition; the space is inundated with options right now. This can mean there is a lot of noise out there, but it also means you have a good chance of finding the tool that works for you.

Take realtime analytics as an example. If you’re in the market for some realtime analytics, you have a huge glut of options, from stream processors, to Database-as-a-service (DBaaS) to full realtime analytics platforms, such as:

Tinybird - realtime analytics platform (powered by ClickHouse)

StarTree - DBaaS (powered by Apache Pinot)

Imply - DBaaS (powered by Apache Druid)

Firebolt - DBaaS (a custom fork of ClickHouse)

Rockset - realtime analytics platform (proprietary tech ontop of RocksDB)

Decodable - stream processing platform (powered by Apache Flink)

These are some of the hottest vendors & technologies in the realtime analytics space right now. But how do you chose between them?

You of course need to start with your functional requirements, what do you actually need from the tool? Work out what is absolutely necessary before you ever look at tools! It’s way too easy to fall into a feature trap, where your mind is clouded by all of the shiny bells and whistles that a vendor advertises, but you don’t actually need. So, have a pretty clear list of what you actually need before you start assessing tools.

When you’re happy that you know what features you absolutely need to achieve your goals, you can start exploring tools. Try to categorise additional features as “don’t care” and “could be useful later”, but prioritise crossing off your requirements list.

Create a short-list of tools that fit your needs, and feel free to rank it by how well the tool meets you current needs and your vision for the future - but, don’t get too attached to that ranking just yet!

It’s easy to buy based purely on features alone, but this neglects the fact that you, and your team, are going to be spending hours of your lives working with this tool. If using it makes you miserable, the features don’t really matter.

Developers that have been in the game for a while know that a tool that makes your life easier is often worth way more than a tool with millions of features. So, you need to start looking at the tool as if you’re about to commit the next year of your life to looking at it almost every day.

Most vendors have blogs, go take a look at it. You don’t need to read every post, but get an idea of the things they talk about. Is it all features, benchmarks, and competitor comparisons? Or do they also post about common patterns, traps developers fall into, or how they approach Developer Experience? This is a sign that the vendor actually talks to, and learns from, it’s customers. That’s a good sign.

Next, start looking at their documentation pages. Is it a barren wasteland? How well written is it? Do they include screenshots or code snippets? What’s the Doc Search like, can you actually find things?

Pricing tends to be as simple as the products, so look at the Pricing pages. Some vendors have complex pricing models based around how much memory and CPU you need, or making you pick instance types for the underlying hardware.  This is a good indicator for how much complexity you might have to deal with when using the platform.

Start emailing the company and ask them the hardest questions about your problem, and how their product solves it. Give as much detail as you can so that they can actually answer you, but don’t feel like you need to take part in their sales push just yet. See how tough it is to get a technical answer.

Do they let you use the product for free? Many vendors now have free tiers that let you sign up and use the full-featured products (with usage-limits) for free with no credit card. Having these totally free and non-time-limited trials can be a good sign that the vendor is confident in their ability to impress you long term. Time limited trial could mean the vendor is concerned about you discovering the rough-edges; or that their running costs are too high, which could mean they are less incentivised to help you optimise costs long term.

Whether its time limited or not, if a free trial exists, use it and get your team to use it as well. Remember that this tool is going to become part of your life. Experience is often the biggest differentiator between tools. It’s not enough to use it just once, come back to it multiple times over a couple of weeks. Sometimes it only takes 2 weeks for you dread having to go back to it - if that happens, features don’t matter, walk away.

Ultimately, the question is:

Does the tool help you succeed, or get in the way?