Fixing stuff on live server is wrong? Not always.

Controversial topic I want to talk about. Among web-developers, there’s a rule of processes how things should be pushed to live version – they need to go through local testing, then staging server and only then deployed to live. Same even with production bugs – you need to reproduce them locally and only then properly fix and deploy. I believed that too, until started our QuickAdmin. Here’s our story.

Our customer support reality

Example from real life – for example, there is a production bug, it appears in our BugSnag and Slack channel, Trello card is automatically created, alarm mode on. We need to fix it asap. But that’s only the beginning of the story.

Then a user goes into our live-chat support box and raises question, something like:

“I’ve tried your service and encountered an error, what can you/I do that I could resume testing and potentially buy premium”

Now this is where REAL alarm-mode is on. There is a real person who potentially wants to give you money and it depends on your customer support abilities whether they will convert into customer.

And you know what’s the most important – it’s all about TIME. It’s not about whether you would fix the bug – it’s about HOW FAST. Preferably in minutes – within the timeframe when they are still online and be able to resume using your service immediately. If not, then they’re gone and it’s really hard to get them back.

So, in real business world you should do whatever it takes to keep them as customer, for business it’s more important than code integrity, version control workflow and other technical stuff.

Developer saying “we cannot push to production cause we need to go through all the testing and it will take a few days” actually is saying “we cannot take the money that we’ve been offered”.

In our QuickAdmin journey, we’ve had potential customers like this, and sometimes we had to go into – oh my God – production database and change some values manually. Another case – SSH into a production server and run some command there to regenerate some files directly on production.

Experienced Senior developers would send us directly to hell with one-way ticket. And they would be partly right. But you know what.. they are not our customers! And our primary goal is to make our customers happy.

Imagine a dialog which pretty much happened:

– Hi, how can I help you?
– Yes, I’ve got an error on this page and I cannot proceed.
– Let me take a look.. (5 minutes later) Ok, problem solved, you can proceed now.
– Wow, great customer support, thank you! And awesome tool, by the way!

Person on the other end didn’t even know that I actually deleted one row from the database which failed to generate CRUD. They don’t need to know. All they care about is they’ve been served!

It’s not about technology

Let’s compare it to something from non-IT world – let’s say, McDonald’s. Imagine you go there, order a Chicken Burger and for some reason you get a Cheese Burger. What would you say?

– Sorry, you gave me wrong burger, I order a Chicken one.

So imagine in this situation the answer would be:

– Oh, I apologize, we need to fix the process bug then, count the amount of chicken left, restart the process and then will change the burger for you.
– What? How much would it take?
– I don’t know. We will send you an email when we’re ready.

Absurd, right?

But that’s exactly how high are the standards of customer service these days. People expect to be served in minutes, or even seconds, especially online where it seems that everything happens “by magic” and instantly.

Technical debt still need to be paid

Now, there’s another side of the story.

Developer workflow rules still need to be followed. Just after the customer is satisfied. Developers STILL need to reproduce the bug locally, fix it and run all necessary tests, merge and fix code conflicts between live and staging environments if needed etc.

I’m not saying that process is wrong or unnecessary. It’s the basis of all quality software and, as a developer, I understand its impact on the technical delivery.

But as a now-CEO and a customer support person for a small startup like QuickAdmin, I care more about customers and their happiness. And flawless experience.

Even if it looks like this:

And if for that I need to break or bend some tech-rules, so be it – I hope I won’t burn in hell for that.