Blog

COVID-19 and its effect on our work

COVID-19 And its effect on our work

This pandemic has been already quite tough on our services and our projects. The extra effort required  due to the huge increase in support requests has had a huge impact on our product release as well as customer project delays.
Network capacity wise (61%) we have managed to cope very easily since we were under-utilising our infrastructure after our last upgrade in August of 2019, our server capacity is now at 86% utilisation and we can afford to wait a little more before requiring new equipment.
I will break up individual projects and issues we are having currently in the following:
Relevant Links

First pandemic for us, we will have something here for the next one.

Customer Support

We have always been very proud on the speed of which we resolve support tickets and requests for all of our customers. Since 2009 we have been proposing to customers to never buy our expedited ticket option since the difference in time to solution was less than 5 minutes.
At the start of the pandemic a few clients chose to use the expedited support option to get faster response but this had the effect of extremely long times for free support customers.
In order to flatten the response times we eliminated the option of expedited support for all customers and we fully refunded even used tickets to all customers.
I believe this is the best option for everyone since it allows us to set ticket importance according to actual customer needs and everyone receives the same amount of support from us.
Why is this happening?
It is pretty simple, everyone is at home, they have less things to do and they instead work on their online presence. It is interesting to see that web shops that would get a few updates per year before, now get updated as many times per day.
The avg rate of web shop updates/changes has gone from 6.3 per year to 37.6 per day.

For comparison:

2020
15th of Jan - 15th of Feb:
AVG Tickets per day: 73 15th of Feb - 15th of Mar:
AVG Tickets per day: 618 15th of Mar - 15th of Apr:
AVG Tickets per day: 802

Halcyon Engine

Unfortunately our most important project will be delayed and we do not have an accurate estimate anymore, we hope it will be ready by October 2020 but we cant set it in stone yet. The main issue is lack of development time due to the support requests.
How will this affect you our customer?
Technically not at all, the release is only bringing in new functionality, any security issues are backported in our previous release and we do not have any outstanding known issue.

Reseller One

The delay in providing this release is the worst result we could have had. The new management features are extremely important to our clients and you have been expecting them since February 2020. We are still fully committed in releasing ASAP but we are already 2 months over the delivery time and we expect this to be delayed at least 45 more days.

How can we fix this?

The easy answer would be to hire more people. But this is not a good solution
Even though our requirements have increased to our current customers and we plan to keep offering free support instead of charging for it, our income has not increased proportionately.
New purchases from companies have dropped to a trickle, if we exclude the new web shops getting made at an increased pace, everything else has been paused, spending is now at its lowest point ever, which forces us to pause spending too due to the uncertainty the next 16-24 months bring.
It is not that we do not have a perfect solution at this time, we do not have any solution, but we do the best we can under the circumstances.

Our plans for the future

We are extremely lucky that we have done our infrastructure update last year, and most importantly that we overdid it by quite a lot. If I could plan last years update with what I know now I would easily have doubled down even more on every part, man power, servers and network capacity.
Why? Because of the costs involved now, everything is more expensive by up to 15% and we expect it to go up to 22%, everything takes at least double the time to be delivered and integrated/installed, that cost can only be reflected to the final pricing to the customer who at this point either does not want to increase spending because of fear or due to lack of funding.
At this rate we will be able to add customers and services without issues till November 2020, at that point we will be up to server capacity, we already calculate a drop of 9% on on-boarding of new customers. We have a server order that is to be delivered on June 2020 that has been order and paid for already but the provider has informed us that we should expect it at early August instead.
How will this affect you? It shouldn't, we are a private owned company, we are profitable since 2012 and we own zero debt inducing assets as well as zero loans. We have the capability to keep fully functioning for many many years without any issues with up to 36% of our customer base (assuming the same costs as now).

We hope this pandemic will make us all collectively stronger and smarter in the future.
Best Regards,
George Pantazis
eSGR CTO

Next Generation of Halcyon

Next Generation of Halcyon

The last 2.5 years we have been working on our new Cloud Hosting platform. We have done extensive testing of the whole stack and today we are starting deployment of the service to our fleet. We expect to have the new platform up and running by mid-November 2019 and we will start moving clients to it after the 2019 holiday season.

What you as a customer should expect from this new platform? More control of your information, a significant change in the elasticity of our network, 50% reduction of planned downtime and instant resource allocation.
Relevant Links

How ready is it?

The new platform has been tested thoroughly by eSGR and all of its services are hosted on it for the last 14 months. Additionally, we have provided the service to select hosting providers we host that requested access to it since April 2019 and we even created two products on the end of July (Reseller One, Provider One) that has been utilized by clients already.

Feature Parity

The platform has every single feature you currently have with the addition of a huge amount of friendly and easy to implement additions. We are introducing a new mechanism that will allow you to enable features on your hosting plan using our custom panel as well as automate hundreds of functions.

GUI

We have designed a very simplified panel and tried to hide all the complexity from our panel, at the same time we wanted to add a lot more features. Our solution is to provide you by default all standard features in the panel and just hide the ones you haven't used on our current panel. At any time you can switch to the advanced feature set and just select the additional modules that you need or just stay in advanced mode.

More control of your Data

In our current platform, your data is located "somewhere" in our network, each block you utilise is stored in 3 different storage nodes on our network. In the new platform, you can choose in real-time the nodes, move your data locally to your node or even just attach it directly to your instance. At the same time, we are providing is the ability to snapshot at any time part of your data but it is not limited to your VPS, it includes DB instances, single files or even just selected blocks that you own.
During any of those events, you do not lose access to your data and there is no downtime to your services.

Network Elasticity

As of now, you have been automatically using our CDN services to offer local caching for your customers in other regions. In the next iteration of Halcyon, you can choose to instantly create local instances of your whole stack on any of our locations with zero downtime, during the process your clients will still be served using our CDN.

Reduced Planned Downtime

Currently, we have between 5 and 15 minutes per month were we have to pause services for planned updates, not including any security-related issues that have to be patched in between. Our new platform allows us to migrate instances in real-time without downtime by moving your service to an already patched node. On our current testing, we have managed to reduce planned downtime by over 50%.

Instance Resource Allocation

We have been extremely careful in providing extra resources to our clients when they needed due to extra traffic on their network. Our current solution allows a client to bubble inside their plan for up to 15% their bandwidth for up to two months without any additional costs. This procedure although automated it still requires users to lose valuable time during a hectic period and it takes up to two hours. Our new platform, will offer instant resource allocation doubling for up to a month and also a new addon that allows the client to prepurchase extra bandwidth at any time of their plan lifetime that will only be used during periods of need.

Next Steps

After the 2019 holiday season, we will start migrating all clients on our Indy plan to the new platform, immediately after SOHO and Standard clients will move. Anything above standard will have a longer moving period since those accounts will move to individual instances with dedicated resources.
Clients will be notified and have the option of staying in our current platform for up to their current plan expiration. If they would like to stay on the legacy plan after their contract is up they will have additional costs added to their hosting plan. Halcyon engine v6.4 will receive security patches until December 2028.

PHP 5.6, 7.1

The minimum PHP version for Halcyon v6.5+ is PHP 7.2 . PHP 7.1 will be using our compat layer (web sites will be slower).
PHP 5.6 is NOT supported. On February 2020 clients using PHP 5.6 will be charged to keep running their older code even on the Legacy platform.

Our plans for the future

The whole mechanism we have designed for our new platform allows us to slowly move to a dedicated instance per client. Each client has more dedicated resources and is slowly getting fully compartmentalised. We plan to move to this direction for every client on our platform. At this time any customer on Biz and up will be having effectively their own unique instance of the resources they are allocated, providing better performance and better security at the same cost.
We will continue adding functionality and features on our platform but always in a way that will not require our clients to get a CS degree to host a fast and reliable website or application.

PS: If you are one of those interested in running your Golang, Rust, Node JS, Python services in the same easy way you host a PHP website on our servers stay in touch ;)
Best Regards,
George Pantazis
eSGR CTO

Plan Upgrades (Free)

Plan Upgrades (Free)

Since May (2019/05/12) we have introduced upgrades to all of our plans but we did it silently since we wanted to check how they would perform.
Since our inception in 2009 we have been doing bandwidth and storage upgrades to all of our plans each year whithout increasing our pricing with the goal of keeping our older clients on the same plan they originaly signed up for.
For the first 7 years we used a formula that increased the available bandwidth based upon the 97th percentile of customer bandwidth consumption on the same plan and it was working relevantly ok since we never had more than 3-5% of customers needing to upgrade to the next plan.
In early 2017 we noticed that a big percentage of our customers have had huge spikes of traffic that slowly turned to a standard stream of more bandwidth consumption near mid-year.
After analysing the traffic we noticed it was a higher number of visitors in 99%+ of the cases and not just heavier site content thus for our free upgrade in 2018 we had to change our algorithm to account for that. Unfortunately even after our upgrades we noticed a lot of customers were month after month very close to their limits and more than a few had to use our Elastic Expansion service quite often.
After talking with a lot of those customers we realised that this was creating two issues to them.
Firstly, they were on a constant mode of anxiety, monitoring traffic hourly in fear of loosing access and asking for an upgrade even though our support told them there is no reason.
Secondly, started trying to "optimize" their sites to make them "lighter" by removing functionality or even hiding/removing relevant content which obviously would impact sales.

Elastic Expansion

This is a free automated service we provide that allows your account to bloat by up to 15% on Bandwidth and Storage for up to 2 consecutive months without requiring an upgrade to a bigger plan.

As long as after the 2 month period your account returns to its normal usage then the system will not offer/require an upgrade. The service can be re-used after 6 months.

Our Solution

In early 2019 we did our normal plan upgrades as ususal but then started working out on how much data would be fare for the amount of money we ask per client regarding to our costs per client.
Our original plan was to update our algorithm one more time and account for the avg extra bandwidth which would be a solution we tried before and kinda worked. But our last 2 year data showed that by the end of 2019 we would be in the same possition which would just delay the customer anxiety to the end of the year. In the end the decision was made to move each bandwidth tier to one slot on the right and then double the bandwidth.

In the following graph you can see how our most purchased plan (Indy) was affected after the plan upgrades:

Bandwidth

2018 2018
Early 2019 Early 2019
May 2019 May 2019
Bandwidth is now 285.7% more.

Storage

2018 2018
Early 2019 Early 2019
May 2019 May 2019
Storage is now 200% larger.

Our plans for the future

We will continue providing free bandwidth upgrades each year in the future as we have been doing but from now the plans will be getting their upgrade in May instead of the end of the current year, so we can have a better idea of current bandwidth requirements.

But we will also be introducing some changes to our plans:
  • Indy limited to 1 Database

    Effective 2020/01/01

    A lot of our customers have been hosting more than one dynamic websites on our smaller plan due to the extra bandwidth and storage. The extra database was never intended for this use, it was so that the customer can host a supporting service to their website.
    Current customers will keep the 2 databases on their accounts till 2025.
  • Application for extra database

    EFFECTIVE 2020/01/01

    We will be providing a form for customers to request a database on Indy for use on a service that is not a website or public accessible service. This will be free of charge and will be monitored for access, if the DB is accessed for a non allowed use case it will be terminated immediately.
Best Regards,
George Pantazis
eSGR CTO

eSGR Services in Australia

eSGR Services in Australia

Last December, the Australian Parliament passed a law requiring that all tech companies that use/develop software utilising encryption need to add backdoors to their software/services and provide access to law enforcement.

This has created quite a few issues for us, some technical but mostly ethical ones, I will try to explain the issues and what we have been working to mitigate them as best as possible.
Electronic Frontier Foundation Articles

Technical Issues

We have been proudly providing secure and encrypted services to our clients since 2009, our full cloud stack is communicating internally over fully encrypted channels to provide integrity and security to your data. In order to accomodate the Australian law we would have to:
  • Stop syncing data between all other regions and Australia (that includes WAF and ACLs), block customer data from moving between zones and terminate out of country backup services for Australian customers.
  • Create another instance of Halcyon, Coeus and Soteria (Mnemosyne can switch to the standard mariadb protocol which would make its traffic unencrypted if desired), without any encryption (we refuse to provide a backdoor in any case).
  • Shield our own services in Australia from contacting any of our servers in other countries since we would have no control on what the Australian secret services would try to do.

Ethical Issues

We believe that removing/disabling encryption is not only unsafe for our clients but also completely unethical, nobody should have access to a citizens personal information without proper cause. It is 100% certain that other 3rd parties will get access to the same data by identifying and exploiting the backdoor or even malicious actors from within the same security services could/would distribute the information for profit or other nefarious uses.

Our "Solution"

Since 12th of december we have started working on moving infrastructure to New Zealand, a full stack has been fully functional since 16th of December. On 23rd of December we contacted all of our Australian clients explaining to them the issue and that we can move their services, by 28th of December all of out clients had been notified and informed of the issue and chose to move or remain. On the 4th of January we started moving the vast majority of clients that chose to move and the last one was moved on 7th of January.

On 13th of January we started moving the few remaining clients to an optimized stack of LNMP, by the 17th of January no software we ever wrote was residing in Australia anymore.

As explained to our remaining customers in the region, we are in the process of deciding if we will continue services in the country even in the current state, every customer had/has the option of getting a full refund and move to another host in the country at any point.

Issues with our Solution

Due to quite technical issues although we host clients in NZ, the AU government could still impact the services we and others offer in the future.

We have been working on finding a solution in case that event occurs.

Our plans for the future

As of today our current plan is to keep the current service until the last customers contract is finished, we will not be renewing any contracts in the region. I would like to mention that this isn't a political decision and we are too small of a company (especially on that region) to exert any political power in any case, it is an ethical decision based on what we have promised to provide as service.
Best Regards,
George Pantazis
eSGR CTO

Storage & Bandwidth Upgrades for Presence, Indy, Standard and SOHO.

After our larger plan upgrades almost 2 months ago we will be officially upgrading our smaller plans too including an upgrade for Presence for the first time in 3 years.

Completely refactored our News section

Due to the heavy work load the last year we have completely ignored our news section. Since our last blog post on July 2017 we have completely switched to posting everything in our announcement section which was technically the only news source that was properly functioning.

Halcyon v5.6 Release

We are proud to release Halcyon v5.6, this release marks the complete separation of function from Coeus and Sotetria. Now all SQL caching requests are served from Coeus and Halcyon is always proxied by Soteria.

2017 Update: Go, NodeJS, MS SQL, PHP Composer, Dynamic DNS Updates

In this post I will be updating you on some of the most important planned changes for the rest of the year and including 2017. Be warned that this post is mostly directed to developers and IT departments hosting or are planning to host with us.

Guides for compiling and installing server software

Since 2010 we have been publishing various guides for compiling software and installing/configuring server software in our knowledge base. These have been gathering quite a lot of traffic and we used them to point our customers to during support.

HTTP/2 Support

After almost 16 years a new http protocol version has been defined and its name HTTP 2.0 (draft RFC, as of this time at rev 17). HTTP/2 has its origins in the SPDY protocol proposed and originally implemented by google, which we have supported and provided to our customers since its official release up to now.

Annual Safety Announcement 2015-01

Dear friends,
as you have noticed all these years we are very frugal in sending you emails, we believe our place of work is to just provide you with a service and stay out of your way. We do tent to continue this trend but we would like to make you aware of something important and its about the security of your services.

Premium SSD Hosting Plans

eSG has been providing speed optimized services for more than 6 years now, we have been implementing numerous tools to provide our customers with the most advance caching and optimization tools available in the market.
During that time we have been providing a hybrid SSD cache on all of our servers that could deliver 45% and better I/O on every type of site, in case of heavily I/O bound sites the boost can go up to 80%.

World IPv6 Launch

eSG published its AAAA record for its portal.
Copyright © 2009-2022
eServices Greece