Hosting & security

Hosting of applications always raises questions of safety and security, especially in these times where prying eyes are everywhere. In this white paper we give insight in our hosting strategies.

Hosting of normal websites/web applications

Whenever a customer wants us to develop a website/web application, we will develop this on our own machines and use our testing servers for testing purposes. For development and testing we use test data which is defined together with our customers. For example we may use data which is defined for educational purposes.

During development on our own machines the customer cannot gain insight in the progress, therefore we periodically publish our developments on a test server. When agreed in the project contract, our customer can “look over our shoulder” in the testing environment and comment on the progress. Each project can make it’s own agreements on the customer involvement during development.

When developments are done and the website/web application is ready to be made live we either deploy this on:

• our own production VPS (virtual private server) which we hire from a Dutch provider which is specialized in cloud hosting
• a customer’s dedicated server which we define, deploy and maintain.

Our own production VPS's are backed up every night, while the MySQL databases (which contain the customer content) are replicated to 2 owned servers on a live basis, meaning that when a page/record with new content is saved, it is immediately replicated to two other servers that are in our control and our control only. Besides that a full database dump is made every morning to one of our own servers in a 7-days rotation scheme. And on top of that the Linux production VPS is cloned once a week using the cloning mechanism of our cloud hosting provider. We also make a full, ‘bare metal’ backup of the Windows servers every Sunday.

When the application and data are stored on a customer’s dedicated server, we also backup periodically and replicate the databases on a live basis to our controlled servers, thus minimizing loss of service and/or loss of data when failures should occur.

Normally a customer’s dedicated server is also controlled by dedicated people within the customer’s organization. Contractually we agree then that all maintenance is done by Tools for Research and that all actions of others than people of Tools for Research will breach our guarantees for undisturbed service.

Hosting of Session Portals

The hosting of session portals is of another nature, since these portals contain confidential patient/client data and therefore need to be secured even more than our other web applications.

Next to security matters, these Session Portals also contains up to thousands of hours of video material, which needs to be available on short term and which of course must safeguarded against technical failures as well as prying eyes.

We deploy Session Portals using our own servers, since only then we can be sure of the safety and the controlled access.

A session portal consists of three distinct services, running on different servers:

1. Database server with all session and video metadata. This is hosted on a production VPS that we hire from a Dutch hosting provider that is specialized in cloud servers. All databases are replicated to one of our dedicated servers (no VPS, but real machines) in either Amsterdam or Rotterdam. And a full database dump is being made every morning in a 7-days rotation scheme. We call this the "Session Portal Database Server"
2. Primary VPS video server, containing only the most recently uploaded videos. All video files are backed up every night to at least 2 (currently 3) of our own servers. Uploaded videos are removed from the primary video server 2-3 days after they have been uploaded. If a customer wants to view a video that is not on the primary server anymore, the video file will be fetched from one of the backup locations. Sometimes referred to as the "Session Portal Video Player"
3. Storage server(s), containing all videos of the project. The purpose of these servers is predominantly video storage, but in case the primary server fails one of the storage servers can act as database & video server. Some people call this the "Session Portal Video Jukebox"

All transferals of video files between our servers are done over encrypted connections. We are using the following protocols for data transfer: SSH, Rsync, SFTP.

Cloud storage elsewhere

Amazon, Google, Microsoft and others all offer cloud storage for large quantities of data. It is a high competition market, so prices are dropping. If a customer wants to use such a solution for the storage of the video recordings, we can accommodate that. Cloud storage has some advantages, but also some disadvantages:

• Tools for Research/Kleine Stappen has no control over the security measures a cloud provider applies. Usually cloud providers have prior knowledge of security issues before the public disclosure, so you can expect them to maintain high standards. On our own servers, we have installed things like intrusion detection and have tailored the firewall to our needs. This will be a lot harder, if not impossible when using cloud storage.
• Cloud providers sometimes suffer from outages, when their services go down. The worst example is already quite a few years ago when the Amazon storage in North America failed for 3 days in a row. But if you search for ‘cloud outage’ you will see that outages still occur. See this publication for the uptime figures of the cloud storage providers in 2014: 
  http://www.networkworld.com/article/2866950/cloud-computing/which-cloud-providers-had-the-best-uptime-last-year.html 
  Of course our own services also could suffer from incidents. We try to minimize the risks, but cannot guarantee 100% uptime. 
  Moral: make a backup in 2 separate locations, like we do with our own storage server (Amsterdam & Rotterdam). Many of the cloud providers let you choose in which region the storage is located.
• US companies and companies with an US subsidiary (like Google, Amazon, Microsoft, but not TFR) can be obliged to disclose customer data to US authorities. It is not likely that this will happen, but you can never be sure. Cloud providers are fighting this obligation by all means. Just read the first sentences at this blog post by Microsoft’s General Counsel:http://blogs.microsoft.com/on-the-issues/2014/12/08/microsoft-appeal-ponders-u-s-reaction-foreign-data-demand/

Cloud Storage can only be used as alternative for our storage servers (background storage of large amounts of video).

Due to our high security standards and the need for adaptability by us, external cloud storage cannot be used for our session portals & databases. Cloud storage is not an option either for for the streaming video services (primary and fallback). For these services we need our own (or fully controllable by us) servers.

Hosting infrastructure

Currently (2018) Tools for Research/Kleine Stappen hires 3 VPS’s, two of which are equipped with Linux, one with Windows server (primary video server). Besides that we own 3 servers (2 Linux and 1 Windows 2008), which are located in 2 data centers that are 60 kilometers apart.

Security

On all servers firewalls are actively monitoring attempts to gain access and, if needed, block the IP-address of would-be intruders. To give you an impression: there are a couple of hundred attempts to log in to our servers each day. Constantly about 20-30 IP-addresses or IP-ranges are totally blocked by the firewall on each server.

In all decisions we make we put data safety and security in first place. Within the safety and security constraints we try to make the life of our customers and ourselves as easy as possible.

Our service is ‘best-effort’. We cannot guarantee any uptime percentage. Nor can we promise that the content will never be compromised. We are using many industry-standard components. One of these industry-standard components is OpenSSL, the key component for the protection of the https protocol. In 2014 several bugs were detected in OpenSSL. ‘Heartbleed’ was one of them. We patched our servers within 24 hours after the public announcement and carefully examined the log files. No attempts to compromise the content were found. We will do the same at the moment new vulnerabilities are announced.

Except for the reboots needed in cases like ‘Heartbleed’ and the monthly Windows updates we had no downtime in 2014. The Windows updates are always applied between 0:00 AM and 7:00 AM CEST.