Proof of concept

Several years ago (I think it was 2002) I was involved in a project to create a Business Intelligence solution for a Short Message Service (SMS) system. 

The solution inserted a copy of the log records from the SMS system to an Oracle database and used the data to create reports for the marketing department of the telecom provider (for instance to find patterns in demographics - when is the time most teenagers use SMS to text to their friends). The system would also be used by the helpdesk of the provider to answer questions of end users (for instance when a text message was sent but not delivered, why was it not delivered?).

The project was already running for a few months and BI specialists were working on data models, reporting, user interfaces and the like. I was asked to look at the infrastructural aspects. One of the first questions I asked was how many log records the system was supposed to insert in the Oracle database and process. The answer was stunning.

10,000 records per second.

The system was supposed to insert 10,000 records in an Oracle database each and every second 24/7. Of course the next question was how they were going to do this. The answer was also stunning.

By inserting them.

The project never had a clue this was quite a challenge. When looking for information on the maximum speed records could be inserted in an Oracle database I found that the maximum speed reported at that time was around 1,000 inserts per second; 10 times too slow for us. I suggested to perform a proof of concept to find out how fast we could insert records in our database setup. The outcome: 500. A bit disappointing and a clear threat to the project.

We solved the problem partly by doing some fancy Oracle tricks and in the end using the same proof of concept setup we reached an acceptable 5,000 inserts per second (I think maybe a world record at the time).

The point I want to make is that apparently the project needed an architect to show them the pitfall of the solution and that a proof of concept was needed to find out how the solution would behave. Such a proof of concept should have been as one of the first things in the project.

I have very good experience with using proof of concepts in projects. A proof of concept should be used to test the most challenging parts of your solution early in the project. This is not a natural thing to do. Most people start with the part of the project they feel most familiar with. The more challenging part usually is addressed at a later stage. But these challenging parts need to be addressed anyway and could lead to a delay in the project or even a halt. A proof of concept shows this at a time not too much money is spent yet and shows the project team and the customer that the project's highest risk is been taken care of.

This entry was posted on Monday 15 March 2010

Earlier articles

The cloud is as insecure as its configuration

Infrastructure as code

My Book

DevOps for infrastructure

Infrastructure as a Service (IaaS)

(Hyper) Converged Infrastructure

Object storage

Software Defined Networking (SDN) and Network Function Virtualization (NFV)

Software Defined Storage (SDS)

What's the point of using Docker containers?

Identity and Access Management

Using user profiles to determine infrastructure load

Public wireless networks

Supercomputer architecture

Desktop virtualization

Stakeholder management

x86 platform architecture

Midrange systems architecture

Mainframe Architecture

Software Defined Data Center - SDDC

The Virtualization Model

What are concurrent users?

Performance and availability monitoring in levels

UX/UI has no business rules

Technical debt: a time related issue

Solution shaping workshops

Architecture life cycle

Project managers and architects

Using ArchiMate for describing infrastructures

Kruchten’s 4+1 views for solution architecture

The SEI stack of solution architecture frameworks

TOGAF and infrastructure architecture

The Zachman framework

An introduction to architecture frameworks

How to handle a Distributed Denial of Service (DDoS) attack

Architecture Principles

Views and viewpoints explained

Stakeholders and their concerns

Skills of a solution architect architect

Solution architects versus enterprise architects

Definition of IT Architecture

What is Big Data?

How to make your IT "Greener"

What is Cloud computing and IaaS?

Purchasing of IT infrastructure technologies and services

IDS/IPS systems

IP Protocol (IPv4) classes and subnets

Infrastructure Architecture - Course materials

Introduction to Bring Your Own Device (BYOD)

IT Infrastructure Architecture model

Fire prevention in the datacenter

Where to build your datacenter

Availability - Fall-back, hot site, warm site

Reliabilty of infrastructure components

Human factors in availability of systems

Business Continuity Management (BCM) and Disaster Recovery Plan (DRP)

Performance - Design for use

Performance concepts - Load balancing

Performance concepts - Scaling

Performance concept - Caching

Perceived performance

Ethical hacking

The first computers

Open group ITAC /Open CA Certification

Sjaak Laan

Recommended links

Ruth Malan
Gaudi site
Esther Barthel's site on virtualization
Eltjo Poort's site on architecture


XML: RSS Feed 
XML: Atom Feed 


The postings on this site are my opinions and do not necessarily represent CGI’s strategies, views or opinions.


Copyright Sjaak Laan